Archive for January 2010

Server Outages

It’s not my fault! Yes, I did kill a few sites last week, and I did choose not to recover all of them. But the issue with several of my sites (ccphillips.net, archcon.archlinux.ca, schwag.archlinux.ca) being out all weekend was not my fault. That was caused by a MySQL upgrade on my shared webhost. The Python MySQL connector was unable to find the new library, and thus, all Python DB-driven sites on that host were hosed.
“Ok, so why didn’t you just fix it instead of leaving it out all weekend?” you ask.

I’d like to have a story like Judd, Aaron, or Dale used to use, “I was out camping/drinking/enjoying myself, and that was the weekend the server chose to go down.”

The sad truth is, I was stuck in the winter’s worst blizzard. While the weather dumped two feet of snow on us, drifted into 11 foot banks, my satellite internet connection was mostly down, and my power was off as much as it was on, all weekend. I wasn’t able to get a ssh session to last long enough to track down a fix for the issue.

storm 3

I apologize for the inconvenience, but there was nothing I could do but sit tight and wait the storm out.

Death of Arch Bounty

Last September I posted about Arch Bounty, a project I’d written to allow people to post ‘bounties’ to have specific ArchLinux bugs fixed. I didn’t promote it and interest faded quickly. I’ve been thinking of pulling the plug on it for a while now, but it happened unexpectedly yesterday when I accidentally killed half a dozen of the sites on my webhost. I’ve recovered most of them, but I decided that ArchBounty won’t be coming back.

There was one donation to the project; I will be forwarding it directly to the Arch Linux Donations fund.

A Python 3 Powered Blog

Last week, I posted my intent to port the Tornado web framework to Python 3. Not only have I done that (sort of), but I’ve hacked it to pieces; soon it will be unrecognizable to the original developers, and possibly, to me!

It didn’t take long to get the Hello World example included with Tornado running. Quite a bit more work was the blog demo, but I now have the template, auth, and httpclient modules working, along with the core modules for a working async server. I was able to log into my example blog with my google account, compose some entries, view, and edit them, including the feed.

That doesn’t sound like much if you’re coding blog software for an existing framework (10 minutes in django, probably about 12 in web.py). But if you’re coding a web framework for existing blog software, it’s an accomplishment. I’m proud to have a “working” framework (even though it’s full of bugs and working just means “those things I’ve fixed”) for Python 3 in such a short amount of time.

I’ve named the project psyclone, a play on ‘tornado’ –> ‘cyclone,’ and the fact that Python projects oughta have a ‘p’ and ‘y’ in them somewhere. The code is on github for all to play with. Patches welcome! :-)

I’m having a lot of fun with this project, so it’s taking more of my time than I ought to be devoting to it… on the positive side, it’s progressing rapidly!

My plans:

  • Go over the existing code and improve some of the rather messy unicode/str –> str/unicode hacks I had to make to get it working.
  • Write some tests. (The Tornado team seems not to value tests.) I’ll use py.test and may need to write a test client.
  • Write a session framework and auth framework; the current auth framework uses openID only; but I like a more local solution to be available as well.
  • Consider writing an ORM. Likely, I’ll discard this idea, arguing that Judd was right to design a frameworkwith SQL only. The truth behind the argument will be laziness, of course.

Arch Linux Pens

I’d like to announce the arrival of Arch Linux Pens to the Schwag shop. These are nicer than I imagined, very opulent’ I’m very happy with them and hope you will be too.

Pens are deep blue and gold with “Arch Linux” and “www.archlinux.org” engraved on them. They have a soft black grip and fine ball-point black ink. They are available for $5 individually, or as low as $3.50 in bulk.

Order them now from http://schwag.archlinux.ca/product/pen/.

pens

The Utility Of Python Coroutines

Coroutines are a mysterious aspect of the Python programming language that many programmers don’t understand. When the first came out I thought, “Cool, now you can send values into generators to reset the sequence… when would I use that?” The examples in most books and tutorials are academic and unhelpful.

Last year, I attended David Beazley’s course A Curious Course On Coroutines along with a fellow Archer. We agreed that it was an exceptionally interesting course (Beazley built an OS scheduler in Python, with just a minimal amount of code: how cool is that), but that we didn’t see any practical application of it in our regular work.

Yesterday, I started working with the Tornado code to port it to Python 3. Tornado uses an async framework; I hate async because I hate working with code like this:

def somemethod(self):
    #
    self.stream.read_until("\r\n\r\n", self.callback)
 
def self.callback(self, content):
    # handle content read from the stream

I understand the utility of this code; while the stream is being read, the app can take care of other stuff, like accepting new connections, until the stream has been read. You receive high speed concurrency without the overhead of threads, or the confusion of GIL. When the read is complete, it calls the callback function. It makes perfect sense, but when you read code with a lot of such callbacks, you’re constantly trying to figure out where the code went next.

In my mind, the above code is really saying:

def somemethod(self):
    #
    self.stream.read_until("\r\n\r\n")
    # give up the CPU to let other stuff happen
    # but let me know as soon as the stream has finished reading
    # handle content read from the stream

I find this paradigm much easier to read; everything I want to do surrounding content is in one place. After pondering different ways to write a language in which this was possible, it hit me that this is what coroutines are for, and it’s possible in my preferred language.

Because coroutines use generator syntax, I thought they had something to do with iterators. They don’t, really. The above code can be written like so:

def somemethod(self):
    #
    self.stream.read_until("\r\n\r\n")
    content = (yield)
    # handle the content

The calling code would call somemethod() and somemethod().next(), and eventually, when content is available, somemethod().send(content) to drive it.

A generator compiles to an object with an iterator interface. The coroutine above (sort of, but not really, at all) compiles to a function with a callback interface (you could say it is an iterator over callbacks). You can use yield multiple times in one method to receive more data (or to send it; put the value on the right side of yield, like in a generator).

The mainloop that called this code would still be at least as complicated to read as it is using a callback syntax, but the objects on the async loop are now much easier to read.

This paradigm has been implemented in the Diesel web framework. I’ve looked at it before and thought it was an extremely bizarre way to design a web framework. I still do, but now I understand what their goals were. If you’ve ever struggled with the, “why would I ever use this?” question when it comes to coroutines, now you understand too.

I have no immediate plans to rewrite my tornado port using coroutines, but maybe someday if I’m bored, I’ll give it a try.

Python 3 Web Framework

I got it in my head this weekend that it was about time someone wrote a web framework for Python 3. My head is kind of stubborn about these things, so I asked it some questions:

Does the world need another web framework?
Do I need another web framework?
Do I have time to do this?

The answers were all “no.” Still, I’m planning to go ahead with it until I get bored. Then the project can sit and collect dust with all my others.

A bit of discussion with The Cactus, led to a few ideas:

I discovered that QP is apparently the “first Python-3 enabled web framework.” I didn’t try it, so I was perhaps unfair in discarding it, but it doesn’t look… suitable.

I looked around some more, and found that CherryPy is about to release a Python 3 enabled version. I’m sure that will spawn a whole slough of Python 3 frameworks built around CherryPy. I considered such a plan (I’d call it ChokeCherryPy based on a receipe my mom devised): create some kind of templating system based on str.format, some session support, and some kind of database module wrapped around py-postgresql. Could be fun. But I’d end up with a mess of third-party technologies much like TurboGears, and that would be embarrassing, plus I’m sure the TG team already has people working on this.

Then I came back to my original plan, which was to port either Tornado or web.py to Python 3. Tornado looks like a smaller codebase (easier to port) and I’ve never used it before, so it’s also a chance to learn something new. So today I forked Tornado on github and run 2to3 on it. I’ve already got the “hello world” demo running; it wasn’t too hard once I figured out the difference between bytes and strings. At least, I think I did that part correctly.

The project is named psyclone, a little play on the ‘destructive weather patterns’ genre. I was close to p3clone, but it’s too hard to convince people it should be pronounced ‘cyclone’.

This isn’t a project I expect to go anywhere; django will be ported to Python 3 soon enough, and other frameworks will be popping up all over. But I’ve been working with Python 3 a lot lately, and I thought it was time to tackle the ‘scary’ job of porting an existing app. It’s tedious, but not difficult.

Copyright Dichotomy

In the so-called “copyright wars,” we see a spectrum having the MPAA, RIAA, Jack Valenti, and “all rights reserved” on one side, with the Pirate Parties, Pirate Bay, Rick Falkvinge, and “no rights reserved” on the other side. In the middle, we have Creative Commons, Lawrence Lessig, and “some rights reserved”.

I’d like to momentarily expand this line to one that places “no rights reserved” in the middle, in a way that shifts Lessig closer to Valenti, and opens up a whole new area of creative exploration beyond the pirates, who are no longer extremists.

First, a disclaimer: I don’t claim to have any answers. I don’t even believe what I’m suggesting is the right path. I am simply suggesting an idea that frames a long-standing and long-term discussion in a different light.

The spectrum above defines the opposite of a right as “the absense of a right.” This only goes halfway. The opposite of a right is a responsibility.

Image, for a moment, a society where there is no such thing as, “the right to my creation,” but there is a massive, “responsibility to create.” In this society, people would have free access to all the materials of the world, all the patents, blueprints, and software, all the films, songs, and books, all the photos, paintings, and sketches the world has ever seen. In exchange for this free access, individuals would be required (responsible) to create a certain amount of new material every year. Some of this material would be innovative and fresh, some would be a new presentation of old stories and ideas, some of it would be interpretations of those old stories in new media. We’d see new designs for existing products, we’d see new products that merge old technologies. We’d see Android phones with iphone gestures, and we’d see Mickey Mouse saving Princess Peach from the evil Bowser the Hedgehog.

Such a world may excite some, bore others, and scare many. Would these same people be less excited, bored, or scared by the Pirate Party? by Creative Commons? Maybe those deals aren’t so bad after all (to those demanding rights)… or maybe they aren’t so good (to the promoters of creativity).

This responsibility to create idea seems radical in the context of entertainment media, but it is not new. It’s a long-standing scientific tradition, best encompassed by Newton’s overused quote about giants. Academics have “free” access to the entire compendium of academic knowledge; in exchange for this access, they are expected (responsible) to generate new ideas and innovations. Some are good and some are bad, but if a scientist neglects to publish a few new papers a year, they fade from the academic community.

This idea is also an unofficial motivator in open source communities. Within the Arch Linux community, my home, I’ve made some effort recently to verbalize this norm. The story goes thus: Arch Linux has had contributions from many thousands of users. Each of us that uses the distribution is somehow indebted to all those other users. Further, we can never, as individuals, pay off the debt in its entirety. Even the well-known user with 8000 posts on the forum, thousands of package updates to his name, and dozens of Arch Linux tools under his belt has contributed but a drop in the bucket compared to the efforts of the entire community. And Aaron is aware of this debt. So should we all be.

Yes, in the academic and open source world, the implied responsibility to create is known to work. Creativity in both worlds spreads more quickly than anywhere else. Compare to the communities creating ideas whose soul purpose is entertainment. Even the liberated Jamendo is mired way over in the (Some) Rights Reserved end of the scale.

SimpleHTTPServer in Python 3

If you’ve been doing any testing of client code that uses urllib or httplib, you probably know about this command:

python -m SimpleHTTPServer

This starts a very simple server in the current working directory; it serves all files from that directory, and is, quite simply, the quickest way to get something set up if you want to test some kind of web parsing or client code. (It’s also handy if you want to fire up a server to easily share files from your hard drive for a few minutes).

SimpleHTTPServer has been merged with BaseHttpServer into the http.server package in Python 3. I couldn’t easily find documentation for the new command, and ended up writing the following simple code:

from http.server import HTTPServer, SimpleHTTPRequestHandler
 
httpd = HTTPServer(('127.0.0.1', 8000), SimpleHTTPRequestHandler)
httpd.serve_forever()

Then I did a bit more digging around and realized that this command does what the old one did.

python3 -m http.server

My code performs a little differently (it only serves on the localhost interface), but if anyone is looking for the old SimpleHTTPServer command line, there you have it.

The http.server module is normally supposed to be as a base for creating more complicated server environments (see your favourite web framework, for example), but the fact that it can be executed directly has a great deal of utility as well.

By the way, if you didn’t know about SimpleHTTPServer, you might also be interested in the built-in smtpd server as well. I use this command frequently:

python -m smtpd -n -c DebuggingServer localhost:2525

This runs a simple smtp server on the given interface and port, and outputs all mail sent to that port to the console. It is very useful for testing and debugging web-based send-mail forms and such. You can, of course, run a standard smtpd server by not passing the -c DebuggingServer.

Fixing Git Bash Completion

I didn’t know until yesterday about the __git_ps1 command. You can include it in your bash PS1 like this:

PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '

and whenever you’re in a git directory, it will include the current branch in your prompt, along with a few other goodies.

I did this and it didn’t work. It just displayed __git_ps1 in my prompt all the time, which is ugly and not terribly useful.

I couldn’t find an answer on Google, so I ended up just disabling lines in my .bashrc until I could figure out what was wrong. I ended up having to disable this line:

shopt -u promptvars

I don’t know why it was on; perhaps I had a reason for it once and then copied the bashrc from computer to computer, but it’s gone now and my git bash prompt works.

So if you’ve recently heard about __git_ps1 and it’s not working for you, look for the promptvars shopt.

Intellectually Dispossessed

Ursula K. Le Guin is, or had been, one of my favourite authors. In 1974, she published an excellent thought experiment, set in a science fiction setting, titled, “The Dispossessed.” The book discusses a group of people who built a culture and society around the idea of non-possession; nothing belonged to anyone. People lived in whichever house was vacant, people worked together to feed and shelter themselves. Their language did not include concepts of “my” or “mine,” and their children were raised by the community at large.

In some ways, “The Dispossessed” picks up where Richard Stallman’s Short Story, The Right To Read, published 23 years later, left off. The similarity is striking, yet the current stance of the two authors is startlingly different.

“The Dispossessed,” was a masterpiece, yet it is only one of several books Le Guin has written that seem to support cultures of freedom and creativity. I always believed this author was one who supported freedom and creativity.

Apparently, her works are fiction after all.

In December, 2009 Ursula K. Le Guin resigned from the Author’s Guild due to their settlement with Google on their book scanning policies.

I question how a woman who so clearly understood and documented the benefits of “dispossession” could now be in favour of intellectual property and copyrights. How could she write such an innovative novel, one that she apparently believed in, and yet, now that the world she describes is within reach, she fights it?

Yes, the culture described in Le Guin’s 1970s-era book is similar to a culture the open source and creative commons movements are now so effectively living. Her dream, nearly forty years later, is now becoming a reality.

I’m not sure what has changed in the decades since The Dispossessed was originally written and published, but I would like Ms. Le Guin to reconsider her stance, to study these new movements. Please, ask Lawrence Lessig to explain his views. Most importantly, I sincerely encourage her to publish her next work under a creative commons license. I think she’ll find that she will profit, rather than lose, from such a venture.