Dusty's Diverse Domain

For those that don’t know, the first ever ArchCon will be held in Toronto, Ontario, Canada on July 22 and 23. Ralvez and I are the primary organizers. We’ve got some great presentations lined up and are becoming more confident that we’ll have enough attendees to fund conference activities.

The earlybird registration deadline is fast approaching; you have four days to register for only $100. Register at the registration page: http://archcon.archlinux.ca/

For more information, visit the ArchCon website: http://www.archlinux.ca/archcon2010/

Dieter has set up a couple of wiki pages to help people collaboratively plan the conference, and more importantly, social and vacation activities around the conference. He’s coming all the way from Belgium, and is looking for other Archers to do some road trips and tourist attractions with while in North America.

http://wiki.archlinux.org/index.php/Archcon#Archcon_2010
http://wiki.archlinux.org/index.php/Archcon_extra_activities

After far too many delays, I’ve finally shipped all the outstanding case badge orders. The badges arrived today, after a delay in production and mailing, and all envelopes have been packaged; they’ll be in the mail tomorrow. I spent a lot of time stuffing envelopes this evening!

I would like to apologize to everyone who has been waiting for badges; the preorder form has been up since early December. I originally said they’d be shipping in early January, and a 1.5 month delay is truly unacceptable. Thanks to everyone for their patience; I will try to perform better in the future. I’ve certainly learned not to rely on shipping estimates!

The new badges are a more modern looking than the old ones; I’ll try to update the pictures on http://schwag.archlinux.ca/ tomorrow.

Just because something can be validated does not mean it should be validated. It’s very easy to validate form fields in django and most other web frameworks. That does not mean we should always take advantage of this feature.

Why did I just get this error when creating a user account on a website?:


Error. The password field can contain only letters and numbers


I had included a couple of punctuation characters in my password, because that makes it harder to guess, right?

From a technical standpoint, there is absolutely no reason for this website to tell me I can’t use punctuation characters. If they’re encrypting ascii, then any ascii character should be legit. If they’re encrypting bytestreams, then any unicode byte should be legit.

The only validation a password field should have is to test if the password is ‘too easy.’ Typically a minimum length test is enough, but ensuring the user didn’t enter five 1s or their username as a password can be good validation too (although it’ll annoy the user, not often a good thing). You may also need a maximum length if your database is poorly designed, but make it a very high maximum in case anal-retentive people with 64 character passwords want to buy something from you. After all, why shouldn’t you let them?

Further, don’t force your users to have passwords that conform to arbitrary rules like ” must contain at least one number, one lower case letter, and one capital.” This actually cuts down the total number of options a brute force attacker needs to check if they want to break the password; they now know that eerieairplane is not an option they have to test. After all, eerieairplane and EerieAirPlane are totally different passwords, neither is more “guessable” than the other (unless you are a pilot for Lake Eerie Air, in which case you’re probably better off using pokertoMatotoOthpaste).

Users have different systems for creating their passwords; some of these systems aren’t very intelligent (same password everywhere, or prepend the name of the site to a common word), but forcing a single system of our own on them is even less intelligent.

While we’re on the subject, what’s up with all the corporate sites who believe that having security questions in case you forgot your password helps make things more secure? Honestly, do you think that my random password with extra punctuation is easier to guess than my first dog’s breed, my mother’s maiden name, or my favourite author? What’s the point of having a password at all if both I and any given attacker can just look up these values instead?

It’s not my fault! Yes, I did kill a few sites last week, and I did choose not to recover all of them. But the issue with several of my sites (ccphillips.net, archcon.archlinux.ca, schwag.archlinux.ca) being out all weekend was not my fault. That was caused by a MySQL upgrade on my shared webhost. The Python MySQL connector was unable to find the new library, and thus, all Python DB-driven sites on that host were hosed.
“Ok, so why didn’t you just fix it instead of leaving it out all weekend?” you ask.

I’d like to have a story like Judd, Aaron, or Dale used to use, “I was out camping/drinking/enjoying myself, and that was the weekend the server chose to go down.”

The sad truth is, I was stuck in the winter’s worst blizzard. While the weather dumped two feet of snow on us, drifted into 11 foot banks, my satellite internet connection was mostly down, and my power was off as much as it was on, all weekend. I wasn’t able to get a ssh session to last long enough to track down a fix for the issue.

I apologize for the inconvenience, but there was nothing I could do but sit tight and wait the storm out.

Last September I posted about Arch Bounty, a project I’d written to allow people to post ‘bounties’ to have specific ArchLinux bugs fixed. I didn’t promote it and interest faded quickly. I’ve been thinking of pulling the plug on it for a while now, but it happened unexpectedly yesterday when I accidentally killed half a dozen of the sites on my webhost. I’ve recovered most of them, but I decided that ArchBounty won’t be coming back.

There was one donation to the project; I will be forwarding it directly to the Arch Linux Donations fund.

Last week, I posted my intent to port the Tornado web framework to Python 3. Not only have I done that (sort of), but I’ve hacked it to pieces; soon it will be unrecognizable to the original developers, and possibly, to me!

It didn’t take long to get the Hello World example included with Tornado running. Quite a bit more work was the blog demo, but I now have the template, auth, and httpclient modules working, along with the core modules for a working async server. I was able to log into my example blog with my google account, compose some entries, view, and edit them, including the feed.

That doesn’t sound like much if you’re coding blog software for an existing framework (10 minutes in django, probably about 12 in web.py). But if you’re coding a web framework for existing blog software, it’s an accomplishment. I’m proud to have a “working” framework (even though it’s full of bugs and working just means “those things I’ve fixed”) for Python 3 in such a short amount of time.

I’ve named the project psyclone, a play on ‘tornado’ –> ‘cyclone,’ and the fact that Python projects oughta have a ‘p’ and ‘y’ in them somewhere. The code is on github for all to play with. Patches welcome!

I’m having a lot of fun with this project, so it’s taking more of my time than I ought to be devoting to it… on the positive side, it’s progressing rapidly!

My plans:

• Go over the existing code and improve some of the rather messy unicode/str –> str/unicode hacks I had to make to get it working.
• Write some tests. (The Tornado team seems not to value tests.) I’ll use py.test and may need to write a test client.
• Write a session framework and auth framework; the current auth framework uses openID only; but I like a more local solution to be available as well.
• Consider writing an ORM. Likely, I’ll discard this idea, arguing that Judd was right to design a frameworkwith SQL only. The truth behind the argument will be laziness, of course.

I’d like to announce the arrival of Arch Linux Pens to the Schwag shop. These are nicer than I imagined, very opulent’ I’m very happy with them and hope you will be too.

Pens are deep blue and gold with “Arch Linux” and “www.archlinux.org” engraved on them. They have a soft black grip and fine ball-point black ink. They are available for $5 individually, or as low as $3.50 in bulk.

Order them now from http://schwag.archlinux.ca/product/pen/.

Coroutines are a mysterious aspect of the Python programming language that many programmers don’t understand. When the first came out I thought, “Cool, now you can send values into generators to reset the sequence… when would I use that?” The examples in most books and tutorials are academic and unhelpful.

Last year, I attended David Beazley’s course A Curious Course On Coroutines along with a fellow Archer. We agreed that it was an exceptionally interesting course (Beazley built an OS scheduler in Python, with just a minimal amount of code: how cool is that), but that we didn’t see any practical application of it in our regular work.

Yesterday, I started working with the Tornado code to port it to Python 3. Tornado uses an async framework; I hate async because I hate working with code like this:

def somemethod(self):
    # 
    self.stream.read_until("\r\n\r\n", self.callback)
 
def self.callback(self, content):
    # handle content read from the stream

I understand the utility of this code; while the stream is being read, the app can take care of other stuff, like accepting new connections, until the stream has been read. You receive high speed concurrency without the overhead of threads, or the confusion of GIL. When the read is complete, it calls the callback function. It makes perfect sense, but when you read code with a lot of such callbacks, you’re constantly trying to figure out where the code went next.

In my mind, the above code is really saying:

def somemethod(self):
    # 
    self.stream.read_until("\r\n\r\n")
    # give up the CPU to let other stuff happen
    # but let me know as soon as the stream has finished reading
    # handle content read from the stream

I find this paradigm much easier to read; everything I want to do surrounding content is in one place. After pondering different ways to write a language in which this was possible, it hit me that this is what coroutines are for, and it’s possible in my preferred language.

Because coroutines use generator syntax, I thought they had something to do with iterators. They don’t, really. The above code can be written like so:

def somemethod(self):
    # 
    self.stream.read_until("\r\n\r\n")
    content = (yield)
    # handle the content

The calling code would call somemethod() and somemethod().next(), and eventually, when content is available, somemethod().send(content) to drive it.

A generator compiles to an object with an iterator interface. The coroutine above (sort of, but not really, at all) compiles to a function with a callback interface (you could say it is an iterator over callbacks). You can use yield multiple times in one method to receive more data (or to send it; put the value on the right side of yield, like in a generator).

The mainloop that called this code would still be at least as complicated to read as it is using a callback syntax, but the objects on the async loop are now much easier to read.

This paradigm has been implemented in the Diesel web framework. I’ve looked at it before and thought it was an extremely bizarre way to design a web framework. I still do, but now I understand what their goals were. If you’ve ever struggled with the, “why would I ever use this?” question when it comes to coroutines, now you understand too.

I have no immediate plans to rewrite my tornado port using coroutines, but maybe someday if I’m bored, I’ll give it a try.

I got it in my head this weekend that it was about time someone wrote a web framework for Python 3. My head is kind of stubborn about these things, so I asked it some questions:

Does the world need another web framework?
Do I need another web framework?
Do I have time to do this?

The answers were all “no.” Still, I’m planning to go ahead with it until I get bored. Then the project can sit and collect dust with all my others.

A bit of discussion with The Cactus, led to a few ideas:

I discovered that QP is apparently the “first Python-3 enabled web framework.” I didn’t try it, so I was perhaps unfair in discarding it, but it doesn’t look… suitable.

I looked around some more, and found that CherryPy is about to release a Python 3 enabled version. I’m sure that will spawn a whole slough of Python 3 frameworks built around CherryPy. I considered such a plan (I’d call it ChokeCherryPy based on a receipe my mom devised): create some kind of templating system based on str.format, some session support, and some kind of database module wrapped around py-postgresql. Could be fun. But I’d end up with a mess of third-party technologies much like TurboGears, and that would be embarrassing, plus I’m sure the TG team already has people working on this.

Then I came back to my original plan, which was to port either Tornado or web.py to Python 3. Tornado looks like a smaller codebase (easier to port) and I’ve never used it before, so it’s also a chance to learn something new. So today I forked Tornado on github and run 2to3 on it. I’ve already got the “hello world” demo running; it wasn’t too hard once I figured out the difference between bytes and strings. At least, I think I did that part correctly.

The project is named psyclone, a little play on the ‘destructive weather patterns’ genre. I was close to p3clone, but it’s too hard to convince people it should be pronounced ‘cyclone’.

This isn’t a project I expect to go anywhere; django will be ported to Python 3 soon enough, and other frameworks will be popping up all over. But I’ve been working with Python 3 a lot lately, and I thought it was time to tackle the ’scary’ job of porting an existing app. It’s tedious, but not difficult.

In the so-called “copyright wars,” we see a spectrum having the MPAA, RIAA, Jack Valenti, and “all rights reserved” on one side, with the Pirate Parties, Pirate Bay, Rick Falkvinge, and “no rights reserved” on the other side. In the middle, we have Creative Commons, Lawrence Lessig, and “some rights reserved”.

I’d like to momentarily expand this line to one that places “no rights reserved” in the middle, in a way that shifts Lessig closer to Valenti, and opens up a whole new area of creative exploration beyond the pirates, who are no longer extremists.

First, a disclaimer: I don’t claim to have any answers. I don’t even believe what I’m suggesting is the right path. I am simply suggesting an idea that frames a long-standing and long-term discussion in a different light.

The spectrum above defines the opposite of a right as “the absense of a right.” This only goes halfway. The opposite of a right is a responsibility.

Image, for a moment, a society where there is no such thing as, “the right to my creation,” but there is a massive, “responsibility to create.” In this society, people would have free access to all the materials of the world, all the patents, blueprints, and software, all the films, songs, and books, all the photos, paintings, and sketches the world has ever seen. In exchange for this free access, individuals would be required (responsible) to create a certain amount of new material every year. Some of this material would be innovative and fresh, some would be a new presentation of old stories and ideas, some of it would be interpretations of those old stories in new media. We’d see new designs for existing products, we’d see new products that merge old technologies. We’d see Android phones with iphone gestures, and we’d see Mickey Mouse saving Princess Peach from the evil Bowser the Hedgehog.

Such a world may excite some, bore others, and scare many. Would these same people be less excited, bored, or scared by the Pirate Party? by Creative Commons? Maybe those deals aren’t so bad after all (to those demanding rights)… or maybe they aren’t so good (to the promoters of creativity).

This responsibility to create idea seems radical in the context of entertainment media, but it is not new. It’s a long-standing scientific tradition, best encompassed by Newton’s overused quote about giants. Academics have “free” access to the entire compendium of academic knowledge; in exchange for this access, they are expected (responsible) to generate new ideas and innovations. Some are good and some are bad, but if a scientist neglects to publish a few new papers a year, they fade from the academic community.

This idea is also an unofficial motivator in open source communities. Within the Arch Linux community, my home, I’ve made some effort recently to verbalize this norm. The story goes thus: Arch Linux has had contributions from many thousands of users. Each of us that uses the distribution is somehow indebted to all those other users. Further, we can never, as individuals, pay off the debt in its entirety. Even the well-known user with 8000 posts on the forum, thousands of package updates to his name, and dozens of Arch Linux tools under his belt has contributed but a drop in the bucket compared to the efforts of the entire community. And Aaron is aware of this debt. So should we all be.

Yes, in the academic and open source world, the implied responsibility to create is known to work. Creativity in both worlds spreads more quickly than anywhere else. Compare to the communities creating ideas whose soul purpose is entertainment. Even the liberated Jamendo is mired way over in the (Some) Rights Reserved end of the scale.