Posts tagged ‘open source’

Guido Van Rossum Should Retire (and focus on python)

At the two previous Pycons I’ve attended (2009 and 2012), Guido Van Rossum’s keynotes sounded bored and uninterested, even though the content was meaningful. I was actually wondering if this would be the year that he would step down from BDFL of Python. Thankfully, I was dead wrong.

Instead, he presented a highly technical and very exciting addition to the Python language. Alfredo told me this started when he took a month off between stepping down at Google and starting at DropBox. Now, when normal people take a month off, they relax or travel or visit friends and family. Not our BDFL. He writes a callback-free asynchronous event loop API and reference implementation that is expected to massively alleviate Python’s oft-maligned lack of a consistent, unhackish concurrency solution.

Let’s have more of that. What if Mr. Van Rossum could hack on Python full time? Would we see quantum progress in Python every month?

Anyone who knows about the Gittip project likely thinks they can guess where this is going. We, the people, can each tip our BDFL a few cents or dollars per week so he can focus on whatever he deems worthy. It’s safe to assume that a man who spends his vacation time drafting a new Python library would choose to work on Python full time if we funded him.

This plan is great, and I actually think that Guido could easily earn enough to quit his day job if he endorsed Gittip and invited individuals to tip him. But I’d like to discuss a different plan: Not individuals, but companies should tip Guido the maximum gittip amount as a sort of “partial salary”. At $1248 per year, most companies wouldn’t even notice this expense, and they would get a better programming language and standard library in return. The rate of accelerated development would be even higher if each of these companies chose to invest an entire salary, split between a hundred Python core and library developers. If a hundred companies chose to do this, those hundred people could work on Python full time. The language and library would improve so vastly and so rapidly that the return on investment for each of those companies would be far greater than if they had paid that same salary to a single developer working on their in-house product, full time.

It might take some convincing to justify such a strategy to these corporations. Companies tend to like to know what is happening to their money, and simply throwing a hefty developer salary at Gittip would be hard to justify. Obviously “goodwill” could support some of it, in the same way that so many companies sponsored Pycon in exchange for exposure.

Cutthroat CEOs should perhaps consider not just the value that having Guido working exclusively on Python is, but also the cost of having him work for the competition. I’m sure Box.com CEO Aaron Levie was a little nervous when he found out that the first and greatest Python programmer of all time had recently hired on at a major competitor. Perhaps Box.com can’t afford to steal Guido from Dropbox, but if all the companies currently involved in cloud storage were to tip Guido $24 per week on Gittip, this incredible programmer could be working on an open source product that directly and indirectly benefits their company rather than improving the competing product on a full-time basis.

Most of the arguments that Gittip will fail are based on the premise that not enough money can be injected into the platform to sustain full time development by open source programmers. However, if an open and caring relationship can be built such that the corporate world is also funding the system, I think it can become extremely successful. Everyone will benefit: Open source projects will improve at a rapid pace. Exceptional developers will get to pursue their passions. End users will get better products. The overall level of happiness in the world will be higher.

I would like to see a world where brilliant young software engineers are not risking their mental health (and consequently, their lives) on startup ideas in the hopes of being bought out for a few billion dollars. I would like to see a world where those engineers are not working for large corporations that have neither their employees nor their end users (but rather, their stockholders and advertisers) interests at heart. I would like to see a world where those developers can choose to invest their passion in open source products that will change the world.

Offline Enabled Webapps And Open Source

I’m currently working on a rather simple Todo list application intended to meet the requirements I outlined earlier. I’m developing this as an offline-enabled webapp and while I haven’t gotten very far (don’t try it, it barely satisfies the most elemental requirements), I have some interesting observations to make:

First, the app is currently 100% Javascript. Its wrapped in a Django project because I’m going to have to make it online enabled at some point, but I wrote this app from the ground up to run in offline mode. This is a huge departure from traditional web 2.0 development (as of now, web 2.0 is to be considered ‘traditional’), where logic is mostly stored on the server and an ugly mess of ajax calls run that business logic on different events. Instead, I have a Javascript app that is surprisingly elegant. MVC under this architecture is basically SQLite (model), XHTML/CSS (view), and javascript (controller). Then I will have a workerpool to sync up with the server in the background.

As a result, the entire code-base for this app is being served to the web browser. So if you personally want to hack this app, you can download the source files from the browser. Basically, I can’t release this app as closed source. I could put a license on it forbidding you to modify or redistribute it, but you can still read the code; the source is open.

Of course, that’s not a problem for me, as I release most of my code as open source and its already up on github. But it could kill corporate migration toward offline-enabled web 3.0. Because while its one thing to keep your trade secrets locked up on the server and provide a few incoherent javascript calls that interact with that API, it is a completely different beast to put your entire app available for download as a .js file.

I suspect the open source movement has gained enough momentum that any companies who have these fears will simply be put out of business by more modern outfits that will spring up to take their place. Otherwise, either web 3.0 will fail to grab market share, or somebody will come up with a way to ‘compile’ javascript into a standards compliant byte-code so they can pretend nobody can reverse engineer their app. This would be a damn shame as its a politically-motivated technically useless layer of complexity on a web based architecture that I am finally happy to be working with.

Offline-Enabled Web Apps: The Future

I was reluctant to join the world of web development. I started in high school with a few sites and realized several things: Javascript sucks, Internet Explorer sucks; therefore web development sucks.

Fast-forward through a couple academic degrees. Job hunting with one requirement: Python. Python jobs all require Django.

So I learned Django, assuming, incorrectly, that if I was developing python backends, I wouldn’t need to work with the horrors of Javascript or Internet Explorer. I earned money. I relearned Javascript and became a first rate web developer.

In the back of my mind I still felt that web development sucks. So a few weeks back when deciding on a platform for a personal project, I thought I’d try something new. The Android platform was in my hands and I gave it a whirl.

I didn’t enjoy it much and I am now rewriting the app as an offline enabled webapp using Google Gears.

Then Chrome OS was announced and I realized that I’ll probably be doing a lot of offline enabled webapps using Google gears and/or HTML 5. Like it or not, it’s the future. Me, I like it. There are a lot of advantages to this kind of setup: I can access the apps from my phone, my laptop, my parent’s desktop, or Phrakture’s hacked computer whenever and wherever I want. I don’t have to write a different client for each one. Its true ‘write once, run anywhere’. I can upgrade each of those clients automatically as long as there’s a network connection.

On that note, you don’t need a network connection to run HTML 5 or Google Gears based apps. They both provide a ‘localserver’ that caches pages and javascripts, and give you an SQLite database for data caching. Typically offline versions of apps are not as powerful as their networked counterparts, but they do not require network access to run. Further, because they are locally cached, they can be made to run as fast as a “standard” (old fashioned) desktop app. The apps run in the browser, but the browser is just a container, a window manager, to hold the application.

In traditional webapps, you code most of the logic on the server side. In this new model, you end up coding most of the logic in the client, because the app needs to run without a guaranteed server connection. For me, this has a massive, nearly show-stopping drawback: A large portion of the app must be written in Javascript. JQuery makes Javascript suck lest, but it still sucks. I’m a Python programmer.

For years, I’ve dreamed of browsers supporting tags that allow me to write my DOM manipulation scripts in Python rather than the ubiquitous and annoying Javascript. This wasn’t possible because python can’t be adequately sandboxed such that arbitrary scripts running on the web don’t have access to, say, your entire hard drive.

This is no longer true. The PyPy project finally has a complete Python 2.5 interpreter that can be safely sandboxed. Since discovering this at Pycon 2009, I’ve been thinking about interfacing it with a web browser.

I figured “somebody must have started this already”. Google didn’t help much, but when I logged into #pypy on freenode I was told “fijal started doing that with webkit yesterday”. I’ve been following up trying to get the project to build (I was warned that the build process is a mess and was invited to wait until it is cleaned up a bit). So far, no luck, but I am optimistic that python support is finally coming to the browser. Granted, it won’t be much use for public webapps (at first) since browsers won’t want to be distributing pypy, but a lot of my projects are personal, and satisfying the general public will be far lower on my priorities list than ‘developing in my preferred language’.

I’ll have to install a pypy interpreter into Chrome Lite under Android before this is useful to me. That may be tricky.

Recent Developments

I’ve been awfully busy the past few weeks, but finally had three separate evenings to sit down and code on some of my little projects this week. I’m anticipating having more time in a couple weeks, as I gave notice on my job on Thursday. I am, however planning a move and exploring numerous job opportunities in my home province.

I managed to fix a couple bugs on WhoHasMy. As previously reported this project was originally coded in 48 hours for the Django Dash competition. We tied for fifth place and have traded the resulting bitbucket account for a github prize. I’m very happy with the placement given that my brilliant co-developers had nil django experience going into the competition and I hadn’t touched it professionally in months. I added a TOS to the page as requested in a comment on my earlier post, fixed some ordering bugs in the lists, fixed a couple broken links, and made it easier to add information about friends when you loan an item to someone not currently in the system. And here we thought it was 100% bug free when we finished our 48 hour stint and stumbled off to bed.

I have also spent a fair bit of time improving Quodroid, the Android app for controlling quod libet on my laptop from my phone. It now uses fancy icon buttons, allows you to specify the host and port you want to connect to, lists the currently playing song whenever you perform an action, allows volume control, and gives a semi-sane error message when the phone can’t connect. In short, its actually useful and usable by someone other than myself. I’ve been using it regularly the past few days. I still have to arrange it to perform the network stuff in a service instead of the main activity, which occasionally becomes unresponsive if the server is slow to respond. I’m actually becoming more comfortable with Java again as I develop this, its not as evil as I thought, but it certainly cuts into productivity.

Today, I made a few changes to opterator. I wrote my first app (a contrived example code-test for a job I’m pursuing) that actually used opterator a couple weeks back and found it was missing a few features. It now the ability to have multiple copies of a single option. Turns out this actually worked, all you had to do was use the ‘append’ action. I wrote three tests, didn’t change a line of code and poof, I had append support! I then realized that storing the action in the docstring was unnecessary as it could be introspected from the type of the keyword argument. This makes the @param docstrings a lot more readable and informative. As simple as this little module is, I feel its one of my more brilliant innovations.

I’ve also tossed around the idea of having multiple opterated main methods in a single module and allow the decorator to pick which one to call depending on the options. This seemed cool at first, but I think it may violate the ‘one best way’ policy of Python. I also realized that deriving sensible error messages and usage strings would be really painful, from the end user’s perspective, so I’m holding off on this until I’ve decided how best to do it.

Quodroid, an app just for me

I’ve spent a few days reading through the Android developer docs and trying to get Eclipse to cooperate with my outlook on software development. It took giving up on Eclipse to get any real coding done. It was a bit tricky, as I’m not the all-star Java programmer I once was, and I got bogged down on silly syntax errors and library issues. For example, I forgot that ‘new’ is a keyword in Java, and trying to instantiate a class without it results in errors that don’t say, “what do you think this is, Python?”.

The pet project I started is an extremely simple music remote control to interact with Quod Libet, my music player of choice, running on my laptop. So now if I have my phone at the couch I don’t have to get up and walk six steps to my desk to pause or skip to the next song.

The overall architecture is mildly embarrassing, but works well. I’ve got a very simple web.py server running on my laptop that accepts three urls — prevous, next, and play-pause. When one of these is received, it sends a control command to quod libet to perform the associated action. The android client simply displays three ultra-ugly buttons that send requests to a (hard-coded, no less) IP and Presto! I can skip tracks and pause my music!

It’s barely working yet and the design is atrocious, but I’ve posted the initial version to github for peer review and patches storage.

I really just wrote this for myself, the fact that I’m releasing it is mostly because a) I have a blog now and b) I have a github account now.

I’ve got several things on my to-do list, most importantly improving the client interface. I want to add volume control as well, and the ability to return the current song whenever an action is invoked. I might even add a polling feature to update the displayed song on the phone. I also need to do some config stuff, hard-coding the url may actually not be that flexible.

Introducing Opterator

When I attended Pycon 2009, I saw a lightning talk on a neat little tool called Argparse. Its a python option parsing tool designed to be easier to use than optparse.

In the first few seconds of the talk, I thought to myself “I know where this is going. It’s brilliant.” I said so to the guy next to me. Then I took it back because while argparse is a pretty cool idea, its not the brilliant idea I thought it was.

See, I thought the tool would automatically introspect a script’s main method and use the docstring to provide any missing information. That’s what I thought would be brilliant.

Since it didn’t do that, I did it. Its certainly not full-featured yet, but it works for basic options and arguments. For 90% of scripts, that’s all you need. I called it opterator and put it on github:

http://github.com/buchuki/opterator/tree/master.

git clone git://github.com/buchuki/opterator.git

Here’s an example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
from opterator import opterate
@opterate
def main(filename1, filename2, recursive=False, backup=False,
        suffix='~', *other_filenames):
    '''An example copy script with some example parameters that might
    be used in a copy command.
 
    @param recursive store_true -r --recursive copy directories
        recursively
    @param backup store_true -b --backup backup any files you copy over
    @param suffix store -S --suffix override the usual backup
        suffix '''
    filenames = [filename1, filename2] + list(other_filenames)
    destination = filenames.pop()
 
    print "You asked to move %s to %s" % (filenames, destination)
    if recursive:
        print "You asked to copy directories recursively."
    if backup:
        print "You asked to backup any overwritten files."
        print "You would use the suffix %s" % suffix
 
if __name__ == '__main__':
    main()

Obviously, that only parses the options, its not doing the actual copying because implementing that would do nothing to illustrate how opterator is working. Here’s one run of it:

dusty:opterator $ python copy.pyc -b a b c d -r
You asked to move ['a', 'b', 'c'] to d
You asked to copy directories recursively.
You asked to backup any overwritten files.
You would use the suffix ~

Here’s the automatically generated help:

dusty:opterator $ python copy.py -h
Usage: copy.py [options] filename1 filename2 [other_filenames]

An example copy script with some example parameters that might
    be used in a copy command.

Options:
  -h, --help            show this help message and exit
  -r, --recursive       copy directories recursively
  -b, --backup          backup any files you copy over
  -S SUFFIX, --suffix=SUFFIX
                        override the usual backup suffix

I named it opterator because I wanted a name that didn’t have ‘parse’ in it. I was going to go for operator but I realized if you heard about this app called operator and searched google for ‘python operator’ you’d get swamped. ‘opterator’ is unique and highlights the option part of the task. Plus it allowed me to invent a new verb: ‘opterate’. Days when I get to invent new verbs are always good days.

I used py.test for testing (another gem I discovered at pycon). You don’t need to run the tests, but the tests are useful examples to see what opterator can do, so have a look.

So I hope somebody finds this useful or even awesome and contributes some patches to make it more of the above. Enjoy!

Todo List

I’ve spent a lot of time thinking about the proper way to design a calendar and to-do list application.  The irony: I never got around to putting ‘write to-do list application’ on my to-do list.

The best to-do list I’ve seen was designed by Kim Hoyer with input from myself and another developer. Its part of Kim’s proprietary Pursuits XRM System, a comprehensive sales and company management system. Oprius also has a terrific to-do and appointment management system. Google’s Calendar, on the other hand, seems to have done everything all wrong, by my standards.

I’ve read about several of the web-based options and discarded them for various reasons, usually too much complexity. Remember The Milk is a notable exception in that its complexities can be easily ignored. However, its still due-date based, and that’s not the way I work.

Stephen Covey’s well-known ‘The Seven Habits of Highly Effective People’ describes a slightly over-engineered, but otherwise workable paper-based to-do system that really jives with the way I think.

I’ve tried numerous solutions and always fall back to writing stuff on a scrap of paper. I’ve been actively monitoring exactly how I really do things (instead of trying to imagine how I should do things) in the 5.5 months since getting a day-book for Christmas. I’m ready to design a to-do list. I’d probably have sat down and started coding by now if I didn’t have this blog thingy to lay out some ideas. Just a little groundwork.

Day-oriented, not due-date oriented

When I plan what I want to do, its always about what I want to do today. I don’t care that the due date of a task is in two weeks, I care only about choosing whether I am or am not working on that task today. When I’m done working on the task today, I cross it off my todo list, even if the task isn’t complete. If its not complete, I add a NEW task for the next day. Its good to break big tasks into bite-sized sub-tasks, but often I just write the same task down for each day that I work on it.

Only plan a few days in advance

I need to be able to add, reorder, and move tasks between all days, but typically I won’t have tasks listed for something more than a week in advance. Unless I’m specifically meeting someone or planning a vacation, I don’t have stuff filled out for two months from now.

Area for planned but scheduled tasks

I currently add tasks I intend to do in the future for other days and then ‘move’ them by crossing out and rewriting under another day. This is suboptimal. I want a separate section for tasks that I don’t want to forget. It needs to be easy to move them into a specific day.

Recurring Tasks Suck

Most of the tasks I do on a recurring basis don’t happen at the same date and time each week. I just know I need to do them once a week. Having them auto occur makes me easily ignore them.

Instead, I need an area (possibly same area from previous point) to store tasks that I do repeatedly. These would be generic tasks and whenever I need to put one on a specific date I can just select it and add a date.

Super minimalist

I don’t need to attach much info to a task. I don’t need priorities, descriptions, notes, durations, locations, contacts. This shit clogs up the interface and the task name itself usually helps me recall all I need to remember about this stuff. Maybe if I was a sales person with 90 contacts per day that I can’t remember their names and faces I’d need those details, but in my life, its just extra cruft.

Task Ordering

On any given day, I want the completed tasks at the bottom of the list. Uncompleted tasks are at the top in a semi-ordered fashion. Currently when I sit down at my daybook, I cross out the last item I did and pick another one based on my current priorities. Sometimes I draw numbers beside them to note the next three things I’m going to do. In software, I want to use either drag and drop or tap-to-raise to easily order the next few things I plan to do. When I finish one, I want it to be easy to change my mind about what I’m doing next. Keep it agile!

My Phone

I expect I would make this a web app in the long run so I can access it anywhere, but I definitely need to be able to access it on my phone (Android Dev Phone). Since I want to play with the android APIs anyway, my first attempt is going to be for Android. Later I’ll tie it up to a mobile-oriented webapp similar to Choncho.