Archive for the ‘Python’ Category.

Trembling Aspen

Aspen is a unique Python web framework developed by Chad Whitacre. I am still evaluating it, but one thing I can say unequivocally is that it’s different. Very different. Aspen divorces itself from the MVC pattern that every major — and almost every minor — web framework has been relying on for the past decade. While each framework has its perks and annoyances, they all have this underlying MVC-like interface. Aspen throws that all away.

Aspen lodges itself in a small niche between php-like spaghetti code and a complicated package structure full of python models, views, controllers, and templates. In Aspen, template and control code live in the same file, but in separate pages, called Simplates. In a rebellion against regular expression url parsing, Aspen uses the filesystem to define url structure and paths; like in the old days of statically served HTML files.

One thing I love about Aspen is how easy it is to get up and running. Aspen will serve a single HTML file with no python code without any boilerplate. It serves static files for css, javascript, and images just as easily. To add a dynamic page to your Aspen powered site, you simply use the aforementioned simplates, which include python controllers and Tornado inspired templates in the same file. Aspen provides numerous service connectors, so deployment can also be trivial.

Another great feature of Aspen is its documentation. At first glance, the docs seem sparse and rather limited. However, they are well organized, and every page seems to answer every question I have on the topic succinctly and quickly. Chad seems to have a terrific minimalist policy of telling you exactly what you want to know in as few words as possible. As a writer, I know just how difficult this is to accomplish without repeating oneself.

On the negative side, I do have misgivings as to how well Aspen’s filesystem urls it can scale. Chad has assured me that Aspen is up for running large sites. I appreciate the power of regex urls, and I don’t necessarily like having arbitrary file extensions in URLS. I think locators should be succinct and divorced from implementation details. Ideally, I believe there may be a happy medium between Aspen’s enforced folder structure, and the overly-abstract and overly-complicated url scheme Django and its kin provide.

Another downside is that I haven’t figured out a good way to unit test Aspen simplates. You can refactor much of the code into backend libraries and test that, of course, or you can start the service running and mock requests. However, I miss the TestClient feature from Django that allows me to mock a request without running the server. It provides a better analysis of code coverage, and is closer to the actual code.

Now, with the review over, I’d like to introduce a little project I’ve started to provide some vital features that Aspen core is missing.
Trembling is kind of a catch-all project with a few basic helpers and a few useful projects. Namely, trembling provides:

  • Basic sessions
  • Basic user authentication
  • Less than basic single request messages
  • A simple redirect response

Aspen itself does not define or require a model layer; you’re free to use the ORM, DBAPI connector, or NoSQL database of your choice. Trembling arbitrarily chooses mongoengine for the models. I made no effort to make pluggable backends like Django does; Aspen does not encourage excessive layers of abstraction.

I also don’t know how secure trembling’s session and auth features are. If you find a security hole, pull requests are welcome. Overall, I think the combination of Trembling and Aspen will be very useful for my future basic sites. I would also like to try it on a more complicated project just to see how well it performs.

Hacking on PyPy

In another great Pycon2012 keynote, David Beazely asked the question, “is PyPy easily hackable?” After a great talk, he answered with a decisive, “I still don’t know.” Having sprinted on Python I’d like to answer his question in a bit more detail.

I love David’s presentation style. He has a novel method of using phrases like, “blow your mind” and “this is really scary” repeatedly until they lose their meaning and you no longer feel mindblown or scared. A variety of factors, including Beazely’s thorough keynote address motivated me to join the PyPy team during the Pycon developer sprints.

I’d like to clear up one oversight in Dave’s otherwise impeachable talk. One of the PyPy devs, Holger Kregel explained to me that PyPy does not have over 1 million lines of code. I don’t have exact numbers, but for “historical reasons”, a non-python file containing Base64 encoded data was given a .py extension. When excluding this file from the line count, around half a million lines of actual Python code exist, and about a quarter of these are tests.

I was surprised how trivial it was to get started hacking on PyPy. I don’t really grok the many layers of the translation toolset and PyPy interpreter, but it’s pretty clear that the layers are well separated. I was hacking on the py3k branch of PyPy. I am happy to admit I was working primarily on changing print statements to print() functions and commas in exceptions to the as keyword.

Here are the steps to start hacking on PyPy. Notice that the hour-long translation step is not part of the procedure. PyPy has a solid test framework, and the PyPy crew are focused on a 100% test-driven-development paradigm.

  1. Clone pypy (this takes a while):
    hg clone https://bitbucket.org/pypy/pypy/
  2. Pick a branch to work on. There are about 80 branches. I don’t know what they all do. Popular ones during the sprints included py3k and numpy-ufuncs2
  3. Pick a feature to work on. For py3k support, the list of failing tests in the buildbot is a good place to start. Numpy programmers had a list of fuctions that needed implementing, but I can’t find the link. Ask on IRC, the PyPy crew are very helpful. The bug tracker contains many features and issues that need addressing
  4. Add /path/to/pypy/ to your path so you can run the pytest.py command
  5. cd into the directory indicated in the buildbot output and run pytest.py path/to/test.py -k testname
  6. The test will likely fail. Hack away and fix it.
  7. When the test passes, commit, push to a bitbucket repo, and issue a pull request.
  8. Repeat!
  9. There are quite a few cons to working on this project. If you run hg in the pypy/modules/ directory, it will try to pick standard library modules from pypy and choke horribly. The pypy developers don’t really believe in documenting their code. Being able to tell the difference between rpython and python (which have identical syntax) is important. In general, if a module starts with “interp_” it contains rpython, but if it starts with “app_” it contains python. The code does not appear to be well-documented.

    If you are hacking on Python 3 support, you need to bear in mind that the PyPy interpreter is written in Python 2. You are working on a Python 2 application that executes Python 3 bytecode!

    On the positive side, rPython and Python are much easier to read and write than C. The PyPy devs are brilliant, but not intimidating. They are so confident in their test suite that they are comfortable programming in a “cowboy coding” style, hacking randomly until all the tests pass. Any one layer in the toolchain is easy to understand and develop. The IRC channel is full of friendly, knowledgable, helpful people at any time of day.

    Overall, I am much less intimidated by this project than I was before I started the dev sprints. I still can’t answer the, “Is Python easily hackable?” question fully. It’s certainly easy to get started, but I don’t know how easy it is to become intimate with the project. Dave Beazely’s keynote made PyPy more approachable, and I approached it. Hopefully this article will encourage you to do the same.

Why we need Python in the Browser

In his Pycon 2012 keynote speech on Sunday, Guido van Rossum covered many of the open “Troll” questions related to the Python community. I’ve had occasion to either complain about or defend all of the topics he covered, and he did a wonderful job of addressing them. The issues could largely be divided into two categories: those that come from a fundamental misunderstanding of why Python is wonderful (e.g. whitespace), and those that are not an issue for the core python dev team, but are being actively worked on outside of core Python (e.g event loop).

And then there’s the question of supporting Python in the web browser. I honestly thought I was the only one that cared about this issue, but apparently enough people have complained about it that Guido felt a need to address it. His basic assertion is that the browsers aren’t going to support this because nobody uses it and that nobody uses it because the browsers don’t support it.

This is a political problem. Politics shouldn’t impact technical awesomeness.

The fallacious underlying assumption here is that modern HTML applications must be supported on all web browsers in order to be useful. This is no longer true. Web browser applications are not necessarily deployed to myriad unknown clients. In a way, HTML 5, CSS 3, and DOM manipulation have emerged as a de facto standard MVC and GUI system. For example, many mobile apps are developed with HTML 5 interfaces that are rendered by a packaged web library rather than an unknown browser. Client side local storage has created fully Javascript applications that require no or optional network connectivity. There are even situations where it may not be necessary to sandbox the code because it’s trusted. Many developers create personal or private projects using HTMl 5 because it’s convenient.

Convenient. It would be more convenient if we could code these projects in Python. Web browsers can be viewed as a zero install interface, a virtual machine for these applications. Such a VM has no reason to be language dependent.

It is simply unfair to all the other programming languages and coders of those languages to say, “we can’t displace Javascript, so we won’t try.” Web browsers have evolved into a virtualization layer more like operating systems than single programs. While it is true that the most restrictive operating systems only permit us to code in Objective C, in general, it is not considerate to restrict your developers a single language or environment.

It is time (in fact, long overdue) for Python to be supported in the browser, not necessarily as an equal to Javascript, but as an alternative. The web is a platform, and we must take Guido’s statement as a call to improve this platform, not to give up on it.

Update: I just stumbled across http://www.gnu.org/software/pythonwebkit/ and I can’t wait to play with it!

Update 2: From the number of comments on this article, it appears that my article has hit some web aggregator. To those users suggesting python to javascript compilers, I’ve been a minor contributor to the pyjaco project, a rewrite of the pyjs library. It has the potential to be a great tool, and the head developer, Christian Iversen is very open to outside contributions. Let’s make it better!

Pyjaco in a real app: Todos with local storage

I didn’t get the memo, but there appears to be a movement to demonstrate emerging web technologies with a simple todo list application, much as hello world is used to introduce programming languages.

In my last post, I introduced using jQuery with Pyjaco, the PYthon JAvascript COmpiler. Since then, I’ve made several contributions to the project and have been involved in diverse discussions with Pyjaco developers regarding the current and future status of the project. This post goes further by acting as a tutorial for writing a basic todos app using Pyjaco.

Pyjaco is alpha software. It is hard to write valid code, and harder to debug. I’ve managed to both lock up Firefox and hard crash it while using the Pyjaco library.

On the positive side, Pyjaco is under active, rapid development. The head developer, Christian Iversen is extremely responsive to both questions about Pyjaco, and to code contributions. This is a project with a lot of potential, and I see it as the current best bet for Python programmers hoping to avoid javascript one day in the future.

In spite of the hiccups, it is possible to generate a working javascript app using just Pyjaco. Here’s how.

Let’s start:

mkdir pyjados
cd pyjados
virtualenv2 venv --distribute --no-site-packages
source venv/bin/activate
pip install git+https://buchuki@github.com/buchuki/pyjaco.git@run_script

First we create a directory to work in and install a virtualenv. Pyjaco does not currently work with python 3, so in Arch Linux, I use the virtualenv2 command. We then activate the virtualenv and install the pyjaco package. Here I am installing from my personal fork, as it contains some changes for generating the built-in standard library that have not yet been merged upstream. You should normally install directly from chrivers’s git repository using pip install git+git://github.com/chrivers/pyjaco.git.

Now let’s create a basic HTML 5 page with jQuery loaded:

<!DOCTYPE html>
<html>
  <head>
    <title>PyJaco Todo List Example</title>
    <script type="text/javascript" src="http://code.jquery.com/jquery-1.6.4.min.js"></script>
  </head>
  <body>
    <h1>PyJaco Todo List Example</h1> 
  </body>
</html>

We can load this in our web browser using a file:// URL. This is the only HTML page in our app, and it can be refreshed to load our changes as we work.

Pyjaco doesn’t simply translate Python into Javascript. Rather, it creates a basic standard library of Python-like objects that are utilized in the compiled javascript code. I wasn’t too keen on this idea when I first heard it, as it introduces a dependency that currently weighs in at 65K before minification and compression. While this is not a terribly heavy library, there are efforts under way to shrink the builtins or to dynamically generate it to contain only those builtins that your code actually touches. At any rate, we need to ensure this library is available to our code. First we generate the library:

pyjs.py --builtins=generate --output=py-builtins.js

pyjs.py is the name of the pyjaco command. It is expected to be renamed to pyjaco in the future. The --builtins=generate option tells pyjaco to generate the standard library, while the --output flag provides the filename for the new library file:

$ ls
index.html  py-builtins.js  venv

We then need to load this library in the head of our html file. Let’s also load the future pyjados.js script at this time:

  <head>
    <title>PyJaco Todo List Example</title>
    <script type="text/javascript" src="http://code.jquery.com/jquery-1.6.4.min.js"></script>
    <script type="text/javascript" src="py-builtins.js"></script>
    <script type="text/javascript" src="pyjados.js"></script>
  </head>

Now, before we start coding the Python file that will be compiled to Javascript, I want to discuss what I consider to be the most confusing aspect of Pyjaco development. There are basically two types of variables in Pyjaco, Javascript variables, and Python variables. Javascript variables refer to “normal” variables that you would call in Javascript. These include alert, window, document and the like, as well as variables in third-party Javascript libraries, such as the ubiquitous jQuery. Further, any attributes on those objects are also Javascript variables, and the return value of any methods will also be Javascript variables.

Python variables, on the other hand, refer to any variables that you define in your Python source code. If you create a dict or a list, for example, it will be compiled to a list or dict object from the standard library we just generated. In the compiled script, of course these Python variables are represented by Javascript objects, but from the point of view of a Pyjaco coder, it is important to keep the two types of files separate. Almost all the bugs I have encountered in my Pyjaco code have been caused by confusing the two types of variables.

The distinction between python and javascript variables introduces a couple of complications to writing Pyjaco compatible python code. First we need to flag all of our Javascript variables using a decorator on methods that access them. Second, we need to explicitly convert our variables between Javascript and Python any time we access one from the other. I’m told that this conversion can — and one day will — be done automatically by the pyjaco multiplexer, but in the meantime, we need to make it explicit. We do this by using two javascript functions supplied with the standard library we just generated, appropriately named js() and py(). You will see examples of these shortly.

When I finally figured out the distinction, my first thought was, “ok, let’s prefer to always work with python variables.” Therefore, in my initialization code, I tried jQ=py(jQuery). Unfortunately, jQuery is a rather large object, and the py function apparently recursively converts all attributes from javascript to python. I ended up with a stack overflow.

Now, let’s create our first python code and watch it compile to Javascript. Name the file pyjados.py:

def setup():
    print "Pyjados Hello World"
 
jQuery(js(setup));

First we write a python function named setup. This function is a python object. jQuery is a javascript object that expects a javascript object as input. Therefore, we wrap setup in a js() call and pass the result into the jQuery function. jQuery will now run setup when document.ready is fired.

Now we compile the code using the following command inside our activated virtualenv:

pyjs.py --watch pyjados.py --output pyjados.js

You’ll notice the command doesn’t exit. That is the --watch option at work. If you now make a change to pyjados.py and save it, it will automatically recompile it. The output file pyjados.js is regenerated each time. This is the file we included in our html file. So now, open that html file in a web browser using a file:// url. Make sure the Javascript console is displayed and reload the page. You should see the words “Pyjados Hello World” printed on the console. Pyjaco automatically compiles print statements into console.log output.

Before we start implementing our Todo list, let’s look at an example of accessing a javascript variable inside a function. Change setup.py to utilize alert, as follows:

 
@JSVar("alert")
def setup():
    alert(js("Pyjados Hello Alert")
 
jQuery(js(setup));

Did you look closely at that code? There is a missing close bracket on the alert line. You’ll see the syntax error in your console where pyjs.py is watching the compiled code. Add the bracket and let it automatically recompile itself:

 
@JSVar("alert")
def setup():
    alert(js("Pyjados Hello Alert"))
 
jQuery(js(setup));

Let’s analyze this snippet. First, notice how we told the compiler that alert is a Javascript variable when used inside setup(). This is a bit odd, since the JSVar decorator is never actually imported into the namespace. This is a bit of magic in the Pyjaco compiler, just pretend it has been imported.

Second, notice that since alert has been flagged as a JSVar, it must accept a Javascript variable. However, the string “Pyjados Hello Alert” is a Python variable. Therefore, we convert it using js() as we pass it into the alert call.

Now let’s prepare to create some working todo-list code. Start by adding a form for submitting todos and a list to render the todos to the html body:

  <body>
    <h1>PyJaco Todo List Example</h1> 
    <form id="add_todo_form">
      <input type="text" id="add_box" placeholder="Add Todo", autofocus="autofocus">
      <button id="add_button">Add Todo</button>
    </form>
    <ul id="todo_items"></ul>
  </body> 

Nothing too exciting here. Note the ids on the elements, since we’ll be utilizing these from Pyjaco using jQuery selectors.

Now back into the python file. Let’s create a class to manage the various todo elements:

class TodosApp:
    @JSVar("jQuery", "js_add_form")
    def __init__(self):
        js_add_form = jQuery(js("#add_todo_form"))
        js_add_form.submit(js(self.add_todo))
 
    def add_todo(self, event):
        print "form submitted"
        return js(False)
 
def setup():
    todo_app = TodosApp()
 
jQuery(js(setup));

The __init__ function hooks up the form’s submit button to a method on the object. Notice that we need to flag not just jQuery, but also js_add_form as a javascript variable. Pyjaco does not (currently) know that a javascript variable is returned when calling a method on an existing javascript variable. I like to add the js_ prefix to variable names to help remind myself that this is a javascript variable.

In an ideal world, we could convert this variable to a Python variable using py(), but as noted earlier, calling py on a jQuery object results in a stack overflow or browser crash.

Also pay attention to the way we wrap the self.add_todo method name in a js() call when we pass it into the submit handler. The submit method is a javascript function expecting a javascript object.

The def add_todo method has its single parameter flagged as a @JSVar, since the method is being called internally by jQuery when the event occurs. We also wrap the False return value (to prevent event propogation on the submit handler) in a js() call so that jQuery recognizes it as a javascript false rather than a (true) object named False.

Try the code. Ensure the compiler recompiled it, and reload the html file. Enter some characters into the text box and use the Enter key or the Add Todo button to submit the form. The words form submitted should be displayed in the javascript console.

Now let’s actually store and render a newly added todo. The todos are stored in memory in a python dict object. Initialize this object by adding the following two lines of code to the end of __init__:

    self.todos = {}
    self.next_id = 1

And rewrite add_todo as follows as well as a new method named render

    @JSVar("event", "js_add_box")
    def add_todo(self, js_event):
        js_add_box = jQuery(js("#add_box"))
        self.todos[self.next_id] = py(js_add_box.val())
        js_add_box.val('')
        js_add_box.focus()
        self.next_id += 1
        self.render()
        return js(False)
 
    @JSVar("js_todo_items")
    def render(self):
        js_todo_items = jQuery(js("#todo_items"))
        js_todo_items.html("")
        for id, todo in sorted(self.todos.items()):
            js_todo_items.append(js('<li>%s</li>' % (id, todo)))

Note that the todos dict is a Python object, so when we insert the value of the js_add_box into it, we must convert it from a javascript object using py(). Also note how, because we are writing in a python function, manipulating the python value self.next_id requires no conversion, and calling the python function self.render is also clean.

In the render function itself, I think it’s pretty cool that string formatting using % is supported by pyjaco (as an aside, the str.format method introduced in python 2.6 is not yet available) and that the python sorted() function is available. Note also how we can loop over items() on the self.todos dictionary just as if we were using a normal python dictionary.

Now let’s add the ability to complete todos. Let’s start by adding a template string as a class variable, and use that string inside the render function. This illustrates that pyjaco supports class variables:

class TodosApp:
    list_item_template = """<li>
    %(text)s
    </li>"""

and we change the for loop in render to:

        for id, todo in sorted(self.todos.items()):
            js_todo_items.append(js(TodosApp.list_item_template % {
                "id": id,
                "text": todo}))

Reload the page again and notice how checkboxes have been displayed beside each todo. The next step is to make clicking these boxes actually complete the todos. We add a couple lines to our __init__ method to connect a live click event to the checkbox items, which now looks like this:

    @JSVar("jQuery", "js_add_form", "js_checkbox")
    def __init__(self):
        js_add_form = jQuery(js("#add_todo_form"))
        js_add_form.submit(js(self.add_todo))
        js_checkbox = jQuery(js("input[type=checkbox]"))
        js_checkbox.live("click", js(self.complete_todo))
        self.todos = {}
        self.next_id = 1

Don’t forget to add js_checkbox to the JSVar decorator.

The complete_todo method looks like this:

    @JSVar("event", "jQuery", "todo_item")
    def complete_todo(self, event):
        todo_item = jQuery(event.target).parent()
        id = int(py(todo_item.attr("id"))[5:])
        del self.todos[id]
        todo_item.delay(1500).fadeOut("slow")

The first line is using exclusively javascript arguments, and returns the <li> element containing the checkbox that was clicked. The id = line converts the javascript string id attribute of this element (which looks like “todo_5“, as defined in list_item_template) into the python integer id of the todo. The remaining lines simply remove that todo from the internal list and from the DOM, after a 1.5 second delay.

In fact, we now have a fully functional todo list that allows adding todos and checking them off. Now, as a bonus, let’s try hooking this up to the HTML 5 localStorage object so that the list is maintained across page reloads. We start by adding a store() method to our class:

    @JSVar("localStorage", "JSON")
    def store(self):
        localStorage.setItem("todolist", JSON.stringify(js(self.todos)))

The main line of code is easiest to read from the inside out. First we convert the self.todos dict to a normal javascript object using the js() function. Then we call JSON.stringify on this object to create a string suitable for insertion into localStorage.

Now add this call to the end of the two methods that manipulate the todo list, add_todo and complete_todo:

        self.store()

.

Refresh the page, add a couple todos, and inspect the localStorage object in your console. You should see the stringified dict in the todolist value.

Now all we have to do is ensure the self.todos dict is loaded from localStorage when the app is initialized. Add the following to the end of the __init__ method (make sure to add js_stored_todos to the JSVars decorator):

        js_stored_todos = localStorage.getItem("todolist")
 
        if js_stored_todos:
            stored_dict = dict(py(JSON.parse(js_stored_todos)))
            self.todos = dict([(int(i), stored_dict[i]) for i in stored_dict.keys()])
            self.next_id = max(self.todos.keys()) + 1
 
        self.render()

Note that calling py() on the output of JSON.parse creates a python object, not a python dict. The code is therefore wrapped in a call to dict(), which converts the object to a dictionary.

Unfortunately, the resultant dict contains keys that are strings, whereas our original dict used integer keys. So a pure-python list comprehension is used to convert the dictionary to one with integer keys. This line is a bit hard to read, but I wanted to include it to demonstrate that Pyjaco can parse list comprehensions. Finally, we set self.next_id using the python max() call, which Pyjaco also automatically translates into javascript.

Try it out. Load the pyjados HTML file, add some todos, check a few of them off, then close and reload the web browser. Your todos will be stored!

I hope you’ve enjoyed this introduction to Pyjaco. It is a nice tool with a lot of potential. Currently, I find writing Pyjaco code to be approximately equally tedious to writing Javascript code. However, I feel that as I learn the ins and outs of Pyjaco, and as the developers continue to refine and improve the compiler, Pyjaco may one day be a perfectly viable alternative to writing pure Javascript or to the rather too Ruby-esque, but otherwise excellent Coffeescript.

pyjaco and jQuery

After giving up on CoffeeScript, I decided to play around with Pyjaco the Python to Javascript Compiler.

While the code is readable, there is very little in the way of end-user documentation. I hope to address this with this blog post. The Pyjaco examples all embed generated javascript in an html page. I needed a way to generate an external Javascript file as I would include in an HTML file. I also wanted to find out if I could use Pyjaco with jQuery.

The first step was to install a development version of Pyjaco:

git clone https://github.com/chrivers/pyjaco.git

Pyjaco normally requires a generated Javascript file mapping Python builtins to Javascript to be included with the created Javascripts. This must be generated:

cd pyjaco
python2 generate_library.py
cp py-builtins.js ~/pyjaco_test # directory for my new page

The next step was to create an HTML file that included jquery, the py-builtins.js script above, and a yet-to be defined javascript file named clicker.js that will be generated from a yet-to-be-defined python file. I also add a couple of DOM elements (a heading and paragraph) that are to be manipulated via jQuery:

<!DOCTYPE html>
<html>
<head>
<script type=”text/javascript” src=”jquery-1.6.4.min.js”></script>
<script type=”text/javascript” src=”py-builtins.js”></script>
<script type=”text/javascript” src=”clicker.js”></script>
</head>
<body>
<h1>Click me</h1>
<p id=”when_clicked”></p>
</body>
</html>

That’s the easy part. Writing python code that compiles to correct Javascript is the hard part. Pyjaco doesn’t currently provide very useful compile-time errors, and it also does not yet map javascript errors back to the input python.

There are (at least) two ways to compile python code in Pyjaco. The first, which is used in the Pyjaco examples, and appears to be the preferred method at this time is to create a custom main() method that uses various pyjaco.Compiler methods to combine the functions into a string of text. See https://github.com/chrivers/pyjaco/tree/devel/examples for three examples.

However, I was looking to have a complete python file that compiles to a complete javascript file. Pyjaco supports this as well, using the provided pyjs.py script. It took some investigating to understand how to reference javascript variables inside python functions. Decorator syntax is used to expose the variables for jQuery, Math.random, and Math.floor in the following example. The mystifying bit is that because we will be compiling this file to javascript as a string, it is not necessary (or possible) to import the JSVar decorator, as was done in the Pyjaco examples linked above.

# clicker.py
@JSVar("jQuery")
def ready():
    jQuery('h1').click(on_click)
 
@JSVar("jQuery", "Math.random", "Math.floor")
def on_click():
    if jQuery('#when_clicked').html():
        r = Math.floor(Math.random() * 255)
        g = Math.floor(Math.random() * 255)
        b = Math.floor(Math.random() * 255)
        color = "rgb(%d, %d, %d)" %(r,g,b)
        jQuery('#when_clicked').attr('style', 'background-color: ' + color)
    else:
        jQuery('#when_clicked').html("you clicked it!")
 
jQuery(ready)

Notice that I’m using the jQuery function instead of the $ alias, since $ is not a valid variable name in Python. This is a rather odd looking mix of Python and Javascript functions, but it works.

I had to repeat the compile and test step a few times before coming up with the above file. The script can be compiled using the pyjs.py that comes with the pyjaco source distribution (and, thanks to a simple patch I submitted, will come with the binary distribution in the next release.) Here’s how the script is run:

python2 ~/code/pyjaco/pyjs.py -N --output clicker.js clicker.py

The -N option tells pyjs not to generate the builtin library that we created manually in the first step.

This translation step creates a clicker.js file that looks like this:

var ready = function() {
    var __kwargs = __kwargs_get(arguments);
    var __varargs = __varargs_get(arguments);
    var $v1 = Array.prototype.slice.call(arguments).concat(js(__varargs));
    jQuery("h1").click(on_click);
return None;
}
var on_click = function() {
    var __kwargs = __kwargs_get(arguments);
    var __varargs = __varargs_get(arguments);
    var $v2 = Array.prototype.slice.call(arguments).concat(js(__varargs));
    if (bool(jQuery("#when_clicked").html()) === True) {
        var r = Math.floor((Math.random()) * (255));
        var g = Math.floor((Math.random()) * (255));
        var b = Math.floor((Math.random()) * (255));
        var color = str('rgb(%d, %d, %d)').PY$__mod__(tuple([r, g, b]));
        jQuery("#when_clicked").attr("style", ("background-color: ") + (color));
    } else {
        jQuery("#when_clicked").html("you clicked it!");
    }
return None;
}
jQuery(ready);

I find this rather unfortunately difficult to read. There is code for argument parsing that would not have been needed if I had hand-written javascript. Further the use of “mock” python builtins makes the javascript look less javascripty. However, the original python file looks much more readable than an equivalent javascript one would. I am hopeful that improvements to pyjaco will cause it to generate more readable javascript with less extraneous code.

The entire example can be found on my github fork

Christian Iversen is actively working on Pyjaco right now. I am excited about this project and hope that further community involvement will help it evolve into a practical and useful tool. I intend my next patch to be an autocompile tool that monitors files in one directory for change and outputs .js files in another directory, one of CoffeeScript’s killer features. I am also considering a port to Python 3.

I can’t use CoffeeScript

I had some space between wrapping up my last contract on December 20, and starting my new job on January 16. I decided it was finally time to build a simple task management system, something I’ve attempted to do and never finished several times before. I currently use RememberTheMilk, which is a lovely service, but I don’t like paying them for mobile access. Further, even though I trust this small company, I see no reason to share intimate information about the tasks I accomplish every day with them. My long term goal is to pull as much of my personal data out of the cloud as I possibly can. This project is a step towards that goal.

There are numerous open source task manager apps out there that I’m sure would suit my not-too-exotic tastes. However, I also wanted to take this opportunity to learn a bunch of new technologies for an offline-enabled and mobile-enabled web application.

Therefore, I’ve spent most of my vacation time so far researching some technologies I haven’t had a chance to explore in the past year. JQuery Mobile was at the top of the list. I think it’s a lovely framework and I expect to continue using it.

I also wanted to try out CoffeeScript, as I’ve always hated writing Javascript, and I wanted to use some sort of client-side ORM for localstorage. I looked at backbone.js, but was more attracted to Spine.js. I have spent three days studying and playing with these two technologies. I am still undecided about Spine.js, but I have come to the conclusion that CoffeeScript is not for me.

I understand all the hype around the project. JavaScript really does suck. And CoffeeScript does suck less, it has pretty language features and it is much more succinct than JavaScript. I can imagine a lot of people being really excited about CofeeScript, especially Ruby and PERL programmers, and possibly even php programmers.

But not Python programmers. I tried to learn Ruby several times, and each time I was left with a foul taste in my mouth. It’s not a bad language, it just doesn’t fit in well with my personal philosophy. My personal philosophy happens to coincide almost exactly with the Zen Of Python. I chose Python because it matched my philosophy… not the other way around.

Like Ruby and PERL, CoffeeScript violently violates what I consider the most important rule of Python: “There should be one — and preferably only one — obvious way to do it.” I’m not going to argue why this is a good idea, I understand that some programmers prefer the “even if I don’t understand it, I can write code that will probably work” paradigm that Ruby promotes.

The simple truth is, writing CoffeeScript leaves me feeling like I’ve done something dirty, no less dirty than writing JavaScript. There is no incentive for me to add a layer of complexity (the CoffeeScript to JavaScript compile step) to my code when I know my code is going to be “ugly” either way.

CoffeeScript is a wonderful idea. It’s far better than JavaScript. It’s just not good enough. Luckily, the Python community is already working on pythonic answers, including the evilly poorly documented pyjaco and the less-than-well maintained pyvascript and pyjamas projects. I hope one of these or a new upstart will soon gain community momentum so frontend development is no longer painful.

Finally happy with folding in vim (or how to get an overview of a python file)

I’ve always found vim folding to be somewhat mysterious. No matter which of the foldmethods I used, it seemed like it was always too tedious or too manual.

Today, while editing a python file over a remote connection that has high enough latency to interfere with rapidly scrolling through the file, I finally set out to solve this problem. My goal was to get an overview of the file. Yes, there are plugins or ctags to do this, but I don’t have ctags installed and have tried different plugins that didn’t really work as I wanted.

Indent folding does essentially what I want; when all folds are closed, I can see the overview of classes and module-level methods in the python module. When they are open, I see the whole file. I always felt that Indent mode should be ideal with whitespace-driven python, but in the past, I found that the default settings left far too many folds to work with.

I already new about the foldminlines setting which only creates a fold if more than a specific number of lines is displayed. But today, I discovered foldnestmax.

This setting is the maximum number of folds in the file. In a python file, if I set it to 1, folding happens only at the classes or module-level functions in the module. If I set it to two, it happens at each method in each class. I don’t get a bunch of internal folds that I don’t want.

Another setting that I discovered today is foldlevelstart. If set to 99, opening a new file will always have all the folds expanded, which is what I want.

So now I have exactly what I want: a few top-level folds are created automatically based on indentation, but I don’t have to fuss with interior indentation in other folds. Now I needed a refresher on how best to interact with these folds.

  1. zM Close All Folds Switch to “overview” mode
  2. zR Open All Folds Switch to normal editing
  3. zo Open Fold Open the class or method under the cursor
  4. zc Close Fold Close the class method I’m currently editing

The last one is the most useful, because it works from anywhere in the method. If I’m done with that method, I just zc and it’s hidden from site. I don’t have to worry about collapsing multiple levels of indentation or creating manual folds. It just works.

There are other fold commands; see :help fold for an overview, but just using these settings and memorizing these four commands seems to be all I need.

I suspect that javascript would also work well using a foldnestmax of 1. HTML would probably not work as well since normally the entire body is the top level of intent. Perhaps a foldnestmax of 2 or 3 would be helpful here. Luckily, I can change these values on per filetype basis using autocmd.

When you use a Django query keyword as a field name

I need to model a location in the Alberta Township System coordinate space. The model is extremely simple:

class Location(models.Model):
    project = models.ForeignKey(Project)
    lsd = models.PositiveIntegerField(null=True, blank=True)
    section = models.PositiveIntegerField(null=True, blank=True)
    township = models.PositiveIntegerField(null=True, blank=True)
    range = models.PositiveIntegerField(null=True, blank=True)
    meridian = models.PositiveIntegerField(null=True, blank=True)

There’s a rather subtle problem with this model, that came up months after I originally defined it. When querying the foreign key model by a join on location, having a field named range causes Django to choke:

>>> Project.objects.filter(location__range=5)
------------------------------------------------------------
Traceback (most recent call last):
  File "", line 1, in
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 141, in filter
    return self.get_query_set().filter(*args, **kwargs)
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/query.py", line 556, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/query.py", line 574, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1152, in add_q
    can_reuse=used_aliases)
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1092, in add_filter
    connector)
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/sql/where.py", line 67, in add
    value = obj.prepare(lookup_type, value)
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/sql/where.py", line 316, in prepare
    return self.field.get_prep_lookup(lookup_type, value)
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/fields/related.py", line 136, in get_prep_lookup
    return [self._pk_trace(v, 'get_prep_lookup', lookup_type) for v in value]
TypeError: 'int' object is not iterable

That’s a pretty exotic looking error in Django’s internals, but it didn’t take long to figure out that using location__range is making Django think I want to use the range field lookup on Location.id instead of the field I defined in the model. I expect a similar problem would arise if I had a field named “in”, “gt”, or “exact”, for example.

The solution is simple enough, but didn’t occur to me until searching Google and the Django documentation, and ultimately scouring the Django source code failed to yield any clues. If you ever encounter this problem, simply explicitly specify an exact lookup:

>>> Project.objects.filter(location__range__exact=5)
[< Project: abc>, > Project: def >]

Django: don’t use distinct and order_by across relations

I needed to get a list of project objects that had Time objects attached to them that had been updated by a specific user. I wanted the list to be ordered by the most recently updated Time object, and importantly, I wanted the list of project objects to be distinct (since there are multiple time objects attached to any one project).

I was trying to make the following query work in django:

Project.objects.filter(time__user=user).distinct().order_by('-time__date')

As the note here describes, this particular combination (distinct and order_by on a related field) doesn’t work so well. The related table (Time, in this case) columns are being added to the query’s SELECT clause, giving me multiple copies of projects that I wanted to be distinct.

There is a Django feature request to support named fields in the call to distinct, but it is not incorporated into trunk yet, mostly due to database backend support.

After some searching and pondering, I was able to get the same list of projects using aggregates instead:

Project.objects.filter(time__user=user).annotate(
                models.Max("time__date")).order_by('-time__date__max')

This solution to the problem doesn’t seem to be suggested often, so I thought I’d take the time to mention it.

Rendering Django models to JSON

I recently discussed a simple template tag that allows rendering “raw” jquery templates inside a Django template, even though the syntax overlaps. A commenter asked for a Pypi package. It was such a simple template tag that I didn’t want to maintain a separate package for it, but it turns out, this tag fits in very well with the next step in the project. The short story is that the tag is now available on Pypi.

The tag is distributed with a project I called Djason (named for Django, JSON, and former Arch Linux Developer, Jason Chu, and because djson was already taken). Djason’s primary purpose is to serialize Django models to a json client format that is friendly to client libraries, especially to jquery-tmpl. It is essentially just a fork of Django Full Serializers with a patch applied, and some minor edits to simplify the output json. I install it as a Django serializer in settings.py using:

SERIALIZATION_MODULES = {
    'djason': 'djason.json'
}

And in my code:

from django.core.serializers import serialize

    content = serialize("djason", project,
            relations=["location_set", "company", "project_type"])
    return HttpResponse(content)

I’ll probably be adding some additional code to this package to fit in with my still cloudy ideas about the proper separation between client and server code. I’ll also likely be rewriting portions of the forked project to appease my personal sense of style.