Posts tagged ‘python’

Pushing Python Past the Present

This is the first time I’ve inserted myself into an exchange between bloggers I don’t know. This topic interests me and I have something to add. Most importantly, I found an alliterative title. So I thought I’d give it a go.

I first saw Calvin Spealman’s article I am worried about the future of Python. I suspect that upon reading this article, Guido got into his famous time machine to give his Pycon 2012 Keynote speech, accusing Calvin of trolling.

This article was shortly followed by Tim McNamara’s Python is doing just fine which (likely unintentionally) summarizes Guido’s talk.

Finally, Nick Coghlan came out with an extensive summary article called Python’s Future: A Global Perspective.

And now there’s me. All I want to do is push people to start supporting PyPy. The PyPy developers have worked on a variety of technologies that can be made into production-ready products that address most issues (real or imaginary) that people see with Python.

They’ve already solved speed and I suspect the next release of PyPy will solve memory. They’ve made huge but mostly incomplete progress in a lot of other areas, including the ones the above bloggers have mentioned.

Concurrency

I have had a lot of trouble dealing with parallelism in Python. Everything I have tried has either been a hack or required a hack to work with it. In my experience, the multiprocessing module does not work in large-scale production. However, it works a lot better on PyPy than it does on cPython, at least for me. The various async solutions can only use one processor and are therefore, in my opinion, no more exciting than GIL-throttled Threads.

In addition to better multiprocessing, PyPy also supports stackless extensions out of the box. And I haven’t even mentioned Armin Rigo’s mysterious Software Transactional Memory implementation.

Concurrency is a sore point in Python. There are solutions. PyPy is capable of doing those solutions better.

Web

I think it’s unfortunate that the only way to code for a web browser is to use JavaScript or a language that compiles to JavaScript. I feel there is no real reason browsers can’t support arbitrary <script type=""> tags. I discussed this last March.

With development, PyPy can support Python in the browser. The mostly-finished but untested sandboxing feature of PyPy can be adapted for in-browser execution. I know some experimenting was done on this in 2009 that proved it’s possible. We just need to take it up and make it happen.

Mobile

PyPy’s speed makes it theoretically possible to run it effectively on mobile platforms. Further, it can be compiled to arbitrary execution environments including JVM and .Net. There is no reason that with some development, PyPy couldn’t run on the Dalvik JVM. I’m sure it’s even possible to run it on iOS.

Interestingly, one of the trolls Guido mentioned in his talk was “PyPy should become the default Python implementation”. He debunked this readily by asking the audience how many people use PyPy in production. Nobody moved.

PyPy is ready for production use, but it is not widely used. I think this is largely because the primary PyPy developers are way more excited about creating new features than they are about marketing the current platform or polishing up nearly finished features like sandboxing or a Dalvik backend. They are visionaries, not finishers.

I say this with a great deal of respect. I am not calling for these guys to change their focus. What I want is for other people from the Python community to join the PyPy project as finishers. It needs people that can make sure these nearly-finished features are working, production-ready and more importantly: documented.

I’ve been meaning, for months, to become one of these people. Unfortunately, I’ve prioritized other things including my job and my upcoming book. It’s still on my radar, though, and I hope that after reading this article, you, too, are thinking about make PyPy the future of Python.

So, what can you do?

Use PyPy
PyPy is capable of being your default Python 2 interpreter for most common tasks. Use it. If it doesn’t work for a given project, get on #pypy, they will help you fix it. It’s even more exciting if you can use PyPy features that are not currently available in cPython, such as the stackless extensions.
Evangelize PyPy
Tell people how well PyPy is working for you. Write articles like this one.
Document PyPy
The PyPy website contains a lot of documentation, but it’s rather intimidating and unreadable to the uninitiated. PyPy is just Python, but it’s also much more than Python. Somebody with some writing skills needs to get in there and make it digestable.
Design PyPy
Seriously, pypy.org does not need to look like something out of 2005.
Develop PyPy
I’ve hacked on PyPy. It’s not scary. Work on features like numpy or Python 3 support that are the current developer’s focus. Better yet, work on finalizing features like sandboxing or alternative backends that are finished but not quite tested.

If you have an hour or two free this weekend, instead of writing about how Python is great or not great, or not going to continue to be great, do something with PyPy. Make it happen.

How (not) to screw up timezone processing in Python

If you use pytz without reading the documentation, you might think you can do this:

cltime = datetime.datetime(
        2012,9,26,1,15,0,
        tzinfo=pytz.timezone('America/Santiago'))

This is an easy way to utterly screw up your timezones.

Never pass a tzinfo into the datetime constructor. Here’s why:

>>> cltime = datetime.datetime(
        2012,9,26,1,15,0,
        tzinfo=pytz.timezone('America/Santiago'))
>>> cltime.astimezone(pytz.timezone('US/Pacific')).isoformat()
'2012-09-25T22:58:00-07:00'

That’s simply the wrong time. 1:15 Chilean time should be 21:15 Pacific the previous day, not 22:58.

For whatever reason, the wrong tzinfo is attached to the object. Compare:

>>> cltime.tzinfo
<DstTzInfo 'America/Santiago' SMT-1 day, 19:17:00 STD>
>>> datetime.datetime.now(pytz.timezone('America/Santiago')).tzinfo
<DstTzInfo 'America/Santiago' CLST-1 day, 21:00:00 DST>

The correct way is to use pytz.localize():

>>> correct_cltime = pytz.timezone('America/Santiago'
        ).localize(datetime.datetime(2012,9,26,1,15,0))
>>> correct_cltime.astimezone(pytz.timezone('US/Pacific')).isoformat()
'2012-09-25T21:15:00-07:00'
>>> correct_cltime.tzinfo
<DstTzInfo 'America/Santiago' CLST-1 day, 21:00:00 DST>

The pytz documentation does not indicate if datetime.datetime.now() creates the correct timezone, so I tested it:

>>> for tz in pytz.all_timezones:
...     assert datetime.datetime.now(pytz.timezone(tz)
...         ).tzinfo == pytz.UTC.localize(datetime.datetime.utcnow()
...         ).astimezone(pytz.timezone(tz)).tzinfo
...     
>>>

The assertion never failed, so it is safe to use datetime.datetime.now(pytz.timezone("[timezonename]")) to generate the current date in a specific timezone.

In general, you should store dates in UTC format and convert them to the user’s timezone at the latest possible time. Try to never store naive datetimes. if you need the current time in UTC, use

pytz.UTC.localize(datetime.datetime.utcnow())

because utcnow() returns a naive datetime. Localizing it to UTC adds the correct timezone to it.

pytz.UTC.localize(datetime.datetime(2012,9,26,1,15))

is the correct way to construct a datetime in the UTC timezone. Technically, you CAN pass tzinfo to the datetime constructor if tzinfo is UTC. This is because UTC does not have daylight savings time. However, for consistency, do not do this; do not ever pass tzinfo into a datetime constructor.

Converting to or from UTC is simple, using astimezone:

>>> utctime = cltime.astimezone(pytz.UTC)
>>> utctime
datetime.datetime(2012, 9, 26, 4, 15, tzinfo=<UTC>)
>>> utctime.astimezone(pytz.timezone('America/Santiago'))
datetime.datetime(2012, 9, 26, 1, 15, tzinfo=<DstTzInfo 'America/Santiago' CLST-1 day, 21:00:00 DST>)

Trembling Aspen

Aspen is a unique Python web framework developed by Chad Whitacre. I am still evaluating it, but one thing I can say unequivocally is that it’s different. Very different. Aspen divorces itself from the MVC pattern that every major — and almost every minor — web framework has been relying on for the past decade. While each framework has its perks and annoyances, they all have this underlying MVC-like interface. Aspen throws that all away.

Aspen lodges itself in a small niche between php-like spaghetti code and a complicated package structure full of python models, views, controllers, and templates. In Aspen, template and control code live in the same file, but in separate pages, called Simplates. In a rebellion against regular expression url parsing, Aspen uses the filesystem to define url structure and paths; like in the old days of statically served HTML files.

One thing I love about Aspen is how easy it is to get up and running. Aspen will serve a single HTML file with no python code without any boilerplate. It serves static files for css, javascript, and images just as easily. To add a dynamic page to your Aspen powered site, you simply use the aforementioned simplates, which include python controllers and Tornado inspired templates in the same file. Aspen provides numerous service connectors, so deployment can also be trivial.

Another great feature of Aspen is its documentation. At first glance, the docs seem sparse and rather limited. However, they are well organized, and every page seems to answer every question I have on the topic succinctly and quickly. Chad seems to have a terrific minimalist policy of telling you exactly what you want to know in as few words as possible. As a writer, I know just how difficult this is to accomplish without repeating oneself.

On the negative side, I do have misgivings as to how well Aspen’s filesystem urls it can scale. Chad has assured me that Aspen is up for running large sites. I appreciate the power of regex urls, and I don’t necessarily like having arbitrary file extensions in URLS. I think locators should be succinct and divorced from implementation details. Ideally, I believe there may be a happy medium between Aspen’s enforced folder structure, and the overly-abstract and overly-complicated url scheme Django and its kin provide.

Another downside is that I haven’t figured out a good way to unit test Aspen simplates. You can refactor much of the code into backend libraries and test that, of course, or you can start the service running and mock requests. However, I miss the TestClient feature from Django that allows me to mock a request without running the server. It provides a better analysis of code coverage, and is closer to the actual code.

Now, with the review over, I’d like to introduce a little project I’ve started to provide some vital features that Aspen core is missing.
Trembling is kind of a catch-all project with a few basic helpers and a few useful projects. Namely, trembling provides:

  • Basic sessions
  • Basic user authentication
  • Less than basic single request messages
  • A simple redirect response

Aspen itself does not define or require a model layer; you’re free to use the ORM, DBAPI connector, or NoSQL database of your choice. Trembling arbitrarily chooses mongoengine for the models. I made no effort to make pluggable backends like Django does; Aspen does not encourage excessive layers of abstraction.

I also don’t know how secure trembling’s session and auth features are. If you find a security hole, pull requests are welcome. Overall, I think the combination of Trembling and Aspen will be very useful for my future basic sites. I would also like to try it on a more complicated project just to see how well it performs.

Easily changing virtualenvs

The workon script available in virtualenvwrapper is a useful tool for switching virtual environments on the fly. However, it fails for the workflow I prefer, so I wrote my own.

I prefer to keep my virtualenvs in the same folder as whatever source code or project they apply to. Normally, I put them in a folder named venv, although sometimes I use different names, most often when I have multiple virtualenvs applying to the same source tree (e.g. venv27, venvpypy).

My directory structure therefore looks like this:

/home/dusty/code/
+ project1
| + src
| | + pkg1
| | + pkg2
| + doc
| + venv
+ project2
| + src
| + venv27
| + venvpypy
+ project3
| + venv

… and so on

The problem is, I may be anywhere in the folder tree when I want to activate a virtualenv. This means I’m doing stuff like . ../../venv/bin/activate one time and venv/bin/activate another time. Trying to remember which parent directory I want often requires several tab completions to see the directory listing. This is annoying. So I wrote a bash function to activate the optionally named virtualenv in the current directory or any parent directory.

I avoid bash whenever I can, so this may not be the prettiest bash script you’ve ever seen. ;-)

function v {
  # activate a virtualenv
  # usage: v
  #   activate the virtualenv named venv
  # usage: v venvname
  #   activate the virtualenv named venvname
  name=venv
  if [ $1 ] ; then
    name=$1
  fi
  olddir=$(pwd)
  quit=0
  deactivate &amp;&gt;/dev/null
  cwd=$(pwd)
  while [ $quit -eq 0 ]
  do
    cd $cwd
    if [ $cwd == '/' ] ; then
      quit=1
    fi
    if [ -e $name ]  ; then
      source "$name/bin/activate"
      quit=1
    fi

Put this in your ~/.bashrc. Typing v on the command line will look in the current and all parent directories for a directory named venv, and will activate that venv. Typing v [venvname] will walk the same tree looking for a directory named [venvname] to activate.

If you want to make it prettier, feel free to fork it on github.

Hacking on PyPy

In another great Pycon2012 keynote, David Beazely asked the question, “is PyPy easily hackable?” After a great talk, he answered with a decisive, “I still don’t know.” Having sprinted on Python I’d like to answer his question in a bit more detail.

I love David’s presentation style. He has a novel method of using phrases like, “blow your mind” and “this is really scary” repeatedly until they lose their meaning and you no longer feel mindblown or scared. A variety of factors, including Beazely’s thorough keynote address motivated me to join the PyPy team during the Pycon developer sprints.

I’d like to clear up one oversight in Dave’s otherwise impeachable talk. One of the PyPy devs, Holger Kregel explained to me that PyPy does not have over 1 million lines of code. I don’t have exact numbers, but for “historical reasons”, a non-python file containing Base64 encoded data was given a .py extension. When excluding this file from the line count, around half a million lines of actual Python code exist, and about a quarter of these are tests.

I was surprised how trivial it was to get started hacking on PyPy. I don’t really grok the many layers of the translation toolset and PyPy interpreter, but it’s pretty clear that the layers are well separated. I was hacking on the py3k branch of PyPy. I am happy to admit I was working primarily on changing print statements to print() functions and commas in exceptions to the as keyword.

Here are the steps to start hacking on PyPy. Notice that the hour-long translation step is not part of the procedure. PyPy has a solid test framework, and the PyPy crew are focused on a 100% test-driven-development paradigm.

  1. Clone pypy (this takes a while):
    hg clone https://bitbucket.org/pypy/pypy/
  2. Pick a branch to work on. There are about 80 branches. I don’t know what they all do. Popular ones during the sprints included py3k and numpy-ufuncs2
  3. Pick a feature to work on. For py3k support, the list of failing tests in the buildbot is a good place to start. Numpy programmers had a list of fuctions that needed implementing, but I can’t find the link. Ask on IRC, the PyPy crew are very helpful. The bug tracker contains many features and issues that need addressing
  4. Add /path/to/pypy/ to your path so you can run the pytest.py command
  5. cd into the directory indicated in the buildbot output and run pytest.py path/to/test.py -k testname
  6. The test will likely fail. Hack away and fix it.
  7. When the test passes, commit, push to a bitbucket repo, and issue a pull request.
  8. Repeat!
  9. There are quite a few cons to working on this project. If you run hg in the pypy/modules/ directory, it will try to pick standard library modules from pypy and choke horribly. The pypy developers don’t really believe in documenting their code. Being able to tell the difference between rpython and python (which have identical syntax) is important. In general, if a module starts with “interp_” it contains rpython, but if it starts with “app_” it contains python. The code does not appear to be well-documented.

    If you are hacking on Python 3 support, you need to bear in mind that the PyPy interpreter is written in Python 2. You are working on a Python 2 application that executes Python 3 bytecode!

    On the positive side, rPython and Python are much easier to read and write than C. The PyPy devs are brilliant, but not intimidating. They are so confident in their test suite that they are comfortable programming in a “cowboy coding” style, hacking randomly until all the tests pass. Any one layer in the toolchain is easy to understand and develop. The IRC channel is full of friendly, knowledgable, helpful people at any time of day.

    Overall, I am much less intimidated by this project than I was before I started the dev sprints. I still can’t answer the, “Is Python easily hackable?” question fully. It’s certainly easy to get started, but I don’t know how easy it is to become intimate with the project. Dave Beazely’s keynote made PyPy more approachable, and I approached it. Hopefully this article will encourage you to do the same.

Why we need Python in the Browser

In his Pycon 2012 keynote speech on Sunday, Guido van Rossum covered many of the open “Troll” questions related to the Python community. I’ve had occasion to either complain about or defend all of the topics he covered, and he did a wonderful job of addressing them. The issues could largely be divided into two categories: those that come from a fundamental misunderstanding of why Python is wonderful (e.g. whitespace), and those that are not an issue for the core python dev team, but are being actively worked on outside of core Python (e.g event loop).

And then there’s the question of supporting Python in the web browser. I honestly thought I was the only one that cared about this issue, but apparently enough people have complained about it that Guido felt a need to address it. His basic assertion is that the browsers aren’t going to support this because nobody uses it and that nobody uses it because the browsers don’t support it.

This is a political problem. Politics shouldn’t impact technical awesomeness.

The fallacious underlying assumption here is that modern HTML applications must be supported on all web browsers in order to be useful. This is no longer true. Web browser applications are not necessarily deployed to myriad unknown clients. In a way, HTML 5, CSS 3, and DOM manipulation have emerged as a de facto standard MVC and GUI system. For example, many mobile apps are developed with HTML 5 interfaces that are rendered by a packaged web library rather than an unknown browser. Client side local storage has created fully Javascript applications that require no or optional network connectivity. There are even situations where it may not be necessary to sandbox the code because it’s trusted. Many developers create personal or private projects using HTMl 5 because it’s convenient.

Convenient. It would be more convenient if we could code these projects in Python. Web browsers can be viewed as a zero install interface, a virtual machine for these applications. Such a VM has no reason to be language dependent.

It is simply unfair to all the other programming languages and coders of those languages to say, “we can’t displace Javascript, so we won’t try.” Web browsers have evolved into a virtualization layer more like operating systems than single programs. While it is true that the most restrictive operating systems only permit us to code in Objective C, in general, it is not considerate to restrict your developers a single language or environment.

It is time (in fact, long overdue) for Python to be supported in the browser, not necessarily as an equal to Javascript, but as an alternative. The web is a platform, and we must take Guido’s statement as a call to improve this platform, not to give up on it.

Update: I just stumbled across http://www.gnu.org/software/pythonwebkit/ and I can’t wait to play with it!

Update 2: From the number of comments on this article, it appears that my article has hit some web aggregator. To those users suggesting python to javascript compilers, I’ve been a minor contributor to the pyjaco project, a rewrite of the pyjs library. It has the potential to be a great tool, and the head developer, Christian Iversen is very open to outside contributions. Let’s make it better!

PyCon 2012

Hello Arch Linux community! If any of you are attending Pycon 2012 in Santa Clara, CA this week, make sure to bump into me. Or contact me to schedule a meetup.

Pyjaco in a real app: Todos with local storage

I didn’t get the memo, but there appears to be a movement to demonstrate emerging web technologies with a simple todo list application, much as hello world is used to introduce programming languages.

In my last post, I introduced using jQuery with Pyjaco, the PYthon JAvascript COmpiler. Since then, I’ve made several contributions to the project and have been involved in diverse discussions with Pyjaco developers regarding the current and future status of the project. This post goes further by acting as a tutorial for writing a basic todos app using Pyjaco.

Pyjaco is alpha software. It is hard to write valid code, and harder to debug. I’ve managed to both lock up Firefox and hard crash it while using the Pyjaco library.

On the positive side, Pyjaco is under active, rapid development. The head developer, Christian Iversen is extremely responsive to both questions about Pyjaco, and to code contributions. This is a project with a lot of potential, and I see it as the current best bet for Python programmers hoping to avoid javascript one day in the future.

In spite of the hiccups, it is possible to generate a working javascript app using just Pyjaco. Here’s how.

Let’s start:

mkdir pyjados
cd pyjados
virtualenv2 venv --distribute --no-site-packages
source venv/bin/activate
pip install git+https://buchuki@github.com/buchuki/pyjaco.git@run_script

First we create a directory to work in and install a virtualenv. Pyjaco does not currently work with python 3, so in Arch Linux, I use the virtualenv2 command. We then activate the virtualenv and install the pyjaco package. Here I am installing from my personal fork, as it contains some changes for generating the built-in standard library that have not yet been merged upstream. You should normally install directly from chrivers’s git repository using pip install git+git://github.com/chrivers/pyjaco.git.

Now let’s create a basic HTML 5 page with jQuery loaded:

<!DOCTYPE html>
<html>
  <head>
    <title>PyJaco Todo List Example</title>
    <script type="text/javascript" src="http://code.jquery.com/jquery-1.6.4.min.js"></script>
  </head>
  <body>
    <h1>PyJaco Todo List Example</h1> 
  </body>
</html>

We can load this in our web browser using a file:// URL. This is the only HTML page in our app, and it can be refreshed to load our changes as we work.

Pyjaco doesn’t simply translate Python into Javascript. Rather, it creates a basic standard library of Python-like objects that are utilized in the compiled javascript code. I wasn’t too keen on this idea when I first heard it, as it introduces a dependency that currently weighs in at 65K before minification and compression. While this is not a terribly heavy library, there are efforts under way to shrink the builtins or to dynamically generate it to contain only those builtins that your code actually touches. At any rate, we need to ensure this library is available to our code. First we generate the library:

pyjs.py --builtins=generate --output=py-builtins.js

pyjs.py is the name of the pyjaco command. It is expected to be renamed to pyjaco in the future. The --builtins=generate option tells pyjaco to generate the standard library, while the --output flag provides the filename for the new library file:

$ ls
index.html  py-builtins.js  venv

We then need to load this library in the head of our html file. Let’s also load the future pyjados.js script at this time:

  <head>
    <title>PyJaco Todo List Example</title>
    <script type="text/javascript" src="http://code.jquery.com/jquery-1.6.4.min.js"></script>
    <script type="text/javascript" src="py-builtins.js"></script>
    <script type="text/javascript" src="pyjados.js"></script>
  </head>

Now, before we start coding the Python file that will be compiled to Javascript, I want to discuss what I consider to be the most confusing aspect of Pyjaco development. There are basically two types of variables in Pyjaco, Javascript variables, and Python variables. Javascript variables refer to “normal” variables that you would call in Javascript. These include alert, window, document and the like, as well as variables in third-party Javascript libraries, such as the ubiquitous jQuery. Further, any attributes on those objects are also Javascript variables, and the return value of any methods will also be Javascript variables.

Python variables, on the other hand, refer to any variables that you define in your Python source code. If you create a dict or a list, for example, it will be compiled to a list or dict object from the standard library we just generated. In the compiled script, of course these Python variables are represented by Javascript objects, but from the point of view of a Pyjaco coder, it is important to keep the two types of files separate. Almost all the bugs I have encountered in my Pyjaco code have been caused by confusing the two types of variables.

The distinction between python and javascript variables introduces a couple of complications to writing Pyjaco compatible python code. First we need to flag all of our Javascript variables using a decorator on methods that access them. Second, we need to explicitly convert our variables between Javascript and Python any time we access one from the other. I’m told that this conversion can — and one day will — be done automatically by the pyjaco multiplexer, but in the meantime, we need to make it explicit. We do this by using two javascript functions supplied with the standard library we just generated, appropriately named js() and py(). You will see examples of these shortly.

When I finally figured out the distinction, my first thought was, “ok, let’s prefer to always work with python variables.” Therefore, in my initialization code, I tried jQ=py(jQuery). Unfortunately, jQuery is a rather large object, and the py function apparently recursively converts all attributes from javascript to python. I ended up with a stack overflow.

Now, let’s create our first python code and watch it compile to Javascript. Name the file pyjados.py:

def setup():
    print "Pyjados Hello World"
 
jQuery(js(setup));

First we write a python function named setup. This function is a python object. jQuery is a javascript object that expects a javascript object as input. Therefore, we wrap setup in a js() call and pass the result into the jQuery function. jQuery will now run setup when document.ready is fired.

Now we compile the code using the following command inside our activated virtualenv:

pyjs.py --watch pyjados.py --output pyjados.js

You’ll notice the command doesn’t exit. That is the --watch option at work. If you now make a change to pyjados.py and save it, it will automatically recompile it. The output file pyjados.js is regenerated each time. This is the file we included in our html file. So now, open that html file in a web browser using a file:// url. Make sure the Javascript console is displayed and reload the page. You should see the words “Pyjados Hello World” printed on the console. Pyjaco automatically compiles print statements into console.log output.

Before we start implementing our Todo list, let’s look at an example of accessing a javascript variable inside a function. Change setup.py to utilize alert, as follows:

 
@JSVar("alert")
def setup():
    alert(js("Pyjados Hello Alert")
 
jQuery(js(setup));

Did you look closely at that code? There is a missing close bracket on the alert line. You’ll see the syntax error in your console where pyjs.py is watching the compiled code. Add the bracket and let it automatically recompile itself:

 
@JSVar("alert")
def setup():
    alert(js("Pyjados Hello Alert"))
 
jQuery(js(setup));

Let’s analyze this snippet. First, notice how we told the compiler that alert is a Javascript variable when used inside setup(). This is a bit odd, since the JSVar decorator is never actually imported into the namespace. This is a bit of magic in the Pyjaco compiler, just pretend it has been imported.

Second, notice that since alert has been flagged as a JSVar, it must accept a Javascript variable. However, the string “Pyjados Hello Alert” is a Python variable. Therefore, we convert it using js() as we pass it into the alert call.

Now let’s prepare to create some working todo-list code. Start by adding a form for submitting todos and a list to render the todos to the html body:

  <body>
    <h1>PyJaco Todo List Example</h1> 
    <form id="add_todo_form">
      <input type="text" id="add_box" placeholder="Add Todo", autofocus="autofocus">
      <button id="add_button">Add Todo</button>
    </form>
    <ul id="todo_items"></ul>
  </body> 

Nothing too exciting here. Note the ids on the elements, since we’ll be utilizing these from Pyjaco using jQuery selectors.

Now back into the python file. Let’s create a class to manage the various todo elements:

class TodosApp:
    @JSVar("jQuery", "js_add_form")
    def __init__(self):
        js_add_form = jQuery(js("#add_todo_form"))
        js_add_form.submit(js(self.add_todo))
 
    def add_todo(self, event):
        print "form submitted"
        return js(False)
 
def setup():
    todo_app = TodosApp()
 
jQuery(js(setup));

The __init__ function hooks up the form’s submit button to a method on the object. Notice that we need to flag not just jQuery, but also js_add_form as a javascript variable. Pyjaco does not (currently) know that a javascript variable is returned when calling a method on an existing javascript variable. I like to add the js_ prefix to variable names to help remind myself that this is a javascript variable.

In an ideal world, we could convert this variable to a Python variable using py(), but as noted earlier, calling py on a jQuery object results in a stack overflow or browser crash.

Also pay attention to the way we wrap the self.add_todo method name in a js() call when we pass it into the submit handler. The submit method is a javascript function expecting a javascript object.

The def add_todo method has its single parameter flagged as a @JSVar, since the method is being called internally by jQuery when the event occurs. We also wrap the False return value (to prevent event propogation on the submit handler) in a js() call so that jQuery recognizes it as a javascript false rather than a (true) object named False.

Try the code. Ensure the compiler recompiled it, and reload the html file. Enter some characters into the text box and use the Enter key or the Add Todo button to submit the form. The words form submitted should be displayed in the javascript console.

Now let’s actually store and render a newly added todo. The todos are stored in memory in a python dict object. Initialize this object by adding the following two lines of code to the end of __init__:

    self.todos = {}
    self.next_id = 1

And rewrite add_todo as follows as well as a new method named render

    @JSVar("event", "js_add_box")
    def add_todo(self, js_event):
        js_add_box = jQuery(js("#add_box"))
        self.todos[self.next_id] = py(js_add_box.val())
        js_add_box.val('')
        js_add_box.focus()
        self.next_id += 1
        self.render()
        return js(False)
 
    @JSVar("js_todo_items")
    def render(self):
        js_todo_items = jQuery(js("#todo_items"))
        js_todo_items.html("")
        for id, todo in sorted(self.todos.items()):
            js_todo_items.append(js('<li>%s</li>' % (id, todo)))

Note that the todos dict is a Python object, so when we insert the value of the js_add_box into it, we must convert it from a javascript object using py(). Also note how, because we are writing in a python function, manipulating the python value self.next_id requires no conversion, and calling the python function self.render is also clean.

In the render function itself, I think it’s pretty cool that string formatting using % is supported by pyjaco (as an aside, the str.format method introduced in python 2.6 is not yet available) and that the python sorted() function is available. Note also how we can loop over items() on the self.todos dictionary just as if we were using a normal python dictionary.

Now let’s add the ability to complete todos. Let’s start by adding a template string as a class variable, and use that string inside the render function. This illustrates that pyjaco supports class variables:

class TodosApp:
    list_item_template = """<li>
    %(text)s
    </li>"""

and we change the for loop in render to:

        for id, todo in sorted(self.todos.items()):
            js_todo_items.append(js(TodosApp.list_item_template % {
                "id": id,
                "text": todo}))

Reload the page again and notice how checkboxes have been displayed beside each todo. The next step is to make clicking these boxes actually complete the todos. We add a couple lines to our __init__ method to connect a live click event to the checkbox items, which now looks like this:

    @JSVar("jQuery", "js_add_form", "js_checkbox")
    def __init__(self):
        js_add_form = jQuery(js("#add_todo_form"))
        js_add_form.submit(js(self.add_todo))
        js_checkbox = jQuery(js("input[type=checkbox]"))
        js_checkbox.live("click", js(self.complete_todo))
        self.todos = {}
        self.next_id = 1

Don’t forget to add js_checkbox to the JSVar decorator.

The complete_todo method looks like this:

    @JSVar("event", "jQuery", "todo_item")
    def complete_todo(self, event):
        todo_item = jQuery(event.target).parent()
        id = int(py(todo_item.attr("id"))[5:])
        del self.todos[id]
        todo_item.delay(1500).fadeOut("slow")

The first line is using exclusively javascript arguments, and returns the <li> element containing the checkbox that was clicked. The id = line converts the javascript string id attribute of this element (which looks like “todo_5“, as defined in list_item_template) into the python integer id of the todo. The remaining lines simply remove that todo from the internal list and from the DOM, after a 1.5 second delay.

In fact, we now have a fully functional todo list that allows adding todos and checking them off. Now, as a bonus, let’s try hooking this up to the HTML 5 localStorage object so that the list is maintained across page reloads. We start by adding a store() method to our class:

    @JSVar("localStorage", "JSON")
    def store(self):
        localStorage.setItem("todolist", JSON.stringify(js(self.todos)))

The main line of code is easiest to read from the inside out. First we convert the self.todos dict to a normal javascript object using the js() function. Then we call JSON.stringify on this object to create a string suitable for insertion into localStorage.

Now add this call to the end of the two methods that manipulate the todo list, add_todo and complete_todo:

        self.store()

.

Refresh the page, add a couple todos, and inspect the localStorage object in your console. You should see the stringified dict in the todolist value.

Now all we have to do is ensure the self.todos dict is loaded from localStorage when the app is initialized. Add the following to the end of the __init__ method (make sure to add js_stored_todos to the JSVars decorator):

        js_stored_todos = localStorage.getItem("todolist")
 
        if js_stored_todos:
            stored_dict = dict(py(JSON.parse(js_stored_todos)))
            self.todos = dict([(int(i), stored_dict[i]) for i in stored_dict.keys()])
            self.next_id = max(self.todos.keys()) + 1
 
        self.render()

Note that calling py() on the output of JSON.parse creates a python object, not a python dict. The code is therefore wrapped in a call to dict(), which converts the object to a dictionary.

Unfortunately, the resultant dict contains keys that are strings, whereas our original dict used integer keys. So a pure-python list comprehension is used to convert the dictionary to one with integer keys. This line is a bit hard to read, but I wanted to include it to demonstrate that Pyjaco can parse list comprehensions. Finally, we set self.next_id using the python max() call, which Pyjaco also automatically translates into javascript.

Try it out. Load the pyjados HTML file, add some todos, check a few of them off, then close and reload the web browser. Your todos will be stored!

I hope you’ve enjoyed this introduction to Pyjaco. It is a nice tool with a lot of potential. Currently, I find writing Pyjaco code to be approximately equally tedious to writing Javascript code. However, I feel that as I learn the ins and outs of Pyjaco, and as the developers continue to refine and improve the compiler, Pyjaco may one day be a perfectly viable alternative to writing pure Javascript or to the rather too Ruby-esque, but otherwise excellent Coffeescript.

pyjaco and jQuery

After giving up on CoffeeScript, I decided to play around with Pyjaco the Python to Javascript Compiler.

While the code is readable, there is very little in the way of end-user documentation. I hope to address this with this blog post. The Pyjaco examples all embed generated javascript in an html page. I needed a way to generate an external Javascript file as I would include in an HTML file. I also wanted to find out if I could use Pyjaco with jQuery.

The first step was to install a development version of Pyjaco:

git clone https://github.com/chrivers/pyjaco.git

Pyjaco normally requires a generated Javascript file mapping Python builtins to Javascript to be included with the created Javascripts. This must be generated:

cd pyjaco
python2 generate_library.py
cp py-builtins.js ~/pyjaco_test # directory for my new page

The next step was to create an HTML file that included jquery, the py-builtins.js script above, and a yet-to be defined javascript file named clicker.js that will be generated from a yet-to-be-defined python file. I also add a couple of DOM elements (a heading and paragraph) that are to be manipulated via jQuery:

<!DOCTYPE html>
<html>
<head>
<script type=”text/javascript” src=”jquery-1.6.4.min.js”></script>
<script type=”text/javascript” src=”py-builtins.js”></script>
<script type=”text/javascript” src=”clicker.js”></script>
</head>
<body>
<h1>Click me</h1>
<p id=”when_clicked”></p>
</body>
</html>

That’s the easy part. Writing python code that compiles to correct Javascript is the hard part. Pyjaco doesn’t currently provide very useful compile-time errors, and it also does not yet map javascript errors back to the input python.

There are (at least) two ways to compile python code in Pyjaco. The first, which is used in the Pyjaco examples, and appears to be the preferred method at this time is to create a custom main() method that uses various pyjaco.Compiler methods to combine the functions into a string of text. See https://github.com/chrivers/pyjaco/tree/devel/examples for three examples.

However, I was looking to have a complete python file that compiles to a complete javascript file. Pyjaco supports this as well, using the provided pyjs.py script. It took some investigating to understand how to reference javascript variables inside python functions. Decorator syntax is used to expose the variables for jQuery, Math.random, and Math.floor in the following example. The mystifying bit is that because we will be compiling this file to javascript as a string, it is not necessary (or possible) to import the JSVar decorator, as was done in the Pyjaco examples linked above.

# clicker.py
@JSVar("jQuery")
def ready():
    jQuery('h1').click(on_click)
 
@JSVar("jQuery", "Math.random", "Math.floor")
def on_click():
    if jQuery('#when_clicked').html():
        r = Math.floor(Math.random() * 255)
        g = Math.floor(Math.random() * 255)
        b = Math.floor(Math.random() * 255)
        color = "rgb(%d, %d, %d)" %(r,g,b)
        jQuery('#when_clicked').attr('style', 'background-color: ' + color)
    else:
        jQuery('#when_clicked').html("you clicked it!")
 
jQuery(ready)

Notice that I’m using the jQuery function instead of the $ alias, since $ is not a valid variable name in Python. This is a rather odd looking mix of Python and Javascript functions, but it works.

I had to repeat the compile and test step a few times before coming up with the above file. The script can be compiled using the pyjs.py that comes with the pyjaco source distribution (and, thanks to a simple patch I submitted, will come with the binary distribution in the next release.) Here’s how the script is run:

python2 ~/code/pyjaco/pyjs.py -N --output clicker.js clicker.py

The -N option tells pyjs not to generate the builtin library that we created manually in the first step.

This translation step creates a clicker.js file that looks like this:

var ready = function() {
    var __kwargs = __kwargs_get(arguments);
    var __varargs = __varargs_get(arguments);
    var $v1 = Array.prototype.slice.call(arguments).concat(js(__varargs));
    jQuery("h1").click(on_click);
return None;
}
var on_click = function() {
    var __kwargs = __kwargs_get(arguments);
    var __varargs = __varargs_get(arguments);
    var $v2 = Array.prototype.slice.call(arguments).concat(js(__varargs));
    if (bool(jQuery("#when_clicked").html()) === True) {
        var r = Math.floor((Math.random()) * (255));
        var g = Math.floor((Math.random()) * (255));
        var b = Math.floor((Math.random()) * (255));
        var color = str('rgb(%d, %d, %d)').PY$__mod__(tuple([r, g, b]));
        jQuery("#when_clicked").attr("style", ("background-color: ") + (color));
    } else {
        jQuery("#when_clicked").html("you clicked it!");
    }
return None;
}
jQuery(ready);

I find this rather unfortunately difficult to read. There is code for argument parsing that would not have been needed if I had hand-written javascript. Further the use of “mock” python builtins makes the javascript look less javascripty. However, the original python file looks much more readable than an equivalent javascript one would. I am hopeful that improvements to pyjaco will cause it to generate more readable javascript with less extraneous code.

The entire example can be found on my github fork

Christian Iversen is actively working on Pyjaco right now. I am excited about this project and hope that further community involvement will help it evolve into a practical and useful tool. I intend my next patch to be an autocompile tool that monitors files in one directory for change and outputs .js files in another directory, one of CoffeeScript’s killer features. I am also considering a port to Python 3.

I can’t use CoffeeScript

I had some space between wrapping up my last contract on December 20, and starting my new job on January 16. I decided it was finally time to build a simple task management system, something I’ve attempted to do and never finished several times before. I currently use RememberTheMilk, which is a lovely service, but I don’t like paying them for mobile access. Further, even though I trust this small company, I see no reason to share intimate information about the tasks I accomplish every day with them. My long term goal is to pull as much of my personal data out of the cloud as I possibly can. This project is a step towards that goal.

There are numerous open source task manager apps out there that I’m sure would suit my not-too-exotic tastes. However, I also wanted to take this opportunity to learn a bunch of new technologies for an offline-enabled and mobile-enabled web application.

Therefore, I’ve spent most of my vacation time so far researching some technologies I haven’t had a chance to explore in the past year. JQuery Mobile was at the top of the list. I think it’s a lovely framework and I expect to continue using it.

I also wanted to try out CoffeeScript, as I’ve always hated writing Javascript, and I wanted to use some sort of client-side ORM for localstorage. I looked at backbone.js, but was more attracted to Spine.js. I have spent three days studying and playing with these two technologies. I am still undecided about Spine.js, but I have come to the conclusion that CoffeeScript is not for me.

I understand all the hype around the project. JavaScript really does suck. And CoffeeScript does suck less, it has pretty language features and it is much more succinct than JavaScript. I can imagine a lot of people being really excited about CofeeScript, especially Ruby and PERL programmers, and possibly even php programmers.

But not Python programmers. I tried to learn Ruby several times, and each time I was left with a foul taste in my mouth. It’s not a bad language, it just doesn’t fit in well with my personal philosophy. My personal philosophy happens to coincide almost exactly with the Zen Of Python. I chose Python because it matched my philosophy… not the other way around.

Like Ruby and PERL, CoffeeScript violently violates what I consider the most important rule of Python: “There should be one — and preferably only one — obvious way to do it.” I’m not going to argue why this is a good idea, I understand that some programmers prefer the “even if I don’t understand it, I can write code that will probably work” paradigm that Ruby promotes.

The simple truth is, writing CoffeeScript leaves me feeling like I’ve done something dirty, no less dirty than writing JavaScript. There is no incentive for me to add a layer of complexity (the CoffeeScript to JavaScript compile step) to my code when I know my code is going to be “ugly” either way.

CoffeeScript is a wonderful idea. It’s far better than JavaScript. It’s just not good enough. Luckily, the Python community is already working on pythonic answers, including the evilly poorly documented pyjaco and the less-than-well maintained pyvascript and pyjamas projects. I hope one of these or a new upstart will soon gain community momentum so frontend development is no longer painful.