Posts tagged ‘python’

Gesture Recognition In Kivy

Kivy is a modern GUI platform that runs on Windows, Linux, Mac, iOS, and Android.

Kivy supports gestures, but the documentation is a as to how to use them. Reviewing the Gesture Board example provides most of the missing pieces if you’re willing to experiment. I have done these experiments and hope this article will make it easier for the next coder.

The kivy.gesture module contains two main classes, the Gesture and GestureDatabase. A Gesture represents the stroke or strokes in a gesture. It maps a sequence of (x,y) coordinates to a normalized representation and can be compared to another sequence of points to determine if the two sequences “match”. In Kivy, gestures can be encoded in base64. This provides an easy way to store and load gestures in source code.

The GestureDatabase is essentially a collection of Gesture objects. Its primary purpose is to compare a new gesture as input by the user to those stored in the GestureDatabase and return the closest matching gesture, if one exists.

Before we can recognize whether a user’s gesture is meaningful, we need some gestures to compare it to. Luckily, the gesture_board.py that ships in Kivy’s examples directory does this for us. Run python gesture_board.py from a terminal. A blank window opens up. Draw a gesture on it.

Have a look in the terminal. There is a variety of output there, but the important one is the long base64 encoded string following the words “gesture_representation:”. Copy that string into a variable in a basic Kivy app:

from kivy.app import App

down_stroke = "eNq1l91u4zYQhe/1IslNjPkfzgtkb<snip>"
square = "eNq1mEluIzcYRvd1EXsT4Z+HC6i3AXyAwG<snip>"


class TestApp(App):
    pass

TestApp().run()

Now let’s set up a quick and dirty GestureDatabase from the given strings. Normally, I’d put these as an instance variable on the App or a specific widget, but for easy illustration, I’ll just toss them into the module:

from kivy.app import App
from kivy.uix.widget import Widget
from kivy.graphics import Line
from kivy.gesture import Gesture, GestureDatabase

down_stroke = "eNq1l91u4zYQhe/1IslNjPkfzgtkb<snip>"
square = "eNq1mEluIzcYRvd1EXsT4Z+HC6i3AXyAwG<snip>"

gestures = GestureDatabase()
gesture = gestures.str_to_gesture(down_stroke)
gesture.name = "down_stroke"
gestures.add_gesture(gesture)
gesture = gestures.str_to_gesture(square)
gesture.name = "square"
gestures.add_gesture(gesture)

.
.
.

The next step is recording the gesture that the user makes. This requires keeping track of the touch down, move, and up events. Let’s create a new widget to handle this (Note, for clarity, I’ve omitted collision detection and error conditions):

from kivy.uix.widget import Widget
from kivy.graphics import Line


class TestWidget(Widget):

    def on_touch_down(self, touch):
        touch.ud['gesture_line'] = Line(points=(touch.x, touch.y))

    def on_touch_move(self, touch):
        touch.ud['gesture_line'].points += [touch.x, touch.y]

    def on_touch_up(self, touch):
        # compare the gestures

The final step is to implement on_touch_up to convert the stroke the user created into a gesture and compare that gesture to those in the database:

    def on_touch_up(self, touch):
        gesture = Gesture()
        gesture.add_stroke(
            zip(touch.ud['gesture_line'].points[::2],
                touch.ud['gesture_line'].points[1::2]))
        gesture.normalize()
        match = gestures.find(gesture, minscore=0.70)
        if match:
            print("{} gesture occured".format(match[1].name))
        else:
            print("No gesture recognized")

This code requires some explanation. The Line object stores it’s points in a one dimensional list, where alternate indexes represent the x and y coordinates of each point. However, add_stroke expects a list of tuples of x and y values. In code, Line stores [x1, y1, x2, y2, x3, y3] while add_stroke expects [(x1, y1), (x2, y2), (x3, y3)]. Hence, the rather complicated call to zip.

The gestures.find call accepts a minscore (1.0 would mean the gesture matched perfectly). It will return the gesture that matches with the maximum matching score, but only if that maximum is above minscore. 0.70 seems to be suitable for basic gestures, though I have been able to confuse a ‘square’ with a ‘circle’ in my experiments.

gestures.find returns either None or a tuple of (score, gesture). Thus, if there is a match we need to pull out the Gesture via match[1].

And there you have it: basic gesture recognition in Kivy.

Unfortunately, the touch events are gobbled up by the gesture code, so if you have a gesture widget that contains other widgets, they won’t receive any events. I have taken a stab at creating a GestureBox widget that passes events through to child widgets. It seems to work for touch events, but deciding whether a motion event

Python on Android? First impressions of Kivy

Kivy is a modern cross-platform Python GUI toolkit that runs on mobile devices (Android and IPhone) and supports modern input events (multitouch, accelerometor). I was really skeptical at first, but I spent the weekend developing an app in Kivy, and now I’m hooked.

In the past, I have tried most of the available Python GUI toolkits (even Swing, through jython). In recent history, I’ve mostly done HTML based interfaces, partially because that’s where the money has been, but mostly because pyQT, pyGTK,wxpython, and tk are all painful to work with.

In Kivy, interfaces are defined in a language called (aptly) the Kivy Language. The best way to describe the Kivy Language is “what HTML5 wishes it was.” It is mostly just a layout language, but it also supports binding properties and events using inline Python snippets. It is indentation defined, like Python, and utilizes a minimal amount of syntax with a maximal amount of readability. It is vaguely reminiscent of CoffeeKup but cleaner, and doesn’t suffer from having to be compiled to HTML. I love how it mimics Python syntax, but also clearly separates presentation from control code.

My favourite feature of Kivy is that properties are events. You can easily hook a property on one widget to the property on another. For example, I wrote a simple demo that connects the value of a slider to the value of a progressbar:

BoxLayout:
    orientation: 'vertical'
    ProgressBar:
        id: bar
        value: 14
        max: 20
    Slider:
        id: slider
        max: 200
        value: 140
        on_value: bar.value = self.value / 10

When the value property on the slider changes (whether because the user moved the slider, or because it was adjusted programmatically in response to some other event), the on_value event is fired. This is connected to the value of the progress bar using a Python snippet.

Only a few lines of Python code are required to to get a Kivy application running once its interface has been defined in Kivy language. The Kivy Hello World example is actually overly complicated because it doesn’t actually use the Kivy language. The bare essentials, if the above file is named sample.kv is:

from kivy.app import App
class SampleApp(App): pass
SampleApp().run()

Just three lines of code. Kivy introspects the class name (convention over configuration) and automatically loads sample.kv as the interface for a class named SampleApp.

The Kivy API is very simple. In fact, my earliest impression was that it was almost amateurish. It felt like if I wanted to do anything serious with it, I’d be disappointed. It’s the kind of API I would teach a child. However, the few features of the API (basically properties, events, and graphics instructions) are extremely well thought out and well-defined. They can be combined like Lego bricks to create widgets as complicated as one could possibly wish. The fact that the library includes a ReStructured Text rendering widget is testament to that fact!

Kivy is extremely well documented. I actually found their pong tutorial exciting. Seeing how simple it was to do each step actually made me giggle out loud. The programmer’s guide and API are well presented as well. Further, if you get stuck, the developers on IRC are terrific. I felt like I was an accepted member of the team after asking only one question.

Drawbacks

It took a bit of fiddling to get Kivy to install correctly under Arch Linux. I suppose if I’d followed the instructions, all would have been well. However, I like to do all my development in a virtualenv, and there is no trivial way to track down and install all Kivy’s dependencies. This is especially compounded by the fact that pygame currently needs to be patched under Arch in order to compile.

There are instructions for packaging your Kivy app onto Android, but I wasn’t able to pull it off, yet. I’ll have to revisit it when I have more time.

I tried some sample Kivy apps in the Android market. They work just fine once they are loaded, but it takes each app several seconds to initialize on my Galaxy Nexus. Hopefully the Kivy team can reduce this load time. I don’t know if Kivy supports displaying a splash screen while stuff is loading; that would certainly enhance the usability.

Most of these issues are solvable. I’m sure the team is actively working on some of them. Kivy is a quite young project, but it is remarkably mature.

Kivy Catalog

My weekend project was the Kivy Catalog, an interactive showcase of Kivy widgets. It allows you to view the Kivy language code used to define the widgets and to edit it and see what effects your changes have. I think it will be a tremendous aid to users looking to get started with the Kivy language. Developing it certainly was for me!

I am expecting to have this code included with the Kivy distribution in the examples directory in a future release.

Pushing Python Past the Present

This is the first time I’ve inserted myself into an exchange between bloggers I don’t know. This topic interests me and I have something to add. Most importantly, I found an alliterative title. So I thought I’d give it a go.

I first saw Calvin Spealman’s article I am worried about the future of Python. I suspect that upon reading this article, Guido got into his famous time machine to give his Pycon 2012 Keynote speech, accusing Calvin of trolling.

This article was shortly followed by Tim McNamara’s Python is doing just fine which (likely unintentionally) summarizes Guido’s talk.

Finally, Nick Coghlan came out with an extensive summary article called Python’s Future: A Global Perspective.

And now there’s me. All I want to do is push people to start supporting PyPy. The PyPy developers have worked on a variety of technologies that can be made into production-ready products that address most issues (real or imaginary) that people see with Python.

They’ve already solved speed and I suspect the next release of PyPy will solve memory. They’ve made huge but mostly incomplete progress in a lot of other areas, including the ones the above bloggers have mentioned.

Concurrency

I have had a lot of trouble dealing with parallelism in Python. Everything I have tried has either been a hack or required a hack to work with it. In my experience, the multiprocessing module does not work in large-scale production. However, it works a lot better on PyPy than it does on cPython, at least for me. The various async solutions can only use one processor and are therefore, in my opinion, no more exciting than GIL-throttled Threads.

In addition to better multiprocessing, PyPy also supports stackless extensions out of the box. And I haven’t even mentioned Armin Rigo’s mysterious Software Transactional Memory implementation.

Concurrency is a sore point in Python. There are solutions. PyPy is capable of doing those solutions better.

Web

I think it’s unfortunate that the only way to code for a web browser is to use JavaScript or a language that compiles to JavaScript. I feel there is no real reason browsers can’t support arbitrary <script type=""> tags. I discussed this last March.

With development, PyPy can support Python in the browser. The mostly-finished but untested sandboxing feature of PyPy can be adapted for in-browser execution. I know some experimenting was done on this in 2009 that proved it’s possible. We just need to take it up and make it happen.

Mobile

PyPy’s speed makes it theoretically possible to run it effectively on mobile platforms. Further, it can be compiled to arbitrary execution environments including JVM and .Net. There is no reason that with some development, PyPy couldn’t run on the Dalvik JVM. I’m sure it’s even possible to run it on iOS.

Interestingly, one of the trolls Guido mentioned in his talk was “PyPy should become the default Python implementation”. He debunked this readily by asking the audience how many people use PyPy in production. Nobody moved.

PyPy is ready for production use, but it is not widely used. I think this is largely because the primary PyPy developers are way more excited about creating new features than they are about marketing the current platform or polishing up nearly finished features like sandboxing or a Dalvik backend. They are visionaries, not finishers.

I say this with a great deal of respect. I am not calling for these guys to change their focus. What I want is for other people from the Python community to join the PyPy project as finishers. It needs people that can make sure these nearly-finished features are working, production-ready and more importantly: documented.

I’ve been meaning, for months, to become one of these people. Unfortunately, I’ve prioritized other things including my job and my upcoming book. It’s still on my radar, though, and I hope that after reading this article, you, too, are thinking about make PyPy the future of Python.

So, what can you do?

Use PyPy
PyPy is capable of being your default Python 2 interpreter for most common tasks. Use it. If it doesn’t work for a given project, get on #pypy, they will help you fix it. It’s even more exciting if you can use PyPy features that are not currently available in cPython, such as the stackless extensions.
Evangelize PyPy
Tell people how well PyPy is working for you. Write articles like this one.
Document PyPy
The PyPy website contains a lot of documentation, but it’s rather intimidating and unreadable to the uninitiated. PyPy is just Python, but it’s also much more than Python. Somebody with some writing skills needs to get in there and make it digestable.
Design PyPy
Seriously, pypy.org does not need to look like something out of 2005.
Develop PyPy
I’ve hacked on PyPy. It’s not scary. Work on features like numpy or Python 3 support that are the current developer’s focus. Better yet, work on finalizing features like sandboxing or alternative backends that are finished but not quite tested.

If you have an hour or two free this weekend, instead of writing about how Python is great or not great, or not going to continue to be great, do something with PyPy. Make it happen.

How (not) to screw up timezone processing in Python

If you use pytz without reading the documentation, you might think you can do this:

cltime = datetime.datetime(
        2012,9,26,1,15,0,
        tzinfo=pytz.timezone('America/Santiago'))

This is an easy way to utterly screw up your timezones.

Never pass a tzinfo into the datetime constructor. Here’s why:

>>> cltime = datetime.datetime(
        2012,9,26,1,15,0,
        tzinfo=pytz.timezone('America/Santiago'))
>>> cltime.astimezone(pytz.timezone('US/Pacific')).isoformat()
'2012-09-25T22:58:00-07:00'

That’s simply the wrong time. 1:15 Chilean time should be 21:15 Pacific the previous day, not 22:58.

For whatever reason, the wrong tzinfo is attached to the object. Compare:

>>> cltime.tzinfo
<DstTzInfo 'America/Santiago' SMT-1 day, 19:17:00 STD>
>>> datetime.datetime.now(pytz.timezone('America/Santiago')).tzinfo
<DstTzInfo 'America/Santiago' CLST-1 day, 21:00:00 DST>

The correct way is to use pytz.localize():

>>> correct_cltime = pytz.timezone('America/Santiago'
        ).localize(datetime.datetime(2012,9,26,1,15,0))
>>> correct_cltime.astimezone(pytz.timezone('US/Pacific')).isoformat()
'2012-09-25T21:15:00-07:00'
>>> correct_cltime.tzinfo
<DstTzInfo 'America/Santiago' CLST-1 day, 21:00:00 DST>

The pytz documentation does not indicate if datetime.datetime.now() creates the correct timezone, so I tested it:

>>> for tz in pytz.all_timezones:
...     assert datetime.datetime.now(pytz.timezone(tz)
...         ).tzinfo == pytz.UTC.localize(datetime.datetime.utcnow()
...         ).astimezone(pytz.timezone(tz)).tzinfo
...     
>>>

The assertion never failed, so it is safe to use datetime.datetime.now(pytz.timezone("[timezonename]")) to generate the current date in a specific timezone.

In general, you should store dates in UTC format and convert them to the user’s timezone at the latest possible time. Try to never store naive datetimes. if you need the current time in UTC, use

pytz.UTC.localize(datetime.datetime.utcnow())

because utcnow() returns a naive datetime. Localizing it to UTC adds the correct timezone to it.

pytz.UTC.localize(datetime.datetime(2012,9,26,1,15))

is the correct way to construct a datetime in the UTC timezone. Technically, you CAN pass tzinfo to the datetime constructor if tzinfo is UTC. This is because UTC does not have daylight savings time. However, for consistency, do not do this; do not ever pass tzinfo into a datetime constructor.

Converting to or from UTC is simple, using astimezone:

>>> utctime = cltime.astimezone(pytz.UTC)
>>> utctime
datetime.datetime(2012, 9, 26, 4, 15, tzinfo=<UTC>)
>>> utctime.astimezone(pytz.timezone('America/Santiago'))
datetime.datetime(2012, 9, 26, 1, 15, tzinfo=<DstTzInfo 'America/Santiago' CLST-1 day, 21:00:00 DST>)

Trembling Aspen

Aspen is a unique Python web framework developed by Chad Whitacre. I am still evaluating it, but one thing I can say unequivocally is that it’s different. Very different. Aspen divorces itself from the MVC pattern that every major — and almost every minor — web framework has been relying on for the past decade. While each framework has its perks and annoyances, they all have this underlying MVC-like interface. Aspen throws that all away.

Aspen lodges itself in a small niche between php-like spaghetti code and a complicated package structure full of python models, views, controllers, and templates. In Aspen, template and control code live in the same file, but in separate pages, called Simplates. In a rebellion against regular expression url parsing, Aspen uses the filesystem to define url structure and paths; like in the old days of statically served HTML files.

One thing I love about Aspen is how easy it is to get up and running. Aspen will serve a single HTML file with no python code without any boilerplate. It serves static files for css, javascript, and images just as easily. To add a dynamic page to your Aspen powered site, you simply use the aforementioned simplates, which include python controllers and Tornado inspired templates in the same file. Aspen provides numerous service connectors, so deployment can also be trivial.

Another great feature of Aspen is its documentation. At first glance, the docs seem sparse and rather limited. However, they are well organized, and every page seems to answer every question I have on the topic succinctly and quickly. Chad seems to have a terrific minimalist policy of telling you exactly what you want to know in as few words as possible. As a writer, I know just how difficult this is to accomplish without repeating oneself.

On the negative side, I do have misgivings as to how well Aspen’s filesystem urls it can scale. Chad has assured me that Aspen is up for running large sites. I appreciate the power of regex urls, and I don’t necessarily like having arbitrary file extensions in URLS. I think locators should be succinct and divorced from implementation details. Ideally, I believe there may be a happy medium between Aspen’s enforced folder structure, and the overly-abstract and overly-complicated url scheme Django and its kin provide.

Another downside is that I haven’t figured out a good way to unit test Aspen simplates. You can refactor much of the code into backend libraries and test that, of course, or you can start the service running and mock requests. However, I miss the TestClient feature from Django that allows me to mock a request without running the server. It provides a better analysis of code coverage, and is closer to the actual code.

Now, with the review over, I’d like to introduce a little project I’ve started to provide some vital features that Aspen core is missing.
Trembling is kind of a catch-all project with a few basic helpers and a few useful projects. Namely, trembling provides:

  • Basic sessions
  • Basic user authentication
  • Less than basic single request messages
  • A simple redirect response

Aspen itself does not define or require a model layer; you’re free to use the ORM, DBAPI connector, or NoSQL database of your choice. Trembling arbitrarily chooses mongoengine for the models. I made no effort to make pluggable backends like Django does; Aspen does not encourage excessive layers of abstraction.

I also don’t know how secure trembling’s session and auth features are. If you find a security hole, pull requests are welcome. Overall, I think the combination of Trembling and Aspen will be very useful for my future basic sites. I would also like to try it on a more complicated project just to see how well it performs.

Easily changing virtualenvs

The workon script available in virtualenvwrapper is a useful tool for switching virtual environments on the fly. However, it fails for the workflow I prefer, so I wrote my own.

I prefer to keep my virtualenvs in the same folder as whatever source code or project they apply to. Normally, I put them in a folder named venv, although sometimes I use different names, most often when I have multiple virtualenvs applying to the same source tree (e.g. venv27, venvpypy).

My directory structure therefore looks like this:

/home/dusty/code/
+ project1
| + src
| | + pkg1
| | + pkg2
| + doc
| + venv
+ project2
| + src
| + venv27
| + venvpypy
+ project3
| + venv

… and so on

The problem is, I may be anywhere in the folder tree when I want to activate a virtualenv. This means I’m doing stuff like . ../../venv/bin/activate one time and venv/bin/activate another time. Trying to remember which parent directory I want often requires several tab completions to see the directory listing. This is annoying. So I wrote a bash function to activate the optionally named virtualenv in the current directory or any parent directory.

I avoid bash whenever I can, so this may not be the prettiest bash script you’ve ever seen. ;-)

function v {
  # activate a virtualenv
  # usage: v
  #   activate the virtualenv named venv
  # usage: v venvname
  #   activate the virtualenv named venvname
  name=venv
  if [ $1 ] ; then
    name=$1
  fi
  olddir=$(pwd)
  quit=0
  deactivate &amp;&gt;/dev/null
  cwd=$(pwd)
  while [ $quit -eq 0 ]
  do
    cd $cwd
    if [ $cwd == '/' ] ; then
      quit=1
    fi
    if [ -e $name ]  ; then
      source "$name/bin/activate"
      quit=1
    fi

Put this in your ~/.bashrc. Typing v on the command line will look in the current and all parent directories for a directory named venv, and will activate that venv. Typing v [venvname] will walk the same tree looking for a directory named [venvname] to activate.

If you want to make it prettier, feel free to fork it on github.

Hacking on PyPy

In another great Pycon2012 keynote, David Beazely asked the question, “is PyPy easily hackable?” After a great talk, he answered with a decisive, “I still don’t know.” Having sprinted on Python I’d like to answer his question in a bit more detail.

I love David’s presentation style. He has a novel method of using phrases like, “blow your mind” and “this is really scary” repeatedly until they lose their meaning and you no longer feel mindblown or scared. A variety of factors, including Beazely’s thorough keynote address motivated me to join the PyPy team during the Pycon developer sprints.

I’d like to clear up one oversight in Dave’s otherwise impeachable talk. One of the PyPy devs, Holger Kregel explained to me that PyPy does not have over 1 million lines of code. I don’t have exact numbers, but for “historical reasons”, a non-python file containing Base64 encoded data was given a .py extension. When excluding this file from the line count, around half a million lines of actual Python code exist, and about a quarter of these are tests.

I was surprised how trivial it was to get started hacking on PyPy. I don’t really grok the many layers of the translation toolset and PyPy interpreter, but it’s pretty clear that the layers are well separated. I was hacking on the py3k branch of PyPy. I am happy to admit I was working primarily on changing print statements to print() functions and commas in exceptions to the as keyword.

Here are the steps to start hacking on PyPy. Notice that the hour-long translation step is not part of the procedure. PyPy has a solid test framework, and the PyPy crew are focused on a 100% test-driven-development paradigm.

  1. Clone pypy (this takes a while):
    hg clone https://bitbucket.org/pypy/pypy/
  2. Pick a branch to work on. There are about 80 branches. I don’t know what they all do. Popular ones during the sprints included py3k and numpy-ufuncs2
  3. Pick a feature to work on. For py3k support, the list of failing tests in the buildbot is a good place to start. Numpy programmers had a list of fuctions that needed implementing, but I can’t find the link. Ask on IRC, the PyPy crew are very helpful. The bug tracker contains many features and issues that need addressing
  4. Add /path/to/pypy/ to your path so you can run the pytest.py command
  5. cd into the directory indicated in the buildbot output and run pytest.py path/to/test.py -k testname
  6. The test will likely fail. Hack away and fix it.
  7. When the test passes, commit, push to a bitbucket repo, and issue a pull request.
  8. Repeat!
  9. There are quite a few cons to working on this project. If you run hg in the pypy/modules/ directory, it will try to pick standard library modules from pypy and choke horribly. The pypy developers don’t really believe in documenting their code. Being able to tell the difference between rpython and python (which have identical syntax) is important. In general, if a module starts with “interp_” it contains rpython, but if it starts with “app_” it contains python. The code does not appear to be well-documented.

    If you are hacking on Python 3 support, you need to bear in mind that the PyPy interpreter is written in Python 2. You are working on a Python 2 application that executes Python 3 bytecode!

    On the positive side, rPython and Python are much easier to read and write than C. The PyPy devs are brilliant, but not intimidating. They are so confident in their test suite that they are comfortable programming in a “cowboy coding” style, hacking randomly until all the tests pass. Any one layer in the toolchain is easy to understand and develop. The IRC channel is full of friendly, knowledgable, helpful people at any time of day.

    Overall, I am much less intimidated by this project than I was before I started the dev sprints. I still can’t answer the, “Is Python easily hackable?” question fully. It’s certainly easy to get started, but I don’t know how easy it is to become intimate with the project. Dave Beazely’s keynote made PyPy more approachable, and I approached it. Hopefully this article will encourage you to do the same.

Why we need Python in the Browser

In his Pycon 2012 keynote speech on Sunday, Guido van Rossum covered many of the open “Troll” questions related to the Python community. I’ve had occasion to either complain about or defend all of the topics he covered, and he did a wonderful job of addressing them. The issues could largely be divided into two categories: those that come from a fundamental misunderstanding of why Python is wonderful (e.g. whitespace), and those that are not an issue for the core python dev team, but are being actively worked on outside of core Python (e.g event loop).

And then there’s the question of supporting Python in the web browser. I honestly thought I was the only one that cared about this issue, but apparently enough people have complained about it that Guido felt a need to address it. His basic assertion is that the browsers aren’t going to support this because nobody uses it and that nobody uses it because the browsers don’t support it.

This is a political problem. Politics shouldn’t impact technical awesomeness.

The fallacious underlying assumption here is that modern HTML applications must be supported on all web browsers in order to be useful. This is no longer true. Web browser applications are not necessarily deployed to myriad unknown clients. In a way, HTML 5, CSS 3, and DOM manipulation have emerged as a de facto standard MVC and GUI system. For example, many mobile apps are developed with HTML 5 interfaces that are rendered by a packaged web library rather than an unknown browser. Client side local storage has created fully Javascript applications that require no or optional network connectivity. There are even situations where it may not be necessary to sandbox the code because it’s trusted. Many developers create personal or private projects using HTMl 5 because it’s convenient.

Convenient. It would be more convenient if we could code these projects in Python. Web browsers can be viewed as a zero install interface, a virtual machine for these applications. Such a VM has no reason to be language dependent.

It is simply unfair to all the other programming languages and coders of those languages to say, “we can’t displace Javascript, so we won’t try.” Web browsers have evolved into a virtualization layer more like operating systems than single programs. While it is true that the most restrictive operating systems only permit us to code in Objective C, in general, it is not considerate to restrict your developers a single language or environment.

It is time (in fact, long overdue) for Python to be supported in the browser, not necessarily as an equal to Javascript, but as an alternative. The web is a platform, and we must take Guido’s statement as a call to improve this platform, not to give up on it.

Update: I just stumbled across http://www.gnu.org/software/pythonwebkit/ and I can’t wait to play with it!

Update 2: From the number of comments on this article, it appears that my article has hit some web aggregator. To those users suggesting python to javascript compilers, I’ve been a minor contributor to the pyjaco project, a rewrite of the pyjs library. It has the potential to be a great tool, and the head developer, Christian Iversen is very open to outside contributions. Let’s make it better!

PyCon 2012

Hello Arch Linux community! If any of you are attending Pycon 2012 in Santa Clara, CA this week, make sure to bump into me. Or contact me to schedule a meetup.

Pyjaco in a real app: Todos with local storage

I didn’t get the memo, but there appears to be a movement to demonstrate emerging web technologies with a simple todo list application, much as hello world is used to introduce programming languages.

In my last post, I introduced using jQuery with Pyjaco, the PYthon JAvascript COmpiler. Since then, I’ve made several contributions to the project and have been involved in diverse discussions with Pyjaco developers regarding the current and future status of the project. This post goes further by acting as a tutorial for writing a basic todos app using Pyjaco.

Pyjaco is alpha software. It is hard to write valid code, and harder to debug. I’ve managed to both lock up Firefox and hard crash it while using the Pyjaco library.

On the positive side, Pyjaco is under active, rapid development. The head developer, Christian Iversen is extremely responsive to both questions about Pyjaco, and to code contributions. This is a project with a lot of potential, and I see it as the current best bet for Python programmers hoping to avoid javascript one day in the future.

In spite of the hiccups, it is possible to generate a working javascript app using just Pyjaco. Here’s how.

Let’s start:

mkdir pyjados
cd pyjados
virtualenv2 venv --distribute --no-site-packages
source venv/bin/activate
pip install git+https://buchuki@github.com/buchuki/pyjaco.git@run_script

First we create a directory to work in and install a virtualenv. Pyjaco does not currently work with python 3, so in Arch Linux, I use the virtualenv2 command. We then activate the virtualenv and install the pyjaco package. Here I am installing from my personal fork, as it contains some changes for generating the built-in standard library that have not yet been merged upstream. You should normally install directly from chrivers’s git repository using pip install git+git://github.com/chrivers/pyjaco.git.

Now let’s create a basic HTML 5 page with jQuery loaded:

<!DOCTYPE html>
<html>
  <head>
    <title>PyJaco Todo List Example</title>
    <script type="text/javascript" src="http://code.jquery.com/jquery-1.6.4.min.js"></script>
  </head>
  <body>
    <h1>PyJaco Todo List Example</h1> 
  </body>
</html>

We can load this in our web browser using a file:// URL. This is the only HTML page in our app, and it can be refreshed to load our changes as we work.

Pyjaco doesn’t simply translate Python into Javascript. Rather, it creates a basic standard library of Python-like objects that are utilized in the compiled javascript code. I wasn’t too keen on this idea when I first heard it, as it introduces a dependency that currently weighs in at 65K before minification and compression. While this is not a terribly heavy library, there are efforts under way to shrink the builtins or to dynamically generate it to contain only those builtins that your code actually touches. At any rate, we need to ensure this library is available to our code. First we generate the library:

pyjs.py --builtins=generate --output=py-builtins.js

pyjs.py is the name of the pyjaco command. It is expected to be renamed to pyjaco in the future. The --builtins=generate option tells pyjaco to generate the standard library, while the --output flag provides the filename for the new library file:

$ ls
index.html  py-builtins.js  venv

We then need to load this library in the head of our html file. Let’s also load the future pyjados.js script at this time:

  <head>
    <title>PyJaco Todo List Example</title>
    <script type="text/javascript" src="http://code.jquery.com/jquery-1.6.4.min.js"></script>
    <script type="text/javascript" src="py-builtins.js"></script>
    <script type="text/javascript" src="pyjados.js"></script>
  </head>

Now, before we start coding the Python file that will be compiled to Javascript, I want to discuss what I consider to be the most confusing aspect of Pyjaco development. There are basically two types of variables in Pyjaco, Javascript variables, and Python variables. Javascript variables refer to “normal” variables that you would call in Javascript. These include alert, window, document and the like, as well as variables in third-party Javascript libraries, such as the ubiquitous jQuery. Further, any attributes on those objects are also Javascript variables, and the return value of any methods will also be Javascript variables.

Python variables, on the other hand, refer to any variables that you define in your Python source code. If you create a dict or a list, for example, it will be compiled to a list or dict object from the standard library we just generated. In the compiled script, of course these Python variables are represented by Javascript objects, but from the point of view of a Pyjaco coder, it is important to keep the two types of files separate. Almost all the bugs I have encountered in my Pyjaco code have been caused by confusing the two types of variables.

The distinction between python and javascript variables introduces a couple of complications to writing Pyjaco compatible python code. First we need to flag all of our Javascript variables using a decorator on methods that access them. Second, we need to explicitly convert our variables between Javascript and Python any time we access one from the other. I’m told that this conversion can — and one day will — be done automatically by the pyjaco multiplexer, but in the meantime, we need to make it explicit. We do this by using two javascript functions supplied with the standard library we just generated, appropriately named js() and py(). You will see examples of these shortly.

When I finally figured out the distinction, my first thought was, “ok, let’s prefer to always work with python variables.” Therefore, in my initialization code, I tried jQ=py(jQuery). Unfortunately, jQuery is a rather large object, and the py function apparently recursively converts all attributes from javascript to python. I ended up with a stack overflow.

Now, let’s create our first python code and watch it compile to Javascript. Name the file pyjados.py:

def setup():
    print "Pyjados Hello World"
 
jQuery(js(setup));

First we write a python function named setup. This function is a python object. jQuery is a javascript object that expects a javascript object as input. Therefore, we wrap setup in a js() call and pass the result into the jQuery function. jQuery will now run setup when document.ready is fired.

Now we compile the code using the following command inside our activated virtualenv:

pyjs.py --watch pyjados.py --output pyjados.js

You’ll notice the command doesn’t exit. That is the --watch option at work. If you now make a change to pyjados.py and save it, it will automatically recompile it. The output file pyjados.js is regenerated each time. This is the file we included in our html file. So now, open that html file in a web browser using a file:// url. Make sure the Javascript console is displayed and reload the page. You should see the words “Pyjados Hello World” printed on the console. Pyjaco automatically compiles print statements into console.log output.

Before we start implementing our Todo list, let’s look at an example of accessing a javascript variable inside a function. Change setup.py to utilize alert, as follows:

 
@JSVar("alert")
def setup():
    alert(js("Pyjados Hello Alert")
 
jQuery(js(setup));

Did you look closely at that code? There is a missing close bracket on the alert line. You’ll see the syntax error in your console where pyjs.py is watching the compiled code. Add the bracket and let it automatically recompile itself:

 
@JSVar("alert")
def setup():
    alert(js("Pyjados Hello Alert"))
 
jQuery(js(setup));

Let’s analyze this snippet. First, notice how we told the compiler that alert is a Javascript variable when used inside setup(). This is a bit odd, since the JSVar decorator is never actually imported into the namespace. This is a bit of magic in the Pyjaco compiler, just pretend it has been imported.

Second, notice that since alert has been flagged as a JSVar, it must accept a Javascript variable. However, the string “Pyjados Hello Alert” is a Python variable. Therefore, we convert it using js() as we pass it into the alert call.

Now let’s prepare to create some working todo-list code. Start by adding a form for submitting todos and a list to render the todos to the html body:

  <body>
    <h1>PyJaco Todo List Example</h1> 
    <form id="add_todo_form">
      <input type="text" id="add_box" placeholder="Add Todo", autofocus="autofocus">
      <button id="add_button">Add Todo</button>
    </form>
    <ul id="todo_items"></ul>
  </body> 

Nothing too exciting here. Note the ids on the elements, since we’ll be utilizing these from Pyjaco using jQuery selectors.

Now back into the python file. Let’s create a class to manage the various todo elements:

class TodosApp:
    @JSVar("jQuery", "js_add_form")
    def __init__(self):
        js_add_form = jQuery(js("#add_todo_form"))
        js_add_form.submit(js(self.add_todo))
 
    def add_todo(self, event):
        print "form submitted"
        return js(False)
 
def setup():
    todo_app = TodosApp()
 
jQuery(js(setup));

The __init__ function hooks up the form’s submit button to a method on the object. Notice that we need to flag not just jQuery, but also js_add_form as a javascript variable. Pyjaco does not (currently) know that a javascript variable is returned when calling a method on an existing javascript variable. I like to add the js_ prefix to variable names to help remind myself that this is a javascript variable.

In an ideal world, we could convert this variable to a Python variable using py(), but as noted earlier, calling py on a jQuery object results in a stack overflow or browser crash.

Also pay attention to the way we wrap the self.add_todo method name in a js() call when we pass it into the submit handler. The submit method is a javascript function expecting a javascript object.

The def add_todo method has its single parameter flagged as a @JSVar, since the method is being called internally by jQuery when the event occurs. We also wrap the False return value (to prevent event propogation on the submit handler) in a js() call so that jQuery recognizes it as a javascript false rather than a (true) object named False.

Try the code. Ensure the compiler recompiled it, and reload the html file. Enter some characters into the text box and use the Enter key or the Add Todo button to submit the form. The words form submitted should be displayed in the javascript console.

Now let’s actually store and render a newly added todo. The todos are stored in memory in a python dict object. Initialize this object by adding the following two lines of code to the end of __init__:

    self.todos = {}
    self.next_id = 1

And rewrite add_todo as follows as well as a new method named render

    @JSVar("event", "js_add_box")
    def add_todo(self, js_event):
        js_add_box = jQuery(js("#add_box"))
        self.todos[self.next_id] = py(js_add_box.val())
        js_add_box.val('')
        js_add_box.focus()
        self.next_id += 1
        self.render()
        return js(False)
 
    @JSVar("js_todo_items")
    def render(self):
        js_todo_items = jQuery(js("#todo_items"))
        js_todo_items.html("")
        for id, todo in sorted(self.todos.items()):
            js_todo_items.append(js('<li>%s</li>' % (id, todo)))

Note that the todos dict is a Python object, so when we insert the value of the js_add_box into it, we must convert it from a javascript object using py(). Also note how, because we are writing in a python function, manipulating the python value self.next_id requires no conversion, and calling the python function self.render is also clean.

In the render function itself, I think it’s pretty cool that string formatting using % is supported by pyjaco (as an aside, the str.format method introduced in python 2.6 is not yet available) and that the python sorted() function is available. Note also how we can loop over items() on the self.todos dictionary just as if we were using a normal python dictionary.

Now let’s add the ability to complete todos. Let’s start by adding a template string as a class variable, and use that string inside the render function. This illustrates that pyjaco supports class variables:

class TodosApp:
    list_item_template = """<li>
    %(text)s
    </li>"""

and we change the for loop in render to:

        for id, todo in sorted(self.todos.items()):
            js_todo_items.append(js(TodosApp.list_item_template % {
                "id": id,
                "text": todo}))

Reload the page again and notice how checkboxes have been displayed beside each todo. The next step is to make clicking these boxes actually complete the todos. We add a couple lines to our __init__ method to connect a live click event to the checkbox items, which now looks like this:

    @JSVar("jQuery", "js_add_form", "js_checkbox")
    def __init__(self):
        js_add_form = jQuery(js("#add_todo_form"))
        js_add_form.submit(js(self.add_todo))
        js_checkbox = jQuery(js("input[type=checkbox]"))
        js_checkbox.live("click", js(self.complete_todo))
        self.todos = {}
        self.next_id = 1

Don’t forget to add js_checkbox to the JSVar decorator.

The complete_todo method looks like this:

    @JSVar("event", "jQuery", "todo_item")
    def complete_todo(self, event):
        todo_item = jQuery(event.target).parent()
        id = int(py(todo_item.attr("id"))[5:])
        del self.todos[id]
        todo_item.delay(1500).fadeOut("slow")

The first line is using exclusively javascript arguments, and returns the <li> element containing the checkbox that was clicked. The id = line converts the javascript string id attribute of this element (which looks like “todo_5“, as defined in list_item_template) into the python integer id of the todo. The remaining lines simply remove that todo from the internal list and from the DOM, after a 1.5 second delay.

In fact, we now have a fully functional todo list that allows adding todos and checking them off. Now, as a bonus, let’s try hooking this up to the HTML 5 localStorage object so that the list is maintained across page reloads. We start by adding a store() method to our class:

    @JSVar("localStorage", "JSON")
    def store(self):
        localStorage.setItem("todolist", JSON.stringify(js(self.todos)))

The main line of code is easiest to read from the inside out. First we convert the self.todos dict to a normal javascript object using the js() function. Then we call JSON.stringify on this object to create a string suitable for insertion into localStorage.

Now add this call to the end of the two methods that manipulate the todo list, add_todo and complete_todo:

        self.store()

.

Refresh the page, add a couple todos, and inspect the localStorage object in your console. You should see the stringified dict in the todolist value.

Now all we have to do is ensure the self.todos dict is loaded from localStorage when the app is initialized. Add the following to the end of the __init__ method (make sure to add js_stored_todos to the JSVars decorator):

        js_stored_todos = localStorage.getItem("todolist")
 
        if js_stored_todos:
            stored_dict = dict(py(JSON.parse(js_stored_todos)))
            self.todos = dict([(int(i), stored_dict[i]) for i in stored_dict.keys()])
            self.next_id = max(self.todos.keys()) + 1
 
        self.render()

Note that calling py() on the output of JSON.parse creates a python object, not a python dict. The code is therefore wrapped in a call to dict(), which converts the object to a dictionary.

Unfortunately, the resultant dict contains keys that are strings, whereas our original dict used integer keys. So a pure-python list comprehension is used to convert the dictionary to one with integer keys. This line is a bit hard to read, but I wanted to include it to demonstrate that Pyjaco can parse list comprehensions. Finally, we set self.next_id using the python max() call, which Pyjaco also automatically translates into javascript.

Try it out. Load the pyjados HTML file, add some todos, check a few of them off, then close and reload the web browser. Your todos will be stored!

I hope you’ve enjoyed this introduction to Pyjaco. It is a nice tool with a lot of potential. Currently, I find writing Pyjaco code to be approximately equally tedious to writing Javascript code. However, I feel that as I learn the ins and outs of Pyjaco, and as the developers continue to refine and improve the compiler, Pyjaco may one day be a perfectly viable alternative to writing pure Javascript or to the rather too Ruby-esque, but otherwise excellent Coffeescript.