Archive for January 2012

Learning hg as a git user

As my friend, Jason Chu recently noted, I am primarily a git user who has discovered a need to understand and use Mercurial. I am trying to refrain from judgment on Mercurial, as I’m easily bored by bikeshed discussions and holy wars. I have a pragmatic “use what you like and let me use what I like” philosophy, but when you are interacting with other people’s code, you occasionally have to use what they like.

I have read several articles that I do not intend to link to discussing the differences, and cheat sheets of hg equivalents of common git commands. These are utterly useless. Mercurial and git have different design philosophies, as Jason noted, even though the end result of their usage is much the same. If you’re comfortable with git, and interested in learning Mercurial, you may find my own eureka moment helpful.

hg commit is not the same as git commit.

Most comparisons of git and hg do not notice this distinction, but I was really puzzled by how hg could be more powerful than Subversion and supposedly equally powerful to my beloved git until I came to this realization.

In git, when you make a commit, you are creating local history that can be easily changed, modified, or ratified. You can rebase over those changes as many times as you like. You can use git commit –amend to change the commit or add changes to it. The history is not remotely considered “permanent” until you push it to a public repo, and even then, there are times when it is acceptable to rewrite it.

Conversely, in hg, when you commit, you are doing what the word actually says: committing. You are saying “this commit looks the way I want it to, I am finished with it.” You may not be pushing the commit to a remote repo any time soon, you may not be publishing it, but you have written in sandstone that this commit is complete.

I say written in sandstone, rather than stone because there are a variety of hg commands and extensions that allow local history editing, rebasing, and rollbacks. I haven’t learned how fluid these extensions are compared to equivalent history modification in git, but the feeling I am getting is that such changes would be considered much more invasive in hg than in git. History editing is a third party extension; this says to me “not officially supported” (as compared to built-in extensions like Mercurial queues). Mercurial typically desires us to think of a commit as an object that is permanently in the history. Many of the other slightly-deeper-than-cosmetic differences between the two systems seem to stem from this same basic difference.

In git, I have gotten quite used to coding first, and then creating an appropriate history later. There are numerous other potential workflows with git, but that’s the one I like. At first, I thought this was impossible or very difficult with Mercurial. However, when I realized that “commit” falls somewhere between the commands “commit” and “push” in git, things started to fall into place.

Mercurial has a powerful tool called queues that allow you to manipulate history to your heart’s content before you call commit. I’ve been using these effectively to create a workflow that I am comfortable with. It’s not the same as what I’d do in git, not remotely, but the overall outcome is similar.

A related basic understanding that is a little better documented than the difference between hg commit and git commit is the following:

hg branch is not the same as git branch.

Once again, hg branch lies somewhere between git branch and git push origin. When you call hg branch, you are stating an intent that the branch will be public. In git, you can have as many unpublished branches as you want. In Mercurial, this behavior is better achieved by the use of bookmarks, although I’ve found that Mercurial queues are easier to work with.

There are many tutorials for new hg users coming from a svn background, and a few tutorials for those coming from a git background. If you are hoping to learn Mercurial effectively, I suggest avoiding most of those options. It is much better to study Mercurial from the perspective of a programmer who hasn’t seen version control before. Such coders don’t exist (I hope!), but this attitude allows you to learn how the new system should be used, not how to make it behave like a system you are previously used to.

For mercurial basics, I strongly recommend http://hginit.com/, an irreverent and entertaining tutorial on the simpler concepts.

I had a lot of trouble understanding hg queues until I read the hg book chapter on the topic I intend to read the entire red-bean book at some point, as it appears to be much more coherent than the official Mercurial documentation. Now that I’ve been playing with hg queues for a day or so, I have come to understand that they can cover several common git tasks that appear to be missing from hg, including stashing, rebase -i, and similar. The key takeaway is you don’t commit your queues until you are quite certain you want them to become permanent history.

I haven’t yet figured out just when to choose hg queues over hg bookmarks, but a good read for getting used to hg bookmarks can be found here.

I strongly recommend enabling the hgk extension Just add the following to your ~/.hgrc. This will enable an hg view command that is more similar to gitk than it should be, considering the basic differences between branches in the two systems.

[extensions]
hgk =
mq = 
bookmarks =

(The second line is for enabling hg queues and third enables bookmarks.)

Overall, I suspect that I will always prefer git to hg. However, unlike subversion, I think Mercurial does supply me with tools I need to work effectively. Different tools from git, but effective nonetheless.

One beef I have with both git and Mercurial is that they violate the “one best way to do things” principle which makes learning, communicating about, and deciding how to use them more complicated than it needs to be.

My first Blackberry

I don’t often post reviews of hardware. However, RIM is getting a bad rap lately, and all the cool kids are using Android or IPhone these days. I have been severely disappointed with the three Android phones I’ve had to date, and I have had too many bad experiences with touch keyboards to take the IPhone seriously. That left me with a choice between a Nokia Windows device or a Blackberry. I purchased the Blackberry Curve 9360. I wasn’t expecting much, but having used it, I don’t think I’ll ever go back.

From all the market hype, I expected the Blackberry to be inferior to Android. My expectations were therefore blown away. This is hands down, the best device I have ever used. Unlike Android, everything just works, and there are no force closes. The battery lasts a good 2.5 days under normal usage, compared to less than a day on my Android devices. RIM has put a LOT of thought into their OS design, and everything just flows. This phone is a joy to use, instead of an irritant.

The Curve doesn’t have a touch screen, but I never miss it. I’m willing to say, in fact, that the trackpad is a superior method of input for serious usage. Granted, a touch screen is great for mobile games, but I don’t waste time with that sort of thing. While pinch to zoom action on maps and images is quite pretty, on my Android phones it tended to be flaky and annoying (I’ve heard this is not the case on iPhones). One click zooming on my blackberry is more effective.

Actually, I did miss the touch screen when using my web browser at first. Trying to use the trackpad as a mouse was irritating until I increased the sensitivity. Now that the cursor moves as far and fast as I expect it to, I’d say that the interaction is about comparable to touching links and buttons. The touch action is more intuitive overall, but I used to find under Android’s browser that 1 time in 3 the phone would activate a different link from what I had thought I touched. The trackpad pointer never does this. The tradeoff leaves me sitting on the fence; either method works, but neither is optimal.

Of course, there are Blackberry devices with touchscreensI originally wanted to get a Torch 9810 to get the best of both worlds. I doubt I will do that when I upgrade, simply because I’ve unexpectedly proven to myself that the touch screen is not the wonderful idea that Steve Jobs told us it was.

The Blackberry OS 7 is extremely responsive, even though the Curve hardware specifications are modest compared to high power Androids, or even higher power Blackberries. I never have to wait for text to show up on the screen or for activities to complete. The tools that Blackberry is known for, namely messaging and email are even more pleasant to use than I had been led to expect.

The Blackberry market app is a bit of a disappointment. It doesn’t seem to find apps that I know exist, and I normally find it easier to search google on the phone and install an OTA link than to download through the market. Another annoyance is that you need to perform a Windows-like reboot whenever you install new apps. However, this is a rare task for me; I have never been one to download a lot of apps. On my Androids, I tended to replace the stock apps with sexier ones from the Android market (The Go dev team’s contacts, messaging, launcher, and other tools are particularly nice). I feel no need to do this under Blackberry; the stock apps are too good to replace. I have heard there is a lack of selection of Blackberry apps, but I personally haven’t missed it. To be honest, the argument that Android has a much bigger selection of apps in its market is severely watered down when you realize that many of these are fart sound boards or sexy lady wallpaper collections.

The Curve’s camera takes very clear still pictures. The flash is small, but indoor shots seem to be high quality. Digital camera technology is pretty much a solved problem these days; I doubt that any one phone is going to outperform another in this regard. I haven’t bothered with the video camera, so I don’t know how it compares.

Call quality is definitely clearer on this phone than any other device I’ve ever used, including landlines. I have no idea what sort of noise cancellation or other technology is involved, or if the towers in this area have been improved since I last used them. I know they’ve been updated to 4G, but I don’t know how that is expected to affect call quality. I like that the phone usage is a single button press; it highlights that the dialer is not just another app on the phone. I’ve had Android get stuck in another app when trying to answer an incoming call. This doesn’t happen with the Blackberry; calls take priority.

All the reviewers that say they are far superior to any other hardware manufacturer’s keyboard are understating the truth. In just a couple weeks of usage, I am already more accurate than I have been on any soft or hard keyboard on the Android devices I’ve had. I’m also quite a bit faster. Typing an e-mail on a soft Android keyboard used to be annoying enough that I’d wait until I was sitting at my computer. The two Android hardware keyboards were much better, but I could easily write entire book chapters on this phone if I wanted to. The keys are easy to press and give a satisfying click on each keystroke. The keys seem deceptively small, but are easy to locate and press. It took a bit of practice to get used to the keyboard. I was dissatisfied at first, but if you use a BlackBerry phone for more than a day, the learning curve is gone. Compared to the three months I spent trying to learn a soft Android keyboard, this is exceptional!

One other thing I like about the phone is that it acquires a wireless connection in almost no time at all. My ancient laptop takes a couple minutes to connect to a WPA protected network, and my Android phones have all been equally slow. But the Blackberry, for some reason, can do it in no time at all.

I don’t want to be sounding too harsh on Android. If you like Android or like Google’s offerings, than by all means, stick with it. However, if you’re like me, and stumped for a good alternative, don’t let the negative media attention RIM has received fool you. This is a high quality piece of hardware running a high quality collection of software.

Pyjaco in a real app: Todos with local storage

I didn’t get the memo, but there appears to be a movement to demonstrate emerging web technologies with a simple todo list application, much as hello world is used to introduce programming languages.

In my last post, I introduced using jQuery with Pyjaco, the PYthon JAvascript COmpiler. Since then, I’ve made several contributions to the project and have been involved in diverse discussions with Pyjaco developers regarding the current and future status of the project. This post goes further by acting as a tutorial for writing a basic todos app using Pyjaco.

Pyjaco is alpha software. It is hard to write valid code, and harder to debug. I’ve managed to both lock up Firefox and hard crash it while using the Pyjaco library.

On the positive side, Pyjaco is under active, rapid development. The head developer, Christian Iversen is extremely responsive to both questions about Pyjaco, and to code contributions. This is a project with a lot of potential, and I see it as the current best bet for Python programmers hoping to avoid javascript one day in the future.

In spite of the hiccups, it is possible to generate a working javascript app using just Pyjaco. Here’s how.

Let’s start:

mkdir pyjados
cd pyjados
virtualenv2 venv --distribute --no-site-packages
source venv/bin/activate
pip install git+https://buchuki@github.com/buchuki/pyjaco.git@run_script

First we create a directory to work in and install a virtualenv. Pyjaco does not currently work with python 3, so in Arch Linux, I use the virtualenv2 command. We then activate the virtualenv and install the pyjaco package. Here I am installing from my personal fork, as it contains some changes for generating the built-in standard library that have not yet been merged upstream. You should normally install directly from chrivers’s git repository using pip install git+git://github.com/chrivers/pyjaco.git.

Now let’s create a basic HTML 5 page with jQuery loaded:

<!DOCTYPE html>
<html>
  <head>
    <title>PyJaco Todo List Example</title>
    <script type="text/javascript" src="http://code.jquery.com/jquery-1.6.4.min.js"></script>
  </head>
  <body>
    <h1>PyJaco Todo List Example</h1> 
  </body>
</html>

We can load this in our web browser using a file:// URL. This is the only HTML page in our app, and it can be refreshed to load our changes as we work.

Pyjaco doesn’t simply translate Python into Javascript. Rather, it creates a basic standard library of Python-like objects that are utilized in the compiled javascript code. I wasn’t too keen on this idea when I first heard it, as it introduces a dependency that currently weighs in at 65K before minification and compression. While this is not a terribly heavy library, there are efforts under way to shrink the builtins or to dynamically generate it to contain only those builtins that your code actually touches. At any rate, we need to ensure this library is available to our code. First we generate the library:

pyjs.py --builtins=generate --output=py-builtins.js

pyjs.py is the name of the pyjaco command. It is expected to be renamed to pyjaco in the future. The --builtins=generate option tells pyjaco to generate the standard library, while the --output flag provides the filename for the new library file:

$ ls
index.html  py-builtins.js  venv

We then need to load this library in the head of our html file. Let’s also load the future pyjados.js script at this time:

  <head>
    <title>PyJaco Todo List Example</title>
    <script type="text/javascript" src="http://code.jquery.com/jquery-1.6.4.min.js"></script>
    <script type="text/javascript" src="py-builtins.js"></script>
    <script type="text/javascript" src="pyjados.js"></script>
  </head>

Now, before we start coding the Python file that will be compiled to Javascript, I want to discuss what I consider to be the most confusing aspect of Pyjaco development. There are basically two types of variables in Pyjaco, Javascript variables, and Python variables. Javascript variables refer to “normal” variables that you would call in Javascript. These include alert, window, document and the like, as well as variables in third-party Javascript libraries, such as the ubiquitous jQuery. Further, any attributes on those objects are also Javascript variables, and the return value of any methods will also be Javascript variables.

Python variables, on the other hand, refer to any variables that you define in your Python source code. If you create a dict or a list, for example, it will be compiled to a list or dict object from the standard library we just generated. In the compiled script, of course these Python variables are represented by Javascript objects, but from the point of view of a Pyjaco coder, it is important to keep the two types of files separate. Almost all the bugs I have encountered in my Pyjaco code have been caused by confusing the two types of variables.

The distinction between python and javascript variables introduces a couple of complications to writing Pyjaco compatible python code. First we need to flag all of our Javascript variables using a decorator on methods that access them. Second, we need to explicitly convert our variables between Javascript and Python any time we access one from the other. I’m told that this conversion can — and one day will — be done automatically by the pyjaco multiplexer, but in the meantime, we need to make it explicit. We do this by using two javascript functions supplied with the standard library we just generated, appropriately named js() and py(). You will see examples of these shortly.

When I finally figured out the distinction, my first thought was, “ok, let’s prefer to always work with python variables.” Therefore, in my initialization code, I tried jQ=py(jQuery). Unfortunately, jQuery is a rather large object, and the py function apparently recursively converts all attributes from javascript to python. I ended up with a stack overflow.

Now, let’s create our first python code and watch it compile to Javascript. Name the file pyjados.py:

def setup():
    print "Pyjados Hello World"
 
jQuery(js(setup));

First we write a python function named setup. This function is a python object. jQuery is a javascript object that expects a javascript object as input. Therefore, we wrap setup in a js() call and pass the result into the jQuery function. jQuery will now run setup when document.ready is fired.

Now we compile the code using the following command inside our activated virtualenv:

pyjs.py --watch pyjados.py --output pyjados.js

You’ll notice the command doesn’t exit. That is the --watch option at work. If you now make a change to pyjados.py and save it, it will automatically recompile it. The output file pyjados.js is regenerated each time. This is the file we included in our html file. So now, open that html file in a web browser using a file:// url. Make sure the Javascript console is displayed and reload the page. You should see the words “Pyjados Hello World” printed on the console. Pyjaco automatically compiles print statements into console.log output.

Before we start implementing our Todo list, let’s look at an example of accessing a javascript variable inside a function. Change setup.py to utilize alert, as follows:

 
@JSVar("alert")
def setup():
    alert(js("Pyjados Hello Alert")
 
jQuery(js(setup));

Did you look closely at that code? There is a missing close bracket on the alert line. You’ll see the syntax error in your console where pyjs.py is watching the compiled code. Add the bracket and let it automatically recompile itself:

 
@JSVar("alert")
def setup():
    alert(js("Pyjados Hello Alert"))
 
jQuery(js(setup));

Let’s analyze this snippet. First, notice how we told the compiler that alert is a Javascript variable when used inside setup(). This is a bit odd, since the JSVar decorator is never actually imported into the namespace. This is a bit of magic in the Pyjaco compiler, just pretend it has been imported.

Second, notice that since alert has been flagged as a JSVar, it must accept a Javascript variable. However, the string “Pyjados Hello Alert” is a Python variable. Therefore, we convert it using js() as we pass it into the alert call.

Now let’s prepare to create some working todo-list code. Start by adding a form for submitting todos and a list to render the todos to the html body:

  <body>
    <h1>PyJaco Todo List Example</h1> 
    <form id="add_todo_form">
      <input type="text" id="add_box" placeholder="Add Todo", autofocus="autofocus">
      <button id="add_button">Add Todo</button>
    </form>
    <ul id="todo_items"></ul>
  </body> 

Nothing too exciting here. Note the ids on the elements, since we’ll be utilizing these from Pyjaco using jQuery selectors.

Now back into the python file. Let’s create a class to manage the various todo elements:

class TodosApp:
    @JSVar("jQuery", "js_add_form")
    def __init__(self):
        js_add_form = jQuery(js("#add_todo_form"))
        js_add_form.submit(js(self.add_todo))
 
    def add_todo(self, event):
        print "form submitted"
        return js(False)
 
def setup():
    todo_app = TodosApp()
 
jQuery(js(setup));

The __init__ function hooks up the form’s submit button to a method on the object. Notice that we need to flag not just jQuery, but also js_add_form as a javascript variable. Pyjaco does not (currently) know that a javascript variable is returned when calling a method on an existing javascript variable. I like to add the js_ prefix to variable names to help remind myself that this is a javascript variable.

In an ideal world, we could convert this variable to a Python variable using py(), but as noted earlier, calling py on a jQuery object results in a stack overflow or browser crash.

Also pay attention to the way we wrap the self.add_todo method name in a js() call when we pass it into the submit handler. The submit method is a javascript function expecting a javascript object.

The def add_todo method has its single parameter flagged as a @JSVar, since the method is being called internally by jQuery when the event occurs. We also wrap the False return value (to prevent event propogation on the submit handler) in a js() call so that jQuery recognizes it as a javascript false rather than a (true) object named False.

Try the code. Ensure the compiler recompiled it, and reload the html file. Enter some characters into the text box and use the Enter key or the Add Todo button to submit the form. The words form submitted should be displayed in the javascript console.

Now let’s actually store and render a newly added todo. The todos are stored in memory in a python dict object. Initialize this object by adding the following two lines of code to the end of __init__:

    self.todos = {}
    self.next_id = 1

And rewrite add_todo as follows as well as a new method named render

    @JSVar("event", "js_add_box")
    def add_todo(self, js_event):
        js_add_box = jQuery(js("#add_box"))
        self.todos[self.next_id] = py(js_add_box.val())
        js_add_box.val('')
        js_add_box.focus()
        self.next_id += 1
        self.render()
        return js(False)
 
    @JSVar("js_todo_items")
    def render(self):
        js_todo_items = jQuery(js("#todo_items"))
        js_todo_items.html("")
        for id, todo in sorted(self.todos.items()):
            js_todo_items.append(js('<li>%s</li>' % (id, todo)))

Note that the todos dict is a Python object, so when we insert the value of the js_add_box into it, we must convert it from a javascript object using py(). Also note how, because we are writing in a python function, manipulating the python value self.next_id requires no conversion, and calling the python function self.render is also clean.

In the render function itself, I think it’s pretty cool that string formatting using % is supported by pyjaco (as an aside, the str.format method introduced in python 2.6 is not yet available) and that the python sorted() function is available. Note also how we can loop over items() on the self.todos dictionary just as if we were using a normal python dictionary.

Now let’s add the ability to complete todos. Let’s start by adding a template string as a class variable, and use that string inside the render function. This illustrates that pyjaco supports class variables:

class TodosApp:
    list_item_template = """<li>
    %(text)s
    </li>"""

and we change the for loop in render to:

        for id, todo in sorted(self.todos.items()):
            js_todo_items.append(js(TodosApp.list_item_template % {
                "id": id,
                "text": todo}))

Reload the page again and notice how checkboxes have been displayed beside each todo. The next step is to make clicking these boxes actually complete the todos. We add a couple lines to our __init__ method to connect a live click event to the checkbox items, which now looks like this:

    @JSVar("jQuery", "js_add_form", "js_checkbox")
    def __init__(self):
        js_add_form = jQuery(js("#add_todo_form"))
        js_add_form.submit(js(self.add_todo))
        js_checkbox = jQuery(js("input[type=checkbox]"))
        js_checkbox.live("click", js(self.complete_todo))
        self.todos = {}
        self.next_id = 1

Don’t forget to add js_checkbox to the JSVar decorator.

The complete_todo method looks like this:

    @JSVar("event", "jQuery", "todo_item")
    def complete_todo(self, event):
        todo_item = jQuery(event.target).parent()
        id = int(py(todo_item.attr("id"))[5:])
        del self.todos[id]
        todo_item.delay(1500).fadeOut("slow")

The first line is using exclusively javascript arguments, and returns the <li> element containing the checkbox that was clicked. The id = line converts the javascript string id attribute of this element (which looks like “todo_5“, as defined in list_item_template) into the python integer id of the todo. The remaining lines simply remove that todo from the internal list and from the DOM, after a 1.5 second delay.

In fact, we now have a fully functional todo list that allows adding todos and checking them off. Now, as a bonus, let’s try hooking this up to the HTML 5 localStorage object so that the list is maintained across page reloads. We start by adding a store() method to our class:

    @JSVar("localStorage", "JSON")
    def store(self):
        localStorage.setItem("todolist", JSON.stringify(js(self.todos)))

The main line of code is easiest to read from the inside out. First we convert the self.todos dict to a normal javascript object using the js() function. Then we call JSON.stringify on this object to create a string suitable for insertion into localStorage.

Now add this call to the end of the two methods that manipulate the todo list, add_todo and complete_todo:

        self.store()

.

Refresh the page, add a couple todos, and inspect the localStorage object in your console. You should see the stringified dict in the todolist value.

Now all we have to do is ensure the self.todos dict is loaded from localStorage when the app is initialized. Add the following to the end of the __init__ method (make sure to add js_stored_todos to the JSVars decorator):

        js_stored_todos = localStorage.getItem("todolist")
 
        if js_stored_todos:
            stored_dict = dict(py(JSON.parse(js_stored_todos)))
            self.todos = dict([(int(i), stored_dict[i]) for i in stored_dict.keys()])
            self.next_id = max(self.todos.keys()) + 1
 
        self.render()

Note that calling py() on the output of JSON.parse creates a python object, not a python dict. The code is therefore wrapped in a call to dict(), which converts the object to a dictionary.

Unfortunately, the resultant dict contains keys that are strings, whereas our original dict used integer keys. So a pure-python list comprehension is used to convert the dictionary to one with integer keys. This line is a bit hard to read, but I wanted to include it to demonstrate that Pyjaco can parse list comprehensions. Finally, we set self.next_id using the python max() call, which Pyjaco also automatically translates into javascript.

Try it out. Load the pyjados HTML file, add some todos, check a few of them off, then close and reload the web browser. Your todos will be stored!

I hope you’ve enjoyed this introduction to Pyjaco. It is a nice tool with a lot of potential. Currently, I find writing Pyjaco code to be approximately equally tedious to writing Javascript code. However, I feel that as I learn the ins and outs of Pyjaco, and as the developers continue to refine and improve the compiler, Pyjaco may one day be a perfectly viable alternative to writing pure Javascript or to the rather too Ruby-esque, but otherwise excellent Coffeescript.