Posts tagged ‘javascript’

pyjaco and jQuery

After giving up on CoffeeScript, I decided to play around with Pyjaco the Python to Javascript Compiler.

While the code is readable, there is very little in the way of end-user documentation. I hope to address this with this blog post. The Pyjaco examples all embed generated javascript in an html page. I needed a way to generate an external Javascript file as I would include in an HTML file. I also wanted to find out if I could use Pyjaco with jQuery.

The first step was to install a development version of Pyjaco:

git clone https://github.com/chrivers/pyjaco.git

Pyjaco normally requires a generated Javascript file mapping Python builtins to Javascript to be included with the created Javascripts. This must be generated:

cd pyjaco
python2 generate_library.py
cp py-builtins.js ~/pyjaco_test # directory for my new page

The next step was to create an HTML file that included jquery, the py-builtins.js script above, and a yet-to be defined javascript file named clicker.js that will be generated from a yet-to-be-defined python file. I also add a couple of DOM elements (a heading and paragraph) that are to be manipulated via jQuery:

<!DOCTYPE html>
<html>
<head>
<script type=”text/javascript” src=”jquery-1.6.4.min.js”></script>
<script type=”text/javascript” src=”py-builtins.js”></script>
<script type=”text/javascript” src=”clicker.js”></script>
</head>
<body>
<h1>Click me</h1>
<p id=”when_clicked”></p>
</body>
</html>

That’s the easy part. Writing python code that compiles to correct Javascript is the hard part. Pyjaco doesn’t currently provide very useful compile-time errors, and it also does not yet map javascript errors back to the input python.

There are (at least) two ways to compile python code in Pyjaco. The first, which is used in the Pyjaco examples, and appears to be the preferred method at this time is to create a custom main() method that uses various pyjaco.Compiler methods to combine the functions into a string of text. See https://github.com/chrivers/pyjaco/tree/devel/examples for three examples.

However, I was looking to have a complete python file that compiles to a complete javascript file. Pyjaco supports this as well, using the provided pyjs.py script. It took some investigating to understand how to reference javascript variables inside python functions. Decorator syntax is used to expose the variables for jQuery, Math.random, and Math.floor in the following example. The mystifying bit is that because we will be compiling this file to javascript as a string, it is not necessary (or possible) to import the JSVar decorator, as was done in the Pyjaco examples linked above.

# clicker.py
@JSVar("jQuery")
def ready():
    jQuery('h1').click(on_click)
 
@JSVar("jQuery", "Math.random", "Math.floor")
def on_click():
    if jQuery('#when_clicked').html():
        r = Math.floor(Math.random() * 255)
        g = Math.floor(Math.random() * 255)
        b = Math.floor(Math.random() * 255)
        color = "rgb(%d, %d, %d)" %(r,g,b)
        jQuery('#when_clicked').attr('style', 'background-color: ' + color)
    else:
        jQuery('#when_clicked').html("you clicked it!")
 
jQuery(ready)

Notice that I’m using the jQuery function instead of the $ alias, since $ is not a valid variable name in Python. This is a rather odd looking mix of Python and Javascript functions, but it works.

I had to repeat the compile and test step a few times before coming up with the above file. The script can be compiled using the pyjs.py that comes with the pyjaco source distribution (and, thanks to a simple patch I submitted, will come with the binary distribution in the next release.) Here’s how the script is run:

python2 ~/code/pyjaco/pyjs.py -N --output clicker.js clicker.py

The -N option tells pyjs not to generate the builtin library that we created manually in the first step.

This translation step creates a clicker.js file that looks like this:

var ready = function() {
    var __kwargs = __kwargs_get(arguments);
    var __varargs = __varargs_get(arguments);
    var $v1 = Array.prototype.slice.call(arguments).concat(js(__varargs));
    jQuery("h1").click(on_click);
return None;
}
var on_click = function() {
    var __kwargs = __kwargs_get(arguments);
    var __varargs = __varargs_get(arguments);
    var $v2 = Array.prototype.slice.call(arguments).concat(js(__varargs));
    if (bool(jQuery("#when_clicked").html()) === True) {
        var r = Math.floor((Math.random()) * (255));
        var g = Math.floor((Math.random()) * (255));
        var b = Math.floor((Math.random()) * (255));
        var color = str('rgb(%d, %d, %d)').PY$__mod__(tuple([r, g, b]));
        jQuery("#when_clicked").attr("style", ("background-color: ") + (color));
    } else {
        jQuery("#when_clicked").html("you clicked it!");
    }
return None;
}
jQuery(ready);

I find this rather unfortunately difficult to read. There is code for argument parsing that would not have been needed if I had hand-written javascript. Further the use of “mock” python builtins makes the javascript look less javascripty. However, the original python file looks much more readable than an equivalent javascript one would. I am hopeful that improvements to pyjaco will cause it to generate more readable javascript with less extraneous code.

The entire example can be found on my github fork

Christian Iversen is actively working on Pyjaco right now. I am excited about this project and hope that further community involvement will help it evolve into a practical and useful tool. I intend my next patch to be an autocompile tool that monitors files in one directory for change and outputs .js files in another directory, one of CoffeeScript’s killer features. I am also considering a port to Python 3.

I can’t use CoffeeScript

I had some space between wrapping up my last contract on December 20, and starting my new job on January 16. I decided it was finally time to build a simple task management system, something I’ve attempted to do and never finished several times before. I currently use RememberTheMilk, which is a lovely service, but I don’t like paying them for mobile access. Further, even though I trust this small company, I see no reason to share intimate information about the tasks I accomplish every day with them. My long term goal is to pull as much of my personal data out of the cloud as I possibly can. This project is a step towards that goal.

There are numerous open source task manager apps out there that I’m sure would suit my not-too-exotic tastes. However, I also wanted to take this opportunity to learn a bunch of new technologies for an offline-enabled and mobile-enabled web application.

Therefore, I’ve spent most of my vacation time so far researching some technologies I haven’t had a chance to explore in the past year. JQuery Mobile was at the top of the list. I think it’s a lovely framework and I expect to continue using it.

I also wanted to try out CoffeeScript, as I’ve always hated writing Javascript, and I wanted to use some sort of client-side ORM for localstorage. I looked at backbone.js, but was more attracted to Spine.js. I have spent three days studying and playing with these two technologies. I am still undecided about Spine.js, but I have come to the conclusion that CoffeeScript is not for me.

I understand all the hype around the project. JavaScript really does suck. And CoffeeScript does suck less, it has pretty language features and it is much more succinct than JavaScript. I can imagine a lot of people being really excited about CofeeScript, especially Ruby and PERL programmers, and possibly even php programmers.

But not Python programmers. I tried to learn Ruby several times, and each time I was left with a foul taste in my mouth. It’s not a bad language, it just doesn’t fit in well with my personal philosophy. My personal philosophy happens to coincide almost exactly with the Zen Of Python. I chose Python because it matched my philosophy… not the other way around.

Like Ruby and PERL, CoffeeScript violently violates what I consider the most important rule of Python: “There should be one — and preferably only one — obvious way to do it.” I’m not going to argue why this is a good idea, I understand that some programmers prefer the “even if I don’t understand it, I can write code that will probably work” paradigm that Ruby promotes.

The simple truth is, writing CoffeeScript leaves me feeling like I’ve done something dirty, no less dirty than writing JavaScript. There is no incentive for me to add a layer of complexity (the CoffeeScript to JavaScript compile step) to my code when I know my code is going to be “ugly” either way.

CoffeeScript is a wonderful idea. It’s far better than JavaScript. It’s just not good enough. Luckily, the Python community is already working on pythonic answers, including the evilly poorly documented pyjaco and the less-than-well maintained pyvascript and pyjamas projects. I hope one of these or a new upstart will soon gain community momentum so frontend development is no longer painful.

codemirror > editarea

So you’re working on some kind of webapp that needs some in-browser code editing functionality of some sort. You search google and discover that most folks are using editarea these days. You figure it must work to be so popular and decide to try it out.

Then as you start to use it, you realize it’s unmaintained, doesn’t work terribly well in Webkit browsers, doesn’t work at all in Internet Explorer 9, is poorly documented and hard to extend, expand, or fix. You’re frustrated.

But repeated web searches do not yield anything as functional, and you resign yourself to a half-working editor, or maybe even decide to write something from scratch.

I’ve been that person, and I was recently lucky enough, while searching for something completely unrelated, to stumble across CodeMirror a standards-compliant javascript-powered code editor that is well documented. It’s not as featureful as editarea, but if you add CodeMirror UI to the mix, you end up with a comparable featureset that actually works.

I’m not normally one for simply reposting links, but as I had so much trouble finding CodeMirror in the first place, I hope this post will either help others to find it, or at least increase CodeMirror’s page rank in search results.

Rendering Django models to JSON

I recently discussed a simple template tag that allows rendering “raw” jquery templates inside a Django template, even though the syntax overlaps. A commenter asked for a Pypi package. It was such a simple template tag that I didn’t want to maintain a separate package for it, but it turns out, this tag fits in very well with the next step in the project. The short story is that the tag is now available on Pypi.

The tag is distributed with a project I called Djason (named for Django, JSON, and former Arch Linux Developer, Jason Chu, and because djson was already taken). Djason’s primary purpose is to serialize Django models to a json client format that is friendly to client libraries, especially to jquery-tmpl. It is essentially just a fork of Django Full Serializers with a patch applied, and some minor edits to simplify the output json. I install it as a Django serializer in settings.py using:

SERIALIZATION_MODULES = {
    'djason': 'djason.json'
}

And in my code:

from django.core.serializers import serialize

    content = serialize("djason", project,
            relations=["location_set", "company", "project_type"])
    return HttpResponse(content)

I’ll probably be adding some additional code to this package to fit in with my still cloudy ideas about the proper separation between client and server code. I’ll also likely be rewriting portions of the forked project to appease my personal sense of style.

Django and jquery.tmpl

Lately, I’ve been finding Django increasingly inappropriate for the web applications I develop. I have several complaints: the forms library doesn’t extend well to ajax requests, any extensive customization to the admin requires obtuse inspection of the admin source code, many admin customizations simply aren’t possible, the “reusable apps” philosophy has added a layer of complexity to a lot of things that really should not be there, there are no obvious best practices for ajax support.

In spite of all this Django is still better than other frameworks (Python or not) that I have investigated or tested. I’ve considered writing my own web framework, but I wouldn’t maintain interest in it long enough to get it off the ground. So I’m letting my complaints bounce around in the back of my mind with the hopes that I can improve Django so that it continues to work for me as a platform.

I’m currently trying to come up with a better system for ajax requests. I have nothing concrete in mind, but I’ve started with the premise that ajax requests should never return rendered html, but should return only json (hence my issue with the django forms library). With that in mind, I need a templating library for json data. JQuery is a must, and the officially supported JQuery templating library is jquery.tmpl http://api.jquery.com/category/plugins/templates/

The problem with jquery.tmpl is that it uses django-like syntax. The following is a valid block that may be rendered in a jquery.tmpl page:

<script id="project_tmpl" type="text/x-jquery-tmpl">
    {{each projects}}<li>${$value}</li>{{/each}}
</script>

If you try to include this in a django template, the {{ and }} tags will be replaced with (probably) empty variables. Django has {% templatetag %} to render these individual items, but what we really need is a way to tell the django templating system to leave complete sections of code alone. So I wrote the jqtmpl template tag. It allows us to wrap code in a block that tells Django not to render that block as template code. So the above would show up in a Django template as follows:

<script id="project_tmpl" type="text/x-jquery-tmpl">
{% jqtmpl %}
    {{each projects}}<li>${$value}</li>{{/each}}
{% endjqtmpl %}
</script>

Here’s the template tag:

from django.template import Library, TextNode, TOKEN_BLOCK, TOKEN_VAR
 
register = Library()
 
@register.tag
def jqtmpl(parser, token):
    nodes = []
    t = parser.next_token()
    while not (t.token_type == TOKEN_BLOCK and t.contents == "endjqtmpl"):
        if t.token_type == TOKEN_BLOCK:
            nodes.extend(["{%", t.contents, "%}"])
        elif t.token_type == TOKEN_VAR:
            nodes.extend(["{{", t.contents, "}}"])
        else:
            nodes.append(t.contents)
 
        t = parser.next_token()
 
    return TextNode(''.join(nodes))

This doesn’t handle Django’s {# comments #}, as token.contents doesn’t return a valid value for comment items. As far as I know, you wouldn’t use the comment construct inside a jquery.tmpl template anyway, so it’s still functional.

Next on my list is a better forms validation library to suit my theory that validation should be done client side. I’ve got a server-side system in mind that returns json responses, and does not return user-facing error messages, since those should have been caught client side. With these tools, and hopes for Grappelli to eventually create a decent admin, Django may continue to serve me.

Validate as you type in javascript

This is a common task, but has several facets that are easy to miss. I needed an input element that would validate as you type to ensure that an integer value was always entered. This was easily done by doing a replace against a regular expression, linked to the keyup event:

function validate_number(event) {
    this.value = this.value.replace([/[^0-9]/g,'');
}
 
$('jquery selector').keyup(validate_number);

This works fine provided the user is typing at the end of the box. The problem is that if the user has clicked in the middle of the number, aiming to insert a new digit, each time this.value is set, the cursor is moved to the end of the box, regardless of whether or not a valid character was entered. This happens on every keypress. That is bad.

The solution is to save and reset the selection range, as follows:

function validate_number(event) {
    if (this.value.match(/[^.0-9]/g)) {
        var start = this.selectionStart-1;
        var end = this.selectionEnd-1;
        this.value = this.value.replace(/[^.0-9]/g,'');
        this.setSelectionRange(start, end);
    }
}

I haven’t found any situations where this code doesn’t work, but there is one other caveat: pasting. If the user pastes a non-numeric value into the field without issuing a keypress (ie: using the right click–> paste menu), the value will not be validated. I haven’t found a way to stop this at the point when the value is pasted, but I’ve found it sufficient to additionally call the validation function when the textbox loses focus:

$('jquery selector').keyup(validate_number).blur(validate_number);

A variation of this code can be used to validate most inputs as the user types; a regular expression that matches whitespace can remove whitespace, phone numbers would look for digits and hyphens, decimal numbers would additionally search for a period, etc.

Offline Enabled Webapps And Open Source

I’m currently working on a rather simple Todo list application intended to meet the requirements I outlined earlier. I’m developing this as an offline-enabled webapp and while I haven’t gotten very far (don’t try it, it barely satisfies the most elemental requirements), I have some interesting observations to make:

First, the app is currently 100% Javascript. Its wrapped in a Django project because I’m going to have to make it online enabled at some point, but I wrote this app from the ground up to run in offline mode. This is a huge departure from traditional web 2.0 development (as of now, web 2.0 is to be considered ‘traditional’), where logic is mostly stored on the server and an ugly mess of ajax calls run that business logic on different events. Instead, I have a Javascript app that is surprisingly elegant. MVC under this architecture is basically SQLite (model), XHTML/CSS (view), and javascript (controller). Then I will have a workerpool to sync up with the server in the background.

As a result, the entire code-base for this app is being served to the web browser. So if you personally want to hack this app, you can download the source files from the browser. Basically, I can’t release this app as closed source. I could put a license on it forbidding you to modify or redistribute it, but you can still read the code; the source is open.

Of course, that’s not a problem for me, as I release most of my code as open source and its already up on github. But it could kill corporate migration toward offline-enabled web 3.0. Because while its one thing to keep your trade secrets locked up on the server and provide a few incoherent javascript calls that interact with that API, it is a completely different beast to put your entire app available for download as a .js file.

I suspect the open source movement has gained enough momentum that any companies who have these fears will simply be put out of business by more modern outfits that will spring up to take their place. Otherwise, either web 3.0 will fail to grab market share, or somebody will come up with a way to ‘compile’ javascript into a standards compliant byte-code so they can pretend nobody can reverse engineer their app. This would be a damn shame as its a politically-motivated technically useless layer of complexity on a web based architecture that I am finally happy to be working with.

Offline-Enabled Web Apps: The Future

I was reluctant to join the world of web development. I started in high school with a few sites and realized several things: Javascript sucks, Internet Explorer sucks; therefore web development sucks.

Fast-forward through a couple academic degrees. Job hunting with one requirement: Python. Python jobs all require Django.

So I learned Django, assuming, incorrectly, that if I was developing python backends, I wouldn’t need to work with the horrors of Javascript or Internet Explorer. I earned money. I relearned Javascript and became a first rate web developer.

In the back of my mind I still felt that web development sucks. So a few weeks back when deciding on a platform for a personal project, I thought I’d try something new. The Android platform was in my hands and I gave it a whirl.

I didn’t enjoy it much and I am now rewriting the app as an offline enabled webapp using Google Gears.

Then Chrome OS was announced and I realized that I’ll probably be doing a lot of offline enabled webapps using Google gears and/or HTML 5. Like it or not, it’s the future. Me, I like it. There are a lot of advantages to this kind of setup: I can access the apps from my phone, my laptop, my parent’s desktop, or Phrakture’s hacked computer whenever and wherever I want. I don’t have to write a different client for each one. Its true ‘write once, run anywhere’. I can upgrade each of those clients automatically as long as there’s a network connection.

On that note, you don’t need a network connection to run HTML 5 or Google Gears based apps. They both provide a ‘localserver’ that caches pages and javascripts, and give you an SQLite database for data caching. Typically offline versions of apps are not as powerful as their networked counterparts, but they do not require network access to run. Further, because they are locally cached, they can be made to run as fast as a “standard” (old fashioned) desktop app. The apps run in the browser, but the browser is just a container, a window manager, to hold the application.

In traditional webapps, you code most of the logic on the server side. In this new model, you end up coding most of the logic in the client, because the app needs to run without a guaranteed server connection. For me, this has a massive, nearly show-stopping drawback: A large portion of the app must be written in Javascript. JQuery makes Javascript suck lest, but it still sucks. I’m a Python programmer.

For years, I’ve dreamed of browsers supporting tags that allow me to write my DOM manipulation scripts in Python rather than the ubiquitous and annoying Javascript. This wasn’t possible because python can’t be adequately sandboxed such that arbitrary scripts running on the web don’t have access to, say, your entire hard drive.

This is no longer true. The PyPy project finally has a complete Python 2.5 interpreter that can be safely sandboxed. Since discovering this at Pycon 2009, I’ve been thinking about interfacing it with a web browser.

I figured “somebody must have started this already”. Google didn’t help much, but when I logged into #pypy on freenode I was told “fijal started doing that with webkit yesterday”. I’ve been following up trying to get the project to build (I was warned that the build process is a mess and was invited to wait until it is cleaned up a bit). So far, no luck, but I am optimistic that python support is finally coming to the browser. Granted, it won’t be much use for public webapps (at first) since browsers won’t want to be distributing pypy, but a lot of my projects are personal, and satisfying the general public will be far lower on my priorities list than ‘developing in my preferred language’.

I’ll have to install a pypy interpreter into Chrome Lite under Android before this is useful to me. That may be tricky.