Posts tagged ‘jquery’

pyjaco and jQuery

After giving up on CoffeeScript, I decided to play around with Pyjaco the Python to Javascript Compiler.

While the code is readable, there is very little in the way of end-user documentation. I hope to address this with this blog post. The Pyjaco examples all embed generated javascript in an html page. I needed a way to generate an external Javascript file as I would include in an HTML file. I also wanted to find out if I could use Pyjaco with jQuery.

The first step was to install a development version of Pyjaco:

git clone https://github.com/chrivers/pyjaco.git

Pyjaco normally requires a generated Javascript file mapping Python builtins to Javascript to be included with the created Javascripts. This must be generated:

cd pyjaco
python2 generate_library.py
cp py-builtins.js ~/pyjaco_test # directory for my new page

The next step was to create an HTML file that included jquery, the py-builtins.js script above, and a yet-to be defined javascript file named clicker.js that will be generated from a yet-to-be-defined python file. I also add a couple of DOM elements (a heading and paragraph) that are to be manipulated via jQuery:

<!DOCTYPE html>
<html>
<head>
<script type=”text/javascript” src=”jquery-1.6.4.min.js”></script>
<script type=”text/javascript” src=”py-builtins.js”></script>
<script type=”text/javascript” src=”clicker.js”></script>
</head>
<body>
<h1>Click me</h1>
<p id=”when_clicked”></p>
</body>
</html>

That’s the easy part. Writing python code that compiles to correct Javascript is the hard part. Pyjaco doesn’t currently provide very useful compile-time errors, and it also does not yet map javascript errors back to the input python.

There are (at least) two ways to compile python code in Pyjaco. The first, which is used in the Pyjaco examples, and appears to be the preferred method at this time is to create a custom main() method that uses various pyjaco.Compiler methods to combine the functions into a string of text. See https://github.com/chrivers/pyjaco/tree/devel/examples for three examples.

However, I was looking to have a complete python file that compiles to a complete javascript file. Pyjaco supports this as well, using the provided pyjs.py script. It took some investigating to understand how to reference javascript variables inside python functions. Decorator syntax is used to expose the variables for jQuery, Math.random, and Math.floor in the following example. The mystifying bit is that because we will be compiling this file to javascript as a string, it is not necessary (or possible) to import the JSVar decorator, as was done in the Pyjaco examples linked above.

# clicker.py
@JSVar("jQuery")
def ready():
    jQuery('h1').click(on_click)
 
@JSVar("jQuery", "Math.random", "Math.floor")
def on_click():
    if jQuery('#when_clicked').html():
        r = Math.floor(Math.random() * 255)
        g = Math.floor(Math.random() * 255)
        b = Math.floor(Math.random() * 255)
        color = "rgb(%d, %d, %d)" %(r,g,b)
        jQuery('#when_clicked').attr('style', 'background-color: ' + color)
    else:
        jQuery('#when_clicked').html("you clicked it!")
 
jQuery(ready)

Notice that I’m using the jQuery function instead of the $ alias, since $ is not a valid variable name in Python. This is a rather odd looking mix of Python and Javascript functions, but it works.

I had to repeat the compile and test step a few times before coming up with the above file. The script can be compiled using the pyjs.py that comes with the pyjaco source distribution (and, thanks to a simple patch I submitted, will come with the binary distribution in the next release.) Here’s how the script is run:

python2 ~/code/pyjaco/pyjs.py -N --output clicker.js clicker.py

The -N option tells pyjs not to generate the builtin library that we created manually in the first step.

This translation step creates a clicker.js file that looks like this:

var ready = function() {
    var __kwargs = __kwargs_get(arguments);
    var __varargs = __varargs_get(arguments);
    var $v1 = Array.prototype.slice.call(arguments).concat(js(__varargs));
    jQuery("h1").click(on_click);
return None;
}
var on_click = function() {
    var __kwargs = __kwargs_get(arguments);
    var __varargs = __varargs_get(arguments);
    var $v2 = Array.prototype.slice.call(arguments).concat(js(__varargs));
    if (bool(jQuery("#when_clicked").html()) === True) {
        var r = Math.floor((Math.random()) * (255));
        var g = Math.floor((Math.random()) * (255));
        var b = Math.floor((Math.random()) * (255));
        var color = str('rgb(%d, %d, %d)').PY$__mod__(tuple([r, g, b]));
        jQuery("#when_clicked").attr("style", ("background-color: ") + (color));
    } else {
        jQuery("#when_clicked").html("you clicked it!");
    }
return None;
}
jQuery(ready);

I find this rather unfortunately difficult to read. There is code for argument parsing that would not have been needed if I had hand-written javascript. Further the use of “mock” python builtins makes the javascript look less javascripty. However, the original python file looks much more readable than an equivalent javascript one would. I am hopeful that improvements to pyjaco will cause it to generate more readable javascript with less extraneous code.

The entire example can be found on my github fork

Christian Iversen is actively working on Pyjaco right now. I am excited about this project and hope that further community involvement will help it evolve into a practical and useful tool. I intend my next patch to be an autocompile tool that monitors files in one directory for change and outputs .js files in another directory, one of CoffeeScript’s killer features. I am also considering a port to Python 3.

Django and jquery.tmpl

Lately, I’ve been finding Django increasingly inappropriate for the web applications I develop. I have several complaints: the forms library doesn’t extend well to ajax requests, any extensive customization to the admin requires obtuse inspection of the admin source code, many admin customizations simply aren’t possible, the “reusable apps” philosophy has added a layer of complexity to a lot of things that really should not be there, there are no obvious best practices for ajax support.

In spite of all this Django is still better than other frameworks (Python or not) that I have investigated or tested. I’ve considered writing my own web framework, but I wouldn’t maintain interest in it long enough to get it off the ground. So I’m letting my complaints bounce around in the back of my mind with the hopes that I can improve Django so that it continues to work for me as a platform.

I’m currently trying to come up with a better system for ajax requests. I have nothing concrete in mind, but I’ve started with the premise that ajax requests should never return rendered html, but should return only json (hence my issue with the django forms library). With that in mind, I need a templating library for json data. JQuery is a must, and the officially supported JQuery templating library is jquery.tmpl http://api.jquery.com/category/plugins/templates/

The problem with jquery.tmpl is that it uses django-like syntax. The following is a valid block that may be rendered in a jquery.tmpl page:

<script id="project_tmpl" type="text/x-jquery-tmpl">
    {{each projects}}<li>${$value}</li>{{/each}}
</script>

If you try to include this in a django template, the {{ and }} tags will be replaced with (probably) empty variables. Django has {% templatetag %} to render these individual items, but what we really need is a way to tell the django templating system to leave complete sections of code alone. So I wrote the jqtmpl template tag. It allows us to wrap code in a block that tells Django not to render that block as template code. So the above would show up in a Django template as follows:

<script id="project_tmpl" type="text/x-jquery-tmpl">
{% jqtmpl %}
    {{each projects}}<li>${$value}</li>{{/each}}
{% endjqtmpl %}
</script>

Here’s the template tag:

from django.template import Library, TextNode, TOKEN_BLOCK, TOKEN_VAR
 
register = Library()
 
@register.tag
def jqtmpl(parser, token):
    nodes = []
    t = parser.next_token()
    while not (t.token_type == TOKEN_BLOCK and t.contents == "endjqtmpl"):
        if t.token_type == TOKEN_BLOCK:
            nodes.extend(["{%", t.contents, "%}"])
        elif t.token_type == TOKEN_VAR:
            nodes.extend(["{{", t.contents, "}}"])
        else:
            nodes.append(t.contents)
 
        t = parser.next_token()
 
    return TextNode(''.join(nodes))

This doesn’t handle Django’s {# comments #}, as token.contents doesn’t return a valid value for comment items. As far as I know, you wouldn’t use the comment construct inside a jquery.tmpl template anyway, so it’s still functional.

Next on my list is a better forms validation library to suit my theory that validation should be done client side. I’ve got a server-side system in mind that returns json responses, and does not return user-facing error messages, since those should have been caught client side. With these tools, and hopes for Grappelli to eventually create a decent admin, Django may continue to serve me.

HTML Form validation

I’ve been experimenting with pylons oven the past week. It’s a nice framework, but the recommended html form validation engine, formencode sucks.

I researched alternative form engines, and looked at the form engine I’m most familiar with, in Django with new eyes. It is also a pain to work with. I just never noticed it before. I can’t find anything that works as seamlessly as it should. I think Flatland may be a good solution for pylons, but I have decided that nobody is doing it right yet.

Why is this obviously common problem is so unsolved. What do html form validation frameworks do?

  • (optionally) generate the form in the first place.
  • Validate form data on the server side when form data is supplied
  • Convert the incoming string form data to Python objects or data structures
  • If the form data is not valid, issue a GET request that redisplays the form with error messages

The last step is the messy one. The two possible solutions I’ve seen are to either generate the response (ie: create an entire form with values and error messages) or to modify the response in some sort of middleware (the form is generated from a template, and then the middleware adds value=”" attributes and error messages to the result). Either way, we end up with controller code that is too tightly coupled to view code.

Yet, this isn’t how the web works anymore. The “POST to server and if it’s invalid, display the form with error messages” paradigm is what you might call “uncool.” In awesome apps, we validate the code on the client side, using Javascript, or, even better, HTML 5 form validation (I mean, we would use it if the browsers supported it consistently), preferably as the user types or as soon as the widget is blurred. This is a far better interaction experience.

Wouldn’t it be nice if we could do away with server-side validation altogether? Obviously, we can’t, because we need to respond appropriately if someone issues a malicious POST or tries to POST to our form from a script or scraper instead of the fancy validated web page we displayed to a web browser. The point I’d like to emphasize is that the appropriate response in these situations is NOT to redisplay the HTML form with values filled in and error messages. If it’s a malicious request, we just want to make sure we don’t do anything with it. If it is an invalid request from a script, the script programmer doesn’t want to see in the response, they will just want a succinct error message telling them how to fix the validation in their script.

I’d therefore like to propose properly decoupling the controller and view. The view (html+javascript) does validation, while the server side focuses on converting the incoming data to python objects. If it encounters an error while doing the conversion, it can bail with an appropriate error message (probably a 406 status code). Getting this status code means your webapp is broken –just like a 500 error would indicate– because your webapp should have validated the form input before submitting it. This would also be a suitable response in a REST based design. If a REST client sends invalid data, it should get an HTTP 406 with appropriate error message.

This simplifies the server side code a lot. It no longer has to worry about generating the html form, filling it with values, or returning error messages. The client side code becomes more complicated, but it’s the kind of more complicated that should be done anyway, to enhance the user experience. Ideally, it should be easier to write client-side validation than to default to server-side validation on form requests. It’s too easy to use django’s forms framework to create server-side validation without ever bothering with client-side validation.The coolest websites tell us our passwords don’t match *before* you submit the form. This should be the norm. It should be easier to write this kind of client-side validation than it is to default to server-side validation, so people default to using it.

Going forward, browsers need to support the new HTML 5 input types with proper validation and error messages. I also think there needs to be better support in the html spec (maybe html 5 has it and I don’t know about it?) for displaying validation errors. something like . For more complicated validation, we need a nice javascript framework. I haven’t used it yet, but I’m looking forward to experimenting with the jQuery Validation plugin.

With that in mind, I’m planning to continue using formencode for datatype conversion, but to ignore it’s buggy htmlfill stuff. Maybe eventually I’ll make a simpler stand-alone library that does this stuff and returns a 406 on invalid input.

Validate as you type in javascript

This is a common task, but has several facets that are easy to miss. I needed an input element that would validate as you type to ensure that an integer value was always entered. This was easily done by doing a replace against a regular expression, linked to the keyup event:

function validate_number(event) {
    this.value = this.value.replace([/[^0-9]/g,'');
}
 
$('jquery selector').keyup(validate_number);

This works fine provided the user is typing at the end of the box. The problem is that if the user has clicked in the middle of the number, aiming to insert a new digit, each time this.value is set, the cursor is moved to the end of the box, regardless of whether or not a valid character was entered. This happens on every keypress. That is bad.

The solution is to save and reset the selection range, as follows:

function validate_number(event) {
    if (this.value.match(/[^.0-9]/g)) {
        var start = this.selectionStart-1;
        var end = this.selectionEnd-1;
        this.value = this.value.replace(/[^.0-9]/g,'');
        this.setSelectionRange(start, end);
    }
}

I haven’t found any situations where this code doesn’t work, but there is one other caveat: pasting. If the user pastes a non-numeric value into the field without issuing a keypress (ie: using the right click–> paste menu), the value will not be validated. I haven’t found a way to stop this at the point when the value is pasted, but I’ve found it sufficient to additionally call the validation function when the textbox loses focus:

$('jquery selector').keyup(validate_number).blur(validate_number);

A variation of this code can be used to validate most inputs as the user types; a regular expression that matches whitespace can remove whitespace, phone numbers would look for digits and hyphens, decimal numbers would additionally search for a period, etc.

Authenticated Threaded Comments in Django 1.1

I’m looking to include threaded comments in my latest project, such that only authenticated users can use them. The Django ThreadedComments project seems to have lost momentum; as the old site says that the new version will use the comment hooks provided in Django 1.1 and will be backwards incompatible, but very little fresh development has been done.

Turns out, almost all the required functionality is available for free in the standard Django comments system, and you only need one slightly messy hack.

Authenticated Comments

I want a system that doesn’t ask for name/email/url; you can only post if you’re logged in, and the only visible box is one for comments. Behind the scenes, django automatically fills in these details from the currently logged in user if they are not supplied. It also kindly attaches that user to the comment (so it can be retrieved as comment.user). All you have to do is restrict commenting to logged in users and render the template directly to ignore the fields you don’t need. Like this:

some_template.html

{% if user.is_authenticated %}
    <h4>Submit New Comment:</h4>
    {% get_comment_form for project as comment_form %}
    {% include 'comment_form.html' %}
{% endif %}

comment_form.html

    {% load comments %}
    <form action="{% comment_form_target %}" method="POST">
        {{comment_form.content_type.errors}}{{comment_form.content_type}}
        {{comment_form.object_pk.errors}}{{comment_form.object_pk}}
        {{comment_form.timestamp.errors}}{{comment_form.timestamp}}
        {{comment_form.security_hash.errors}}{{comment_form.security_hash}}
        {{comment_form.comment.errors}}{{comment_form.comment}}
        <span>{{comment_form.honeypot}}</span>
        <br />
    </form>

Most of the variables in comment_form.html are hidden fields to help keep spammers at bay. If you prefer, you can do this same customization a bit higher in the abstraction layer by using a Custom Comment Form that doesn’t contain the url, e-mail and name fields at all. The django.contrib.comments.forms.CommentSecurityForm holds everything you need except a honeypot.

Threaded Comments

Django comments can be attached to any object… including existing comments. That idea and some thoughtful template design provides threaded comments:

some_template.html

{% get_comment_list for project as comments %}
{% for comment in comments %}
    {% include 'comment.html' %}
{% endfor %}

Here I’m getting all the comments attached to a specific “project”. For non-threaded comments, comment.html would basically just contain {{comment}} to render the comment. But I want threaded comments:

comment.html

{% load comments %}
<div class="comment">
    {{comment.user}}
    {{comment.comment}}
    {% get_comment_list for comment as comments %}
    {% with 'comment.html' as comment_include %}
        {% for comment in comments %}
            {% include comment_include %}
        {% endfor %}
        {% if user.is_authenticated %}
        <br /><a href="#">Reply to This Comment</a>
            {% get_comment_form for comment as comment_form %}
            {% include 'comment_form.html' %}
        {% endif %}
    {% endwith %}
</div>

Here, for each comment, I get the list of comments attached to that comment, and render them, along with a box to reply to the comment. This box is hidden by default in my stylesheet and displayed with a bit of jquery. I recursively include the same template for each comment so you can have as many levels of comments as you like (that may be a bad thing).

The recursive include is the messy hack — by default, Django includes templates at parse time. This means the template is included regardless of whether any comments exist, and the Python stack quickly fills up with too many recursive calls. But if the template being included is a variable instead of a string, it has to know what that variable value is and can thus only be executed at render time. So I simply wrap the include with a with template tag. This forces django to use the IncludeNode instead of ConstantIncludeNode (see django/templates/loader_tags.py), which renders the template at render instead of parse time.

Some uncreative styling in my css makes the comments indent a bit and hides the reply box until the jquery link is clicked.

.comment {
    margin-left: 1em;
    border-style: solid;
    border-width: 1px;
    border-color: #aaa;
    border-top-width: 0px;
    border-right-width: 0px;
    padding: 1ex;
}
.comment_form textarea {
    height: 3em;
    width: 40em;
}
.comment .comment_form {
    display: none;
}

Obviously, it could do with a makeover from a talented web stylist, but here’s how the general idea looks:

Nearly-unstyled threaded comments.

Nearly-unstyled threaded comments.