Posts tagged ‘web development’

Pyjaco in a real app: Todos with local storage

I didn’t get the memo, but there appears to be a movement to demonstrate emerging web technologies with a simple todo list application, much as hello world is used to introduce programming languages.

In my last post, I introduced using jQuery with Pyjaco, the PYthon JAvascript COmpiler. Since then, I’ve made several contributions to the project and have been involved in diverse discussions with Pyjaco developers regarding the current and future status of the project. This post goes further by acting as a tutorial for writing a basic todos app using Pyjaco.

Pyjaco is alpha software. It is hard to write valid code, and harder to debug. I’ve managed to both lock up Firefox and hard crash it while using the Pyjaco library.

On the positive side, Pyjaco is under active, rapid development. The head developer, Christian Iversen is extremely responsive to both questions about Pyjaco, and to code contributions. This is a project with a lot of potential, and I see it as the current best bet for Python programmers hoping to avoid javascript one day in the future.

In spite of the hiccups, it is possible to generate a working javascript app using just Pyjaco. Here’s how.

Let’s start:

mkdir pyjados
cd pyjados
virtualenv2 venv --distribute --no-site-packages
source venv/bin/activate
pip install git+https://buchuki@github.com/buchuki/pyjaco.git@run_script

First we create a directory to work in and install a virtualenv. Pyjaco does not currently work with python 3, so in Arch Linux, I use the virtualenv2 command. We then activate the virtualenv and install the pyjaco package. Here I am installing from my personal fork, as it contains some changes for generating the built-in standard library that have not yet been merged upstream. You should normally install directly from chrivers’s git repository using pip install git+git://github.com/chrivers/pyjaco.git.

Now let’s create a basic HTML 5 page with jQuery loaded:

<!DOCTYPE html>
<html>
  <head>
    <title>PyJaco Todo List Example</title>
    <script type="text/javascript" src="http://code.jquery.com/jquery-1.6.4.min.js"></script>
  </head>
  <body>
    <h1>PyJaco Todo List Example</h1> 
  </body>
</html>

We can load this in our web browser using a file:// URL. This is the only HTML page in our app, and it can be refreshed to load our changes as we work.

Pyjaco doesn’t simply translate Python into Javascript. Rather, it creates a basic standard library of Python-like objects that are utilized in the compiled javascript code. I wasn’t too keen on this idea when I first heard it, as it introduces a dependency that currently weighs in at 65K before minification and compression. While this is not a terribly heavy library, there are efforts under way to shrink the builtins or to dynamically generate it to contain only those builtins that your code actually touches. At any rate, we need to ensure this library is available to our code. First we generate the library:

pyjs.py --builtins=generate --output=py-builtins.js

pyjs.py is the name of the pyjaco command. It is expected to be renamed to pyjaco in the future. The --builtins=generate option tells pyjaco to generate the standard library, while the --output flag provides the filename for the new library file:

$ ls
index.html  py-builtins.js  venv

We then need to load this library in the head of our html file. Let’s also load the future pyjados.js script at this time:

  <head>
    <title>PyJaco Todo List Example</title>
    <script type="text/javascript" src="http://code.jquery.com/jquery-1.6.4.min.js"></script>
    <script type="text/javascript" src="py-builtins.js"></script>
    <script type="text/javascript" src="pyjados.js"></script>
  </head>

Now, before we start coding the Python file that will be compiled to Javascript, I want to discuss what I consider to be the most confusing aspect of Pyjaco development. There are basically two types of variables in Pyjaco, Javascript variables, and Python variables. Javascript variables refer to “normal” variables that you would call in Javascript. These include alert, window, document and the like, as well as variables in third-party Javascript libraries, such as the ubiquitous jQuery. Further, any attributes on those objects are also Javascript variables, and the return value of any methods will also be Javascript variables.

Python variables, on the other hand, refer to any variables that you define in your Python source code. If you create a dict or a list, for example, it will be compiled to a list or dict object from the standard library we just generated. In the compiled script, of course these Python variables are represented by Javascript objects, but from the point of view of a Pyjaco coder, it is important to keep the two types of files separate. Almost all the bugs I have encountered in my Pyjaco code have been caused by confusing the two types of variables.

The distinction between python and javascript variables introduces a couple of complications to writing Pyjaco compatible python code. First we need to flag all of our Javascript variables using a decorator on methods that access them. Second, we need to explicitly convert our variables between Javascript and Python any time we access one from the other. I’m told that this conversion can — and one day will — be done automatically by the pyjaco multiplexer, but in the meantime, we need to make it explicit. We do this by using two javascript functions supplied with the standard library we just generated, appropriately named js() and py(). You will see examples of these shortly.

When I finally figured out the distinction, my first thought was, “ok, let’s prefer to always work with python variables.” Therefore, in my initialization code, I tried jQ=py(jQuery). Unfortunately, jQuery is a rather large object, and the py function apparently recursively converts all attributes from javascript to python. I ended up with a stack overflow.

Now, let’s create our first python code and watch it compile to Javascript. Name the file pyjados.py:

def setup():
    print "Pyjados Hello World"
 
jQuery(js(setup));

First we write a python function named setup. This function is a python object. jQuery is a javascript object that expects a javascript object as input. Therefore, we wrap setup in a js() call and pass the result into the jQuery function. jQuery will now run setup when document.ready is fired.

Now we compile the code using the following command inside our activated virtualenv:

pyjs.py --watch pyjados.py --output pyjados.js

You’ll notice the command doesn’t exit. That is the --watch option at work. If you now make a change to pyjados.py and save it, it will automatically recompile it. The output file pyjados.js is regenerated each time. This is the file we included in our html file. So now, open that html file in a web browser using a file:// url. Make sure the Javascript console is displayed and reload the page. You should see the words “Pyjados Hello World” printed on the console. Pyjaco automatically compiles print statements into console.log output.

Before we start implementing our Todo list, let’s look at an example of accessing a javascript variable inside a function. Change setup.py to utilize alert, as follows:

 
@JSVar("alert")
def setup():
    alert(js("Pyjados Hello Alert")
 
jQuery(js(setup));

Did you look closely at that code? There is a missing close bracket on the alert line. You’ll see the syntax error in your console where pyjs.py is watching the compiled code. Add the bracket and let it automatically recompile itself:

 
@JSVar("alert")
def setup():
    alert(js("Pyjados Hello Alert"))
 
jQuery(js(setup));

Let’s analyze this snippet. First, notice how we told the compiler that alert is a Javascript variable when used inside setup(). This is a bit odd, since the JSVar decorator is never actually imported into the namespace. This is a bit of magic in the Pyjaco compiler, just pretend it has been imported.

Second, notice that since alert has been flagged as a JSVar, it must accept a Javascript variable. However, the string “Pyjados Hello Alert” is a Python variable. Therefore, we convert it using js() as we pass it into the alert call.

Now let’s prepare to create some working todo-list code. Start by adding a form for submitting todos and a list to render the todos to the html body:

  <body>
    <h1>PyJaco Todo List Example</h1> 
    <form id="add_todo_form">
      <input type="text" id="add_box" placeholder="Add Todo", autofocus="autofocus">
      <button id="add_button">Add Todo</button>
    </form>
    <ul id="todo_items"></ul>
  </body> 

Nothing too exciting here. Note the ids on the elements, since we’ll be utilizing these from Pyjaco using jQuery selectors.

Now back into the python file. Let’s create a class to manage the various todo elements:

class TodosApp:
    @JSVar("jQuery", "js_add_form")
    def __init__(self):
        js_add_form = jQuery(js("#add_todo_form"))
        js_add_form.submit(js(self.add_todo))
 
    def add_todo(self, event):
        print "form submitted"
        return js(False)
 
def setup():
    todo_app = TodosApp()
 
jQuery(js(setup));

The __init__ function hooks up the form’s submit button to a method on the object. Notice that we need to flag not just jQuery, but also js_add_form as a javascript variable. Pyjaco does not (currently) know that a javascript variable is returned when calling a method on an existing javascript variable. I like to add the js_ prefix to variable names to help remind myself that this is a javascript variable.

In an ideal world, we could convert this variable to a Python variable using py(), but as noted earlier, calling py on a jQuery object results in a stack overflow or browser crash.

Also pay attention to the way we wrap the self.add_todo method name in a js() call when we pass it into the submit handler. The submit method is a javascript function expecting a javascript object.

The def add_todo method has its single parameter flagged as a @JSVar, since the method is being called internally by jQuery when the event occurs. We also wrap the False return value (to prevent event propogation on the submit handler) in a js() call so that jQuery recognizes it as a javascript false rather than a (true) object named False.

Try the code. Ensure the compiler recompiled it, and reload the html file. Enter some characters into the text box and use the Enter key or the Add Todo button to submit the form. The words form submitted should be displayed in the javascript console.

Now let’s actually store and render a newly added todo. The todos are stored in memory in a python dict object. Initialize this object by adding the following two lines of code to the end of __init__:

    self.todos = {}
    self.next_id = 1

And rewrite add_todo as follows as well as a new method named render

    @JSVar("event", "js_add_box")
    def add_todo(self, js_event):
        js_add_box = jQuery(js("#add_box"))
        self.todos[self.next_id] = py(js_add_box.val())
        js_add_box.val('')
        js_add_box.focus()
        self.next_id += 1
        self.render()
        return js(False)
 
    @JSVar("js_todo_items")
    def render(self):
        js_todo_items = jQuery(js("#todo_items"))
        js_todo_items.html("")
        for id, todo in sorted(self.todos.items()):
            js_todo_items.append(js('<li>%s</li>' % (id, todo)))

Note that the todos dict is a Python object, so when we insert the value of the js_add_box into it, we must convert it from a javascript object using py(). Also note how, because we are writing in a python function, manipulating the python value self.next_id requires no conversion, and calling the python function self.render is also clean.

In the render function itself, I think it’s pretty cool that string formatting using % is supported by pyjaco (as an aside, the str.format method introduced in python 2.6 is not yet available) and that the python sorted() function is available. Note also how we can loop over items() on the self.todos dictionary just as if we were using a normal python dictionary.

Now let’s add the ability to complete todos. Let’s start by adding a template string as a class variable, and use that string inside the render function. This illustrates that pyjaco supports class variables:

class TodosApp:
    list_item_template = """<li>
    %(text)s
    </li>"""

and we change the for loop in render to:

        for id, todo in sorted(self.todos.items()):
            js_todo_items.append(js(TodosApp.list_item_template % {
                "id": id,
                "text": todo}))

Reload the page again and notice how checkboxes have been displayed beside each todo. The next step is to make clicking these boxes actually complete the todos. We add a couple lines to our __init__ method to connect a live click event to the checkbox items, which now looks like this:

    @JSVar("jQuery", "js_add_form", "js_checkbox")
    def __init__(self):
        js_add_form = jQuery(js("#add_todo_form"))
        js_add_form.submit(js(self.add_todo))
        js_checkbox = jQuery(js("input[type=checkbox]"))
        js_checkbox.live("click", js(self.complete_todo))
        self.todos = {}
        self.next_id = 1

Don’t forget to add js_checkbox to the JSVar decorator.

The complete_todo method looks like this:

    @JSVar("event", "jQuery", "todo_item")
    def complete_todo(self, event):
        todo_item = jQuery(event.target).parent()
        id = int(py(todo_item.attr("id"))[5:])
        del self.todos[id]
        todo_item.delay(1500).fadeOut("slow")

The first line is using exclusively javascript arguments, and returns the <li> element containing the checkbox that was clicked. The id = line converts the javascript string id attribute of this element (which looks like “todo_5“, as defined in list_item_template) into the python integer id of the todo. The remaining lines simply remove that todo from the internal list and from the DOM, after a 1.5 second delay.

In fact, we now have a fully functional todo list that allows adding todos and checking them off. Now, as a bonus, let’s try hooking this up to the HTML 5 localStorage object so that the list is maintained across page reloads. We start by adding a store() method to our class:

    @JSVar("localStorage", "JSON")
    def store(self):
        localStorage.setItem("todolist", JSON.stringify(js(self.todos)))

The main line of code is easiest to read from the inside out. First we convert the self.todos dict to a normal javascript object using the js() function. Then we call JSON.stringify on this object to create a string suitable for insertion into localStorage.

Now add this call to the end of the two methods that manipulate the todo list, add_todo and complete_todo:

        self.store()

.

Refresh the page, add a couple todos, and inspect the localStorage object in your console. You should see the stringified dict in the todolist value.

Now all we have to do is ensure the self.todos dict is loaded from localStorage when the app is initialized. Add the following to the end of the __init__ method (make sure to add js_stored_todos to the JSVars decorator):

        js_stored_todos = localStorage.getItem("todolist")
 
        if js_stored_todos:
            stored_dict = dict(py(JSON.parse(js_stored_todos)))
            self.todos = dict([(int(i), stored_dict[i]) for i in stored_dict.keys()])
            self.next_id = max(self.todos.keys()) + 1
 
        self.render()

Note that calling py() on the output of JSON.parse creates a python object, not a python dict. The code is therefore wrapped in a call to dict(), which converts the object to a dictionary.

Unfortunately, the resultant dict contains keys that are strings, whereas our original dict used integer keys. So a pure-python list comprehension is used to convert the dictionary to one with integer keys. This line is a bit hard to read, but I wanted to include it to demonstrate that Pyjaco can parse list comprehensions. Finally, we set self.next_id using the python max() call, which Pyjaco also automatically translates into javascript.

Try it out. Load the pyjados HTML file, add some todos, check a few of them off, then close and reload the web browser. Your todos will be stored!

I hope you’ve enjoyed this introduction to Pyjaco. It is a nice tool with a lot of potential. Currently, I find writing Pyjaco code to be approximately equally tedious to writing Javascript code. However, I feel that as I learn the ins and outs of Pyjaco, and as the developers continue to refine and improve the compiler, Pyjaco may one day be a perfectly viable alternative to writing pure Javascript or to the rather too Ruby-esque, but otherwise excellent Coffeescript.

codemirror > editarea

So you’re working on some kind of webapp that needs some in-browser code editing functionality of some sort. You search google and discover that most folks are using editarea these days. You figure it must work to be so popular and decide to try it out.

Then as you start to use it, you realize it’s unmaintained, doesn’t work terribly well in Webkit browsers, doesn’t work at all in Internet Explorer 9, is poorly documented and hard to extend, expand, or fix. You’re frustrated.

But repeated web searches do not yield anything as functional, and you resign yourself to a half-working editor, or maybe even decide to write something from scratch.

I’ve been that person, and I was recently lucky enough, while searching for something completely unrelated, to stumble across CodeMirror a standards-compliant javascript-powered code editor that is well documented. It’s not as featureful as editarea, but if you add CodeMirror UI to the mix, you end up with a comparable featureset that actually works.

I’m not normally one for simply reposting links, but as I had so much trouble finding CodeMirror in the first place, I hope this post will either help others to find it, or at least increase CodeMirror’s page rank in search results.

Rendering Django models to JSON

I recently discussed a simple template tag that allows rendering “raw” jquery templates inside a Django template, even though the syntax overlaps. A commenter asked for a Pypi package. It was such a simple template tag that I didn’t want to maintain a separate package for it, but it turns out, this tag fits in very well with the next step in the project. The short story is that the tag is now available on Pypi.

The tag is distributed with a project I called Djason (named for Django, JSON, and former Arch Linux Developer, Jason Chu, and because djson was already taken). Djason’s primary purpose is to serialize Django models to a json client format that is friendly to client libraries, especially to jquery-tmpl. It is essentially just a fork of Django Full Serializers with a patch applied, and some minor edits to simplify the output json. I install it as a Django serializer in settings.py using:

SERIALIZATION_MODULES = {
    'djason': 'djason.json'
}

And in my code:

from django.core.serializers import serialize

    content = serialize("djason", project,
            relations=["location_set", "company", "project_type"])
    return HttpResponse(content)

I’ll probably be adding some additional code to this package to fit in with my still cloudy ideas about the proper separation between client and server code. I’ll also likely be rewriting portions of the forked project to appease my personal sense of style.

HTML Form validation

I’ve been experimenting with pylons oven the past week. It’s a nice framework, but the recommended html form validation engine, formencode sucks.

I researched alternative form engines, and looked at the form engine I’m most familiar with, in Django with new eyes. It is also a pain to work with. I just never noticed it before. I can’t find anything that works as seamlessly as it should. I think Flatland may be a good solution for pylons, but I have decided that nobody is doing it right yet.

Why is this obviously common problem is so unsolved. What do html form validation frameworks do?

  • (optionally) generate the form in the first place.
  • Validate form data on the server side when form data is supplied
  • Convert the incoming string form data to Python objects or data structures
  • If the form data is not valid, issue a GET request that redisplays the form with error messages

The last step is the messy one. The two possible solutions I’ve seen are to either generate the response (ie: create an entire form with values and error messages) or to modify the response in some sort of middleware (the form is generated from a template, and then the middleware adds value=”" attributes and error messages to the result). Either way, we end up with controller code that is too tightly coupled to view code.

Yet, this isn’t how the web works anymore. The “POST to server and if it’s invalid, display the form with error messages” paradigm is what you might call “uncool.” In awesome apps, we validate the code on the client side, using Javascript, or, even better, HTML 5 form validation (I mean, we would use it if the browsers supported it consistently), preferably as the user types or as soon as the widget is blurred. This is a far better interaction experience.

Wouldn’t it be nice if we could do away with server-side validation altogether? Obviously, we can’t, because we need to respond appropriately if someone issues a malicious POST or tries to POST to our form from a script or scraper instead of the fancy validated web page we displayed to a web browser. The point I’d like to emphasize is that the appropriate response in these situations is NOT to redisplay the HTML form with values filled in and error messages. If it’s a malicious request, we just want to make sure we don’t do anything with it. If it is an invalid request from a script, the script programmer doesn’t want to see in the response, they will just want a succinct error message telling them how to fix the validation in their script.

I’d therefore like to propose properly decoupling the controller and view. The view (html+javascript) does validation, while the server side focuses on converting the incoming data to python objects. If it encounters an error while doing the conversion, it can bail with an appropriate error message (probably a 406 status code). Getting this status code means your webapp is broken –just like a 500 error would indicate– because your webapp should have validated the form input before submitting it. This would also be a suitable response in a REST based design. If a REST client sends invalid data, it should get an HTTP 406 with appropriate error message.

This simplifies the server side code a lot. It no longer has to worry about generating the html form, filling it with values, or returning error messages. The client side code becomes more complicated, but it’s the kind of more complicated that should be done anyway, to enhance the user experience. Ideally, it should be easier to write client-side validation than to default to server-side validation on form requests. It’s too easy to use django’s forms framework to create server-side validation without ever bothering with client-side validation.The coolest websites tell us our passwords don’t match *before* you submit the form. This should be the norm. It should be easier to write this kind of client-side validation than it is to default to server-side validation, so people default to using it.

Going forward, browsers need to support the new HTML 5 input types with proper validation and error messages. I also think there needs to be better support in the html spec (maybe html 5 has it and I don’t know about it?) for displaying validation errors. something like . For more complicated validation, we need a nice javascript framework. I haven’t used it yet, but I’m looking forward to experimenting with the jQuery Validation plugin.

With that in mind, I’m planning to continue using formencode for datatype conversion, but to ignore it’s buggy htmlfill stuff. Maybe eventually I’ll make a simpler stand-alone library that does this stuff and returns a 406 on invalid input.

Get your head out of the clouds: Local web applications

I spent this weekend with two friends crazy enough to join me in a 48 hour coding sprint for the Django Dash. We competed in the dash last year and placed 5th. Our goal was to move up in the rankings this year (competition is stiff, wish us luck!). Our team had the highest number of commits, but I can’t say how many of them can be justified as quality commits… especially since we keep track of our TODO file inside the git repository!

This year, we created a project called Great Big Crane. (I don’t know why we called it this.) The code is stored on Github, and we set up a splash page at greatbigcrane.com. We don’t have a live demo for reasons I’ll get into shortly.

This project’s primary purpose is to help managing buildouts for Python projects, most especially Django projects. It helps take care of some of the confusing boilerplate in buildout configuration. It also allows one click access to common commands like running bootstrap or buildout, syncdb, manage.py, or migrate, and running the test suite associated with a buildout. It does most of these actions as jobs in the background, and pops up a notification when it completes. It even keeps track of the results of the latest test suite run so you can see at a glance which of your projects are failing their tests.

One of the most intriguing things this application does is open a text editor, such as gvim, to edit a buildout if you need more control than our interface provides. It does this be queuing a job that executes the text editor command on the server.

Wait, What? It can be a bit creepy when clicking a button in a web application fires up an arbitrary program on your computer.

This entire app is designed to run on localhost. It’s set up for developers to manage their own projects. It doesn’t support authentication (this is why we don’t have a live demo), and the server has full access to the local filesystem. It’s meant to support your local IDE, not to provide an online IDE. The entire app is therefore super fast (no network delay), and switching from it to my text editor to several terminals became quite normal as I was developing on it (yes, the buildout for Great Big Crane runs just fine from inside Great Big Crane ;).

So yes, you’re expected to run this web app locally. Why would anybody want to do this? Is it a sensible thing to do?

The alternative to what we’ve done here would be to code the whole thing up as a GUI application of some sort. I have experience with most of the Python GUI toolkits, and I can’t say that I “enjoy” working in any of them. I’m not sure I enjoy working in HTML either, but I do a lot of it. HTML 5 with CSS 3 is certainly a powerful and reasonable alternative to modern graphical toolkits.

I’ve been coding HTML for so long that I don’t know what the learning curve is for it, but I’m definitely more comfortable working with it than I am with TK, QT, GTK, or WxWidgets, all of which take a long time to learn how to code properly. Possibly I’m just stagnating, but I think I’d prefer to develop my next “desktop” app as a webapp intended to run locally, rather than study these toolkits again. Indeed, because I think I’d prefer to do that, I started coding my last project in PyQT, just to fight the stagnation tendency. PyQT is an incredibly sensible toolkit, after you have learned how to make sense of it, but it’s not as sensible as the new web standards. Another advantage is that if you ever decide you want to make the app network enabled, you’re already running an app server, and using standard web technologies to push it to the cloud.

So my gut feeling at this point is that yes, it is sensible to design “traditional” desktop apps using HTML 5, CSS, and javascript for the interface, and your choice of webserver and web framework for the backend. Perhaps it’s not any more sensible than using a GUI toolkit, but it’s certainly not insane.

If it makes sense to replace local desktop apps with a local server, does it also make sense to replace web applications with a local system?

I’m not a huge fan of web applications because they are slow for me. I have a good connection (by Canadian standards, which aren’t high…). Yet Gmail is slower than Thunderbird, Freshbooks is too slow for me to justify paying for it, and github, while fascinating, is also slow compared to local access. The only webapp I have tested that I consider responsive is Remember The Milk, a popular todo list. I’m not certain what they do to make it so responsive, but I suspect Google Gears or HTML 5 localstorage must be involved.

Local storage. I’ve written about this before (I must be getting repetitive). My idea then was that offline enabled webapps are just as responsive as desktop apps. But the current available paradigm, using HTML5 localstorage, requires a lot of overhead normally involving manual syncing between the browser data and the server. What if I was running the app locally instead? Then I could just design it as a “normal” web app, without having to put extra thought into designing and maintaining local storage in the browser. It would be super responsive when I access it from my computer. More interestingly, it would also be available from remote computers. If I accessed it across my LAN using another laptop or my phone’s wifi, it would still be acceptably responsive. And if I happen to need access from the library or my friend’s computer, I can log in remotely, and still have approximately the same level of responsiveness that I currently get by logging into a server in the cloud.

This isn’t a new idea. It’s been presented as a “gain control of your own data” alternative to the privacy and control fears that Google, Facebook, and Apple (among others) have been creating. (<a href="http://www.h-online.com/open/features/Interview-Eben-Moglen-Freedom-vs-the-Cloud-Log-955421.html"this Is a nice discussion). There are a lot of clear advantages of moving data local, but there are also disadvantages. The nice thing about cloud storage is not having to worry about data backup. The “access anywhere” paradigm is nice, too, although that is not ruled out with running a home webserver. Zero install and end users not having to think about dependencies is also nice.

Overall, I’m finding more and more reasons to bring our apps home, where we have control of them. Such cycles are common in the technology industry. Dumb terminals mainframes. Personal computers. Business networks. The Internet. The cloud. Off-board video/On-board video. Network cards? On-board nic. Hardware modems or Software modems. Personally, I think the cycle away from the cloud is just beginning. I think the company that finally conquers Google will be doing it by giving you back control of your data. I’ve never been totally comfortable with the whole web application idea (as a user — they’re fine to develop!). I’m still trying to identify what my reasons are, but in the meantime, we experimented with the idea by developing Great Big Crane as a local web application.

Validate as you type in javascript

This is a common task, but has several facets that are easy to miss. I needed an input element that would validate as you type to ensure that an integer value was always entered. This was easily done by doing a replace against a regular expression, linked to the keyup event:

function validate_number(event) {
    this.value = this.value.replace([/[^0-9]/g,'');
}
 
$('jquery selector').keyup(validate_number);

This works fine provided the user is typing at the end of the box. The problem is that if the user has clicked in the middle of the number, aiming to insert a new digit, each time this.value is set, the cursor is moved to the end of the box, regardless of whether or not a valid character was entered. This happens on every keypress. That is bad.

The solution is to save and reset the selection range, as follows:

function validate_number(event) {
    if (this.value.match(/[^.0-9]/g)) {
        var start = this.selectionStart-1;
        var end = this.selectionEnd-1;
        this.value = this.value.replace(/[^.0-9]/g,'');
        this.setSelectionRange(start, end);
    }
}

I haven’t found any situations where this code doesn’t work, but there is one other caveat: pasting. If the user pastes a non-numeric value into the field without issuing a keypress (ie: using the right click–> paste menu), the value will not be validated. I haven’t found a way to stop this at the point when the value is pasted, but I’ve found it sufficient to additionally call the validation function when the textbox loses focus:

$('jquery selector').keyup(validate_number).blur(validate_number);

A variation of this code can be used to validate most inputs as the user types; a regular expression that matches whitespace can remove whitespace, phone numbers would look for digits and hyphens, decimal numbers would additionally search for a period, etc.

Django Registration Paypal Backend

One of my clients needed integration with Paypal for his site. The django-paypal module is great for this sort, but it didn’t quite suit our needs because we wanted to disallow account logins until payment had been collected. We had been using django-registration to support registration, and I decided to stick with it. The development version of django-registration has support for different registration backends. The default version sends an e-mail to verify the user’s e-mail address before they can get in. A simpler backend included with the project allows direct registrations with no verification. We needed a backend that required, not only e-mail address verification, but also confirmation that a paypal payment had been made.

And so, django-registration-paypal was born:

http://pypi.python.org/pypi/django-registration-paypal/0.1.1

http://github.com/buchuki/django-registration-paypal

It is, like all my first releases, pretty rough around the edges, and contains at least one glaring security hole. But it’s out in the wild now, and patches, as always, are welcome.

Web Applications in Python 3

Everyone knows that there aren’t any decent web stacks in Python 3, right? Nobody seems interested in porting Django or Turbogears or Pylons or Plone or Zope to the new, improved much more fun to write language.

Almost true. There are a few less popular frameworks for Python 3, including my own half-finished psyclone: http://archlinux.me/dusty/2010/01/17/python-3-web-framework/

But there aren’t any full stack frameworks. But it’s not too hard to hack one together using available tools.

In my mind, a full stack web framework requires several features:

  • The core server
  • A database engine
  • A templating languages

In addition, it’d be nice if it supported sessions out of the box and had a built-in auth framework. Basically, the more like Django it is, the better.

The good news is, we can hack one together with popular Python 3 ready tools including:

CherryPy provides a working web application engine. SQLAlchemy, which supports sqlite and posgres under Python 3 provides us an extremely flexible and robust ORM. Jinja is an advanced Django-like templating system. I implemented a very basic blog (sans authentication) in these tools in a matter of an hour or so, and it wasn’t any worse to work with than Turbogears. This isn’t a big surprise, since Turbogears is built on CherryPy and SQLAlchemy as well. We don’t have access to the extensive library support Turbogears has to offer, but it is now accurate to say that Python 3 supports a (nearly) full stack web framework.

You’ll need the svn version of CherryPy, and the newly released 0.6 version of SQLAlchemy. Jinja 2 cooperates well.

I don’t find it as pleasant as working with Django (I’m actually not a huge fan of Alchemy’s syntax), but it’s certainly a working solution. It wouldn’t overly much library support to turn it into a really nice system.

Password validation hints

Just because something can be validated does not mean it should be validated. It’s very easy to validate form fields in django and most other web frameworks. That does not mean we should always take advantage of this feature.

Why did I just get this error when creating a user account on a website?:


Error. The password field can contain only letters and numbers

I had included a couple of punctuation characters in my password, because that makes it harder to guess, right?

From a technical standpoint, there is absolutely no reason for this website to tell me I can’t use punctuation characters. If they’re encrypting ascii, then any ascii character should be legit. If they’re encrypting bytestreams, then any unicode byte should be legit.

The only validation a password field should have is to test if the password is ‘too easy.’ Typically a minimum length test is enough, but ensuring the user didn’t enter five 1s or their username as a password can be good validation too (although it’ll annoy the user, not often a good thing). You may also need a maximum length if your database is poorly designed, but make it a very high maximum in case anal-retentive people with 64 character passwords want to buy something from you. After all, why shouldn’t you let them?

Further, don’t force your users to have passwords that conform to arbitrary rules like ” must contain at least one number, one lower case letter, and one capital.” This actually cuts down the total number of options a brute force attacker needs to check if they want to break the password; they now know that eerieairplane is not an option they have to test. After all, eerieairplane and EerieAirPlane are totally different passwords, neither is more “guessable” than the other (unless you are a pilot for Lake Eerie Air, in which case you’re probably better off using pokertoMatotoOthpaste).

Users have different systems for creating their passwords; some of these systems aren’t very intelligent (same password everywhere, or prepend the name of the site to a common word), but forcing a single system of our own on them is even less intelligent.

While we’re on the subject, what’s up with all the corporate sites who believe that having security questions in case you forgot your password helps make things more secure? Honestly, do you think that my random password with extra punctuation is easier to guess than my first dog’s breed, my mother’s maiden name, or my favourite author? What’s the point of having a password at all if both I and any given attacker can just look up these values instead?

A Python 3 Powered Blog

Last week, I posted my intent to port the Tornado web framework to Python 3. Not only have I done that (sort of), but I’ve hacked it to pieces; soon it will be unrecognizable to the original developers, and possibly, to me!

It didn’t take long to get the Hello World example included with Tornado running. Quite a bit more work was the blog demo, but I now have the template, auth, and httpclient modules working, along with the core modules for a working async server. I was able to log into my example blog with my google account, compose some entries, view, and edit them, including the feed.

That doesn’t sound like much if you’re coding blog software for an existing framework (10 minutes in django, probably about 12 in web.py). But if you’re coding a web framework for existing blog software, it’s an accomplishment. I’m proud to have a “working” framework (even though it’s full of bugs and working just means “those things I’ve fixed”) for Python 3 in such a short amount of time.

I’ve named the project psyclone, a play on ‘tornado’ –> ‘cyclone,’ and the fact that Python projects oughta have a ‘p’ and ‘y’ in them somewhere. The code is on github for all to play with. Patches welcome! :-)

I’m having a lot of fun with this project, so it’s taking more of my time than I ought to be devoting to it… on the positive side, it’s progressing rapidly!

My plans:

  • Go over the existing code and improve some of the rather messy unicode/str –> str/unicode hacks I had to make to get it working.
  • Write some tests. (The Tornado team seems not to value tests.) I’ll use py.test and may need to write a test client.
  • Write a session framework and auth framework; the current auth framework uses openID only; but I like a more local solution to be available as well.
  • Consider writing an ORM. Likely, I’ll discard this idea, arguing that Judd was right to design a frameworkwith SQL only. The truth behind the argument will be laziness, of course.