HTML Form validation

I’ve been experimenting with pylons oven the past week. It’s a nice framework, but the recommended html form validation engine, formencode sucks.

I researched alternative form engines, and looked at the form engine I’m most familiar with, in Django with new eyes. It is also a pain to work with. I just never noticed it before. I can’t find anything that works as seamlessly as it should. I think Flatland may be a good solution for pylons, but I have decided that nobody is doing it right yet.

Why is this obviously common problem is so unsolved. What do html form validation frameworks do?

  • (optionally) generate the form in the first place.
  • Validate form data on the server side when form data is supplied
  • Convert the incoming string form data to Python objects or data structures
  • If the form data is not valid, issue a GET request that redisplays the form with error messages

The last step is the messy one. The two possible solutions I’ve seen are to either generate the response (ie: create an entire form with values and error messages) or to modify the response in some sort of middleware (the form is generated from a template, and then the middleware adds value=”" attributes and error messages to the result). Either way, we end up with controller code that is too tightly coupled to view code.

Yet, this isn’t how the web works anymore. The “POST to server and if it’s invalid, display the form with error messages” paradigm is what you might call “uncool.” In awesome apps, we validate the code on the client side, using Javascript, or, even better, HTML 5 form validation (I mean, we would use it if the browsers supported it consistently), preferably as the user types or as soon as the widget is blurred. This is a far better interaction experience.

Wouldn’t it be nice if we could do away with server-side validation altogether? Obviously, we can’t, because we need to respond appropriately if someone issues a malicious POST or tries to POST to our form from a script or scraper instead of the fancy validated web page we displayed to a web browser. The point I’d like to emphasize is that the appropriate response in these situations is NOT to redisplay the HTML form with values filled in and error messages. If it’s a malicious request, we just want to make sure we don’t do anything with it. If it is an invalid request from a script, the script programmer doesn’t want to see in the response, they will just want a succinct error message telling them how to fix the validation in their script.

I’d therefore like to propose properly decoupling the controller and view. The view (html+javascript) does validation, while the server side focuses on converting the incoming data to python objects. If it encounters an error while doing the conversion, it can bail with an appropriate error message (probably a 406 status code). Getting this status code means your webapp is broken –just like a 500 error would indicate– because your webapp should have validated the form input before submitting it. This would also be a suitable response in a REST based design. If a REST client sends invalid data, it should get an HTTP 406 with appropriate error message.

This simplifies the server side code a lot. It no longer has to worry about generating the html form, filling it with values, or returning error messages. The client side code becomes more complicated, but it’s the kind of more complicated that should be done anyway, to enhance the user experience. Ideally, it should be easier to write client-side validation than to default to server-side validation on form requests. It’s too easy to use django’s forms framework to create server-side validation without ever bothering with client-side validation.The coolest websites tell us our passwords don’t match *before* you submit the form. This should be the norm. It should be easier to write this kind of client-side validation than it is to default to server-side validation, so people default to using it.

Going forward, browsers need to support the new HTML 5 input types with proper validation and error messages. I also think there needs to be better support in the html spec (maybe html 5 has it and I don’t know about it?) for displaying validation errors. something like . For more complicated validation, we need a nice javascript framework. I haven’t used it yet, but I’m looking forward to experimenting with the jQuery Validation plugin.

With that in mind, I’m planning to continue using formencode for datatype conversion, but to ignore it’s buggy htmlfill stuff. Maybe eventually I’ll make a simpler stand-alone library that does this stuff and returns a 406 on invalid input.

3 Comments

  1. Xentac says:

    What about cases where you can’t check the validity based solely on the form data, such as checking for duplicates on the server? Would you be able to return enough information in a 500 or a 406 that told the form which part was wrong? Trying to do on-the-fly duplicate checking would probably become expensive too, wouldn’t it be better to do it at submit time instead of as-you-type?

    • dusty says:

      It could be done at submit time, but still client side (ie: in on-submit). You’d need an ajax request, but the purpose of that request would be a single action: find out if there is a duplicate. This is much better division of responsibility than handing it off to the server.

      I imagine there are situations where the server needs to do validation and return a form, but I feel that this needs to be discouraged, rather than encouraged.

  2. cactus says:

    I found pylons to be kind of messy, and didn’t end up getting too far ‘into it’.
    I was put off by their routing implementation. First the default version doesn’t separate the http methods, but just gloms them together in one handler class. I really don’t like that.
    There is a REST controller, but it uses docstrings, which seems like a poor choice to me.

    I vastly prefer tornado (or cyclone if you need twisted) and web.py’s take on routing — I even like bottle’s methodology of using http verbs as decorators more than pylon’s goofiness.