Archive for September 2010

Introducing Prickle, a time tracking tool

I’ve been doing a lot of Django coding lately, and I’ve been doing it very well. But I’ve felt bit of stagnation settling in. I’ve only been coding what I know. I needed to explore, experiment.

So I picked a couple of technologies I’ve heard about but haven’t tried, and started coding. The goal wasn’t to make something useful, it was to learn some new APIs and frameworks. I accidentally made something useful.

So here I present Prickle. Prickle is a time tracking tool for freelancers written using Pylons, CouchDB, and Jinja2. (Those are the technologies I chose to play with.) I decided to write an app to replace Freckle, a stay-out-of-your-way time tracking tool that has been highly recommended to me by favorite alternate freelancer, Dan McGee. Freckle is a great service, but I have a couple problems with it: I hate hosting my data on other people’s machines, and I hate paying money to use a service. Granted, at $12 a month, it’s going to take me years to earn back the hours I put into developing Prickle, but that’s not the point. Replacing Freckle in my daily use was just a little bonus I got while studying Pylons and CouchDB.

All Prickle does is allow me to enter the number of hours I’ve worked on a project, along with a description. It makes entering hours as simple as possible; the date is selected semi-automatically. Hours can be entered in a variety of formats (0:30, 30, 0.5, and .5 all map to 30 minutes 1, 1:00 1.0 all map to 1 hour), so it tends to “just work”. Project names are autocompleted, so it’s just a few keypresses to get to the description box. Now I type a description, press and I’m done.

There are a bunch of views of the timesheets that allow me to see how much uninvoiced time has been spent on a particular project or during a particular month or day. If I set a rate for a project, I also get a summary of how much money I’ve earned on a given day or month, or how much I’m going to have to bill a client for a specific project.

And Prickle does invoicing. It’s very simple invoicing, but very simple is all that I need. It summarizes the hours since the last invoice was created, displays them in an invoice template (I currently use print to pdf to generate the invoice. Maybe later I’ll automate sending an e-mail to clients). Even here, Prickle has some handy helpers to speed up the process. It keeps track of billing addresses for a given project, so I don’t have to enter it each time, and it guesses what the next invoice number should be so I often don’t have to type it in.

Prickle doesn’t have all the fancy graphs and reports that Freckle has, but it quickly answers the primary questions that I ask of my data: How many hours have I billed today? How much money have I earned this month? Am I on target for this project’s budget?

Prickle is open source, and I’m hoping some people will find it useful enough to contribute back to it. I use it daily already, and don’t really have any complaints. Some things I intend to add include:

* A timer with a pause button. I hated this when I first tried Freckle, but it grew on me.
* improved historic view of invoices
* editing or deleting time entries

Some things I invite others to add include:
* authentication. I like to run things locally, so auth wasn’t important to me. It’s also confusing to implement in Pylons.
* styling. I applied some semi-random CSS rules, but I know it’s ugly.
* browser support. This thing works great in chromium, but I’m using some of the most experimental stuff: html 5 inputs, CSS 3, etc. It’d be nice to add some Javascript to make it work on other browsers.

So, Dusty, now that you’ve played with Pylons and CouchDB, what’s your take?

CouchDB is pretty cool to work with. Map/Reduce based queries take some getting used to, but once you’ve learned them, that’s all you have to know. There’s no tuning SQL, the ORM is a very thin layer, it just works. If CouchDB is as scalable as they say it is, I think I’d like to use it some more.

Pylons is kinda nice. Formencode sucks, but other than that, the libraries bundled with or suggested for pylons are pretty intuitive. I’m finding Django is a bit over-engineered these days, or maybe I’ve just been pushing its limits. I was hoping Pylons would be a less bossy solution, but I don’t think I’ll be switching to it or suggesting it to my clients anytime soon. It seems a bit rough around the center, and doesn’t seem any less complicated than Django, in practice.

Next on my list of tools to play with is node.js. I also want to play with web2py, and I may try a zope 3 app just for fun someday, too.

HTML Form validation

I’ve been experimenting with pylons oven the past week. It’s a nice framework, but the recommended html form validation engine, formencode sucks.

I researched alternative form engines, and looked at the form engine I’m most familiar with, in Django with new eyes. It is also a pain to work with. I just never noticed it before. I can’t find anything that works as seamlessly as it should. I think Flatland may be a good solution for pylons, but I have decided that nobody is doing it right yet.

Why is this obviously common problem is so unsolved. What do html form validation frameworks do?

  • (optionally) generate the form in the first place.
  • Validate form data on the server side when form data is supplied
  • Convert the incoming string form data to Python objects or data structures
  • If the form data is not valid, issue a GET request that redisplays the form with error messages

The last step is the messy one. The two possible solutions I’ve seen are to either generate the response (ie: create an entire form with values and error messages) or to modify the response in some sort of middleware (the form is generated from a template, and then the middleware adds value=”" attributes and error messages to the result). Either way, we end up with controller code that is too tightly coupled to view code.

Yet, this isn’t how the web works anymore. The “POST to server and if it’s invalid, display the form with error messages” paradigm is what you might call “uncool.” In awesome apps, we validate the code on the client side, using Javascript, or, even better, HTML 5 form validation (I mean, we would use it if the browsers supported it consistently), preferably as the user types or as soon as the widget is blurred. This is a far better interaction experience.

Wouldn’t it be nice if we could do away with server-side validation altogether? Obviously, we can’t, because we need to respond appropriately if someone issues a malicious POST or tries to POST to our form from a script or scraper instead of the fancy validated web page we displayed to a web browser. The point I’d like to emphasize is that the appropriate response in these situations is NOT to redisplay the HTML form with values filled in and error messages. If it’s a malicious request, we just want to make sure we don’t do anything with it. If it is an invalid request from a script, the script programmer doesn’t want to see in the response, they will just want a succinct error message telling them how to fix the validation in their script.

I’d therefore like to propose properly decoupling the controller and view. The view (html+javascript) does validation, while the server side focuses on converting the incoming data to python objects. If it encounters an error while doing the conversion, it can bail with an appropriate error message (probably a 406 status code). Getting this status code means your webapp is broken –just like a 500 error would indicate– because your webapp should have validated the form input before submitting it. This would also be a suitable response in a REST based design. If a REST client sends invalid data, it should get an HTTP 406 with appropriate error message.

This simplifies the server side code a lot. It no longer has to worry about generating the html form, filling it with values, or returning error messages. The client side code becomes more complicated, but it’s the kind of more complicated that should be done anyway, to enhance the user experience. Ideally, it should be easier to write client-side validation than to default to server-side validation on form requests. It’s too easy to use django’s forms framework to create server-side validation without ever bothering with client-side validation.The coolest websites tell us our passwords don’t match *before* you submit the form. This should be the norm. It should be easier to write this kind of client-side validation than it is to default to server-side validation, so people default to using it.

Going forward, browsers need to support the new HTML 5 input types with proper validation and error messages. I also think there needs to be better support in the html spec (maybe html 5 has it and I don’t know about it?) for displaying validation errors. something like . For more complicated validation, we need a nice javascript framework. I haven’t used it yet, but I’m looking forward to experimenting with the jQuery Validation plugin.

With that in mind, I’m planning to continue using formencode for datatype conversion, but to ignore it’s buggy htmlfill stuff. Maybe eventually I’ll make a simpler stand-alone library that does this stuff and returns a 406 on invalid input.

Telus Customer Service

In this day of two way communication, most companies know that if they provide poor service, the whole world is going to know about it. Here’s a letter I recently sent to Telus, a pathetic excuse for a Canadian telecom:

TELUS Client Care
PO Box 7575
Vancouver, BC
V6B 8N9

Re: Terrible Service and Over Billing

Dear Telus:

I just received my first telus bill and was surprised that it included $12.90 in charges accrued from August 26 to September 2. I am writing to have you reverse these charges and explain why.

I did originally request that my service be connected on August 26. However, this did not occur. I was told a technician would be coming that day, so I waited at home all day, but nobody showed up. I called Telus Support and they told me that my Internet was hooked up, but my modem had not been shipped. They said that the modem would arrive within three business days.

On August 30th, the modem arrived. I hooked it up but there was no service. I called Telus Support again, and they spent a lot of time looking into it, ultimately telling me that my service would not be available until the 31st, because three business days had not yet passed. They explicitly told me that my billing period would not begin until August 31st.

On August 31st, I still did not have any service. I called Telus Support again, and they took about an hour to discover that my line had not been properly connected and a technician call would be required. They said he would arrive on September 1st. I spent so much time on hold that I was out of cell phone minutes on my plan four days into the billing period.

On September 1st, the technician, one Simon Mourton, called to explain what was wrong (something had not been connected properly outside my condo) and to help me get my service working. Simon was terrific; indeed, he was the only good service your company has provided.

Thus, I request that $12.90 in charges from August 26 to September 2 be erased from my bill.

In addition, I am billing you for the 8 hours (at my consulting rate of $90/hour plus GST) I was unable to work on Thursday August 26th, because I was waiting for a technician. I am also expensing you for the 93 minutes and 14 seconds (at $0.35 per minute) of cell phone usage that your inept service people kept me on hold, but you will be pleased to note I am not invoicing you for the time spent on hold. The total comes to $801.54. Please find my invoice attached.

Sincerely yours,

Dusty Phillips

Great Big Crane now supports pip

This week, I started using Great Big Crane in real life to manage some of my buildout projects. I was surprised to discover how useful, slick, and bug-free it is. When we wrote it in a 48 hour sprint, I did not realize how functional and complete our final product turned out to be.

I filed about a dozen issues on the project as I used it, but surprisingly few of them were bugs; just feature requests and minor enhancements to make it more usable. I don’t think any of us were expecting to maintain this project when the contest was over. However, now that I see how useful it is, and because winning the dash has garnered a lot of interest in the project, I sat down for a few hours and added the one thing people have been requesting for it: pip support.

This new feature is fairly simple, and not fully tested, but the general idea is to be able to run virtualenv, manage your requirements.txt, and install the dependencies from inside greatbigcrane. This required a fairly invasive refactor of certain commands that we had implemented as buildout specific, but overall, it wasn’t a terribly difficult task.

What I have so far is certainly usable, but I suspect in the long run, it’s just a start!

Have a look at the sources here: http://github.com/pnomolos/Django-Dash-2010/