Posts tagged ‘python’

Finally happy with folding in vim (or how to get an overview of a python file)

I’ve always found vim folding to be somewhat mysterious. No matter which of the foldmethods I used, it seemed like it was always too tedious or too manual.

Today, while editing a python file over a remote connection that has high enough latency to interfere with rapidly scrolling through the file, I finally set out to solve this problem. My goal was to get an overview of the file. Yes, there are plugins or ctags to do this, but I don’t have ctags installed and have tried different plugins that didn’t really work as I wanted.

Indent folding does essentially what I want; when all folds are closed, I can see the overview of classes and module-level methods in the python module. When they are open, I see the whole file. I always felt that Indent mode should be ideal with whitespace-driven python, but in the past, I found that the default settings left far too many folds to work with.

I already new about the foldminlines setting which only creates a fold if more than a specific number of lines is displayed. But today, I discovered foldnestmax.

This setting is the maximum number of folds in the file. In a python file, if I set it to 1, folding happens only at the classes or module-level functions in the module. If I set it to two, it happens at each method in each class. I don’t get a bunch of internal folds that I don’t want.

Another setting that I discovered today is foldlevelstart. If set to 99, opening a new file will always have all the folds expanded, which is what I want.

So now I have exactly what I want: a few top-level folds are created automatically based on indentation, but I don’t have to fuss with interior indentation in other folds. Now I needed a refresher on how best to interact with these folds.

  1. zM Close All Folds Switch to “overview” mode
  2. zR Open All Folds Switch to normal editing
  3. zo Open Fold Open the class or method under the cursor
  4. zc Close Fold Close the class method I’m currently editing

The last one is the most useful, because it works from anywhere in the method. If I’m done with that method, I just zc and it’s hidden from site. I don’t have to worry about collapsing multiple levels of indentation or creating manual folds. It just works.

There are other fold commands; see :help fold for an overview, but just using these settings and memorizing these four commands seems to be all I need.

I suspect that javascript would also work well using a foldnestmax of 1. HTML would probably not work as well since normally the entire body is the top level of intent. Perhaps a foldnestmax of 2 or 3 would be helpful here. Luckily, I can change these values on per filetype basis using autocmd.

My Linux Rig Interview

It’s pretty rare for me to link to other people’s articles in this blog. I don’t believe in regurgitating information. However, this link is not regurgitated, it’s an interview I gave to so it’s my own words:

When you use a Django query keyword as a field name

I need to model a location in the Alberta Township System coordinate space. The model is extremely simple:

class Location(models.Model):
    project = models.ForeignKey(Project)
    lsd = models.PositiveIntegerField(null=True, blank=True)
    section = models.PositiveIntegerField(null=True, blank=True)
    township = models.PositiveIntegerField(null=True, blank=True)
    range = models.PositiveIntegerField(null=True, blank=True)
    meridian = models.PositiveIntegerField(null=True, blank=True)

There’s a rather subtle problem with this model, that came up months after I originally defined it. When querying the foreign key model by a join on location, having a field named range causes Django to choke:

>>> Project.objects.filter(location__range=5)
Traceback (most recent call last):
  File "", line 1, in
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/", line 141, in filter
    return self.get_query_set().filter(*args, **kwargs)
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/", line 556, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/", line 574, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/sql/", line 1152, in add_q
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/sql/", line 1092, in add_filter
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/sql/", line 67, in add
    value = obj.prepare(lookup_type, value)
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/sql/", line 316, in prepare
    return self.field.get_prep_lookup(lookup_type, value)
  File "/home/dusty/code/egetime/venv/lib/python2.7/site-packages/django/db/models/fields/", line 136, in get_prep_lookup
    return [self._pk_trace(v, 'get_prep_lookup', lookup_type) for v in value]
TypeError: 'int' object is not iterable

That’s a pretty exotic looking error in Django’s internals, but it didn’t take long to figure out that using location__range is making Django think I want to use the range field lookup on instead of the field I defined in the model. I expect a similar problem would arise if I had a field named “in”, “gt”, or “exact”, for example.

The solution is simple enough, but didn’t occur to me until searching Google and the Django documentation, and ultimately scouring the Django source code failed to yield any clues. If you ever encounter this problem, simply explicitly specify an exact lookup:

>>> Project.objects.filter(location__range__exact=5)
[< Project: abc>, > Project: def >]

Prickle v0.4: CouchDB to MongoDB

I took some free time to port Prickle, my stay out of your way time tracking tool, from CouchDB to MongoDB. I originally wrote it to use Pylons and Couchdb specifically because I felt like studying a couple of technologies I hadn’t tried before. I found CouchDB to be a bit unwieldy to code with, and it didn’t take long for it to get slow on my Prickle datasets. Rather than figure out how to optimize CouchDB, I decided to port it to MongoDB.

The port was relatively straightforward. I was able to use pretty much the same model layout (Prickle has a very simple data layer). I tightened it up a bit to use actual references between models instead of implicit foreign keys by ids.

I chose to use MongoEngine for the database layer. I started with pymongo, but decided that a more abstract object-document-manager would be more useful.

I found querying in MongoEngine to be much simpler than the map-reduce queries in CouchDB. It is very similar to the Django ORM with which I am very familiar. Document creation and mapping is also familiar. Overall, I think Prickle will be a lot easier to maintain and extend using MongoDB than it was with CouchDB.

Prickle v0.4 does not contain any new features over 0.3. It also contains one bugfix to the date validation, submitted by Shelby Munsch.

Django and jquery.tmpl

Lately, I’ve been finding Django increasingly inappropriate for the web applications I develop. I have several complaints: the forms library doesn’t extend well to ajax requests, any extensive customization to the admin requires obtuse inspection of the admin source code, many admin customizations simply aren’t possible, the “reusable apps” philosophy has added a layer of complexity to a lot of things that really should not be there, there are no obvious best practices for ajax support.

In spite of all this Django is still better than other frameworks (Python or not) that I have investigated or tested. I’ve considered writing my own web framework, but I wouldn’t maintain interest in it long enough to get it off the ground. So I’m letting my complaints bounce around in the back of my mind with the hopes that I can improve Django so that it continues to work for me as a platform.

I’m currently trying to come up with a better system for ajax requests. I have nothing concrete in mind, but I’ve started with the premise that ajax requests should never return rendered html, but should return only json (hence my issue with the django forms library). With that in mind, I need a templating library for json data. JQuery is a must, and the officially supported JQuery templating library is jquery.tmpl

The problem with jquery.tmpl is that it uses django-like syntax. The following is a valid block that may be rendered in a jquery.tmpl page:

<script id="project_tmpl" type="text/x-jquery-tmpl">
    {{each projects}}<li>${$value}</li>{{/each}}

If you try to include this in a django template, the {{ and }} tags will be replaced with (probably) empty variables. Django has {% templatetag %} to render these individual items, but what we really need is a way to tell the django templating system to leave complete sections of code alone. So I wrote the jqtmpl template tag. It allows us to wrap code in a block that tells Django not to render that block as template code. So the above would show up in a Django template as follows:

<script id="project_tmpl" type="text/x-jquery-tmpl">
{% jqtmpl %}
    {{each projects}}<li>${$value}</li>{{/each}}
{% endjqtmpl %}

Here’s the template tag:

from django.template import Library, TextNode, TOKEN_BLOCK, TOKEN_VAR
register = Library()
def jqtmpl(parser, token):
    nodes = []
    t = parser.next_token()
    while not (t.token_type == TOKEN_BLOCK and t.contents == "endjqtmpl"):
        if t.token_type == TOKEN_BLOCK:
            nodes.extend(["{%", t.contents, "%}"])
        elif t.token_type == TOKEN_VAR:
            nodes.extend(["{{", t.contents, "}}"])
        t = parser.next_token()
    return TextNode(''.join(nodes))

This doesn’t handle Django’s {# comments #}, as token.contents doesn’t return a valid value for comment items. As far as I know, you wouldn’t use the comment construct inside a jquery.tmpl template anyway, so it’s still functional.

Next on my list is a better forms validation library to suit my theory that validation should be done client side. I’ve got a server-side system in mind that returns json responses, and does not return user-facing error messages, since those should have been caught client side. With these tools, and hopes for Grappelli to eventually create a decent admin, Django may continue to serve me.

Another release of Prickle, the stay out of your way time tracker

I just released version 0.3 of Prickle, a time tracking tool I’ve mentioned before. Prickle is a simple stay-out-of-your-way time tracking program that I find very useful for my day-to-day work as a consultant. In originally developing it, I modeled it after Freckle. With version 0.3, I consider Prickle to be more usable and more powerful than Freckle for my specific purposes. It may be useful for you as well.

The major improvement in this release is the ability to have multiple types of work for a given project. Each type is represented in the invoice by a different line-item, and can optionally have a different rate. So if you bill admin work at a different rate than development, prickle can handle it on one invoice. This required adding one more field to the log it form, which means one extra keypress for logging line items, but the new field is perfectly comfortable staying blank. It also autocompletes the types for a given project, so it is still stay-out-of-your way convenient.

The other major change is the ability to edit timesheet rows. I find this useful when I occasionally bill a timesheet for the wrong date or project. This is done using inline editing and is very ajaxy. At a user request, I’ve also changed the timesheet view to use hours and minutes rather than decimal representation.

Prickle is now working pretty much as I want it. I may add some reportlab pdf generation at some point so I can automatically e-mail pdf invoices, and a little extra customizability may be in order. The entire interface really wants some design love, but I am really hoping another user will jump up and do that. I’m not what you’d call a design guru. But overall, I think the current feature set is what I need, and I believe it may be useful for a few other people in the consulting field as well.

Updating m2m after saving a model in Django admin

I wanted to ensure that a many-to-many field on my model contained a value that was on a foreign key field on the model whenever that model was saved in the admin, like so:


I thought this was a trivial task. I added this code to a post_save signal connected the model, but the participants list was not being updated. I was pretty sure that the reason it didn’t work was that form.save_m2m() must be called somewhere in the admin after the object was saved, which would override my m2m changes with the empty ones from the model.. Reading the Django admin source code confirmed this. But it didn’t show an obvious way to circumvent the behavior.

It is possible to override the change_view and add_view functions on the ModelAdmin object, but I did not want to do that. I would have had to copy the entire change_view contents into the subclass, as there is no way to make a super call do what I wanted here. Here’s the section of code I needed to modify (it’s in django.contrib.admin.options.ModelAdmin.change_view if you want to look):

            if all_valid(formsets) and form_validated:
                self.save_model(request, new_object, form, change=True)
                # Right here is where i want to insert my call
                for formset in formsets:
                    self.save_formset(request, form, formset, change=True)
                change_message = self.construct_change_message(request, form, formsets)
                self.log_change(request, new_object, change_message)
                return self.response_change(request, new_object)

Obviously, I can’t override save_model because save_m2m() is called after that, which would still wipe out my changes. I really need to have a self.after_m2m() call at the point I have commented in the above code. But I don’t.

I really didn’t want to have to copy this entire method into my ModelAdmin subclass (in just to add that one call… so instead, I overroad another method that happens to have access to the new object and is called after save_m2m(). See that call to self.log_change a few lines later? That method updates the admin log db table. But it also happens to have access to the newly created object. I want to emphasize that this is an ugly hack:

# in
class ClassListAdmin(admin.ModelAdmin):
    def log_change(self, request, object, message):
        super(ClassListAdmin, self).log_change(request, object, message)
        # The add and change views happen to call log_addition and
        # log_change after the object has been saved. I update the
        # m2m at this point (in update_participants) because the add
        # and change views call form.save_m2m() which wipes out the
        # changes if I put it in self.save_model().

self.update_participants(), of course, contains the code I originally wanted to run.

This isn’t the most proper way to do this, but if you’re looking for a quick, dirty but DRY hack, it might save you some time.

Prickle version 0.2

Yesterday, I sent out my first invoices using Prickle. Having tested it in production (I had a to fix a couple sort order issues), I decided it was time, already, to release the second version (release early and often, catb says).

In case you missed the introduction: Prickle is a simple time tracking tool I whipped up to avoid fighting with spreadsheets or paying for Freckle. Source is available from github. Dependencies include pylons and couchdb.

So what’s new in v0.2? Mostly minor bug fixes or data display. The invoice summary is a little more useful, most of the display tables have meaningful sort orders (descending by date, usually), and timesheets can be deleted. One bigger feature is a simple javascript timer with start/stop/pause buttons. When you stop the timer, it automatically adds the time to the duration field to allow you to log it. Another is the ability to mark timesheets as “invoiced” without actually creating an invoice. I use this to clean out timesheets that I’m not actually billing for, such as time spent working on prickle, or arch schwag, or editing for my Dad’s new book.

I’m finding Prickle exceptionally useful; If you do a lot of time tracking for freelance work or other tasks, I hope it might be useful to you as well. Please try it out, and if you have any questions, or problems, get in touch!

Here’s a screenshot of Prickle in action:

Introducing Prickle, a time tracking tool

I’ve been doing a lot of Django coding lately, and I’ve been doing it very well. But I’ve felt bit of stagnation settling in. I’ve only been coding what I know. I needed to explore, experiment.

So I picked a couple of technologies I’ve heard about but haven’t tried, and started coding. The goal wasn’t to make something useful, it was to learn some new APIs and frameworks. I accidentally made something useful.

So here I present Prickle. Prickle is a time tracking tool for freelancers written using Pylons, CouchDB, and Jinja2. (Those are the technologies I chose to play with.) I decided to write an app to replace Freckle, a stay-out-of-your-way time tracking tool that has been highly recommended to me by favorite alternate freelancer, Dan McGee. Freckle is a great service, but I have a couple problems with it: I hate hosting my data on other people’s machines, and I hate paying money to use a service. Granted, at $12 a month, it’s going to take me years to earn back the hours I put into developing Prickle, but that’s not the point. Replacing Freckle in my daily use was just a little bonus I got while studying Pylons and CouchDB.

All Prickle does is allow me to enter the number of hours I’ve worked on a project, along with a description. It makes entering hours as simple as possible; the date is selected semi-automatically. Hours can be entered in a variety of formats (0:30, 30, 0.5, and .5 all map to 30 minutes 1, 1:00 1.0 all map to 1 hour), so it tends to “just work”. Project names are autocompleted, so it’s just a few keypresses to get to the description box. Now I type a description, press and I’m done.

There are a bunch of views of the timesheets that allow me to see how much uninvoiced time has been spent on a particular project or during a particular month or day. If I set a rate for a project, I also get a summary of how much money I’ve earned on a given day or month, or how much I’m going to have to bill a client for a specific project.

And Prickle does invoicing. It’s very simple invoicing, but very simple is all that I need. It summarizes the hours since the last invoice was created, displays them in an invoice template (I currently use print to pdf to generate the invoice. Maybe later I’ll automate sending an e-mail to clients). Even here, Prickle has some handy helpers to speed up the process. It keeps track of billing addresses for a given project, so I don’t have to enter it each time, and it guesses what the next invoice number should be so I often don’t have to type it in.

Prickle doesn’t have all the fancy graphs and reports that Freckle has, but it quickly answers the primary questions that I ask of my data: How many hours have I billed today? How much money have I earned this month? Am I on target for this project’s budget?

Prickle is open source, and I’m hoping some people will find it useful enough to contribute back to it. I use it daily already, and don’t really have any complaints. Some things I intend to add include:

* A timer with a pause button. I hated this when I first tried Freckle, but it grew on me.
* improved historic view of invoices
* editing or deleting time entries

Some things I invite others to add include:
* authentication. I like to run things locally, so auth wasn’t important to me. It’s also confusing to implement in Pylons.
* styling. I applied some semi-random CSS rules, but I know it’s ugly.
* browser support. This thing works great in chromium, but I’m using some of the most experimental stuff: html 5 inputs, CSS 3, etc. It’d be nice to add some Javascript to make it work on other browsers.

So, Dusty, now that you’ve played with Pylons and CouchDB, what’s your take?

CouchDB is pretty cool to work with. Map/Reduce based queries take some getting used to, but once you’ve learned them, that’s all you have to know. There’s no tuning SQL, the ORM is a very thin layer, it just works. If CouchDB is as scalable as they say it is, I think I’d like to use it some more.

Pylons is kinda nice. Formencode sucks, but other than that, the libraries bundled with or suggested for pylons are pretty intuitive. I’m finding Django is a bit over-engineered these days, or maybe I’ve just been pushing its limits. I was hoping Pylons would be a less bossy solution, but I don’t think I’ll be switching to it or suggesting it to my clients anytime soon. It seems a bit rough around the center, and doesn’t seem any less complicated than Django, in practice.

Next on my list of tools to play with is node.js. I also want to play with web2py, and I may try a zope 3 app just for fun someday, too.

Great Big Crane now supports pip

This week, I started using Great Big Crane in real life to manage some of my buildout projects. I was surprised to discover how useful, slick, and bug-free it is. When we wrote it in a 48 hour sprint, I did not realize how functional and complete our final product turned out to be.

I filed about a dozen issues on the project as I used it, but surprisingly few of them were bugs; just feature requests and minor enhancements to make it more usable. I don’t think any of us were expecting to maintain this project when the contest was over. However, now that I see how useful it is, and because winning the dash has garnered a lot of interest in the project, I sat down for a few hours and added the one thing people have been requesting for it: pip support.

This new feature is fairly simple, and not fully tested, but the general idea is to be able to run virtualenv, manage your requirements.txt, and install the dependencies from inside greatbigcrane. This required a fairly invasive refactor of certain commands that we had implemented as buildout specific, but overall, it wasn’t a terribly difficult task.

What I have so far is certainly usable, but I suspect in the long run, it’s just a start!

Have a look at the sources here: