Archive for August 2009

Using the dirstack in bash

I’m writing this post because its one of the most useful tools in bash and I always forget the syntax. Its hard to Google for and the piece of paper I write it down on always gets lost. And asking my advanced bash friends to remind me often indicates that they are unaware of this feature.

Most intermediate shell users are aware of the dirstack and how to manipulate it using pushd, popd, and dirs. If you aren’t, you really are missing out on something. Check out the basics here: http://www.faqs.org/docs/bashman/bashref_73.html

Once you’ve been using the dirstack for a while, you start to wish you could use those directories in commands. You might want to ls the directory you were in two jumps ago, or cat a file from a previous directory.

It’s easy to do, but I always forget the syntax. So here it is for my (and your) Alzheimered future reference: ~1 through ~9 refer to the last directory, the second last directory, etc on the stack.

For example:

dusty:dir3 $ dirs
~/test/dir3 ~/test/dir2 ~/test/dir1
dusty:dir3 $ ls ~2 # that refers to dir1
dusty:dir3 $ touch ~2/hello
dusty:dir3 $ ls ~2
hello
dusty:dir3 $ dirs
~/test/dir3 ~/test/dir2 ~/test/dir1
dusty:dir3 $ ls
dusty:dir3 $ ls ../dir1 # told you it was dir1
hello

So that’s it. the whole point of this post is to remind me that the syntax is ~1 and not !1 or -1 or $1 or %1 or !$1 or ~$1 or…

Why I'm quitting GMail

Lots of people have expressed fear at the power Google has over our information. I don’t intend to expound on that. Suffice it to say that they know more about me than I do.

It’s never bothered me. I have always trusted Google to take care of this information. They’ve always been on the same side of privacy, net neutrality, and copyright debates as me. They’re extremely active in the open source world and they seem to value open source rather than simply using us. I trust them.

I used to trust them.

A couple weeks ago, Aaron (head developer of Arch Linux) received an e-mail from Google Adsense telling us our account had been terminated:

While going through our records recently, we found that your AdSense
account has posed a significant risk to our AdWords advertisers. Since
keeping your account in our publisher network may financially damage our
advertisers in the future, we’ve decided to disable your account.

Please understand that we consider this a necessary step to protect the
interests of both our advertisers and our other AdSense publishers. We
realize the inconvenience this may cause you, and we thank you in advance
for your understanding and cooperation.

How are we a risk to advertisers? What are we doing wrong? How do we fix it? No explanation. Aaron, of course, asked for clarification. They regurgitated the response:

Thanks for providing us with additional information. However, after
thoroughly reviewing your account data and taking your feedback into
consideration, we’ve re-confirmed that your account poses a significant
risk to our advertisers. For this reason, we’re unable to reinstate your
account. Thank you for your understanding.

As a reminder, if you have any questions about your account or the actions
we’ve taken, please do not reply to this email. You can find more
information by visiting

https://www.google.com/adsense/support/bin/answer.py?answer=57153

Following instruction for obtaining more info, Aaron posted here: http://www.google.com/support/forum/p/AdSense/thread?tid=029aef7a42e7c4f2&hl=en&fid=029aef7a42e7c4f20004707fc5d9ce9e

Still no additional information. I find this offensive. We are an honest, upstanding, unfunded open source Linux Distribution. We have done nothing wrong.

I respect Google’s right to choose where they place ads. If I was a marketing agency for companies that sell adult products, I wouldn’t post advertisements in a daycare or during Saturday morning cartoons. However to cancel our account without giving us a chance to fix or even understand their rationale is troubling.

The money from Adsense is mediocre at best, compared to our donations and schwag income. But this event caused me to reevaluate my trust of this mammoth company. I’m not suddenly “OMG Google is evil, I must not use their products.” (As a teen I had this tendency toward Microsoft products. I’ve matured a bit and can acknowledge that Microsoft has some good products: their mice and keyboards.) However, I now feel less comfortable giving this company full control of my information.

And thus, I am slowly moving my e-mail account from GMail to my own domain. I’m considering anonymizing my Google cookie. I’m reluctant to put my own or client data on Google App engine. I’m losing enthusiasm for my Android powered phone (that has as much to do with Java as it does with Google, to be honest), and evaluating all new exciting Google services with just a hint of distrust.

Google still creates some of the best technology in the world, and they are still mostly friendly to the open source community. They are a large company and the actions of one department obviously don’t reflect the opinions of others. Adsense is Google’s cash cow. The more exciting Google projects occur in research and innovation. There isn’t much communication between the two.

I am looking forward to Google Wave (I intend to set up my own host, of course) and to an anonymized version of Chromium for Linux. I have no problem with Google Gears. I still use their maps, but I’ll have to stay off Latitude.

Update: This post unexpectedly hit Reddit, and within a few hours, Aaron got another e-mail telling us that our Adsense account had been reinstated. We’re still lacking an explanation, and I’m still not sure I trust them, but I have to give them credit for quick action!

I hadn’t made this post to get action out of Google, though, so we’re probably not putting Adsense back on archlinux.org and I’m probably not going to go back to GMail.

Authenticated Threaded Comments in Django 1.1

I’m looking to include threaded comments in my latest project, such that only authenticated users can use them. The Django ThreadedComments project seems to have lost momentum; as the old site says that the new version will use the comment hooks provided in Django 1.1 and will be backwards incompatible, but very little fresh development has been done.

Turns out, almost all the required functionality is available for free in the standard Django comments system, and you only need one slightly messy hack.

Authenticated Comments

I want a system that doesn’t ask for name/email/url; you can only post if you’re logged in, and the only visible box is one for comments. Behind the scenes, django automatically fills in these details from the currently logged in user if they are not supplied. It also kindly attaches that user to the comment (so it can be retrieved as comment.user). All you have to do is restrict commenting to logged in users and render the template directly to ignore the fields you don’t need. Like this:

some_template.html

{% if user.is_authenticated %}
    <h4>Submit New Comment:</h4>
    {% get_comment_form for project as comment_form %}
    {% include 'comment_form.html' %}
{% endif %}

comment_form.html

    {% load comments %}
    <form action="{% comment_form_target %}" method="POST">
        {{comment_form.content_type.errors}}{{comment_form.content_type}}
        {{comment_form.object_pk.errors}}{{comment_form.object_pk}}
        {{comment_form.timestamp.errors}}{{comment_form.timestamp}}
        {{comment_form.security_hash.errors}}{{comment_form.security_hash}}
        {{comment_form.comment.errors}}{{comment_form.comment}}
        <span>{{comment_form.honeypot}}</span>
        <br />
    </form>

Most of the variables in comment_form.html are hidden fields to help keep spammers at bay. If you prefer, you can do this same customization a bit higher in the abstraction layer by using a Custom Comment Form that doesn’t contain the url, e-mail and name fields at all. The django.contrib.comments.forms.CommentSecurityForm holds everything you need except a honeypot.

Threaded Comments

Django comments can be attached to any object… including existing comments. That idea and some thoughtful template design provides threaded comments:

some_template.html

{% get_comment_list for project as comments %}
{% for comment in comments %}
    {% include 'comment.html' %}
{% endfor %}

Here I’m getting all the comments attached to a specific “project”. For non-threaded comments, comment.html would basically just contain {{comment}} to render the comment. But I want threaded comments:

comment.html

{% load comments %}
<div class="comment">
    {{comment.user}}
    {{comment.comment}}
    {% get_comment_list for comment as comments %}
    {% with 'comment.html' as comment_include %}
        {% for comment in comments %}
            {% include comment_include %}
        {% endfor %}
        {% if user.is_authenticated %}
        <br /><a href="#">Reply to This Comment</a>
            {% get_comment_form for comment as comment_form %}
            {% include 'comment_form.html' %}
        {% endif %}
    {% endwith %}
</div>

Here, for each comment, I get the list of comments attached to that comment, and render them, along with a box to reply to the comment. This box is hidden by default in my stylesheet and displayed with a bit of jquery. I recursively include the same template for each comment so you can have as many levels of comments as you like (that may be a bad thing).

The recursive include is the messy hack — by default, Django includes templates at parse time. This means the template is included regardless of whether any comments exist, and the Python stack quickly fills up with too many recursive calls. But if the template being included is a variable instead of a string, it has to know what that variable value is and can thus only be executed at render time. So I simply wrap the include with a with template tag. This forces django to use the IncludeNode instead of ConstantIncludeNode (see django/templates/loader_tags.py), which renders the template at render instead of parse time.

Some uncreative styling in my css makes the comments indent a bit and hides the reply box until the jquery link is clicked.

.comment {
    margin-left: 1em;
    border-style: solid;
    border-width: 1px;
    border-color: #aaa;
    border-top-width: 0px;
    border-right-width: 0px;
    padding: 1ex;
}
.comment_form textarea {
    height: 3em;
    width: 40em;
}
.comment .comment_form {
    display: none;
}

Obviously, it could do with a makeover from a talented web stylist, but here’s how the general idea looks:

Nearly-unstyled threaded comments.

Nearly-unstyled threaded comments.

Why I don't like reverse and {% url %} in Django

Django has a feature that allows you to look up an url based on a function name and arguments. Here’s how it works:

######
# urls.py
urlpatterns = patterns('',
    (r'^projects/(?P\d+)/$', 'views.view_project'),
)
 
######
# views.py
def view_project(request, project_id):
    # create the view
 
######
# something.py
some_url = reverse("views.view_project", 4)
# some_url = /projects/4/
 
######
# sometemplate.html
<a href="{% url 'views.view_project' 4 %}">My project</a>

The idea behind urls.py is to separate urls, which are not supposed to change (because someone may have bookmarked them, there might be links to the page, etc) from implementation (views.py), which can change at any time (due to refactoring, new features, and bugfixes). You can move functions around at will, and simply change urls.py to point at the right function without breaking people’s links. I love this decoupling.

The idea behind reverse and url is that you should not be hard-coding urls into your code, you should use the names of the functions being called instead. I’ve tried to do this because its the “proper thing to do” if you’re a django coder… but I don’t like it, and I’m going to quit using it in my personal projects.

Here’s why:
1) It’s harder to maintain. As stated above, I often change the name or location of a view function. When I do that, I have to go through all the files that have a url or reverse call and change the view there. If I had hardcoded the url, I’d only have to change it in urls.py. I’ve been stung by this much more often than by changing urls.

2) Its harder to read. When you see “/path/to/something” you know you’re looking at an url. When you see reverse(‘some.module.path’, some, arg), it takes longer for the brain to parse, even if you know what the reverse call does.

3) In the case of {% url %} it exposes implementation details to the template author. Template authors should not know or care what python functions are being called internally. But they know what an url is, and what it represents.

In short, it adds an extra layer of abstraction to url handling. I like abstraction layers to be thin and useful; in the case of reverse(), the added complexity does not, in my minimalist opinion, justify the supposed gain in simplicity.

Py.test funcargs and Django

Holger Krekel just released the 1.0 version of py.test. Py.test is a functional and unit testing tool that cuts out a lot of the annoying boilerplate found in the unittest module included in the Python standard library.

I do a lot of Django coding. Django has a built-in test engine based on unittest. Its annoying, but it does a few things (such as capturing e-mails and creating a test database) automatically so I’ve tended to use it rather than setting up py.test to take care of these things. Today I decided I’d rather use py.test for my latest project. Turns out its not that complicated:

# there are better ways to do these first three lines
import os, sys
sys.path.append('.')
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
from django.conf import settings
from django.test.client import Client
from django.test.utils import setup_test_environment, teardown_test_environment
from django.core.management import call_command
 
def pytest_funcarg__django_client(request):
    old_name = settings.DATABASE_NAME
    def setup():
        setup_test_environment()
        settings.DEBUG = False
        from django.db import connection
        connection.creation.create_test_db(1, True)
        return Client()
    def teardown(client):
        teardown_test_environment()
        from django.db import connection
        connection.creation.destroy_test_db(old_name, 1)
    return request.cached_setup(setup, teardown, "session")
 
def pytest_funcarg__client(request):
    def setup():
        return request.getfuncargvalue('django_client')
    def teardown(client):
        call_command('flush', verbosity=0, interactive=False)
        mail.outbox = []
    return request.cached_setup(setup, teardown, "function")

Put that in your conftest.py and you can write tests like this:

def test_something(client):
    response = client.post('/some_url', {'someparam': 'somevalue'})
    assert "somestring" in response.content
    # Other assertions...

This uses the innovative py.test ‘funcarg’ mechanism to create a test database when testing starts, and refer to that database throughout the test run. The ‘django_client’ funcarg sets up a database when the session starts and deletes it when it finishes. The ‘client’ funcarg creates a similar client, but also resets the database after each test is run to ensure there are no interaction effects between tests.

I haven’t fully tested it yet, but its nice to know I can get the most useful django test functionality so cheaply in py.test.