Posts tagged ‘python 3’

Opterator, revisited

A few years ago, I wrote a simple decorator that introspects a main() method signature and docstring to generate an option parser. I never really used it, mostly because back then, I wasn’t too keen on third party dependencies. Nowadays with distribute being maintained, well-documented, and working (plus I know how to use it properly), I no longer have this concern.

A friend recently linked me to the brilliantly designed docopt and reminded me of my interest in opterator. I revisited my code and decided it’s a pretty neat little design. So I ported it to Python 3. This took a matter of minutes, largely because I originally wrote it shortly after taking the pytest tutorial at Pycon 2009 and it was easy to find the failing code. It now supports Python 2.6, 2.7, and 3.3, according to tox.

I originally designed opterator to be a full replacement for optparse and friends. However, my main purpose for it now is to create quick and dirty command line applications, often for my personal use. These apps usually have a couple of options and it doesn’t seem worth the trouble of setting up optparse. Yet, mucking around with sys.argv is also annoying. Opterator minimizes the boilerplate. Check out this basic example:

from opterator import opterate                      


@opterate                                           
def main(filename, color='red', verbose=False):     
    print(filename, color, verbose)

main()                                              

with three lines of boilerplate code, a function can be turned into a command line program that can be called like so:

  $ python examples/basic.py this_file
  this_file red False

or so:

  python examples/basic.py this_file --color=blue
  this_file blue False

or even so:

  $ python examples/basic.py --color=purple another_file --verbose
  another_file purple True

And you even get a not totally (but somewhat) useless helpfile:

  $ python examples/basic.py -h
  Usage: basic.py [options] filename



Options:
  -h, --help            show this help message and exit
  -c COLOR, --color=COLOR
  -v, --verbose

Hacking on PyPy

In another great Pycon2012 keynote, David Beazely asked the question, “is PyPy easily hackable?” After a great talk, he answered with a decisive, “I still don’t know.” Having sprinted on Python I’d like to answer his question in a bit more detail.

I love David’s presentation style. He has a novel method of using phrases like, “blow your mind” and “this is really scary” repeatedly until they lose their meaning and you no longer feel mindblown or scared. A variety of factors, including Beazely’s thorough keynote address motivated me to join the PyPy team during the Pycon developer sprints.

I’d like to clear up one oversight in Dave’s otherwise impeachable talk. One of the PyPy devs, Holger Kregel explained to me that PyPy does not have over 1 million lines of code. I don’t have exact numbers, but for “historical reasons”, a non-python file containing Base64 encoded data was given a .py extension. When excluding this file from the line count, around half a million lines of actual Python code exist, and about a quarter of these are tests.

I was surprised how trivial it was to get started hacking on PyPy. I don’t really grok the many layers of the translation toolset and PyPy interpreter, but it’s pretty clear that the layers are well separated. I was hacking on the py3k branch of PyPy. I am happy to admit I was working primarily on changing print statements to print() functions and commas in exceptions to the as keyword.

Here are the steps to start hacking on PyPy. Notice that the hour-long translation step is not part of the procedure. PyPy has a solid test framework, and the PyPy crew are focused on a 100% test-driven-development paradigm.

  1. Clone pypy (this takes a while):
    hg clone https://bitbucket.org/pypy/pypy/
  2. Pick a branch to work on. There are about 80 branches. I don’t know what they all do. Popular ones during the sprints included py3k and numpy-ufuncs2
  3. Pick a feature to work on. For py3k support, the list of failing tests in the buildbot is a good place to start. Numpy programmers had a list of fuctions that needed implementing, but I can’t find the link. Ask on IRC, the PyPy crew are very helpful. The bug tracker contains many features and issues that need addressing
  4. Add /path/to/pypy/ to your path so you can run the pytest.py command
  5. cd into the directory indicated in the buildbot output and run pytest.py path/to/test.py -k testname
  6. The test will likely fail. Hack away and fix it.
  7. When the test passes, commit, push to a bitbucket repo, and issue a pull request.
  8. Repeat!
  9. There are quite a few cons to working on this project. If you run hg in the pypy/modules/ directory, it will try to pick standard library modules from pypy and choke horribly. The pypy developers don’t really believe in documenting their code. Being able to tell the difference between rpython and python (which have identical syntax) is important. In general, if a module starts with “interp_” it contains rpython, but if it starts with “app_” it contains python. The code does not appear to be well-documented.

    If you are hacking on Python 3 support, you need to bear in mind that the PyPy interpreter is written in Python 2. You are working on a Python 2 application that executes Python 3 bytecode!

    On the positive side, rPython and Python are much easier to read and write than C. The PyPy devs are brilliant, but not intimidating. They are so confident in their test suite that they are comfortable programming in a “cowboy coding” style, hacking randomly until all the tests pass. Any one layer in the toolchain is easy to understand and develop. The IRC channel is full of friendly, knowledgable, helpful people at any time of day.

    Overall, I am much less intimidated by this project than I was before I started the dev sprints. I still can’t answer the, “Is Python easily hackable?” question fully. It’s certainly easy to get started, but I don’t know how easy it is to become intimate with the project. Dave Beazely’s keynote made PyPy more approachable, and I approached it. Hopefully this article will encourage you to do the same.

Python 3 Object Oriented Programming

For the past eight months, I’ve been working hard on a project that’s a little out of the ordinary, for me. It’s the reason there’s been such a drastic reduction in number of blog posts here. It’s the reason I haven’t been earning enough money to cover my expenses each month. It’s my biggest accomplishment to date.

I’ve written a book (an entire book!) on object oriented programming, with a focus on syntax and libraries supported in the exciting new Python version 3. It’s designed for beginner to intermediate Python developers who are more familiar with Python as a scripting language than as an object oriented programming language.

As a byproduct, it also introduces Python 3 syntax, and will be a great reference for programmers wanting to upgrade their Python 2 skills. For the most part, Python 3 is a simpler, more elegant language. The learning curve is shallow, but it takes some getting used to.

It also summarizes the state of the most exciting libraries available for Python 3 at this time. If you’ve been wondering when it’s time to start migrating to the new language, it is now!

I’m currently in the rewrite phase on the book (it’s time consuming!) but it’s already available for preorder directly from my publisher:

https://www.packtpub.com/python-3-object-oriented-programming/book

I’m not great at marketing, so to put it bluntly: I hope you all buy a copy! I’ve put a great deal of effort into this project, and I’m very proud of the result. This book is a great resource and fills a void in the available references. It also fills a void in my available writings, as my blog posts tapered off over the past few months!

Web Applications in Python 3

Everyone knows that there aren’t any decent web stacks in Python 3, right? Nobody seems interested in porting Django or Turbogears or Pylons or Plone or Zope to the new, improved much more fun to write language.

Almost true. There are a few less popular frameworks for Python 3, including my own half-finished psyclone: http://archlinux.me/dusty/2010/01/17/python-3-web-framework/

But there aren’t any full stack frameworks. But it’s not too hard to hack one together using available tools.

In my mind, a full stack web framework requires several features:

  • The core server
  • A database engine
  • A templating languages

In addition, it’d be nice if it supported sessions out of the box and had a built-in auth framework. Basically, the more like Django it is, the better.

The good news is, we can hack one together with popular Python 3 ready tools including:

CherryPy provides a working web application engine. SQLAlchemy, which supports sqlite and posgres under Python 3 provides us an extremely flexible and robust ORM. Jinja is an advanced Django-like templating system. I implemented a very basic blog (sans authentication) in these tools in a matter of an hour or so, and it wasn’t any worse to work with than Turbogears. This isn’t a big surprise, since Turbogears is built on CherryPy and SQLAlchemy as well. We don’t have access to the extensive library support Turbogears has to offer, but it is now accurate to say that Python 3 supports a (nearly) full stack web framework.

You’ll need the svn version of CherryPy, and the newly released 0.6 version of SQLAlchemy. Jinja 2 cooperates well.

I don’t find it as pleasant as working with Django (I’m actually not a huge fan of Alchemy’s syntax), but it’s certainly a working solution. It wouldn’t overly much library support to turn it into a really nice system.

Image Manipulation in Python 3

Enough libraries have been ported to Python 3 to finally make it seriously possible to write real world code in this modern Python interpreter. Sure, we don’t have django or really any decent web framework yet (CherryPy runs, but it’s not a full web stack), and database support is limited (SQLAlchemy supports postgres and sqlite3), but for the most part, if you need to do something Python 3, you can.

The major exception I’ve discovered is Image manipulation. The Python Imaging Library has not been ported to Python 3, and there is no indication when it will be. The latest version of PIL, 1.1.7 was released in late 2009, with an indication that it would be “made available for Python 3,” but no estimate as to timeline. There are no mailing list posts answering an increasingly popular question “when will PIL be available for python 3?” I found no source repositories that indicate that it has been started. There is a patch floating around that can supposedly be applied to PIL 1.1.6 to make it Python 3 compatible, but it didn’t work for me.

I tried doing a port of PIL 1.1.7 myself, but was unable to find documentation for the modifications to the C extension API in Python 3. My C is pretty rusty, and my schedule is way too full for the next two months, so I gave up on the task. Because of the lack of support from the official PIL developers (I hold nothing against them; we’re all busy in the open source world, and can only contribute what we have time and interest to contribute), I’m hoping this post will motivate someone to attempt a port of PIL, making Python 3 that much more attractive.

If you came to this post looking for some kind of Python 3 image manipulation library and were disappointed, don’t be! The Pygame image module allows us to load a few image formats, and we can manipulate the resulting surfaces in a variety of ways. It’s lower level and not as comprehensive as the Python Imaging Library, but it is one useful alternative if you need to do image manipulation in Python 3.

A Python 3 Powered Blog

Last week, I posted my intent to port the Tornado web framework to Python 3. Not only have I done that (sort of), but I’ve hacked it to pieces; soon it will be unrecognizable to the original developers, and possibly, to me!

It didn’t take long to get the Hello World example included with Tornado running. Quite a bit more work was the blog demo, but I now have the template, auth, and httpclient modules working, along with the core modules for a working async server. I was able to log into my example blog with my google account, compose some entries, view, and edit them, including the feed.

That doesn’t sound like much if you’re coding blog software for an existing framework (10 minutes in django, probably about 12 in web.py). But if you’re coding a web framework for existing blog software, it’s an accomplishment. I’m proud to have a “working” framework (even though it’s full of bugs and working just means “those things I’ve fixed”) for Python 3 in such a short amount of time.

I’ve named the project psyclone, a play on ‘tornado’ –> ‘cyclone,’ and the fact that Python projects oughta have a ‘p’ and ‘y’ in them somewhere. The code is on github for all to play with. Patches welcome! :-)

I’m having a lot of fun with this project, so it’s taking more of my time than I ought to be devoting to it… on the positive side, it’s progressing rapidly!

My plans:

  • Go over the existing code and improve some of the rather messy unicode/str –> str/unicode hacks I had to make to get it working.
  • Write some tests. (The Tornado team seems not to value tests.) I’ll use py.test and may need to write a test client.
  • Write a session framework and auth framework; the current auth framework uses openID only; but I like a more local solution to be available as well.
  • Consider writing an ORM. Likely, I’ll discard this idea, arguing that Judd was right to design a frameworkwith SQL only. The truth behind the argument will be laziness, of course.

Python 3 Web Framework

I got it in my head this weekend that it was about time someone wrote a web framework for Python 3. My head is kind of stubborn about these things, so I asked it some questions:

Does the world need another web framework?
Do I need another web framework?
Do I have time to do this?

The answers were all “no.” Still, I’m planning to go ahead with it until I get bored. Then the project can sit and collect dust with all my others.

A bit of discussion with The Cactus, led to a few ideas:

I discovered that QP is apparently the “first Python-3 enabled web framework.” I didn’t try it, so I was perhaps unfair in discarding it, but it doesn’t look… suitable.

I looked around some more, and found that CherryPy is about to release a Python 3 enabled version. I’m sure that will spawn a whole slough of Python 3 frameworks built around CherryPy. I considered such a plan (I’d call it ChokeCherryPy based on a receipe my mom devised): create some kind of templating system based on str.format, some session support, and some kind of database module wrapped around py-postgresql. Could be fun. But I’d end up with a mess of third-party technologies much like TurboGears, and that would be embarrassing, plus I’m sure the TG team already has people working on this.

Then I came back to my original plan, which was to port either Tornado or web.py to Python 3. Tornado looks like a smaller codebase (easier to port) and I’ve never used it before, so it’s also a chance to learn something new. So today I forked Tornado on github and run 2to3 on it. I’ve already got the “hello world” demo running; it wasn’t too hard once I figured out the difference between bytes and strings. At least, I think I did that part correctly.

The project is named psyclone, a little play on the ‘destructive weather patterns’ genre. I was close to p3clone, but it’s too hard to convince people it should be pronounced ‘cyclone’.

This isn’t a project I expect to go anywhere; django will be ported to Python 3 soon enough, and other frameworks will be popping up all over. But I’ve been working with Python 3 a lot lately, and I thought it was time to tackle the ‘scary’ job of porting an existing app. It’s tedious, but not difficult.

SimpleHTTPServer in Python 3

If you’ve been doing any testing of client code that uses urllib or httplib, you probably know about this command:

python -m SimpleHTTPServer

This starts a very simple server in the current working directory; it serves all files from that directory, and is, quite simply, the quickest way to get something set up if you want to test some kind of web parsing or client code. (It’s also handy if you want to fire up a server to easily share files from your hard drive for a few minutes).

SimpleHTTPServer has been merged with BaseHttpServer into the http.server package in Python 3. I couldn’t easily find documentation for the new command, and ended up writing the following simple code:

from http.server import HTTPServer, SimpleHTTPRequestHandler
 
httpd = HTTPServer(('127.0.0.1', 8000), SimpleHTTPRequestHandler)
httpd.serve_forever()

Then I did a bit more digging around and realized that this command does what the old one did.

python3 -m http.server

My code performs a little differently (it only serves on the localhost interface), but if anyone is looking for the old SimpleHTTPServer command line, there you have it.

The http.server module is normally supposed to be as a base for creating more complicated server environments (see your favourite web framework, for example), but the fact that it can be executed directly has a great deal of utility as well.

By the way, if you didn’t know about SimpleHTTPServer, you might also be interested in the built-in smtpd server as well. I use this command frequently:

python -m smtpd -n -c DebuggingServer localhost:2525

This runs a simple smtp server on the given interface and port, and outputs all mail sent to that port to the console. It is very useful for testing and debugging web-based send-mail forms and such. You can, of course, run a standard smtpd server by not passing the -c DebuggingServer.