Pushing Python Past the Present

This is the first time I’ve inserted myself into an exchange between bloggers I don’t know. This topic interests me and I have something to add. Most importantly, I found an alliterative title. So I thought I’d give it a go.

I first saw Calvin Spealman’s article I am worried about the future of Python. I suspect that upon reading this article, Guido got into his famous time machine to give his Pycon 2012 Keynote speech, accusing Calvin of trolling.

This article was shortly followed by Tim McNamara’s Python is doing just fine which (likely unintentionally) summarizes Guido’s talk.

Finally, Nick Coghlan came out with an extensive summary article called Python’s Future: A Global Perspective.

And now there’s me. All I want to do is push people to start supporting PyPy. The PyPy developers have worked on a variety of technologies that can be made into production-ready products that address most issues (real or imaginary) that people see with Python.

They’ve already solved speed and I suspect the next release of PyPy will solve memory. They’ve made huge but mostly incomplete progress in a lot of other areas, including the ones the above bloggers have mentioned.

Concurrency

I have had a lot of trouble dealing with parallelism in Python. Everything I have tried has either been a hack or required a hack to work with it. In my experience, the multiprocessing module does not work in large-scale production. However, it works a lot better on PyPy than it does on cPython, at least for me. The various async solutions can only use one processor and are therefore, in my opinion, no more exciting than GIL-throttled Threads.

In addition to better multiprocessing, PyPy also supports stackless extensions out of the box. And I haven’t even mentioned Armin Rigo’s mysterious Software Transactional Memory implementation.

Concurrency is a sore point in Python. There are solutions. PyPy is capable of doing those solutions better.

Web

I think it’s unfortunate that the only way to code for a web browser is to use JavaScript or a language that compiles to JavaScript. I feel there is no real reason browsers can’t support arbitrary <script type=""> tags. I discussed this last March.

With development, PyPy can support Python in the browser. The mostly-finished but untested sandboxing feature of PyPy can be adapted for in-browser execution. I know some experimenting was done on this in 2009 that proved it’s possible. We just need to take it up and make it happen.

Mobile

PyPy’s speed makes it theoretically possible to run it effectively on mobile platforms. Further, it can be compiled to arbitrary execution environments including JVM and .Net. There is no reason that with some development, PyPy couldn’t run on the Dalvik JVM. I’m sure it’s even possible to run it on iOS.

Interestingly, one of the trolls Guido mentioned in his talk was “PyPy should become the default Python implementation”. He debunked this readily by asking the audience how many people use PyPy in production. Nobody moved.

PyPy is ready for production use, but it is not widely used. I think this is largely because the primary PyPy developers are way more excited about creating new features than they are about marketing the current platform or polishing up nearly finished features like sandboxing or a Dalvik backend. They are visionaries, not finishers.

I say this with a great deal of respect. I am not calling for these guys to change their focus. What I want is for other people from the Python community to join the PyPy project as finishers. It needs people that can make sure these nearly-finished features are working, production-ready and more importantly: documented.

I’ve been meaning, for months, to become one of these people. Unfortunately, I’ve prioritized other things including my job and my upcoming book. It’s still on my radar, though, and I hope that after reading this article, you, too, are thinking about make PyPy the future of Python.

So, what can you do?

Use PyPy
PyPy is capable of being your default Python 2 interpreter for most common tasks. Use it. If it doesn’t work for a given project, get on #pypy, they will help you fix it. It’s even more exciting if you can use PyPy features that are not currently available in cPython, such as the stackless extensions.
Evangelize PyPy
Tell people how well PyPy is working for you. Write articles like this one.
Document PyPy
The PyPy website contains a lot of documentation, but it’s rather intimidating and unreadable to the uninitiated. PyPy is just Python, but it’s also much more than Python. Somebody with some writing skills needs to get in there and make it digestable.
Design PyPy
Seriously, pypy.org does not need to look like something out of 2005.
Develop PyPy
I’ve hacked on PyPy. It’s not scary. Work on features like numpy or Python 3 support that are the current developer’s focus. Better yet, work on finalizing features like sandboxing or alternative backends that are finished but not quite tested.

If you have an hour or two free this weekend, instead of writing about how Python is great or not great, or not going to continue to be great, do something with PyPy. Make it happen.

15 Comments

  1. cactus says:

    Until gevent/eventlet works on pypy (unlikely in the near future), pypy is a sadly a no-go for much of what I use python for on a daily basis.

  2. drcouzelis says:

    OK, you talked me into it! I installed PyPy, but I’m afraid I’m having a hard time knowing where to go from here…

    Does PyPy support standard CPython libraries, such as Pygame? (I assume it does) Where can I learn how to install it?

    Is there a PyPy tutorial you recommend?

    Also, I totally ordered your Object Oriented Programming in Python book. :D

    • Justin Alan Ryan says:

      Last I checked, there’s some work to support traditional CPython apis, except for those related to refcounting, but the true golden path here is for C extensions to be developed using ctypes, which allows runtime dynamic linking. Pretty sure anything that uses ctypes should work on PyPy.

  3. The problem with Pypy is that it does not support most of the commonly used libraries.
    I use Python for scientific computing. I need numpy, scipy and matplotlib (also scikit-learn).
    These do not run on PyPy.
    If you answer “So make it work.”, please have a look at how many people already work on this and how little progress they are making. Also: is it really worth it? Is it sensible to have two implementation of every library?
    Porting Numpy to PyPy is pretty crazy.

    Once there is decent Cython support, we might start talking about writing scientific python in Cython, so that it runs on both CPython and PyPy. But until then – no way!

  4. Thomas Kluyver says:

    Unfortunately, for a lot of the scientific community using Python, PyPy is still essentially an interesting research project. We have a lot of development invested in C extensions, which PyPy has no interest in supporting well. And since the critical parts are already in compiled code, there’s little payoff for the massive rewriting that would be necessary.

  5. It’s been said better elsewhere, but for the community of Python users with whom I most often interact (that is, scientific users), PyPy is currently a no-go as well. Part of what makes NumPy a bedrock is its extensive C API upon which mountains of packages are based; rewriting all of those packages will take an immense amount of time and effort, and even then, easy integration with legacy C and Fortran libraries is immensely important. Rewriting such libraries in Python is simply not a serious option for a lot of reasons. See Peter Wang’s NumPy isn’t about fast arrays and More thoughts on arrays in PyPy for details. Succinctly put, a NumPy C API is not an implementation detail, it is a necessity.

    I really do wish this were not the case. Interpreted CPython performance sucks, and PyPy and the STM patch seems like the best hope for real multi-threaded parallelism in Python. If I were writing a conventional Python web application or something like that, or doing non-numerical data crunching, I might consider PyPy. But array computation benchmarks alone are not going to rip me away from CPython for numerical work, simply because I’d be leaving behind the ecosystem and not gaining the tools necessary to rebuild it (setting aside, for the moment, the manpower problem).

  6. Volker Birk says:

    Why I stopped using PyPy? Well, with my tool chain, it was slower than CPython, so I saw no reason.

    I think, PyPy is a cool project, though.

  7. Dmitri Lebedev says:

    Guido’s keynote was quite dissapointing for me, despite the content of the message. When you say “Python is everywhere”, saying “It’s not our fault that were not on mobile” sounds really bad. Then, I heard nothing about why Python is adopted this wide, like it just happened. So I get Guido’s message as “The adoption of Python is out of our control.”

    • Dmitri Lebedev says:

      Just to compare, see Jakob Kaplan-Moss’ keynote at DjangoCon 12, where he presented Meteor.js, a very threatening future competitor, said “this is too hard in Django”, yet the whole talk was very inspirational and made me personally to start using Sphinx finally.

  8. tehwalrus says:

    I work in Python, Cython, C (using the cython interface) and I write wrappers for other peoples’ FORTRAN (in cython, via direct C access). If all of that works in PyPy, then I’d be happy to try it.

    Oh, I also require pygame, pyopengl, cjson (or a json library that doesn’t use 6GB RAM to open a 100MB document, like the default CPython module,) and so on.

    Seriously, my entire PhD is (indirectly) trying to make python faster – Support this and I’ll switch.