Python 3 Object-oriented Programming Second Edition

One of several reasons this blog has been so quiet this year is the time I’ve invested in the second edition of my first book.

I am extremely proud of the end result. The first edition of Python 3 Object Oriented Programming was great and garnered 30 five star reviews on Amazon. This edition is better. So much better. I’ve restructured the content so the topics flow much more evenly into each other. I improved examples, clarified content, and added a new chapter. All in all, this book is a great resource to teach a new generation of programmers the advantages and pitfalls of object-oriented programming.

I’ve matured a lot in the five years since the book was originally published. I’m a better coder, a better writer, and a better person. All of this is reflected in this, my best work to date. In all honesty, if someone else had written this exact book, I would probably be recommending it even more heartily! So if you’re interested in advancing your knowledge of object-oriented programming, or Python 3 in general, take a look at Python 3 Object-oriented programming, second edition. You won’t be disappointed.

With this project done, I am optimistic that you will see more updates to this blog in the future. I love researching and writing, and with no book to invest those activities in, this blog will, once again, become my outlet.

Opterator 0.5 Supports Function Annotations

Opterator is a decorator to make quick and clean command line interfaces for scripts. It’s not designed to compete with argparse or click. Instead it supports that common usecase where you’re writing a reusable script and don’t want to write all the bits required for argument parsing, but also don’t want to use the maligned ‘just access sys.argv and pray’ technique.

Say you’ve written a script that makes a thumbnail of an image. It requires the image name, the output thumbnail name, and an optional algorithm. Just do this:

from opterator import opterate

@opterate
def main(image, thumbnail, algorithm="cubic"):
    # do the scaling stuff

if __name__ == "__main__":
    main()

There’s no mucking about with options, and you even get a helptext that is borderline useful:

python3 meh.py -h
usage: meh.py [-h] [-a ALGORITHM] image thumbnail

positional arguments:
  image
  thumbnail

optional arguments:
  -h, --help            show this help message and exit
  -a ALGORITHM, --algorithm ALGORITHM

If that’s not good enough, you can customize the helptext by editing the docstring, as described in the
README.

But opterator has supported that for years. Let’s talk about some new stuff. I recently released version 0.5, which has a few bugfixes and a couple exciting new features that I want to introduce.

First, I ported it from optparse to argparse. This makes opterator handle positional arguments much more neatly, both in the code and in the resulting helptext. It doesn’t really change your interaction with opterator as an api client, but the users will see improved helptext.

Second, I’ve added support for function annotations so you can change the variable name output by opterator. This obviously only works on Python 3, since function annotations are a new feature in Python. Opterator itself, however, is Python 2 and 3 compatible as it has been since 2012. Here’s the same example, but we changed the algorithm argument to be passed in as –scaling or -s:

from opterator import opterate

@opterate
def main(image, thumbnail, algorithm:["--scaling", "-s"]="cubic"):
    # do the scaling stuff
    pass

if __name__ == "__main__":
    main()

Here’s the resulting helptext:

usage: meh.py [-h] [--scaling ALGORITHM] image thumbnail

positional arguments:
  image
  thumbnail

optional arguments:
  -h, --help            show this help message and exit
  --scaling ALGORITHM, -s ALGORITHM

The main reason you might want to do this is that you’ve got options that start with the same letter (the first letter is what opterator uses as a short option, by default). Function annotations let you supply other options if you like. Or possibly you just have a burning desire to play with function annotations. Either way, opterator has you covered.

Opterator is well-tested, so I’ve (belatedly) upgraded the classifier from beta to production. It’s a small tool that has saved me a lot of time and many lines of ugly code over the years. I think it’s worth having installed in site- packages so you can write quick scripts without thinking about getting data from the user.

Creating Apps In Kivy: The Book

I apologize for my near-silence over the past few months. I’ve packed house and moved twice in that time, started a new job, and, relevant to this post, written a book.

My Creating an application in Kivy blog series has been by far the most popular set of posts on this blog. It garners about 200 views per day, and has been doing so for nearly a year. This taught me a very important lesson: posts with controversial titles, (notably this one, but basically anything that has hit hacker news) are much less valuable than posts with meaningful long-term content. Getting a few thousand hits in one day is pretty satisfying, but when most of your “readers” simply comment on the post without having read the article, that hit-count becomes meaningless.

“Creating an application in Kivy”, on the other hand, has had sixteen thousand visits to part 1, and nearly three thousand of those have made their way all the way to part 9. I could definitely achieve more hits per day by writing a new minimum-content controversial article every day, but these visitors keep coming with no effort on my part. This is lucky, as the focus on writing solid content for my book has taken time away from writing solid content for this blog.

And with that, I’d like to announce that Creating Apps In Kivy is now available. This project started when the aforementioned tutorial caught the eye of an O’Reilly editor. They helped me put together a great book that I am proud of and confident you will enjoy.

Creating Apps in Kivy covers many of the topics discussed in my tutorial, but in more depth and with much more polish. Like the tutorial, it takes you from “new to Kivy” to creating and deploying a complete application to your Android or iOS device. It’s a fun, short read and I’m proud to present it to you today. If you liked the tutorial, buying my book would be a great way to support my work!

Have you ever been afraid to get out of bed?

In my past writings about mental health, I’ve mostly discussed depression and suicide. I’ve been meaning to add ‘anxiety’ to that list of topics, but have been waiting until I could write about it from the first person. One of the peculiar things about my personal struggles with mental illness is that when I’m feeling well, it’s really hard to remember what it’s like to be feeling poorly. The inverse is also true, which is why it’s possible to feel suicidal after just a few hours of depression; I can’t imagine what it felt like yesterday when I’d been feeling fine for months on end.

I’ve got a great handle on what I need to do in my life to deal with, prevent, and accommodate for my depressive, suicidal, and bipolar tendencies. I’ve also made great strides in reducing anxiety levels, but, for me, anxiety is a more difficult beast to tame. The following was written a few days ago when I was in the middle of a high-anxiety experience (not an anxiety attack). I’ve edited it a bit for grammar (typing on a phone is a horrible experience), but it’s mostly straight from the heart. I hope it illustrates what anxiety feels like to those who have not experienced it, so you can support those in your life who are suffering. I would also like it to demonstrate to those who may not be aware that they are dealing with anxiety that their symptoms are treatable, and perhaps you will seek further help. And, as always, I want those of you who deal with anxiety on a regular basis to remember that you are not alone.

From Inside Anxiety

Let me tell you what anxiety feels like. It feels like being on a small rock amidst a boiling vat of lava. I have to type this on a phone because the laptop in the other room is no less inaccessible than if I had to jump a giant ravine. Anxiety is fear, pure simple fear. But it’s not normal fear; anxiety is fear of the normal.

Have you ever experienced an adrenaline rush? Some threatening situation: the grill of an oncoming bus, falling down the stairs, being approached by a thug with knife in hand, slipping on ice, a shove from behind, a car accident?

Anxiety is that same feeling, but all the time. Every step is a slip, every car is going to hit you, all your friends and co-workers are thugs with knives. The worst part is that the brain can see that these sensations are not true. I’m not imagining real lava around the bed. I can see the perfectly safe carpet. But my legs feel like numb jello and I cannot step onto it. I’m not hallucinating that all the people outside my street want to kill me. I’m afraid of them even though I know they have no interest in me at all. I know in my mind that my office is full of people I can trust and rely on, yet my body is reacting as though I will be entering a war zone, every sense alert, every muscle tensed against the next attack.

Once upon a time, I lived the majority of my life’s moments in such a constant state of heightened awareness. Fear. It was exhausting. I am lucky that days like this are now rare. I am lucky that today, my anxiety is not accompanied by depression (it’s hard to feel worthy when you spend all your time hiding in bed.).
Some people look for adrenaline rushes outside their normal state of being. They sky dive, they become immersed in video games or movie thrillers, they join extreme sports. While they do this, I look for quiet rooms, I meditate, I practice yoga. Anything to calm the blood down. I will experience the same thrill climbing into the shower this morning that you feel jumping out of that plane. I will feel the same fear walking into my office today that you feel as you step onto a public stage.

And here’s the kicker: nobody thinks I’m brave. My walk to work today will mirror an Indian Jones escape sequence or Aladdin’s carpet ride out of the Cave Of Wonders. I’ll walk into my office and be surrounded by people who do that kind of thing every day. Anxiety is living your life inside a thriller movie. The music is always loud, rushing your heart beat. Something awful is always going to happen around the next corner. This movie doesn’t end and you can’t walk out of the theater.

The world is going to end if I don’t get out of bed and into the shower. How much is a world worth?

Dealing With Anxiety

I have had less success in defeating anxiety than I have had with depression. There are a few types of treatments to deal with it, and you should consult with a medical professional to work out what the best one is for you. The two things that have had the greatest success in taming (if not controlling, yet) my anxiety have been yoga/meditation, and exposure therapy. The latter isn’t as bad as it sounds, although like many cures, it’s not always pleasant. The heart of exposure therapy is, as you can expect from the name, placing yourself in situations that cause anxiety. However, there are a few caveats. You have to place yourself in such situations until you become comfortable with them. If you exit an anxiety-inducing situation while you still feel afraid, you will have reinforced that this is a fearful situation, and that “fight or flight” is an appropriate response.

Instead, you need to pick situations that only cause a small amount of anxiety. A good way to do this is to make a list of as many situations as you can think of that cause anxiety, and then rank them in order of how frightening they are. Over the course of a year or so, put yourself into the easiest situations and work your way up the chain. It’s slow progress, but it’s measurable, and the feedback you give yourself from one encounter can help you refine your technique in a later one.

I’ve been utilizing this technique for three years. As illustrated above, I still have days with a lot of anxiety. However, I lead a functional life, now. I used to work from home, I now work for a high-profile company and I’m doing well at it. I used to avoid all social situations, I now regularly go out with friends or even strangers and generally have a good time. I used to be especially afraid to interact with single women; I’m now in a stable, loving relationship. There are still a few things on my list, such as presenting at conferences and dealing better with conflict, but for the most part my coping skills are better than many people who are considered not to have psychological disorders. It wasn’t easy, but it was worth it.

I’m Not Selling Out The Gittip Dream

A couple weeks ago, I signed a contract with O’Reilly to publish a book inspired by my extremely popular Creating An Application In Kivy blog series. On Monday, I start work at Facebook, a full-time salaried position that includes a substantial raise over my last position.

Signing these two contracts, two very traditional ways of obtaining money in exchange for services, is in stark contrast to the excitement I have held for the Gittip project. I marketed (some might say ‘begged’) for Gittip contributions in return for the service I performed in writing the Kivy articles. Gittip provided; I really didn’t expect to make more than a couple dollars per week, but my current income from Gittip is over $11 per week. I’m touched and grateful for these donations, and I don’t feel I deserve them at all, considering that I’ll be spending most of my time on my new book and job for the next few months.

It would be easy to argue that Gittip has failed. If I had somehow made more money off Gittip, I might have chosen to publish a book under a creative commons license or through other platforms, rather than signing a traditional book contract. If I was making so much money off Gittip that I could have laughed off Facebook’s salary and stayed out of the horrifically over-priced silicon valley, the open source world (and Facebook’s competitors!) would certainly have benefited.

However, there is a better way to look at this. I have committed (to myself) to contribute a substantial portion of the royalties from my new book back to the Kivy team. I will naturally use Gittip to distribute these funds. I am also planning to increase my funding of other projects out of my Facebook salary, and I’ll be using Gittip to fund the Arch Linux development team (if they ever set one up on Gittip) out of the income I already make from Arch Linux Schwag.

I believe one of the biggest problems Gittip has to solve is not increasing it’s member base, but increasing the amount of money being injected into the system. If small tips are just moving from one open source developer to another and back again (I’ve gotten into the unfortunate habit of regifting everything I make plus a few dollars), the money really has no value. It is a small gesture, but it’s not something that can be turned into food on the table if everyone just turns around and gives it away. I would definitely like to live a life, someday, where my writing and coding activities are exciting enough to the world that my entire salary comes from Gittip. However, I think the best thing I can do for the tool right now is to put money in, rather than take it out.

So no, Gittip has not failed me. It is succeeding in a different way from what I originally anticipated. Nor am I selling out. I am excited and passionate about my current and future prospects. Gittip was designed to allow content creators to pursue their passions. I’m able to do that within the traditional framework. I’m very lucky to be able to express my gratitude for my current life in the form of microdonations to other developers.

Arch Linux Schwag site is back up (a comment on privacy)

Hey all,

http://schwag.archlinux.ca has been down for at least a week. I’m really sorry about that. I’ve been working on transferring to a new web host and they were having trouble getting all the Python dependencies I needed.

My old host was based in the United States, and while they provided exceptionally good service for the price, I have become uncomfortable with storing any data in the US. The websites are not private, of course. However I am also hosting e-mail on these domains. As a Canadian citizen, I am not covered by the NSA’s (possibly false) claim that they do not spy on US citizens, so I felt that I should store it in my own country.

I don’t do anything that is currently illegal, but I don’t know that these arrogant governments aren’t going to change their laws to restrict my freedoms. I may not be a person of interest today, but I am outspoken about freedom, privacy, and transparency. This is currently legal and acceptable, but that may not always be the case and I want to be proactive about keeping my private data out of the hands of those who would use it against me.

The importance of “One Obvious Way”

In the face of ambiguity, refuse the temptation to guess.
There should be one– and preferably only one –obvious way to do it.
Although that way may not be obvious at first unless you’re Dutch.

— Tim Peters, The Zen of Python
>>> import this

While Python has numerous advantages over other languages that make me reach for it first when starting a new project, I feel the two most important are its emphasis on readability and the community belief in Tim’s words above, “there should be one– and preferably only one –obvious way to do it.” While it’s rare to stumble upon a Python coder that doesn’t implicitly “get” this principle, I find that is frequently misinterpreted when I try to explain Python’s charm to outsiders.

This may be because I often misquote the principle as “the one best way” principle. Many languages have a “one best way” principle. Java, for example, insists that the best way to solve any problem is object oriented programming. Similarly, prolog insists that logic programming is the best way, and Haskell programmers suggests that functional is always the way to go.

Python is not like this at all. Python is pragmatically multi-paradigm. Python has a long history of stealing great ideas from other languages (and, given Guido’s genius, making them better). Python does the very opposite of insisting that there is a best way to do everything.

Orthogonally, I have often had my defense of this principle misinterpreted as having said, “Python is always the best solution to any problem.” This makes me sound arrogant and misinformed, and is patently false. I am keenly aware of Python’s shortcomings. My day job has me pushing Python’s limits on a regular basis, and often, it fails me.

No, Python is not always the best way. The “one obvious way” principle refers to something quite different. It’s not about the language or its libraries at all, it’s more about the programmer’s mindset. Given any specific problem, there are invariably multiple ways to solve it. A Python developer will start out by assuming that one of those assorted solutions is the best solution. The goal of both the Python language and the libraries developed by the community is to make that best solution the obvious one to implement.

This is a form of consistency. Just as pep-8 strongly encourages us to be consistent with all other Python developers in our coding style, the “one obvious way” principle encourages us to be consistent in our choices of algorithms, data structures, and design patterns. In an ideal world, this means that any two Python programmers, given the same problem, would write virtually the same solution. Further, the solution that they proposed would be the best possible solution for the given problem, and all other pythonistas would find it elegantly readable.

This never happens. However, I believe that Python programs are closer to this ideal than many (if not most) other languages. This cohesion means that Python code is often more maintainable, not just by the original developer, but by all members of the Python community.

Some might suggest that strict adherence to the “one obvious way” principle is narrow-minded and restricts experimentation. I heartily disagree. There is clearly not one best web framework in Python. There are numerous web engines available, some more suitable to specific purposes than other. And more are being written every day. I think this is because nobody strictly believes that the most obvious implementation has been found, yet. The poem says there should be one obvious way. It doesn’t say that there always is one. The Python community as a whole is working together to find these best solutions and make them into the obvious solution. A few problems are solved. If you need an iterator, Python’s got you covered. Others are still up in the air, as evidenced by the collection of web frameworks or packaging tools or xml parsers on pypi. Some have only been recently solved; the Requests library is a terrific example. And still more are completely unsolved. The most obvious way I’ve found to handle concurrency and parallelism in Python is: Use Golang. Kivy is a terrific library, but the obvious way to do Android development is still (currently) Java.

In summary, while the ideal of “obviousness” is far, far away, I believe that it is a worthy goal to pursue. I think the mindset attracts a certain kind of programmer, and that those programmers are all as delighted when they discover Python as I was almost a decade ago. I don’t think Python is the best tool for all jobs, nor do I think it is the most enjoyable tool for all programmers. However, for those coders who like a certain kind of obvious, readable, elegance, Python is the language of choice.

Creating an Application in Kivy: Part 9

Welcome to the final part of this series on creating a working Jabber client in Kivy. While there are numerous things we could do with our code to take it to Orkiv version 2, I think it’s currently in a lovely state that we can call “finished” for 1.0.

In this article, we’re going to talk about releasing and distributing Kivy to an Android mobile device. We’ll use a tool called buildozer that the Kivy developers have written. These commands will, in the future, extend to other operating systems.

We aren’t going to be writing a lot of code in this part (some adaptation to make it run on android is expected). If you’re more interested in deploying than coding, you can check out my state of the orkiv repository as follows:

git clone https://github.com/buchuki/orkiv.git
git checkout end_part_eight

Remember to create and activate a virtualenv populated with the appropriate dependencies, as discussed in part 1.

Table Of Contents

Here are links to all the articles in this tutorial series:

  1. Part 1: Introduction to Kivy
  2. Part 2: A basic KV Language interface
  3. Part 3: Handling events
  4. Part 4: Code and interface improvements
  5. Part 5: Rendering a buddy list
  6. Part 6: ListView Interaction
  7. Part 7: Receiving messages and interface fixes
  8. Part 8: Different views for different screens
  9. Part 9: Deploying your kivy application

Restructuring the directory

Back in Part 1, we chose to put all our code in __main__.py so that we could run it from a zip file or directly from the directory. In some ways, this is pretty awesome for making a package that can easily be distributed to other systems (assuming those systems have the dependencies installed). However, Kivy’s build systems presume that the file is called main.py.

I’m not overly happy about this. Kivy sometimes neglects Python best practices and reinvents tools that are widely used by the Python community. That said, it is possible for us to have the best of both worlds. We can move our code into orkiv/main.py and still have a __main__.py that imports it.

First, move the entire __main__.py file into main.py, using git:

    git mv orkiv/__main__.py orkiv/main.py

Next, edit the new main.py so that it can be imported from other modules without automatically running the app. There is a standard idiomatic way to do this in python. Replace the code that calls Orkiv().run() with the following:

(commit)

    def main():
        Orkiv().run()

    if __name__ == "__main__":
        main()

There are two things going on here. First, we moved the code that instantiates the application and starts it running in a function called main(). There is nothing exciting about this function. It does the same thing that used to happen as soon as the module was imported. However, nothing will happen now unless that function is called.

One way to regain the previous functionality of immediately running the app when we type python main.py would be to just call main() at the bottom of the module. However, that is not what we want. Instead, our goal is to make the application automatically run if the user runs python orkiv/main.py, but if they run import main from a different module or the interpreter, nothing is explicitly run.

That’s what the if __name__ conditional does. Every module that is ever imported has a magic __name__ attribute. This is almost always the name of the module file without the .py extension (eg: main.py has a __name__ of main). However, if the module is not an imported module, but is the script that was run from the command line, that __name__ attribute is set to __main__ instead of the name of the module. Thus, we can easily tell if we are an invoked script or an imported script by testing if __name__ is __main__. Indeed, we are taking advantage of the same type of magic when we name a module __main__.py. When we run python zipfile_or_directory it tries to load the __main__ module in that directory.

However, we do not currently have a __main__ module, since we just moved it. Let’s rectify that:

    from main import main
    main()

Save this as a new orkiv/__main__.py and test that running python orkiv from the parent directory still works.

You’ll find that the window runs, however, if you try to connect to a jabber server, you get an exception when it tries to display the buddy list. This is because our orkiv.kv file was explicitly importing from __main__. Fix the import at the top of the file so that it imports from main instead:

#:import la kivy.adapters.listadapter
#:import ok main

OrkivRoot:

While we’re editing code files, let’s also add a __version__ string to the top of our main.py that the Kivy build tools can introspect:

(commit)

    __version__ = 1.0

Deploying to Android with buildozer

Deploying to Android used to be a fairly painful process. Back in December, 2012, I described how to do it using a bulky Ubuntu virtual machine the Kivy devs had put together. Since then, they’ve designed a tool called buildozer that is supposed to do all the heavy lifting. Once again, I have some issues with this plan; it would be much better if Kivy integrated properly with setuptools, the de facto standard python packaging system than to design their own build system. However, buildozer is available and it works, so we’ll use it. The tool is still in alpha, so your mileage may vary. However, I’ve discovered that Kivy products in alpha format are a lot better than final releases from a lot of other projects, so you should be ok.

Let’s take it out for a spin. First activate your virtualenv and install the buildozer app into it:

    . venv/bin/activate
    pip install buildozer

Also make sure you have a java compiler, apache ant, and the android platform tools installed. This is OS specific; these are the commands I used in Arch Linux:

    sudo pacman -S jdk7-openjdk
    sudo pacman -S apache-ant
    yaourt android-sdk-platform-tools

Then run the init command to create a boilerplate buildozer.spec:

    buildozer init

Now edit the buildozer.spec to suit your needs. Most of the configuration is either straightforward (such as changing the application name to Orkiv) or you can keep the default values. One you might overlook is that source.dir should be set to orkiv the directory that contains main.py.

The requirements should include not only kivy and sleekxmpp but also dnspython, a library that sleekxmpp depends on. You can use the pip freeze command to get a list of packages currently installed in your virtualenv. Then make a conscious decision as to whether you need to include each one, remembering that many of those (cython, buildozer, etc) are required for development, but not Android deployment.

For testing, I didn’t feel like creating images for presplash and the application icon, so I just pointed those at icons/available.png. It’s probably prettier than anything I could devise myself, anyway!

If you want to see exactly what changes I made, have a look at the diff.

The next step is to actually build and deploy the app. This takes a while, as buildozer automatically downloads and installs a wide variety of tools on your behalf. I don’t recommend doing it over 3G!

First, Enable USB debugging on the phone and plug it into your development machine with a USB cable. Then tell buildozer to do all the things it needs to do to run a debug build on the phone:

buildozer android debug deploy run

I had a couple false starts before this worked. Mostly I got pertinent error messages that instructed me to install the dependencies I mentioned earlier. Then, to my surprise, the phone showed a Kivy “loading…” screen. Woohoo! It worked!

And then the app promptly crashed without any obvious error messages or debugging information.

Luckily, such information does exist. With the phone plugged into usb, run adb logcat from your development machine. Review the Python tracebacks and you’ll discover that it is having trouble finding the sound file in.wav:

I/python  (25368):    File "/home/dusty/code/orkiv/.buildozer/android/platform/python-for-android/build/python-install/lib/python2.7/site-packages/kivy/core/audio/__init__.py", line 135, in on_source
I/python  (25368):    File "/home/dusty/code/orkiv/.buildozer/android/platform/python-for-android/build/python-install/lib/python2.7/site-packages/kivy/core/audio/audio_pygame.py", line 84, in load
I/python  (25368):    File "/home/dusty/code/orkiv/.buildozer/android/platform/python-for-android/build/python-install/lib/python2.7/site-packages/android/mixer.py", line 202, in __init__
I/python  (25368):  IOError: [Errno 2] No such file or directory: '/data/data/ca.archlinux.orkiv/files/orkiv/sounds/in.wav'
I/python  (25368): Python for android ended.

The problem here is that the code specifies a relative path to the sound file as orkiv/sounds/in.wav. This worked when we were running python orkiv/ from the parent directory, but python for android is running the code as if it was inside the orkiv directory. We can reconcile this later, but for now, let’s focus on getting android working and just hard code the file location:

(commit)

    self.in_sound = SoundLoader.load("sounds/in.wav")

While we’re at it, we probably also need to remove “orkiv” from the location of the icon files in orkiv.kv:

    source: "icons/" + root.online_status + ".png"

Finally, running buildozer and adb logcat again indicates the error message hasn’t changed. This is because we aren’t explicitly including the .wav file with our distribution. Edit buildozer.spec to change that:

    # (list) Source files to include (let empty to include all the files)
    source.include_exts = py,png,jpg,kv,atlas,wav

Run the buildozer command once again and the app fire up and start running! Seeing your Python app running on an Android phone is a bit of a rush, isn’t it? I was able to log into a jabber account, but selecting a chat window goes into “narrow” mode because the phone’s screen has a much higher pixel density than my laptop. We’ll have to convert our size handler to display pixels somehow.

I was able to send a chat message, which was exciting, but when the phone received a response, it crashed. Hard. adb logcat showed a segmentation fault or something equally horrifying. I initially guessed that some concurrency issue was happening in sleekxmpp, but it turned out that the problem was in Kivy. I debugged this by putting print statements between each line in the handle_xmpp_message method and seeing which ones executed before it crashed. It turned out that Kivy is crashing in its attempt to play the .wav file on an incoming message. Maybe it can’t handle the file format of that particular audio file or maybe there’s something wrong with the media service. Hopefully the media service will be improved in future versions of Kivy. For now, let’s use the most tried and true method of bugfixing: pretend we never needed that feature! Comment out the line:

(commit)

    #self.in_sound.play()

and rerun buildozer android debug deploy run.

Now the chat application interacts more or less as expected, though there are definitely some Android related quirks. Let’s fix the display pixel issue in orkiv.kv:

(commit)

    <OrkivRoot>:
        mode: "narrow" if self.width < dp(600) else "wide"
        AccountDetailsForm:

All I did was wrap the 600 in a call to dp, which converts the 600 display pixels into real pixels so the comparison is accurate. Now when we run the app, it goes into “narrow” mode because the screen is correctly reporting as “not wide enough for wide mode”. However, if you start the app in landscape mode, it does allow side-by-side display. Perfect!

And that’s the app running on Android. Unfortunately, in all honesty, it’s essentially useless. As soon as the phone display goes to sleep, the jabber app closes which means the user is logged out. It might be possible to partially fix this using judicious use of Pause mode. However, since the phone has to shut down internet connectivity to preserve any kind of battery life, there’s probably a lot more involved than that.

On my phone, there also seems to be a weird interaction that the BuddyList buttons all show up in a green color instead of the colors specified in the Kivy language file and the args_converter.

Third, the app crashes whenever we switch orientations. This is probably a bug fixable in our code rather than in Kivy itself, but I don’t know what it is.

Also, touch events are erratic on the phone. Sometimes the scroll view won’t allow me to scroll when I first touch it, but immediately interprets a touch event on the ListItem. Sometimes touch events appear to be forgotten or ignored and I have to tap several times for a button to work. Sometimes trying to select a text box utterly fails. I think there must be some kind of interaction between Kivy’s machinery and my hardware here, but I’m not certain how to fix it.

Occasionally, when I first log in, the BuddyList refuses to act like a ScrollView and the first touch event is interpreted as opening a chat instead of scrolling the window. This is not a problem for subsequent touch events on the buddy list.

Finally, there are some issues with the onscreen keyboard. When I touch a text area, my keyboard (I’m using SwiftKey) pops up, and it works well enough. However, the swipe to delete behavior, which deletes an entire word in standard android apps, only deletes one letter here. More alarmingly, when I type into the password field, while the characters I typed are astrisk’d out in the field, they are still showing up in the SwiftKey autocomplete area. There seems to be some missing hinting between the OS and the app for password fields.

Before closing, I’d like to fix the problem where the app is no longer displaying icons on the laptop when I run python orkiv/. My solution for this is not neat, but it’s simple and it works. However, it doesn’t solve the problem if we put the files in a .zip. I’m not going to worry about this too much, since buildozer is expected, in the future, to be able to make packages for desktop operating systems. It’s probably better to stick with Kivy’s tool. So in the end, this __main__.py feature is probably not very useful and could be removed. (Removing features and useless code is one of the most important parts of programming.) However, for the sake of learning, let’s make it work for now! First we need to add a root_dir field to the Orkiv app. This variable can be accessed as app.root_dir in kv files and as Orkiv.get_running_app().root_dir in python files.

(commit)

    class Orkiv(App):
        def __init__(self, root_dir):
            super(Orkiv, self).__init__()
            self.root_dir = root_dir
            self.xmpp = None

Of course, we also have to change the main() function that invokes the app to pass a value in. However, we can have it default to the current behavior by making the root_dira keyword argument with a default value:

    def main(root_dir=""):
        Orkiv(root_dir).run()

Next, we can change the __main__.py to set this root_dir variable to orkiv/:

    from main import main
    main("orkiv/")

The idea is that whenever a file is accessed, it will be specified relative to Orkiv.root_dir. So if we ran python main.py from inside the orkiv/ directory (this is essentially what happens on android), the root_dir is empty, so the relative path is relative to the directory holding main.py. But when we run python orkiv/ from the parent directory, the root_dir is set to orkiv, so the icon files are now relative to the directory holding orkiv/.

Finally, we have to change the icons line in the kivy file to reference app.root_dir instead of hardcoding the path (a similar fix would be required for the sound file if we hadn’t commented it out):

    Image:
        source: app.root_dir + "icons/" + root.online_status + ".png"

And now, the app works if we run python orkiv/ or python main.py and also works correctly on Android.

There are a million things that could be done with this Jabber client, from persistent logs to managed credentials to IOS deployment. I encourage you to explore all these options and more. However, this is as far as I can guide you on this journey. Thus part 9 is the last part of this tutorial. I hope you’ve enjoyed the ride and have learned much along the way. Most importantly, I hope you’ve been inspired to start developing your own applications and user interfaces using Kivy.

Financial Feedback

Writing this tutorial has required more effort and consumed more time than I expected. I’ve put on average five hours per tutorial (with an intended timeline of one tutorial per week) into this work, for a total of around 50 hours for the whole project.

The writing itself and reader feedback has certainly been compensation enough. However, I’m a bit used up after this marathon series and I’m planning to take a couple months off before writing anything like this tutorial again in my free time.

That said, I have a couple of ideas for further tutorials in Kivy that would build on this one. I may even consider collecting them into a book. It would be ideal to see this funded on gittip, but I’d want to be making $20 / hour (far less than half my normal wage, so I’d still be “donating” much of my time) for four hours per week before I could take half a day off from my day job to work on such pursuits — without using up my free time.

A recent tweet by Gittip founder Chad Whitacre suggests that people hoping to be funded on gittip need to market themselves. I don’t want to do that. I worked as a freelancer for several years, and I have no interest in returning to that paradigm. If I have to spend time “selling” myself, it is time I’m not spending doing what I love. In my mind, if I have to advertise myself, then my work is not exceptional enough to market itself. So I leave it up to the community to choose whether to market my work. If it is, someone will start marketing me and I’ll see $80/week in my gittip account. Then I’ll give those 4 hours per week to tutorials like this or other open source contributions. If it’s not, then I’ll feel more freedom to spend my free time however I like. Either way, I’ll be doing something that I have a lot of motivation to do.

Regardless of any financial value, I really hope this tutorial has been worth the time you spent reading it and implementing the examples! Thank you for being an attentive audience, and I hope to see you back here next time, whenever that is!

Creating Apps In Kivy: The Book

My request for crowd funding above didn’t take off. I’m making just under $12 per week on Gittip, all of which I’m currently regifting. Instead of my open contribution dream being fulfilled, I opted to follow the traditional publishing paradigm and wrote a book Creating Apps In Kivy, which was published with O’Reilly. I’m really excited about this book; I think it’s the highest quality piece of work I have done. If you enjoyed this tutorial, I encourage you to purchase the book, both to support me and because I think you’ll love it!

Bad Guy Discrimination

On my flight back from an absolutely amazing Python Canada 2013, I watched “42″ (Jackie Robinson’s story) and about half of “Skyfall”. The former warmed my heart because the racial discrimination was so disturbing. You’re probably reading that twice and scratching your head, so let me explain: It’s heartwarming that we have made progress in fighting racism, and that things that were considered normal 70 years are truly disgusting today. While nauseating racism, sexism, homophobia, and other forms of othering still exist today, in Canada, at least, it is no longer considered normal. For example, last year, the NHL media was in uproar because of an alleged racial slur against PK Subban. Yes that slur should not have happened, but 60 years ago, it would have been encouraged.

Skyfall naturally reminded me of past decades of James Bond movies. Traditionally, the bad guy in a Bond film is a dude with an accent reflecting whatever country the American media (while Bond is British, the films are Hollywood) was hyperventilating about at the time. This “bad guy with an accent” portrayal is a form of xenophobia that Hollywood has been projecting for decades.

In Skyfall, as well as a majority of the very few other action movies I’ve seen in the past few years, the bad guy is not black like in the 70s, Russian like in the 80s, or middle Eastern like in the 90s. In movies such as Skyfall, Iron Man 2 and 3, all three Batman movies, and a super hero movie that I can’t remember the name of (citation required) the bad guy as a deranged local white man. Some of those insane white dudes have been phenomenal villains. Heath Ledger’s incredible portrayal of The Joker was particularly poignant.

Deranged local white men. These movies are silently teaching us, in the same way that movies of the 60s and 70s taught us that blacks are gangsters and women simper, that mentally ill people are dangerous, different, frightening. And you know what? We are! Even with the empathy I have developed from group therapy sessions and being hospitalized alongside seriously mentally ill individuals, I still get nervous when “crazy people” approach me on the street. I personally identified with The Joker, and my girlfriend at the time even commented that he reminded her of me.

The bad guy in a film normally has to be a little demented. I don’t want to live in a society where the desire to blow up entire cities of peace loving citizens is considered sane. I’m just here to remind people to encourage, rather than suspend, disbelief when watching all movies. Disbelieve the gender roles typified in Disney films. Disbelieve that people with accents are bad guys. Disbelieve that Asians can kill you with their pinky. Disbelieve that crazy men, men like me, are necessarily evil.

Creating an Application in Kivy: Part 8

Sorry for the delay in posting this, folks. I’ve been away on vacation and then at Pycon Canada, which was amazing. And another apology, in advance, that this article is going to be bloody short compared to parts 1 through 7. That’s not only because I’m short on time; it also turned out that what I wanted to do was easier than I expected!

Anyway, time for content! Now that we have a working jabber client, is our job done? The client is certainly functional, but I’d still like to experiment with displaying a different representation depending on the size of the window. Specifically, if it is wider than a certain number of pixels, let’s show the buddy list beside the chat window; otherwise, let’s stick with the switching back and forth that we currently have.

If you haven’t been following along or want to start this example at the same point at which I’m starting, you can clone the github repository with my examples:

git clone https://github.com/buchuki/orkiv.git
git checkout end_part_seven

Remember to create and activate a virtualenv populated with the appropriate dependencies, as discussed in part 1.

Table Of Contents

Here are links to all the articles in this tutorial series:

  1. Part 1: Introduction to Kivy
  2. Part 2: A basic KV Language interface
  3. Part 3: Handling events
  4. Part 4: Code and interface improvements
  5. Part 5: Rendering a buddy list
  6. Part 6: ListView Interaction
  7. Part 7: Receiving messages and interface fixes
  8. Part 8: Width based layout
  9. Part 9: Deploying your Kivy application

Thinking through design

Since I’m not entirely certain how I want this code (or interface) to look, let’s spend a few paragraphs describing it so we all know we’re developing the same thing. It’s simple to say “we want chat windows to display beside the buddy list when the screen is big enough”, but what does that mean exactly? At what points in the code do we have to handle it? I’m actually “thinking out loud” here so you can see the various paths a brain can follow when programming.

Let’s follow the process from program startup. When the account credentials are displayed, we may as well just show the form no matter how big the screen is. When the Buddy List is first displayed after login, it can take up the whole screen as well. It’s not until we open our first chat that we have to decide if we are going to add the new chat beside the buddy list or to replace it. Of course, it’s not just a matter of adding the chat window, we also have to remove any existing chat windows, since we don’t just want to keep adding windows every time they click a new buddy. One way we could do this is add a ChatWindowContainer and ensure that widget only contains one chat window. Another is to keep track of which widows are loaded and unload them as needed. A third option is to clear all widgets and add both the Buddy List and Chat Window to a clean slate.

Regardless of the width of the screen, when the user clicks the “Buddy List” button in the chat window, it should do the same thing: close all chat windows and show only the buddy list, full width. One option worth exploring is to hide that button when we are in wide screen view. Or perhaps the button should be removed altogether; a gesture might be a better interface. While gestures are all the rage in mobile devices right now, I’m not a fan of them from an affordance perspective. It’s never clear to the end user when or where a gesture is appropriate. While I use them extensively in my highly customized AOKP installation, I wouldn’t want to explain a gesture based interface to my parents.

In addition to deciding what to render when chat windows are added or removed, we need to do something when the window changes size. If it is in wide screen mode while a chat window is displayed, and then gets smaller, we have to remove the buddy list to switch to the original behavior. If it is in narrow mode while a chat window is displayed and it gets wider, we have to add the buddy list back.

I feel like it would be wise to have a “mode” property bound to the window size, so that we always know whether we are currently supposed to be in wide or narrow mode. However binding that property to size means that the mode will change without telling us what the “old” mode was. Thus, we have no way to know whether the mode we are supposed to be in is the one we are currently in. On the other hand, such information can be inferred from the size or contents of the child list on the OrkivRoot object.

Finally, it seems most of our calculations can be simplified by noticing that we only have to do things different if a chat window is currently displayed or about to be displayed.

Now I think I have a plan in my head. You may or may not have come to the same conclusions while reading the above, and it’s possible you’ve had some better ideas. That said, this is how I intend to implement this. It looks like it’s going to be simpler than I originally expected. Hooray for thinking things through in advance!

1. Add a mode property to OrkivRoot that is automatically set from the size of the window
2. Add chat_visible and budddy_list_visible python properties (not kivy properties) that tells us if there are buddy list or chat windows visible.
3. Check the mode when adding a chat window and decide whether to replace the buddy list or not
4. On a size event, if a chat window is currently visible, determine whether the buddy list needs to be added or removed.

Properties

We’ve looked at Kivy properties in past parts of this tutorial, and you may already be familiar with Python properties. It’s somewhat unfortunate that the same word is being used to explain two different programmatic concepts. However, there are only so many words to describe the idea of a “characteristic”, and this idea is extremely common in many different programming libraries. So some overloading of terminology is inevitable. I tend to use the phrase “Kivy property” to refer to the kinds of properties that Kivy supplies. Kivy properties are pretty cool because they can automatically update themselves when other properties change and events can be bound to them so that those other properties also update themselves. In other words, Kivy properties are smart.

Python properties are not stupid, mind you. People sometimes have trouble wrapping their mind around Python properties, but they’re actually pretty simple. It might be convenient to think of python properties as just a way to make a method look like an attribute. Thus, you can access something like node.color and underneath the hood, it’s actually behaving as though node.color() was called. However, they’re even more interesting than that, because you can actually call a completely different method when you set the value of the property. Thus node.color = "red" can do about the same thing as node.color("red").

The confusing thing about python properties is not so much “how do I do this?” as “why would I do this”? There are many problem-specific answers, but two general ones come to mind. The first is the Python idiom that readability counts. In general, when you access a property on an object, you are doing something with an adjective on a noun. In contrast, when you call a method on an object, you are doing something (a verb) to a noun. However, sometimes when you are manipulating an adjective, you need to do a little extra calculation or manipulation when the value is set or accessed. Using properties, it’s possible for the code accessing the property to read “naturally”, as though you were doing something with an adjective on a noun, even though there is actually a little “doing to” going on as well.

The second reason is actually an evolution of the first. Often, in the first version of an API, we just add an attribute to an object and allow client code to manipulate it. Then, in a later iteration of the code, we want to do some kind of calculation, caching, or validation on these attributes. However, we can’t suddenly change the attribute into a method or the old client code will break. However, we can use properties to add these new manipulations and keep the original API, which is probably more readable from the calling code’s perspective, anyway.

At any rate, our OrkivRoot class needs to have one Kivy property and two Python properties in order to perform the calculations for treating the screen differently depending on its size:

(commit)

    class OrkivRoot(BoxLayout):
        mode = StringProperty("narrow")

        @property
        def chat_visible(self):
            return ChatWindow in {c.__class__ for c in self.children}

        @property
        def buddy_list_visible(self):
            return self.buddy_list in self.children

So we have a simple mode property that is exactly the same as any other Kivy property. We also have two read only python properties named buddy_list_visible and chat_visible. The former just checks if the buddy_list object is one of the children. The latter loops through the children of OrkivRoot and checks if any of them are instances of the ChatWindow class. Any time you access orkivroot.chat_visible, this method is executed.

The odd notation in the chat_visible property may need some explanation. (Note: If you’re running python 2.6 or older, it won’t even run) Python has a variety of comprehension syntaxes that allow us to create new iterator objects such as lists, generators, dictionaries, and sets based on existing iterators. This syntax is optimized for efficiency as compared to looping over one iterator and manually adding it to another. It’s also more readable if you know the syntax.

The braces, which you are probably used to denoting a dictionary, in this case actually generate a set, based on the contents of another iterator: self.children. For each element in self.children it adds the class of that element to the new set. Then the in operator is used to determine if a ChatWindow class is in that set.

In theory, we use a set instead of a list for efficiency. However, in this case, I’m actually only using it for pedagogy. The in keyword supports much faster lookups in a large set than it does in a large list. However, in this case, the number of children of a OrkivRoot will never be large. I’m just trying to drive home the point that different data structures have different strengths and weaknesses, and you need to choose them wisely. For example, a few months ago, one of my colleagues shaved nearly an hour off our loading process by converting a list to a set in this very fashion.

I hope you find my digressions educational! As I’ve said before, my goal is to not just explain how to build a Jabber application in Kivy, but to demonstrate the various thought processes behind designing your own application in whatever programming language or library you might choose.

Let’s ignore those python properties for a bit and look back at that Kivy property. While the mode StringProperty has been defined, it’s not yet being set to anything. Let’s fix that in the Kivy language file:

(commit)

    <OrkivRoot>
        mode: "narrow" if self.width < 600 else "wide"
        AccountDetailsForm:

That one new line of code is all that is needed to have a dynamically updating mode attribute. Any time the window size changes, the mode will automatically (it’s almost automagic, in fact) be set to either “narrow” or “wide” depending on what the width is. Kivy properties are very powerful. The width property of widget is automatically updated whenever the window gets resized. Then the Kivy language does a wonderful job of hooking up the event bindings such that any time that width changes, a callback that executes the line of code above is invoked.

For the record, I tested these properties by adding a couple print() statements when the show_buddy_chat method is invoked to render the current mode.

Make it work

Now we need to rework the show_buddy_chat method just a little:

(commit)

    def show_buddy_chat(self, jabber_id):
        self.clear_widgets()
        if self.mode == "wide":
            self.add_widget(self.buddy_list)
        self.add_widget(self.get_chat_window(jabber_id))
        self.buddy_list.new_messages.discard(jabber_id)
        self.buddy_list.force_list_view_update()

Instead of explicitly removing the buddy_list, we now start with a clean slate by calling clear_widgets. Then we check if the mode is “wide” and add the buddy_list if it is. Then the chat window is added just like before, but now if the view is wide, it will peacefully slide in beside the buddy_list instead of replacing it.

The code for changing the window contents when the mode changes is a bit longer. The logic is simple, it’s just verbose. We need to add an on_mode method to the OrkivRoot. This method will only be fired when the mode changes thanks, once again, to the magic of Kivy properties.

(commit)

def on_mode(self, widget, mode):
    if mode == "narrow":
        if self.chat_visible and self.buddy_list_visible:
            self.remove_widget(self.buddy_list)
    else:
        if self.chat_visible and not self.buddy_list_visible:
            self.add_widget(self.buddy_list, index=1)

When I first tested this method, it failed because I didn’t accept any arguments on the event. So I had to figure out what the expected arguments were. I did this by making the method accept *args, which says “put all the non-keyword arguments in a list named args”. Then I printed that list to see what they were. So I knew that the arguments passed were a widget object and the current value of mode. Introspection in Python is so easy!

If the mode is narrow, it checks if both widgets are currently visible and removes the buddy_list if this is the case. The only way both widgets could be visible on an on_mode event is if the mode just switched from wide to narrow.

If the mode is wide, it checks if a chat window is visible and the buddy list is not. My first attempt only checked if the buddy_list was not visible and added it arbitrarily. This failed on startup, since OrkivRoot has a size, and therefore a mode, right from startup, even when the login form is being displayed. Thus, we only do the widget juggling if a chat_window is currently visible.

The index=1 argument to add_widget tells it to insert the buddy_list before the already visible chat_window rather than after.

And that’s it! Test that, try switching back and forth between the buddy list and the chat window, try resizing the window, try switching chats while the buddy list is visible. I’m a little dubious that it’s perfect, but I can’t find a situation where these few lines of code aren’t enough to do what I want. That’s rather unfortunate, because even with all my detours and excursions, this tutorial is incredibly short! However, I’m going to publish it as is, since you’ve all been waiting patiently while I was taking vacations and wandering around at Pycon Canada (where Kivy was mentioned both in the keynote speech and in the first talk of the day). Next week, let’s try to run this sucker on Android!

Financial Feedback

If you are enjoying this tutorial series and would like to see similar work published in the future, please support me. I dream of a future where open companies like gittip and open contributions help the world run in harmony instead of in competition. I believe that my work is high quality and worth backing. If you think this article held enough value that you would have paid for it, please consider backing me in one or more of the following ways: