Hacking Happy

I am extremely proud to announce the release of my new book, Hacking Happy. I believe every geek should read this book to help them find the joy and happiness in their lives that I have in mine.

The foundations of this book started with a hospital stay, as I outlined a couple of months ago. It was on the psychiatric ward that I started learning about the skills that ultimately led to my recovery. These techniques are as learnable as any programming language. My book helps you discover them for yourself.

Hacking Happy will motivate you to find techniques that work for you to optimize the happiness levels in your life. It is not a book of ready-made solutions. It is a true hacker’s manual. It provides processes. You find the solutions for the system you are hacking: yourself. The entire book is a series of analogies (some suitable, others simply amusing) to common software and systems development situations. However, the systems presented have been scientifically proven by psychologists and psychiatrists.

I have to admit that about halfway through writing Hacking Happy, I decided not to publish it. It seemed corny and the tools I was describing appeared so obvious to me. I couldn’t see how anyone would pay to read it. Then I asked myself, “If they are so obvious, why did it take me two decades and several mental breakdowns to find them?”

So I rewrote the book. It is now the best writing and most important work I have created, yet. It is hilarious. It is motivating, moving, and inspiring. It is educational and instructive. It is essential.

Marketing is a skill I’m still trying to learn. So I’m turning to you, “the crowd” to help market it! If you don’t buy the book, at least share it. Tell your friends about it. Retweet it. Like it. Plus One it. I think “Hacking Happy” is vital reading for every coder and systems professional out there. If you don’t love yourself completely and unconditionally, buy this book.

Hacking Happy is available in print and eBook formats. I’ll be adding more links to the home page as it propagates to more retailers. As of today, you can buy the book in print from CreateSpace or Amazon.com, or download eBooks in the following formats:

Gitifyhg: Accessing Mercurial repos from GIT

My company uses Mercurial for internal hosting. Other than that, it’s an absolutely terrific place to work.

About three quarters of my colleagues are sick of hearing the rest of us complain about Mercurial’s inadequacies. They mention tutorials like this one that simply don’t work in real life. I’ve done my research, and I have not been able to find a viable git to hg pipeline anywhere. There is a git-hg repo, but it appears to be un-maintained and not overly well documented. It’s also written in bash, so I’m not eager to take over maintenance.

Instead, I’ve been spending some of my non-working hours trying to make a python wrapper around hg-git named gitifyhg. It started out as an automated version of the hg-git tutorials, but is now expanding to provide additional support and tools.

Right now I’m following a git-svn style of workflow, where your git master branch can be synced up with the hg default branch. It seems to be working somewhat ok. I think at this point, the pain of using gitifyhg is about equal to the pain of using hg alone. Hopefully future improvements will reduce that pain, and as always, patches are most welcome!

The instructions are pretty clear and the unit tests provide a pretty good description of basic usage. Non-basic usage probably doesn’t work (yet). Here’s a tutorial of what does work:

Acme corporation uses Mercurial for all their repositories. Fred and Wilma are developing a project named AcmeAdmin together. Fred is content to use Mercurial for hosting, but Wilma is a git advocate who is going crazy with the restrictions Mercurial places on her. She decides she’s going to try gitifyhg on this new project to see how much trouble it is.

Fred starts the project by creating a Mercurial repo that will serve as the remote repo for both of them:

mkdir acmeadmin
cd acmeadmin
hg init

Fred and Wilma are doing a very strange sort of pair programming where they have separate repositories on the same machine. They are doing this so if you were reading along, you could type in the same commands they are typing and the demo would work. Rest assured that in spite of this queer practice, Fred and Wilma are hotshot programmers. So Fred now clones this remote repo and starts working. In the meantime, Wilma is browsing the gitifyhg README, so she hasn’t cloned anything yet.

cd ..
hg clone acmeadmin fredacme
cd fredacme
echo "Write Documentation" >> TODO
hg add TODO
hg commit -m "write the documentation for this project."
echo "Write Unit Tests" >>TODO
hg commit -m "unit tests are done"
hg push

Wilma’s caught up now and eager to try gitifyhg. She takes over the keyboard and clones the remote repository, which already contains Fred’s changes. Then she runs gitifyhg and verifies that a git repository exists with Fred’s two commits.

cd ..
hg clone acmeadmin wilmaacme
cd wilmaacme
gitifyhg
git log

Fred’s gone for coffee. Wilma now starts her own hacking and pushes the commits using gitifyhg.

echo "Implement code until tests pass" >>TODO
git commit -m "code implemented" -a
git hgpush

Wilma got kind of lucky here because she implemented her code on the master branch and there were no conflicts with Fred’s work. If there had been, I’m not sure what would have happened. Fred’s finished his coffee and pulls in Wilma’s change via Mercurial. He then commits some more changes.

cd ../fredacme
hg pull -u
echo "deploy" >> TODO 
hg commit -m "it's up and running"
hg push

Meanwhile, Wilma is working on a separate feature. She remembers to create a new git branch this time, which is good because she’s going to want to do some rebasing when Fred’s commit comes in.

cd ../wilmaacme
git checkout -b feature_branch
echo "new feature documentation" >> FEATURE
git add FEATURE
git commit -m "start new feature by documenting it"

Before pushing her changes, Wilma pulls to see if Fred has done anything. He has! But the hgpull merges all that into her master branch. She checks out her working branch and rebases it onto master. Then she pushes it upstream.

git hgpull
git checkout feature_branch
git rebase master
git checkout master
git merge feature_branch
git hgpush

And that’s the basic workflow! I’m probably going to add an hgrebase command to take care of that checkout rebase checkout merge step, since I expect it to be common. It should probably use git rebase -i, which is probably the most amazing thing that ever happened to version control.

Like I said, this is how gitifyhg works when all goes well. When things don’t go well it’s still a bit of a mess. I’ve had to manually sync up the hg and git working directories using git reset --hard and hg update -C a couple times. Once, my hgpush ended up putting a bookmarked branch on my hg repo that I didn’t want to go public (luckily, hg push fails by default when there are multiple heads); I ended up having to use hg strip to clean it up. I’m hoping to automate or prevent some of this in the future. For now, try it out and submit patches or at least issues!

Gesture Recognition In Kivy

Kivy is a modern GUI platform that runs on Windows, Linux, Mac, iOS, and Android.

Kivy supports gestures, but the documentation is a as to how to use them. Reviewing the Gesture Board example provides most of the missing pieces if you’re willing to experiment. I have done these experiments and hope this article will make it easier for the next coder.

The kivy.gesture module contains two main classes, the Gesture and GestureDatabase. A Gesture represents the stroke or strokes in a gesture. It maps a sequence of (x,y) coordinates to a normalized representation and can be compared to another sequence of points to determine if the two sequences “match”. In Kivy, gestures can be encoded in base64. This provides an easy way to store and load gestures in source code.

The GestureDatabase is essentially a collection of Gesture objects. Its primary purpose is to compare a new gesture as input by the user to those stored in the GestureDatabase and return the closest matching gesture, if one exists.

Before we can recognize whether a user’s gesture is meaningful, we need some gestures to compare it to. Luckily, the gesture_board.py that ships in Kivy’s examples directory does this for us. Run python gesture_board.py from a terminal. A blank window opens up. Draw a gesture on it.

Have a look in the terminal. There is a variety of output there, but the important one is the long base64 encoded string following the words “gesture_representation:”. Copy that string into a variable in a basic Kivy app:

from kivy.app import App

down_stroke = "eNq1l91u4zYQhe/1IslNjPkfzgtkb<snip>"
square = "eNq1mEluIzcYRvd1EXsT4Z+HC6i3AXyAwG<snip>"


class TestApp(App):
    pass

TestApp().run()

Now let’s set up a quick and dirty GestureDatabase from the given strings. Normally, I’d put these as an instance variable on the App or a specific widget, but for easy illustration, I’ll just toss them into the module:

from kivy.app import App
from kivy.uix.widget import Widget
from kivy.graphics import Line
from kivy.gesture import Gesture, GestureDatabase

down_stroke = "eNq1l91u4zYQhe/1IslNjPkfzgtkb<snip>"
square = "eNq1mEluIzcYRvd1EXsT4Z+HC6i3AXyAwG<snip>"

gestures = GestureDatabase()
gesture = gestures.str_to_gesture(down_stroke)
gesture.name = "down_stroke"
gestures.add_gesture(gesture)
gesture = gestures.str_to_gesture(square)
gesture.name = "square"
gestures.add_gesture(gesture)

.
.
.

The next step is recording the gesture that the user makes. This requires keeping track of the touch down, move, and up events. Let’s create a new widget to handle this (Note, for clarity, I’ve omitted collision detection and error conditions):

from kivy.uix.widget import Widget
from kivy.graphics import Line


class TestWidget(Widget):

    def on_touch_down(self, touch):
        touch.ud['gesture_line'] = Line(points=(touch.x, touch.y))

    def on_touch_move(self, touch):
        touch.ud['gesture_line'].points += [touch.x, touch.y]

    def on_touch_up(self, touch):
        # compare the gestures

The final step is to implement on_touch_up to convert the stroke the user created into a gesture and compare that gesture to those in the database:

    def on_touch_up(self, touch):
        gesture = Gesture()
        gesture.add_stroke(
            zip(touch.ud['gesture_line'].points[::2],
                touch.ud['gesture_line'].points[1::2]))
        gesture.normalize()
        match = gestures.find(gesture, minscore=0.70)
        if match:
            print("{} gesture occured".format(match[1].name))
        else:
            print("No gesture recognized")

This code requires some explanation. The Line object stores it’s points in a one dimensional list, where alternate indexes represent the x and y coordinates of each point. However, add_stroke expects a list of tuples of x and y values. In code, Line stores [x1, y1, x2, y2, x3, y3] while add_stroke expects [(x1, y1), (x2, y2), (x3, y3)]. Hence, the rather complicated call to zip.

The gestures.find call accepts a minscore (1.0 would mean the gesture matched perfectly). It will return the gesture that matches with the maximum matching score, but only if that maximum is above minscore. 0.70 seems to be suitable for basic gestures, though I have been able to confuse a ‘square’ with a ‘circle’ in my experiments.

gestures.find returns either None or a tuple of (score, gesture). Thus, if there is a match we need to pull out the Gesture via match[1].

And there you have it: basic gesture recognition in Kivy.

Unfortunately, the touch events are gobbled up by the gesture code, so if you have a gesture widget that contains other widgets, they won’t receive any events. I have taken a stab at creating a GestureBox widget that passes events through to child widgets. It seems to work for touch events, but deciding whether a motion event

Python on Android? First impressions of Kivy

Kivy is a modern cross-platform Python GUI toolkit that runs on mobile devices (Android and IPhone) and supports modern input events (multitouch, accelerometor). I was really skeptical at first, but I spent the weekend developing an app in Kivy, and now I’m hooked.

In the past, I have tried most of the available Python GUI toolkits (even Swing, through jython). In recent history, I’ve mostly done HTML based interfaces, partially because that’s where the money has been, but mostly because pyQT, pyGTK,wxpython, and tk are all painful to work with.

In Kivy, interfaces are defined in a language called (aptly) the Kivy Language. The best way to describe the Kivy Language is “what HTML5 wishes it was.” It is mostly just a layout language, but it also supports binding properties and events using inline Python snippets. It is indentation defined, like Python, and utilizes a minimal amount of syntax with a maximal amount of readability. It is vaguely reminiscent of CoffeeKup but cleaner, and doesn’t suffer from having to be compiled to HTML. I love how it mimics Python syntax, but also clearly separates presentation from control code.

My favourite feature of Kivy is that properties are events. You can easily hook a property on one widget to the property on another. For example, I wrote a simple demo that connects the value of a slider to the value of a progressbar:

BoxLayout:
    orientation: 'vertical'
    ProgressBar:
        id: bar
        value: 14
        max: 20
    Slider:
        id: slider
        max: 200
        value: 140
        on_value: bar.value = self.value / 10

When the value property on the slider changes (whether because the user moved the slider, or because it was adjusted programmatically in response to some other event), the on_value event is fired. This is connected to the value of the progress bar using a Python snippet.

Only a few lines of Python code are required to to get a Kivy application running once its interface has been defined in Kivy language. The Kivy Hello World example is actually overly complicated because it doesn’t actually use the Kivy language. The bare essentials, if the above file is named sample.kv is:

from kivy.app import App
class SampleApp(App): pass
SampleApp().run()

Just three lines of code. Kivy introspects the class name (convention over configuration) and automatically loads sample.kv as the interface for a class named SampleApp.

The Kivy API is very simple. In fact, my earliest impression was that it was almost amateurish. It felt like if I wanted to do anything serious with it, I’d be disappointed. It’s the kind of API I would teach a child. However, the few features of the API (basically properties, events, and graphics instructions) are extremely well thought out and well-defined. They can be combined like Lego bricks to create widgets as complicated as one could possibly wish. The fact that the library includes a ReStructured Text rendering widget is testament to that fact!

Kivy is extremely well documented. I actually found their pong tutorial exciting. Seeing how simple it was to do each step actually made me giggle out loud. The programmer’s guide and API are well presented as well. Further, if you get stuck, the developers on IRC are terrific. I felt like I was an accepted member of the team after asking only one question.

Drawbacks

It took a bit of fiddling to get Kivy to install correctly under Arch Linux. I suppose if I’d followed the instructions, all would have been well. However, I like to do all my development in a virtualenv, and there is no trivial way to track down and install all Kivy’s dependencies. This is especially compounded by the fact that pygame currently needs to be patched under Arch in order to compile.

There are instructions for packaging your Kivy app onto Android, but I wasn’t able to pull it off, yet. I’ll have to revisit it when I have more time.

I tried some sample Kivy apps in the Android market. They work just fine once they are loaded, but it takes each app several seconds to initialize on my Galaxy Nexus. Hopefully the Kivy team can reduce this load time. I don’t know if Kivy supports displaying a splash screen while stuff is loading; that would certainly enhance the usability.

Most of these issues are solvable. I’m sure the team is actively working on some of them. Kivy is a quite young project, but it is remarkably mature.

Kivy Catalog

My weekend project was the Kivy Catalog, an interactive showcase of Kivy widgets. It allows you to view the Kivy language code used to define the widgets and to edit it and see what effects your changes have. I think it will be a tremendous aid to users looking to get started with the Kivy language. Developing it certainly was for me!

I am expecting to have this code included with the Kivy distribution in the examples directory in a future release.

Coming out on Mental Health

In December, 2010, I was admitted to my local psychiatric ward to be treated for depression. I was hours away from suicide. Luckily, my parents took my symptoms seriously and drove two and a half hours to ensure I obtained treatment that I had desperately needed for about two decades.

I have carefully guarded my mental illness from online circles. My career and reputation are built entirely on what I have done on the web. From the essays published on this blog and my books to my github commits and mailing list postings, my internet presence is carefully crafted. I am not ashamed of my mental illness, but because of the stigma against mental health patients, I chose to keep it private from prospective clients, employers, and readers.

It is now time to fight that stigma. Encouraged by publicly mentally ill figures such as Jeph Jacques, and Mathew Good, I’ve decided to place myself as a counter-example to the stereotype, rather than allowing myself to be victimized by it.

For years, I have been successful as a software developer, and more recently as an author in spite of the depression. Now that I have been treated, the effects of my illness have been minimized, and I am even better at what I do. I say this, not to distance myself from the crazy people I met on the psych ward, but so that you will see them as people with a lot of potential, people like me.

I am offended when people claim or imply that depression such as I suffer from is not as “bad” as other forms of mental illness. This allows them to interact with me as a normal person, while marginalizing people who suffer from bipolar disorder, schizophrenia, OCD, or other illnesses that they consider more serious. It’s like they’re saying, “Sure, Dusty, you’re sort of normal. We’ll let you play on our team, but we won’t have anything to do with those freaks.”

Those freaks are my friends. I stand by them. Their illnesses are also treatable and they are just as capable as I am. Further, the implication that depression is not as serious as other disorders is an insult to those other friends that have not yet managed to successfully treat it.

 

My life in the last two years has been incredible. The changes — partially therapeutically and partially chemically induced — in my psyche have been phenomenal. I am now able to enjoy the daily aspects of life. Every day is an adventure, positive and full of hope and meaning.

Sometimes I am terrified to think that in another quantum reality, I died, tragically, almost two years ago. I generally succeed at those things I attempt. If I had attempted suicide, I wouldn’t have survived.

However, I am even more horrified that in this reality, a million people a year turn a highly treatable illness into a terminal one. I was heartbroken last summer when four prominent hockey figures took their own lives. Ilya Zhitomirskiy’s suicide hit particularly close to home. I once had an argument with someone who insisted that these million people, “had a choice.” I know otherwise. When you are that sick, you have no choices. You’ve exhausted them. Death through suicide is no more a choice than death through brain cancer. Both are illnesses in the brain. Both can be treated with varying levels of success. Both are tragic.

Neither were decisions on the part of the deceased.

About one in four of my readers will be affected by mental illness at some point in their life. I am here to tell you that you are not alone. You need not suffer alone. You can be treated, and your life will be amazing in the future. I care about you. You are incredible, you are successful. Take the steps you need to honour yourself, and don’t be too proud to obtain treatment. Mental illness is almost completely treatable. Take the steps you need to before it is too late. Like me, you are capable of enjoying every day. You just need to find it within yourself. You are loved.

Pushing Python Past the Present

This is the first time I’ve inserted myself into an exchange between bloggers I don’t know. This topic interests me and I have something to add. Most importantly, I found an alliterative title. So I thought I’d give it a go.

I first saw Calvin Spealman’s article I am worried about the future of Python. I suspect that upon reading this article, Guido got into his famous time machine to give his Pycon 2012 Keynote speech, accusing Calvin of trolling.

This article was shortly followed by Tim McNamara’s Python is doing just fine which (likely unintentionally) summarizes Guido’s talk.

Finally, Nick Coghlan came out with an extensive summary article called Python’s Future: A Global Perspective.

And now there’s me. All I want to do is push people to start supporting PyPy. The PyPy developers have worked on a variety of technologies that can be made into production-ready products that address most issues (real or imaginary) that people see with Python.

They’ve already solved speed and I suspect the next release of PyPy will solve memory. They’ve made huge but mostly incomplete progress in a lot of other areas, including the ones the above bloggers have mentioned.

Concurrency

I have had a lot of trouble dealing with parallelism in Python. Everything I have tried has either been a hack or required a hack to work with it. In my experience, the multiprocessing module does not work in large-scale production. However, it works a lot better on PyPy than it does on cPython, at least for me. The various async solutions can only use one processor and are therefore, in my opinion, no more exciting than GIL-throttled Threads.

In addition to better multiprocessing, PyPy also supports stackless extensions out of the box. And I haven’t even mentioned Armin Rigo’s mysterious Software Transactional Memory implementation.

Concurrency is a sore point in Python. There are solutions. PyPy is capable of doing those solutions better.

Web

I think it’s unfortunate that the only way to code for a web browser is to use JavaScript or a language that compiles to JavaScript. I feel there is no real reason browsers can’t support arbitrary <script type=""> tags. I discussed this last March.

With development, PyPy can support Python in the browser. The mostly-finished but untested sandboxing feature of PyPy can be adapted for in-browser execution. I know some experimenting was done on this in 2009 that proved it’s possible. We just need to take it up and make it happen.

Mobile

PyPy’s speed makes it theoretically possible to run it effectively on mobile platforms. Further, it can be compiled to arbitrary execution environments including JVM and .Net. There is no reason that with some development, PyPy couldn’t run on the Dalvik JVM. I’m sure it’s even possible to run it on iOS.

Interestingly, one of the trolls Guido mentioned in his talk was “PyPy should become the default Python implementation”. He debunked this readily by asking the audience how many people use PyPy in production. Nobody moved.

PyPy is ready for production use, but it is not widely used. I think this is largely because the primary PyPy developers are way more excited about creating new features than they are about marketing the current platform or polishing up nearly finished features like sandboxing or a Dalvik backend. They are visionaries, not finishers.

I say this with a great deal of respect. I am not calling for these guys to change their focus. What I want is for other people from the Python community to join the PyPy project as finishers. It needs people that can make sure these nearly-finished features are working, production-ready and more importantly: documented.

I’ve been meaning, for months, to become one of these people. Unfortunately, I’ve prioritized other things including my job and my upcoming book. It’s still on my radar, though, and I hope that after reading this article, you, too, are thinking about make PyPy the future of Python.

So, what can you do?

Use PyPy
PyPy is capable of being your default Python 2 interpreter for most common tasks. Use it. If it doesn’t work for a given project, get on #pypy, they will help you fix it. It’s even more exciting if you can use PyPy features that are not currently available in cPython, such as the stackless extensions.
Evangelize PyPy
Tell people how well PyPy is working for you. Write articles like this one.
Document PyPy
The PyPy website contains a lot of documentation, but it’s rather intimidating and unreadable to the uninitiated. PyPy is just Python, but it’s also much more than Python. Somebody with some writing skills needs to get in there and make it digestable.
Design PyPy
Seriously, pypy.org does not need to look like something out of 2005.
Develop PyPy
I’ve hacked on PyPy. It’s not scary. Work on features like numpy or Python 3 support that are the current developer’s focus. Better yet, work on finalizing features like sandboxing or alternative backends that are finished but not quite tested.

If you have an hour or two free this weekend, instead of writing about how Python is great or not great, or not going to continue to be great, do something with PyPy. Make it happen.

How (not) to screw up timezone processing in Python

If you use pytz without reading the documentation, you might think you can do this:

cltime = datetime.datetime(
        2012,9,26,1,15,0,
        tzinfo=pytz.timezone('America/Santiago'))

This is an easy way to utterly screw up your timezones.

Never pass a tzinfo into the datetime constructor. Here’s why:

>>> cltime = datetime.datetime(
        2012,9,26,1,15,0,
        tzinfo=pytz.timezone('America/Santiago'))
>>> cltime.astimezone(pytz.timezone('US/Pacific')).isoformat()
'2012-09-25T22:58:00-07:00'

That’s simply the wrong time. 1:15 Chilean time should be 21:15 Pacific the previous day, not 22:58.

For whatever reason, the wrong tzinfo is attached to the object. Compare:

>>> cltime.tzinfo
<DstTzInfo 'America/Santiago' SMT-1 day, 19:17:00 STD>
>>> datetime.datetime.now(pytz.timezone('America/Santiago')).tzinfo
<DstTzInfo 'America/Santiago' CLST-1 day, 21:00:00 DST>

The correct way is to use pytz.localize():

>>> correct_cltime = pytz.timezone('America/Santiago'
        ).localize(datetime.datetime(2012,9,26,1,15,0))
>>> correct_cltime.astimezone(pytz.timezone('US/Pacific')).isoformat()
'2012-09-25T21:15:00-07:00'
>>> correct_cltime.tzinfo
<DstTzInfo 'America/Santiago' CLST-1 day, 21:00:00 DST>

The pytz documentation does not indicate if datetime.datetime.now() creates the correct timezone, so I tested it:

>>> for tz in pytz.all_timezones:
...     assert datetime.datetime.now(pytz.timezone(tz)
...         ).tzinfo == pytz.UTC.localize(datetime.datetime.utcnow()
...         ).astimezone(pytz.timezone(tz)).tzinfo
...     
>>>

The assertion never failed, so it is safe to use datetime.datetime.now(pytz.timezone("[timezonename]")) to generate the current date in a specific timezone.

In general, you should store dates in UTC format and convert them to the user’s timezone at the latest possible time. Try to never store naive datetimes. if you need the current time in UTC, use

pytz.UTC.localize(datetime.datetime.utcnow())

because utcnow() returns a naive datetime. Localizing it to UTC adds the correct timezone to it.

pytz.UTC.localize(datetime.datetime(2012,9,26,1,15))

is the correct way to construct a datetime in the UTC timezone. Technically, you CAN pass tzinfo to the datetime constructor if tzinfo is UTC. This is because UTC does not have daylight savings time. However, for consistency, do not do this; do not ever pass tzinfo into a datetime constructor.

Converting to or from UTC is simple, using astimezone:

>>> utctime = cltime.astimezone(pytz.UTC)
>>> utctime
datetime.datetime(2012, 9, 26, 4, 15, tzinfo=<UTC>)
>>> utctime.astimezone(pytz.timezone('America/Santiago'))
datetime.datetime(2012, 9, 26, 1, 15, tzinfo=<DstTzInfo 'America/Santiago' CLST-1 day, 21:00:00 DST>)

Arch Linux Handbook for Kindle

I have had a handful of requests that the Arch Linux Handbook be made available for the Kindle platform. It seemed like an odd request, given that the latest version of the Beginners’ Guide is already freely available in electronic format online. However, I had some free time this week and tried the conversion. It wasn’t difficult and I uploaded a version of the Handbook to the Kindle app store. I’ve helped publish other books to Kindle, so I already knew the process.

I received an e-mail from Amazon Customer Service to the following effect:

During a review of your KDP submission(s), we found content that is freely available on the web. You can do an online search for the content inside your book(s) to discover which sites are offering the content for free. Copyright is important to us – we want to make sure that no author or other copyright holder has their work claimed and sold by anyone else.

To confirm you have publishing rights to and control where you distribute the book(s), please provide all of the following information:

1. The URLs for all websites where this content is published
2. An explanation as to why the content is available online

If the books are in the public domain, please confirm this and include the information you used to make this determination. We may request additional information to confirm the public domain status.

Please respond within 5 days to title-submission@amazon.com, and include the title and ID of your books in your reply. Your book has been moved to a blocked status on your bookshelf and will not be available for sale in the Kindle store until we receive the documentation requested.

Sure, no problem. I responded to explain that it was freely redistributable under the Free Documentation License:

This content is indeed freely available on the web at https://wiki.archlinux.org/index.php/Beginners%27_Guide although I have done a certain amount of editing to get it into its current format.

However, this freely available content is published under the GNU Free Documentation License 1.3 or later. (http://www.gnu.org/copyleft/fdl.html) which explicitly states:

“The purpose of this License is to make a manual, textbook, or other functional and useful document “free” in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.”

Their response makes me sound like some kind of criminal:

Hello,

We’ve reviewed the information you provided and have decided to block these books from being sold in the Kindle Store. The books closely match content that is freely available on the web and we are not confident that you hold exclusive publishing rights. This type of content can create a poor customer experience, and is not accepted. As a result, we have blocked the books listed below from being sold in the Kindle Store.

Arch Linux Handbook 3.0 by Phillips, Dusty (EDITOR) (ID: 2884216)

Please be advised that you must hold exclusive publishing rights for books that closely match content that is freely available on the web. If your catalog continues to contain books that fail to comply with these conditions or do not meet our Content Guidelines, your account may be terminated.

The content guidelines applicable to all Kindle Direct Publishing (KDP) publishers can be found here: https://kdp.amazon.com/self-publishing/help?topicId=A2TOZW0SV7IR1U

Best regards,

Megan B.
Amazon.com
Your feedback is helping us build Earth’s Most Customer-Centric Company.

I was insulted and hurt by this message. It’s arrogant and it is confrontational. It makes no effort to address the specifics of the e-mail. But I cheered up at the end, or at least I laughed, “Customer-Centric Company”? Don’t advertise what you aren’t.

The original message made no mention of the fact that I should have “exclusive” publishing rights. It even said public domain work was acceptable. I have published Creative Commons books that are freely available online on the Kindle before (although they are a non-commercial creative commons license, so we still have “exclusive” rights to publish on Kindle).

In other news, the Arch Linux Handbook can be downloaded in .mobi format, free of charge, from http://archlinux.ca/arch_linux_handbook_3.mobi.

UPDATE:

After this article unexpectedly hit Reddit and Hacker News (my target audience was the Arch Linux Community), Amazon let the title go through and it is now available from http://www.amazon.com/dp/B0098R71V2. They have been in contact with me but have not been able to explain why the book has been allowed to be published! The best explanation I got was:

We’ve re-reviewed your content and have determined it may be published to the Kindle Store. We generally can’t accept content that closely matches content that is freely available on the web, for which you do not hold the sole publishing rights, or that which is not in the public domain. For example, content from Wikipedia and content with private label rights are not allowed since it disappoints our customers to pay for content that is freely available on the web.

Since the Arch Linux Beginners’ Guide is a wiki-developed article no different from Wikipedia, I’m not certain why the Handbook is now allowed.

I originally posted this article to explain why the handbook is not available in the Kindle store. My intent was not to complain about Amazon’s policies (although I was not happy with the suggestion that I am prone to criminal activity). The new purpose is to mention it’s availability through the store, although you can still download my quick one-off conversion if it suits you as well. In all honesty, I can’t imagine why anyone would want to purchase something that is already freely available in electronic format, but at least now that choice is open.

Copyresponsibility

There has been a great deal of discussion about copyrights in the new world, where the ability to copy or distribute any piece of information without destroying the original is virtually infinite.

Much effort has been put into preserving the “old model”, where the right to copy is restricted based on the author (or more often, the publisher/distributor), in the form of DRM, civil lawsuits, new laws, and intellectual property. Overall, this effort has failed, and continues to fail. The laws are becoming more complex, but enforcement is still impossible.

We live in an exciting time. Society has not yet decided whether copying information is moral or not. We already know what a world in which copying data is immoral is like — the “old model”. But what would happen if a world where it was both accepted and expected that anyone who has access to a piece of information has the legal and moral right to redistribute it?

Personally, I think we must plan for such a world, because it seems inevitable. Perhaps the various organizations attempting to hold onto the old model will succeed, but not forever. I think coming generations will demand the right to access and distribute information as they see fit.

Ultimately, this would mean that anyone who has access to (“read”) a piece of information has an equal right to share that information as the author. We already see models where this is the case; very few people know or care who the authors of Wikipedia articles are, for example. When a YouTube video goes viral, you rarely look to see who uploaded it. Even if you do, you don’t know if they actually created the original.

I think this will place a great deal of responsibility on people who have privileged knowledge. The author of a document has exclusive right to its content until she shares it with another person. At that point, those two people would have an equal share in the right to the content. This is implicit and obvious, but something that has not been discussed much to date is that they also have an equal share in the responsibility of deciding who else may view that document.

If every single person who has access to the information agrees that no-one else should see it, then the information is private to that group. However, if the weakest link chooses to share it with outsiders, nobody else in the group would have the right to stop them.

Thus, in the copy-responsibility world, a huge amount of trust is required before sharing information that you consider sensitive. I know people who claim, “if it’s on the Internet then it’s public”, even if it’s inside a private Facebook group or Google Plus circle. This claim simply acknowledges that every member of that circle has an equal right to share that data, or equal responsibility to keep it within the cartel. (They are overlooking the fact that employees at Facebook and Google are also implicit members of that circle that may not be so reliable.)

In summary, I think it’s fair to say we live in (or will soon live in) a world where we no longer have a right to privacy. Instead, we will have a responsibility to privacy. We are responsible for our own “secrets” as well as the secrets of anyone who has shared their secrets with us.

Arch Linux Handbook 3.0

After some unexpected delays, I am proud to present the latest version of the Arch Linux Handbook: https://www.createspace.com/3904652. This Handbook is based off the Arch Linux Beginners’ Guide as it was on June 10, 2012. Due to complaints with previous editions, I put extra effort into the interior of this version, removing the styling on things that look like links to other wiki pages, and ensuring that such links at least give a hint as to what you should search for on the Arch Wiki for further information.

Even so, the handbook is little more than a printout of the Beginners’ Guide. It does not contain any new information aside from a short forward I authored. However, if you are looking for a paper copy of the Beginners’ Guide in a convenient handbook size, this book is exactly what you are looking for.

As with version 2.0 of the handbook, the cover design for version 3.0 was done by Branko Vukelic, a gifted artist, designer, and web developer.