1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

#!/bin/bash

set -x

# Update the master branch.
git checkout master
git pull upstream master

# Synchronize the remote repository.
git push origin master

# Find all merged branches and delete them in the local and remote
# repositories.
for branch in `git branch --merged | grep -v '*' | tr -d ' '`
do
    git branch -d $branch
    git push origin :$branch
done

It’s available in this gist. You can wire it up as a cleanbranches command by adding the following to your ~/.gitconfig file:

1
2

[alias]
    cleanbranches = !/home/jkakar/bin/gitcleanbranches.sh

Now you can run git cleanbranches in any repository and local and remote branches that have already been merged will be deleted automatically. It’s a simple thing, but one of those simple things that makes a nice difference in the daily workflow.

Some time ago during a sprint, I noticed that Free Ekanayaka, one of my Landscape teammates, had excellent pyflakes/flymake integration that dramatically improved the experience of editing Python files in emacs. When I asked him about it he kindly offered to share his configuration and I was excited to pick out the flymake settings he had. When I actually looked at what he’d sent me I was blown away. He has the most elegant emacs configuration I’ve ever seen. He uses a simple pattern composed of individual configuration files, each focused on a particular set of customizations such as appearance, interaction, programming, etc. that are all wired up to create the final result. It’s very clean and avoids the blob-of-crap problem that most emacs configuration suffers from.

This evening I finally took his ideas and refactored them to match my emacs setup. The structure is the same, but the details are, in some cases, quite different. I’m managing the configuration with Bazaar and have pushed it to Launchpad at lp:~jkakar/+junk/emacs. Hopefully others will find this as inspiring as I have.

I’ve been working on the Landscape Team since early 2006. I was one of the founding members of the project along with Gustavo and Chris. Landscape is a web-based tool that eases the task of managing large deployments of Ubuntu. It is one of the many services that Canonical provides, as a value add to Ubuntu, and is proprietary software. At this point you’re probably thinking, “How can proprietary software help the goal of spreading free software? That’s crazy talk.” Well, in at least two ways.

Landscape makes it possible for people that could never have chosen Ubuntu to consider it, and in many cases to adopt it. Our enterprise customers look at Ubuntu and they like what it offers, but that’s only the beginning of the story. Management is a big concern for organizations that roll out tens, hundreds or thousands of Ubuntu machines. Without a management solution, Ubuntu is a non-starter no matter how good the user experience.

Landscape is a revenue generator for Canonical and, though we’d all love to release it as free software, that revenue is very important. It contributes to the sustainability of the company and thus, to the sustainability of Ubuntu itself. We do as much work as we can in the open. For example, Storm, an object relational mapper for Python, was developed by the Landscape Team and released as free software. We’ve sent patches to projects such as Twisted, txAWS and others.

Although my work may be controversial in the wider free software community I’m a free software developer at heart. The effort I put into Landscape is good for free software in general. It helps Ubuntu gain adoption in places where it simply wouldn’t be considered without a tool like Landscape. Canonical has been, and continues to be, an amazing place to work. It is full of passionate people all doing their part to get free software out into the world.

We’re always interested in hearing about your experiences with Storm. If you have comments or questions please feel free to hop into #storm on Freenode and talk to us, or post a message on the mailing list.

I couldn’t find the answers I wanted in the documentation so I wrote a simple tool to invoke a method with an arbitrary set of parameters against an EC2 API endpoint. After sending the request, it prints the HTTP status code and response to the screen. It’s called txaws-discover and is part of the txAWS project. For example, to run the DescribeRegions method simply provide credentials, an endpoint, the method name and whatever parameters you want to pass with it:

1
2
3
4
5

txaws-discover --key NUJOZS2V6RWQUJ3G8JUT \
    --secret +Ki9Wk5Y4kudFh1EcnB3hthC9PtVU+CcfMJZ4DVl \
    --endpoint https://ec2.us-east-1.amazonaws.com \
    --action DescribeRegions \
    --RegionName.0 us-west-1

This command produces the following output:

1
2
3
4
5
6
7
8
9
10
11
12

HTTP status code: 200


<DescribeRegionsResponse xmlns="http://ec2.amazonaws.com/doc/2008-12-01/">
    <requestId>1e9beacc-0044-4ad6-bd2c-c615351ae46d</requestId>
    <regionInfo>
        <item>
            <regionName>us-west-1</regionName>
            <regionEndpoint>ec2.us-west-1.amazonaws.com</regionEndpoint>
        </item>
    </regionInfo>
</DescribeRegionsResponse>

The credentials and API endpoint can be defined in environment variables to make txaws-discover easier to use:

1
2
3

export AWS_ACCESS_KEY_ID=NUJOZS2V6RWQUJ3G8JUT
export AWS_SECRET_ACCESS_KEY=+Ki9Wk5Y4kudFh1EcnB3hthC9PtVU+CcfMJZ4DVl
export AWS_ENDPOINT=https://ec2.us-east-1.amazonaws.com

With those defined, the command above can be shortened:

1

txaws-discover --action DescribeRegions --RegionName.0 us-west-1

It’s in lp:txaws and has proven to be very helpful as a learning tool.

1

result = store.find(Account)

Clauses to limit the scope of the query can be passed to find. Finding all accounts owned by Vince Offer, the equivalent of SELECT * FROM account WHERE owner = ‘Vince Offer', is also simple:

1

result = store.find(Account, Account.owner == "Vince Offer")

Another way to achieve the same thing is to take the result from the first find operation and refine it:

1
2

result1 = store.find(Account)
result2 = result1.find(Account.owner == "Vince Offer")

Storm doesn’t run a query until you do something with the result, so the impact on the database is exactly the same as in the previous example. This is the simplest form of what Jonathan and I have been calling the collection pattern. The basic idea is that you start with a collection of all objects, and then refine it, until you have the collection you want. This pattern is possible using pure Storm, as shown above, but we’ve found it useful to implement objects that hide this logic and provide a more user friendly API. For example, an AccountCollection would provide filtering logic as named methods:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

class AccountCollection(object):

    def owned_by(self, salesman):
        """
        Get a new collection with accounts owned by
        salesman.
        """

    def has_unpaid_invoices(self):
        """
        Get a new collection with accounts that have
        unpaid invoices.
        """

    def find(self):
        """Get a result set of matching accounts."""

Each filtering method, such as owned_by and has_unpaid_invoices returns a new AccountCollection instance. This isn’t strictly necessary but creating a new instance is both easier to understand and implement. Finding all accounts owned by Vince Offer that have unpaid invoices is quite easy:

1
2
3
4

collection1 = AccountCollection()
collection2 = collection1.owned_by("Vince Offer")
collection3 = collection2.has_unpaid_invoices()
result = collection3.find()

This pattern provides several benefits:

• It names filtering options which makes them easy to understand and use.

• Query building logic is in one place which eliminates duplication.

• All the criteria used to build a query are available when the query is generated, which makes it possible to generate optimized queries. All users of the collection benefit from optimizations.

• It creates a clean separation between finding data and using it. This helps keep application logic more focused, because it isn’t intermingled with the particulars of generating queries.

We’ve used this pattern to good effect in both Landscape and Launchpad. In a future post I’ll talk about some different strategies we’ve used to implement the pattern.

• You only have to know that the program is called bzr to start discovering it, because its default behaviour is to teach you how to use the builtin help system.
• Each command has a clear name that makes its purpose easy to discern.
• Each command has a succinct summary and a well written long description, along with clear information about the arguments and options the command accepts.
• Help topics provide general information about the program that isn’t command-specific. The topics describe conceptual and technical aspects of the program that help the user better understand how to use it.
• The interface conventions are very simple. It doesn’t take long to figure out how things work.

I wanted, as much as possible, to start writing application logic without having to do much work to get a working user interface. With that in mind, the basic goal for Commandant is to provide a Bazaar-like user interface with the least effort possible. I started by trying to extend Bazaar but I ran into tricky integration problems that made it hard to make progress, so I switched gears and started implementing the logic from scratch. Reinventing the wheel got me further than trying to integrate, but I quickly reached a point where progress slowed down because I had to deal with tricky problems that Bazaar had already solved. In the meantime, Robert landed changes in Bazaar that removed the integration hurdles that I’d run into.

With the integration problems out of the way, I stopped wasting time reinventing the wheel, and Commandant is now a thin layer on top of bzrlib itself. Being able to take advantage of the work that has been done in Bazaar has been a blessing. There’s more that can be done to improve Commandant, but it is very usable in its current state. If you’re interested in learning more, the README file should get you started.

First, some advice: if you don’t know what SQL query you want to run, using Storm will probably be hard. When I get asked a question about writing a query using Storm I almost always respond with, “What does the query you want to run look like?” The querying part of Storm is essentially just an expression compiler that takes a list of expressions and converts them to SQL. Once you know what end result you want, working backwards to figure out the Storm parlance is relatively straight forward.

Today I was asked about using COALESCE. The person asking the question found the storm.expr.Coalesce expression but it wasn’t obvious how to use it. He knew he wanted to run a query like:

1

SELECT * FROM table WHERE COALESCE(col1, col2) = 'foo'

With that knowledge, figuring out how to use Storm was quite easy:

1

store.find(Table, Coalesce(Table.col1, Table.col2) == "foo")

Furthermore, Coalesce is just a helper expression to aid usability and readability. If we didn’t have a built-in Coalesce expression the query could have been written using the generic storm.expr.Func expression:

1

store.find(Table, Func("COALESCE", Table.col1, Table.col2) == "foo")

If you want to use a SQL function in a Storm query, and there is no built-in expression for it, you can use Func to call it.

In retrospect, I should have come up with much better example code. I didn’t make enough time to prepare and, due to rushing at the end, the topics I covered were rather basic. I did an experiment and wrote a psuedo-doctest to use as slides. The doctest and a script to run it are available at lp:~jkakar/+junk/storm-talk. I used a slightly hacked version of Michael Hudson-Doyle’s very awesome console-presenter, which is available at lp:~jkakar/console-presenter/two-line-separator.

The format was great for getting me to focus on content, even though I didn’t have enough time to do a great job preparing it, but the slides were a bit lackluster being grey text on a black background. Also, a common problem with doctests is that you need to keep track of the state of the program as you read the document. I think this problem made the format awkward for the audience. It was a good experiment, but in the future I’ll use traditional slides, and try to avoid a situation that requires the audience to keep program state in their head as the talk progresses. Funny how these things seem so obvious in hindsight.

Doug Latornell gave a nice talk about his experiences using PyYAML, flickerapi and Tkinter to build an application that displays random photos on his living room TV, both from his personal collection and from his favourite groups on Flickr. I enjoyed the talk and the discussion it spurred.

All in all it was a good experience and from which I’ve learnt some lessons, which I can hopefully use to make the next talk better.

Unfortunately, the karmic and lucid packages don’t work properly because they install files in /usr/lib/python2.6/site-packages, which is no longer included in sys.path by default. In the next couple of days I’ll build new packages that correctly install files in /usr/lib/python2.6/dist-packages, which should fix the problem. For now, adding the site-packages path to sys.path or including it in PYTHONPATH will workaround the issue.

We’re always interested in hearing about your experiences with Storm. If you have comments or questions please feel free to hop into #storm on Freenode and chat us up, or post a message to the mailing list.