What I Learned from the apache.org Break-in

Here’s my initial take-away after reading about the recent apache.org XSS-initiated break-in:

  • If you’re going to allow caching of credentials (Subversion or otherwise) on a server, don’t use an account that shares credentials with any superuser account. Personally, I can’t think of a good reason for these credentials to be cached in the first place (except on a development machine). As an aside, by default, Mercurial doesn’t do this; I suppose the fact that every ‘svn commit’ is also a push makes this more “necessary” with Subversion.
  • If you have an organization-wide login (say a Windows login that is automatically sync’ed with Subversion, your enterprise RDBMS, and who knows what else[1]), if at all possible use a different  password on any server where you’ve got superuser access.
  • All superuser accounts on servers should have different passwords; at a minimum, if you use a common password for superuser accounts across servers, don’t use this password for other accounts.
  • Use Trac instead of Jira. [2]

That’s a bare minimum; I’m still thinking about how vulnerable the organization I work for might be. Most of this probably seems obvious, but I’m betting that these and other less-than-best practices are extremely common.

Ref: https://blogs.apache.org/infra/entry/apache_org_04_09_2010

[1] This is a purely hypothetical scenario. ;)

[2] Just kidding?

Eclipse for Python Web Development on Ubuntu

Updated 14 July 2009 for Eclipse 3.5. Now using Aptana to install PyDev.

Updated 5 March 2010 for Eclipse 3.5.2 and Subversion 1.6.

Updated 20 July 2010 for Eclipse 3.6

Updated 1 August 2011 for Eclipse 3.7 and Aptana Studio 3


So, I think I’ve finally decided that I prefer Eclipse for Python Web development over NetBeans. (I prefer Wing IDE over both for straight Python development, but that’s another post.) Eclipse’s Python support, via PyDev, seems more advanced and NetBeans has some annoying issues. Eclipse also seems a bit snappier, at least on my machine (YMMV, blah blah).

Eclipse takes a bit more effort to set up, but once you’ve done it a couple times, it’s pretty straightforward. The hard part is keeping track of the links to the Eclipse update sites and remembering a few odd bits of configuration. This document gives details on installing Eclipse for Python Web development with Aptana, PyDev, and Subclipse.

The instructions here are Linux/Ubuntu-centric, but the instructions for getting to the Platform Binary–the smallest possible Eclipse download, as far as I can tell–are applicable to all platforms.

Download Eclipse

If you go to the Eclipse downloads page, you’ll see packages for Java and C++ along with some other options. If you’re doing ONLY Python development, you might wonder which version to download. These versions install cruft I don’t want or need. I finally found what I think is the smallest possible Eclipse package, the so-called Platform Runtime Library.

Install Eclipse

Extract the downloaded package: tar xvzf eclipse-platform-3.7-linux-gtk.tar.gz. I rename the resulting eclipse directory to eclipse-3.6, move it to ~/.local, and create a symlink in ~/.local/bin to the eclipse executable: ln -s ~/.local/eclipse-3.7/eclipse ~/.local/bin/eclipse

Configure Eclipse

Edit ~/.local/eclipse-3.7/eclipse.ini. Find the line containing “-vmargs”. Add a new line directly below that: -Djava.library.path=/usr/lib/jni. Save and close. Fire up Eclipse.

Install Aptana Studio

Install PyDev

PyDev is installed as part of the Apatana Studio install.

Install MercurialEclipse

Eclipse update URL: http://cbes.javaforge.com/update

Supervisor now supports Python 2.6

I’ve been using Supervisor for a few months now to manage some app servers at work. Good stuff; solid. My only gripe with it was that it didn’t run properly under Python 2.6[1,2]. I had to install Python 2.5 just to run Supervisor, which isn’t a super big deal but is kind of annoying. The latest release, 3.0a7, fixes this by including a “patched version of Medusa to allow Supervisor to run on Python 2.6.”

Note: I had to `wget http://dist.supervisord.org/supervisor-3.0a7.tar.gz` and easy_install that due to a network timeout[3]. I assume that’s because Supervisor is super popular and everyone’s upgrading.

[1] It did run under 2.6 but issued an error when starting up.
[2] This was due to changes in the Python 2.6 stdlib, not a problem with Supervisor itself.
[3] Suggestion: If possible, remove link from PyPI to http://supervisord.org/ so easy_install doesn’t get stuck there.

Implementation of Dijkstra’s Single-Source Shortest-Paths in JavaScript

I’m working on a project where the client wants a cool sliding navigation effect. We’re implementing this with JavaScript/AJAX/DHTML.

One of the constraints is that pages can only be reached via certain other pages. For example, if you’re on the /portland/contact page and want to go to the /seattle/contact page, you’ll first slide up to /portland, then over to /seattle, then finally down to /seattle/contact.

After a while, it occurred to me that there were some similarities with another project I’ve been working on off and on for the last few years, byCycle.org, which is a bicycle trip planner a la Google Maps.

I had written a Python version of Dijkstra’s Single-Source Shortest-Paths (SSSP) for byCycle.org. That’s available on Bitbucket as Dijkstar (so named because it also does has the potential to do A*). I figured it wouldn’t be too hard to port the Python version to JavaScript, and it wasn’t.

There were a few snags, though. Most of it was just syntactic and semantic differences between the two languages. The biggest issue was that I use “heapq“ in the Python version to maintain the costs to previously visited nodes in sorted order. JavaScript has no priority queue implementation that I could find, so I came up with a different solution that involves updating an Object (AKA hash) with costs to newly visited nodes and sorting the keys to pick the next node to visit. I’m assuming/hoping the underlying sort implementation is highly optimized.

Interestingly, I think I found at least one bug in the Python version, although I’ve been using that version for a couple years now with no known problems, so it must only be applicable in certain edge (no pun intended) cases (or maybe it’s due to some difference in the languages–need to take a closer look). I think the JS version came out cleaner, too.

If anyone’s interested, I’m releasing this under an MIT license. You can get it from here. Note that it depends on the util module that you can get from here. The util module contains some other Python-inspired JavaScript, in particular a couple of functions for generating namespaces and classes. I might write another post about that at some point.

YUI TreeView with Ruby on Rails

Here’s some code I’m using to generate a dynamic tree view using an acts_as_tree model with slug and title fields, the TreeView widget from YUI, and a Rails helper. I chopped out some of the code for clarity, so all this does is create a menu with the titles from the model, but the basic idea is there to expand on.

Rails view/JavaScript

<script type="text/javascript">
  var page_tree;
  page_tree_init = function () {
    page_tree = new YAHOO.widget.TreeView('page_tree');
    var root = page_tree.getRoot();
    <% generate_page_nodes(@root) {} %>


def generate_page_nodes(node, &block)
  parent = node.parent
  node_name = node.slug.gsub('-', '_')
  parent_node_name = parent.nil? ? 'root' : parent.slug.gsub('-', '_')
  js = <<-JS
    var #{node_name} = new YAHOO.widget.MenuNode('<span class="node_title">#{node.title}</span>',
  concat(js, block.binding)
  children = node.children
  children.each { |c| generate_page_nodes(c, &block) } unless children.empty?

Ruby on Rails… Revisited

Updated with links and a couple typo corrections.

Update: It wasn’t long before the project got too complex on the back end (SOAP blech) for my limited Ruby knowledge. I switched it back to Python/Pylons and never looked back. The Pylons => Rails migration was straightforward. I guess I could have pushed through with Ruby/Rails, but with deadlines looming, it made more sense for me to go with what I knew best. Being familiar Python and its ecosystem was far more pertinent than the deficiency of any particular library. There’s probably another blog post or two in here…

I’ve been working on a fairly big Web site project lately. My partner and I initially decided to use Django to build the site, mainly because I’m a Python “expert” and Django is (apparently) the #1 Python Web framework. We were also lured by the easy admin interface.

After trying to use Django and not really enjoying it, I tried switching to Pylons because I’ve had a good amount of experience with it in the building of byCycle.org. It’s gone through two fairly major releases since then, and so have a bunch of the libraries that tend to get used with it, like SQLAlchemy, Elixir, etc.

I was having a hard time with the Pylons docs, and so I ended screwing around with Grok (which actually looks fairly interesting) and even took a look at the Zope 3 site. I’m sure Zope is really awesome or whatever, but it might as well suck. Every time I look at that site, I’m just like “WTF! This shit has been around for like five years!” Anyway, I might just not be smart enough for Zope.

This led us back toward Rails (even if it is a ghetto). I used Rails a bit last year but never did anything too serious with it. Diving into it today was quite a pleasure. There are issues to be sure, but overall I’m enjoying it by far over any of the other options we had tried. I’m also enjoying learning/relearning Ruby.

If Pylons had good docs, we’d probably be using that.

So, I don’t know if this is a particularly useful post, since I didn’t get into much in the way of reasons (what, i have back this up?!). This subject’s been hashed and rehashed, but I just wanted (needed) to make a qualitative statement about my/our experience, which, of course, is purely personal.

Erlang Linked List Exercise

Yesterday, my copy of Programming Erlang arrived in the mail1. w00t! I’m already part way through chapter three. I don’t know what it is about this language – maybe it’s all hype and a passing fad – but I haven’t been this interested in learning a new language since I started with Python over two and half years ago.

The day before yesterday, I took a shot at implementing a linked list in Erlang. I had one basic rule, which was that I wasn’t allowed to use the built in list type. Getting started was fairly difficult, but once I started to “get it” (e.g., pattern matching, recursion), the task got much easier.

Previously, I had only played around in the shell, so this is the first Erlang module I’ve written. Erlang modules are similar to Python modules, though to make functions available outside an Erlang module, they must be explicitly exported. I haven’t yet come across whether Erlang modules can be organized into packages, although I imagine there must be some kind of higher level system for organizing Erlang programs.

Recently, I did a similar exercise with Python as way to experiment with Behavior Driven Development. That version uses the familiar “destructive assignment” operation throughout. Erlang allows single assignment only, so I had to think about the problem in a different way. For example, an item can’t be appended to a list by manipulating a couple of object references as in Python—instead I used recursion to build up a new list.

After I get further into the book, it will be fun to come back to this implementation and see how it can be improved given a better understanding of the language. I’m also looking forward to exploring Concurrency Oriented Programming in depth. With Python, I tend to not think in terms of concurrency, though I’m sure I’ve got code that could be improved by using it.

Finally, here’s the code. It was written in Emacs, which has a nice Erlang mode. There are two other IDEs available, one based on Eclipse and the other on NetBeans.


-record(list, {head, length=0}).
-record(item, {data, next}).

new() ->

new_item(Data) ->

head(List) ->


tail(List) when List == #list{} ->
tail(List) ->
    Length = linkedlist:length(List) - 1,
    #list{head=next(head(List)), length=Length}.

append(Data, List) when List == #list{} ->
    List#list{head=new_item(Data), length=1};
append(Data, List) ->
    Item = append_item(Data, head(List)),
    NewLength = linkedlist:length(List) + 1,
    List#list{head=Item, length=NewLength}.

append_item(Data, Item) when Item#item.next == undefined ->
append_item(Data, Item) ->

    Item#item{next=f(Data, next(Item))}.

next(Item) ->

% Get the Nth item from List
% N: Index of item to get
% List: List to get from
nth(N, List) when N < 1 ->
nth(N, List) ->
    nth(N, 1, head(List)).

% N: Index of item to get
% I: Current index
% Item: #item in List corresponding to index I
nth(N, I, Item) when I == N ->
nth(N, I, Item) ->

    nth(N, I + 1, next(Item)).

last(List) ->
    nth(linkedlist:length(List), List).

length(List) ->

is_empty(List) ->
    List == new().

p() ->
p(Object) ->

main() ->

    L = new(),
    test_list(L, "New list"),
    L1 = append(data1, L),
    test_list(L1, "List with one item"),
    L2 = append(data2, L1),
    test_list(L2, "List with two items"),
    L3 = append(data3, L2),
    test_list(L3, "List with three items"),

test_list(List, Description) ->
    io:format("~s~n", [Description]),
    Length = linkedlist:length(List),
    p({'list', List}),
    p({'head', head(List)}),
    p({'tail', tail(List)}),
    p({'first', nth(1, List)}),
    p({'nth', nth(Length, List)}),
    p({'last', last(List)}),
    p({'length', Length}),
    p({'is_empty', is_empty(List)}),

1 A new blade and O-ring for my old-fashioned Oster blender came also. Double w00t!

Erlang for Python Programmers?

I’ve been hearing about Erlang for several months now. It has seemed interesting, but up until recently I hadn’t felt any great need to look into it.

Last week, however, my interest was piqued, and I spent much of the weekend reading about Erlang and playing around in its interactive shell. That continued into this week when I decided to buy the new book.

While I’ve been sitting around waiting for the book to show up, I’ve been wondering if there’s an Erlang for Python Programmers tutorial floating around anywhere (I haven’t come across one yet). I’m thinking I might try to put something like that together. I think that will be an interesting challenge as there’s not necessarily an obvious one-to-one correspondence between the two languages.

Here’s how I initially learned about and got interested in Erlang:

These things made it easier to get beyond the yeah-that-looks-neat stage and into the ok-what-the-hell-I’ll-give-it-a-try stage:

  • It was easy to install with apt-get on Ubuntu (`sudo apt-get install erlang`)
  • It has an interactive shell (`erl`)
  • There’s good introductory documentation at erlang.org

Here’s what’s keeping me interested:

  • It’s a functional language, which makes it more interesting in some ways than learning, say, Ruby—in fact, I wouldn’t really need to learn much to start using Ruby, whereas using Erlang will require new ways of thinking about program structure (in terms of functions and processes)
  • I like the term “Concurrency Oriented Programming”
  • It’s been used to build some massive real-world systems
  • There’s a Web framework [link died]
  • It has a planet Planet Erlang [link died]
  • There’s a recent book by one of the original creators of the language
  • It seems to be hot amongst alpha geeks, and I’m a wannabe alpha geek

See erlang.org for pointers to more info.

Erlang is the first functional language I’ve really gotten into (for whatever reason), so regardless of whether I ever end up using it in the real world, the learning experience will be valuable.

Fun With Python BDD

I was testing my algorithm/data structures chops by implementing a Linked List from scratch using no reference material (not very difficult, I know), and I must say that using BDD was incredibly helpful. I’m calling it “BDD” instead of “TDD” just because the test class names describe a context and the method names in the test classes are behavior-oriented, inspired by the style found on the RSpec site.

I didn’t write the tests/specs up front like you’re “supposed” to. Instead, I thought about what I was trying to do, came up with an initial idea for the implementation, started writing code, and then added a test to make sure I was on the right track. This worked out pretty well. I also used the tests during refactoring. Writing the tests first might have worked out better, but I can’t say for sure.

Now, for some real fun, implement a linked list based on only the test code below.

class LinkedList(object):
    def __init__(self, *values):
        self._size = 0
        if values:
            for v in values:
            self.head = self.tail = None

    def is_empty(self):
        return len(self) == 0

    def add(self, val):
        node = Node(val)
        if self.is_empty():
            self.head = node
            self.tail.next = node
        self.tail = node
        self._size += 1

    def get(self, index, get_prev=False):
        if index < 0:
            index = self._size + index
        if index < 0 or index >= self.length:
            raise IndexError
        prev = None
        for i, node in enumerate(self):
            if i == index:
                if get_prev:
                    return node, prev
                    return node
            prev = node

    def remove(self, index):
        node, prev = self.get(index, get_prev=True)
        if node is self.head:
            self.head = node.next
            prev.next = node.next
        self._size -= 1
        return node

    def pop(self):
        return self.remove(self.length - 1)

    def __len__(self):
        return self._size
    length = property(__len__)
    size = property(__len__)

    def __iter__(self):
        curr = self.head
        while curr is not None:
            yield curr
            curr = curr.next
        raise StopIteration

    def __str__(self):
        return ', '.join([str(node.value) for node in self])

class Node(object):
    def __init__(self, val, next=None):
        self.value = val
        self.next = next
import unittest
class Test_A_New_Linked_List(unittest.TestCase):

    def test_given_no_values_should_be_empty(self):
        list_ = LinkedList()
        assert list_.is_empty()

    def test_given_no_values_should_have_size_0(self):
        list_ = LinkedList()
        assert list_.size == list_.length == 0

    def test_given_values_should_not_be_empty(self):
        list_ = LinkedList(1, 2, 3)
        assert not list_.is_empty()
        assert [node.value for node in list_] == [1, 2, 3]

    def test_given_4_values_should_have_size_4(self):
        list_ = LinkedList(1, '2', 3, '4th value')
        assert list_.size == 4

class Test_An_Empty_List(unittest.TestCase):

    def test_should_have_size_1_after_add(self):
        list_ = LinkedList()
        val = 15
        assert list_.head.value == list_.tail.value == val
        assert len(list_) == list_.length == list_.size == 1

    def test_should_raise_an_index_error_on_get(self):
        list_ = LinkedList()
        self.assertRaises(IndexError, list_.get, 1)
        list_ = LinkedList(1, 2, 3)
        list_.remove(0); list_.remove(0); list_.remove(0)
        self.assertRaises(IndexError, list_.get, 0)

    def test_should_raise_an_index_error_on_remove(self):
        list_ = LinkedList()
        self.assertRaises(IndexError, list_.remove, 1)
        list_ = LinkedList(1, 2, 3)
        list_.remove(0); list_.remove(0); list_.remove(0)
        self.assertRaises(IndexError, list_.remove, 0)

    def test_should_raise_an_index_error_on_pop(self):
        list_ = LinkedList()
        self.assertRaises(IndexError, list_.pop)

class Test_A_Non_Empty_List(unittest.TestCase):

    def test_should_increase_its_size_by_1_on_add(self):
        list_ = LinkedList(1, 2, 4, 5)
        starting_size = list_.size
        assert len(list_) == list_.length == list_.size == (starting_size + 1)

    def test_should_return_a_node_on_get(self):
        list_ = LinkedList(1, 2, '3rd value')
        node = list_.get(0)
        assert isinstance(node, Node)
        assert node.value == 1
        node = list_.get(1)
        assert isinstance(node, Node)
        assert node.value == 2
        node = list_.get(2)
        assert isinstance(node, Node)
        assert node.value == '3rd value'

    def test_should_decrease_its_size_by_1_on_remove(self):
        list_ = LinkedList(1, 2, 4, 5)
        starting_size = list_.size
        assert len(list_) == list_.length == list_.size == (starting_size - 1)

    def test_should_decrease_its_size_by_1_on_pop(self):
        list_ = LinkedList(1, 2, 4, 5)
        starting_size = list_.size
        assert len(list_) == list_.length == list_.size == (starting_size - 1)

Google Maps Encoded Polylines

Update 6/30/07: Fixed links to glineenc.py so that it’s actually accessible.
Update 7/15/08: Fixed links to glineenc.py again because of change to HTTPS on Trac site.

Here’s some Python code (complete with unit and doc tests) for converting a series of latitude/longitude points (i.e., a polyline) to the Base64 encoding that Google Maps understands. It’s particularly useful for long and/or complicated lines.

It’s based on the algorithm listed here and the JavaScript code here [page disappeared].

This site gives some more insight into it and has a pretty cool example of a fractal line here.

Here’s the code:

Previously, I had pasted the Python code right into this article, but I recently made a bunch of revisions and it was way too long. Here’s a link to the Bitbucket project instead:

glineenc on Bitbucket

Please note that this code is still in somewhat of a rough state. I have plans to polish and package it up, but for now, I’m using it as is and it’s working quite well (you’ll have to be patient to click that link as it takes ~20-30 seconds to generate the route, even though the line drawing itself is almost instantaneous).

JavaScript that uses results from `glineenc` looks something like this (assuming you’ve returned some JSON, say, with `encoded_points` and `encoded_lines` keys):

 map.addOverlay(new GPolyline.fromEncoded({ color: "#0000ff", weight: 4, opacity: 0.8, points: result.encoded_points, levels: result.encoded_levels, zoomFactor: 32, numLevels: 4 })); 

`points` is the encoded lat/long points. `levels` indicates which zoom levels each point should displayed at; there is one character per point. See the links above for a more complete explanation.