Where To Find A (Physical) Map of Brisbane

Posted on Monday 31 May 2010 by J. D. Bartlett

A quick one for today: I wanted a fold-out map of greater Brisbane which was detailed enough to be able to see individual suburban roads. I didn’t need every road to be labelled, just to be shown. Of course the main roads should (you would expect) be labelled with their names. I searched the Internet for such a map, but to no avail. I remember quite a few years ago I found a pdf version of such a map on the Internet somewhere. I had a look at a map for sale on the Queensland Government Bookshop website, but it turned out it was not as detailed as I wanted.

Finally, I found a map which suited my purposes: RACQ‘s Brisbane and District map. I couldn’t find it on their website, but I went in to a branch office, and it is free for members, or $4 (AUD) otherwise.

Details of the map:

  • 1 : 80 000 scale
  • about 66 cm wide, 80 cm high
  • extends north as far as Ocean View, Moorina, Morayfield and Burpengary
  • extends South as far as South Ripley, Spring Mountain, Greenbank, Park Ridge South, Holmview, Stapylton and Woongoolba
  • extends West as far as Kings Scrub, Dayboro, Samsonvale, Cedar Creek, Highvale, Lake Manchester, Kholo, Bundamba, Blackstone and Ripley

On the other side, there’s some more detailed maps of the city centre and surrounding areas, but I was more interested in the map of suburbs.

Posted in short | Tagged , , | Comments Off on Where To Find A (Physical) Map of Brisbane

What If: Trivia Show Software

Posted on Monday 24 May 2010 by J. D. Bartlett

Earlier this year I was involved in hosting a kind of trivia evening. In order to make things more exciting, each round was worth more points than the previous. The evening was a huge success, but since there were about 15 teams, adding up the scores became a bit of a chore. So in today’s article, I imagine some software which would make such events easier to manage in future.

The Set-up

Imagine that the host of the event has a laptop set up, hooked up to a projector. A couple of helpers have smart phones which are connected to a wireless network which allows them to connect to the laptop via http. The laptop is running software which acts as a web server, so any web-capable phone will work.

Before the Game

Before the event, the host enters into the laptop the planned number of rounds and how many points the questions will be worth in each round. The host may also set up some kind of authentication for the helpers to use when first connecting to the laptop. This may be a username/password combinations for each helper, a single master password, or just matching the IP addresses or MAC addresses (if the setup allows) of the smartphones to be used.

Let the Games Begin

As the game starts, the helpers go around to each table to get the team names. As these are entered into the smart phones, the team names come appear on the projector (which is displaying an AJAX webpage hosted on the laptop). Perhaps the projector even shows the table layout with the team names on the correct table.

Things Progress

After each round, the teams pass their answers to another team to be marked. The helpers then enter into their phones how many questions each team got correct in that round. These are transmitted to the laptop which calculates the team scores and shows a leader board and perhaps even a graph. The host of the event also has the ability to add bonus points and subtract penalty points using either the laptop, or the host’s own phone.

Other Thoughts

If the wireless network is provided by an access point connected directly to the host’s laptop, other exciting possibilities emerge. For instance, the laptop could be running a DNS server, so that the helpers can simply go to http://www.trivia/ in their browsers. Also, if it’s an open access point with an innocent-looking name, you could try to catch a few cheaters—simply set up a capture portal which uses javascript / flash / some other technology to play a loud alarm sound the minute anyone on the wireless network goes to any webpage except the trivia page.

Posted in midlength | Tagged , , , , , , | Comments Off on What If: Trivia Show Software

Mercurial Woes

Posted on Monday 17 May 2010 by J. D. Bartlett

Introduction

I mentioned previously that the Trosnoth programmers have been looking at migrating from subversion to mercurial for source control. I took it upon myself to convert the subversion history to mercurial history as seamlessly as possible. Here I document those adventures.

Background

Mercurial comes with a good tool for converting subversion history to mercurial history. There is a very clear explanation of how to use it in Bryan O’Sullivan’s Mercurial Book. So why was conversion such an adventure? Well, the Trosnoth repository has not been well managed. It was something of an experiment for us initially: none of us had much experience maintaining a subversion repository. So the automated conversion stumbled over our various crimes against convention…

  • When we first started the project, the trunk was in a directory called “Trosnoth” (yes, it had a capital T). Convention dictates that the trunk should be in the “trunk” directory. The branches were in subdirectories of the “branches” directory as is commonly the case.
  • Later on we wanted to version control non-Trosnoth code. For technical reasons we put it all in the same repository, moving the Trosnoth trunk to “trosnoth/trunk” and the Trosnoth branches to “trosnoth/branches”.
  • At a few stages during development, we tagged code, then realised that we’d forgotten to make some minor change (e.g. changing the version number in the code), so we committed a changeset to the tag. Conventionally, tags exist as snapshots and shouldn’t be modified. Although mercurial does support the notion of changing which revision a tag name refers to, having a changeset applying to a tag seems to confuse the conversion utility.

Final Conclusion

Let me insert a spoiler here, so that you only have to read to the end if you’re really keen to see how I went about things. There are two dead easy options when converting a subversion repository of dubious background to a mercurial repository:

  1. decide that you don’t need to be able to access old revisions from mercurial anyway; or
  2. keep old revisions as if they were all in the same branch, and don’t try to convert the branch structure to mercurial.

In the end, because of all the trouble involved in the complete conversion, I decided to go with option 2. Read on if you want the gruesome details.

Preparation

The first thing I did was to mirror the subversion repository. I did that according to the instructions in the Mercurial Book. I should note that at time of writing, there is an error in the Mercurial Book—where it says to type

$ svnsync --init file://pwd/memcached-mirror \
  http://code.sixapart.com/svn/memcached

it should read

$ svnsync init file://pwd/memcached-mirror \
  http://code.sixapart.com/svn/memcached

Note the correct spelling of init.

As well as the Mercurial Book, my attempts were assisted by typing the following command:

$ hg help convert

My Attempts

It took me many attempts to get things to a workable state, so I won’t go through every detail of everything I tried. I will do my best to summarise what I did and what I learnt.

Attempt 1: Naïve Branchmap and Filemap

My first attempt was to construct branchmap and filemap files which provided mappings for how the repository was structured at each step of its history.

File Map

rename trosnoth .

Branch Map

trosnoth/tags/release-1.0.1 release-1.0.1
trosnoth/tags/release-1.0.0 release-1.0.0
trosnoth/tags/postcamp-2009-winter postcamp-2009-winter

Result

File "/usr/lib/pymodules/python2.6/mercurial/localrepo.py", line 777, in _filecommit
    self.ui.debug(_(" %s: copy %s:%s\n") % (fname, cfname, hex(crev)))
TypeError: b2a_hex() argument 1 must be string or read-only buffer, not None

Lesson Learnt

Perhaps the fact that “Trosnoth” originally meant “trunk” confused it. I’m not really sure. But the real lesson learnt was to try again if it doesn’t work.

Attempt 2: Filemap with no Branchmap, First Leg Only

I decided to see if I could get things to work up to the first point of interest only, then perform subsequent conversion steps one at a time. The branch map looked the same as for my first attempt, but I used no file map.

Result

This attempt failed. I suspect this is again because of something to do with the trunk being called “Trosnoth” at some point in its past.

Lesson Learnt

Um… when you’re trying to delete the last conversion attempt, don’t accidentally delete the subversion mirror or you’ll have to wait for the whole mirroring process to finish again. Trust me, this is an important one.

Attempt 3: Using “Rename” in the Filemap

Filemap

rename trosnoth/trunk trunk
rename trosnoth/branches branches
rename trosnoth/tags tags
rename trosnoth/camperblocks camperblocks

Result

This attempt failed. It no longer recognised subversion branches as branches.

Lesson Learnt

For hg convert to realise that a subversion branch is a branch, it must be in /branches. Making it end up there as the result of a rename is not enough.

Attempts 4 and 5: Combining Branchmap with Filemap

Branchmap

branches/josh-Trosnoth-restructure josh-trosnoth-restructure

Filemap

rename trosnoth/trunk trunk
rename trosnoth/branches branches
rename trosnoth/tags tags
rename trosnoth/camperblocks camperblocks
rename Trosnoth trunk

Result

Attempt 4 was pretty good, but I did not use the final line of the file map, so it didn’t account for the fact that our trunk was called “Trosnoth”. Attempt 5 still didn’t accept the trunk as trunk, even though it had been renamed from “Trosnoth” to “trunk”.

Lesson Learnt

Hg convert does not realise that a subversion trunk is the trunk if it’s been renamed to /trunk in the file map. It must be called /trunk already.

Attempt 7: Using –config convert.svn.trunk=

In attempt 6 I tried using –config convert.svn.trunk=Trosnoth, but that made the default branch called “Trosnoth”. I got things to work by using –config convert.svn.trunk= with nothing after the equals sign, then using the branch map to map “Trosnoth” to “default”. Of course by this stage I was just trying to get things to work up to the first branch, so there was still work to be done.

Lesson Learnt

Make use of –config convert.svn.*= if you can.

Attempts 8 to 11: Getting Rid of a Branch

At one point in the history of Trosnoth, we created a branch which should not really have been a branch. Really it was a side project. But it resided in the “branches” directory. So I wanted to get rid of it when I did the conversion. I tried using various “exclude” directives in the file map, but no matter how hard I tried, the branch was always created by the conversion (even though I could get it to have no revisions after the initial creation of the branch).

Lesson Learnt

If you find yourself spending many hours on something, stop and ask yourself if it’s really that important.

Final Attempt: Ignore Branches and Tags, Fix it at the End

So finally, after messing with things for hours, I decided this: I would not try to convert subversion branches and tags to mercurial branches and tags. Rather, I would reproduce the entire subversion directory structure, branches and all, as a single branch of the mercurial repository. I would add tags to this repository at the revision at which tags were added to the subversion repository.

Converting Without Any Branches or Tags

$ hg convert file://pwd/trosnoth-mirror trosnoth-hg --config convert.svn.trunk= --config convert.svn.branches= --config convert.svn.tags=

The command above tells mercurial to ignore all branches and tags during the conversion.

Finding Braches and Tags

I knew that the first subversion tag had been made at revision 486. I wanted to create a corresponding mercurial tag.

$ cd trosnoth-hg
$ cat .hg/shamap | grep @486
svn:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx@486 <hg-revision-id>
$ hg tag <tag-name> -r <hg-revision-id>

Finally Cleaning Up

At the end of everything, I wanted the trunk (and any open branches) to not contain the full subversion directory layout, but only the branch content. So I did a combination of hg rm and hg move commands, and everything worked out fine.

Posted in long | Tagged , , , | Comments Off on Mercurial Woes

HATE: Python-based Terminal Emulator

Posted on Monday 10 May 2010 by J. D. Bartlett

Introduction

A while ago I came across a terminal emulator called HATE: the High Availability Terminal Emulator. The original idea was for it to be a terminal emulator that could survive X crashing, but what caught my eye was the fact that it was written in Python. Therefore, I reasoned, it would be easy to extend.

What I’ve Done With It

Step 1: Refactoring

I got a copy of HATE from Jerub’s repository and had a look at the source code. Seeing that it was a small yet complete Python program, the first thing I decided to do with it was to use it as an example of my Pyj extension, which I wrote about recently. So I carefully refactored the code, updating Pyj as I went. After this step, my new version of HATE:

  • did not construct any classes;
  • was made up of a whole bunch of small files each defining one function, object or constant; and
  • still did exactly the same thing as the original did.

Step 2: Terminal Stack and Colours

Inspired by the Tab Kit extension for Firefox, I added a vertical stack of terminals on the left-hand side of the window. Each entry in the stack has an icon, a title, and a letter which can be typed in the “Go” box in order to switch to that terminal. Pressing Ctrl+Shift+G will jump to the “Go” box, Ctrl+Shift+T will open a new terminal, and Ctrl+Shift+B will switch to the terminal at the bottom of the stack (least recently used). I also made each terminal have a random colour to make it easier to remember which terminal was used for what purpose.

You can get my code on launchpad here. Or if you have bazaar, you can type bzr branch lp:~jd-bartlett/+junk/hate. To see what the project looked like after refactoring, have a look at the tag 10.4. You may wish to compare it with the 8.7 tag, which is how things were before the refactoring. The 10.5 tag is what the project looked like at time of writing. Please note though that:

  • in 10.4, setup.py does not work because it picks up .py files (of which there is only one) and ignores .pyj files;
  • you will need to get a copy of pyj for this to work—hg clone https://talljosh@bitbucket.org/talljosh/pyj; and
  • the 0.9 tag of Pyj works with the 10.4 tag of HATE, and the 0.9.1 tag of Pyj works with the 10.5 tag of HATE, but other combinations of revisions may not work together.

What I May Yet Do

Inspired by the MiniBufExplorer plugin for vim, and the Terminator terminal emulator, I would like to add viewports to HATE. When using Terminator, you can split the current viewport either vertically or horizontally, ending up with multiple viewports in the one window. I’d like to see this in HATE. There should also be key combinations to switch to each viewport.

And of course, the more work I do on HATE, the more fun I have with Pyj!

Posted in midlength | Tagged , , , , | Comments Off on HATE: Python-based Terminal Emulator

Pyj: Experimenting with Code Design

Posted on Monday 3 May 2010 by J. D. Bartlett

Introduction

A few weeks ago I proposed a new concept for not-quite-OO programming. Recently I’ve been working on turning this into a reality. I haven’t written a new programming language. Rather, I’ve extended Python using import hooks. I’ve called this extension “Pyj”. In this article I’ll describe what Pyj does (or will someday do) and how I’ve implemented it.

The Pyj Philosophy

Pyj is quite experimental and I’ve invented it based on thoughts I’ve had that usually start with “Wouldn’t it be cool if…”. Only time will tell whether these ideas are useful and worthwhile. Here are the main principles I’ve based Pyj upon:

  • small files, declaring all dependencies;
  • files shouldn’t need to be organised perfectly;
  • behaviour of objects is more important than some kind of class hierarchy; and
  • functions should be able to easily access ancestors of the current “context” object—see this article for more details.

It is worth noting that Pyj files are actually valid Python files. The difference lies in how they are treated when imported into Python.

Small Files

I’ve designed Pyj around the idea of keeping each file simple and small, so that programmers don’t have to search through a whole bunch of code to find the part of the file that they want. In practice what this looks like is each Pyj file representing a single function or object. Pyj is implemented so that Pyj modules behave almost identically to the function or object that they represent. So if you write /stoop/heap/heart.pyj and you make it define a function, you can call the function using stoop.heap.heart() rather than having to add the name of the function, as in stoop.heap.heart.myFunction(). Within a Pyj source file, whichever function or object is called “this” will be the function or object represented by that module.

As an example, if you wanted to write a function to add two numbers together, your Pyj file wolud look like:

def this(a, b):
    return a + b

You would not put any other functions in the same file. And because you have named the function “this”, you would be able to access it using the module’s name. For example, if the file was called addTwoNumbers.pyj, you would simply type addTwoNumbers(x, y) to invoke the function.

No Need to Organise Perfectly

In going with small files, I’ve assumed that developers have tools (such as grep) available to quickly search and find which file they want. Small files mean lots of files, and without this assumption, developers would be forever creating sub-packages to organise all their pyj modules, and would have much less time to do the actual programming.

Behaviour is More Important than Hierarchy

Because of the idea of functions running in contexts (I assume you’ve already read my previous post), and because of the flexible nature of Python, I’ve decided to throw most of object-oriented programming (OOP) out the window and resort to an object model similar to that used in JavaScript. That is, we don’t care about classes and inheritance, we care about what an object does. So rather than a class with a constructor, we have factory functions—functions which create and return objects. This has allowed me to reinvent a very flexible kind of OOP using the context in place of the current object. I hope to present a complete working example of a useful Pyj program in a few weeks, and you’ll be able to see what I mean.

Functions Can Access Ancestors of the Context

For this one you really should read my previous post. Basically though, each object has a name and can have a parent. Every function is called in a context, and that context may be any object. If a function is running in a particular context and the context has a parent named breathe which has a parent named imposter which has a parent named wisdom, the function can access wisdom by the name wisdom rather than having to type self.breathe.imposter.wisdom. Too complicated? My previous post explains in more depth.

Python Import Hooks

The idea of import hooks in Python is that you can customise the way in which Python imports modules. Pyj makes use of import hooks in order to distinguish Pyj modules from ordinary Python modules. Ordinary Python modules have a .py extension, and the compiled byte-code is saved in a file with a .pyc or .pyo extension (.pyo for optimised bytecode). Pyj modules have a .pyj extension, and compiled byte-code is saved in files with .pyjc or .pyjo extensions. I’ve written an import hook, so that when an import statement is executed:

  • if there is a .py, .pyo, or .pyc file, that will be imported as a normal Python module;
  • otherwise, if there is a .pyj, .pyjc, or .pyjo file, that will be imported as a Pyj module.

(As an aside, I’m considering changing it so that if there is a .pyc and .pyj but no .py it will load the .pyj rather than the .pyc, but for now the default is always normal Python modules over Pyj modules.)

To figure out how to write the import hook, I exploited these useful resources:

  • PEP 302 describes in detail how import hooks work; and
  • the source code of the ihooks and pkgutil modules (found in your Python directory—I’m using Python 2.6).

So if you were writing a Python package which includes some Pyj modules, how would you install the import hooks? Simply put the following line in the __init__.py of your package.

import pyj

That’s it! Your __init__.py file will be loaded first before any attempt to import a sub-package or module within the package. When Pyj is imported, it installs the import hook.

How Pyj Works

The main challenge in writing Pyj was figuring out how to change the way a Python function looked up global names. Normally if you try to access a global name from within a function, the globals dictionary of the module is looked up and that’s it. But for Pyj, I needed the lookup to depend on the current context. It took me a few attempts to get this right.

Subclassing Dict

My first attempt was to create a subclass of the dict built-in type, and customise the way that lookups are done. My reasoning was that if the function looks up global variables in a dict, looking them up in my subclass of dict should allow me to change the way that look-ups are done. Unfortunately this plan was thwarted by the fact that Python does its global lookups using the default dict lookup even if your subclass customises dict lookups.

Intelligent Name Substitution

The approach which I had to go with in the end was this: at the time that a function is called, copy any relevant names into the global namespace, then when the function ends, replace them with what they were before. A relevant name is a name which:

  • is used in a function in the Pyj module (this is determined by inspecting function.func_code.co_names); and
  • is the name of an ancestor of the current context, or a special name like self.

In order to actually achieve this, every function declared in a Pyj module is wrapped in a simple class which intercepts calls. And performs the appropriate substitutions before and after the execution of the function.

Can I try this out?

At the time of writing, I’m still working hard on getting Pyj to do exactly what I want it to. But in a couple of weeks I hope to release a working example of a program written in Pyj, for you to have a look at. For the time being, you can have a look at my code here. I’ve also set up a project page for Pyj here.

In Closing

I’ve explained the general concepts behind Pyj and a few of the specifics about how Pyj works. Pyj is entirely experimental, but I’m hoping that it’s a lot of fun to play (erm… I mean work…) with. I also hope that my explanation of Pyj has inspired you to imagine other ways in which conventional approaches to programming might be challenged. Certainly, some such challenges will be prove unworthy, and be defeated. But the more people dream about what programming could be like, the more chance we’ll have of making real improvements to how we do things.

Posted in long | Tagged , , , , | Comments Off on Pyj: Experimenting with Code Design