Introduction
I mentioned previously that the Trosnoth programmers have been looking at migrating from subversion to mercurial for source control. I took it upon myself to convert the subversion history to mercurial history as seamlessly as possible. Here I document those adventures.
Background
Mercurial comes with a good tool for converting subversion history to mercurial history. There is a very clear explanation of how to use it in Bryan O’Sullivan’s Mercurial Book. So why was conversion such an adventure? Well, the Trosnoth repository has not been well managed. It was something of an experiment for us initially: none of us had much experience maintaining a subversion repository. So the automated conversion stumbled over our various crimes against convention…
- When we first started the project, the trunk was in a directory called “Trosnoth” (yes, it had a capital T). Convention dictates that the trunk should be in the “trunk” directory. The branches were in subdirectories of the “branches” directory as is commonly the case.
- Later on we wanted to version control non-Trosnoth code. For technical reasons we put it all in the same repository, moving the Trosnoth trunk to “trosnoth/trunk” and the Trosnoth branches to “trosnoth/branches”.
- At a few stages during development, we tagged code, then realised that we’d forgotten to make some minor change (e.g. changing the version number in the code), so we committed a changeset to the tag. Conventionally, tags exist as snapshots and shouldn’t be modified. Although mercurial does support the notion of changing which revision a tag name refers to, having a changeset applying to a tag seems to confuse the conversion utility.
Final Conclusion
Let me insert a spoiler here, so that you only have to read to the end if you’re really keen to see how I went about things. There are two dead easy options when converting a subversion repository of dubious background to a mercurial repository:
- decide that you don’t need to be able to access old revisions from mercurial anyway; or
- keep old revisions as if they were all in the same branch, and don’t try to convert the branch structure to mercurial.
In the end, because of all the trouble involved in the complete conversion, I decided to go with option 2. Read on if you want the gruesome details.
Preparation
The first thing I did was to mirror the subversion repository. I did that according to the instructions in the Mercurial Book. I should note that at time of writing, there is an error in the Mercurial Book—where it says to type
$ svnsync --init file://pwd/memcached-mirror \
http://code.sixapart.com/svn/memcached
it should read
$ svnsync init file://pwd/memcached-mirror \
http://code.sixapart.com/svn/memcached
Note the correct spelling of init.
As well as the Mercurial Book, my attempts were assisted by typing the following command:
$ hg help convert
My Attempts
It took me many attempts to get things to a workable state, so I won’t go through every detail of everything I tried. I will do my best to summarise what I did and what I learnt.
Attempt 1: Naïve Branchmap and Filemap
My first attempt was to construct branchmap and filemap files which provided mappings for how the repository was structured at each step of its history.
File Map
rename trosnoth .
Branch Map
trosnoth/tags/release-1.0.1 release-1.0.1 trosnoth/tags/release-1.0.0 release-1.0.0 trosnoth/tags/postcamp-2009-winter postcamp-2009-winter
Result
File "/usr/lib/pymodules/python2.6/mercurial/localrepo.py", line 777, in _filecommit
self.ui.debug(_(" %s: copy %s:%s\n") % (fname, cfname, hex(crev)))
TypeError: b2a_hex() argument 1 must be string or read-only buffer, not None
Lesson Learnt
Perhaps the fact that “Trosnoth” originally meant “trunk” confused it. I’m not really sure. But the real lesson learnt was to try again if it doesn’t work.
Attempt 2: Filemap with no Branchmap, First Leg Only
I decided to see if I could get things to work up to the first point of interest only, then perform subsequent conversion steps one at a time. The branch map looked the same as for my first attempt, but I used no file map.
Result
This attempt failed. I suspect this is again because of something to do with the trunk being called “Trosnoth” at some point in its past.
Lesson Learnt
Um… when you’re trying to delete the last conversion attempt, don’t accidentally delete the subversion mirror or you’ll have to wait for the whole mirroring process to finish again. Trust me, this is an important one.
Attempt 3: Using “Rename” in the Filemap
Filemap
rename trosnoth/trunk trunk rename trosnoth/branches branches rename trosnoth/tags tags rename trosnoth/camperblocks camperblocks
Result
This attempt failed. It no longer recognised subversion branches as branches.
Lesson Learnt
For hg convert to realise that a subversion branch is a branch, it must be in /branches. Making it end up there as the result of a rename is not enough.
Attempts 4 and 5: Combining Branchmap with Filemap
Branchmap
branches/josh-Trosnoth-restructure josh-trosnoth-restructure
Filemap
rename trosnoth/trunk trunk rename trosnoth/branches branches rename trosnoth/tags tags rename trosnoth/camperblocks camperblocks rename Trosnoth trunk
Result
Attempt 4 was pretty good, but I did not use the final line of the file map, so it didn’t account for the fact that our trunk was called “Trosnoth”. Attempt 5 still didn’t accept the trunk as trunk, even though it had been renamed from “Trosnoth” to “trunk”.
Lesson Learnt
Hg convert does not realise that a subversion trunk is the trunk if it’s been renamed to /trunk in the file map. It must be called /trunk already.
Attempt 7: Using –config convert.svn.trunk=
In attempt 6 I tried using –config convert.svn.trunk=Trosnoth, but that made the default branch called “Trosnoth”. I got things to work by using –config convert.svn.trunk= with nothing after the equals sign, then using the branch map to map “Trosnoth” to “default”. Of course by this stage I was just trying to get things to work up to the first branch, so there was still work to be done.
Lesson Learnt
Make use of –config convert.svn.*= if you can.
Attempts 8 to 11: Getting Rid of a Branch
At one point in the history of Trosnoth, we created a branch which should not really have been a branch. Really it was a side project. But it resided in the “branches” directory. So I wanted to get rid of it when I did the conversion. I tried using various “exclude” directives in the file map, but no matter how hard I tried, the branch was always created by the conversion (even though I could get it to have no revisions after the initial creation of the branch).
Lesson Learnt
If you find yourself spending many hours on something, stop and ask yourself if it’s really that important.
Final Attempt: Ignore Branches and Tags, Fix it at the End
So finally, after messing with things for hours, I decided this: I would not try to convert subversion branches and tags to mercurial branches and tags. Rather, I would reproduce the entire subversion directory structure, branches and all, as a single branch of the mercurial repository. I would add tags to this repository at the revision at which tags were added to the subversion repository.
Converting Without Any Branches or Tags
$ hg convert file://pwd/trosnoth-mirror trosnoth-hg --config convert.svn.trunk= --config convert.svn.branches= --config convert.svn.tags=
The command above tells mercurial to ignore all branches and tags during the conversion.
Finding Braches and Tags
I knew that the first subversion tag had been made at revision 486. I wanted to create a corresponding mercurial tag.
$ cd trosnoth-hg $ cat .hg/shamap | grep @486 svn:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx@486 <hg-revision-id> $ hg tag <tag-name> -r <hg-revision-id>
Finally Cleaning Up
At the end of everything, I wanted the trunk (and any open branches) to not contain the full subversion directory layout, but only the branch content. So I did a combination of hg rm and hg move commands, and everything worked out fine.