This weekend I finally bit the bullet and upgraded our production Subversion server to 1.4. The upgrade was painless, after the usual running of autogen.sh on the Neon libraries that for some reason are necessary when building on the Solaris 9 environment. I also had some weird test failures that wound up being caused by hook scripts being called on the test repositories. For some reason, Python couldn’t find one of the standard libraries, even though I had set my path and library path in my environment. A quick Google search later and all tests passed.

We have been running Subversion with the Berkeley 2.2 back-end since May of 2004, when the Subversion team first released 1.0. We’ve done a few upgrades since then, when I found the time, but wound up getting as far as 1.2.3 and my time to actually do the upgrades never really materialized. Part of that had to do with some pretty large projects that were going on that I didn’t want to interrupt the teams with an upgrade.

So yesterday, finally, I did the upgrade to 1.4. As part of the upgrade, I decided to dump and load the repositories (11 of them) to take advantage of the new SVN Diff format that 1.4 brings to the table. As part of that, I also decided to abandon the Berkeley back-end in favor of the FSFS file system.

The dumps and loads took quite a long time, about sixteen hours. However, the results were worth it. After all was said and done, we saw a reduction in server disk space usage of about 46.8%.

Not bad guys!

As I had mentioned in a previous post, one of the motivators for this upgrade were the working copy improvements, which according to the release notes, should cut down on inode usage on machines running the client. In our environment, we have about 20 people all working on the same machine (in their home directories) and we have had problems with inode max-outs in the past. Again, I’ll have to reserve judgement on these changes until we see the results, but this is one of the things that motivated me to bite the bullet and take a Saturday to do the upgrades.

The other big motivation for me, especially after yesterday was this statement from the FSFS document referenced above:

An FSFS repository can be backed up with standard backup software.
Since old revision files don’t change, incremental backups with
standard backup software are efficient. (See “Note: Backups” for
caveats.)

(BDB repositories can be backed up using “svnadmin hotcopy” and can be
backed up incrementally using “svnadmin dump”. FSFS just makes it
easier.)

Aside from normal system backups done nightly by the admin team, our nightly backups have also consisted of doing nightly dumps of all of the repositories so that, in the event that the repository server goes down, we can just load the repository on another machine (you can never have too many backups – in my opinion). The wake up call from yesterday is that it takes much longer to reload a repository on Solaris 9 than it does on the Linux work-group server I have sitting here that I did all of my original testing on a few years ago. What I thought would be a trivial load (3 hours was the estimate that I had in my head) wound up being 5-6 times that on the 2×2 Solaris box at work. It will be nice to be able to rely on the nightly system backups for these now — and perhaps just a quick tarring of the existing repository structure for quick backups (using hotcopy), rather than the dump / load process we have planned to use up until now. I’ve just never been comfortable for some reason with the Berkeley backend and backups outside of repository dumps.

Finally, I’m really looking forward to playing around with svnsync to set up a shadow fail-over repository in the event of an outage. This was something I threw on a wish list back in 2004, and I’m excited to see a tool to synchronize a “stand by repository” included in this release. There apparently is not much documentation on this new feature yet, but this article should be a good starting point.

Thanks to the Subversion team for another great release of their software. As anyone who reads this blog with any regularity knows, this is one of those pieces of software that I continuously receive value from and cannot say enough about.

Now I’m off to search for more svnsync documentation …