SVK – Distributed Version Control – Part III

Introduction

Last week we explored using a local branch and keeping it in sync with the main repository. We also explored the incremental synchronization of local mirrors on our machine. This week we will be merging our changes back into the main repository and wrapping up our discussion of SVK for now.

Notation

Some of the lines in the command are truncated because they are too long. I have used the ‘_’ character as a line continuation character. If you see this character, look at the next line.

Merging From Your Local Branch To The Mirror

At the end of our last article, we had finished the work that we wanted to do and now we would like to merge our changes into the main repository so that others can access our changes. The assumption is that we have performed all of the due dillegence that software developers perform when they finalize a change and are ready to have those changes activated in the main repository, including keeping our local branch up to date with the main repository using the repository synchronization techniques outlined in previous articles and merging them into our local branch to keep our workarea current.

The time has finally come where everyone who uses the main repository gets to see the changes we have been working on.

When merging to a mirrored tree in SVK, we use the smerge command just like we would in our previous work, only this time the source and destination are switched. We are now merging to the mirror, however the process looks the same. The really cool thing about SVK however, is that when you perform a merge to a mirrored path, you not only update your mirror, but the repository that the mirror points to is also updated simultaneously.

So lets do it. Before actually performing the merge, connect to the network and sync your local repository.


svk sync //bieberlabs/trunk
Syncing http://subversion.bieberlabs.com/svn/bieberlabs/trunk
rbieber:/~>

Now that we are synced up, we will perform an svk smerge -C command to make sure we have no conflicts.


rbieber:/~>svk smerge -C //bieberlabs/new-feature-x _
//bieberlabs/trunk
Auto-merging (0, 409) /bieberlabs/new-feature-x _
to /bieberlabs/trunk (base /bieberlabs/trunk:408).
Checking locally against mirror source _
http://subversion.bieberlabs.com/svn/bieberlabs/trunk.
U website/index.php
D website/familypictures
New merge ticket: fd3a5cf1-f4e9-0310-b907-bd1e11f8034a: _
/bieberlabs/new-feature-x:409
rbieber:/~>

Some things to notice. Previously we saw quite a few changes coming from the mirrored repository to our local branch, including the re-adding of a photos directory with 68 pictures in the main repository. None of these changes are being merged back to the mirror. SVK is keeping track of the merges we have made from the main repository to our local branch and filtering them out (or skipping them) for us.

It winds up that with all the activity between the main repository and the local branch, the final result is that the only changes made by us in this whole process were the deletion of a directory, and modifications to our index.php file — and SVK knows that. We didn’t have to spend a lot of our valueable time towards an impossible schedule to figure it out!

Now that we know there are no conflicts (it’s nice to live in a perfect world!) we can actually perform a real merge to the mirror. We do this by substituting (in this case) the -C option with the -l option to get an initial log message that reflects each of our individual commits so we know what changes have been made through the time we have been developing on our local branch.


rbieber:/~>svk smerge -l //bieberlabs/new-feature-x //bieberlabs/trunk
Auto-merging (0, 409) /bieberlabs/new-feature-x to _
/bieberlabs/trunk (base /bieberlabs/trunk:408).
Waiting for editor...
Merging back to mirror source _
http://subversion.bieberlabs.com/svn/bieberlabs/trunk.
U website/index.php
D website/familypictures
New merge ticket: fd3a5cf1-f4e9-0310-b907-bd1e11f8034a: _
/bieberlabs/new-feature-x:409
Merge back committed as revision 24.
Syncing http://subversion.bieberlabs.com/svn/bieberlabs/trunk
Retrieving log information from 24 to 24
Committed revision 420 from revision 24.
rbieber:/~>

As you can see from the above messaging, a few things have happened during the actual merge:

  1. We were prompted for a log message. Even though this is prepopulated, you have to add something to it to let the tool know you want to continue with the merge, just like SVN and CVS.
  2. Our changes were merge to the mirrored path as we specified.
  3. Our changes were then merged from the mirrored repository to the main repository — automatically exposing them in the main repository for other developers to subsequently pull down.
  4. Our mirror was automatically resynced to be current.

Just to verify that our changes made it to the repository, we can go to our local installation of ViewCVS to verify the commit:

ViewCVS Screenshot

In Conclusion

In the last three articles, we have the basic areas of distributed, activity based development necessary to get someone new to SVK productive. I hope that as we have walked through the use of this tool, I have illustrated the huge gains you can receive from beginning to use it.

There are obvious areas in which I have neglected to cover from a conceptual level like conflict resolution, however this process under SVN is covered quite extensively in the Resolving Conflicts chapter of the SVN book written by the Subversion developers. There are also other commands and types of merges that SVK can perform that I will leave to the reader to explore.

SVK is a very powerful tool. The main disadvantage of it is that is written in PERL, and therefore can be difficult to install, and slow to start up. However, these disadvantages are far outweighed by the advantages that the tool brings to the table.

The ability to work while disconnected from a central repository and the automatic merge tracking features of SVK are of incredible value to anyone wanting to give flexibility and true distributed capabilities to their development staff. I highly recommend that you take the time to explore this tool and explore for yourself the new capabilities and flexibility, not to mention just plain time savings SVK brings to the table.

Back to part II | Start Over at Part I

SVK – Distributed Version Control – Part II

Introduction

In our last article, we installed SVK and created our local mirror of our repository. Since the writing of the previous article, we have caught up on a lot of TV by making code changes to our local SVK workarea and using the svk commit command to commit our changes to our local repository. During this time, one of my other personalities has been making code changes to our main repository using straight SVN. This scenario will illustrate keeping your local repository in sync, updating your local branch with changes to your mirrored repository, and merging your local changes back to your mirrored repository when we are finished.

Notation

Some of the lines in the command are truncated because they are too long. I have used the ‘_’ character as a line continuation character. If you see this character, look at the next line.

Keeping your mirror current

Some changes have happened in the main repository since we mirrored it, including changes to the main index.php file and a directory removal that was backed out from 8 revisions ago. We have also made some code changes to our local branch, including removing a familypictures directory and modifying the index.php file and now we want to sync our local branch with what is currently in production. To do this we have to update our mirrored repository, requiring a network connection. Once connected to the network, the svk sync command will bring down changes to the main repository to our local mirror just like it did in the first article, only quite a bit faster now since it is only pulling deltas from the last synchronization.

svk sync //bieberlabs/trunk
Syncing https://subversion.bieberlabs.com/svn/bieberlabs/trunk
Retrieving log information from 21 to 23
Committed revision 406 from revision 21.
Committed revision 407 from revision 22.
Committed revision 408 from revision 23.

Now our local mirror is current, so we can disconnect from the network again and go sit in front of the TV.

Keeping your local branch current

To make sure we are changing current code, we need to merge changes made to the main repository to our local branch. In straight SVN, this is done with the SVN merge command and specific revisions must be specified in order to make sure you are merging only changes that were made since your last merge from the trunk. This requires use of the SVN log command to figure out what you merged last in order to get the correct revision range to specify in the merge command.

With SVK, things get a little easier. SVK implements the star merge algorithm first introduced in the arch version control tool. This algorithm figures out what has already been merged from one branch to another, alieviating the need to go through and manually figure out what you need to merge back and forth to the branch.

The first thing we need to do in order to execute a star merge is the equivelent of a –dry-run merge in SVN. Using the -C option of the svk smerge command, we can find out whether there will be any conflicts during our merge to prepare ourselves for any conflict resolution that may have to take place.

svk smerge -C //bieberlabs/trunk //bieberlabs/new-feature-x
Auto-merging (301, 408) /bieberlabs/trunk to
/bieberlabs/new-feature-x (base /bieberlabs/trunk:301).
U   website/wordpress/index.php
A   website/photos
A   website/photos/raytracing
A   website/photos/raytracing/yinyang2.jpg
A   website/photos/raytracing/grapes.jpg
A   website/photos/raytracing/mexico-sunset.jpg
A   website/photos/raytracing/castle.jpg
A   website/photos/raytracing/yyisland.jpg
A   website/photos/jonna
A   website/photos/jonna/jonnaandyairhockey.jpg
A   website/photos/jonna/jonnajake.jpg
A   website/photos/jonna/jonnatattoo.jpg
A   website/photos/jonna/family.jpg
A   website/photos/jonna/ronandandyracing.jpg
A   website/photos/jonna/jonnaboys.jpg
A   website/photos/jonna/ronandbeetle.jpg
A   website/photos/jonna/andysbirthday.jpg
A   website/photos/jonna/jonna.jpg
A   website/photos/jonna/boysandpumpkins.jpg
A   website/photos/jonna/jonnacouch.jpg
A   website/photos/jonna/ronintn.jpg
A   website/photos/kelsi
A   website/photos/kelsi/kelsianddaddy.jpg
A   website/photos/kelsi/cute.jpg
A   website/photos/kelsi/girls1.jpg
A   website/photos/kelsi/kelsiandandrea.jpg
A   website/photos/kelsi/kelsiandandy.jpg
A   website/photos/kelsi/2minutes.jpg
A   website/photos/kelsi/kelsiandmommy.jpg
A   website/photos/kelsi/halloween.jpg
A   website/photos/kelsi/firstbirthday.jpg
A   website/photos/kelsi/toocool.jpg
A   website/photos/kelsi/batdadandbatkelsi.jpg
A   website/photos/kelsi/longhaireddaddy.jpg
A   website/photos/kelsi/ozzytatoo.jpg
A   website/photos/kelsi/newbaby.jpg
A   website/photos/kelsi/snowwhite.jpg
A   website/photos/kelsi/indeed!.jpg
A   website/photos/kelsi/kelsinow.jpg
A   website/photos/kelsi/computer.jpg
A   website/photos/kelsi/atthepool.jpg
A   website/photos/index.php
A   website/photos/uploaded
A   website/photos/uploaded/portraitweb.jpg
A   website/photos/uploaded/newTV.jpg
A   website/photos/uploaded/ronjonnaedsidcheyney1.jpg
A   website/photos/uploaded/signlicense.jpg
A   website/photos/uploaded/newtatto.jpg
A   website/photos/uploaded/wedding-picture-web.jpg
A   website/photos/uploaded/neutralzone.jpg
A   website/photos/uploaded/house-front.jpg
A   website/photos/uploaded/happy kelsi.jpg
A   website/photos/uploaded/house-left.jpg
A   website/photos/uploaded/biography-cover-jacob.jpg
A   website/photos/uploaded/mvc-007s.jpg
A   website/photos/uploaded/jonna-tattoo.jpg
A   website/photos/uploaded/biography-cover-andrew.jpg
A   website/photos/uploaded/ronring.jpg
A   website/photos/uploaded/house-downstreet.jpg
A   website/photos/uploaded/bridekiss.jpg
A   website/photos/uploaded/conehead kids.jpg
A   website/photos/uploaded/familyatdisney.jpg
A   website/photos/uploaded/biography-cover-ronjonna.jpg
A   website/photos/uploaded/rings.jpg
A   website/photos/uploaded/ronjonna.jpg
A   website/photos/uploaded/livingroom.jpg
A   website/photos/uploaded/andy and dwarf jonna.jpg
A   website/photos/uploaded/vaibox.jpg
A   website/photos/uploaded/andy books.jpg
A   website/photos/uploaded/kids.jpg
A   website/photos/uploaded/ronfirstbirthday.jpg
A   website/photos/uploaded/welcomemat.jpg
A   website/photos/uploaded/house-right.jpg
A   website/photos/uploaded/edcheyney2.jpg
A   qf/source/wildcard.h
A   qf/source/compress.h
A   qf/source/qfw32.c
A   qf/source/qf.c
A   qf/source/qf.h
A   qf/source/crc.h
A   qf/source/qfdos.c
UU  qf/source/QF2.MAK
UU  qf/source/DOSTYPES.H
A   qf/source/qfdos.h
A   qf/source/qfprint.c
UU  qf/source/DOC/QF.DOC
UU  qf/source/QFW32.MAK
A   qf/source/qfllf.8
A   qf/source/wildcard.c
A   qf/source/qfos2.c
A   qf/source/compress.c
UU  qf/source/MAKEIT.CMD
A   qf/source/qfvfy.c
UU  qf/source/QF.MAK
UU  qf/source/QF.STS
D   qf/source/COMPRESS.C
D   qf/source/COMPRESS.H
D   qf/source/CRC.H
D   qf/source/QF.C
D   qf/source/QF.H
D   qf/source/QFDOS.C
D   qf/source/QFDOS.H
D   qf/source/QFLLF.8
D   qf/source/QFOS2.C
D   qf/source/QFPRINT.C
D   qf/source/QFVFY.C
D   qf/source/QFW32.C
D   qf/source/WILDCARD.C
D   qf/source/WILDCARD.H
New merge ticket: 4ddbc19d-f0cf-0310-94ad-fbf59656ec37:/trunk:23

As you can see from the output, we have about 110 changes that have happened to the main SVN repository since we created our local mirror, and all of them are now in our mirrored repository. Luckily during this time, none of the changes we have made to our local branch have conflicted with the changes made to the main repository.

One thing to note as we execute these commands is that the merge ‘from’ and merge ‘to’ repository locations on the command line are actually specified from left to right. The from repository is always the first one specified, and the ‘to’ repository is always specified last.

Now that we have this information, we can merge these changes into our local branch to continue our work. We do this with the same svk smerge command as above, this time without the -C option specified.

svk smerge -l //bieberlabs/trunk //bieberlabs/new-feature-x
Auto-merging (301, 408) /bieberlabs/trunk to
/bieberlabs/new-feature-x (base /bieberlabs/trunk:301).
Waiting for editor...

D   qf/source/WILDCARD.H
New merge ticket: 4ddbc19d-f0cf-0310-94ad-fbf59656ec37:/trunk:23
Committed revision 409.

You will notice that we specified an -l option to the smerge command. This command prepopulates your commit messages with the individual commit messages for the revisions you are merging and allows you to add additional message text to them. I find this very handy for keeping track of the actual changes that have been merged locally. The initial commit message when using this option during our merge looked like this:

 r373@compaq (orig r13):  rbieber | 2004-12-11T13:22:26.271503Z
 Change svn:eol-style to native
 r374@compaq (orig r14):  rbieber | 2004-12-11T13:31:27.979404Z
 Continue updating header with GNU license.
 r375@compaq (orig r15):  rbieber | 2004-12-11T13:36:57.421569Z
 Continue code reformatting
 r376@compaq (orig r16):  rbieber | 2004-12-11T13:39:56.256824Z
 Rename files to lowercase equivelent
 r377@compaq (orig r17):  rbieber | 2004-12-11T14:04:28.117857Z
 Code reformatting
 r379@compaq (orig r18):  rbieber | 2004-12-11T14:24:48.136330Z
 Code reformatting
 r380@compaq (orig r19):  rbieber | 2004-12-11T14:30:28.926752Z
 Code Reformatting
 r381@compaq (orig r20):  rbieber | 2004-12-11T14:55:07.571027Z
 Code Reformatting
 r406@compaq (orig r21):  rbieber | 2004-12-31T13:35:35.719204Z
 Update index.php with new content from production
 r407@compaq (orig r22):  rbieber | 2004-12-31T13:37:40.697066Z
 Back out production changes made in revision 21 and
 update correct index.php
 r408@compaq (orig r23):  rbieber | 2004-12-31T13:44:13.339651Z
 Undelete photos from old web site

Updating your working copy

Now we need to update our working copy from our local branch. We do this by changing to our working copy and executing an svk up command:

rbieber@compaq:~/svk/website> svk up
Syncing //bieberlabs/new-feature-x/website
(/bieberlabs/new-feature-x/website)
in /home/rbieber/svk/website to 409.
U   wordpress/index.php
A   photos
A   photos/raytracing
A   photos/raytracing/yinyang2.jpg

Wrap Up

This article has detailed the process of keeping your local mirror up to date with activity going on in your production Subversion repository. Keep in mind that all of the changes have happened on your local machine and that the main repository has not yet been updated. This process can be repeated as often as you like until such time as you are ready to merge your local changes back into the production repository, which will be covered in part 3 in this series. The real beauty of SVK that this article should illustrate is that at no time were we mining for specific revision ranges to merge. Using the smerge command this is all taken care of for us. The software knows about every merge that has happened between your local mirror and your local branches. When we smerge our changes back to the production repository, you will also not be mining for revision numbers. SVK knows what has been merged to your local branch and will automatically skip them on our merge back to the repository.

It is also very important to point out that the only time in which network connectivity was required was during the initial synchronization of the mirror (the execution of the svk sync command). All other actions were performed disconnected from any network.

In part 3, we will cover the final portion of our process, which is taking our local branch and merging our work back to the repository.

Back to Part I | On to part III

Posted in SVK

SVK – Distributed Version Control – Part I

Just when you thought version control couldn’t get any cooler (and we all think it’s cool, right??), along comes SVK. SVK is a distributed version control system (a la Bitkeeper) written by Chia-liang Kao. The tool allows you to mirror existing remote repositories, create branches on your machine, work on these branches locally, and when you are ready, merge them back into your mirrored trunk, which transparently updates the remote repository.

I have spent only a small amount of time with SVK, and have limited knowledge thus far on it’s capabilities. However, in the couple of days that I have been playing around with it, I prefer it over straight Subversion for a number of reasons including:

  1. Full local repository on my machine – I can commit to save, back out, or merge code as if I was sitting at my desk, even in the most remote location.
  2. No more long URLs – Once you mirror the repository, svk aliases your repository to something short and more manageable for those of us who hate typing.
  3. Incremental synchronization of mirrored repositories. Depending on the size of your repository the initial synchronization can take a while. Subsequent syncs are very fast though
  4. Repeated merge support – SVK implements the ‘star-merge’ algorithm introduced in arch – and does it without being as cryptic. It keeps track of merges that have already been done from branch to branch and eliminates the need to skip synchronizations of the trunk to the development branch when merging your changes back into the main line. So, you can synch your mirrored branch, merge the changes into your local branch, and keep working. When you merge your changes back, the tool knows which revisions have been merged from the destination and doesn’t merge them – reducing a load of conflicts and / or manual work in figuring what to exclude.
  5. Simple command set – the commands ‘mirror’ Subversions command set. The whole mirroring of the repository and the merge syntax can be intimidating at first, but once you do it a few times it makes complete sense.
  6. Lightweight Workareas – SVK does not use the .svn directory in the local workarea. The workarea is very lightweight and does not use the space that the ‘vanilla’ Subversion tool uses.
  7. The freedom to walk away knowing that if you don’t have a network connection, you are not dead in the water.

This is the first in a series of articles that will build on one another to show the capabilities of SVK as I learn them. Just in the last couple of days, I’ve done enough to know that it cannot be covered in one article. I hope you enjoy these and stay tuned for Part 2.

Why Distributed Version Control?

The first question people might ask when talking about a tool like SVK is “What does distributed version control get me?”

Well, consider this simple but common scenario. You have a deadline that you have to meet but hate being in the office late at night. You would rather be sitting on your couch watching the latest installment of Mythbusters and cutting code. Or better yet, you would like to leave the office for a bit to get away from interruptions and go to the nearest Starbucks, have a coffee and work there. Unfortunately, you need a connection to the source repository in order to continue your work. In either scenario, your life would be more pleasant being somewhere outside your office, but you need the full source repository in order to perform your work (such as merging or backing out some ideas you were trying out). This might sound a little rediculous, but it is a real scenario, articulated very well in a user story posted to the Subversion development list by by Eric Raymond.

The ideal situation is to be able to mirror the repository (or branch you are working on) on your local machine in a working repository, disconnect from the network, continue your work, and later be able to synchronize your local repository with the production repository. This is what you get with SVK.

Environment Notes

Everything done in this article was done on a Compaq Presario 3000 laptop running SuSE Linux 9.1. There are currently Windows binaries for the SVK tool, however I do most of my work on Linux, and cannot speak to the installation complexity or performance of the tool in the Windows environment.

I have Subversion 1.1.1 installed with both Perl and Python bindings. The Subversion server is running over http/DAV.

The version of SVK I have installed is 0.26.

Notation

Some of the lines in the command are truncated because they are too long. I have used the ‘_’ character as a line continuation character. If you see this character, look at the next line.

Installing SVK

The tool is written in PERL, and consequently is kind of a pain to install. I took me a bit to whittle down the install to the commands you will see below, but I’m one of those people that RTFM as a last resort. You can find detailed install instructions on the SVK Wiki.

First you have to make sure you have the Subversion perl bindings installed. I didn’t and when I initially tried to install SVK from CPAN I had to scour the output when an error occurred to figure this out. However, since I build subversion from the source tarballs I had my last build lying around and was able to type the following to get the bindings installed:

make swig-pl
make install-swig-pl

Once the bindings were installed, installation of SVK turned out to be a matter of installing the latest version (and all of it’s many dependencies) from CPAN with the following command:

perl -MCPAN -e 'install SVK'

This one takes a while, so relax a little bit (after making sure to answer all the prompts to install the many dependencies).

Setting up your local ‘depotmap’

To set up your local repository, type the following command:

svk depotmap --init

This command will check to see if you have a local repository, and if not, prompt you to create it. Answer in the affirmative.

Mirroring your Repository

Mirroring your repository consists of two commands. First, you have to tell SVK what repository path you would like to mirror and what you would like to refer to it locally. The root of your SVK repository is referred to as ‘//’. So to set up a mirror for my web site repository, I do the following:

svk mirror http://subversion.bieberlabs.com/svn/bieberlabs/trunk //bieberlabs/trunk
Committed revision 290.

To ensure that my mirror has indeed been set up, I can use the svk mirror command to list my current mirrors:

svk mirror --list
Path                  Source
===============================================
//bieberlabs/trunk      http://subversion.bieberlabs.com/svn/bieberlabs/trunk

Now that we have a mirror set up, we need to sync our local copy with the remote repository:

svk sync //bieberlabs/trunk
Syncing http://subversion.bieberlabs.com/svn/bieberlabs/trunk
Retrieving log information from 1 to 12
Committed revision 291 from revision 1.
Committed revision 292 from revision 3.
Committed revision 293 from revision 4.
Committed revision 294 from revision 5.
Committed revision 295 from revision 6.
Committed revision 296 from revision 7.
Committed revision 297 from revision 8.
Committed revision 298 from revision 9.
Committed revision 299 from revision 10.
Committed revision 300 from revision 11.
Committed revision 301 from revision 12.

At this point, we can disconnect and go sit on the couch, turn on Cold Case Files, and get some real work done.

Creating a Local Branch

The first thing we want to do before we start working is create a local branch to work in. Doing this is the same as you would do in Subversion, the difference being the paths that you will specify. For this change we will do something simple just to illustrate the point.

svk cp -m "Create local branch for new feature X"  //bieberlabs/trunk //bieberlabs/new-feature-x
Committed revision 302.

Yes, it’s that simple. Since SVK uses the Subversion filesystem to do its work, this operation is very fast. Let’s check to see that we actually have a branch. After all, we’re on the couch and not at our desk, remember?

svk ls //bieberlabs/new-feature-x
network/
qf/
website/

Checking Out Our Development Branch

Let’s check out the web site code and make some changes. I’ve created a directory called svk in my home directory in which I will place all of my workareas. Checking out a workarea is the same as doing it with Subversion. The main difference is that you will be using ‘svk’ as the command and your repository path will be the name you used during the copy (remember the name starting with “//”??):

svk co //bieberlabs/new-feature-x/website
Syncing //bieberlabs/new-feature-x/website (/bieberlabs/new-feature-x/website) _
in /home/rbieber/svk/website to 302.
A   website/wordpress
A   website/wordpress/wp-config.php
A   website/wordpress/wp-rss.php
A   website/wordpress/print.css
A   website/wordpress/styles
A   website/wordpress/styles/panther
A   website/wordpress/styles/panther/style.css

A   website/wordpress/wp-admin/edit-form-comment.php
A   website/wordpress/wp-admin/edit-form.php
A   website/wordpress/wp-admin/import-mt.php
A   website/wordpress/wp.php
A   website/wordpress/wp-feed.php
A   website/new.html
A   website/index.php

… and we’re done for now

So far we have installed SVK, created a mirror of our source repository, synchronized it, created a development branch, and checked it out to a workarea that we can begin coding in. Hopefully, in the work done so far, you can already see SVK’s usefulness. Just wait — it gets bettter. In Part II we’ll actually start using the tool to help us manage our work — all in front of the TV with no network.

Stay tuned.

On to Part II …

Posted in SVK