Subversion and SSL Troubles

I decided to upgrade my home Subversion repository to version 1.4.3 as soon as it was released. Since then, my ViewVC application has ceased to work, getting a Python exception every time I try to execute it. Creating a small Python program that just imports the library (from svn import fs) gave me the following error:

ImportError: /usr/local/lib/libsvn_ra_dav-1.so.0: undefined symbol: SSL_load_error_strings

Thinking it was an SSL library problem, I upgraded SSL – a few times. I kept mucking with the options, rebuilding Subversion, only to get everything installed and get that same error:

ImportError: /usr/local/lib/libsvn_ra_dav-1.so.0: undefined symbol: SSL_load_error_strings

Over, and over and over again I repeated the process and got the same result. The absolute definition of insanity. This has been going on for a couple of months and I’ve been trying to address it in my spare time, as I’ve been pretty busy lately during the week and gone to the Relaxation Unit the last few weekends.

I googled my ass off to find the error, but to no avail. Finally today I ran across this thread that explained the problem. After going through my distribution directory for 1.4.4 (which I upgraded at the beginning of the month only to receive the same error) I realized that I hadn’t pulled down the Subversion dependencies tarball and rebuilt neon. So, basically I was using an old version of the neon libraries.

I finally settled on the configure statement listed here, after downloading and untarring the deps file:

./configure --with-ssl --with-apxs=/usr/local/apache2/bin/apxs 
            --with-apr=/usr/local/apache2 --with-apr-util=/usr/local/apache2 
            --enable-shared --with-libs=/usr/local/ssl

This uses the already installed apr libraries that I build with my Apache server, and ensures that the neon shared libraries are built. A quick configure/make/make install/make swig-py/make install-swig-py sequence later and my Python libraries were working fine.

I made it a point this time to document this on the Labs internal wiki, but thought I should throw this out here in public so that others can find it. Hope it helps save the weeks of frustration that I have been suffering for someone out there.

Happy building …

Building Scalable Web Sites by Cal Henderson

I have about three books that I am reading on and off but have been unable to focus on any of them for any length of time. Tom The Architect mentioned a book to me a few months ago called Building Scalable Web Sites: Building, Scaling, and Optimizing the Next Generation of Web Applications by Cal Henderson, engineering manager for the Flickr photo service, a service that I have used extensively since being turned on to it by, you guessed it, Tom The Architect.

This was the first book in a long time that I couldn’t put down, mainly because everything in the book is geared towards teaching you about how to create really, really, big web sites and the issues involved in scaling them. It was also quite intriguing because the book covers tools you use all of the time, like PHP and MySQL that are hard to find really good books about how they scale.

Cal covers a lot of material in this book, from layering your web application architecture, to creating an environment for developers to work in, which includes source control, issue tracking, coding standards and the like. This section was quite encouraging to me, as we have implemented almost everything that Cal mentions in the book (sometimes its nice to get some external validation). Cal then goes on to talk about internationalization and localization, data integrity and security, using email as an alternate entrance into your application, and how to build remote services.

All of this was great, but the next few chapters I found really valuable. Cal talks about identifying bottlenecks in your web application, scaling applications such as MySQL (where he covers quite a few replication strategies) and scaling storage. He also covers measurements, statistics and monitoring. Finally, Cal talks about adding API’s into your application to support mobile applications, web services, etc.

Cal references quite a few tools that are freely available in these discussions – tools that I didn’t even know were out there, that you can use to simplify your monitoring environment. I was most intrigued with the Spread Toolkit, a self described “a unified message bus for distributed applications” that allows you to unify logging across your applications. Anyone who has tried to debug an issue on a site that has more than one box would appreciate knowing about this tool.

This is the first book that I’ve read in a long time, technology wise, that hit the sweet spot between talking about real issues that I have been facing and possible solutions. I highly recommend grabbing this book and in the very least just keeping it on your book shelf for future reference. This is one thats going to be a constant companion for me in the coming months.

The Shoemakers Son Always Goes Barefoot

The other night the ignition switch on the furnace went out in the house. I watched as Jonna spent a ton of time searching for the contact information for the guy who came out the last time we had a problem. It took quite a while to find the information, but finally she found it. When she got a hold of him, he started asking questions about a blinking light on the furnace. We had no idea what he was talking about, but I did remember he gave us information last time he was here – I just couldn’t remember what it was.

Yesterday as I was driving to work, I was reflecting on the activities of the night before. Why did we not have this information available when we needed it? Where could we put it so that if something happened again, we could have it readily available? How can we take these kinds of notes effortlessly and ensure that we know where we put them?

Then a stark realization hit me. We’ve already solved this problem – at work.

In early 2004, at the urging of one of my direct reports, we installed wiki software at the office to solve just this problem. All of our information was scattered around network drives, none of it really searchable. Doug was very into Python at the time so we chose ZWiki, a wiki package that runs on the Zope application server. We used that for about 1 1/2 years until we finally bit the bullet and moved to MediaWiki, where our information repository lives today.

We actually have quite a knowledge base going there now, everything from detailed process information, to configuration information, to even some projects that are being managed on the platform, with detailed information about all of the issues encountered, configuration information, and the like. It has become a one stop shop for all information related to our environment.

And I’ve been the primary champion since it was installed.

This was when, as I was sitting in the car pondering this, that the title of this post came to me. The old adage is true. There are so many problems that we solve in our daily business lives that never get resolved in our personal lives, and vice versa. Its amazing to me that while we’ve done so much at work to centralize the information in our department (while decentralizing the authoring so that if something is found to be wrong it can be corrected) that I never thought to apply this at home to keep all of our information straight here. Instead, Jonna spends countless amounts of time searching through kitchen drawers for information on service providers and I sit trying to remember that one valuable piece of information that the furnace guy absolutely needs so that he can arrive and fix the part, rather than wasting trips to and from our house to first diagnose the problem, then go get the parts to fix it.

So, I’ve spent this morning getting MediaWiki running here at the Labs. Hopefully, I can motivate the family to use it as we have motivated our employees to use it at work to keep all of our important information centralized and updated. Its a simple thing to set up, but can be rather difficult to socialize. Luckily, we only have 5 people here, so the socialization might be a tad bit easier to do.

How many things do you struggle with at home that have been solved for years at work? Maybe you even had a hand in solving them, but the solution never seeped into your life outside of work?

This was a major “AHA” moment for me this week and I’d love to hear about other people who might have similar stories.