Implementing Open Source Defect Tracking as a Corporate Tool – An Update

Note: This article was started around March or April of this year – and I just got the motivation to finish it – because I thought it was important. I hope I did the original idea justice all these months later.

Back in October of 2007 I had written about starting to implement the Eventum Defect Tracking System in place of the defect tracking system that we had been using for about 8-9 years. I thought I’d throw up an update on how we’re doing, and what advantages we are seeing from taking the Open Source route over the proprietary route.

Let me start off by saying that the more we use Eventum, the more impressed I am with the thought that went into the development of the application. There were only a few minor tweaks that we had to do in order to get it working in a way that we could be much more productive. These tweaks were things like enabling basic LDAP authentication on the application and most recently enabling email integration and source control integration, which are actually already implemented in the system. Some little tweaks for our environment to alleviate the change required were necessary, but as you will see, these tweaks were small.

Basic Install and Feedback Process

I had the basic install of Eventum up in about thirty minutes. I then sent the link to the application out to small pieces of the team in order to receive targeted feedback. Most of the feedback revolved around things that were missing that people were used to in the current application – but these things were largely why I wanted to remove it in the first place. They caused a lot of extra work. I made some tweaks to add custom fields, and sent it out to another small group. Once everyone had a chance to play with it a bit, I assembled the team to talk about what we needed to do to cut over. Much of the conversation was around how the new process would work and some were around tweaks that I had made that people didn’t feel we needed (which came directly from feedback from the team). Some of the tweaks were undone and we decided to move ahead with cutover on the next build cycle.

A conscious decision was made to just get the system up and people to start using it without all of the bells and whistles like email integration, which I would add incrementally as I had the time to do so. The next section will talk about some of the things we did in order to expedite the enabling of email integration.

Some of the initial feedback that we received on the tool was that the “screen was too cluttered” or people didn’t understand the states. Much of this, I knew, was simply because the tool was unfamiliar. People do not like change. I spent a lot of time reassuring people and asking them to “hang in there” as we familiarized ourselves with the tool and the new processes that would evolve through its use. To their credit, they did.

Enabling Email Integration

One of the reasons I chose Eventum from the many defect tracking systems I looked at was the capability to integrate email conversations at the ticket level. In April of 2007, I wrote an article called Metrics as a Side Effect in which I talked about how I would love to see metrics be able to be updated without having to make a concerted effort to switch contexts in order to do so. I viewed conversations around a particular ticket the same way – and wanted a system that would record all email conversations with the ticket so that we had something to refer to as things went live, were closed, or were killed. The thought behind this is that if you are required to go into the system in order to document a conversation, chances are you won’t. The conversation needs to be recorded as a side-effect of normal work activities.

One of the killer features Eventum offered was this capability. Unfortunately, the documented ways to enable it were a little unclear and required changes to the mail server in order to enable it. Eventum uses custom email addresses like issue_123@example.com or note_123@example.com to do email association. Common solutions require setting up blanket aliases (issue-*) or using ‘+’ addresses in your mail server to be able to intercept these issues. Also, POP3 or IMAP mailboxes are required. Neither of these were options for us, as we do not control the email servers. We had to find a work around – since we only had one email address for our “build system” and I didn’t want to have the myriad of conversations that I knew I would have to have to create these blanket aliases on the corporate server.

As it turns out, tweaking Eventum only took commenting out a few lines in the notification class and a small procmail recipe to allow the integration without touching the existing mail server configuration.

The code changes look like this, starting at line 159 in class.notification.php and consisted of commenting out 5 lines:

  // RCB:  Don't use custom from addresses
	
  //      if (@$setup[$routing]['status'] != 'enabled') {
            // let's use the custom outgoing sender address
            $project_info = Project::getOutgoingSenderAddress($project_id);
            if (empty($project_info['email'])) {
                /// no project email, use main email address
                $from_email = $setup['smtp']['from'];
            } else {
                $from_email = $project_info['email'];
            }
   //     } else {
   //         $from_email = $setup[$routing]['address_prefix'] . $issue_id . 
   //                               "@" . $setup[$routing]['address_host'];
   //     }
        if (empty($info['sender_name'])) {
            // no sender name, check if this email address belongs to a user and if so use that
            $usr_id = User::getUserIDByEmail($info['email']);
            if (!empty($usr_id)) {
                $info['sender_name'] = User::getFullName($usr_id);
            } else {
                // no name exists, use email address for name as well
                $info['sender_name'] = $info['email'];
            }
        }

These changes force the application to, instead of setting the outgoing sender address to issue-1234@example.com, to send it using the default project email address.

The next problem to face was intercepting these emails and processing them as Eventum would when pulling them from a POP3 or IMAP inbox. Luckily, our system uses procmail quite extensively to process emails for software deployments, and other update notifications for things like branches that are created, etc. We just had to add two procmail receipes and a perl script that looks like the following:

:0
* ^Subject:.*[#[0-9]+] Note: .*$
	| $HOME/preprocessEventumMessages.pl | /usr/local/bin/php /path/to/eventum/misc/route_notes.php

:0
* ^Subject:.*[#[0-9]+].*$
	| $HOME/preprocessEventumMessages.pl | /usr/local/bin/php /path/to/eventum/misc/route_emails.php

These recipes look at the subject and check to see if there is an issue number in it – as well as whether it has the word ‘Note:’ in it. If they do, they pipe the message to the following perl script, which acts as an email filter, translating the from address from our blanket address to the issue specific address and in turn piping it to the Eventum script to process the email:

#!/usr/bin/perl
#--------------------------------------------------------------------------------------------
# preprocessEventumMessages.pl
#
# This script is called by procmail when Eventum recipes are fired.  It pulls the 
# issue id from the subject line and manufactures an issue or note email address, basically
# faking Eventum into thinking that it is routing an issue specific email, without having to 
# change the mail server.
#
# This is basically a filter.   It reads from stdin and writes to stdout, so that procmail
# can process the message and send it to the appropriate routing script.
#
# The procmail rules look like this:
#
# :0
# * ^Subject:.*[#[0-9]+] Note: .*$
#   | $HOME/preprocessEventumMessages.pl | /usr/local/bin/php /path/to/eventum/misc/route_notes.php
#
# :0
# * ^Subject:.*[#[0-9]+].*$
#   | $HOME/preprocessEventumMessages.pl | /usr/local/bin/php //path/to/eventum/misc/route_emails.php
#
#--------------------------------------------------------------------------------------------
@contents = ;

# grab the full email into one big buffer ...
$mailContents = join('', @contents);

# grab the issue number and note designation (if present)
if ($mailContents =~ m/Subject:.*[#(d+)] (Note:)*.*$/mi) {
    # strip the issue number of leading and trailing spaces
    $issueNumber = $1;
    $issueNumber =~ s/^s+//sig;
    $issueNumber =~ s/s+$//sig;
    
    # if the note designation was in the subject, create an email address of 
    # note_xxx@example.com.   If not, its an email, so use issue_xxx@example.com
    if ($2 eq "Note:") {
        $newEmail = "note_" . $issueNumber . "@example.com";
    } else {
        $newEmail = "issue_" . $issueNumber . "@example.com";
    }
 
} 
        
# Find the To: header and replace the umbrella email address with the issue specific
# one and write the email back out to stdout line by line.   This will then be fed
# by procmail to the appropriate routing script (see module comments above)
foreach (@contents) {
    if (m/^To:/sig) {
        s/blanket-email@example.com/$newEmail/sig;
    }       
    
    print $_;
} 

The Eventum settings are set just like you are using address based routing, and these scripts do the rest to fake it.

Now, once again, modifying the code directly – probably not the best way to do it. But it was the fastest and allowed us to make changes independently, without having to make changes to the corporate mail server.

Now we did find another issue once email association was working. There were some emails that were coming in that had pieces of the issue id missing. So we had one more tweak to make, again in class.notification.php in the block starting at line 952:

if ($type == 'notes') {
$extra_subject = $data['note']['not_title'];
// don't add the "[#3333] Note: "
// prefix to messages that already have that in the subject line
if (strstr($extra_subject, "[#$issue_id] $subject: ")) {
// RCB: 12-13-2008: This code was lopping off the issue id, which breaks email
// association.
//$pos = strpos($extra_subject, "[#$issue_id] $subject: ");
//$full_subject = substr($extra_subject, 4);
// keep the full subject instead
$full_subject = $extra_subject;
} else {
$full_subject = "[#$issue_id] $subject: $extra_subject";
}
} elseif (($type == 'new_issue') && ($is_assigned)) {
$full_subject = "[#$issue_id] New Issue Assigned: " . $data['iss_summary'];
} else {
$extra_subject = $data['iss_summary'];
$full_subject = "[#$issue_id] $subject: $extra_subject";
}

I’m not entirely sure what this was doing and why the hardcoded ‘4’ is in line 959, but its definitely a bug, which I submitted to the Eventum project as a fix. I’m not sure if it was ever corrected.

The benefit that these changes give us, however, is that now when an internal note or email is sent from the Eventum tool and an email is sent out, the default behavior of people to respond to the email rather than go into the tool still results in an updated conversation about the item in question. No longer are decisions made without documentation – the documentation is created as a side effect.

Enabling Source Control Integration

The final piece to the puzzle was source control integration. We use Subversion for our source control system and ViewVC as a means to browse the source code (both using mod_ldap and mod_svn_authz to authenticate with the corporate LDAP store). Eventum includes a script that you can use in your Subversion post-commit hook to log each commit targeted to a particular issue (specified in the commit comments) that automatically logs commits to a particular issue. You can then set up patterns to build URL’s to your repository so that from each issue a user can view the particular commits through this URL. We set this up to link to our ViewVC application so that each commit will link to the diff of the commit. This allows us to streamline our code review process by being able to view each commit to an issue in context. This has proven to be a huge win – at least for me when I want to see what is going on for a particular issue.

Getting The Business Users Involved in the Process

The biggest benefit I believe we have received from switching to Eventum is that, while it looks complex at the beginning, once you start using it it is quite easy to learn. So much so, that for the first time that I can remember, we have our business users engaged in the defect tracking system to approve work that they feel are a priority. Some of this is still a manual process, but being able to distribute the entry and maintenance of items has been a big win for us, removing a bottleneck that we had of one person on the team designated as the “issue generator” for other teams. We now have our business users engaged in the process – everywhere from entering tickets themselves, to responding to issue emails and having the discussions automatically logged for historical purposes. This is something we had tried to do with our previous tool, but was unsuccessful due to the complexity of the tool and licensing costs involved in deploying the tool (or licenses) to people outside of our group – since ultimately these costs hit our (IT’s) budget. We can now add as many users to the system as we want and let them begin engaging in the actual work being done – rather than being passive participants.

Huge win.

Upgrading

We started with Eventum 2.0.1 and had multiple “projects” set up. The main project was the work queue for the group, with a few specialized projects that might have unique items, or may have to be re-entered into the main project for the development team to have visibility to (each group wanted to work from one specific queue – not check multiple projects for work to be done). The 2.1 release of Eventum included the functionality to be able to move items between projects – which for us was a big deal – as we were re-entering items when they were applicable to the main project and not specific to the sub-project they were entered into.

With this one feature alone, its given us the ability to set up multiple work queues, in which we can begin to segment “nice to haves” from musts – and move these “nice to haves” into the main work flow when they become things our users want to address. This will, in the future, also be a big win for us, as we can segment the demand and begin to have the development teams focus on the things that are important now, while enabling our business users to continue to queue work for us without creating additional noise in the main work queue.

Conclusion

For us, moving from proprietary software with licensing limitations to open source software has been a big win in changing the way we work. The absence of licensing costs has enabled us to engage our users in ways we couldn’t before, while the ability to integrate email and source control has enabled us to collect information as a side effect of our teams normal work process rather than adding additional record keeping work to their duties. The fact that we have the source code has enabled us to customize the software in small ways to our environment, avoiding cross functional work that could take longer to implement due to the additional hand offs between groups – and there is always the possibility that a change to the email server could be viewed as something that shouldn’t be done. Access to source code allowed us to work around that.

Overall, in my view, the implementation of Eventum for work tracking has had a huge positive effect on the way we work. Hopefully, this article summarizes the benefits that we’ve seen in a way that encourages people to give open source a shot in their corporate environments.

I would encourage members of my team to respond to this post as well to keep me honest. What I see may be very different than how it works “in real life” – but I think what I’ve documented here is pretty accurate.

LDAP Enabling The Eventum Defect Tracking System

Due to a recent reorg, I have the opportunity to replace our defect tracking system, which has quite a bit of really wasteful process baked into the tool, with a new one. I’ve been looking at defect tracking software for a while, and chose Eventum, an open source project by MySQL AB for a number of reasons, some of them including:

  • Its open source
  • Its written in PHP, so I don’t have to worry about messing with fastcgi, mod_perl, or mod_python
  • It is extensible (you can add custom fields, etc)
  • It uses MySQL, rather than SQLLite or something like that, so we can integrate it into the rest of our home-grown build software
  • It supports email integration. While we won’t be using this right away, we’ll be implementing it in a later iteration
  • Its simple to use, with a very simple interface, once you get use to it. Everything is essentially on one screen.
  • It has time tracking, along with some basic reporting built in

One thing it doesn’t have built in is LDAP authentication. I wrote a previous article about all of the work we’ve done to integrate both our home grown applications and a few open source applications in with our LDAP store, to minimize the management of multiple passwords across systems, so this was very important to me. I started with many, many Google searches to see if someone else has done this, only to hit one dead end after another. At first I was being lazy and decided to just forget about it. One system not tied to the LDAP tree isn’t that big of a deal, but then my perfectionism set in. Why would I settle for that when LDAP authentication should be really easy to integrate into an Open Source package?

So I decided to spend a few hours to get it working. Since I had no success finding an implementation, I figured I could do my part and post what I have. There are a couple of caveats that I want to throw out before we actually get to the code though:

  1. It isn’t done “right”. This is all extra work for me, so I got enough done so that it would work. The right way to do this would to refactor the auth stuff out into a workflow like hierarchy that could be pluggable (see this post in the eventum mailing list). I’ll get to that someday, but right now this solution hacks the auth module to get authentication working.
  2. LDAP Settings are not configurable through the interface. I don’t have time for that, so a set of defines at the top of the LDAPAuthenticator class contains all of the configuration information for the LDAP server. Bummer, but like I said, I’m on a schedule.
  3. Users still have to be added to the Eventum database – they are not added automagically when they log in. I want control of who is in the system, so I’ve elected to leave this functionality out and just do authentication.

With these three caveats in place though, given my experience looking around for this stuff, at least this code works and will be able to be used by others. Its a starting point – which is more than is out there today. Anyone is free to use this and take the time to do it right. With that said, I’d love to receive updates if someone actually takes this up. For now though, this works for me.

So, now to the code. I wrote a small PHP class called “class.LDAPAuthenticator.php. There are two functions in it. Because Eventum uses email address as the login, we need a way to get the full user DN from the email address. This is what the email_to_dn function does. Given an email address, it returns the full distinguished name of the user. This is called by the main class function, ldap_authenticate. The ldap_authenticate function takes the same arguments as the class.auth.php function isCorrectPassword, which consist of the email address and the password. It binds to the LDAP authentication tree using the full DN of the user and the password supplied. If authentication is successful, it returns TRUE, otherwise it returns FALSE, just like the isCorrectPassword function used to validate the password from the Eventum database.

The code looks like this:

# Change these values to access another LDAP server.
define("LDAP_PORT", 636);
define("LDAP_HOST", 'ldaps://ldapserver.example.com:' . LDAP_PORT);
define("LDAP_BIND_DN", 'PUT THE BIND DN HERE');
define("LDAP_BIND_PASSWORD", 'PUT THE BIND PASSWORD HERE');
define("LDAP_SEARCH_DN", "PATH OF THE TREE TO SEARCH FOR USERS");

class LDAPAuthenticator {

# Look up a users full distinguised name from
# their email address, since Eventum uses
# email address as the login name.
function email_to_dn($emailAddress) {
$returnDN = "";

$server = ldap_connect(LDAP_HOST);

if ($server == FALSE) {
return($returnDN);
}

ldap_set_option($server, LDAP_OPT_PROTOCOL_VERSION, 3) ;

$ldapbind = ldap_bind($server, LDAP_BIND_DN, LDAP_BIND_PASSWORD);

# verify binding
if ($ldapbind) {
# find the user based on the entered email address.
$result = ldap_search($server,
LDAP_SEARCH_DN,
"(&(mail=$emailAddress))",
array("dn"));

$info = ldap_get_entries($server, $result);

# if we actually got a value back, return the users DN
if ($info["count"] > 0) {
$returnDN = $info[0]["dn"];
}

ldap_unbind($server);
}

return($returnDN);
}

# Authenticate with the LDAP server. Function returns true
# if authentication was successful, false otherwise.
function ldap_authenticate($email, $password) {
$returnValue = FALSE;
$userDN = LDAPAuthenticator::email_to_dn($email);

if ($userDN == "") {
return($returnValue);
}

$server = ldap_connect(LDAP_HOST);

if ($server == FALSE) {
return($returnValue);
}

ldap_set_option($server, LDAP_OPT_PROTOCOL_VERSION, 3) ;

$ldapbind = ldap_bind($server,
LDAPAuthenticator::email_to_dn($email),
$password);

if ($ldapbind) {
$returnValue = TRUE;
ldap_unbind($server);
}

return($returnValue);
}
}

Save this file as class.LDAPAuthenticator.php and put it in your Eventum includes directory. Modify the define statements at the top to contain your LDAP server information.

Now, to use it. Go to your Eventum includes directory and add the following line to the top of the class.auth.php file:

require_once(APP_INC_PATH . "class.LDAPAuthenticator.php");

I have this at the end of all of the rest of the require statements.

Now, replace the isCorrectPassword function in class.auth.php with the following function:

 /**
* Checks whether the provided password match against the email
* address provided.
*
* @access public
* @param string $email The email address to check for
* @param string $password The password of the user to check for
* @return boolean
*/
function isCorrectPassword($email, $password) {
return(LDAPAuthenticator::ldap_authenticate($email, $password));
}

… and VOILA. You can now authenticate off of your LDAP tree.

Now, I know it isn’t pretty, hacking the code directly – but it works, and its more of a starting point than I can find anywhere else. I hope its useful to others. Again, if anyone takes this further and does it “right”, I would be really happy to get a copy of the modifications.

One more thing – don’t forget to require SSL on the URL to your Eventum installation by using the SSLRequireSSL directive in your Apache server. You don’t want these passwords floating around in the clear across the network.

Download the Eventum LDAP hack here and happy authenticating.

Subversion, MediaWiki, Wordpress, and LDAP

One of the biggest arguments you’ll get in deploying open source software in a corporate environment perception that they are extra, standalone applications. If your corporation uses an LDAP server, you can get some big wins by ensuring that your open source applications can authenticate with your corporate LDAP store, showing integration with the main systems.

I recently went through this exercise with a number of applications in our environment, including:

  • Subversion
  • MediaWiki
  • WordPress

I thought I’d throw up an entry here outlining the tools I used to make this integration possible.

Subversion was a no-brainer, since we host our repositories using mod_dav_svn. Configuring the mod_auth_ldap module in the Apache server and converting all access to SSL made this integration painless, once I figured out how to build Apache to use OpenLDAP and Secure LDAP. For MediaWiki, the Mediawiki LDAP Extension worked flawlessly. The key problem with Mediawiki is that there is no mechanism built in to ensure that logins are performed via SSL. A quick rewrite rule in the Apache server took care of this for me. A complete explanation of this process can be found at Library Web Chic.

For WordPress, I found a great plugin from Kane IT Consulting that was extremely easy to configure. I had the plugin installed and configured in minutes. I highly recommend this one. The Admin-SSL plugin, gave us the security around the login that we needed.

What has been interesting to me is seeing the subtle shift in perception of these applications as we integrated them into the authentication system. They almost seem like legitimate pieces of the system now … even to me.