While browsing Slashdot this morning, I found an article that states that “95% of all IT groups are not delivering some amount of projects on time or to the full satisfaction of the business executive”. The most interesting part of the article for me was that while a survey of the projects stated understaffing, unrealistic timeframes, and poorly defined project scope as the cause, the article takes a different perspective and talks about how much of the “lateness” could account for vendors overpromising the ability of their software in order to make sales, causing IT departments to not be able to deliver.

This is an interesting perspective, but I have another one.

Wednesday night we watched The Incredibles. The kids had set some pretty high expectations for the movie, but I went in skeptical. I didn’t think I was going to like it, but I went in with an open mind. I sat through the movie and absolutely loved it. Great story, great animation. The movie had everything.

Once the movie was over, we went through some of the extras. One was a “making of” in which the guy in charge of the project was talking about how difficult it is making movies, because all you have is an idea and you aren’t sure how you are going to do it. You are inventing things as you go along. He then compared it to an assembly line, where things are predictable, and stated that movie making like this wasn’t the same thing.

This statement hit me pretty hard, and I remember saying to myself, “That’s what we do in IT”.

I believe that the main problem with IT projects being late is that executives fail to realize that the development of an IT system is new product development, not assembly line work. IT is treated as though we are making cars, rather than as if we are creating something that hasn’t been done before. The funny thing is that if it had actually been done before, why aren’t they buying it rather than developing it themselves?

As I thought more about this over the last couple of days, and then read the article mentioned above, I realized that you never hear this kind of talk about Open Source projects. I remember when Subversion was at 0.34 and the only talk you ever heard about going to 1.0 was for people, like me, whose management wouldn’t settle for using software that was not 1.0. Otherwise, I and other people on the list were actually using it for production work, we just didn’t tell management we were doing so.

The same goes for SVK. I have started to have my staff look at it even though it not a 1.0 product yet. Why? Because it is useful the way it is. The “release early and often” model, and the responsiveness of the two development teams mentioned above make it irrelevant to anyone except the executives as to whether the product is 1.0. You know there are new features coming. You know there are bugs that will be fixed if you report them. The project is treated, by it’s developers and it’s users as an organic thing that grows over time, but is still useable in its current state.

Software projects do not have a defined ending. They are never done, they just change state. There is a lifecycle involved in software development and by the time a software project actually “ends” it is retired. Software development is a creative and ongoing process. This is why we have version numbers.

I think the main problem with IT “being late” or “not meeting expectations” lies with the people setting the expectations, not the people doing the work. Executives are stuck in a model that doesn’t work, and they are afraid to acknowledge it because it challenges everything they have been taught as executives. However, if they would spend some time in IT, they would realize that software projects never end, and it is quite difficult to meet expectations that do not understand the underlying problems or process involved in the software lifecycle.

Also, in order to incrementally develop software in a way that adds value, priorities for functional requirements must be set. The persons requesting the work would have to decide what is most important to have now, rather than just saying they want everything, now.

The way to do software development where everyone wins is to acknowledge that it is an ongoing process and release early and often, prioritizing the work based on what is most important at the time you set the priorities. You must also, as a business user, have the flexibility to change the priority at the end of each incremental release. Doing this allows you to begin to realize value from the project sooner, and the stress level goes down because you know there is always work to do, but it will get done. Corporations need to begin to acknowledge that the open source model works for a reason and that if you want your projects to be “on time” you have to redefine what your meaning of a software project is.

This is agile software development.

Eric Raymond as a great article called the Cathedral and the Bazzar, which explains the software development model used in the open source world. I would encourage everyone involved in software development (whether you are doing it or you interface with IT to get it done) to read it and really think about what your expectations are around the IT organization. If you don’t like reading these kinds of things online, you can always buy it at Amazon or pick it up at your local book store.

The bottom line is that in order for executives to feel that they are getting their moneys worth from IT, they have to acknowledge that the work is different than they think it is. The problem, in my opinion, has less to do with IT meeting expectations as it does the expectations being set based on a false set of assumptions.

Now, how to teach executives this is another story altogether. When I figure it out, I’ll let you know.