Mitigating Software Project Failure using Lean Startup Techniques

Introduction

Most software projects fail.

According to the Standish group, the software project development failure rate has been around 70% over the course of the last 20 years. This is a staggering statistic. There is no simple solution to ensure software development success, but a lot can be done to mitigate the failure rate.

Most software projects are not the public web services we use everyday, and not shrink-wrapped software. Many of the projects that fail are custom software projects built for governments and corporations. The teams involved in these projects usually have little to no exposure to the Silicon Valley technology startup scene. Some of the greatest software successes have come out of the Silicon Valley, and I believe a lot of the problems that software projects encounter can be solved by following strategies that have become common-place in the Bay Area.

In 2011, Eric Ries, a Silicon Valley entrepreneur published a book called “The Lean Startup”, describing a methodology for building businesses and products. Although there is no silver bullet, I believe that following this methodology can mitigate most important aspects of development failure and increase the success rate of all software projects, even when they have nothing to do with startups.

Solving Problems with Lean Methods

One of the most common problems in failing software projects is the lack of clear requirements and specifications. This is a problem startups face as well. It is often uncertain whether the market wants the product that the startup is building or whether the project will ultimately turn out to be unsuccessful. The lean startup methodology suggests using the MVP (Minimum Viable Product) strategy to first test the market. A simple mock product is built with a very limited set of features, satisfying only the most important requirements, and shown to early adopters. Feedback is collected, and the potential of the product is then evaluated. Similarly, when the specifications are unclear for software projects, building a product akin to the MVP can be helpful in collecting early feedback from the clients, and then using the feedback to clarify the specifications.

Another common problem causing software project failure is the lack of project status updates. Clients cannot easily find out how much progress the development team is making. This is because it often takes a long time before developers can show the first version of the project in a usable state to the client. Ideally developers should build upon the MVP, continuously improving the product and collecting feedback from the client. The lean method recommends that startups implement continuous deployment to achieve this, immediately releasing a new version whenever any small change is made. In that way, any issues encountered by the client can then be taken into account as soon as they are noticed, rather than after the project is deemed finished.

Continuous deployment also allows the client to see how fast the project is developing. By measuring the pace of development, and taking note of which features still need to be implemented, it becomes easier to set a realistic estimate for the project. Obviously this does not completely solve the problem, but any estimation method is better than pulling the deadline out of thin air.

The project requirements may be unclear, but if a software development team has a way to continuously collect feedback from the client, based on it, the team can decide whether to pivot or persevere. As in, decide whether major changes need to be made to the ‘MVP’, or whether it is fine to develop the project further in the same direction. As long as the feedback from the client is taken into account, to some extent this shields the development team from the politics involved in project planning and management: every change is being made in order to satisfy the client. Rather than following some manager’s guidance on what feature to implement next, developers can decide on that simply by looking at the feedback: figuring out what clients complain about the most, what bugs they find the most annoying, and what features they demand the most.

Other common reasons for project failures are poor project management and sloppy development practices. If the management team is incompetent, there is not much developers can do about that, but at least when it comes to development practices, a lean methodology forces the development team to adopt agile development techniques. Surveys have shown that the usage of agile methods usually results in better software quality and productivity gains, hence, improving the chance of success for the project.

Conclusion

Clearly the lean startup method is much more general than you would expect it to be, and it can also be applied to non-startup projects. The methodology is proven to work, as many successful companies such as DropBox and Intuit were built upon it. The methodology ensures that the pace of development is fast and that client’s feedback is taken into account from the very beginning of the project. There is hope that as time passes more and more software teams will become aware of the lean method, and that many software project failures will be avoided by adopting it.

Response to “Hack and Slash” by s1371464

Introduction

This article is a response to Hack and Slash by s1371464.

The author of the article states that, based on his or her experience, it’s possible to save a late project by reducing the number of developers in the team, and claims that this is a preposterous but real proposition. I agree with the author’s proposition, but would like to clarify and explain the statements made, showing why the proposition is not preposterous at all.

Brooks’s law

The author states that the reasons for the project failing to meet the deadline, were:

  • too many developers assigned to the project;
  • a large code base consisting of heavyweight components.

Both of these are simply the consequences of Brooks’s law, as presented by Frederick Brooks in the book “The Mythical Man-Month”. Brooks’s law can be summarised as: “adding manpower to a late software project makes it later” – that, although the work to be done can be split among the new developers, the complexity of coordinating and merging all of the work will slow the progress of the whole project down.

Too much manpower

This simple law also applies to the situation detailed in the original article. The planning and estimation for the project were done inadequately: there was too much manpower assigned to the project from the beginning. The law is supposed to be applied to projects which are late, and obviously the project mentioned in the author’s article could not be late on the first day of its existence, but over time the unneeded complexity of coordinating work among a large number of developers clearly caused the project to miss its deadline.

The main factor of the Brooks’s law is communication overhead. With more people working on the same project, it takes more time to figure out where your place in the project is and what others are working on. Clearly, this was slowing the project down, as cutting down the team much improved the outcome.

Code base size

The other aspect discussed in the article, that caused the project to miss its deadline, was flaws in the project’s architecture. Namely, the software consisted of heavyweight components and the code base of the project was too large. This is also a consequence of communication overhead. As maintaining many inter-relationships across a large team is hard, developers tend to cluster into smaller groups which end up responsible for only a specific component of the project. Developers are well aware on what everybody is working on inside that cluster, and they believe they do not need to care about what is going on outside their own circle. Unless there are some pre-agreed notions about the architecture of the project, each of these developer clusters ends up building a very heavyweight component, to the extent that different teams sometimes end up with different implementations of the same functionality.

Conclusion

The problems that the author encountered are clearly the consequences of Brooks’s law. Projects with too many people assigned have a large communication overhead, which results in missed deadlines along with a flawed architecture. Since Brooke’s law is a well-known software development principle, I do not consider the author’s proposition to be preposterous, but just a simple observation.

Startups and Development Methodologies

Introduction

When it comes to early stage technology startups, software development is usually done in an ad-hoc way, without adhering to any specific rules. This is not surprising, considering startups usually have 2-3 co-founders, and not all of them are necessarily doing development. One of the co-founders is often non-technical, and handles the business side of the startup. Generally, you don’t expect kids operating a company from a garage to thoroughly follow a software development methodology.

google
Sergey and Larry were running Google from a garage and were lousy coders.
Fortunately they now have the resources to fix their code.

However, my experience working for a few startups, including one early stage startup, has shown that not adopting a software development methodology as early as possible causes a lot of headaches later. Despite the low number of people involved at the beginning, most startups develop both long-term and large-scale projects. Not adhering to any software development methodology when working on large projects leads to flaws in the implementation, buggy and poorly documented code, and loss of productive development time in the long run.

Startup Problems

At a startup most of the code ends up being written in an ad-hoc manner due to lack of direction. Whenever a customer requires a new feature, and the architecture is found to be unsuitable for it, the developer usually ends up implementing the feature as a hack, rather than properly integrating it into the software. If this kind of development continues, and the number of hacks in the implementation increases, the code base becomes fairly hard to navigate.

As early stage startups usually do not have the resources needed to produce complete hand-written software documentation, they rely on automatic documentation generators (such as Doxygen) to at least document the APIs. However, multitudes of hacks greatly increase the complexity of the code base, such that the quality of documentation produced by the tools suffers. When a startup hires a new developer, it will take a lot of time for him or her to comprehend the code base.

fb
Facebook also suffered from code quality problems: it was written in PHP.
They are still dealing with that.

Similar problems affect the testing side of startup software. Few test cases are written, so a lot of bugs are missed during development, and only encountered by the customer once the software is shipped. If such a bug is deemed critical (e.g. it may cause data loss to the customer), the developer has to rush to fix it, leading to even more hacks in the software code.

To some extent this can be avoided by having a dedicated tester on the startup team. Testers are usually cheaper to hire than developers, and can find some of the more critical bugs before customers encounter them, hopefully preventing dangerous code from being shipped. However, due to lack of funding, hiring even one tester may prove difficult for an early stage startup.

Implementing a Development Methodology at a Startup

Now that we have established how the lack of a methodology leads to various problems during software development, we can look at what methodology suits a startup best.

Although still common in large corporations, the Waterfall methodology is too rigid to work in a startup environment. In lean startups, product specifications have to be revised constantly to adapt to changes in how customers perceive the product. If the viability of the product becomes questionable, the whole concept of the product may have to be changed. This is incompatible with Waterfall, as it is a sequential software development process.

The alternative to sequential development processes is agile methods. One such method is Extreme Programming (XP). Obviously, due to low head count in early stage startups, not all elements of XP can be implemented completely.

Planning in an agile way can usually be carried out without too many problems, as to some extent startups are aware of their potential customers, and hence, are able to produce the needed user stories. As one of the founders usually takes on more of a business-side role, he or she can be considered the “on-site customer”, and able to consult the developers about software requirements.

Short release cycles can also be implemented. Startups often develop web apps, which can usually be shipped instantly. When it comes to mobile, Android market does not have a pre-approval process. Even for iOS, App Store approval time is about 6 days, which also allows for a relatively quick release cycle. This limits the scope of newly implemented features, encourages keeping the code base clean, and reduces the number of hacks in the code.

Regarding testing, software quality can be much improved by implementing test-driven development (i.e. writing test cases before implementing the needed functionality to make them pass). Designing a few manual test cases to be executed before major changes are shipped to ensure no major bugs are introduced, is also a good idea, as long as the test cases are kept simple enough so that developers (or testers) do not lose a lot of time while executing them.

Since early stage startup teams are so small, implementing Pair Programming and the 40 hour work week can be impractical. Human resources are already too scarce to limit the focus on code and restrict working hours. These agile methods can be implemented later, once the startup team becomes larger.

Conclusion

Obviously, following agile practices during startup development slows down short-term progress due to some overhead. However, considering the number of problems which are eventually avoided, this is more than worth the effort. Unless you are building a startup you know is doomed to fail, adopting an agile development methodology early on will eventually save both time and money.