Response to “Agile Methodologies in Large-Scale Projects: A Recipe for Disaster”

This is a response article to an  an article Agile Methodologies in Large-Scale Projects: A Recipe for Disaster.

Introduction

People tend to think about Agile as a set of rules which following will lead your team to achieving an easy success. Unfortunately this is not the case as none available software development methodology guarantees success.

The main difference between Agile and other methodologies is focus on its core values described in agile manifesto [4]. Agile creators developed it having in mind some of the important issues reported during the usage of standard development practices. They decided that during the process of software development we should care about three major factors:

  • Creativity.
  • People.
  • Working Software.

First of all, programming is a task much more similar to art rather than any other type of engineering. Code can be written in infinite numbers of ways, so it demands constant flow of creativity from the programmer.

Secondly, we must care about people during software development – Agile is designed to enhance people productivity and maximize teamwork effort. It serves an idea of proper knowledge distribution and setting bonds between team members.

Lastly, Agile designers realised that producing software and writing new code is much easier (and far more enjoyable) than fixing bugs, therefore testing process beforehand is crucial task during software design process.

It is easy to see that those principles can be used both in small projects (where Agile is believed to be designed for) as well as big projects. After all in both types of projects, there is a team of people that want to produce innovative and working software.

 Working Software over Comprehensive Documentation

Author of [1] states that it is far more desirable to work on comprehensive documentation and start with complex planning in order increase chances of delivering a working solution.

First and foremost, big projects tend to fail too often regardless of documentation and planning involved [8]. You can find a lot of examples in the web [7]:

In a landmark 1995 study, the Standish Group established that only about 17% of IT projects could be considered “fully successful,” another 52% were “challenged” (they didn’t meet budget, quality or time goals) and 30% were “impaired or failed.” In a recent update of that study conducted for ComputerWorld, Standish examined 3,555 IT projects between 2003 and 2012 that had labour costs of at least $10 million and found that only 6.4% of them were successful.

This citation can be used as naïve explanation that standard development methodologies do not focus on producing working software, therefore we should use Agile. Unfortunately the answer is much more difficult to form properly.

Looking at this from different angle – Agile does produce Comprehensive Documentation. The documentation itself will be done by properly designed unit tests and by following coding standards (commenting your code extensively). Those things can be then used to automatically generate  proper documentation with minimal effort (eg. Doxygen).

The idea behind “Working Software over Comprehensive Documentation” is to start working on a software development as early as possible, rather than writing everything in specs before the project stars.

Lastly, author of [1] states that when failure occurs, shareholders will try to find explanation. It is implied that without proper documentation there won’t be a proof of what went wrong.

This means that “Working Software over Comprehensive Documentation” point was misunderstood by an author of [1]. There is a program documentation and process documentation. These two are completely different and serve for different purposes – one is used to produce software another one is to avoid structural issues. The second one will be used to document the whole process by project manager and it will be a first thing to look for errors.

Individuals and Interactions over Processes and Tools

Agile care about people, because after all it is a Team of people which will produce a software. I would say that proper communication is the most important part of software development in a team, especially important when talking about bigger projects. From [6] we can derive:

Studies of “communication saturation” during one project showed that, when no communication problems exist, teams can perform 50 times better than the industry average

Agile principles were introduced to enhance human interaction and build team spirit. This mean that stand-up meetings, pairing, energized workspace and code sharing were designed to boost communication inside a team.

Moreover, author of [1] says that we should always reuse tools and libraries instead of prototyping them. This unfortunately is also misunderstood – good communication between programmers may boost code reuse, because someone may already know the answer to a problem you are working on. Agile was never about not using existing tools and building them on your own, but about relying on human interaction during data gathering process rather than using sophisticated tools and processes.

I can easily think about an example of relying on processes rather than talking with other team mates. During my summer internship I was a witness of an employee filling bug report on piece of code which was developed by a team mate sitting next to him. This obviously is a recipe for disaster.

In addition, author claims that in larger projects it is wrong to go from standards and use innovative tools. This seems like a valid point in general, but there are moments when prototyping may be useful. Let’s think that there are two possible solution to an existing problem – one will be standard another new and innovative. It may be a case that quickly developing standard solution and testing will enlighten some important faults. We know that this is always difficult to estimate everything beforehand even in standardised big projects [3].

Customer Collaboration over Contract Negotiation

Let’s think about using on-site customer for a minute. Author of [1] explains importance of iron-clad contract in big software projects, rather than using customers to deliver what they really need.

We should stop for a second… We need to deliver a software which is doing what was intended to do. This is stated on a contract…

The idea behind Agile Development is to use constant feedback from customers (or shareholders) and make sure that our software does what it is intended to do (WHAT CUSTOMER REALLY NEED).

Authors of both [2][3] explains a solution during contract negotiation. During planning game project manager (the most important person in the group) will discuss cost for delivering individual features.

In the end – what is Customer Collaboration after all? Let’s say that the goal is to deliver enormous web system – what stops us than to test first features as soon as possible and gather a data from users?

Responding to Change over Following a Plan

Author of [1] claims that following a plan in big projects is a good practice, because everything can be crossed from a list. In addition according to the author in large projects it is much more difficult to set a proper ways of communication.

I would say that it is everyone wish to code everything according to a list. It would make our work much easier (and probably far more boring). Unfortunately even in big projects requirements are vague and not specified very well. By following agile practices we can iteratively prepare demos which may be presented to a customer (or shareholder) for verification.

Communication in huge projects is a significant and main problem for Agile being difficult to implement. Fortunately there are several solutions proposed from Agile experts to deal with this problem. Those solutions introduces a bigger overhead when it comes to software development, but are designed to distribute knowledge among teams and (what is probably much more important) to create a feeling of ownership of the project. One of those ideas (which easily may serve as a topic for a new blog article) is Scrum of all Scrums presented in [9]. The idea can be explained by the usage of Scrum principles for teams in addition to Scrum for individuals.

Summary

It is obvious that Agile is much more difficult to implement for larger project. On the other hand I believe that problems discussed by an author of [1] can be easily solve even in big, government projects.

Agile is not a best solution for everything nor a set of magical rules which grants success to every project. Values written down in Agile Manifesto serve a purpose to focus on extremely important aspects which tend to be overlooked in standard ways of software development methods. After all rules and resources in software engineering are completely different than in electric or civil engineering.

Secondly, big projects tend to suffer from the same kind of problems that the smaller ones. Agile Manifesto described set of rules which will boost productivity of a team, so there is no reason to apply them for everyone. In addition there are a lot of discussion how to use Agile successfully for bigger projects [10][11].

Last, but not least is an idea of applying some agile principles even in standard waterfall process can boost productivity, because we can provide constant knowledge sharing, satisfaction of individuals and much greater feeling of project ownership.

Note

The author of [1] chose very broad subject for his article. It was difficult to answer and discuss every part, so I decided to comment some problems and misunderstandings which may be found in the author’s article . There is much more to be said in this subject, so I am open to discuss points which were only mentioned in this article in the comments section below.

Reference

[1] Agile Methodologies in Large-Scale Projects: A Recipe for Disaster. SAPM Blog, 2014

[2] Shore, James. The art of agile development. ” O’Reilly Media, Inc.”, 2007.

[3] Martin, Robert Cecil. Agile software development: principles, patterns, and practices. Prentice Hall PTR, 2003.

[4] Manifesto for Agile Software Development http://agilemanifesto.org/

[5] What Does the Agile Manifesto Mean?  http://www.scrumalliance.org/community/articles/2013/2013-april/what-does-the-agile-manifesto-mean

[6] Agile Principles and Values, by Jeff Sutherland http://msdn.microsoft.com/en-us/library/dd997578.aspx

[7] Why Do Big IT Projects Fail So Often? http://www.informationweek.com/strategic-cio/executive-insights-and-innovation/why-do-big-it-projects-fail-so-often/d/d-id/1112087?

[8] Why Big Software Projects Fail: The 12 Key Questions  http://www.cs.ucc.ie/~herbert/CS4504/Xtra%20Software%20Risks/laacm0512-Why-Big-SW-Projects-Fail0503Humphrey.pdf

[10] Scrum of Scrums: Running Agile on Large Projects  http://www.scrumalliance.org/community/articles/2013/june/scrum-of-scrums-running-agile-on-large-projects

[11] Yes, Agile works in larger enterprise projects too. http://www.zdnet.com/yes-agile-works-in-larger-enterprise-projects-too-7000020875/

[10] Agility@Scale: Strategies for Scaling Agile Software Development  https://www.ibm.com/developerworks/community/blogs/ambler/entry/large_agile_teams?lang=en