As a result of software development being a fairly new undertaking, no one methodology has yet been able to claim the throne and be the one methodology to rule them all. In fact, there are many methodologies in use and new ones appearing frequently enough to cause shifts in the ways we develop software. One can at least discern ’categories’ in which these methodologies belong, agile vs. heavyweight [1], each of which has their distinctive characteristics.
Before presenting an overview of these characteristics, it needs to be stated that this post assumes that a methodology is needed for any kind of large-scale software project. That being said, it seems as if choosing a methodology is as much a stage in the development process as the development of the software itself, but more on that in a while.
The key characteristics
There are many agile methods to choose from, such as scrum [2] and extreme programming [3]. Boehm [4] points out some common characteristics among them all. He notes the idea that the lack of documentation is made up for by implicit developer knowledge and that, therefore, agile methods may require a group of more experienced developers. Instead of adhering to a set of processes and tools, agile methods emphasize the individuals working in a project, as well the project itself being amenable to changes during its duration. The last characteristic explains the name, as there is inherent agility in agile methods.
Heavyweight methodologies on the other hand, places emphasis on documentation, standardized processes and capturing the requirements correctly from the get go. This is often done through the use of modeling languages such as UML. Boehm highlights the fact that such projects emphasize efficiency, predictability and a process with a clear goal, in which one matures a software product for its release [4]. Instead of agility, developers get the discipline inherent in such standardizations.
So, which methodology should we opt for?
Supporters at both ends of the spectrum like to point out the flaws of the other methodology. However, Boehm suggests that one should try to balance agility with discipline, especially in sectors with companies needing both ”rapid value” and ”high assurance” [4]. This statement highlights what should be a top priority for developers, namely, what does the customer need?
Heavyweight methodologies are suitable for when companies desire a low risk and when rapid development is not the first priority. This could for instance be applicable to ATM’s, power plant controllers, ERP systems or other, similar, large-scale enterprise software. The constraints are, in most cases, clearer, leading to less ambiguity in the requirements. In the first two examples, the constraints can even outline some of the requirements.
Another factor can be the nature of the development within a certain sector, where a specific system can function for many years and therefore might not need rapid and continuous development, as opposed to say, web development. In web development, new technologies and development trends makes rapid prototyping and development necessary, since the changes are frequent. The factors, both internal and external, in the large scale examples mentioned affecting a given project are conceived to be rarer than in settings that are “plagued” by change.
Agile methods on the other hand can work in those change plagued environments, where high risks are accepted, and where developers can work with cutting-edge technologies for rapid prototyping, such as rails, node.js etc. for the development of commercial apps. Therefore, it might be more suitable for ”startups” and small companies with fast development cycles. Requirements can be vague or even unknown to a lesser extent and then discovered during development. In such environments small teams can be more productive not dealing with bureaucracy and adhering to standardized processes.
An example could be a new technology allowing for a new type of product. At this stage, or in fact at any stage, the priority is not to perfect a product, but rather to rapidly deploy a functioning product and then re-iterate within a continuous development and deployment cycle, enabling a rapid growth of the company in question [5].
Concluding remarks
In essence, I have argued that different methodologies have their time and place and that one should select the right tools for the job, the right methodology for the setting. Boehm may well be right in saying that one can even combine the two approaches in some settings. In any case, one needs to analyze the context; what is being developed and for whom? The choices developers make are not made in a vacuum and thus have to be made with the context taken into consideration.
Developers should be comfortable using different approaches to different projects, and this is true not only for methodologies, but also for programming languages and API’s. In that sense, with regard to the comment made in the beginning of this post, the choice of methods and tools for a projects is as much a part of the project as the actual development.
There is a need for pragmatism and objectivity, and it is therefore easier said than done, since many will presumably fall back to what they are comfortable with, even if another method is better suited for the task. Therein lies the danger, in the zealotry of developers, regardless of which methodology one happens to favor.
So how does one know which methods and tools are the right ones? Well, as in any project, some estimations are necessary. If one is to trust thoughts presented here on the matter, one could make a choice based on an analysis of the context of the project, e.g. constraints, requirements, risk of potential changes during the course of the project and so on. This of course introduces further problems of the difficulties of estimating risk for example, however, thoughts on that is best left to a post of its own.
References
[1] A. Clark. ”Software Development Methodologies”. Lecture slides, University of Edinburgh. Feb 2014 [Online]. Available: http://www.inf.ed.ac.uk/teaching/courses/sapm/2013-2014/sapm-all.html#/Methodologies_Lecture_Start
[2] K. Schwaber, J.Sutherland. ”The Scrum Guide”. Scrum.org. Jul 2013 [Online]. Available: https://www.scrum.org/Portals/0/Documents/Scrum%20Guides/2013/Scrum-Guide.pdf
[3] T. Parr. ”Object-Oriented Software Development”. Lecture notes, University of San Francisco. Jan 2009 [Online]. Available: http://www.cs.usfca.edu/~parrt/course/601/lectures/xp.html
[4] B. Boehm, ”Get Ready for Agile Methods, with Care”, IEEE Computer, vol. 35, no. 1, pp. 64-69, Jan 2002
[5] P. Graham. “Startup = Growth”. paulgraham.com. Sep 2012 [Online]. Available: http://paulgraham.com/growth.html
Hey Simon,
Thanks for the interesting post!
Reading through your essay, I get the feeling you are saying agile programming is connected with a lack of discipline. Did I get that right?
Cheers
Aimo
Hi Mo,
Thanks for the comment.
What I’m trying to say is that, in comparison to heavyweight methodologies, agile methods aren’t as strict in enforcing standardized processes, such as modeling. I think it’s for the better in certain circumstances, such as in rapid development cycles, as it allows a project to better adapt to unforeseen events.
It’s not a lack of discipline per se, but rather a “we can adapt and change requirements as we go”-sort of mindset, making it less disciplined than a highly standardized heavyweight methodology. Again, I’m trying to argue that both of them have their use cases (which also serves as a limitation by the way).
/Simon
Hi Simon,
I’m interested to see if you agree with the position of Eric Ries in this Lean Startup presentation (http://www.slideshare.net/500startups/eric-ries); he claims that the decision to select either “traditional” methods (which you refer to here as heavyweight processes) or Agile processes depend on the nature of problem you are tackling.
Ries argues that a project must define its unit of progress; for projects which have “known problems” and “known solutions”, Waterfall (for example) can work well, while for projects with “known problems” and “unknown solutions”, Agile solutions work well.
The argument for Waterfall is that if you know exactly what your problem is and you know exactly how to solve it, you should just plan the stages required to deliver the solution and then execute – the unit of progress is the completion of a stage. For Agile, the argument is that if you know what the problem is, but do not know exactly what the solution is then Agile methods fit well, as it emphasises an iterative and incremental approach to converge on a solution that works. The unit of progress for agile is the delivery of working, tested code.
Ries also argues that for “unknown problems” with “unknown solutions”, the Lean Startup method is appropriate. In this method, the solutions (implemented with Agile methodologies) feed back into the refining the definition of the problem (such as by the gathering of metrics from a released product), for which the unit of progress is validated learning; the intention is to minimise the amount of time between cycles of learning what to do, building a solution and measuring the results.
Would you agree that the decision for whether to use heavyweight or Agile methods is informed by how well the team understands the problem and the solution?
-Scott
Hi Scott,
First of all, thanks for the comment.
Looking at the slides in the link, it seems as if Ries manages to pinpoint and put words to my thoughts in an uncanny way.
The key thing in his presentation is, I believe, that he manages to accurately exemplify the dynamics of unknown solutions, and even unknown problems in some cases. Startups that begin working on one thing, often evolve into something completely different as a result of one or multiple development cycles. I tried to shed light on that fact when I referred to agile methods as enablers of rapid growth of companies, rather than particular products, which seems to be a common trait inherent in startups.
As for the heavyweight methods, e.g. a waterfall method, I also tried arguing for the feasibility of such methods when developers have clear constraints and, in some sense clear requirements. One can maybe also see the relation between these “knowns” and the lesser risk of these projects as explained in the post.
Another interesting aspect is the second myth of lean startups. I used web development as an example in my post and I think that the notion of uncertainty is a better way of understanding it in more general terms.
Definitely a very insightful presentation. So to answer your question; yes, I do agree with that, although I didn’t manage to express it in such an eloquent way. May I ask what your own position on the matter is?
/Simon
Hi Simon,
Thanks for your clarification. I’m glad you appreciated the Eric Ries slides, his Lean Startup book is well worth a read.
My position is that when reading your post I felt that you were in broad agreement with Ries (which I also agree with). I wanted to clarify your point that “heavyweight methodologies are suitable for when companies desire a low risk and when rapid development is not the first priority”; I would argue that all companies would, all other things being equal, prefer low risk and rapid development, but that heavyweight methods may actually offer the most straightforward (and, rapid) route to the solution, so long as the problem and solution are very well understood.
My other point to clarify is that I believe that Agile processes (and “lean” style processes with continuous feedback and learning from the iteration cycle) require a lot of discipline, it’s just another form of discipline. Arguably a successful Agile team must be highly disciplined, to prevent losing track of the current target. I do agree with you that Agile teams have less requirement for the types of bureaucratic artefacts produced in more heavyweight methods.
-Scott
Hey Scott,
From my point of view, agile development requires more discipline than in a more heavyweight approach. In agile methodology, small intermediate goals tend to be less clear. So the developer is more responsible for his own intermediate results.
Due to those responsibility shifts, more discipline is necessary to be successful in agile development.
What do you think?
Aimo
Hi Scott and Aimo,
I think that you’re on to something saying that it’s “another form of discipline”. The discipline is internal to the developers, but I wouldn’t argue that it requires more discipline, but rather maybe more communication and coordination within the team.
Discipline: “the practice of training people to obey rules or a code of behaviour, using punishment to correct disobedience.” – Just to avoid having to argue semantics.
Thinking about it, heavyweight methods require discipline in creating the models, as well as in adhering to them. That all these are externally represented by some sort of artefacts is, in my mind, not equivalent to it being less disciplined. On the contrary, the punishment for not adhering to principles is often harsher in that everything might have to be redone.
As Reis so neatly highlights: maybe it does not matter as much when agile development teams lose track, seeing as they can often morph their initial idea into something previously not thought of?
I agree with you on companies preferring both low risk and rapid development, but in most realistic cases there seems to be a tradeoff. Maybe rapid development in itself induces higher risk?