Developers are just as Responsible for Feature Creep as Clients

“Feature Creep” is the tendency for extra, unnecessary features to  gradually “creep in” to the basic requirements of a product. It is frequently cited as one of the top reasons why software projects become bloated, run late, and fail.featureCreep

Who is to blame for feature creep? As developers, it is tempting to blame indecisive clients, or project managers eager to please their bosses by promising new functionality. I would argue however that many common behaviours of developers are equally to blame, and that it is our responsibility to alter our bad habits with either self discipline or by advocating for systematic changes in the workplace.

The three key patterns in developer behaviour which I believe contribute to feature creep are:

  1. Boredom, or the tendency to want to work on interesting problems rather than necessary work
  2. Perfectionism and needless attention to detail
  3. Fear of asserting their own knowledge and expertise

Developer Boredom

Most developers I know, including myself, would rather work on interesting technical challenges than mundane tasks. A passion for the difficult can sometimes be a benefit, but often this desire to work on complex tasks or new technologies can bias a developer’s judgement. Sometimes the tasks which are most important to the success of a project are the boring ones such as implementing business logic or adding widgets to the GUI.

boredomThe obvious way to combat this behaviour is to practice disciplined self-reflection. When considering working on some complex algorithm, ask yourself : “Is this really necessary to the success of the project? Or do I just think this is cool?”

This solution may be unsatisfactory for some people. Perhaps they believe that it is impossible to squash a programmer’s natural intellectual curiosity, or that it would be bad for employee morale to  always prioritize the boring-but-necessary tasks. An alternative solution for these people might be to see if your workplace is open to the idea of compartmentalizing  this curiosity.

Many companies (most notably Google) are known for adopting a practice they call “20% Time“. This is where employees are given a small percentage of their work week to work on whatever tasks related to the company that they feel like. The benefits of this policy are twofold: Firstly, employees are more amenable to doing some of the boring-but-necessary bits of software development, as they now have breathing space to explore more interesting technical challenges in their own time. Secondly, If something an employee is working on in their 20% time does turn out to be valuable, the company can reap the benefits.

Perfectionism and Procrastination

The flip side of my previous point is the tendency of developers to get bogged down implementing frivolous or needless features. The most notorious example of this is a developer’s tendency to add “Performance” as a requirement in projects which have little to no performance constraints.

Steve McConnell tells an entertaining anecdote regarding this behaviour in his book Code Complete. He recalls a time when he was sent in to fix a software project which was long over time and budget, but still not even providing basic functionality. Upon demonstrating his completed version of the project to the old engineering team, a senior engineer criticised his implementation for taking over twice as long to run as their original implementation.

At this point, McConnell quipped that the difference was that his system actually worked. If he was to relax that constraint, he could deliver a product that ran in zero seconds!
The point here is that developers will often subconsciously add their own set of requirements without considering their benefit to the main aims of the project.

A systematic way to combat this behaviour is to introduce a technique from the Scrum methodology known as Sprint Iterations. This is where software iterations are broken down into small – usually weekly – focussed tasks. In contrast to gradually implementing features over a long period of time, this technique makes it more difficult for developers to drift onto unnecessary features, as they have a clear short-term goal to deliver by the end of the sprint.

Having the backbone to say what you know

Even with the most competent, focussed developers in the world, people still often work in an environment with clients who are constantly suggesting “one more feature” and project managers who are eager to please by over-selling what can be done in the time available. Many developers I have spoken to believe these to be “the facts of life” and that there is nothing that can really be done about this. I don’t think this is entirely true.workersUnite

In my time working as an intern at Amazon, I was exposed to a lot of their philosophy. One term which came up time and again was the concept of “Learned Helplessness”. The premise of “Learned Helplessness” is that even though developers are frequently in the strongest position to give advice on how difficult a new feature will be to implement, or how likely it is that the current workload will be finished on time, they will often not speak up because of a few bad experiences dealing with management in the past.

The developers at Amazon were determined to not let this happen to them, and made a point of actively “Pushing back”. “Pushing back” was their term for joining together as a group and actively resisting management requests for additional features when they thought that they were unnecessary.

This was perhaps one of the most successful working environments I have experienced because developers had just as much decision making power as the management. This allowed both groups to feel comfortable sharing their relevant knowledge and expertise.

Developers in all workplaces need to have the courage to speak up when they think that the project scope is running away from them, and shed the notion that “The customer is always right”.

Conclusion

Feature creep is a force that has been responsible for a multitude of project failures, and is often attributed to indecisive clients and failures on the part of management. However, I have hopefully convinced you that we developers often behave in ways that subtly contribute to feature creep. Developers need to be disciplined, take personal responsibility for these behaviours, and campaign to make changes in their workplace which discourage others from contributing to the creep.

“Conservatism” in project management is more valuable than you might think

The following article is a response to the blog post “Conservatism has no place in project management” by s0952140.

recycle-binIn this post, the author laments that project managers frequently ignore the value of upgrading to a new language, framework or tool.  Their central assertion is that it is almost always preferable to throw out old code in favour of a new pattern, framework or language:

“…You have to make a choice: be conservative, keep what you have, or try the new things. At many times in all projects these moments will arise. Should I keep to what I am doing or should I try something new. Always try something new”

Their reasoning stems from the belief that once a project reaches a certain age, it will reach an unwieldy complexity , at which point you have only two options: Work fruitlessly on improving old code or throw it all out and start fresh. The author believes this is an easy decision:

Chuck your old code. If you had wrote some ground breaking algorithms and whatnot, you can still take that and insert in your new project. You lose nothing, other than the cancer.”

Dealing with technical debt is important, but the author puts forward a radically simplified view of the situation in most large scale, long term software projects. I can think of three reasons why “Conservatism” in project management is valuable.

  1. Working with new languages and frameworks comes with considerable risk
  2. The old software actually works and has been strengthened by years of development
  3. Developers are often biased towards using new technologies even when that technology may offer little benefit to the project

Just to clarify, the author’s definition of “Conservatism” is wide ranging and is defined not only as a bias against throwing out old code in favour of using new languages, patterns and frameworks, but also as a bias against using new tools. I actually agree with the author on the point about tools. Setting up a bug tracking tool or version control for a project takes hours at most, and the benefits of doing so are so enormous that any manager arguing against them is a fool.

For the other aspects however, I believe a small amount of “Conservatism” may have its place.

Dangers of working on the bleeding edge

Using a brand new framework or language for your project undoubtedly comes with advantages: They allow you to take advantage of exotic paradigms or useful features not present in old frameworks. However, adopting a new technology also comes with significant risks.

Firstly, new technologies often have less libraries and tools. Here’s an experiment : Go to Google and type “J” followed by any verb. Chances are that one of the top results is a Java library. If you decide to scrap your Java system in order to leverage the power of the Julia language, you may find that the benefits of the language’s power are offset by the lack of libraries and developers.

Secondly, newer technology is more likely to have bugs and be subject to change. Developers often forget that many new technologies they are excited about are still in their experimental stages, and the framework implementation may be rapidly morphing during the life cycle of your main product. If one of those changes introduces a bug, you may find yourself spending more time fixing the framework than working on your own product. Remember, even popular and stable new frameworks like NodeJS are still technically in beta (version 0.10.26) .

Old software *works* and has a wealth of knowledge

Throwing out messy code is not as costless an option as the author suggests. At every point in the lifetime of a product, the developers will have learned more about the best way to implement such a product. Surely, the author reasons, if we were to throw out all the current complex old code and start again, we would build a far cleaner, maintainable, and robust system.startOver

This line of reasoning is flawed because it assumes that the developers can remember all the lessons learned during the entire life of the product. For any sufficiently large system, it is definitely not the case that you can keep all of the implementation details in your head at once. However, one thing that can remember all these lessons is the code base itself. Much of the code that looks like a “mess” in a long lasting piece of software is actually code that acts as a workaround for browser compatibility issues, guards against a particularly subversive bug, or deals with a specific corner case for some users.

Often, the more likely scenario when starting a code base from scratch is that you will waste time making all the same mistakes again. Sometimes throwing out old code is the correct course of action, but the author’s assertion that it is always the correct action is misleading.

Inappropriate lust for new technology

Programmers are often so excited by the prospect of using a new and interesting new technology that they fail to consider whether adopting that new technology would have any tangible benefit to developing their business’ product. This was exemplified for me by a light-hearted conversation I overheard during my time interning at Amazon:

Programmer A: “<Complaining about an old perl web system> Look at this code – Its a complete mess! We should switch to NodeJS. It’s way more maintainable AND it scales better”

Programmer B :”We’ve have been adding features to this code base no problem, and this system has already scaled to millions of users. What are you talking about?”

Programmer A: “Look – I just really want to use Node <laughs>”

The point here is that a programmer’s motivations for adopting a new technology may not line up with the needs of a project.  While increasing programmer morale definitely has some tangible benefits, sometimes having a “Convservative” project manager to consider the business impact of such a decision (e.g Will changing to this new technology save us money? Will the effort to make a change cost take time away from other tasks?), is of value.

Conclusion

Managing technical debt and adopting new tools and technologies undoubtedly has value, and the author of the original article is right to criticise project managers who may undervalue it. However, the first line of their article is telling: They admit that there may be some cases in which “Conservatism” has a place, but only “Very Rarely”.

Hopefully this article has convinced you that there are several reasons to apply a little “Conservatism” in project management, and that these reasons may be more prevalent in long term, large scale software than the author believes.

Incentives Poison Extreme Programming Values

Agile methods such as extreme programming outline a set of principles and practices for developing software projects. They are ideal for small projects with volatile requirements as they favour flexibility and working software over rigorous planning and extensive documentation.

Reading over the list of rules for Extreme Programming, I couldn’t help but agree that many of the rules seemed like brilliant ways to ensure the quality and success of a project:

  • Having developers take collective ownership of the codebase
  • Integrating code with the main project build often
  • Writing unit tests before writing the code
  • Making frequent, small releases
  • Refactoring code whenever possible
  • …and many more.

What surprised me however was that on both the Extreme Programming site and the Agile Manifesto page, little is said about how to effectively get people to follow these principles, or systems which might come in conflict with these principles. It is as if the authors believe that if they just pick the right set of principles, then implementing them in any context will be trivial.donkeycarrotcliff

My argument is that even if you intend to adhere to extreme programming’s principles, incentive structures put in place by the organization in which you work can have a strong impact on whether you follow them or not. By relating my personal experience of such external incentives, I will hopefully convince you of just how powerful an influence they can have, and to be more aware of how incentives may be motivating your behaviour in the future.

As part of the software development course at the University of Edinburgh, I was placed in a team of 12 other students with the task of creating an autonomous, football-playing robot in a single semester. The robot would use images of the pitch received via bluetooth from a web camera in order to make tactical gameplay decisions. This was a project with vague requirements which would require lots of rapid prototyping to be successful, so it seemed like a perfect fit for the extreme programming methodology. We agreed to commit fully – Pair programming, Trello cards for user stories, stand up meetings, merciless re-factoring, shared code ownership. Everything

We soon found however that the assessment criteria would pressure us into abandoning many of these principles.

Incompatible, Mandatory Milestones

To ensure that teams were making tangible progress on their project as the semester progressed, the course organizers set fortnightly milestones on which a percentage of the teams total mark for the course was assessed. This seems like a good incentive to encourage students to work on the project, but two key features of the system led to negative behaviour:

  •  The milestones were set by the course organisers at the beginning of the semester  and didn’t change as the semester progressed.
  • Regardless of other functionality your team’s robot had, if it couldn’t pass the milestone, then the entire team got zero marks.

This created a conflict between following our Extreme Programming principles and achieving the highest course mark:

In principle we would like to write unit tests for all our code, refactor old modules and ensure overall system quality by following a coding standard.

In reality, if we can’t hack together some code to make our robot intercept an oncoming ball by Friday, we all lose 5% of our mark.

When pushed for time, we would always choose to hack together a façade of functionality to pass the current week’s milestone rather than follow solid development principles. We were rewarded for this behaviour with a higher course mark, but the code used for the milestone was often of such poor quality that it was of no use in the main project’s goal of building a quality football playing robot.

The conclusion we reached from this experience was that milestones are only useful if they correlate with the end goal of producing a quality final product. Being forced to meet arbitrary milestones often means developers have to choose between producing principled, quality code and meeting an arbitrary deadline.

Adding a competition element to frequent releases

Rather than fixed milestones, Extreme Programming advocates small, frequent releases which gradually display new functionality. Our robot football course actually had such a system in the form of fortnightly “friendly matches”.

Every two weeks, every team on the course would take part in a friendly tournament where 2 opposing teams would use the current version of their robot to play against each other in a game of one on one football. This ticks a lot of boxes in the extreme programming framework: It allows teams to demonstrate visible progress to their clients (the course markers) and gives teams a chance to get feedback on their systems by exposing its current design flaws.

Again however, an incentive was added which corrupted things a little. Instead of being consequence free matches, the position in which you finished in the previous fortnight’s tournament “seeded” you for the next tournament. In other words, if your robot was in the top 3, you would be allowed to “skip” the first round in the next tournament. Additionally, your robot’s position in the final tournament contributed a significant portion to your course mark, so being able to skip a round in this tournament was a massive reward.

To understand why this incentive is so insidious, it is important to recognise that the value of showcasing frequent iterations under the extreme programming methodology is to expose flaws in your system and get the necessary feedback to improve them. This incentive warps this goal so that the value chiefly lies in beating as many teams as possible. Teams would often accomplish this by explicitly coding gameplay strategies which they knew the other teams couldn’t handle at their current stage of development instead of working on the problems in their final overall system. In this environment, it is optimal to hide your system’s flaws rather than expose them.

Ranking Group Participation

Even without external influence, enforcing extreme programming principles such as pair programming and shared code ownership requires willpower. In a disciplined, trusting environment, it can certainly be done, but in an environment which forces you to consider your participation in the project as a competition against others in your team, it is nearly impossible.

In order to encourage participation from every member of a team , each team was subject to a weekly performance review, where students were given a score to grade their level of contribution and made up a small percentage of their final course mark . A score of 5 meant you had contributed exceptionally, while a 1 meant your contribution was minimal.  The key detail of this incentive system was that not everyone in the team was allowed to receive a high mark . Students were ranked from those who contributed most to those who contributed least and assigned a score respectively.

stackRankingThis was perhaps the most poisonous incentive of all, as it encouraged us to think of our teammates not as collaborators, but as competitors. As a result, it made numerous extreme programming principles difficult to uphold:

  • Collective ownership of code – “Sharing code” now means others may be able to take credit for part of your contribution . The optimal strategy to get a high rank is to take sole ownership of a module in order to say in the meeting that you were “entirely responsible for progress on the planning system this week”.
  • Code the unit tests first / Refactor whenever possible – Again, you are forced to consider what sounds better in a meeting:  “I added functionality so that the robot can now kick while moving” or “I refactored some code which was already working so that it has a clearer structure”.
  • Integrate Often – It now becomes advantageous to sometimes not integrate your work with the current build. This way, you can create the impression that you have made a large amount of progress by explaining all the new code you have just “not yet pushed to the main build”. If you were to continually integrate your work, there would be clear evidence of what you had and had not done.

Communication and trust are vital to upholding many agile practices, so incentives such as this which create an environment of distrust will suffocate such practices.

This system of “Stack Ranking” developers has become notorious in industry because of its use by many high profile companies. Many reports claim that such a system was responsible for the majority of problems at Microsoft over the last decade.

Conclusion

While agreeing to commit to an agile process such as Extreme Programming (or indeed any process) is a positive step towards a successful project,  I have shown through relaying my own experiences that the mere choice of principle is not the only factor at play. Teams must be disciplined in adhering to their chosen principles and diligent in identifying structures which pressure them to do otherwise.

I hope I have emphasised just how powerful and poisonous such influences can be, and have given enough examples to encourage you to look out for similar systems which may influence you in the future.