Knowing When to Let Go

Joel Spolsky once referred to the act of rewriting code from scratch as “the single worst strategic mistake that any software company can make”.

In this article I aim to investigate whether this judgement applies as a sweeping statement. Are there situations where the correct choice is to start over? Is there a replacement strategy that should be adopted instead?

Motivation for a rewrite

The downward slope of code quality

Code is rarely seen as flawed at conception. Otherwise, the sensible choice would be to immediately fix any design problems — nipping any issues in the bud.
Furthermore, code does not physically degrade. If the requirements for a project stay fixed, a perfectly constructed solution would remain so throughout its lifetime.

Instead, there are many factors related to shifting scope that may cause even the greatest designs to wither. In “Programs, Life Cycles, and Laws of Software Evolution”, the author describes three classes of software. Condensed, these are:

  • Programs that are strictly specified, such as calculators for mathematical functions.
  • Programs that solve a real world problem, perhaps a machine competing against chess grandmasters.
  • Programs that automate or ease existing human activities.

The final category comprises the majority of large software projects within the commercial sector. Unfortunately, this category is also the most demanding when taking the cost of keeping a solution relevant into account.

A telling, albeit dated, statistic is presented in the paper:

“Of the total U.S. expenditure for 1977, some 70 percent was spent on program maintenance and only about 30 percent on program development. This ratio is generally accepted by the software community as characteristic of the state of the art.”

The root cause is supposedly intrinsic to the field:

“A program that is used and that is an implementation of its specification […] undergoes continual change or becomes progressively less useful. [This] continues until it is judged more cost effective to replace the system with a recreated version.”

“As an evolving program is continually changed, its complexity, reflecting deteriorating structure, increases unless work is done to maintain or reduce it.”

It is worth noting that the Extreme Programming movement attempts to counteract this final truism with its guidance of “Refactor whenever and wherever possible.”

Recognising rock-bottom

Often, a degraded design will result in application growing-pains.
There are several warning signs that programmers may recognise before considering a codebase rewrite:

Time taken for new developers to get up to speed.

A recently hired developer may be unable to produce useful code even after a period at the company. If this is due to the application’s unwieldy design, it is usually beneficial to devote resources to fixing flaws. Effort expended will be refunded in the future, as developer turnover is a stark reality.

Deployment of the code cannot be automated.

Testing-and-deployment automation can provide enormous productivity gains. Allowing developers to avoid carrying out menial tasks leaves them with more time to add product value, sidestepping unproductive context-switches.

Automation may be impossible due to assumptions made earlier on in the system’s design. Alternatively, it may become difficult due to fragmentation of system components over time.

Test suites take too long.

Alternatively, writing good tests is hard because of code interdependencies.

This flag is self descriptive and common, resulting in many developers searching the internet for quick fixes.

Arguments against a rewrite

Joel initially argues that programmers may be over-eager to begin rewriting a codebase. In his own words:

“The reason that they think the old code is a mess is because of a cardinal, fundamental law of programming:
It’s harder to read code than to write it.”

People spend the majority of their reading time absorbing prose, able to visualise concepts at the rate that eyes scan a page.
Code is often much more concept-dense. It is easy to assume that any lack of understanding after study is due to an overcomplicated implementation.

Additionally, he believes that developers are more keen to produce new code instead of fixing existing code. Creation is often seen as more appealing:

“Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand. We’re not excited by incremental renovation: tinkering, improving, planting flower beds.”

Problems caused by rewriting

Overzealous rewriting can lead to a multitude of issues, even when assuming that no new bugs are introduced.

Firstly rewriting, like most software development, will likely take longer than initially expected. This becomes more important when combined with the second issue: Rewriting adds no value to a product.

Ideally, customers should not notice that a rewrite has occurred. Clients may become disenfranchised with the lack of product feature progression. In the worst case, by the time that the rewrite is complete, many customers may have switched to a competitor’s system.

Thirdly, a subtle disadvantage to the rewrite process is that it invalidates any bug-tracking progress. There is no guarantee that any bug reports or failing test-cases produced will be useful to the maintainers of the replacement system. In an open source project, where these contributions are often made by the system’s users, the community may not be particularly pleased if hard work is casually thrown away. This sentiment is reflected by the infamous “Cascade of Attention-Deficit Teenagers” rant.

Finally, unless the system is entirely documented, it is possible that functionality may be lost. This may range from missing undocumented features, to removing hidden side-effects that customers discovered.

However, you cannot make the assumption that no new bugs will be introduced.

Joel states:

“The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they’ve been fixed.”

“It’s important to remember that when you start from scratch there is absolutely no reason to believe that you are going to do a better job than you did the first time”

Any churn within development teams will result in new programmers — that are not necessarily more experienced than the previous authors — being tasked with a rewrite. This could lead to an underwhelming rewrite of code, suffering many problems that were present before.

 Arguments for a rewrite

If the article concluded here, you would be forgiven for assuming that rewriting code is a foolish idea, spawned by a limited attention span. On the contrary, there are several benefits to starting from scratch.

Discarding troublesome code

Many companies, especially startups, have code laying forgotten, created during project’s bootstrapping phase.

This code is often written by less technically adept team-members, such as the CEO, when the growing company did not require their main vocation. In this example, the original maintainer now owns a different sector of the business. As a result, this orphaned contribution can cause political issues for the development team.

With no ownership, maintenance of the module is likely to fall behind acceptable levels. In addition, junior developers may feel intimidated when discovering the initial author via VCS.

In this case it is often a good idea to rewrite the troublesome code. The replacement can follow present-day best-practices. Responsibility for maintenance returns to the full-time development team. Politically, “We needed a new module to handle 2014 business logic.” sounds much better in a developer’s mind than telling a CEO that they should stick to their day job.

Avoiding bugs introduced during significant modification

Empirical studies have investigated the factors shaping code evolution. “An Analysis of Errors in a Reuse-Oriented Development Environment” provides reasoning supporting potential redesign.

It studies evolution through re-use of components, three types of reuse are defined:

  • Verbatim reuse: Parameters to a function differ but the original component is not modified.
  • Reuse with some modification: Altering less than 25% of the original component to provide new functionality.
  • Reuse with extensive modification: Increasing functionality, but altering more than 25% of the original component.

The paper has several interesting findings. Unsurprisingly, parametric reuse of modules is often the most successful:

“There is a clear benefit from reuse in terms of reduced error density when the reuse is verbatim or via slight modification. However, reuse through slight modification only shows about a 59% reduction in total error density, while verbatim reuse results in more than a 90% reduction compared to newly developed code.”’

Unfortunately, heavily modified components lose significant reliability:

“Reuse via extensive modification appears to yield no advantage over new code development.”

It would appear that modules requiring significant modification during maintenance or integration into new components are no more reliable than when starting from scratch. This is at odds with Joel’s praise of “battle-hardened” legacy code.

Furthermore, bugs introduced when modifying extensively are often harder to fix than those caused by rewriting:

“[Extensive modification] results in errors that typically were more difficult to isolate and correct than the errors in newly developed code. In terms of the rework due to the errors in these components, it appears that this mode of development is more costly than new development.”

When coupled with the effects of developer familiarity on bug-isolation, differences in sign-off time between rewrite at reshape can be even more significant. Starting from scratch may result in deploying well-tested code, far before the cause for any modification-induced regressions would have been discovered.

Conclusion

The examined articles suggest that rewriting code is often detrimental. Some also suggest that extensively modifying to avoid a rewrite is detrimental.

If this is the case, are we doomed to fail? Is there a better way?

I think that it is important to note the distinction between throwing away significant working code and rewriting. An incremental rewrite can improve a codebase without causing long periods of stagnation.

Developers should fix, or in worst-case: introduce, a strict interface within the software component. Tests must be utilised to verify existing behaviour and prevent regressions.
Functions within the system undergoing incremental-rewrite should be replaced gradually. Ideally, set to invoke methods of new modular implementation(s).

This process will eventually result with the troublesome component’s implementation being rewritten. Crucially, the catastrophic consequences of throwing away all of the code at once are avoided.

Once targets for incremental rewrite have been identified, the process is akin to refactoring with a few additional rules:

  • Before replacing components, always be certain that the code is bad — avoid cognitive bias.
  • No new features should be introduced. They serve only as a distraction and increase the chance of error.
  • Study the problem’s domain to avoid mistakes made during the previous implementation. Find the sweet-spot between over-complication (YAGNI) and under-specification.

Rewriting at a smaller scale will help developers avoid biting off more than they can chew.

Importantly, development on the system as a whole can carry on and the product continues to present value to its consumers.

There is no silver hammer for fixing organically-grown software behemoths. However, avoiding and all-or-nothing approach can help maintain development velocity and decrease potential risk. With reduced risk comes higher proportional payoff for exploration.
Hopefully, exploration yields a well-constructed next-generation system, perfectly suited to the task at hand — for now.

Response to “Version Control: Important for Individual Projects?”

This article is a response to “Version Control: Important for Individual Projects?”
The aforementioned post suggests that using VCS is imperative when working in a team but optional, albeit helpful, when working alone.

Upon reading the original article, I found myself agreeing with the majority of points. Yet, I felt that the author did not quite do the positive case justice.

It is for that reason that I have written this response to expand on some overlooked points and offer my own insight.

The summary of the linked article starts by suggesting:

“[Version Control] is used to track and record every aspect of the development of software including who has made changes, why they have done so and to what the changes and additions refer to.”

I feel that this statement is misleading as it seems to conflate the role of raw VCS and providers, such as GitHub, that offer extra services.

Tracking the ‘why’ behind software development is often better suited to a project management tool such as Basecamp [1] or Trello [2]. Git simply stores a series of file updates (nodes) and an ordering (edges) in a graph. The stages of development when producing a conceptual solution do not necessarily map to a linear progression, so VCS alone may not provide an adequate record.
In contrast, GitHub [3] provides excellent Issue-Tracking and Pull-Request features that aid in software development.

The summary continues to offer:

“[a VCS’] working copy is the most up-to date version of the software or the ‘head’ but previous history and iterations are saved and can be viewed as needed.”

This describes only a limited usage of a VCS, like Git,’s full potential. When using an approach such as the Feature Branch Workflow[4], there is rarely a most-up-to-date branch. Instead, many development branches provide a partial ordering of changes. Disjoint features are advanced concurrently. There may be a release branch that is most-up-to-date of all integrations, but this would not include changes present in un-merged feature branches.

Later, in the discussion, the author mentions:

“Merging your edits with someone else’s can be very difficult or they can break the code completely, unfortunately there is no way around this.”

This suggests a scenario where the Feature Branch Workflow may come in handy.

Git’s merge features can usually handle changes within the same file admirably. This is especially true when options such as “patience merging”[5] are enabled.
Any remaining merge conflicts may stem from too many people changing the same function or submodule within a file. This is often a bad sign as it is unlikely that a single component is critical enough to need many programmers constructing it in isolation. It is far more likely that it is a case of “too many cooks” combined with a poor separation of concerns.
Instead, the module that is requiring continual adjustment by many can be split into separate components.

Components can be developed on disjoint Feature Branches and merged into an integration branch when completed. By fixing the interfaces of each component, a single component can interact with one from another branch. This lessens the need for continuous merging of partially-completed implementations.

The original article claims:

“[Using VCS] ensures that everyone is consistently up to date with the latest version of the software. These advantages do not apply for an individual developer.”

I disagree with the suggestion that a sole developer cannot benefit from distributed versioning.
During the course of my Master’s project I developed low-level software concurrently on a laptop and PC. The majority of system code was shared, but features and optimisations were developed independently for a particular device.

Using VCS, shared code was pushed into the release branch and merged into both feature branches. This task would have been much more challenging without the ability to juggle dependencies via branching.

Another quote from the original article that I would like to draw attention to is as follows:

“breaking the code [is] no longer such a hassle since you can always revert back to older versions where all the information is present and works.”

I agree that a major benefit of VCS is the ability to rewind in worst-case scenarios. However, I feel uneasy at the idea of ‘rewinding’ being advertised as a killer feature, as opposed to the safety net it provides.

Prior to development branches being merged in, integration tests should be run to ensure that the existing system will not be broken by code addition.

In my opinion, using the ‘rollback’ feature of version control should be avoided in favour of eradicating sloppy development practices.

The author sensibly states:

“Keeping track of who did what can also be quite important when debugging/trying understand what is going on or when creating documentation”

…but then argues that this is not significant when working alone.

I have anecdotal findings to the contrary. I have found the ability to “review history” incredibly useful, both when working in teams and alone. The ability to view a diff over a custom period and see the previous direction of progression has proven invaluable after taking some time away from development. The information it provides eases the act of “getting back into the swing of things”.

Finally:

“If a third party controls your repository, this can also double up as a back up of your project”

This statement is entirely true. However, having your development repository solely in the command of a centralised third-party is generally a Bad Idea™.

There have been many cases of service provides, such as GitHub, suffering from DDoS attacks and technical failures. [6] When this happens, you could be left unable to commit work and progress can grind to a halt.
Instead, work can be pushed to personal, distributed repositories in addition to a single service provider.

In the article’s conclusion, the author stated:

“Version control is not completely necessary for the individual programmer since a lot of the reasons why this tool is so useful do not apply in this circumstance, and those that do are not essential”

I disagree that individual programmers should go it alone without Git as a sidekick.

Yes, Git is definitely poorly designed for beginner usage. For example, you may wrongly expect “$ git branch” to allow you to change between branches instead of the “checkout” command. This contributes to the unwillingness that many new users have towards learning Git.

Yet, the fact that it is so critical for large teams means that it is far more useful to use it in your own time so that you can become accustomed to it. There are plenty of guides available and even some great in-depth learning resources about how Git works [7].

(PS: I’d really recommend reading that last link if you are interested in painlessly discovering how Git really works)

Using Git for personal files in general allows you to have an audit trail, something that you rarely appreciate fully until you need it.
The author stated that they have not yet been saved by VCS on a personal project. I believe that this is akin to stating “Seat-belts aren’t essential as I have never crashed!”

For example, by putting all of my configuration (dot)files into a Git repository, I can easily check which esoteric hack fixed a particular obscure bug.
In addition, you can leverage the power of automation to pinpoint the exact moment that you introduced a flaw into any system — often saving lots of time. [8]

In conclusion, I believe that Git is a valuable tool that is well worth the time investment to become familiar with. I think that although it is technically non-essential, it is a false-economy to not use Git whenever possible. It takes so little time to set up that the benefits largely outweigh the costs.

 

[1] https://basecamp.com/

[2] https://trello.com/

[3] http://github.com/

[4] https://www.atlassian.com/git/workflows#!workflow-feature-branch

[5] http://bryanpendleton.blogspot.co.uk/2010/05/patience-diff.html

[6] https://github.com/blog/1759-dns-outage-post-mortem

[7] http://think-like-a-git.net/

[8] http://clock.co.uk/tech-blogs/git-bisect-simple-examples-and-automation

How Can We Strive for Simplicity when Everything is Complicated?

It Started Out With a K.I.S.S.

You may be familiar with the phrase: “Keep it simple, stupid.”

Software development is a field centred around deliverables and value.
There are rarely rewards for the most ingenious solution — save perhaps a wholesome, academic pat-on-the-back.

The size of encountered problems and their required solutions is continuing to grow. This is a by-product of the human desire to strive for bigger and better. Often the solution is so vast that a single skilled worker cannot produce it. Such endeavours earn the title of large software projects.

Once the task at hand is too much for a single entity, the project’s design process must span the unavoidable hardware gap present between distinct human beings. Here, the majority of code-spewing aficionados perform much worse than expected. Obviously this issue stems from humanity’s concept transmission protocol being neither lossless nor reliable.

When utilising the increased throughput of a multi-person team, code evaluation becomes subjective. A member’s software module that ticks all the objective boxes: efficiency, functionality, reliability, and using-spaces-not-tabs, may cause bottlenecks within the development pipeline if any of the rest of the group cannot understand completely what the unit does.

It is for this reason that we are often reminded to keep software components simple. Is this as trivial as the description implies? In the rest of this article, I will investigate whether simplicity is a concept that can be agreed upon. Ironically, simplicity appears to be yet another concept sometimes lost-in-translation and misunderstood during software design or evaluation.

Simple: How do?

What does simplicity mean in the context of software development? A common interpretation is to consider the plethora of programming languages available to construct a solution from.

Here [1], is a diagram detailing the perceived complexity, measured as the number of distinct ‘concepts’ present within several language’s design. It suggests that the complexity of C++ is far higher than that of the ‘competing’ languages: Ruby and CoffeeScript — due to the size of its feature-set.

I agree that C++ has a reputation for being complicated in nature, with its standards-commitee forever dreaming up new ways for you to blow your foot off [2]. Yet, I do not agree that this method of judging complexity is reliable.

As a counter-example, consider the feature set of the C programming language.

C has by far the fewest language features of today’s widely-used programming languages, but would anyone go as far as saying that it is a shining beacon of simplicity? Should teams chasing concept-clarity write upcoming features solely in a systems language from 1969?

I am arguing neither for or against the C programming language, as I have experienced it as both “simplistically elegant” and “overly cumbersome” across past and current projects.

Instead, I argue that the complexity introduced to any implementations of a given feature stems from the amount of book-keeping that anyone understanding a single module must do: The quantity of interleaving components that must be understood and remembered before contributors can describe the behaviour of the system as a whole.

[1] http://cpprocks.com/cpp-ruby-coffeescript-language-complexity/#!prettyPhoto

[2] http://meetingcpp.com/index.php/br/items/a-look-at-cpp14-papers-part-1.html

Enough Space to Swing a Function Call.

Paul Graham presented the idea of using the count of distinct runtime components requiring consideration, as a metric for code size, in his essay: “Succinctness is Power” [3]. He argues, championing his beloved LISP over Python, that the latter’s quest for explicitly readability does not necessarily produce a software component that is easier to understand.

“What readability-per-line does mean, to the user encountering the language for the first time, is that source code will look unthreatening. […] It’s isomorphic to the very successful technique of letting people pay in instalments. Instead of frightening them with a high upfront price, you tell them the low monthly payment.”

This quote is a slight sidetrack from the point. It is being used for the purpose of claiming that one language is superior to another, something this article does not attempt. Yet, Graham continues and suggests:

“Instalment plans are a net lose for the buyer […] mere readability-per-line probably is for the programmer. The buyer is going to make a lot of those low, low payments; and the programmer is going to read a lot of those individually readable lines.”

Here, we can draw the conclusion that, assuming we need both reading comprehension and memory capacity for processing a line of source-code, language-concept scarcity and single-line simplicity is not necessarily proportional to the mental taxation involved when understanding a software module.

It is documented [4], that an average human has difficulty storing greater than “seven plus or minus two” differing items in short-term memory. With this limit in place, any block of code with a higher number of interacting sub-systems may prove difficult to understand cohesively.

If this is the case, how can modules be structured to aid comprehensibility and interpretation?

[3] http://www.paulgraham.com/power.html

[4] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2657600

Curiosity Confused the Cat.

There is a solution to information-overload regarding the workings of a unit. Abstract away the implementation of modules via the usage of clear interfaces.

An interface is a strict set of entry points to the functionality of a module. An ideal understandable interface is one that is well-documented and a provides a suitable modelling abstraction. Given this, a programmer does not need to worry about the subcomponents that are performing any requested tasks. Instead, they must only maintain the code required to process communication to and from the module.

Blocks of code that perform a common set of tasks should be encapsulated before approaching critical levels of memory-complexity. By presenting only required interfaces to other disjoint blocks, the concept-count at any point can be contained below harmful levels.

A well documented API is not significantly harder to understand than a basic operator, as long as it known what should go in and how this maps to what comes out. Building from interfaces yields greater potential computational outcomes given a tight concept density budget.

Simplicity Does Not Mean Sacrificing Power.

The most important feature of a good interface is well-defined boundaries, its inputs and outputs. This does not mean that the work undertaken by the interface must be trivial. It is often better to have an interface performing an action that is complicated enough to require time set aside to understand, than have many interfaces perform separate parts of a larger transformation. Understanding the applications of a powerful abstraction can make a module’s intention far clearer.

A testament to this is the Haskell language’s ability to reduce many complicated computations into a simple mapping between abstract data types. Once the domain of the problem is understood, it becomes clear which functional primitives will provide a intuitive solution. Often, newcomers to the language complain that the language is difficult to comprehend. I feel that this is often due to a wish for ‘quick fix’. Many are reluctant to invest the upfront cost of studying the task’s domain — despite the fact that would provide a net-reduction to total effort.

I believe the ease of understanding the symbolic representation of a solution an unreliable indicator of true complexity.
This viewpoint is presented, relating to many programmer’s false studying economies, again in Graham’s “Succinctness is Power”[3]:

“If you’re used to reading novels and newspaper articles, your first experience of reading a math paper can be dismaying. It could take half an hour to read a single page. And yet, I am pretty sure that the notation is not the problem, even though it may feel like it is. The math paper is hard to read because the ideas are hard. If you expressed the same ideas in prose (as mathematicians had to do before they evolved succinct notations), they wouldn’t be any easier to read, because the paper would grow to the size of a book.”

By producing interacting software components that map to the problem domain as if they were operators, a team — or even a solo programmer — can utilise a domain-specific set of functions or services tailored to their needs.

Conclusion

In this article I have drawn attention to the most significant factor limiting the scalability of software teams, communication of intent and interconnection of distinct contributions.

The points raised do not attempt to prescribe a catch-all solution to the problem. Instead, they attempt to highlight a common misconception when judging module complexity during review.
I have provided an example of how teams can present subcomponents to avoid code bloat for the sake of ‘readability’. Different teams will find some techniques work better than others when partitioning work between members.

Embrace the abstractions of others. By investing the time to understand their contributions, teams can efficiently divide large problems at the conceptual level.
As a by-product, less work must be undertaken in restructuring the problem and the intent of the program is more clearly presented.

We are already standing on the shoulders of interfaces and abstractions, what harm can a few more do?