The Joel Test: “Do you have testers?” – No, and neither should you (probably)

tl;dr

The times they are a-changin’. In the post-Agile world, not having dedicated testers might just be advantageous for you. Depending on the software your team produces, factors such as time-to-market might be more important than a perfect product. “Build the right it” first, “build it right” later.

Setting the scene

In 2000, the prolific software writer Joel Spolsky proposed the “Joel Test”: twelve simple yes-or-no questions to judge the quality of a software development life-cycle. The test invites the inquirer to ask questions such as “Do you use source control?” or “Do programmers have quiet working conditions?”. According to Spolsky, any good software development process will answer “no” to at most two of the test’s questions.

Fourteen years later, the test is still widely recognized as a useful, low-cost tool to evaluate the efficacy of a software development process. But is it really? A lot has happened in the world of software engineering since the test’s inception. When reading Spolsky’s original proposal nowadays, I can’t help but feel as though some of the questions are overly dogmatic or at least a bit dated.

For instance, question 10 of the tests states that

[i]f your team doesn’t have dedicated testers, at least one for every two or three programmers, you are either shipping buggy products, or you’re wasting money by having $100/hour programmers do work that can be done by $30/hour testers. Skimping on testers is such an outrageous false economy that I’m simply blown away that more people don’t recognize it.

Source: “The Joel Test”.

In the remainder of this blog post, I will analyze this assertion and discuss to what extent its premise is still relevant in 2014.

Spolsky’s argument then versus in the trenches now

Let us first have a look at the argument made by Spolsky for why dedicated testers are imperative for a good software development process and then contrast this with my personal experiences in the software world.

Question 10 of the “Joel Test” (quoted above) observes two issues:

• A team without dedicated testers will release a bad product because of lacking quality assurance (QA) from both technical and functional points of view.
• A team without dedicated testers wastes money since testers are cheaper than developers.

In later posts, Spolsky refines his point further:

• Programmers make bad testers – the required skill sets are not related.
• Having testers shortens the time between developers and getting feedback on product quality.
• Testers will catch problems that the original developer didn’t (or wouldn’t).

Sources: “Why Testers?”, “Top Five (Wrong) Reasons You Don’t Have Testers”.

These assertions do not match my personal experience. Time for some anecdata.

Last summer I worked at the Amazon Development Centre Scotland (ADCS). Most people would likely agree that ADCS is a high quality software development environment. As a matter of fact, ADCS would obtain a perfect score on the “Joel Test” if it were not for the lack of dedicated testers on my team… however, this does not mean that ADCS ships bad code, as Spolsky claims must inevitably happen. Far from it. Software quality is simply assured by other (in my opinion superior) means:

• Strict pair programming when code is being touched.
• Unit and integration tests written for any new feature.
• Before a major feature is released, the entire team pitches in and spends half a day to a day focusing on testing the new feature. If any show-stopper bugs are found, the release of the feature is delayed.
• One person on the team is “on-call” every week, handling bug reports as they come in.

Arguably, this Agile-inspired approach has a number of advantages over having dedicated testers:

• Everyone in the team has a rough idea of what is going on. There is less of a chance of knowledge silos developing.
• There is a strong incentive to produce high quality code as your colleagues are going to be directly affected by the mess you cause (as opposed to testers in some other team you might not know).
• More eyes see the code than with Spolsky’s recommended “1 tester per 2 engineers”, increasing the chance of finding bugs or poor user experience.
• Developers will never have to wait for the testing team. Automated tests give instant feedback. More involved evaluation such as user acceptance testing can be prioritized appropriately by the developers in order to integrate it seamlessly with the rest of the development process.

In addition to these benefits, I also found that foregoing dedicated testers means that the development team is more likely to develop a strong sense of ownership of their product. The team owns the software development life cycle end to end and is therefore more likely to “do the right thing and do it right”: develop reliable solutions that do what they are supposed to do. Contrast this with just implementing some specification, passing it on to testers (or later business analysts) and never looking back… Farewell, “code, compile and forget”. You shall not be missed.

A brave new world?

So how do we consolidate this difference between what the “Joel Test” claims should happen and what actually does happen on the grounds, even at a top-tier, quality obsessed company like Amazon?

It is important to note that the “Joel Test” was written in 2000 and very much reflects the spirit of the time in its attitude towards testing. A lot has happened in the world of software since then; most notably, development methodologies have shifted away from the Waterfall-like methods prevalent at the start of the century to a wide-spread adoption of Agile methods.

Software produced in the times of the “Joel Test” would have one, two, maybe a handful of releases a year, indiscriminately distributed via physical media to a company’s entire client base. Nowadays, the near-ubiquitous use of web-based services means that new software versions can often be rolled out instantly or to only a small part of your company’s customers. Being able to perform split tests on your product by releasing new features gradually into the wild means that the impact of bugs or poor design is less severe than with the old model where a company’s reputation could rise or fall on the newest polycarbonate-plastic pressed version of its developers efforts. Thus reduced the importance of the tester.

Furthermore, the modern-day developer has access to a plethora of new tools that reduce the burden of test. While some of these tools might have been available at the time of the “Joel Test”, they would have been nowhere near as ubiquitous as nowadays when FindBugs is in the top ten most downloaded Eclipse plugins and all major Java IDEs come with JUnit installed by default. Static code analysis finds a plethora of bugs even before the developer hits “compile”. Unit and integration tests take snapshots of the functionality of a system and therefore prevents test regressions by default. This eliminates a big chunk of the grunt work that formerly had to be done manually by QA departments. Thus reduced the importance of the tester.

Additionally, the switch to Agile-like methods brought about a mentality change in the software field. Previously, QA teams were often “separate but unequal” parts of the development apparatus. Under Agile and its offspring like Test Driven Development, everyone is a tester: everyone wears all hats. The role of QA can thus grow to encompass more directly technical or business-related aspects such as focusing on user acceptance testing and performance- or security-testing. From a button-mashing unequal to a respected fellow reliability engineer. Thus reduced the importance of the tester.

However, the shift to Agile can’t explain the entire difference between what the “Joel Test” claims and what I observe in the modern software world. Changes in tools and mentality have shifted some of the burden of test from traditional testers to the entire development team… but traditional Agile is still far from the laissez-faire “ship it” attitude that I see pervading the software world. As a matter of fact, Agile evangelist and writer Don Wells claims that

[q]uality assurance (QA) is an essential part of the [Extreme Programming (XP)] process. On some projects QA is done by a separate group, while on others QA will be an integrated into the development team itself. In either case XP requires development to have much closer relationship with QA.

Source: Extreme Programming Rules

I think that this remaining disjoint can be explained by observing that some companies might have moved on from Agile and are adopting elements from even more “lean” philosophies. “Idea bugs” are harder to fix than software bugs. If you build a product that people really want to use, they will be forgiving (prime example: the Twitter service-outage whale). Getting something out of the door and running it by real-world users is thus increasingly becoming the most important aspect of software development: delivering products, not code – focusing on “building the right it” rather than “building it right”. This is something traditional QA can’t really help with. And therewith died testing.

The future of test and concluding remarks

Yet test will surely live on with “I’m not dead yet” tenacity. The software forest is not only comprised of Agile web development trees. Traditional testing is sure to find respite in havens such as AAA game development with its ever-present big releases or security- and safety-critical programming with its zero-tolerance for bugs (e.g. medical or aeronautical software). For the remaining denizens of QA-land, however, I fear that the time has come to leave testing drugery behind and move on to the greener pastures of product management or reliability engineering.

As a closing note, I’d like to leave you with the following video that makes a similar point to the one labored in this blog post, but in a much more entertaining way: “Test is Dead” by Alberto Savoia.