# The Joel Test: “Do you have testers?” – No, and neither should you (probably)

tl;dr

The times they are a-changin’. In the post-Agile world, not having dedicated testers might just be advantageous for you. Depending on the software your team produces, factors such as time-to-market might be more important than a perfect product. “Build the right it” first, “build it right” later.

## Setting the scene

In 2000, the prolific software writer Joel Spolsky proposed the “Joel Test”: twelve simple yes-or-no questions to judge the quality of a software development life-cycle. The test invites the inquirer to ask questions such as “Do you use source control?” or “Do programmers have quiet working conditions?”. According to Spolsky, any good software development process will answer “no” to at most two of the test’s questions.

Fourteen years later, the test is still widely recognized as a useful, low-cost tool to evaluate the efficacy of a software development process. But is it really? A lot has happened in the world of software engineering since the test’s inception. When reading Spolsky’s original proposal nowadays, I can’t help but feel as though some of the questions are overly dogmatic or at least a bit dated.

For instance, question 10 of the tests states that

[i]f your team doesn’t have dedicated testers, at least one for every two or three programmers, you are either shipping buggy products, or you’re wasting money by having $100/hour programmers do work that can be done by$30/hour testers. Skimping on testers is such an outrageous false economy that I’m simply blown away that more people don’t recognize it.

Source: “The Joel Test”.

In the remainder of this blog post, I will analyze this assertion and discuss to what extent its premise is still relevant in 2014.

## Spolsky’s argument then versus in the trenches now

Let us first have a look at the argument made by Spolsky for why dedicated testers are imperative for a good software development process and then contrast this with my personal experiences in the software world.

Question 10 of the “Joel Test” (quoted above) observes two issues:

• A team without dedicated testers will release a bad product because of lacking quality assurance (QA) from both technical and functional points of view.
• A team without dedicated testers wastes money since testers are cheaper than developers.

In later posts, Spolsky refines his point further:

• Programmers make bad testers – the required skill sets are not related.
• Having testers shortens the time between developers and getting feedback on product quality.
• Testers will catch problems that the original developer didn’t (or wouldn’t).

These assertions do not match my personal experience. Time for some anecdata.

Last summer I worked at the Amazon Development Centre Scotland (ADCS). Most people would likely agree that ADCS is a high quality software development environment. As a matter of fact, ADCS would obtain a perfect score on the “Joel Test” if it were not for the lack of dedicated testers on my team… however, this does not mean that ADCS ships bad code, as Spolsky claims must inevitably happen. Far from it. Software quality is simply assured by other (in my opinion superior) means:

• Strict pair programming when code is being touched.
• Unit and integration tests written for any new feature.
• Before a major feature is released, the entire team pitches in and spends half a day to a day focusing on testing the new feature. If any show-stopper bugs are found, the release of the feature is delayed.
• One person on the team is “on-call” every week, handling bug reports as they come in.

Arguably, this Agile-inspired approach has a number of advantages over having dedicated testers:

• Everyone in the team has a rough idea of what is going on. There is less of a chance of knowledge silos developing.
• There is a strong incentive to produce high quality code as your colleagues are going to be directly affected by the mess you cause (as opposed to testers in some other team you might not know).
• More eyes see the code than with Spolsky’s recommended “1 tester per 2 engineers”, increasing the chance of finding bugs or poor user experience.
• Developers will never have to wait for the testing team. Automated tests give instant feedback. More involved evaluation such as user acceptance testing can be prioritized appropriately by the developers in order to integrate it seamlessly with the rest of the development process.

In addition to these benefits, I also found that foregoing dedicated testers means that the development team is more likely to develop a strong sense of ownership of their product. The team owns the software development life cycle end to end and is therefore more likely to “do the right thing and do it right”: develop reliable solutions that do what they are supposed to do. Contrast this with just implementing some specification, passing it on to testers (or later business analysts) and never looking back… Farewell, “code, compile and forget”. You shall not be missed.

## A brave new world?

So how do we consolidate this difference between what the “Joel Test” claims should happen and what actually does happen on the grounds, even at a top-tier, quality obsessed company like Amazon?

It is important to note that the “Joel Test” was written in 2000 and very much reflects the spirit of the time in its attitude towards testing. A lot has happened in the world of software since then; most notably, development methodologies have shifted away from the Waterfall-like methods prevalent at the start of the century to a wide-spread adoption of Agile methods.

Software produced in the times of the “Joel Test” would have one, two, maybe a handful of releases a year, indiscriminately distributed via physical media to a company’s entire client base. Nowadays, the near-ubiquitous use of web-based services means that new software versions can often be rolled out instantly or to only a small part of your company’s customers. Being able to perform split tests on your product by releasing new features gradually into the wild means that the impact of bugs or poor design is less severe than with the old model where a company’s reputation could rise or fall on the newest polycarbonate-plastic pressed version of its developers efforts. Thus reduced the importance of the tester.

Furthermore, the modern-day developer has access to a plethora of new tools that reduce the burden of test. While some of these tools might have been available at the time of the “Joel Test”, they would have been nowhere near as ubiquitous as nowadays when FindBugs is in the top ten most downloaded Eclipse plugins and all major Java IDEs come with JUnit installed by default. Static code analysis finds a plethora of bugs even before the developer hits “compile”. Unit and integration tests take snapshots of the functionality of a system and therefore prevents test regressions by default. This eliminates a big chunk of the grunt work that formerly had to be done manually by QA departments. Thus reduced the importance of the tester.

Additionally, the switch to Agile-like methods brought about a mentality change in the software field. Previously, QA teams were often “separate but unequal” parts of the development apparatus. Under Agile and its offspring like Test Driven Development, everyone is a tester: everyone wears all hats. The role of QA can thus grow to encompass more directly technical or business-related aspects such as focusing on user acceptance testing and performance- or security-testing. From a button-mashing unequal to a respected fellow reliability engineer. Thus reduced the importance of the tester.

However, the shift to Agile can’t explain the entire difference between what the “Joel Test” claims and what I observe in the modern software world. Changes in tools and mentality have shifted some of the burden of test from traditional testers to the entire development team… but traditional Agile is still far from the laissez-faire “ship it” attitude that I see pervading the software world. As a matter of fact, Agile evangelist and writer Don Wells claims that

[q]uality assurance (QA) is an essential part of the [Extreme Programming (XP)] process. On some projects QA is done by a separate group, while on others QA will be an integrated into the development team itself. In either case XP requires development to have much closer relationship with QA.

Source: Extreme Programming Rules

I think that this remaining disjoint can be explained by observing that some companies might have moved on from Agile and are adopting elements from even more “lean” philosophies. “Idea bugs” are harder to fix than software bugs. If you build a product that people really want to use, they will be forgiving (prime example: the Twitter service-outage whale). Getting something out of the door and running it by real-world users is thus increasingly becoming the most important aspect of software development: delivering products, not code – focusing on “building the right it” rather than “building it right”. This is something traditional QA can’t really help with. And therewith died testing.

## The future of test and concluding remarks

Yet test will surely live on with “I’m not dead yet” tenacity. The software forest is not only comprised of Agile web development trees. Traditional testing is sure to find respite in havens such as AAA game development with its ever-present big releases or security- and safety-critical programming with its zero-tolerance for bugs (e.g. medical or aeronautical software). For the remaining denizens of QA-land, however, I fear that the time has come to leave testing drugery behind and move on to the greener pastures of product management or reliability engineering.

As a closing note, I’d like to leave you with the following video that makes a similar point to the one labored in this blog post, but in a much more entertaining way: “Test is Dead” by Alberto Savoia.

## The Anaemic Domain Model is no Anti-Pattern, it’s a SOLID design

Design Patterns, Anti-Patterns and the Anaemic Domain Model

In the context of Object-Oriented software engineering, a “design pattern” describes a frequently recurring and effective solution to a commonly encountered problem. The utility of formalising and sharing design patterns is to provide a set of “battle-tested” designs as idiomatic solutions for classes of problems, and to increase the shared vocabulary among software engineers working on collaboratively developed software. The term was coined in the seminal book by Gamma et al [5], which named and described a set of common design patterns. The lexicon of design patterns grew from the initial set specified in the book, as the notion gained in popularity [6], [17].

Following the increasing popularity of design patterns as a concept, the idea of “design anti-patterns” entered popular discourse [7][8]. As implied by the name, an anti-pattern is the opposite of a pattern; while it too describes a recurring solution to a commonly encountered problem, the solution is typically dysfunctional or ineffective, and has negative impacts on the “health” of the software (in terms of maintainability, extensibility, robustness, etc.). Anti-patterns serve a similar purpose to patterns; the description of the anti-pattern might illustrate a typical implementation of the anti-pattern, explain the context it generally occurs in, and show how the implementation results in problems for the software.

A potential problem with the concept of a design anti-pattern is that it might discourage critical thought about the applicability of the pattern. A design that may be inappropriate in some contexts may be a sensible decision in others; a solution might be discarded after being recognised as an anti-pattern, even though it would be a good fit for the problem at hand.

Why is the Anaemic Domain model considered by some to be an Anti-Pattern?

Fowler [1] and Evans [2] describe an ADM as consisting of a set of  behaviour-free classes containing business data required to model the domain. These classes typically contain little or no validation of the data as conforming to business rules; instead, business logic is implemented by a domain service layer. The domain service layer consists of a set of types and functions which process the domain models as dictated by business rules. The argument against this approach is that the data and methods are divorced, violating a fundamental principle of Object-Oriented design by removing the capability of the domain model to enforce its own invariants. In contrast, while an RDM consists of the same set of types containing necessary business data, the domain logic is also entirely resident on these domain entities, expressed as methods. The RDM then aligns well with the related concepts of encapsulation and information hiding; as Michael L. Scott states in [9], “Encapsulation mechanisms enable the programmer to group data and the subroutines that operate on them together in one place, and to hide irrelevant details from the users of an abstraction”.

In an RDM, the domain service layer is either extremely thin or non-existent [20], and all domain rules are implemented via domain models. The contention is that domain entities in a RDM are then entirely capable of enforcing their invariants, and therefore the system is sound from an Object-Oriented design perspective.

However, the capability of a domain entity to enforce local data constraints is only a single property in a set of desirable qualities in a system; while the ADM sacrifices this ability at the granularity of the individual domain entities, it does so in exchange for greater potential flexibility and maintainability of the overall implementation by allowing the domain logic to be implemented in dedicated classes (and exposed via interfaces). These benefits are particularly significant in statically typed languages such as Java and C# (where class behaviour cannot simply be modified at run-time) for improving the testability of the system by introducing “seams” [10], [11] to remove inappropriate coupling.

A Simple Example

Consider the back end of an e-commerce website in which a customer may purchase items, and offer items for sale to other customers across the globe. Purchasing an item reduces the purchaser’s funds. Consider the implementation of how a customer places a purchase order for an item. The domain rules state that the customer can only place an order if they have enough funds, and the item must be available in region for that customer. In an RDM, a Customer class would represent the domain entity for the customer; it would encapsulate all the attributes for the customer, and present a method such as PurchaseItem(Item item). Like Customer, Item and Order are domain models representing purchasable items and customer orders for items respectively. The implementation of the Customer (in pseudo-C#) might be something like;

/*** BEGIN RDM CODE ***/

class Customer : DomainEntity // Base class providing CRUD operations
{
// Private data declared here

public bool IsItemPurchasable(Item item)
{
bool shippable = item.ShipsToRegion(this.Region);
return this.Funds >= item.Cost && shippable;
}

public void PurchaseItem(Item item)
{
if(IsItemPurchasable(item))
{
Order order = new Order(this, item);
order.Update();
this.Funds -= item.Cost;
this.Update();
}
}
}

/*** END RDM CODE ***/

The domain entities here implement the Active Record pattern [17], exposing Create/Read/Update/Delete methods (from a framework/base class) to modify records in the persistence layer (e.g., a database). It can be assumed that the PurchaseItem method is invoked in the context of some externally managed persistence layer transaction (perhaps initiated by the HTTP request handler/controller, which has extracted a Customer and an Item from the request data). The role of the Customer domain entity in this RDM is then to model the business data, implement the business logic operating on that data, construct Order objects for purchases, and interface with the persistence layer via the Active Record methods; the model is Croesus-like in its richness, even in this trivial use case.

The following example demonstrates how the same functionality might be expressed using an ADM, in the same hypothetical context;

/*** BEGIN ADM CODE ***/

class Customer { /* Some public properties */ }
class Item { /* Some public properties */ }

class IsItemPurchasableService : IIsItemPurchasableService
{
IItemShippingRegionService shipsToRegionService;

public bool IsItemPurchasable(Customer customer, Item item)
{
bool shippable = shipsToRegionService.ShipsToRegion(item);
return customer.Funds >= item.Cost && shippable;
}
}

class PurchaseService : IPurchaseService
{
ICustomerRepository customers;
IOrderFactory orderFactory;
IOrderRepository orders;
IIsItemPurchasableService isItemPurchasableService;

// constructor initialises references

public void PurchaseItem(Customer customer, Item item)
{
if(isItemPurchasableService.IsItemPurchasable(customer, item))
{
Order order = orderFactory.CreateOrder(customer, item);
orders.Insert(order);
customer.Balance -= item.Cost;
customers.Update(customer);
}
}
}

/*** END ADM CODE ***/

Contrasting the example with respect to the SOLID Principles

At first glance, the ADM is arguably worse than the RDM; there are more classes involved, and the logic is spread out over two domain services (IPurchaseService and IItemPurchasableService) and a set of application services (IOrderFactory, ICustomerRepository and IOrderRepository) rather than resident in the domain model. The domain model classes no longer have behaviour, but instead just model the business data and allow unconstrained mutation (and therefore lose the ability to enforce their invariants!). Given these apparent weaknesses, how can this architecture possibly be better than the altogether more Object-Orientation compliant RDM?

The reason that the Anaemic Domain Model is the superior choice for this use case follows from consideration of the SOLID principles, and their application to both of the architectures [12]. The ‘S’ refers to the Single Responsibility Principle [13], which suggests that a class should do one thing, and do it well, i.e., a class should implement a single abstraction. The ‘O’ refers to the Open/Closed Principle [14], a similar but subtly different notion that a class should be “open for extension, but closed for modification”; this means that, in so far as possible, classes should be written such that their implementation will not have to change, and that the impact of changes is minimised.

Superficially, the Customer class in the RDM appears to represent the single abstraction of a customer in the domain, but in reality this class is responsible for many things. The customer class models the business data and the business logic as a single abstraction, even though the logic tends to change with higher frequency that the data. The customer also constructs and initialises Order objects as a purchase is made, and contains the domain logic to determine if a customer can make an order. By providing CRUD operations through a base class, the customer domain entity is also bound to the persistence model supported by this base implementation. By enumerating these responsibilities it is clear that even in this trivial example, the RDM Customer entity exhibits a poor separation of concerns.

The ADM, on the other hand, decomposes responsibilities such that each component presents a single abstraction. The domain data is modelled in “plain-old” language data structures [18], while the domain rules and infrastructural concerns (such as persistence and object construction) are encapsulated in their own services (and presented via abstract interfaces). As a consequence, coupling is reduced.

Contrasting the flexibility of the RDM and ADM architectures

Consider scenarios in which the RDM Customer class would have to be modified; a new field might be introduced (or the type of an existing field may need changed), or the Order constructor may require an additional argument, or the domain logic for purchasing an item may become more complex, or an alternative underlying persistence mechanism might be required which is unsupported by the hypothetical DomainEntity base class.

Alternatively, consider scenarios in which the ADM types must change. The domain entities which are responsible for modelling the business data will only need modified in response to a requirements change for the business data. If the domain rules determining if an item is purchasable become more complex (e.g., an item is specified to only be sold to a customer above a certain “trust rating” threshold), only the implementation of IsItemPurchasableService must change, while in the RDM the Customer class would require changing to reflect this complexity. Should the ADM persistence requirements change, different implementations of the repository [17], [19] interfaces can be provided to the PurchaseService by the higher-level application services without requiring any changes whereas in the RDM, a base class change would impact all derived domain entities. Should the Order constructor require another argument, the IOrderFactory [5] implementation may be able to accommodate this change without any impact on the PurchaseService. In the ADM each class has a single responsibility and will only require modification if the specific domain rules (or infrastructural requirements) which concern the class are changed.

Now consider a new business requirement was added to support refunds for purchases with which a customer is unsatisfied. In the RDM, this might be implemented by adding a RefundItem method to the Customer domain entity, given the simplistic argument that domain logic related to the Customer belongs as a member function of the Customer domain entity. However, refunds are largely unrelated to purchases, for which the Customer domain entity is already responsible, further mixing the concerns of this type. It can be observed that in an RDM, domain entity classes can accumulate loosely related business logic, and grow in complexity. In an ADM, the refund mechanism could be implemented by introducing a RefundService class, solely concerned with the domain logic for processing refunds. This class can depend on the narrow set of abstractions (i.e., interfaces of other domain and infrastructural services) required to implement its single concern. The new RefundService can be invoked at high level (in response to some refund request), and this new domain behaviour has been implemented without impacting any of the existing functionality.

In the example, the ADM solves the problem of bundling unrelated concerns into the same module identified in the RDM by taking advantage of the ‘I’ and ‘D’ in SOLID, namely the Interface Segregation Principle [15] and the Dependency Inversion Principle [16]. These principles state that an interface should present a cohesive set of methods, and that these interfaces should be used to compose the application (i.e., the domain service layer in the ADM). The interface segregation principle tends to result in small narrowly focussed interfaces such as our IItemShippingRegionService and IIsItemPurchasableService, as well as abstract repository interfaces; the dependency inversion principle compels us to depend on these interfaces, to decouple the implementation of a service from the details of the implementation of another.

The Anaemic Domain Model better supports Automated Testing

As well as more flexible and malleable application composition, adoption of these principles allows the ADM to extract the indirect benefits over RDM of simpler automated testing; this is because highly cohesive, loosely coupled components which communicate via abstract interfaces and are composed via dependency injection allow for trivial mocking of dependencies. This means that in the ADM it is simple to construct a scenario in an automated test which might be more complicated to construct in an RDM, so the maintainability of the automated tests is improved; the effect of this is that automated testing has a lower cost, so developers will be more inclined to create and maintain tests. To illustrate this, consider the example above, such that unit tests are to be written for the IsItemPurchasable.

The (current) domain rules for an item being purchasable are that the customer has sufficient funds, and is in a region that the item ships to. Consider writing a test that checks that when a customer has sufficient funds but is not in a shipping region for the item, the item is not purchasable. In the RDM this test might be written by constructing a Customer and an Item, configuring the customer to have more funds than the item costs, and configuring the customer region to be outside the regions the item ships to, and asserting that the return value of customer.IsItemPurchasable(item) is false. However, the IsItemPurchasable method depends on the implementation details of the ShipsToRegion method of the Item domain entity. A change to the domain logic in Item might change the result of the test. This is undesirable, as the test should be exclusively testing the logic of the customer’s IsItemPurchasable method; a separate test should cover the specifics of the item’s ShipsToRegion method. As domain logic is expressed in the domain entity, and the concrete domain entity exposes the interface to the domain logic, implementations are tightly coupled such that the effects of changes cascade, which in turn makes automated tests brittle.

The ADM, on the other hand, expresses the IsItemPurchasable domain logic on a dedicated service, which depends on an abstract interface (the ShipsToRegion method of IItemShippingRegionService). A stubbed, mock implementation of IItemShippingRegionService can be provided for this test, which simply always returns false in the ShipsToRegion method. By decoupling the implementations of the domain logic, each module is isolated from the others and is insulated from changes in the implementation of other modules. The practical benefits of this are that a logic change will likely only result in the breakage of tests which were explicitly asserting on the behaviour which has changed, which can be used to validate expectations about the code.

Refactoring the RDM to apply SOLID tends to result in an ADM

A proponent of the RDM architecture might claim that the hypothetical example provided is not representative of an true RDM. It might be suggested that a well implemented Rich Domain Model would not mix persistence concerns with the domain entity, instead using Data Transfer Objects (DTO’s) [18, 17] to interface with the persistence layer. The inclusion of directly invoking the Order constructor might be viewed as constructing a straw man to attack; of course no domain model implementation would bind itself directly to the constructor of another object, using a factory is just common sense [5]! However, this appears to be an argument for applying the SOLID principles to the application level infrastructural services, and disregarding the SOLID principles for domain design. As the hypothetical RDM is refactored to apply the SOLID principles, more granular domain entities could be broken out; the Customer domain entity might be split into CustomerPurchase and CustomerRefund domain models. However, these new domain models may still depend on atomic domain rules which may change independently without otherwise affecting the domain entity, and might be depended on by multiple domain entities; to avoid duplication and coupling, these domain rules could then be further factored out into their own modules and accessed via an abstract interface. The result is that as the hypothetical RDM is refactored to apply the SOLID principles, the architecture tends towards the ADM!

Conclusion

By exploring the implementation of a straightforward example, we have observed that an Anaemic Domain Model better adheres to the SOLID principles than a Rich Domain Model. The benefits of adherence to the SOLID principles in the context of domain design were considered, in terms of both loose coupling and high cohesion and resulting increased flexibility of the architecture; evidence of this flexibility was that testability was improved by being able to trivially provide stubbed test implementations of dependencies. By considering a how the benefits of adherence to the SOLID principles might be gained in the RDM, the refactoring tended to result in an architecture resembling an ADM. If adherence to the SOLID principles is a property of well engineered Object-Oriented programs, and an ADM adheres better to these principles than an RDM, the ADM cannot be an anti-pattern, and should be considered a viable choice of architecture for domain modelling.

References

[1] Fowler, Martin. Anaemic Domain Model. http://www.martinfowler.com/bliki/AnemicDomainModel.html, 2003.

[2] Evans, Eric. Domain-driven design: tackling complexity in the heart of software. Addison-Wesley Professional, 2004.

[3] Martin, Robert C. The Principles of Object-Oriented Design. http://butunclebob.com/ArticleS.UncleBob.PrinciplesOfOod, 2005.

[4] Martin, Robert C. Design principles and design patterns. Object Mentor, 2000: 1-34.

[5] Erich, Gamma, et al. Design patterns: elements of reusable object-oriented software. Addison Wesley Publishing Company, 1994.

[6] Wolfgang, Pree. Design patterns for object-oriented software development. Addison-Wesley, 1994.

[7] Rising, Linda. The patterns handbook: techniques, strategies, and applications. Vol. 13. Cambridge University Press, 1998.

[8] Budgen, David. Software design. Pearson Education, 2003.

[9] Scott, Michael L. Programming language pragmatics. Morgan Kaufmann, 2000.

[11] Osherove, Roy. The Art of Unit Testing: With Examples in. Net. Manning Publications Co., 2009.

[12] Martin, Robert C. Agile software development: principles, patterns, and practices. Prentice Hall PTR, 2003.

[13] Martin, Robert C. SRP: The Single Responsibility Principle. http://www.objectmentor.com/resources/articles/srp.pdf, Object Mentor, 1996.

[14] Martin, Robert C. The Open-Closed Principle. http://www.objectmentor.com/resources/articles/ocp.pdf, Object Mentor, 1996.

[15] Martin, Robert C. The Interface Segregation Principle. http://www.objectmentor.com/resources/articles/isp.pdf, Object Mentor, 1996.

[16] Martin, Robert C. The Dependency Inversion Principle, http://www.objectmentor.com/resources/articles/dip.pdf, Object Mentor, 1996.

[17] Fowler, Martin. Patterns of enterprise application architecture. Addison-Wesley Longman Publishing Co., Inc., 2002.

[18] Fowler, Martin. Data Transfer Object. http://martinfowler.com/eaaCatalog/dataTransferObject.html, Martin Fowler site, 2002.

[19] Fowler, Martin. Repository. http://martinfowler.com/eaaCatalog/repository.html, Martin Fowler site, 2002.

[20] Fowler, Martin. Domain Model. http://martinfowler.com/eaaCatalog/domainModel.html, Martin Fowler site, 2002.