A/B Testing: More than Just Sandpaper

In interface design A/B testing is a simple experiment in which randomized groups of users are presented with variations of the same interface and their behaviour is observed to better inform design decisions. This kind of testing is usually used to improve website conversions. That is because the World Wide Web is a media that is uniquely suited for A/B testing – it is comparatively inexpensive to present different users with modified versions of the same website and track their actions. It would not be feasible in the context of traditional media.

With success stories that report 50% increases in clicks by altering phrasing of the link or even helping to win presidential elections a great number of A/B testing services and tools (Google Website Optimizer, Amazon’s A/B Testing and Vanity to name just a few) has emerged. Not to mention the countless web posts.

In 2011 the company [Google] ran more than 7,000 A/B tests on its search algorithm. Amazon.com, Netflix, and eBay are also A/B addicts, constantly testing potential site changes on live (and unsuspecting) users.

— Brian Christian at Wired.com

However it is not the apparent ubiquity of A/B testing or success stories but a particular criticism of split testing that inspired me to write this article. In his blog post entitled Groundhog Day, or, the Problem with A/B Testing Jeff Atwood argues that A/B testing has no feeling, no empathy and only produces websites that are goal driven and can never win hearts and minds. Mr. Atwood quotes his friend’s tweet:

A/B testing is like sandpaper. You can use it to smooth out details, but you can’t actually create anything with it.

Nathan Bowers

I believe this to be a wrong way of looking at it. Obviously, A/B testing in itself cannot produce anything but it can guide the design process and quantify how good the final result is. It is difficult enough to avoid developer’s blindness and to work with the fact, that people do not know what they want, in mind. But in this day and age one also has to navigate the perils of multiculturalism. When developing a website that will potentially be accessed from all around the world, a developer or designer cannot possibly be expected to simply conjure the perfect solution out of thin air.

While Mr. Atwood seems to think of A/B testing purely as a way of monetising, I tend to side with some of the people who have commented on his blog post and think that testing democratises the process of software development and brings better outcomes for both the developers and the users.

I believe these same people cannot read the minds of every single person who visits their web site, or uses their app. Therefore, I think it’s great that these people can test both their ideas, rather than having to make some evidence-free guess and rationalize it after the fact. An A/B test is only as good as your best idea, after all. Ideas still matter!


This is not to say that good designers are unimportant but they cannot always predict what will attract the users to interact with their designs. A/B testing has shown time and time again that in some cases the solutions that violate the rules of visual composition or could even be perceived as vulgar are the most appealing.

To try and boost donations the digital team attempted to improve the design by making them look “prettier”.

That failed, so in response an “ugly” design was tested to see if that made any difference. This involved using yellow highlighting to draw attention to certain text within the email.

To the team’s surprise the ugly design actually proved to be quite effective, though the yellow highlighting had to be used sparingly as the novelty wore off after time.

David Moth

I can see where Jeff Atwood is coming form. It might seem that such scientifically rigorous tests subtract from the artistic, inspired or simply human qualities of design. Or that corporate values might suffer in the face of corporate greed. However, I am a firm believer that benefits of A/B testing far outweigh the risks. With the later being non existent when testing is treated as the irreplaceable source of insight it is rather than the deciding voice.

I might be going a step too far here but — honestly — the existence of A/B testing makes me hopeful that elegant solutions for other difficult software development problems (e.g. project timeline prediction) might be within our reach as well.

One thought on “A/B Testing: More than Just Sandpaper”

Comments are closed.