User Driven Development: The Lean startup principles implemented in enterprise software development

User driven development (UDD or User-Centered Design according to Wikipedia) is not really a set of methodologies but  a philosophy or paradigm that a development team can follow. In today’s world, where interacting with technology becoming part of more and more everyday tasks, the single most important factor that will make a service successful, is the end user satisfaction. This is mainly true for products that people would want to use and not for ones they have to use. An example of the latter is large scale enterprise systems which are built to serve a specific function, based on already known tasks.

Unlike consumer software, enterprise products don’t have to strive for consumer satisfaction in order to grow. The requirements are usually very extensive and mainly concern security, usability, maintainability etc. If UDD is used, these requirements will again be present, but only implemented when needed. Resources can be saved by not spending unnecessary money in procuring expensive software solutions such as Database Management Systems.

In this post I’ll talk about what UDD is, the processes involved and why even enterprise development teams should incorporate its philosophies in their daily processes.

The Process

The basis of UDD is pretty self-explanatory from its name. UDD’s main goal is not to just always make decisions based on user requirements, but also involve the user in the development cycle from the starting point. UDD incorporates the principles of the Lean startup with the methodologies of Agile development. The Lean startup is a method proposed by Eric Ries and is based on Lean manufacturing. A quick Wikipedia check will tell you that

 It’s a production practice that considers the expenditure of resources for any goal other than the creation of value for the end customer to be wasteful, and thus a target for elimination.

Ries translated that idea to software startups. In order to eliminate any wasteful practices, lean startups employ the following techniques:

According to the Lean methodology, a startup has to first create a Minimum Viable Product (MVP). A MVP is the first version of a new product that implements the most basic functionality that can be shown to the customer as quickly as possible and receive feedback. This helps clear up early assumptions and hypotheses so that the team has a firmer foundation to build on. Once this is ready, the team gets constant feedback from the users in order to start Continuous deployment on the product. Continuous deployment also ensures that user feedback will be given early for new features. Once the product grows in complexity, A/B testing is used against groups of users, to determine which features bring more value to the user.  This continuous process is also known as build – measure – learn.

A vital part of this process is that the developer does not rely on feedback acquired through questionnaires or collected by a different team. The developers need to demo the product in person and then collect information while the user is trying it out. This is especially true in the early stages of development, during which tools like Google analytics don’t provide any meaningful data from a handful of users. UDD usually employs Agile practices but, just like features, practices that are deemed to be wasteful are simply scrapped or replaced with more effortless alternatives.

If you’re interested in seeing UDD in practice in its purest form, check this video out where a team of developers iteratively create an app right in front of the users: https://www.youtube.com/watch?v=szr0ezLyQHY

 Why UDD?

It’s easy to find motivations for using UDD in a startup. Startups usually lack extensive funding, specific direction and a user base. The process of UDD helps with overcoming the early hump of choosing direction, building useful software with minimum funds and satisfying the user base in order to help it grow. These concerns however are not something that a team building enterprise software will have to worry about. Nonetheless, there are a few advantages that can be gained by building enterprise software following UDD practices.

We will use Ernie Miller’s article as an example of UDD implemented on an enterprise software. (http://erniemiller.org/2008/03/28/the-joy-of-user-driven-development/) In his article, Miller talks about how he built a replacement software for his company’s trouble ticketing system.  The author followed the process of closely interacting with the software’s users in order to get feedback on the necessary improvements needed. One of the first major changes in the ticketing application, was the removal of all ticket states except for resolved and unresolved. This was done because he realized that all other tickets states usually ended up never changing and remaining active in the database without anyone caring for them. This is a perfect example of wasteful features that can be eliminated.

According to Miller, the first version of the software served the most basic operations that were needed, and was only rolled out to a few employees in order to receive feedback. Within the years, the project was used by more and more employees and was changed radically according to focus group insights and word-of-mouth comments. Once again, this shows the basics of UDD in action. The interesting part of the article comes when the writer describes how upper management handled the news of a custom made system. They were not very happy about it. Miller describes their meetings as “speaking two different languages”, with him trying to explain the user benefits and the managers looking at the solution from a bottom-up view, concerned about database designs etc.

So why would a large company want to use UDD in the process of building their enterprise software?

  • Wasted resources are bad for everyone. It might be more essential for startups to make efficient use of their budget, but just because a team has more funds available, does not mean that more funds should be wasted. Resources also mean man-hours spent on a software. According to some metrics (http://www.slideshare.net/timmcowan/introduction-to-scrum-3285439), up to 45% of software features might never be used and 35% is rarely/sometimes used. This means only 20% of features are always/often used by users. This a huge waste of time by developers that could better be spent improving essential features instead of maintaining and useless ones.
  • The users still matter. The system’s success might not be based on the number of its users, since that number is pre-determined. But why would a company want its employees to use a cumbersome system filled with clutter, instead of something that might make more sense to them, and makes the process more pleasant, even if some minor features are missing (in the beginning; they could be implemented later on if considered to be important).
  • Face time with users. When building an enterprise system, or any other system for that matter, in a large group, developers often do not get any face-to-face time with their end users. In projects like these, it’s easy to get lost in the technical matters that are passed down from managers and completely forget the  conversation about the end-user. A user is not just a set of requirements and features, but a person who might just even have insight that the developer could have never thought of. Spending time with your end-user is a great practice for understanding their point of view in any product.
  • Learning new technologies. This might be more of a perk than an overall advantage of UDD. But if a developer has to solve a problem without pre-determined tools, the possibilities are endless. Maybe the team will finally get rid of the expensive (and often considered outdated) solution of using SQL server just because the company is partnered with Microsoft. The developer might discover that a NoSQL database can yield much better performance results for the required task and gain new skills in implementing the database.

Implementing UDD

So if we assume that the early statements is true, and companies building enterprise software should employ UDD, why can’t an enterprise software strive for user satisfaction as its main goal? As mentioned before, one of the problems is the lengthy list of functional requirements that an enterprise system needs to adhere to.  Another obstacle is the large scale of users the system will need to accommodate from the get-go. Also, since enterprise systems are usually necessary for the user’s daily routine, there is little room for failures.

However, some of the characteristics of enterprise software development can also be used in favor of the process.

  • User knowledge. Startups fight for the product-market fit. Their product is often something users don’t even know they need. Enterprise software on the contrary has to perform specific tasks. This can be used in favour of the feedback the developers receive. The user can compare and contrast with pre-existing solutions and offer valuable input on what they’d like to see from a product before it is even built.
  • Domain experience. This can be a double-sided coin. It’s hard for developers that have been working on the same product for years to suddenly think outside the box. But at the same time, experience is a powerful tool when it comes to figuring out a technical implementation to address a user-requested feature.
  • Funding. If UDD is used correctly, the large funding that a company may have for a specific product, can deliver a significantly more successful product than the result of more conservative methodologies.
  • Testing. Though one of the fundamentals of continuous deployment is to often build first, fix later, this is not necessary. Especially if a team has available testers, their input can be invaluable by automating processes that can ensure new features do not raise bugs. In other words Test-Driven Development can be utilized as a facilitator for UDD.

Conclusions

Agile methodologies are a great way of organizing a team. But UDD goes a step further by going directly to the user instead of building User Stories out of thin air. Large enterprise software companies might have been used to the way things are done, often slow and monotonous. But it is feasible to change this. Slowly and iteratively, even the most cumbersome systems can become a pleasure to use and get rid of the clutter that only makes a user’s experience worse and offers nothing to the overall picture.

Sources

Some more info on UDD and Lean principles:

Ernie Miller’s article:

Response Article to “The Joel Test: “Do you have testers?” – No, and neither should you (probably)”

The article https://blog.inf.ed.ac.uk/sapm/2014/02/11/do-you-have-testers argues that the role of the tester is outdated in modern software development, mainly due to its agile nature of continuous delivery. Clemens Wolff, the writer of the article in question, begins by describing an  “instant remedy” method to evaluate a software development team,  “The Joel Test” as well as other articles by Joel Spolsky that explain why testing is valuable. He then proceeds with countering the opinion for the need for testers by firstly describing a personal experience that led him to this belief and then pointing out several factors in the evolution of software that have rendered testing unnecessary.

Though Clemens’s article makes a few good arguments on how the nature of testing has radically changed, I remain unconvinced that testing as a separate practice is dead and unnecessary. In this article I will try and refute that argument by pointing out some of the points Clemens makes and why they don’t necessarily point to one certain conclusion. I will also briefly describe my own personal experience in the software industry as a tester to demonstrate that testing can be many different things and should not be easily dismissed as a whole.

Hasty Generalizations

The writer’s personal experience seems to be a main factor in his opinion being formed, and he gives a fairly good explanation of his team’s methodology and how it was effective without the use of dedicated testers. I will by no means try to refute that the Amazon Development Centre ships bad code but I can’t help but ask the question: What code does it ship? There is no context given about the product the team is working on, its intended users or anything else for that matter, besides their methodologies on ensuring software quality. Whatever their domain, even if these methodologies were indeed “successful” by some acceptance metric, I have no reason to believe they would be successful in a different team.

Furthermore, some of the points made, in my opinion, further supported Polsky’s statements that the writer is trying to refute. For example, one of the steps taken to assure software quality by the writer’s team was “Unit and integration tests written for any new feature”. If the developers write these tests themselves, then Spolsky’s point that “A team without dedicated testers wastes money” definitely holds within that team. Another one of Spolsky’s points is that programmers make bad testers since they look at the code from the writer’s point of view and trying to think of ways to brake your own code can be difficult.

 QA vs Testing

A main weakness of this article is that it seems to blend the job of QA and testing into one. Plenty of articles have been written that argue about the differences of the two, but they all agree on one thing: they’re different. Quality assurance encapsulates all the processes involved in evaluating not just the product, but also the development and/or maintenance processes used by the team. These can include evaluations like “The Joel Test”, the inclusion of documentation, team communications tools and of course testing. If the writer is trying to argue that testing should not be viewed as part of QA but part of the development process then I wholeheartedly agree with that. But that is not made clear in the article.

Testing is not just using a user interface until something doesn’t work and then reporting it. Testing is in its core software development. The difference is that the software developed by testers serves a very different purpose to the software developed by the developers.  The test code is by itself a whole different project (usually in terms of code as well) which is used to test parts of the product code. A tester is required to employ a very different way of approaching a problem which is why it is better to have testers as a separate unit; so that the developers don’t have to worry about it. It makes both the developer’s job easier, and ensures that the job done by the tester will be superior to that done by a developers/tester.

Testing in an agile environment

Another point that the author makes, is that the new world of agile has reduced the need for testers because of its continuous delivery nature. It’s true that web-based services make a-b user testing much easier than the old “software in a disc” paradigm, due to the ease of releasing different versions to small chunks of the user base. But user-acceptance tests are only a small part of testing. A service’s user will usually neither try and breach a new feature’s security nor will she expect that a certain input value might surface certain edge case problems. A user will simply expect the service to work and maybe give feedback on how it could work better.

Furthermore, in order to utilize a service’s user base to conduct insightful user testing, the service needs to firstly establish a user base. And until that is done, there is no way of knowing if the 1.0 version of that web service will work adequately without a testing team.

My personal experience

I spent a year working for Microsoft Mediaroom as a Software Development Engineer in Test. Just like with Amazon Development Center, most people would expect that the team functions in a high quality software development environment. During my internship, I worked on the delivery of a 1.0 version web service which would for the first time transfer Mediaroom’s capabilities to the cloud. We followed an agile methodology with small scrum teams (10-12 people) and the ratio of testers and developers was usually one to one. Here are a few reasons why testing was necessary in our team and why it did not hinder the agile development process.

  • Testers and developers worked closely. Though it was clear who was a tester and who a developer, the whole team participated in all decision making. Testers reviewed code written by developers and vice versa. If a tester found a bug, it was not necessarily a process of filing the bug and then waiting for it to be fixed, but often a solution could be found with a simple face to face chat. This ensured that both teams had good knowledge of what the other team was doing.
  • The client’s requirements. The product (or service) is B2B. Selling a service to another business and not to the end customer, usually means higher acceptance constraints. It might be OK for a single end user to identify a bug that can be fixed directly from the developer, but the last thing our team wanted was forcing a client to admit that a fault with their service was not due to their processes but due to the service they have acquired from a different company.
  • Testing did not block development. Writing tests for a new feature was considered a separate task to developing the feature. If the developer finished the product code near the end of a sprint cycle, then the testing task would simply be carried over to the next sprint and the developer could start working on a different feature while the tester was taking care of the previous one.
  • The product’s domain. The service was a VOD CMS that needed to be able to handle the whole service of uploading content to the cloud, encoding it in multiple formats and then distributing it to multiple different software clients (mobile, tablet, web, etc). Because of its end-to-end nature, the product had a big variety of testing that needed to be performed in an automated manner like load testing, API testing, component testing, user interface testing (for both the content manager and the end user) etc. The process of creating automated testbeds for all these processes is highly complicated and needs a design of its own. Developers should not have to worry about how the testing will be designed, but only how their code will pass the tests.
  • No user base. Since this was a v1.0 product, no there were no users available for user testing. Some clients would give feedback on product features but it was impossible to know the overall reaction once the product was finally released. Furthermore, since Microsoft would not be handling the end user’s experience eventually, end-user testing would never be 100% efficient from our team’s point of view.

Conclusions

“Testing is dead” is a statement that should not be thrown around with ease. “Testing has changed” is of course true just like every other aspect of software development. New agile methodologies have changed the nature of testing but if anything, they’ve made it more interesting and complicated. How a service will be tested depends on a multitude of factors like the user group, the team’s capability and of course the product itself. There are definitely services which because of their nature can be made available with a few bugs that will not pose security or other serious risks. But there will always be cases where software needs to be heavily tested before it is made available to the public, and the best people to do that are professionals who have dedicated their time in the sole purpose of improving testing technologies and methods.

Sources

  1. “Do you have testers?” –https://blog.inf.ed.ac.uk/sapm/2014/02/11/do-you-have-testers/
  2. Joel Spolsky’s articles:
    “The Joel Test” – http://www.joelonsoftware.com/articles/fog0000000043.html
    “Why Testers?” – http://www.joelonsoftware.com/items/2010/01/26.html
    “Top Five (Wrong) Reasons You Don’t Have Testers” – http://www.joelonsoftware.com/articles/fog0000000067.html
  3. On QA vs Testing
    http://intersog.com/blog/qa-vs-testing/
    http://www.mosaicinc.com/mosaicinc/rmThisMonth.asp

Behaviour Driven Development: Bridging the gap between stakeholders and designers

Introduction

It is obvious that when a large system is being developed, no matter the domain it functions within, many things can go wrong. Different applications will focus more on not getting things wrong in different aspects of the system. But the most common reason behind mistakes is the failure to communicate. In a smaller scale, this will can occur among the developer team or between developers and testers. But the larger the project, the more the people involved and the different aspects of successful delivery they need to address. The tree swing metaphor is a very popular and elegant (and perhaps a bit exaggerated) way to demonstrate this. It might be impossible to ever achieve a harmonious communication system across all teams and stakeholders involved, but there is a way to somewhat bridge the gap between the two main actors in a system’s development: the client and the engineering team.

The classic tree swing metaphor

The tester’s Problem: what do I test?

In an ideal world, the process of delivering requirements seems fairly simple and straightforward. Using a generalized Agile framework as an example to simplify discussion, the process is supposed to follow these steps: The client dedicates a team to creating user stories that should cover the product’s requirements and look a little something like this: “As an X, given that Y,  I want to be able to do Z”. The program manager of the development team receives the user stories and creates more specific requirements (use cases) for each of the stories that look like this “Enable user authentication” or “Implement functionality Z”. The program manager will then get together with the developers and testers of the team at the beginning of the sprint and break down the use cases to very specific tasks like “Implement client side authorization cookie”, “write fuzz tests for user authentication” etc.  Each task is assigned to a developer or tester appropriately and then everyone gets down to work. During the sprint duration the devs and testers will mark their tasks as done and at the beginning of the next sprint the process will be repeated depending on completed work so far.  It all sounds nice and easy, right? Well no. One of the first problems that the engineering team will face, is implementing the client’s user stories. Even though large corporate clients will usually have a team dedicated to defining the user stories appropriately, it is extremely common that some aspects will be left out.

The problem is that clients usually think in plain English just like most users will when using the service. Engineers  and testers on the other hand think in code (hopefully). Using a streaming service as a more specific example, what happens when the client writes the user story “as a service subscriber I want to click on the poster for movie X and is launched in the same window”. When the engineering team gets down to creating tasks the following questions can be raised when it comes to testing the story:

  • What if the user has another window open playing movie Y? Should they be able to stream multiple movies at once?
  • What if the user’s subscription does not allow them to watch the movie? Should they be simply shown a “we’re sorry” message? Or should they be guided to a page that will allow them to extend their subscription?
  • If the user clicks “open in new tab” should the browser just launch the video player or the same page with the player embedded to allow deep linking?
  • If the user had started watching the movie earlier closed the browser in the middle, should the movie start over or continue from where it was left off?

Questions like these might of course be covered by different user stories. But very often they’re not. Very often unanswered questions will block development and the program manager will need to get back to the client and wait for an answer. Besides time issues, this can also cause frustration on both sides and  harm the relationship with the client. The problem can also go both ways when the engineering team attempts to explain their testing scenarios and the client does not understand or does not find use in them.

Wouldn’t it be grand if communication of this type could be simplified?

Enter BDD: a middle-ground framework

During my year as a Software Development Engineer in Test at  a  large development team, we worked on a v1.0 streaming service that was partly tailored to the first client’s requirements.  Problems like the one described above were more than common. This was the first time the team was working using agile methodology and releasing a system that would follow the brave new world of continuous delivery instead of the conventional develop -> ship -> support timeline. So when trouble like this appeared, it really set the schedule behind. I would often receive a testing task, write the automation code and then publish it for review. A developer would then tell me I should also test case X which I had left out. I would then publish the 2nd version of my review and a different developer would come along and comment that this is not a requirement and might change soon so I should remove the test. That would lead to a lengthy discussion that would start from the comment thread in the review and then continue off line in the office until it reached a deadlock and had to be raised to the PM. The rest is history. Until one day, a senior tester proposed implementing BDD to our testing framework.

BDD is really just a fancier TDD framework, build to accommodate both the engineering and the client side. The difference is that the automated test methods are wrapped in a simple natural language parser that can read test scenarios written in plain English. In the streaming service example, the client would be asked to write the story in the following format:

Given I am logged in to the service
When I click on a movie
Then the movie will open within the window

The engineering team then builds a framework where the code looks for the Given – When – Then keywords which trigger methods that setup the testing environment as described in Given, run the steps described in When and make assertions that are given in Then.

This description of course doesn’t solve anything since it has the exact same level of ambiguity as the original story. But when the engineering team spots a constraint that should be added to the testing scenarios, they can extend the language definition and give it back to the client. A more specific and correct example of a test scenario would be:

Given I am logged in to the service
and I have access to “all content”
and I am “not” currently streaming another movie
and I had started watching that movie earlier
When I “left-click” on “movie X”
Then the movie will open “within the window”
and the movie will play “from where I had left it off”

The and clauses simply run more setup methods before the test is run and eventually make more assertions one the automation has been finished. The code behind will look something like this:

public class StreaminClientSteps
{
 private readonly clientPage;

 [Given(@"I am logged in to the service"]
 public void GivenLoggedInUser
 {
 ....
 }

 [Given(@"I have access to '(.*)'"]
 public void GivenAvailableContent(string content)
 {
 ....
 }

 [Given(@"I am '(.*)' currently streaming another movie")]
 public void GivenCurrentlyStreaming(string streaming)
 {
 ....
 }

 [When(@"I '(.*)' on '(.*)'")]
 public void GivenCurrentlyStreaming(string clickType, string movie)
 {
 clientPage.ClickMovie(clickType, movie);
 }

 [Then(@"the move will open '(.*)'")]
 public void AssertMovieOpened(string requestedWindow)
 {
 ....
 } 

 [Then(@" the movie will play '(.*)'")]
 public void AssertMoviePlayedFrom(string time)
 {
 ....
 } 
}

This is just a simplified example, and the definitions in the text might need to be more specific, but it shows how sentences can be reused to test different conditions.

Conclusion: Why it works

“So we spent a massive overhead to build this framework in order to run test methods that we would run anyway and the only difference is that they’re written in plain English. Why the hell did we do that?” a skeptic tester might ask. Since introducing the framework is a lengthy task, it needs to become a user story of its own and take some time from testers or developers until it’s implemented. In the long term though, here are some benefits that this framework will introduce.

  • It reduces testers’ need for initiative. The testers simply have to implement methods every time a new condition, automation or assertion is introduced without wondering what the test will actually do. When an untested occasion appears, they simply suggest implementation and continue with it if accepted.
  • It facilitates the client’s in identifying potential problems. Once they start receiving untested conditions, the clients will get accustomed to writing acceptance tests that will cover ever possible aspect of a situation and then assertions that need to be made.
  • It pushes for better code standards. The conversion from plain English forces the testers to write simple methods that only do one thing; that’s one of the basic criteria of “good code” which is often not followed by testers since their code is not included in the product code.
  • The need for new code is gradually reduced. New conditions will of course keep arising but once the main functionality testing has been implemented, hundreds of test can be quickly made by simply changing values in the Given – When – Then clauses.

And last but not least:

  • It solves the communication problem. Not completely. There will always be some friction between engineers and stakeholders that can originate from many stages of the development cycle. But when it comes to acceptance testing, using a common language removes the need of translation by the engineers and the other way. It makes clear to both sides what needs to be done.

 Sources:

  • http://dannorth.net/introducing-bdd/
  • http://cukes.info/
  • https://github.com/cucumber/gherkin/wiki
  • http://www.specflow.org/