When we take measurements with a pinch of salt

Once we agree that effort estimation for project completion is better when based on actual measurements, we tend to phase out human input. I think we should use measurements to give a perspective to developers and start the estimating discussion from there. This gives more information when estimating, but ultimately leaves the developers responsible for the resulting estimation; this is a good thing, since the measurements cannot capture everything that we, as humans, know about a project and it has other positive consequences that I will discuss.

Context

Consider effort estimation. We do it before starting a new project so that we know how many resources (developers, time) we need to allocate to it. Take the simple example of a team of developers that have worked together before. They have experience of previous projects of different sizes and try to estimate effort directly (together! estimation should not be done only by management). They apply some methods to reach a consensus (such as Three-Point or Wideband-Delphi estimating). After a while of doing this, they realise for each estimation, the discussion always includes some metrics — they need to first get some data before estimating.

Now suppose the team realises that effort is difficult to estimate directly. They want to automate the process a little, based on their findings. They start to base their estimates on other metrics, that they also estimate. Examples of such metrics are the projected size of a project and measures of complexity such as the number of its components. Now they start measuring previous projects in order to have some data to base estimates on. Since they know the resulting values of this process are not very precise, they try to use many metrics to at least get an accurate estimate.

This seems like a natural progression. However, I think we jump too quickly to relying on data directly and phasing out estimates done by developers. We know that however hard we try, the measurements will not be able to capture all that we know about a project’s size and complexity. We also know that project development is not only about continuous coding, it is also about a lot of refactoring, design decisions and collaboration. People have a better feeling about the latter than what our measurements can capture, especially in a familiar team of developers.

Consequences

First, better estimates. Developers cannot blame the methods for bad estimations when the final responsibility is on their shoulders. They will be more involved and this will result in more frequent updates to the project timeline. They will tend to more quickly admit to estimation mistakes because the method tells them to expect the initial estimates might be wrong. Since their input is valued over the measurements, they will be more confident and interested in repairing the mistakes proactively. This avoids the situation where a developer tries to conform to initial estimates at the cost of a bad product.

Second, developers are no longer incestivised wrongly. If the estimation is only based on measurements, developers might develop a bias to confirm those measurements. Suppose they estimated the size of the project to be between 10KLOC and 20KLOC. Then the effort estimates will be based on previous projects of the same size. Even if this project turns out it needs 30KLOC for completion, it does not mean the effort estimation was wrong; there are other factors to take into consideration, the size estimates and measurements should not be analysed individually. However, because the developers estimated less than 30KLOC, they are then incentivised to develop more concisely; this might consume unnecessary effort and end up sabotaging the project. Now also suppose the developers were given responsibility of the final effort estimate. Then the initial LOC estimate is not that important, it was just an aid; if it changes, the developers will be more likely to adjust the effort estimate rather than try to prove they were right about that particular measurement.

Finally, developers are encouraged to follow trends in their behaviour. We can build stronger teams when we treat developers as people who want to get better at what they do, not just as coding machines. They are not encouraged to prove their measurements right, they are now encouraged to use that information to their best interest. This might translate to new creative ways of measuring their development effort. Since they are using the data directly for estimates, they can identify factors better than any outsider and most importantly interpret them better. This could lead for example to developers starting to hashtag their commits so they can easily differentiate between e.g. drafting, refactoring, code writing etc.

Conclusion

Involving the developers in the final effort estimates gives them incentives to make better measurements, adjust more frequently and finally translates to a method of constantly improving estimates — now the developers have the power to repair their mistakes, and they are best informed to do this.

Note: if some of the concepts or names are unfamiliar and Google is not helping, keep in mind I wrote my article around these two lectures: estimation and measurement.

Response to: /* DELETE THIS COMMENT */

This is a response article to Scott Murray’s which can be found at blog.inf.ed.ac.uk/sapm/2014/02/21/delete-this-comment/.

I enjoyed the article so much it made me rethink one of my ongoing projects. The main idea discussed is that in many cases comments within code reflect poor design decisions. The author includes some very illustrative examples and shows how refactoring can render the comments useless and improve the design of the software. The article also discusses other strategies/technologies that can help drive down the number of comments. While I agree with the main point and most of its supportive ideas and arguments, I would like to discuss the cases where comments are really needed.

The simplest situations that recommend comments are those where a third party dependency results in strange behaviour, and they have been recognized in the original article. I will give examples of other situations where comments are desirable, but don’t necessarily reflect design issues within the code or its dependencies. This will support the fact that in some cases, comments are not only needed, but don’t pose risks and thus we should not seek to replace them with other techniques.

If you read the original article, you will note we are running some assumptions here. Obviously depending on the environment used, the perfect design cannot be implemented sometimes. Comments are welcome in these situations, since they explain the unexpected to the reader. We will further ignore these situations in our discussion.

Suppose you are writing a function to analyse a large set of data. The methodology is not well known, and there is no theoretical background you can point to that exactly reflects this situation. Let’s use an example. There is a matrix called features which holds vectors of features; for a given pair of row_index and column_index, you are trying to estimate the expected value of the feature at the corresponding position in the matrix (note you already know its value). To do this, you want to use the information from the nearest neighbours of the vector at row_index, but the high-level decision is that you don’t want to use the attribute information of column_index to compute those neighbours. In MATLAB, the similarity metric could easily manipulate the input data to achieve this goal:

features(:, [1:column_index-1 column_index+1:end])

which is interpreted as “from features, select all rows and all columns except column_index“.

The hiding of this column is dictated by the methodology. Instead of trying to come up with clever names for an intermediate variable that will hold the result of the code example (think features_without_column_to_estimate), a comment explaining what the goal of the code is would be more appropriate. Keep in mind this decision is made once before the code being implemented, and although it might never be changed (or even read again), it is good to have an explanation for posterity. If the methodology changes, it is rarely the case that someone will try to change this code; more often than not, a new function will be created for evaluation of estimates, if there is a new and possibly better idea.

The given example illustrates that in some cases comments might be the best way to document the code (as opposed to automated tools or other techniques). In these cases the users of the code will usually not care about its inner workings, its methodology. If for some reason someone needs to understand the code (be it debugging or methodology improvements), comments can explain the goal and intent of the code; they do not reflect bad design decisions and should not be replaced by other, more complex, tools.

When do conventions become detrimental?

There are many good arguments for creating and following conventions (I am referring to rules and patterns here, not conferences). However, conventions rarely pop up in lists about essential advice to follow in large scale software projects. Think of the common code convention which most teams have, that everyone should follow the same formatting etiquette. Teams tend to a code convention and it is not necessary to impose it. In other words, software developers create conventions when needed and we can find examples where imposing them would be detrimental, even if in good spirit.

Main advantages and drivers

Usually conventions follow from patterns. When possible, humans prefer to follow patterns in many aspects of their lives, as it makes decisions easier. Think of your cutlery drawer: why do you separate knives based on their type? (If you don’t, you should.) It is because it boosts your cooking productivity with a minimal effort. Once you learn where everything goes, you will know where to find it.

The general idea when creating conventions is to find patterns that simplify or abstract tasks; then automate where needed or possible and finally impose conventions such that the automations don’t fail. So when you don’t have to look for the bread knife in your cutlery drawer because you know exactly where you (always) put it, it is like when you don’t have to look for that opening curly bracket in your code because you know exactly where you (always) put it – probably before the extra indentation.

Conventions can be of benefit to project management and productivity in a variety of situations. As we saw earlier, one of the simplest conventions is a code convention – this is an example of a convention that, if not followed, will not break the builds. Some more complex conventions can be made about naming or project structuring – here, if the convention is not followed, the build might break (unless extra configuration is possible and made).

The main advantages of conventions are better productivity and developer happiness. Rules are part of our lives and serve the purpose of making them easier. We don’t like repetitive tasks and we don’t like when confusion gets in the way of our work. When all team members follow the same rules, one member’s code from the previous year can be as easy to understand for another member as their own code from the previous week.

Main dangers and pitfalls

We assumed patterns are universal. Conventions easily fail when a pattern is assumed where it shouldn’t. Think again of your cutlery drawer and suppose you are trying to learn how to use chopsticks; you assumed you would never need room for them, but now you have to break convention to find an appropriate place.

Note that conventions can be self-imposed, but they can also be imposed by external tools. Some you can easily change; maybe there was room for chopsticks in your drawer, but you just used it for something else. Others are more difficult to reconfigure; maybe you simply can’t accommodate your new chopsticks in your cutlery drawer.

We also assumed that once a convention is established, all team members will follow it. However, this might not always be true. Maybe by accident or oversight, someone will break the convention at some point. Maybe by necessity, they will have to work around the convention to achieve a goal that does not fall within the established pattern. The problem here is that a convention also creates expectations. When someone else will try to fix a bug in this code after the author left the team, they will expect the conventions to be followed. Thus they might lose more time getting over the initial confusion than they would have if there weren’t any conventions.

Imposed conventions

After thinking about the good and the bad of using conventions, one might wonder what the signs that tell us to avoid creating conventions are. To better identify them, let us first try to put them in contrast to the signs that tell us we should create conventions.

A good working example here is Ruby on Rails, or Rails for short. Rails is a framework for creating web applications that has become very popular in the last decade. It embraces ‘convention over configuration.’ This means that many decisions have been made to reduce configuration time for application development by establishing conventions. The problem is that it has failed quite a few times as a framework for large scale applications.

This ‘convention over configuration’ paradigm might be a good indicator of why Rails has failed in these large scale environments. There are some arguments about Ruby’s (the underlying language) inefficiencies etc. However, the decisive issues in a Rails application start popping up when you have to work around the imposed conventions in order to achieve your goal. It has been said many times that you can develop demo applications in Rails and you don’t have to know its inner workings. However, it seems Rails’ conventions are too limiting for when the applications become larger.

The problem with adopting Rails for a new large scale project is that is serves a specific category of web applications. Of course, the Rails framework supports plug-ins (called gems) that add lots of features. While this helps you in the beginning, when you are doing what has probably been done at least a few times by others, it will eventually get in your way. For your application to evolve, you need to break out of the conventions imposed by the framework. Since Rails is based on those conventions, this basically breaks the purpose of using it.

We have to also quickly note a very important aspect. Rails’ conventions are imposed by people outside the development team. This is the main reason they easily become disruptive limitations rather than helpful conventions.

What we can learn from Rails is that in large scale applications, a ‘convention over configuration’ approach rarely works. We should let conventions be established slowly by the developers. There is a popular guideline in software engineering that encourages developers to automate processes they do more than three times. Let us try to extend this piece of advice to cover conventions.

If a process can be simplified (be it finding that opening curly bracket or building the application), establish conventions in your team to make life easier. Be careful though, do not overdo it – three times going through the same process gives you a hint that a convention can be established, but you have to think about the next three thousand times before deciding on your convention. You have to leave room for configuration over convention to avoid cases when the convention needs to be broken.

Conclusion

While growing up as a software developer, conventions help one work in environments that they are not familiar with. As the applications become more and more complex, conventions become more and more fragile. The reason convention is not praised as essential in large scale applications is because developers will establish them when needed, but if they are imposed there is a large risk they will get in the way of good productivity.