On makefiles and build scripts

tl;dr Using makefiles for small projects is a significant overhead over simpler solutions such as build scripts. However, ultimately it is worth it because of it encourages re-usable solutions and future-proofs the project.

The virtues of automated builds

Build systems (e.g. Ant, Maven, Make, Rake, Cabal, etc) are a great way to automate a lot of redundant tasks in the software development process:

  • Install dependencies
  • Compile and link
  • Run unit tests
  • Perform static analysis
  • Code style checking
  • Deploy

And all of that with a single shell command: the invocation of your favorite build system.
Furthermore, having a “one-click” way of setting up a piece of software will aid new users and contributors: who hasn’t been discouraged from using a piece of software (or contributing to an open source project) because it required a convoluted process to get running/tested/code reviewed?

But I can do that with a shell script!

Many build systems offer additional advantages that are indispensable for large projects: for instance, many build systems will only re-compile parts of a project that have changed which might save hours of compile-time on large projects.

However, what about small projects? Compile times are instantaneous, development teams are small or just a single person, overall project complexity is lower. Are build systems still worth it over alternatives such as a shell script with similar functionality?

Let us first have a look at some of the disadvantages of using a build system for small projects.

Many developers will already be relatively familiar with shell scripting, for example because they use the command line and Unix utilities to analyse logs. Writing a shell script to compile/deploy/etc will likely not take much time. Build systems, on the other hand, would be yet another skill to learn, hone and maintain.

Build system recipes are often “write once and forget” pieces of code: once a build file works as required, there often is little reason to go back and tinker with things. Therefore, it is likely that developers will have to re-learn the syntax and quirks of their build system whenever they touch it. (I sure do.) Queue context switching, mental overhead, getting distracted with looking up features of the build system rather than getting relevant work done, etc.

Some build systems have a substantial learning curve and often work in non-intuitive ways. GNU Make is an especially striking offender in this regard: superficially makefile recipes look like shell script syntax – but with some surprising gotchas (e.g. separate lines in a recipe fork in different sub-shells, indentation should be tab-stops, etc).

Some tasks are more difficult and non-intuitive to do in a build system than in shell scripts (e.g. handling file names with white-space, trying to do loops or conditionals in declarative makefile style, different ways to set the java classpath in ant, etc).

Basically, using a build system over a simple shell script in a small project will incur a significant mental overhead (yet another infrequently used tool, context switching, etc) at not much directly apparent gain.

And yet you should use a makefile

Nevertheless, using a proper build system over build scripts is very likely the way to go, even for small software projects.

A build system gives the user lots of code for free (e.g. easily referenced command line options, shell tab completion, etc). Not having to re-implement functionality time after time is a good thing.

Using a build system often means that all sorts of functionality is consolidated in one location (e.g. build, deploy, run tests, etc). Handling command line arguments is a bit of a pain in shell scripts – it is therefore likely that multiple shell scripts would be written instead of one build system instruction file. Queue more places to check, more code to maintain, possible code duplication, etc.

Similarly, build system scripts encourage the division of tasks. The natural way of structuring a build system script is to have small targets implementing atomic units of work (e.g. compile or run unit-tests) grouped together by larger recipes (e.g. one target grouping pre-deploy instructions, one target grouping post-deploy instructions, etc). This encourages good design and is likely to make the build system script re-usable, easier to test and maintainable. While the same effect is achievable with shell scripts, the design of the language does not encourage the same extent of division of labor.

On a related note, the complexity of build system scripts grows linearly (just add a target and leave the rest untouched) where build scripts can quickly dissolve into an in-extensible mess (“so I need to add this option here and that flag there, but now I need defaults for this function…”).

Additionally, many build systems operate in a declarative way (as opposed to the imperative nature of scripts). This makes the developer thing about the “what” of their requirements, not the “how”. While initially more difficult to comprehend, this likely leads to more re-usable solutions.

In essence, using build systems over hand-rolled build scripts encourages good development practices, increases re-usability and is likely to lead to maintainable solutions.

Future-proofing for free

The conclusion of the last section ties in nicely with an additional huge advantage of using build systems over build scripts: we get future-proofing for free! A software project being small today does not mean that it has to remain small forever. Using build systems instead of build scripts prepares a project for future growth.

It is much easier to “productionize” a build system recipe than a build script (e.g. GNU Make gives us free parallelism when compiling, running tests, etc with the -J option).

Build systems are an abstraction over the underlying operating system and shell. This makes build system recipes more portable meaning less work when the project has to be deployed to a different environment (e.g. there is a version of GNU Make for Windows).

Many build systems are industry standard ways of managing projects in certain languages (e.g. make for C, Ant for Java, Rake for Rubby, etc). This means that not having a build system recipe means that a project might be difficult to integrate with the rest of the software world (e.g. Qt for Android having a build script instead of a makefile was a show-stopper for integration with mainline Qt).

Using a build system is simply the right thing to do! They are indispensable for large scale software projects. While annoying, using them on small projects is an excellent learning opportunity, gives lots of code and power for free and prepares the project for when it becomes the next big thing.

Response Article to “On scripting languages and rockstar programmers” by s1038803

On boring languages and productive programmers

tl;dr Scripting languages are great for initial development but lack community buy-in (tool support, adoption, etc) in order to be suitable for large-scale projects, especially if those projects are performance critical (which is more commonly the case than you'd think).

Introduction

This is a response article to On scripting languages and rockstar programmers by Jakov Smelkin.

Smelkin's main point is summarized in his article's first paragraph:

[It] is not only possible, but oftentimes faster and cheaper to create a medium to large scale project using one of high-level scripting languages as a core of the system.

Throughout the remainder of the article, Smelkin walks us through the line of reasoning leading to his conclusion, comparing and contrasting the non-scripting language C to the scripting language Python. Smelkin's argument can, in essence, be summarized as follows. Scripting languages are high-level, interpreted, dynamically typed and garbage collected. The author notes that these qualities imply a number of advantages, including the following:

  • New code can be developed faster (leaving more time to fix bugs and add features),
  • Scripting languages are easier to learn and use. New programmers can be brought up to speed faster and that average programmers can reach their full potential (however mediocre that may be) and
  • The hit in program run-time and memory consumption is often inconsequential (and there are workarounds when it does).

While Smelkin's conclusion is tautological due to its vagueness ("scripting languages are great except for when they aren't") I will attempt to deconstruct his argument and disagree with the particulars of his premises.

NB: Since I don't know much of C, I'll illustrate my points by comparing Java and Python, not C and Python like in the original article.

Development speed

Smelkin states that scripting languages are great for medium to large scale software projects because their combination of no compile-time and highly productive syntax offer short development times.

Scripting languages are fast to write. No compile times. Fewer characters needed in order to implement functionality (compare: print "hello world" to public class HelloWorld { public static void main(String[] args) {...). Fact.

However, Smelkin seems to forget that writing code is one of the activities on which the least amount of time is spent during the software life cycle of large scale projects – time spent on software maintenance is much higher [1]. The real question to ask, therefore, is whether scripting languages help us doing software maintenance.

From my experience, maintaining code written in a dynamically typed scripting language like Python is an absolute pain when compared to performing a similar maintenance in a boring static language like Java. To illustrate this point, let us focus on what is maybe the most common software maintenance task: refactoring.

There simply aren't the tools to perform refactoring safely in Python. Point in case: compare the rudimentary refactoring support given by the popular Rope library [2] to more sophisticated, context-aware refactoring tools provided by IDEs like Eclipse [3] or IntelliJ IDEA [4]. They don't even play in the same league. Note that this dearth of tool support is not necessarily caused by the fact that Java is a more popular language than Python (and therefore has a bigger market and higher incentives for tooling to be developed). The simple truth is that due to their dynamic nature, scripting languages are unlikely to be able to enjoy the same intelligent tools that statically typed languages do.

Using a scripting language improves feature development time. However, due to poorer tooling, software maintenance tasks (the bulk of the software life-cycle) are slower. This may result in a net loss in productivity.

Programmer skill

Smelkin furthermore states that scripting languages are a great choice for large scale projects because they are easy to learn, allowing new hires to be productive faster and average programmers to be better contributors.

While I agree with the point that scripting languages are often easy to learn superficially, I disagree with the point that this is enough in order to write good medium to large scale software. Often scripting languages require a good deal of due-diligence and good behavior from the programmer (e.g. resisting the temptation to mess with private variables in Python – which can be enforced by the compiler in a language like Java). On the other hand, static languages help novice programmers through superior tooling (e.g. static analysis, highlighting syntax errors and type errors on the fly) and due to the compiler's ability to enforce conventions and invariants.

Scripting languages are easy to pick up and to write "hello world". However, they are more difficult to master. If anything, the ability to write large scale applications in scripting languages is the mark of a competent programmer rather than a novice.

Also, why would I hire a new programmer and train him/her in Python if I can hire experienced Java developers who can hit the ground running at a dime a dozen?

Execution speed

Smelkin also states that the oft-stated performance hit incurred by using a scripting language over a compiled language is irrelevant except for specialized fields such as scientific computing, embedded systems and so forth. Smelkin argues that for most projects, performance is not critical.

I agree with the point that performance is not relevant for all types of software. However, I think that performance un-critical projects are a lot less common than Smelkin seems to imply with his statement.

For instance, the following quote taken from [5] implies that even web development, a traditional bastion of scripting languages (PHP, JavaScript, Python, etc), could be called a performance critical field:

Amazon found every 100ms of latency cost them 1% in sales. Google found an extra .5 seconds in search page generation time dropped traffic by 20%

Granted, as Smelkin states, performance can always be increased by using better data-structures and minimizing IO operations. However, given the extreme importance of performance in some fields (illustrated by the quote above), I argue that optimal data structures, minimal IO operations and a performant language are the norm for in a lot more software fields than Smelkin's argument implies.

Conclusion

Smelkin states that scripting languages are a good choice for medium to large scale performance un-critical projects because they are easy to learn and quick to write. I agree – as long as the projects will only have to be maintained minimally and are truly not performance critical… arguably a much more narrow niche than Smelkin's original post would have us believe.

References

[1] Glass, Robert L. "Frequently forgotten fundamental facts about software engineering." Software, IEEE 18.3 (2001): 112-111.

[2] zjes "Rope, a python refactoring library …" http://rope.sourceforge.net/

[3] The Eclipse Foundation "Eclipse" https://www.eclipse.org/

[4] JetBrains "IntelliJ IDEA 13" http://www.jetbrains.com/idea/

[5] Hoff, Todd "Latency Is Everywhere And It Costs You Sales – How To Crush It" High Scalability blog (2009) http://highscalability.com/latency-everywhere-and-it-costs-you-sales-how-crush-it

“Do you have testers?”

The Joel Test: “Do you have testers?” – No, and neither should you (probably)

tl;dr

The times they are a-changin’. In the post-Agile world, not having dedicated testers might just be advantageous for you. Depending on the software your team produces, factors such as time-to-market might be more important than a perfect product. “Build the right it” first, “build it right” later.

Setting the scene

In 2000, the prolific software writer Joel Spolsky proposed the “Joel Test”: twelve simple yes-or-no questions to judge the quality of a software development life-cycle. The test invites the inquirer to ask questions such as “Do you use source control?” or “Do programmers have quiet working conditions?”. According to Spolsky, any good software development process will answer “no” to at most two of the test’s questions.

Fourteen years later, the test is still widely recognized as a useful, low-cost tool to evaluate the efficacy of a software development process. But is it really? A lot has happened in the world of software engineering since the test’s inception. When reading Spolsky’s original proposal nowadays, I can’t help but feel as though some of the questions are overly dogmatic or at least a bit dated.

For instance, question 10 of the tests states that

[i]f your team doesn’t have dedicated testers, at least one for every two or three programmers, you are either shipping buggy products, or you’re wasting money by having $100/hour programmers do work that can be done by $30/hour testers. Skimping on testers is such an outrageous false economy that I’m simply blown away that more people don’t recognize it.

Source: “The Joel Test”.

In the remainder of this blog post, I will analyze this assertion and discuss to what extent its premise is still relevant in 2014.

Spolsky’s argument then versus in the trenches now

Let us first have a look at the argument made by Spolsky for why dedicated testers are imperative for a good software development process and then contrast this with my personal experiences in the software world.

Question 10 of the “Joel Test” (quoted above) observes two issues:

  • A team without dedicated testers will release a bad product because of lacking quality assurance (QA) from both technical and functional points of view.
  • A team without dedicated testers wastes money since testers are cheaper than developers.

In later posts, Spolsky refines his point further:

  • Programmers make bad testers – the required skill sets are not related.
  • Having testers shortens the time between developers and getting feedback on product quality.
  • Testers will catch problems that the original developer didn’t (or wouldn’t).

Sources: “Why Testers?”, “Top Five (Wrong) Reasons You Don’t Have Testers”.

These assertions do not match my personal experience. Time for some anecdata.

Last summer I worked at the Amazon Development Centre Scotland (ADCS). Most people would likely agree that ADCS is a high quality software development environment. As a matter of fact, ADCS would obtain a perfect score on the “Joel Test” if it were not for the lack of dedicated testers on my team… however, this does not mean that ADCS ships bad code, as Spolsky claims must inevitably happen. Far from it. Software quality is simply assured by other (in my opinion superior) means:

  • Strict pair programming when code is being touched.
  • Unit and integration tests written for any new feature.
  • Before a major feature is released, the entire team pitches in and spends half a day to a day focusing on testing the new feature. If any show-stopper bugs are found, the release of the feature is delayed.
  • One person on the team is “on-call” every week, handling bug reports as they come in.

Arguably, this Agile-inspired approach has a number of advantages over having dedicated testers:

  • Everyone in the team has a rough idea of what is going on. There is less of a chance of knowledge silos developing.
  • There is a strong incentive to produce high quality code as your colleagues are going to be directly affected by the mess you cause (as opposed to testers in some other team you might not know).
  • More eyes see the code than with Spolsky’s recommended “1 tester per 2 engineers”, increasing the chance of finding bugs or poor user experience.
  • Developers will never have to wait for the testing team. Automated tests give instant feedback. More involved evaluation such as user acceptance testing can be prioritized appropriately by the developers in order to integrate it seamlessly with the rest of the development process.

In addition to these benefits, I also found that foregoing dedicated testers means that the development team is more likely to develop a strong sense of ownership of their product. The team owns the software development life cycle end to end and is therefore more likely to “do the right thing and do it right”: develop reliable solutions that do what they are supposed to do. Contrast this with just implementing some specification, passing it on to testers (or later business analysts) and never looking back… Farewell, “code, compile and forget”. You shall not be missed.

A brave new world?

So how do we consolidate this difference between what the “Joel Test” claims should happen and what actually does happen on the grounds, even at a top-tier, quality obsessed company like Amazon?

It is important to note that the “Joel Test” was written in 2000 and very much reflects the spirit of the time in its attitude towards testing. A lot has happened in the world of software since then; most notably, development methodologies have shifted away from the Waterfall-like methods prevalent at the start of the century to a wide-spread adoption of Agile methods.

Software produced in the times of the “Joel Test” would have one, two, maybe a handful of releases a year, indiscriminately distributed via physical media to a company’s entire client base. Nowadays, the near-ubiquitous use of web-based services means that new software versions can often be rolled out instantly or to only a small part of your company’s customers. Being able to perform split tests on your product by releasing new features gradually into the wild means that the impact of bugs or poor design is less severe than with the old model where a company’s reputation could rise or fall on the newest polycarbonate-plastic pressed version of its developers efforts. Thus reduced the importance of the tester.

Furthermore, the modern-day developer has access to a plethora of new tools that reduce the burden of test. While some of these tools might have been available at the time of the “Joel Test”, they would have been nowhere near as ubiquitous as nowadays when FindBugs is in the top ten most downloaded Eclipse plugins and all major Java IDEs come with JUnit installed by default. Static code analysis finds a plethora of bugs even before the developer hits “compile”. Unit and integration tests take snapshots of the functionality of a system and therefore prevents test regressions by default. This eliminates a big chunk of the grunt work that formerly had to be done manually by QA departments. Thus reduced the importance of the tester.

Additionally, the switch to Agile-like methods brought about a mentality change in the software field. Previously, QA teams were often “separate but unequal” parts of the development apparatus. Under Agile and its offspring like Test Driven Development, everyone is a tester: everyone wears all hats. The role of QA can thus grow to encompass more directly technical or business-related aspects such as focusing on user acceptance testing and performance- or security-testing. From a button-mashing unequal to a respected fellow reliability engineer. Thus reduced the importance of the tester.

However, the shift to Agile can’t explain the entire difference between what the “Joel Test” claims and what I observe in the modern software world. Changes in tools and mentality have shifted some of the burden of test from traditional testers to the entire development team… but traditional Agile is still far from the laissez-faire “ship it” attitude that I see pervading the software world. As a matter of fact, Agile evangelist and writer Don Wells claims that

[q]uality assurance (QA) is an essential part of the [Extreme Programming (XP)] process. On some projects QA is done by a separate group, while on others QA will be an integrated into the development team itself. In either case XP requires development to have much closer relationship with QA.

Source: Extreme Programming Rules

I think that this remaining disjoint can be explained by observing that some companies might have moved on from Agile and are adopting elements from even more “lean” philosophies. “Idea bugs” are harder to fix than software bugs. If you build a product that people really want to use, they will be forgiving (prime example: the Twitter service-outage whale). Getting something out of the door and running it by real-world users is thus increasingly becoming the most important aspect of software development: delivering products, not code – focusing on “building the right it” rather than “building it right”. This is something traditional QA can’t really help with. And therewith died testing.

The future of test and concluding remarks

Yet test will surely live on with “I’m not dead yet” tenacity. The software forest is not only comprised of Agile web development trees. Traditional testing is sure to find respite in havens such as AAA game development with its ever-present big releases or security- and safety-critical programming with its zero-tolerance for bugs (e.g. medical or aeronautical software). For the remaining denizens of QA-land, however, I fear that the time has come to leave testing drugery behind and move on to the greener pastures of product management or reliability engineering.

As a closing note, I’d like to leave you with the following video that makes a similar point to the one labored in this blog post, but in a much more entertaining way: “Test is Dead” by Alberto Savoia.