A Slow Test Suite is Killing your Productivity
I was recently reviewing a merge request on which the author had left a note that the changes would cause the test suite runtime to double from 60s to 120s. I felt a strong urge to club this person over the head reject the changes.
Why is that?
In this short essay I will delve into the costs associated with a slow test suite and give three practical recommendations to speed up your test suite.
The canonical reason given for speeding up test suites is that slow tests cause developers to lose focus[1][2]. Since this argument is covered elsewhere I'll be focusing on alternative arguments as to why slow tests are bad for productivity.
Slow tests encourage workflow anti-patterns
Infrequent-Testing
Slow test suites discourage running the tests frequently. As a result, developers tend to make more changes before running the tests. This inevitably leads to more time spent debugging, since whenever the tests fail there are, on average, more changes that could be to blame.
Selective-Testing
As Mark Seeman notes in his excellent blogpost[1], many people work around slow tests by avoiding full test runs almost entirely, instead relying on running single tests or classes of tests. Mark notes that this pattern "adds a source of friction into your workflow" as you have to select the appropriate tests to run each time you want to make a change. This may sound like a minor inconvenience but in my experience (and Mark's) it tangibly detracts from your focus and draws you out of your flow state.
A second effect of this workflow is that it can take much longer to discover regressions in the tests you didn't run. This is problematic for the same reason that running the tests less frequently is: you find regressions later, and it's harder to isolate the change that was responsible!
Slow tests = ineffective feedback loops
Writing software is an iterative process. You make some changes, verify that everything works, and then refactor. The slower your tests run, the longer your feedback loop. The longer your feedback loop the slower you explore the solution space, meaning it takes longer to find an initial solution and longer to evolve and improve this solution. Slower exploration therefore guarantees either lower quality* or lower velocity.
*Moreover, lower quality practically guarantees lower velocity in the long run[3].
Conclusion
Speeding up your slow test suite will:
improve developer focus
encourage efficient workflows
increase code quality
Since speeding up the tests amplifies the effectiveness of not only yourself but your entire team, this work is very high leverage and will generally yield a great return on investment.
I'm convinced. What can I do?
Find the slowest tests and make them faster
Politely refuse to merge code that unreasonably slows down the tests
Run a competition in your team to see who can shave the most seconds off of the tests*.
*Without weakening the constraints they impose, of course!
Thank you for reading!
What are your strategies for speeding up slow test suites?
References
Mark Seeman "TDD tests should run in 10 seconds or less" https://blog.ploeh.dk/2012/05/24/TDDtestsuitesshouldrunin10secondsorless/
Noell Llopis "What's your pain threshold" https://gamesfromwithin.com/whats-your-pain-threshold
John Ousterhout "A philosophy of software design" Relevant Section: "Tactical vs Strategic Programming"