What the industry does about slow, flaky, all-or-nothing builds
Practices used by organisations with large mono repos and many services, grouped by which part of the problem they address: how many pipelines must agree, how often they fail without cause, how much work each run does, and how long feedback takes.
The ten practices
Each practice has its own page with the mechanism, how it applies to a full-system release from about 100 modules, the GoCD features and gaps involved, its pros and cons for this system, and the sources it comes from. The numbers stay as they are on the practice pages, so the two groups are not numbered in sequence.
Pipeline redesign
Changes to what the pipelines run, in which order, and what a red result holds back.
Build a release manifest from each module's last verified version, so one red module holds back only its own change.
Let fast deterministic checks decide which commits produce candidate artifacts, and run the broad UI suites after the fan-in as evidence for the release decision.
Use the mono repo dependency graph to run only the UI suites a commit can affect, and reuse earlier UI passes for modules whose inputs did not change.
Test changes against the tip of main before they land and bisect a failing batch to the change that broke it.
Canary the full-system release and contain escaped defects with feature flags and automated rollback.
Cut a branch in every repository at the revisions that passed build and unit tests, run the UI suites there, and fix defects on it without new commits from main.
Process and engineering practices
Changes to how tests are written, how flaky tests are handled and who owns the build, which work with the current pipelines or a redesigned one.
Detect flaky tests automatically, take them out of the gate without commenting them out, and give each one an owner and a deadline.
Fix the waits, network dependencies, shared data, clocks and shared environments behind most UI test failures.
Move checks into component and contract tests and verify manifest compatibility with can-i-deploy.
Give the build an owning team and published measures, with a rotating build cop who triages failures until flakes are controlled and then enforces revert-or-fix.
GoCD
Each practice page has an "In GoCD" section listing the GoCD features the practice can use and the gaps that need a plugin, a script or another tool.
Mapping anti-patterns to practices
| Anti-pattern | Primary practices |
|---|---|
| Ice-cream cone | 2. Blocking and non-blocking tiers; 5. Move confidence down the pyramid; 8. Post-deployment controls |
| Non-deterministic tests | 3. Quarantine in place of commenting out; 4. Remove flakiness causes |
| Normalisation of deviance | 3. Retries counted as flakes; 7. Merge queue; 9. Build cop |
| Integrated tests as the gate | 2. UI suites moved to the non-blocking tier; 4. Stub at the network boundary; 5. Contract tests |
| Distributed monolith, in the build | 1. Release manifest of verified versions; 5. can-i-deploy on the manifest |
| Big-bang integration at the fan-in | 1. Last known good per module; 7. Merge queue; 10. Release branch |
| Large batch size | 1. Release train with last known good; 6. Affected UI suites; 7. Merge queue; 10. Fixes on a release branch |
| Shared long-lived environments | 4. Ephemeral per-pipeline environments, isolated test data, no state passed between UI suites |
| Wide blast radius from shared code | 6. Split and version the platform, affected UI suites |
| CI theatre | 6. Faster feedback; 9. Measure time to green |
| Tragedy of the commons | 3. Flake SLOs per team; 9. Build owned as a product |
References
- Jez Humble and David Farley, Continuous Delivery, Addison-Wesley, 2010.
- Nicole Forsgren, Jez Humble and Gene Kim, Accelerate, IT Revolution, 2018.
- Martin Fowler, "Eradicating Non-Determinism in Tests", martinfowler.com, 2011.
- John Micco (Google), "Flaky Tests at Google and How We Mitigate Them", Google Testing Blog, 2016.
- Sam Newman, Building Microservices, 2nd edition, O'Reilly, 2021.
- Ian Robinson, "Consumer-Driven Contracts: A Service Evolution Pattern", martinfowler.com, 2006.
- Mateusz Machalica et al. (Meta), "Predictive Test Selection", ICSE-SEIP, 2019.
- Paul Hammant, "The Rise of Test Impact Analysis", martinfowler.com, 2017.
- Rasmus Vestergaard and Kasper Munck (Uber), "Continuous deployment for large monorepos", Uber Engineering Blog, 2024.
- Sundaram Ananthanarayanan et al. (Uber), "Keeping Master Green at Scale", EuroSys, 2019.
- Graydon Hoare, "The Not Rocket Science Rule of Software Engineering", 2014.
- Titus Winters, Tom Manshreck and Hyrum Wright (Google), Software Engineering at Google, O'Reilly, 2020, chapters on testing and continuous integration.
- Carlos Valdez and Frank Chen (Slack), "Balancing Safety and Velocity in CI/CD at Slack", 2022.
- Wing Lam et al. (Microsoft Research), "A Study on the Lifecycle of Flaky Tests", ICSE, 2020.
- Google Research, "Flake Aware Culprit Finding", ICST, 2023.
- Betsy Beyer et al. (eds.) (Google), The Site Reliability Workbook, O'Reilly, 2018, chapter "Canarying Releases".