Reading list for a full-system release with flaky UI gates
Published case studies and papers selected for our constraints: about 100 modules and services, one GoCD pipeline graph, flaky UI suites, and a release that always ships the full system. Each entry says what to take from it and which practice in this study it supports.
Suggested reading order
- Slack's CI/CD case study, for moving broad UI suites out of the blocking path.
- Google's flaky-test report and Fowler's article on non-determinism, for the quarantine policy.
- Microsoft's study of the flaky-test lifecycle, for setting quarantine deadlines with data.
- Pact documentation, for contract checks across a release manifest.
- Uber's SubmitQueue paper and Google's flake-aware culprit finding, for attributing failures to one change.
- Hammant's article on test impact analysis and Meta's predictive test selection, for running fewer UI tests per change.
- Uber's continuous deployment article, for selecting the newest passing version at each stage.
- Google's canarying chapter, for the decision after deployment.
Moving UI suites out of the blocking path
Slack: Balancing Safety and Velocity in CI/CD
Engineering case study · slack.engineering
Slack describes moving the main webapp monorepo from testing almost everything before merge to a tiered workflow with substantial testing after merge, after small changes had started triggering thousands of tests, including flaky end-to-end tests unrelated to the change. This is the closest published account of our second wave of UI pipelines and what can be done with it.
Take which tests stayed blocking and which moved after merge, how Slack handled failures found on main, and how it measured the change in throughput.
Supports practice 2, blocking and non-blocking tiers.
Making flaky tests a managed defect
Google: Flaky Tests at Google and How We Mitigate Them
Experience report, 2016 · Google Testing Blog
Google reports how much of its test signal is flaky and describes detection by rerunning on unchanged code, quarantine and ownership. Its figures on how flakiness accumulates across many tests support the overview's arithmetic for a 100-wide gate.
Take the detection rule (pass and fail on one revision), and the case for keeping quarantined tests running, which the tests our teams comment out today do not.
Supports practice 3, quarantine with an owner and a deadline.
Martin Fowler: Eradicating Non-Determinism in Tests
Article, 2011 · martinfowler.com
Fowler catalogues the causes of non-determinism (isolation, asynchrony, remote services, time, resource leaks) and the fix for each, and argues for quarantine with a short, actively worked list. The catalogue is a practical checklist for triaging our UI suites.
Take the cause-by-cause fixes, applied to the ten flakiest UI pipelines first.
Microsoft Research: A Study on the Lifecycle of Flaky Tests
Research paper, ICSE 2020 · microsoft.com
The study follows flaky tests across six large proprietary projects and measures recurrence, time to fix and how often claimed fixes work. It gives evidence for policy choices that would otherwise be guesses, such as how long a test may stay in quarantine and whether a returning test should be watched before it blocks again.
Take its recurrence and time-to-fix findings as the basis for our quarantine deadline.
Supports practice 3.
Checking a full-system manifest for compatibility
Pact: consumer-driven contract testing
Documentation · docs.pact.io
Pact records the requests each consumer makes and the responses it depends on, and verifies providers against those contracts. Its can-i-deploy check answers whether a set of versions is compatible, which is the question a release manifest needs answered before system tests start. Ian Robinson's 2006 article "Consumer-Driven Contracts" on martinfowler.com gives the underlying pattern.
Take the broker workflow and can-i-deploy against a named environment or manifest.
Attributing a failure to one change
Uber: Keeping Master Green at Scale
Research paper, EuroSys 2019 · search by title
Uber's SubmitQueue tests changes against the tip of main before they land and uses speculation about likely outcomes to keep throughput high with thousands of changes a day. It shows that a merge queue holds up at a scale well beyond ours.
Take the batching and bisection model, which can sit in front of GoCD as the required check.
Supports practice 7, merge queue.
Google Research: Flake Aware Culprit Finding
Research paper, ICST 2023 · research.google
The paper covers finding the commit responsible for a breakage when some of the results being bisected are flaky, evaluated on a large set of real breakages. Today our two-hour candidates bundle many teams' commits, and the UI tier in practice 2 will need the same capability.
Take the approach of rerunning selectively during bisection instead of trusting a single red result.
Running fewer UI tests per change
Paul Hammant: The Rise of Test Impact Analysis
Article, 2017 · martinfowler.com
Hammant traces test impact analysis from Google's Blaze and Testar to Microsoft's implementation in Visual Studio and Azure DevOps, and shows how a coverage map from source files to tests selects the tests for each change. The article also lists where the method falls short, which matters for UI suites built on a shared platform folder.
Take the coverage-map mechanism and the caveats on stale maps, non-code inputs and heavily shared code.
Supports practice 6, affected UI suites.
Meta: Predictive Test Selection
Research paper, ICSE-SEIP 2019 · arXiv
Meta trains a model on historical results to choose which tests to run for a change, and reports catching nearly all faulty changes while running a fraction of the tests. The approach needs only test history and change metadata, both of which GoCD already produces, and commercial tools such as Launchable and Gradle Develocity package it.
Take how the model is trained and how its miss rate is monitored over time.
Supports practice 6, affected UI suites.
Uber: Continuous deployment for large monorepos
Engineering blog, 2024 · uber.com
Rasmus Vestergaard and Kasper Munck describe how Uber maps each commit to the services it affects through the Bazel graph and moves each stage to the newest commit that passed the stage before it. Uber reports going from about 7,000 to 50,000 production deployments a week between early 2022 and March 2024. Uber deploys services independently, which this system does not, so the parts to reuse are the affected-service analysis and the per-stage selection rule.
Take the commit-to-service analysis and the rule for promoting the newest qualifying version.
Deciding after deployment
Google SRE Workbook: Canarying Releases
Practical guide · sre.google
The chapter covers gradual rollout, comparing a canary against a control population and choosing the health metrics that decide whether to continue. A full-system release can be canaried by region or by a share of users, which lets the pre-release gate rely on the deterministic tiers.
Take the canary-versus-control comparison and the metric selection guidance.
Supports practice 8, risk control after deployment.
DORA: Continuous Delivery capabilities
Research synthesis · dora.dev
DORA's capability pages cover test automation, trunk-based development, deployment automation and monitoring, with the measures it uses for each. They are useful for choosing the metrics in practice 9 and for explaining the programme to people outside engineering.
Supports practice 9, the build as an owned product.
Material reviewed and left out
- Google Research, "Taming the Variants: Multi-Architecture Continuous Testing". It addresses duplicate test runs across CPU architectures, which our system does not have.
- Google Research, "Assessing Transition-Based Test Selection Algorithms at Google". The method assumes Google's internal test infrastructure, and Meta's predictive selection is closer to what we could adopt.
- Google SRE, release engineering material on "push on green". The canarying chapter covers the parts that apply to a full-system release.