Named anti-patterns in the current pipeline

Each entry gives the name used in the literature, where the name comes from, and the specific way it appears in our GoCD setup. Using these names makes it easier to find prior work and to discuss the problems with people outside the programme.

Ice-cream cone (inverted test pyramid)

Also called: inverted pyramid, test cupcake when manual testing is stacked on top

Mike Cohn's test pyramid, described in Succeeding with Agile (2009), puts many fast unit tests at the base, fewer service or component tests in the middle and a small number of end-to-end UI tests at the top. Alister Scott named the inversion of this shape the ice-cream cone in 2012, where the bulk of the confidence comes from slow, broad UI tests and the lower layers are thin.

In our setup the first wave does run unit tests, but the release decision rests on the second wave of roughly 100 UI test pipelines, which carry the system tests, each suite driving the deployed system through one module's UI. Those tests are described as comprehensive, which means they are exercising behaviour that could have been checked at a lower layer, where it would run in milliseconds and would not depend on a browser, a network or a deployed backend.

Source Mike Cohn, Succeeding with Agile (2009); Alister Scott, "Testing Pyramids & Ice-Cream Cones" (watirmelon.blog, 2012); Martin Fowler, "TestPyramid" (bliki, 2012).

Non-deterministic tests, commonly called flaky tests

A flaky test passes and fails on the same code. Martin Fowler's 2011 article "Eradicating Non-Determinism in Tests" lists the usual causes as lack of isolation between tests, asynchronous behaviour handled with fixed waits, dependence on remote services, time and ordering assumptions, and resource leaks. All of these are typical of browser-driven UI tests.

Google reported in 2016 that about 1.5% of all its test runs were flaky and that close to 16% of its tests had shown some flakiness at some point. Fowler's article makes the cost explicit: once a suite is known to be flaky, a red result stops carrying information, and people learn to rerun until green. In a gate that requires 100 pipelines to agree, the overview's arithmetic shows how a few percent of flakiness per pipeline is enough to reduce green candidates to about one a week.

Here the module teams handle a flaky test in one of three ways: they comment it out, fix it, or decide that the issue is not worth fixing. Commenting a test out does lower the pipeline's flake rate, and it also stops the test from running, so the pipeline keeps no results for it, nothing records which behaviour has lost its check, and nothing brings the test back once its cause is fixed. Fowler's quarantine keeps the test running outside the gate for exactly this reason, and practice 3 sets out how that differs from commenting out.

Source Martin Fowler, "Eradicating Non-Determinism in Tests" (2011); John Micco, "Flaky Tests at Google and How We Mitigate Them" (Google Testing Blog, 2016).

Normalisation of deviance, and broken windows on the build

Diane Vaughan coined "normalisation of deviance" in her study of the Challenger launch decision to describe how an organisation comes to accept a signal it once treated as a failure, because nothing bad happened the previous times. In build systems the same pattern appears when a red pipeline is treated as noise, reruns become a standard step, and a genuine regression hides among the flaky failures.

The related "broken windows" framing, applied to software by Hunt and Thomas in The Pragmatic Programmer, describes how one tolerated defect lowers the bar for the next. Once the integration build is usually red, an additional red pipeline does not prompt anyone to act, which is why a real defect can sit in the candidate for days.

Commenting out flaky tests is the same acceptance applied to coverage. Each commented-out test is a small, reasonable choice for the team that makes it, and the suites shrink by tests nobody tracks until a regression reaches the candidate through behaviour that one of them used to check.

Source Diane Vaughan, The Challenger Launch Decision (1996); Andrew Hunt and David Thomas, The Pragmatic Programmer (1999).

Integrated tests used as the release gate

J. B. Rainsberger's talk and essay "Integrated Tests Are A Scam" argues that tests exercising many real components together grow combinatorially, run slowly and fail for reasons outside the code under test, so they cannot be the main source of confidence. He separates these from integration tests in the narrower sense, which check one boundary at a time, and recommends collaboration and contract tests instead.

The second-wave suites are system tests: each drives a frontend module's UI against real microservices and shared data, which makes them integrated tests in Rainsberger's sense. Each one can fail because of any service, any deployment in progress, any other test's data or the state left by the upstream suite it waits on, and the module team that owns the test has no control over most of those causes.

Source J. B. Rainsberger, "Integrated Tests Are A Scam" (2009 talk, later essays on blog.thecodewhisperer.com).

Distributed monolith, in the build

Also called: lockstep build, coupled release candidate

Sam Newman uses "distributed monolith" for a system split into services that still have to be built, tested and released together. The system pays the operational cost of distribution and keeps the coupling of a monolith. Newman's test for independent deployability is whether one service can be changed and deployed without changing or redeploying anything else.

Our release model is fixed: every release ships the full system, and no microservice or frontend module goes to production on its own. This study takes the full-system release as given and applies the anti-pattern to the build, which is coupled in the same way as the release. Parts of the build are already modular: each first-wave pipeline builds one unit, and each module's system tests have their own part of the test code and their own UI pipeline, so a test fix reruns one suite and the other suites keep their results. What stays coupled is the product revision set, because each fan-out runs every UI pipeline against one new set of product revisions, so a product change in a single service sets all 100 suites running again, a red result in any of them keeps the green results of the others from forming a candidate, and a module's green result against an earlier product revision set cannot stand in for it. A system can ship as one unit while each of its parts is still built, verified and versioned separately, and the current pipeline gives up that separation along with independent release.

Source Sam Newman, Building Microservices, 2nd edition (2021) and Monolith to Microservices (2019).

Big-bang integration at the fan-in

Also called: integration hell, all-or-nothing release candidate

"Big-bang integration" is the older software engineering term for integrating all components at once at the end, in contrast to incremental integration. Continuous integration was proposed largely to remove the "integration hell" that followed from it. Humble and Farley's Continuous Delivery treats the deployment pipeline as a way to integrate continuously in small steps, each of which can be released.

The GoCD fan-in recreates a big-bang step inside a CI system. GoCD's fan-in resolves a consistent set of upstream revisions and waits for every upstream pipeline, so one failing or slow pipeline among 100 holds back the other 99 (see the pipeline diagram in the overview). The fan-out then multiplies the problem by starting 100 more pipelines, all of which must also pass for the same revision set. A test fix reruns only its own suite and the chain below it, so recovery from a broken test stays inside one module, while a product fix has to pass through the fan-in and wait for the next manual fan-out like any other product change. The fan-out can only be triggered manually, so the integration step happens when a person decides to run it, which is the scheduled integration that continuous integration set out to replace.

The second wave repeats the same step at a smaller scale. A UI pipeline with another UI pipeline among its upstream materials is subject to the same fan-in resolution, so it waits for that pipeline to pass on the same revisions before it starts. A flake in the upstream suite leaves the downstream suite with no run on that revision, which delays its result and also hides it: the run shows one red pipeline while several have produced nothing.

Source Jez Humble and David Farley, Continuous Delivery (2010); Martin Fowler, "Continuous Integration" (2000, revised 2024); GoCD documentation on fan-in dependency resolution.

Large batch size

Don Reinertsen's The Principles of Product Development Flow shows that large batches increase cycle time, delay feedback and make the cost of a defect grow with the size of the batch it travels in. The DORA research summarised in Accelerate finds that high-performing teams ship small changes frequently, and that this correlates with lower change failure rates.

Here, each manual fan-out collects every commit from every team since the previous fan-out into one candidate, and the batch is never smaller than two hours of work and grows whenever the fan-out is started late. When the candidate fails, finding which commit caused it means reading through every change in the batch, and the fix lands in a later batch that carries its own new changes. Because green candidates are rare, the batch that finally ships may contain a week of work, which raises the risk of that release and makes rollback harder.

Source Donald Reinertsen, The Principles of Product Development Flow (2009); Nicole Forsgren, Jez Humble and Gene Kim, Accelerate (2018).

Shared, long-lived test environments

Related: snowflake environments, environment contention

Fowler's "SnowflakeServer" describes environments whose state has drifted through manual changes until no one can reproduce it. Shared integration environments add contention on top of drift. When many pipelines deploy to, seed data into and test against the same environment at the same time, a test can fail because another pipeline changed a record, redeployed a service mid-run or exhausted a connection pool.

Running 100 UI pipelines in parallel against common backends is a likely contributor to the flakiness. Whether the cause sits in the tests or the environment should be established early, because the fixes differ: test-level waits and isolation for the first, hermetic or per-pipeline environments for the second.

The dependencies between UI pipelines in the second wave should be checked for the same reason. When a downstream suite has to run after a particular upstream suite, it may rely on data or environment state that the upstream suite leaves behind, which is the lack of isolation between tests that Fowler lists first among the causes of flakiness, raised from single tests to whole suites. If a dependency only fixes the order of runs, it can be removed, and if it carries state, the downstream suite needs to create that state itself before the dependency can go.

Source Martin Fowler, "SnowflakeServer" (bliki, 2012); Humble and Farley, Continuous Delivery, chapter on managing environments.

Wide blast radius from shared code

There is no single canonical name for this one. It is usually discussed through Robert C. Martin's package principles, particularly the Stable Dependencies Principle (depend in the direction of stability) and the Common Reuse Principle (classes that are not reused together should not be packaged together). In mono repo tooling the same idea appears as the "affected graph": a change to a module that everything depends on marks everything as affected.

The shared frontend platform folder sits under every UI module. If the build treats any change in it as a reason to rebuild and retest all modules, a one-line platform fix costs the full two hours and 200 pipelines. If the platform is also edited often, the most volatile code is the code with the most dependents, which inverts Martin's Stable Dependencies Principle.

Source Robert C. Martin, Agile Software Development: Principles, Patterns, and Practices (2002) and Clean Architecture (2017); Nx and Bazel documentation on affected and reverse-dependency queries.

CI theatre

Thoughtworks used "CI theatre" (placed in Hold on the Technology Radar) for teams that run a CI server without practising continuous integration: builds run, but the main line is red for long stretches, branches live for days, or the build is too slow to give feedback on each change. The CI server produces the appearance of integration while integration actually happens rarely and in large steps.

Our pipelines run all day, yet a fully integrated, releasable state is reached about once a week. By the definition behind the term, the effective integration frequency is weekly. The manual fan-out adds to this: the first wave runs on every commit, while the system tests, which decide the release, run on product changes only when someone starts them, so the pipelines that run continuously are the ones the release decision does not rest on.

Source Thoughtworks Technology Radar, "CI theatre"; Martin Fowler, "Continuous Integration" (revised 2024), section on keeping the build fast and fixing broken builds immediately.

Tragedy of the commons on the build

The tragedy of the commons describes a shared resource that each participant benefits from and none is responsible for maintaining. Applied to builds, every team needs the integrated pipeline to be green, and each team reasonably fixes only its own pipeline. The aggregate state, which is the only one that matters for release, has no owner. Continuous Delivery and Fowler's CI article both recommend the opposite norm, where a broken main build stops further check-ins until someone fixes or reverts the change, often coordinated by a rotating "build cop".

With 100 module teams contributing to one gate, a flake in another team's pipeline is someone else's problem to each team, while the cost lands on all of them. Without someone accountable for the end-to-end green rate, the flake rate drifts upward until the arithmetic in the overview takes over.

The way flaky tests are handled today shows the same split between who decides and who pays. Each module team decides, when it chooses to work on its suite, whether to comment a flaky test out, fix it, or leave the issue because it is not worth fixing. The team weighs that call against its own pipeline, where a test that flakes once in a thousand runs looks harmless, while the overview's arithmetic shows that tests of that kind across 100 pipelines are enough to hold the gate at about one green run a week.

Source Garrett Hardin, "The Tragedy of the Commons" (1968); Humble and Farley, Continuous Delivery, chapter on continuous integration.

How the anti-patterns reinforce each other

The ice-cream cone puts most of the confidence into integrated UI tests, and those tests run against shared environments, which makes them flaky. The distributed monolith and the big-bang fan-in then require every one of those flaky pipelines to pass at once, which turns modest per-pipeline flakiness into a rare green state. Rare green states and a fan-out that only runs on request both force large batches, and large batches make every red result harder to diagnose. Constant red teaches people to rerun and ignore failures, and because nobody owns the aggregate state, the flake rate keeps rising. Chains of UI pipelines inside the second wave stretch each run and let one flake hold back several results, and the tests that teams comment out to get their pipelines green remove coverage without leaving a record of it. The wide blast radius of the shared platform feeds more changes into the full two-hour cycle and keeps it busy.

The industry practices document addresses these at the points where the loop can be broken most cheaply.