Problems that stay once the UI suites stop flaking

Suppose every module team fixes its flaky tests and the per-pipeline flake rate falls close to zero. This page works through what the pipeline still does to the release, using the same topology as the overview: about 100 build pipelines, a fan-in, a fan-out that runs only when a person triggers it, and about 100 chained UI pipelines that all have to pass for one candidate.

Genuine defects in a large batch

With the flake rate at zero, the formula on the overview stops being the limit, and a second one takes its place. If each commit has a chance q of breaking some UI suite, a candidate built from B commits is clean with probability (1 − q)B. The time between manual fan-outs sets B, because every commit made by any team since the last fan-out goes into the next one, and that time is never shorter than the two-hour run. The figures below are illustrative, since the study has no measured value for q or for the commits per run.

A red result from a genuine defect is the gate doing its job, so the question becomes how long the defect keeps the candidate red. The answer depends on run length and on how fast someone can find and fix the commit, and both stay as they are when flakiness goes away.

One real defect costs a run, a diagnosis and a wait for the fan-out

The timeline is the best case, where the failing suite points straight at the cause, the fix is right first time and the wait for the fan-out is short. Whoever starts the fan-out while the fix is still being written can start it at once, which gives the other teams' commits a run that goes red again on the known defect, or wait for the fix, which leaves those commits untested for the length of the diagnosis. Two defects landing in different modules on the same day can use up most of the five runs a day holds. Each red run also holds back every other team's commits from that run, which grows the batch for the run after the fix and raises the chance that it carries a new defect of its own.

The timeline covers a defect in product code, whose fix has to go through wave 1 and then wait for the next fan-out. When the defect is in a test, the fix is a commit to that module's part of the system test code. Because the suites are modular, the commit reruns that module's UI pipeline and the chain below it straight away against the same product revisions, and the other UI pipelines keep their results, so a test defect costs one chain run after the diagnosis and no wait for a person.

Finding the commit behind a red UI suite

A UI suite reports which screen or journey broke. It does not report which of the 100 units changed the behaviour, and a candidate that bundles commits from many repositories offers several suspects for each failure. When a microservice change breaks the order journey, the orders team sees the red suite first and has to trace it to a service it does not own, then find the owning team and wait for that team's fix. The time spent on this trace sits inside the diagnose bar in the timeline above, and on a cross-team failure it can be longer than a whole run.

Some failures only appear when two changes meet, such as a frontend module and a service that were each fine against the previous version of the other. Neither team's own pipeline in the first wave catches these, and reverting either commit alone may or may not turn the suite green, which makes bisecting by hand slow.

Chains hide what is wrong further down

A genuine failure in a UI pipeline near the top of a chain stops the pipelines below it from running on that revision. Any defect in a downstream module stays hidden until the upstream fix passes, and it then appears one run later as a fresh red result. Two independent defects in one chain therefore cost two sequential fix cycles, where a flat second wave would have reported both in the same run and let both teams fix them in parallel.

System test commits can turn the gate red by themselves

A commit to one module's system tests triggers that module's UI pipeline and the chain below it directly, without waiting for the manual fan-out, so the test code takes part in the gate on the same terms as product code and can break a candidate with no product change at all. These runs use the product revisions from the last fan-out. When a team updates a UI test for a new feature, the test commit runs straight away against a product that does not have the feature yet, and the suite stays red until someone starts a fan-out that carries the feature. Nothing in the pipeline ties a test revision to the product revision it was written against, so the team has to hold back its test commit until the feature has been fanned out, or accept a red suite in the meantime.

Environment failures without a flaky test

A test can be deterministic and still fail when the environment under it is unhealthy. A deployment that fails partway, a service that is restarting, a database that another UI pipeline has just modified, or a downstream system that is down all produce a red suite. While flaky tests are common, these failures get filed with them, and once the tests are fixed they become the main source of red results that no code change explains. They need their own classification in the failure reports, owned by whoever owns the environment and not by the module team whose suite happened to be running.

Tests that were commented out stay out

Fixing the flaky tests that are still running does nothing for the ones that were commented out along the way. Their coverage has been missing for as long as they have been out, and no pipeline result shows it. Getting the flake rate down can therefore leave the gate passing more often while checking less than it did before the teams started. Before the flakiness work is called done, each module team needs to find its commented-out tests in the system test code and decide, one test at a time, whether to restore it, rewrite it lower in the pyramid, or delete it with a note on what it covered.

Run length and pipeline capacity

The two-hour run is set by the longest chain of UI suites plus the first wave and the deploy, and flakiness has no part in it. Five runs a day stays the ceiling on candidates and on fix attempts, and the manual fan-out keeps the real number below it, because each run waits for a person to start it. The study has no record of why the fan-out is manual. If the reason is agent or environment capacity, affected builds and test tiers reduce the load that an automatic fan-out would add. A change to the shared platform folder still rebuilds and retests every module, and each system test commit starts another run of its own suite and the chain below it, so the queue for agents and test environments is set by commit volume across the whole programme. Run length goes up as teams add tests, because no step in the current setup removes a UI test when a lower-level test starts to cover the same behaviour.

Flakiness returning

New UI tests are written every week, and some of them will be flaky. Without a measured flake rate per test and a rule for what happens when a test crosses it, the rate drifts back up, and the module teams are again judging each new flake on its own. The fix work described in this study holds only if the flake rate is tracked as a pipeline measure and published per suite.

Which practice addresses each remaining problem