Industry practices · Practice 10 of 10

Stabilise the release on a branch cut across repositories

Chromium cuts a branch from main for each milestone and stabilises the release there while main keeps moving, and Trunk Based Development describes the same practice as branching for release. Applied here, the cut happens in every repository at once, at the revisions whose build and unit pipelines passed. The UI pipelines, which carry the system tests, then run only against the release branch, and any defect they find is fixed on that branch. Commits merged to main after the cut stay off the branch, so a fix for one failure reaches the next run without new changes that can fail in their own ways.

main release branch committed after the cut, kept off the branch R: build and unit green fix F branch cut at R in every repository 100 UI pipelines 95 pass, 3 red, 2 wait Rerun the 3 red 2 pass, recorded as flakes 1 fails again defect, fixed on main Cherry-pick F rerun affected only Every pipeline passed release candidate Release built from the branch
The branch holds the revisions that passed build and unit tests. Red pipelines are rerun on those revisions, the 2 UI pipelines that wait on a red one start once its rerun passes, and the fix for the one real defect is the only new commit the branch receives.

Rerunning only the red pipelines on a frozen branch

On main, a pass counts only for the revision it ran on, and every commit to main starts a new wave of 100 UI pipelines with a 95% chance of failing somewhere at the overview's 3% flake rate. Waiting for one all-green wave takes about 21 waves on average, or roughly 42 hours of two-hour runs, before counting the commits that arrive during them. On the release branch the revisions stay fixed between fixes, so a pipeline that has passed once does not need to run again, and the candidate is complete when every pipeline has a recorded pass on the branch. A first wave leaves about three pipelines red, and each is rerun on the same revision. A flake keeps a pipeline red only if it fails on both attempts, with probability 0.03 × 0.03, so 91% of candidates have a pass from every pipeline after two attempts per pipeline and 99.7% after three.

The chains of UI pipelines inside the second wave change the count of pipelines without a pass, though not these odds. When an upstream pipeline is red, the pipelines below it have not run at all, so a first wave with three red pipelines can leave more than three without a pass. Rerunning the upstream pipeline starts the ones below it on the same revisions, and their first attempt comes only then, which adds the duration of each chain to the time before the candidate is complete. Reruns on the branch should therefore go to the red pipelines with the most pipelines waiting below them first.

The rerun also tells the two kinds of failure apart: a pipeline that passes on it failed from a flake, and the failing test goes to the quarantine list with an owner (practice 3), while a pipeline that fails again on the same revision points to a defect. Accepting the first pass lets an intermittent defect through, such as a race in a service that fails one run in three, because it passes on a rerun and is recorded as a flake. The owners of the quarantine entries raised on the branch still have to investigate each one before the next cut.

Fixing a defect on the release branch

The fix is made on main first and cherry-picked onto the branch, the order Trunk Based Development recommends, so the next cut from main already contains it. A fix made only on the branch is missing from the next release, and the defect returns with it. Recording the source commit on each cherry-pick, which git cherry-pick -x does, lets a script check before each cut that every commit on the branch has a counterpart on main.

Which pipelines rerun after a fix depends on where the fix lands. A fix in a backend repository moves only that service's revision on the branch, so its own pipelines rerun, along with the UI pipelines that depend on the service if the release pipelines model that dependency. A fix in the frontend mono repo moves the revision of the material that all 100 UI pipelines share, and GoCD triggers every one of them, with the pipelines that wait on other UI pipelines starting only after those pass again, which brings back the full wave the branch was cut to avoid. The affected graph from practice 6 limits the rerun to the modules the fix can reach, and the candidate check then accepts a module's earlier pass on the branch when the fix does not touch that module or its dependencies. A fix to the shared platform folder still reaches every UI module.

Using a release branch with release assembly

Release assembly (practice 1) freezes built artifacts and pins a faulty module back to an earlier version, while a release branch freezes source revisions and fixes the faulty module in place, and each suits a different part of this system. Services that publish versioned artifacts can be pinned back by the manifest. A frontend built from one mono repo revision into a single bundle cannot pin one UI module back, and practice 1 leaves it with a revert on main and a rebuild that picks up every commit merged since the candidate. On a release branch the revert or fix is one commit on the branch, and the bundle is rebuilt from the cut revision plus that commit. The manifest can then list the services' verified versions alongside the frontend bundle built from the branch.

In GoCD

Features to use

  • A git material names the branch it tracks, so a set of release pipelines generated from the same templates as the main pipelines can track the release branch in every repository.
  • The material revisions of each passed build stage, available through the API, give the revision to cut at in each repository.
  • A stage, or selected jobs within it, can be rerun on the same revisions, which reruns only the red UI pipelines on the branch.
  • Path filters on the mono repo material keep a cherry-pick from triggering release UI pipelines whose paths it does not touch. A UI pipeline whose material is another UI pipeline still runs each time that pipeline passes, regardless of its own path filter.

Gaps

  • GoCD cannot cut a branch across repositories, so the cut is a script that reads the passed revisions from the API and pushes a branch in each repository.
  • A fan-in over the release UI pipelines waits for all of them to pass on one revision of the shared mono repo material, so the candidate check has to be a script that reads each pipeline's latest pass on the branch.
  • GoCD has no record linking a branch commit to its commit on main, and the check for missing fixes runs outside it.
  • Moving a fixed branch name to a new cut is a non-fast-forward change, and how the git material handles it should be tested before relying on it. Per-release branch names avoid the problem but need the release pipelines regenerated for each release, for example from a config repository.

Pros and cons for this system

Pros

  • A fix reaches the candidate with no other new commits, so the next run tests the fix alone.
  • A pass on the branch stays valid until a fix touches that module, so a flake costs a rerun of one pipeline, and a candidate needs two or three attempts per pipeline instead of an all-green wave.
  • The approach works for a frontend built as a single bundle, which practice 1 cannot pin back one module at a time.
  • Main stays open to commits while the release is stabilised.
  • Reruns on the same revision classify each red pipeline as a flake or a defect, which gives practice 3 its record of which tests flaked on which revision.

Cons

  • Every fix on the branch needs a matching commit on main, and checking for it is a script outside GoCD.
  • Accepting the first pass records intermittent defects as flakes.
  • If the UI suites run only on the branch, a UI regression on main surfaces at the next cut, inside a batch of every commit since the previous one, unless the blocking tier (practice 2) catches it earlier.
  • A fix in the mono repo reruns all 100 UI pipelines until affected builds (practice 6) are in place.
  • Cutting each repository at its own last passed revision can split a change that spans repositories, which the contract check (practice 5) has to catch at the cut.
  • The longer a branch takes to stabilise, the further main moves on, and its cherry-picks stop applying cleanly.

Sources

Addresses big-bang integration at the fan-in, large batch size.

Sources Paul Hammant, "Branch for release", trunkbaseddevelopment.com, for cutting release branches and fixing on main before cherry-picking; Google Chromium project, "Chrome Release Cycle"; GoCD documentation, "Fan-in"; Git documentation, git cherry-pick.