Industry practices ยท Practice 2 of 10

Split the tests into blocking and non-blocking tiers

Slack's engineering team (Carlos Valdez and Frank Chen, writing on slack.engineering in 2022) described how its main webapp monorepo had required most end-to-end tests to pass before a pull request could merge, which slowed developers and let flaky tests block unrelated work. Slack moved most of those tests to after merge. A small set of high-criticality tests still gates the merge, a larger set gates deploys, and the remainder runs in scheduled batches. Failures in the later tiers alert triage engineers and the owning team and no longer block every merge.

Developers here commit straight to main, so no check runs before a commit lands apart from what the developer runs locally, and Slack's pre-merge gate has no direct counterpart. The tier boundary falls at the fan-in instead, between the build and unit pipelines and the UI pipelines. The blocking tier decides whether a module revision produces a candidate artifact, and a red result there means the revision never becomes a candidate and its team reverts or fixes it on main.

Applied here, the tiers follow from the other practices. The blocking tier holds the affected build, the unit and component tests and the contract checks for the modules a commit touches. The non-blocking tier holds the full UI suites per module, run on main against versioned artifacts, with every result recorded for flake tracking. The full-system release decision then reads the evidence from both tiers: a manifest in which every version passed its blocking tier, contract compatibility across the manifest, and UI results whose only failures belong to tests already in quarantine. A release decided this way still has a gate, and the gate no longer depends on 100 UI pipelines turning green in the same run.

Commit stage decides the candidate affected build unit, component contract checks Candidate build and unit passed versioned artifact per module UI stage reports, does not block UI system tests flake tracking culprit finding Release decision evidence from both tiers green manifest contracts compatible only known flakes fail real regression found in the UI stage: revert or fix on main then canary rollout
Blocking and non-blocking tiers. Only fast, deterministic checks decide whether a commit produces a candidate artifact, while the broad UI suites after the fan-in feed evidence into the release decision.

In GoCD

Features to use

  • The existing build and unit pipelines already carry the blocking tier, so the change lies in what follows them.
  • The non-blocking tier is a separate downstream pipeline per module, triggered by a dependency material on the module's build stage, so a red UI run never fails the module pipeline whose result the release assembly reads. The material should be the module's build stage alone: a UI pipeline that also depends on another UI pipeline, as some do today, still gets no run when that pipeline is red, and the release decision loses its evidence for that module.
  • Jobs that upload JUnit XML as test artifacts get a Tests tab listing the failing tests for that run.
  • Notification plugins for Slack and email can alert the owning team when a UI pipeline fails.

Gaps

  • GoCD has no advisory or allowed-to-fail stage. A failing stage stops its pipeline, so every non-blocking suite needs its own pipeline.
  • A UI pipeline instance can cover several commits, and GoCD has no culprit finding to say which one broke it.
  • Because commits land on main directly, GoCD cannot hold back a commit whose blocking tier fails, so a red build and unit pipeline still needs its team to revert or fix it.

Pros and cons for this system

Pros

  • Flaky UI tests stop deciding whether a candidate exists, so the release no longer depends on the flake rate of 100 pipelines.
  • Developers learn within minutes from the fast tier whether their commit produced a candidate.
  • The full UI suites keep running on main and still contribute evidence to the release decision.

Cons

  • Regressions found in the UI stage are already on main and need a revert or a fix, and someone has to triage them. Slack routes these failures to dedicated triage engineers.
  • A UI-stage failure can span several commits, so the tier needs culprit finding to be useful.
  • Teams have to agree which tests are critical enough to block, and the blocking set tends to grow back unless it is reviewed on a schedule.
  • The release decision reads several kinds of evidence instead of one green light, and the rules for it have to be written down and agreed.

Sources

Addresses ice-cream cone, integrated tests used as the gate, big-bang integration at the fan-in.

Sources Slack Engineering, Carlos Valdez and Frank Chen, "Balancing Safety and Velocity in CI/CD at Slack" (2022).