Industry practices ยท Practice 8 of 10

Move some risk control after deployment

Organisations that release often rely less on exhaustive pre-release UI suites and more on controls that act after deployment, and most of these controls work for a full-system release as well as for a single service. Feature flags separate deploying code from exposing it to users. Canary and progressive delivery of the whole release (Argo Rollouts, Flagger, Spinnaker, or flag-based percentage rollouts) expose a change to a small share of traffic first, and automated rollback reacts to error rates. Synthetic monitoring runs a few critical UI journeys continuously against production. The canarying chapter of the Google SRE Workbook covers how to compare a canary against a control population and which health metrics should decide whether a rollout expands, and that comparison works for a full-system release rolled out by region or by a share of users. Charity Majors and others in the observability community describe this as "testing in production", meaning that production telemetry becomes a test signal alongside pre-release tests.

These controls let the pre-release gate shrink to what is fast and deterministic, because the residual risk is contained by limited exposure and quick rollback.

In GoCD

Features to use

  • Deployment pipelines downstream of the release assembly promote the same manifest through environments, with manual approval stages where the release process needs them.
  • Rerunning the deploy stage of an earlier pipeline instance redeploys that instance's artifacts, which gives a rollback to the previous manifest.
  • A deploy stage can hand a rollout to Argo Rollouts or Flagger and wait on its status, so the GoCD run reflects whether the canary succeeded.

Gaps

  • GoCD has no canary analysis and no rollback driven by metrics. Both come from the rollout tool or the service mesh.
  • Feature flags need a separate flag service, and GoCD does not track which flags a release depends on.

Pros and cons for this system

Pros

  • An escaped defect reaches a small share of users and is rolled back before it reaches the rest.
  • Production telemetry becomes a test signal that pre-release suites cannot provide.
  • The pre-release gate can shrink to its fast and deterministic parts.

Cons

  • Some users do see an escaped defect, which matters more for some journeys than others.
  • Canarying a full-system release means running two versions of the system side by side, including compatible database schemas and message formats.
  • The approach depends on good observability and automated rollback, which may not exist yet.
  • If releases need formal change approval, a canary and its automatic rollback have to be agreed with whoever approves releases.

Sources

Addresses ice-cream cone, large batch size.

Sources Pete Hodgson on martinfowler.com, "Feature Toggles (aka Feature Flags)"; Google, The Site Reliability Workbook, "Canarying Releases"; Argo Rollouts (CNCF Argo project); Flagger (CNCF Flux project); Charity Majors, Honeycomb, "I Test in Prod".