Caught a settlement regression with eleven thousand replays

The problem

Cedarlift's settlement endpoints touch money, and its integration test suite was built on fixtures written in 2023. The fixtures passed; production traffic did not always agree with them.

A rounding change shipped in March produced settlement totals that were correct to the cent for every fixture and wrong for any shipment split across more than four legs — a shape that existed only in real data.

What they did

• A replay slice per release

  Cedarlift selects 11,400 production wakes across the settlement endpoints and replays them against the release candidate in staging.

• Response diffing

  Any structural or numeric difference outside a tolerance is raised in the release checklist with both payloads attached.

• Redaction first

  Field rules strip consignee details at the edge, so the replay slice is safe to run in a lower environment.

What changed

• Three customer-visible regressions were caught before release in the first two quarters.
• There have been no settlement incidents since replay entered the release checklist.
• The fixture suite is now generated from replay slices rather than written by hand.

Their stack

• Java origin services
• GCP Cloud Run
• Flintwake collector
• BigQuery export
• GitHub Actions

Cedarlift Logistics is a fictional company created for this template. Every figure on this page is demonstration data.