Incident: the cache that lied for forty minutes
On 14 August, reads returned stale data for forty-one minutes. What happened, what we did, and what changes...
Between 09:12 and 09:53 UTC on 14 August, about eleven percent of read requests returned data up to forty minutes old. No writes were lost. This is the account, written with Dev, who was on call.
Timeline
- 09:12 A deploy of the cache layer rolled out a change to the invalidation key format.
- 09:14 Writes began invalidating keys in the new format. Reads on nodes not yet deployed were still looking up the old format, and finding stale entries.
- 09:31 A customer reported a dashboard showing yesterday's numbers. Alert thresholds had not fired: the responses were fast and well-formed.
- 09:44 The deploy was identified and the rollback started.
- 09:53 Rollback complete. All nodes on the old format; stale entries expired.
Why it happened
The key format change was correct and the rollout was gradual, which is normally the safe way. Here it was the unsafe way: for the twenty minutes the fleet was mixed, half of it could not see the other half's invalidations.
What changes
- Cache key format changes now deploy in two steps: read both formats, then write the new one. Enforced by a lint on the invalidation code.
- A staleness probe writes a known value every thirty seconds and alerts when a read returns an older one.
Revised 21 August with the root-cause detail on the mixed fleet, which the first version had described as a bug in the invalidation code. It was not; the code was correct and the rollout order was wrong.
Release notes by email
Every release and every incident report, the day it is published. Nothing else.
Digest Engineering
The team that builds Digest. Release notes are signed by the team; incident reports by whoever was on call.
All Posts