# Incident: the cache that lied for forty minutes
Author: Digest Engineering
Author URL: https://dmeodigest.superblog.click/author/digest-engineering/
Published: 2026-08-23
Category: Incidents
Category URL: https://dmeodigest.superblog.click/category/incidents/
Meta Title: Incident: the cache that lied for forty minutes · Digest
Meta Description: On 14 August, reads returned stale data for forty-one minutes. What happened, what we did, and what changes.
Tags: incident, cache, postmortem
Tag URLs: incident (https://dmeodigest.superblog.click/tag/incident/), cache (https://dmeodigest.superblog.click/tag/cache/), postmortem (https://dmeodigest.superblog.click/tag/postmortem/)
URL: https://dmeodigest.superblog.click/incident-the-cache-that-lied-for-forty-minutes/

![A row of server racks in a data centre](https://images.unsplash.com/photo-1691435828932-911a7801adfb?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHwzfHxuZXR3b3JrJTIwY2FibGVzJTIwc3dpdGNofGVufDF8MHx8fDE3ODg2Mzg0NTJ8MA&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop)

Between 09:12 and 09:53 UTC on 14 August, about eleven percent of read requests returned data up to forty minutes old. No writes were lost. This is the account, written with Dev, who was on call.

## Timeline

- **09:12** A deploy of the cache layer rolled out a change to the invalidation key format.
- **09:14** Writes began invalidating keys in the new format. Reads on nodes not yet deployed were still looking up the old format, and finding stale entries.
- **09:31** A customer reported a dashboard showing yesterday's numbers. Alert thresholds had not fired: the responses were fast and well-formed.
- **09:44** The deploy was identified and the rollback started.
- **09:53** Rollback complete. All nodes on the old format; stale entries expired.

## Why it happened

The key format change was correct and the rollout was gradual, which is normally the safe way. Here it was the unsafe way: for the twenty minutes the fleet was mixed, half of it could not see the other half's invalidations.

## What changes

1. Cache key format changes now deploy in two steps: read both formats, then write the new one. Enforced by a lint on the invalidation code.
2. A staleness probe writes a known value every thirty seconds and alerts when a read returns an older one.

_Revised 21 August with the root-cause detail on the mixed fleet, which the first version had described as a bug in the invalidation code. It was not; the code was correct and the rollout order was wrong._


---
This blog is powered by Superblog. Visit https://superblog.ai to know more.
---