Incident: a bad deploy on a Friday
A configuration change took the API down for six minutes on 3 July. Short account, and why we now deploy on Fridays anyway...
On 3 July at 16:40 UTC, a configuration deploy set the connection pool size to zero on every API node. The API returned 503 for six minutes. Nothing was lost; nothing worked.
What happened
A pull request changed a pool setting from a literal to a reference to an environment variable. The variable was set in staging and not in production. The default for an unset variable was zero, which validation did not reject because zero is an integer.
What we did
- 16:40 Deploy completed; health checks failed on every node within thirty seconds.
- 16:41 Automatic rollback started. It takes about five minutes across the fleet.
- 16:46 Rollback complete.
What changes
- Pool size has a minimum of one, enforced at startup. A node with a bad value refuses to start rather than starting and failing.
- Configuration referring to an environment variable fails validation if the variable is unset in the target environment.
- Rollback now runs in parallel across the fleet: about ninety seconds rather than five minutes.
On Fridays
We considered banning Friday deploys and decided not to. A deploy that is unsafe on Friday is unsafe on Tuesday. The fix is a faster rollback, which we now have.
Release notes by email
Every release and every incident report, the day it is published. Nothing else.
Digest Engineering
The team that builds Digest. Release notes are signed by the team; incident reports by whoever was on call.
All Posts