How we test a migration before it runs
Every schema change runs against a copy of production, twice, before it touches production once. The pipeline, in detail...
Digest runs about thirty schema migrations a month against a database of several terabytes. None has caused an outage in two years. That is not luck; it is a pipeline, and this is what is in it.
Step one: the shadow
Every night a copy of production is restored to a shadow cluster from the previous night's snapshot. It is a real copy with real sizes, not a fixture.
Step two: the dry run
A pull request containing a migration triggers a run against the shadow. The run records wall time, lock time per table, and the query plan for every statement. If any lock exceeds two seconds, the check fails and says which statement.
Step three: the second run
The migration runs again on the same shadow, to prove it is idempotent, and then its down migration runs, to prove it is reversible. A migration that cannot go backwards does not merge.
Step four: production, gradually
Migrations that add run first, in a separate deploy from the code that uses the new columns. Migrations that remove run last, one release after the code stopped using them. Nothing adds and removes in one step.
What it costs
The shadow cluster is about four percent of our database spend. A migration takes a day longer to ship than it would with no pipeline. Both are cheap.
Release notes by email
Every release and every incident report, the day it is published. Nothing else.
Digest Engineering
The team that builds Digest. Release notes are signed by the team; incident reports by whoever was on call.
All Posts