The queue we replaced
Three years on a message broker, one month on a table in the database. What we gained, what we gave up, and the numbers...
Digest ran its background jobs through a message broker for three years. In June we replaced it with a table in the main database. This is the reasoning and the result.
What the broker cost
- A second system to run, monitor, upgrade and page on.
- Two sources of truth. A job could be acknowledged in the broker and its row uncommitted in the database, or the reverse. Every such gap was a bug we eventually met.
- Nobody could see the queue. Looking meant a separate tool with a separate login.
What the table does
A jobs table, a worker that selects the next row with FOR UPDATE SKIP LOCKED, and a transaction that wraps the job and its row. A job that fails rolls back with its work. A job that succeeds commits with it. There is one source of truth because there is one system.
The numbers
- Throughput: about 4,000 jobs a second at peak, against a need of about 600. The broker did more; we did not need it.
- Latency from enqueue to start: median 8ms, previously 40ms, because there is no hop.
- Operational pages in the first month: zero. The broker averaged two.
What we gave up
Fan-out to many consumers, which we used once and replaced with a loop. And headroom we would need at about ten times current load, at which point the table becomes a partitioned table, which is a migration and not a rewrite.
Release notes by email
Every release and every incident report, the day it is published. Nothing else.
Digest Engineering
The team that builds Digest. Release notes are signed by the team; incident reports by whoever was on call.
All Posts