# The queue we replaced
Author: Digest Engineering
Author URL: https://dmeodigest.superblog.click/author/digest-engineering/
Published: 2026-07-01
Category: Engineering
Category URL: https://dmeodigest.superblog.click/category/engineering/
Meta Title: The queue we replaced · Digest
Meta Description: Three years on a message broker, one month on a table in the database. What we gained, what we gave up, and the numbers.
Tags: engineering, database, queues
Tag URLs: engineering (https://dmeodigest.superblog.click/tag/engineering/), database (https://dmeodigest.superblog.click/tag/database/), queues (https://dmeodigest.superblog.click/tag/queues/)
URL: https://dmeodigest.superblog.click/the-queue-we-replaced/

![Network cables plugged into a switch](https://images.unsplash.com/photo-1683322499436-f4383dd59f5a?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHwxfHxuZXR3b3JrJTIwY2FibGVzJTIwc3dpdGNofGVufDF8MHx8fDE3ODg2Mzg0NTJ8MA&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop)

Digest ran its background jobs through a message broker for three years. In June we replaced it with a table in the main database. This is the reasoning and the result.

## What the broker cost

- A second system to run, monitor, upgrade and page on.
- Two sources of truth. A job could be acknowledged in the broker and its row uncommitted in the database, or the reverse. Every such gap was a bug we eventually met.
- Nobody could see the queue. Looking meant a separate tool with a separate login.

## What the table does

A jobs table, a worker that selects the next row with `FOR UPDATE SKIP LOCKED`, and a transaction that wraps the job and its row. A job that fails rolls back with its work. A job that succeeds commits with it. There is one source of truth because there is one system.

## The numbers

- Throughput: about 4,000 jobs a second at peak, against a need of about 600. The broker did more; we did not need it.
- Latency from enqueue to start: median 8ms, previously 40ms, because there is no hop.
- Operational pages in the first month: zero. The broker averaged two.

## What we gave up

Fan-out to many consumers, which we used once and replaced with a loop. And headroom we would need at about ten times current load, at which point the table becomes a partitioned table, which is a migration and not a rewrite.


---
This blog is powered by Superblog. Visit https://superblog.ai to know more.
---