# About Name: Digest Description: Release notes, engineering notes and incident reports for Digest, an events API for product teams. URL: https://dmeodigest.superblog.click # Navigation Menu - Releases: https://dmeodigest.superblog.click/category/releases - Engineering: https://dmeodigest.superblog.click/category/engineering - Incidents: https://dmeodigest.superblog.click/category/incidents - Guides: https://dmeodigest.superblog.click/category/guides - About: https://dmeodigest.superblog.click/about - Sign up: https://dmeodigest.superblog.click/about # Blog Posts ## About Digest Author: Digest Engineering Author URL: https://dmeodigest.superblog.click/author/digest-engineering/ Published: 2026-09-01 Meta Title: About Digest · Digest Meta Description: Release notes, engineering notes and incident reports for Digest, an events API for product teams. URL: https://dmeodigest.superblog.click/about/ Digest is an events API: your product sends events, Digest stores them, and you query them from the API, the dashboard or the CLI. These are its release notes. ## What is here - **Releases**: every version, with what changed, what broke and how to upgrade. Numbered, dated, and never edited after the fact except to add a correction that says so. - **Engineering**: how the system is built and why, in enough detail to be useful to someone building something else. - **Incidents**: every outage that affected customers, with a timeline and what changes. Published within a week. - **Guides**: how to do the things people write to support about. ## Cadence A release roughly every two weeks. Engineering posts when there is something to say. Incident reports whenever there is an incident, which we hope is rarely. ## Who writes it The engineering team. Posts are signed by the team rather than by a person, except incident reports, which carry the name of whoever was on call. --- This blog is powered by Superblog. Visit https://superblog.ai to know more. --- ## 2.14: streaming responses, and a breaking change Author: Digest Engineering Author URL: https://dmeodigest.superblog.click/author/digest-engineering/ Published: 2026-08-31 Category: Releases Category URL: https://dmeodigest.superblog.click/category/releases/ Meta Title: 2.14: streaming responses, and a breaking change · Digest Meta Description: Responses can now stream. The v1 batch endpoint is removed, six months after deprecation. Upgrade notes inside. Tags: release, streaming, breaking Tag URLs: release (https://dmeodigest.superblog.click/tag/release/), streaming (https://dmeodigest.superblog.click/tag/streaming/), breaking (https://dmeodigest.superblog.click/tag/breaking/) URL: https://dmeodigest.superblog.click/2-14-streaming-responses-and-a-breaking-change/ ![Code on a dark screen](https://images.unsplash.com/photo-1607706189992-eae578626c86?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHw1fHx0ZXJtaW5hbCUyMGNvZGUlMjBzY3JlZW4lMjBkYXJrfGVufDF8MHx8fDE3ODg2Mzg0NTF8MA&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop) Digest 2.14 is out. One new capability, one removal, four fixes. ## Streaming Every endpoint that returns a list can now stream it. Pass `Accept: application/x-ndjson` and rows arrive as they are produced rather than after the last one. For a 200,000-row export this moves time-to-first-byte from forty seconds to under one. ```bash curl -H "Accept: application/x-ndjson" \ https://api.digest.example/v2/events?since=2026-08-01 ``` ## Breaking: v1 batch endpoint removed `POST /v1/batch` was deprecated in 2.8 and returned a warning header since 2.10. It is gone in 2.14. Requests to it now return 410 with a link to the v2 equivalent. If you are still on it, the migration is one path change and one field rename; the guide is linked from the 410 body. ## Fixed - Cursor pagination skipped a row when two rows shared a timestamp at a page boundary (#4471). - The CLI printed the wrong exit code on a partial diff (#4490). - Webhook retries could fire twice after a restart (#4502). - Rate-limit headers were missing on 304 responses (#4515). ## Upgrade `digest upgrade`, or pull the 2.14 image. Migrations run forward automatically and are reversible for thirty days. --- This blog is powered by Superblog. Visit https://superblog.ai to know more. --- ## 2.13: rate limits you can see Author: Digest Engineering Author URL: https://dmeodigest.superblog.click/author/digest-engineering/ Published: 2026-08-27 Category: Releases Category URL: https://dmeodigest.superblog.click/category/releases/ Meta Title: 2.13: rate limits you can see · Digest Meta Description: Every response now says how much of your limit is left and when it resets. The dashboard shows the same, per key. Tags: release, rate limits, api Tag URLs: release (https://dmeodigest.superblog.click/tag/release/), rate limits (https://dmeodigest.superblog.click/tag/rate-limits/), api (https://dmeodigest.superblog.click/tag/api/) URL: https://dmeodigest.superblog.click/2-13-rate-limits-you-can-see/ ![A screen showing a bar chart of usage](https://images.unsplash.com/photo-1686061592689-312bbfb5c055?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHw1fHxtb25pdG9yaW5nJTIwZGFzaGJvYXJkJTIwc2NyZWVufGVufDF8MHx8fDE3ODg2Mzg0NTN8MA&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop) Digest 2.13 makes the rate limit visible, which it should have been from the start. ## Headers - `X-RateLimit-Limit`: the ceiling for this key in the current window. - `X-RateLimit-Remaining`: what is left. - `X-RateLimit-Reset`: seconds until the window resets. They are on every response, including errors. A 429 additionally carries `Retry-After`. ## Dashboard Keys, then any key, then Usage: a chart of the last 24 hours against the limit, with the window that hit 429 highlighted if one did. ## Changed The window is now sliding rather than fixed. A burst at the top of the minute no longer clears at the bottom of it. Effective limits are unchanged; the shape of them is smoother. ## Fixed - The SDK's retry backoff started at zero, so the first retry was immediate (#4402). - Exports with an empty filter returned 400 rather than everything (#4419). --- This blog is powered by Superblog. Visit https://superblog.ai to know more. --- ## Incident: the cache that lied for forty minutes Author: Digest Engineering Author URL: https://dmeodigest.superblog.click/author/digest-engineering/ Published: 2026-08-23 Category: Incidents Category URL: https://dmeodigest.superblog.click/category/incidents/ Meta Title: Incident: the cache that lied for forty minutes · Digest Meta Description: On 14 August, reads returned stale data for forty-one minutes. What happened, what we did, and what changes. Tags: incident, cache, postmortem Tag URLs: incident (https://dmeodigest.superblog.click/tag/incident/), cache (https://dmeodigest.superblog.click/tag/cache/), postmortem (https://dmeodigest.superblog.click/tag/postmortem/) URL: https://dmeodigest.superblog.click/incident-the-cache-that-lied-for-forty-minutes/ ![A row of server racks in a data centre](https://images.unsplash.com/photo-1691435828932-911a7801adfb?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHwzfHxuZXR3b3JrJTIwY2FibGVzJTIwc3dpdGNofGVufDF8MHx8fDE3ODg2Mzg0NTJ8MA&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop) Between 09:12 and 09:53 UTC on 14 August, about eleven percent of read requests returned data up to forty minutes old. No writes were lost. This is the account, written with Dev, who was on call. ## Timeline - **09:12** A deploy of the cache layer rolled out a change to the invalidation key format. - **09:14** Writes began invalidating keys in the new format. Reads on nodes not yet deployed were still looking up the old format, and finding stale entries. - **09:31** A customer reported a dashboard showing yesterday's numbers. Alert thresholds had not fired: the responses were fast and well-formed. - **09:44** The deploy was identified and the rollback started. - **09:53** Rollback complete. All nodes on the old format; stale entries expired. ## Why it happened The key format change was correct and the rollout was gradual, which is normally the safe way. Here it was the unsafe way: for the twenty minutes the fleet was mixed, half of it could not see the other half's invalidations. ## What changes 1. Cache key format changes now deploy in two steps: read both formats, then write the new one. Enforced by a lint on the invalidation code. 2. A staleness probe writes a known value every thirty seconds and alerts when a read returns an older one. _Revised 21 August with the root-cause detail on the mixed fleet, which the first version had described as a bug in the invalidation code. It was not; the code was correct and the rollout order was wrong._ --- This blog is powered by Superblog. Visit https://superblog.ai to know more. --- ## How we test a migration before it runs Author: Digest Engineering Author URL: https://dmeodigest.superblog.click/author/digest-engineering/ Published: 2026-08-18 Category: Engineering Category URL: https://dmeodigest.superblog.click/category/engineering/ Meta Title: How we test a migration before it runs · Digest Meta Description: Every schema change runs against a copy of production, twice, before it touches production once. The pipeline, in detail. Tags: engineering, database, migrations Tag URLs: engineering (https://dmeodigest.superblog.click/tag/engineering/), database (https://dmeodigest.superblog.click/tag/database/), migrations (https://dmeodigest.superblog.click/tag/migrations/) URL: https://dmeodigest.superblog.click/how-we-test-a-migration-before-it-runs/ ![A whiteboard covered in a system diagram](https://images.unsplash.com/photo-1532619675605-1ede6c2ed2b0?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHw0fHx3aGl0ZWJvYXJkJTIwZGlhZ3JhbSUyMGVuZ2luZWVyaW5nfGVufDF8MHx8fDE3ODg2Mzg0NTJ8MA&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop) Digest runs about thirty schema migrations a month against a database of several terabytes. None has caused an outage in two years. That is not luck; it is a pipeline, and this is what is in it. ## Step one: the shadow Every night a copy of production is restored to a shadow cluster from the previous night's snapshot. It is a real copy with real sizes, not a fixture. ## Step two: the dry run A pull request containing a migration triggers a run against the shadow. The run records wall time, lock time per table, and the query plan for every statement. If any lock exceeds two seconds, the check fails and says which statement. ## Step three: the second run The migration runs again on the same shadow, to prove it is idempotent, and then its down migration runs, to prove it is reversible. A migration that cannot go backwards does not merge. ## Step four: production, gradually Migrations that add run first, in a separate deploy from the code that uses the new columns. Migrations that remove run last, one release after the code stopped using them. Nothing adds and removes in one step. ## What it costs The shadow cluster is about four percent of our database spend. A migration takes a day longer to ship than it would with no pipeline. Both are cheap. --- This blog is powered by Superblog. Visit https://superblog.ai to know more. --- ## 2.12: the CLI gets a diff Author: Digest Engineering Author URL: https://dmeodigest.superblog.click/author/digest-engineering/ Published: 2026-08-13 Category: Releases Category URL: https://dmeodigest.superblog.click/category/releases/ Meta Title: 2.12: the CLI gets a diff · Digest Meta Description: digest diff shows what an apply would change, before it changes it. Also: config validation, and a faster login. Tags: release, cli Tag URLs: release (https://dmeodigest.superblog.click/tag/release/), cli (https://dmeodigest.superblog.click/tag/cli/) URL: https://dmeodigest.superblog.click/2-12-the-cli-gets-a-diff/ ![A laptop showing a terminal and code](https://images.unsplash.com/photo-1555066931-4365d14bab8c?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHwyfHxkZXZlbG9wZXIlMjBsYXB0b3AlMjBjb2RlfGVufDF8MHx8fDE3ODg2Mzg0NTF8MA&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop) Digest 2.12 is a CLI release. ## digest diff Compares your local configuration with what is deployed and prints the change as a unified diff. Nothing is applied. It is the same computation `digest apply` performs, split out so you can read it first. ```diff $ digest diff --- deployed +++ local @@ webhooks @@ - retries: 3 + retries: 5 + backoff: exponential ``` ## digest validate Checks the configuration file against the schema and reports every error, not just the first. Runs automatically before apply. ## Login The browser flow now returns to the terminal in one step. The device-code flow remains for machines without a browser. ## Fixed - `apply` exited 0 on a partial failure (#4380). - Very long config files were truncated in the error output (#4391). --- This blog is powered by Superblog. Visit https://superblog.ai to know more. --- ## Guide: zero-downtime key rotation Author: Digest Engineering Author URL: https://dmeodigest.superblog.click/author/digest-engineering/ Published: 2026-08-08 Category: Guides Category URL: https://dmeodigest.superblog.click/category/guides/ Meta Title: Guide: zero-downtime key rotation · Digest Meta Description: Rotate an API key without a second of failed requests, using overlapping keys and the grace period. Tags: guide, security, api keys Tag URLs: guide (https://dmeodigest.superblog.click/tag/guide/), security (https://dmeodigest.superblog.click/tag/security/), api keys (https://dmeodigest.superblog.click/tag/api-keys/) URL: https://dmeodigest.superblog.click/guide-zero-downtime-key-rotation/ ![A keyboard in low light](https://images.unsplash.com/photo-1598662779094-110c2bad80b5?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHw3fHxrZXlib2FyZCUyMGNsb3NlJTIwdXAlMjBkYXJrfGVufDF8MHx8fDE3ODg2Mzg0NTR8MA&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop) A key should be rotated when someone leaves, when it may have leaked, and on a schedule regardless. Here is how to do it without an outage. ## The principle Two keys are valid at once for as long as it takes to move every client from the old one to the new one. The old one is then revoked. At no point does a client hold a key that does not work. ## Steps 1. Create the new key in the dashboard or with `digest keys create`. It is valid immediately. 2. Deploy the new key to every client. Use the Usage chart on the old key to see when its traffic reaches zero. 3. Set a grace period on the old key: `digest keys expire --in 24h`. Requests still work, and each carries a `Warning` header naming the key. 4. After the grace period, the key is revoked automatically. A request with it returns 401 with a body that says the key was rotated and when. ## If the key leaked Skip the grace period: `digest keys revoke `. Some clients will fail until they have the new key. That is the correct trade. ## FAQs Q: How often should keys rotate? A: Every ninety days on a schedule, and immediately on any suspicion of a leak or when a person with access leaves. Q: Can two keys have different permissions during the overlap? A: Yes. Rotation is a good time to narrow a key. Create the new one with the smaller scope and watch the old key's usage for calls the new one would refuse before you expire it. Q: Does a revoked key affect webhooks? A: No. Webhook signing secrets are separate from API keys and rotate independently, with the same grace mechanism. --- This blog is powered by Superblog. Visit https://superblog.ai to know more. --- ## Why our error messages have a number Author: Digest Engineering Author URL: https://dmeodigest.superblog.click/author/digest-engineering/ Published: 2026-08-02 Category: Engineering Category URL: https://dmeodigest.superblog.click/category/engineering/ Meta Title: Why our error messages have a number · Digest Meta Description: Every error Digest returns carries a stable code. What the codes are for, and how we stop them changing. Tags: api, engineering, errors Tag URLs: api (https://dmeodigest.superblog.click/tag/api/), engineering (https://dmeodigest.superblog.click/tag/engineering/), errors (https://dmeodigest.superblog.click/tag/errors/) URL: https://dmeodigest.superblog.click/why-our-error-messages-have-a-number/ ![A terminal window showing code on a dark screen](https://images.unsplash.com/photo-1617040619263-41c5a9ca7521?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHwzfHx0ZXJtaW5hbCUyMGNvZGUlMjBzY3JlZW4lMjBkYXJrfGVufDF8MHx8fDE3ODg2Mzg0NTF8MA&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop) Every error response from Digest looks like this: ``` { "error": { "code": "E2041", "message": "cursor has expired", "docs": "https://docs.digest.example/errors/E2041" } } ``` The code is the part that matters. Here is why. ## Messages change; codes do not A message is for a person reading a log. We reword them when they are unclear. A client that matches on message text breaks when we do. A code is a contract: E2041 has meant "cursor expired" since 1.0 and will mean it in 4.0. ## Codes are searchable Every code has a page. The page says what it means, why it happens, and what to do. Support answers a question about E2041 by sending the link. ## How we keep them stable - Codes live in one file. Adding one is a pull request that requires a docs page in the same change. - Removing one is not allowed. A code that no longer occurs stays in the file, marked retired, so it can never be reused for something else. - A test fails if a code is returned that is not in the file. ## Ranges E1xxx is authentication, E2xxx is requests, E3xxx is limits, E5xxx is us. If you see an E5, it is our fault and the status page will already say so. --- This blog is powered by Superblog. Visit https://superblog.ai to know more. --- ## 2.11: exports Author: Digest Engineering Author URL: https://dmeodigest.superblog.click/author/digest-engineering/ Published: 2026-07-27 Category: Releases Category URL: https://dmeodigest.superblog.click/category/releases/ Meta Title: 2.11: exports · Digest Meta Description: Export any query to CSV, JSON or Parquet. Runs in the background, notifies when done, and keeps the file for seven days. Tags: release, exports Tag URLs: release (https://dmeodigest.superblog.click/tag/release/), exports (https://dmeodigest.superblog.click/tag/exports/) URL: https://dmeodigest.superblog.click/2-11-exports/ ![Analytics graphs on a laptop screen](https://images.unsplash.com/photo-1551288049-bebda4e38f71?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHwxfHxtb25pdG9yaW5nJTIwZGFzaGJvYXJkJTIwc2NyZWVufGVufDF8MHx8fDE3ODg2Mzg0NTN8MA&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop) Digest 2.11 adds exports. ## What it does Any query you can run in the API or the dashboard can be exported. Small results return inline. Anything over ten thousand rows runs as a job, and you get a webhook or an email when the file is ready. ## Formats - CSV, with a header row and RFC 4180 quoting. - JSON, as an array or newline-delimited. - Parquet, for anything going into a warehouse. Column types follow the schema. ## Limits Ten million rows per export, five concurrent exports per account. Files are kept for seven days and then deleted. The size counts against your storage quota while it exists. ## API ```json POST /v2/exports { "query": { "since": "2026-07-01" }, "format": "parquet" } ``` ## Fixed - Filtering on a nullable field with `is: null` returned nothing (#4301). - The dashboard date picker used the browser's time zone for one field and UTC for another (#4322). --- This blog is powered by Superblog. Visit https://superblog.ai to know more. --- ## Incident: a bad deploy on a Friday Author: Digest Engineering Author URL: https://dmeodigest.superblog.click/author/digest-engineering/ Published: 2026-07-21 Category: Incidents Category URL: https://dmeodigest.superblog.click/category/incidents/ Meta Title: Incident: a bad deploy on a Friday · Digest Meta Description: A configuration change took the API down for six minutes on 3 July. Short account, and why we now deploy on Fridays anyway. Tags: incident, postmortem, deploys Tag URLs: incident (https://dmeodigest.superblog.click/tag/incident/), postmortem (https://dmeodigest.superblog.click/tag/postmortem/), deploys (https://dmeodigest.superblog.click/tag/deploys/) URL: https://dmeodigest.superblog.click/incident-a-bad-deploy-on-a-friday/ ![A rack of equipment in a dark server room](https://images.unsplash.com/photo-1680992046626-418f7e910589?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHwxfHxzZXJ2ZXIlMjByb29tJTIwcmFja3N8ZW58MXwwfHx8MTc4ODYzODQ1MXww&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop) On 3 July at 16:40 UTC, a configuration deploy set the connection pool size to zero on every API node. The API returned 503 for six minutes. Nothing was lost; nothing worked. ## What happened A pull request changed a pool setting from a literal to a reference to an environment variable. The variable was set in staging and not in production. The default for an unset variable was zero, which validation did not reject because zero is an integer. ## What we did - **16:40** Deploy completed; health checks failed on every node within thirty seconds. - **16:41** Automatic rollback started. It takes about five minutes across the fleet. - **16:46** Rollback complete. ## What changes 1. Pool size has a minimum of one, enforced at startup. A node with a bad value refuses to start rather than starting and failing. 2. Configuration referring to an environment variable fails validation if the variable is unset in the target environment. 3. Rollback now runs in parallel across the fleet: about ninety seconds rather than five minutes. ## On Fridays We considered banning Friday deploys and decided not to. A deploy that is unsafe on Friday is unsafe on Tuesday. The fix is a faster rollback, which we now have. --- This blog is powered by Superblog. Visit https://superblog.ai to know more. --- ## Guide: reading a trace Author: Digest Engineering Author URL: https://dmeodigest.superblog.click/author/digest-engineering/ Published: 2026-07-15 Category: Guides Category URL: https://dmeodigest.superblog.click/category/guides/ Meta Title: Guide: reading a trace · Digest Meta Description: Every request has a trace. Here is how to find it, what each span means, and how to tell our latency from yours. Tags: guide, observability, latency Tag URLs: guide (https://dmeodigest.superblog.click/tag/guide/), observability (https://dmeodigest.superblog.click/tag/observability/), latency (https://dmeodigest.superblog.click/tag/latency/) URL: https://dmeodigest.superblog.click/guide-reading-a-trace/ ![A monitor showing a monitoring dashboard with graphs](https://images.unsplash.com/photo-1526628953301-3e589a6a8b74?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHwyfHxtb25pdG9yaW5nJTIwZGFzaGJvYXJkJTIwc2NyZWVufGVufDF8MHx8fDE3ODg2Mzg0NTN8MA&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop) Every response from Digest carries an `X-Trace-Id` header. Paste it into the dashboard, or into `digest trace `, and you get the request's timeline. This guide is how to read it. ## The spans - **edge**: from the request reaching our network to reaching an API node. Usually under 5ms. Larger means a slow path from you to us. - **auth**: key lookup and rate-limit check. Cached; under 1ms. - **query**: the database. This is where time goes. The span carries the query plan. - **serialize**: building the response. Scales with rows. - **transfer**: sending it to you. Scales with bytes and your bandwidth. ## Whose latency is it Add edge and transfer: that is the network. Add the rest: that is us. If the total on your side is much larger than the trace total, the time is in your client, before the request left or after the response arrived. ## A slow query span Open the plan. A sequential scan on a large table means a filter with no index; the plan names the column. Filters on indexed columns are listed in the docs, and the query page in the dashboard warns before you run one that is not. ## Retention Traces are kept for fourteen days. Include the id in any support request and the answer comes faster. --- This blog is powered by Superblog. Visit https://superblog.ai to know more. --- ## 2.10: webhooks retry Author: Digest Engineering Author URL: https://dmeodigest.superblog.click/author/digest-engineering/ Published: 2026-07-08 Category: Releases Category URL: https://dmeodigest.superblog.click/category/releases/ Meta Title: 2.10: webhooks retry · Digest Meta Description: Failed deliveries retry with exponential backoff for 24 hours. The delivery log shows every attempt. Tags: release, webhooks Tag URLs: release (https://dmeodigest.superblog.click/tag/release/), webhooks (https://dmeodigest.superblog.click/tag/webhooks/) URL: https://dmeodigest.superblog.click/2-10-webhooks-retry/ ![Fibre cables plugged into a network switch](https://images.unsplash.com/photo-1594915440248-1e419eba6611?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHw0fHxuZXR3b3JrJTIwY2FibGVzJTIwc3dpdGNofGVufDF8MHx8fDE3ODg2Mzg0NTJ8MA&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop) Digest 2.10 makes webhooks reliable. ## Retries A delivery that receives no 2xx is retried: after one minute, then five, then twenty-five, and so on up to five hours between attempts, for 24 hours in total. Each attempt carries the same event id, so a receiver can de-duplicate. ## The log Webhooks, then any endpoint, then Deliveries: every event, every attempt, the status code and the response body we received. Redeliver any event by hand from the same page. ## Signatures Every delivery is signed with the endpoint's secret in `X-Digest-Signature`. Verification examples for six languages are in the docs. Rotate the secret from the endpoint page; the old one is honoured for an hour. ## Changed An endpoint that fails every delivery for three days is disabled and its owner emailed. Previously it was disabled silently after one day. ## Fixed - Events created during a deploy could be delivered out of order (#4210). --- This blog is powered by Superblog. Visit https://superblog.ai to know more. --- ## The queue we replaced Author: Digest Engineering Author URL: https://dmeodigest.superblog.click/author/digest-engineering/ Published: 2026-07-01 Category: Engineering Category URL: https://dmeodigest.superblog.click/category/engineering/ Meta Title: The queue we replaced · Digest Meta Description: Three years on a message broker, one month on a table in the database. What we gained, what we gave up, and the numbers. Tags: engineering, database, queues Tag URLs: engineering (https://dmeodigest.superblog.click/tag/engineering/), database (https://dmeodigest.superblog.click/tag/database/), queues (https://dmeodigest.superblog.click/tag/queues/) URL: https://dmeodigest.superblog.click/the-queue-we-replaced/ ![Network cables plugged into a switch](https://images.unsplash.com/photo-1683322499436-f4383dd59f5a?ixid=M3w3OTAzOTJ8MHwxfHNlYXJjaHwxfHxuZXR3b3JrJTIwY2FibGVzJTIwc3dpdGNofGVufDF8MHx8fDE3ODg2Mzg0NTJ8MA&ixlib=rb-4.1.0&w=1600&q=80&auto=format&fit=crop) Digest ran its background jobs through a message broker for three years. In June we replaced it with a table in the main database. This is the reasoning and the result. ## What the broker cost - A second system to run, monitor, upgrade and page on. - Two sources of truth. A job could be acknowledged in the broker and its row uncommitted in the database, or the reverse. Every such gap was a bug we eventually met. - Nobody could see the queue. Looking meant a separate tool with a separate login. ## What the table does A jobs table, a worker that selects the next row with `FOR UPDATE SKIP LOCKED`, and a transaction that wraps the job and its row. A job that fails rolls back with its work. A job that succeeds commits with it. There is one source of truth because there is one system. ## The numbers - Throughput: about 4,000 jobs a second at peak, against a need of about 600. The broker did more; we did not need it. - Latency from enqueue to start: median 8ms, previously 40ms, because there is no hop. - Operational pages in the first month: zero. The broker averaged two. ## What we gave up Fan-out to many consumers, which we used once and replaced with a loop. And headroom we would need at about ten times current load, at which point the table becomes a partitioned table, which is a migration and not a rewrite. --- This blog is powered by Superblog. Visit https://superblog.ai to know more. ---