The outbox pattern saves a message in the same database transaction as the data change it describes, then uses a background worker to send it.
Concurrent workers can drain the outbox table with FOR UPDATE SKIP LOCKED, but new messages wait for the next poll.
Reading the Postgres write-ahead log with logical replication delivers them in about a millisecond at low load, at the cost of a replication slot and a cleanup job.
If you build distributed systems, you probably heard about the Outbox pattern.
Your API saves an order, then crashes before it can send the message announcing that order.
The order is in the database, but other services waiting for that message never hear about it.
The outbox pattern handles this by saving the message in a separate table,
in the same database transaction as the order.
Either both are saved or neither is.
This is the A in ACID.
Atomicity.
A background worker reads the saved messages and publishes them to a message broker for other services to receive.
If the API crashes after saving the order, the message is still there for the worker to send.
Querying the table repeatedly for pending messages is called polling.
Polling is a good starting point for an outbox processor.
It's decently fast and easy to implement.
But the delay between checks is a tradeoff.
Checking more often means more database queries, while waiting longer delays new messages.
I want to reduce that wait by having Postgres tell me when there's something to process, and see how that compares with the polling workers.
Concurrent Workers Draining the Table
For this walkthrough, we'll use an order API on Postgres that saves each OrderCreated message in an outbox_messages table, with RabbitMQ as the broker.
I want to run several workers at once, and Postgres row locks stop two of them from grabbing the same message.
Each worker starts a transaction and claims a batch with this query, using Dapper:
SELECT id, type, content
FROM outbox_messages
WHERE processed_on_utc IS NULL
ORDER BY occurred_on_utc
LIMIT 1000
FOR UPDATE SKIP LOCKED
FOR UPDATE locks the rows this worker picked until its transaction ends.
SKIP LOCKED makes the next worker take other rows instead of waiting.
The locks live in Postgres, so these workers can run in one process or in several.
The worker keeps that transaction open while it publishes with RabbitMQ.Client, then sets processed_on_utc and commits once RabbitMQ confirms.
When a worker runs out of rows, it sleeps for one second with Task.Delay and tries again.
If a worker crashes before it commits, those rows are still unprocessed, and a worker picks them up again.
Some of those messages may have already reached RabbitMQ, so the same message can arrive twice.
That makes this at-least-once delivery, so your consumers must be idempotent.
Why Not Just LISTEN/NOTIFY?
Postgres has a built-in way to wake those workers.
With LISTEN/NOTIFY, one connection runs LISTEN outbox_new, and Postgres tells it when another connection runs NOTIFY outbox_new and commits.
My first implementation sent NOTIFY from a database trigger on the outbox table.
It needs no API changes and worked well at 100 orders per second.
But with 100 concurrent users, throughput fell from 8,070 orders per second with polling to 256.
The trigger runs inside the order transaction.
Before that transaction commits, Postgres takes a lock and holds it until after the commit.
Every transaction that sends a notification takes the same lock.
A commit normally waits for a disk flush, and concurrent transactions can share one, which is called group commit.
With the lock, each order waits for the one in front of it to finish its own flush.
Sending NOTIFY after the commit, from a separate connection, avoids that lock.
At 100 users the API handled 7,674 orders per second, about 5% below polling, and at 100 orders per second a message took 1.91 ms from commit to consumer.
But now I'm running a notifier, a listener, and a fallback poll for notifications sent while the listener is disconnected, all to wake the same workers.
Postgres already has a record of every new row, so let's read that instead.
Read the Outbox From the WAL
Postgres writes every change to the write-ahead log (WAL).
Logical replication streams committed changes to my app, so there's no polling and no wake-up to miss.
You need wal_level=logical, which takes a restart on Postgres 18.
Then create a publication and a replication slot:
CREATE PUBLICATION outbox_pub FOR TABLE outbox_messages WITH (publish = 'insert');
SELECT pg_create_logical_replication_slot('outbox_slot', 'pgoutput');
The slot remembers how far my reader has confirmed, so it can resume after a restart.
Let's read the stream with an Npgsql replication connection:
await using var connection = new LogicalReplicationConnection(connectionString);
await connection.Open(cancellationToken);
var slot = new PgOutputReplicationSlot("outbox_slot");
var options = new PgOutputReplicationOptions("outbox_pub", PgOutputProtocolVersion.V1);
await foreach (var message in connection.StartReplication(slot, options, cancellationToken))
{
if (message is InsertMessage insert)
{
await PublishAsync(insert, cancellationToken);
}
else if (message is CommitMessage commit)
{
connection.SetReplicationStatus(commit.TransactionEndLsn);
}
}
PublishAsync publishes to RabbitMQ and waits for the confirm.
I only confirm a position after that, when the CommitMessage arrives, so a crash replays messages instead of losing them.
This loop only suits low rates, because waiting on every message drained about 1,600 per second.
So the reader I measured publishes first and waits for confirms, in order, in a separate task.
While the reader is down, the slot keeps all the server's WAL on disk, so a long outage can fill it.
You can cap that with max_slot_wal_keep_size, but Postgres invalidates a slot past the cap.
This reader is a prototype, and I haven't tested it through a failover or on managed Postgres.
Workers vs the WAL Reader
The main benchmark comes from my scaling article: I seeded a backlog, started the processor, and counted what RabbitMQ had after one minute, in messages per second (MPS).
To keep it fair, the WAL reader also marks rows as processed in batches, so it does the same UPDATE as the workers.
This ran on one machine, with Postgres 18 and RabbitMQ 4 in Docker:
| Setup | MPS after 1 minute | Average commit to consumer, 100 orders/s | Average commit to consumer, 100 users |
|---|
| 1 worker | 14,516 to 15,050 | 500 ms | 8.5 s |
| 5 workers | 22,216 to 24,133 | 419 ms | 54 to 85 ms |
| WAL reader | 27,962 to 34,930 | 0.83 ms | 8.6 ms |
One worker already moves about 15,000 messages per second, and five get to about 23,000.
The WAL reader leads at about 31,000, and it reached about 57,600 without the UPDATE.
Both move thousands of messages per second, so throughput alone isn't a reason to switch.
Latency is the real difference.
New messages wait for a worker's one-second sleep, so five workers average 419 ms at 100 orders per second, while the reader stays under a millisecond.
At 100 users, one worker fell 8.5 seconds behind, five workers averaged 54 to 85 ms, and the reader held 8.6 ms.
A crash goes the other way.
A hard kill partway through 200,000 rows lost nothing, but the workers sent 4,536 duplicates and the WAL reader 31,095, because it replays from its last confirmed position.
Workers also scale out, because you can add more in any process.
A slot has one reader, which keeps commit order, but I can't scale it out.
Summary
Keep polling if the delay is acceptable for your application.
Add workers when you need more throughput, because the row locks already handle the concurrency.
Read the WAL if you need new messages right away and can run a replication slot and a cleanup job.
If you use NOTIFY instead, send it after the commit and keep a fallback poll.
A worker can still crash after publishing but before committing its processed markers, causing a message to be sent again.
Consumers must be idempotent: processing the same message twice must not repeat its effects.
Thanks for reading.
And stay awesome!
Log in
