If you have worked on a distributed system for long enough, you have felt the particular cold sweat of the dual write. You insert an order into the database. The transaction commits. Then you have to tell something outside the database about it, a service over HTTP or a Kafka topic, and the connection drops before you can. The order exists and nobody downstream knows. Or the reverse: the message goes out first, the transaction rolls back, and now a consumer is holding an event for an order that was never created. A ghost.

No amount of try/catch makes the two writes atomic.

Think of paying at a restaurant. The payment is your database transaction. The receipt is the message to the outside world. If you walk out without the receipt, you cannot prove the meal happened. The transactional outbox pattern is the industry's answer to that gap: instead of collecting the receipt at the door, you put it in your pocket at the moment you pay. The message goes into an outbox table in the same transaction as the business row, so both commit or neither does.

That solves the dual write. It also creates the problem this post is about.

The relay is a second system#

Someone has to take the receipt out of your pocket and show it to the world. In the conventional design that is a relay: a separate service, outside the database, that polls the outbox table, delivers each row, and marks it done.

The relay is small on the whiteboard and large in production. It has its own deployment, configuration, connection string and alerts. It has to claim rows without two instances grabbing the same one, retry, back off, give up, and put what it gave up on somewhere a person can look. And because it lives outside the database, its view of the world can drift from the database's, which is how relays end up polling a replica for hours while reporting an empty queue.

I wrote ulak because I kept building that relay, and every time it was the least interesting and most fragile part of the system. ulak is a C extension for PostgreSQL 14 through 18, Apache 2.0, that keeps the outbox pattern and removes the relay. The queue is a table you write to in your own transaction. The delivery is done by PostgreSQL background workers.

What it is not#

ulak is not an event streaming platform and it is not a replacement for change data capture. If you are moving hundreds of thousands of log events a second, use Kafka. If you need every change in a database mirrored into a log, use Debezium. ulak has a narrower job: for a system whose core already lives in PostgreSQL, deliver outbox messages locally and transactionally, without running a broker to do it.

Enqueue is your transaction#

The application side is a single function call inside the transaction you already have:

BEGIN;
  INSERT INTO orders (customer_id, total_cents) VALUES (42, 12990);
 
  SELECT ulak.send(
    'billing-webhook',
    jsonb_build_object('order_id', 17, 'total_cents', 12990)
  );
COMMIT;

ulak.send inserts a row into ulak.queue. That row shares a fate with your order. If the next line of your code throws and the transaction rolls back, the message does not float off into space, it is gone as if it never existed. If the transaction commits, the message is durable before anything has tried to deliver it. The receipt is in your pocket.

Delivery is a PostgreSQL process#

This is where the design gets unusual, and a little bold. The workers that drain the queue are PostgreSQL background workers, registered through shared_preload_libraries when the server starts. Their lifecycle is the server's lifecycle. There is no relay to deploy, and when the database fails over, the workers fail over with it, because they are part of it.

The first reaction most engineers have is the right one. PostgreSQL is the heart of the system and the sole guardian of the data. Injecting a library into it that runs its own loops and makes its own network calls sounds like a risk. What if a bug in a worker locks the database or eats its memory?

That concern is the central trade-off of the whole design, and I am not going to argue it away. You remove the operational cost of an independent service and the network hop between it and the database. In exchange you tie the delivery pipeline's lifecycle to the database's. PostgreSQL's background worker infrastructure has matured a great deal: workers are isolated processes with their own memory contexts under PostgreSQL's rules, so a leak is reclaimed when the transaction ends and a crash does not take the postmaster down. It is still a decision that needs a deliberate yes from whoever runs your database.

How workers share one table#

Say you run ten workers. The obvious question is what stops all ten from charging at the first row in the queue. ulak answers it in two layers.

The first layer is arithmetic. Each worker is given a deterministic sequential id at startup, and its poll query carries one extra condition: the row's id modulo the worker count must equal the worker's id. One large table becomes ten disjoint logical slices, and in the normal case nobody eats from anyone else's plate.

The real world is not as clean as a formula. Change the worker count while the system is live and the slices shift. Let an operator lock a few rows by hand in a console and a worker could stall on them. So the arithmetic is only there to reduce contention. The second layer is the one that makes it safe: FOR UPDATE SKIP LOCKED. When a worker reaches a row that someone else holds, it skips it and moves to the next instead of waiting.

SELECT id, endpoint, payload
FROM   ulak.queue
WHERE  status = 'pending'
  AND  scheduled_at <= now()
  AND  id % 10 = 3        -- this worker's slice
ORDER  BY id
LIMIT  200
FOR UPDATE SKIP LOCKED;

There is a third decision underneath. ulak requires the workers to run at READ COMMITTED, PostgreSQL's default, and refuses REPEATABLE READ. The instinct is to reach for the stricter level. Under REPEATABLE READ, PostgreSQL uses snapshot isolation, and when several workers use SKIP LOCKED on overlapping rows, a worker that sees a row updated by someone else after its snapshot was taken gets serialization failure 40001. The workers would spend their lives aborting and retrying. At READ COMMITTED every statement sees the latest committed data: a locked row is skipped, an unlocked one is claimed, and there is no conflict to raise.

What "delivered" means#

Can ulak promise exactly-once delivery? No, and I want to be unambiguous about it, because it is the golden rule of distributed systems: nothing that crosses a network can guarantee exactly-once on its own.

Picture the worst case. ulak sends a message to your payment API. The API processes it, writes it to its own database, and the connection dies in the instant before it can say "got it". ulak never sees the acknowledgement, so it does the only thing it can and sends the message again. Your consumer therefore has to be idempotent. Receiving the same message twice must not corrupt its data. Correctness does not stop at the database, it extends to whoever consumes the message.

The same problem exists on the way in. Your code calls ulak.send, and the database connection drops just as the transaction is finishing. Your code, reasonably, decides the write failed and retries. A blind retry would enqueue the same order twice. The fix is an idempotency key that you supply:

SELECT ulak.send_with_options(
  'billing-webhook',
  jsonb_build_object('order_id', 17),
  idempotency_key => 'order:17:created'
);

ulak stores the MD5 of the key, not the payload, and enforces it with a partial unique index on ulak.queue. The word partial is doing real work. The queue will accumulate millions of delivered rows over time, and an index across all of them would grow enormous and slow every insert. So the index covers only rows whose status is pending or processing. A second send with the same key while the first is still active hits the index, ulak compares the hashes, ignores the new row, and returns the id of the existing message. No trip to Redis for a distributed lock. It is resolved where the data lives.

The trade-off that should bother you#

Here is the thing that bothered me most when I designed it, and it is documented in the 0.0.3 architecture notes so nobody discovers it in production.

Claiming a batch, making the network call, and updating the status happen inside one open PostgreSQL transaction. If the API you are delivering to takes two seconds to respond, the worker's transaction is open for two seconds. Open transactions hold locks and occupy a connection, and a system doing thousands of deliveries a second against a slow downstream can wedge itself quickly. Making HTTP requests from inside a database is playing with fire, and I am the one who lit it.

Two things contain it. The first is at the storage layer: worker transactions run with synchronous_commit = off. Normally PostgreSQL will not consider a transaction finished until the WAL record is flushed to disk, and the disk is the slowest thing after the network. With the flush deferred, a worker marks a message delivered and moves to the next one without waiting.

Skipping the flush would be reckless for business data. For the queue it is safe, and the reason is the at-least-once model from the previous section. If the server loses power in the millisecond after a worker marks a message delivered and that WAL record is lost, the row comes back as pending on restart, as though it had never been sent. A worker wakes up and sends it again. The consumer is idempotent, so the duplicate is harmless.

The second containment is the circuit breaker, which is the next section. Between them they bound how long a transaction can stay open behind a bad downstream. They do not remove the cost. Timeouts on your endpoints and a modest batch size are your side of the bargain.

One more cost sits in the same place. HTTP is built in, because libcurl is the only hard dependency. Kafka, MQTT, Redis Streams, AMQP and NATS each need their C client library compiled into the extension, which means optional external C dependencies on your database server.

When the other side is down#

If a downstream API dies outright, retrying every message against it at full speed wastes connections and buys nothing. ulak keeps a circuit breaker per endpoint, in memory, with three states. Closed means normal. When consecutive failures cross the threshold the breaker opens, and workers skip every row for that endpoint. Nothing is sent, the downstream gets a breather, and other endpoints keep flowing.

The breaker should not stay open forever. After a cooldown it moves to half-open, and one probe is allowed through. The subtle problem is which worker sends it. Fifty workers scanning the queue can all notice the half-open state at once, and without coordination they would all probe, which is the stampede the breaker exists to prevent.

ulak resolves this with a compare-and-swap on the breaker state. There is one microphone in the room. Fifty workers want to ask whether the other side is back, and the one that wins the swap is the only one that gets to speak. It sends the single probe. The rest see the microphone is taken and keep deferring their rows. If the probe gets a 200, the breaker closes and life resumes. If it fails, the breaker reopens and the cooldown starts again.

Not every failure deserves a retry. ulak classifies them first. A transient error, a timeout or a connection reset, is marked retryable and rescheduled with increasing backoff. A permanent error, an HTTP 400 because the payload failed the other side's validation, is not retried at all, because a million attempts would not change the answer. Both a permanent error and an exhausted retry budget move the message to ulak.dlq, the dead letter queue, where it waits for a person. And when an HTTP server answers with a Retry-After header, ulak throws away its own backoff calculation and waits exactly as long as it was asked to.

When the worker itself dies#

There is one more failure, and it is the one people forget. A worker claims a message, marks it processing, and then the process dies mid-request, from a bug or a hardware fault. The row is now processing forever. Other workers will not touch it, because as far as they can tell someone is working on it.

ulak gives worker 0 an extra duty alongside its normal batches: a periodic crash recovery pass that scans for rows stuck in processing past a timeout and returns them to pending. The living workers pick the row up and finish the job.

The bet#

Put the pieces together and this is what ulak is: the outbox pattern with PostgreSQL's own machinery doing the relay's job from inside the database. In return it asks you to accept one deliberate trade: network calls now run inside your database process, and the transaction stays open while they do.

For a decade the microservices consensus has been that databases are dumb storage and the logic belongs in external tools, in Kafka, in RabbitMQ, in a relay. That was taught as a rule. ulak is a bet that for a database-centric system the rule is backwards, and that the database, the one component that already knows exactly what has been committed, is the right thing to announce it.

Whether that bet is yours to make depends on whether you run your own PostgreSQL, whether your core already lives there, and whether you are willing to hand an old friend that much responsibility again. The code is at github.com/zeybek/ulak. If you run it and it breaks, open an issue, I would rather know.