It was one of those post-release mysteries.
A customer’s order had gone through — payment captured, inventory updated, database neatly committed. Everything looked perfect… until downstream systems started acting like the order never existed.
No confirmation email.
No shipping trigger.
No audit trail.
We checked logs, metrics, dashboards. Kafka showed no trace of the OrderPlaced event. But the DB had the order entry, timestamp and all. Somehow, the message never made it out.
There was no error, no stack trace. Just… silence.
And that silence cost us downstream consistency, delayed customer notification, and a late-night incident call.
That was the moment I realized something fundamental:
In distributed systems, writing to a database and publishing a message are two separate commitments.
But they must be treated as one — because if one succeeds and the other fails, then to the outside world, your system isn’t just inconsistent… it’s blind.
And once I knew I was blind, I went looking for vision.
The pattern that gave it to me — was Outbox.
The Dual Write Problem: Two Systems, One Illusion
It’s a deceptively simple sequence:
saveToDatabase(order);
publishToKafka(orderEvent);
To a developer, it looks atomic.
To a distributed system, it’s two entirely separate operations — each with its own failure modes, latencies, and rollback behaviors.
The database might commit successfully.
The Kafka publish might time out, fail silently, or get retried in a weird state.
Or vice versa — the message gets published, but the DB write fails, leaving downstream systems reacting to something that doesn’t exist.
This is the dual write problem:
You try to update the world in two places, but have no shared transactional boundary.
What Goes Wrong
DB succeeds, Kafka fails → Downstream never sees the event
Kafka succeeds, DB fails → Consumers process an event for something that never existed
Both succeed, but logs don’t reflect it → Support nightmares during debugging
And retries?
Without coordination, they often do more harm than good — reprocessing the same data and creating duplicate side effects like billing charges, emails, or resource allocation.
The illusion of safety in try { db; kafka; } catch (e) { retry(); } is exactly that — an illusion.
The Outbox Pattern: Persist the Intent, Publish Later
The Outbox Pattern flips the problem on its head.
Instead of trying to write to the database and publish to Kafka in one go, it says:
Write to the database — and also persist your intent to publish, in the same transaction.
That “intent” goes into a dedicated outbox table.
The actual message publishing happens later, asynchronously, through a reliable relay process.
This decouples the fragile dual-write step into:
A single, atomic local transaction (data + event recorded together)
A separate, recoverable publisher that processes the outbox table and pushes events to Kafka (or RabbitMQ, or anywhere else)
What Goes Into the Outbox Table?
A typical outbox row looks like:
| id | event_type | payload (JSON) | status | created_at |
| 1 | OrderPlaced | {...} | pending | 2025-05-21 10:05 |
You insert this row in the same DB transaction that saved the order itself.
So if the transaction commits, both the order and the message are guaranteed to be persisted.
Then Comes the Relay
The relay is a background process (or a CDC tool) that:
Reads unsent rows from the outbox table
Publishes them to the message broker
Marks the row as “sent” or deletes it
This ensures:
Kafka publish failures don't affect the DB write
The system can retry until success
No messages are lost or published multiple times — if the broker is idempotent or the consumer is
How It Works: Atomic Writes, Eventual Publishes
The power of the Outbox Pattern lies in how cleanly it separates responsibility — without breaking consistency.
Step 1: Save Everything in One Transaction
@Transactional
public void placeOrder(Order order) {
orderRepository.save(order);
OutboxEvent event = new OutboxEvent(
UUID.randomUUID(),
"OrderPlaced",
serialize(order),
LocalDateTime.now(),
"pending"
);
outboxRepository.save(event);
}
Step 2: Let a Relay Publish the Events
A relay process (either a background thread, a separate service, or a Debezium CDC connector) will:
Poll the outbox_events table for new/pending events
Push them to the message broker
Update the event’s status (e.g., sent, failed) or delete it
This ensures:
Kafka publish failures don't break DB writes
Eventual publish happens safely
You own the retry and visibility lifecycle
When and Why It’s Needed: Don’t Overengineer. Do It Where It Hurts.
The Outbox Pattern isn’t for everything.
But if the delivery of your message is as important as the data you just saved, you don’t want to bet on try-catch blocks and hope.
When You Absolutely Need It
✅ Mission-critical events
- Billing, audit logs, compliance actions
✅ Customer-facing side effects
- Emails, SMS, notifications triggered by business actions
✅ Order/state workflows
- OrderPlaced → ShippingScheduled → InvoiceGenerated
✅ High-scale, distributed event-driven systems
- Microservices where multiple consumers depend on a single event
What Outbox Gives You
✅ Atomicity of data + intent
✅ Durable delivery path, even if broker or service goes down
✅ Eventual consistency without needing distributed transactions
✅ Retry without fear — you own the retry lifecycle
The Outbox Pattern sounds bulletproof in theory — but like any distributed pattern, it has caveats. These aren’t gotchas if you plan for them. But if you don’t, they’ll catch you off guard in prod.
Relay Reliability and Idempotency
Problem: What happens if your relay crashes or restarts mid-publish?
Risk: Duplicate publishes or lost offset tracking.
🛠 Solution:
Make your publisher idempotent (e.g., use event IDs for deduplication on the consumer side)
Keep relay logic stateless and retry-safe
Use transactional producers if your broker supports it (Kafka supports exactly-once delivery with some complexity)
Outbox Table Growth
Problem: Outbox table fills up fast, especially at scale.
Risk: Disk usage, slower polling, worse DB performance.
🛠 Solution:
Implement archival or purging logic for rows marked as sent
Consider moving old events to a cold-storage table
Use time-based partitioning for large-scale deployments
Schema Evolution
Problem: Your payload is stored as JSON (or blob) in the DB.
Risk: Schema changes (e.g., field renamed or removed) may break consumers down the line.
🛠 Solution:
Use versioned payloads or event envelopes
Define event schemas and keep a registry (even informal)
Don’t assume consumers can “guess” your structure
Ordering Guarantees
Problem: The relay may not preserve the order in which events were inserted.
Risk: Consumers process OrderShipped before OrderPlaced.
🛠 Solution:
If order matters, enforce ordering in the relay by created_at
Or assign a logical sequence number in your domain events
Kafka partitioning can help, but only if coordinated correctly
Error Recovery Logic
Problem: What happens when publishing fails repeatedly (e.g., Kafka down for hours)?
Risk: Infinite retries, stuck queue, missed alerts.
🛠 Solution:
Mark rows as failed after N attempts and move them to a dead letter table
Add alerting and monitoring for failed outbox publish attempts
Use backoff strategies (e.g., exponential or linear retry spacing)
Wrap-Up: Persist Intent, Not Just Data
In distributed systems, the real danger isn’t failure — it’s false confidence.
When we write to a DB and push to Kafka in a single method call, we feel like we’ve done our job. But if those two operations can’t fail together, then we haven’t written an atomic operation — we’ve written a gamble.
The Outbox Pattern fixes that — not with magic, but with discipline:
You don’t publish events directly.
You persist intent.
You decouple the blast from the trigger — so recovery becomes a process, not a panic.
Whether you’re building a billing engine, an audit trail, or an orchestration service, eventual consistency doesn’t mean eventual chaos.
It just means you engineer for failure — and recover like you planned it.
🔁 Up Next: When Outbox Didn’t Fit — So I Borrowed Its Mindset
In Part 2, I’ll show you a situation where the Outbox Pattern didn’t apply directly — but its philosophy saved the day.
It wasn’t about publishing messages. It was about consuming them safely.
And it all started with a simple requirement:
“Make sure the billing report email goes out — once, and only once.”
