Designing event-driven systems for failure

Event-driven architecture can reduce coupling and unlock independent delivery. It can also move complexity into places that are harder to see. The difference is rarely the message broker. It is how deliberately the system handles failure.

Start with explicit boundaries

An event is a statement that something meaningful has already happened. Good events follow business boundaries: an order was accepted, a payment was authorized, or inventory was reserved. They should not expose another service's internal data model.

Before choosing Kafka topics or serialization formats, define:

• who owns each state transition;
• which facts other domains actually need;
• what happens when a consumer is unavailable;
• how long events need to remain useful.

This work often reveals that fewer, better-defined events create a more adaptable system than broadcasting every database change.

Design for at-least-once delivery

Most production messaging systems make at-least-once delivery the practical baseline. A consumer must assume it can receive the same event more than once.

Idempotency belongs in the business operation, not only in transport code. A common implementation records the message identifier in the same transaction as the state change:

public async Task Handle(OrderAccepted message, CancellationToken ct)
{
    if (await inbox.Contains(message.Id, ct)) return;
 
    await inventory.Reserve(message.OrderId, message.Items, ct);
    await inbox.MarkProcessed(message.Id, ct);
    await unitOfWork.Commit(ct);
}

The exact mechanism varies, but the invariant is stable: replaying a message must not corrupt the business state.

Make failure observable

A distributed workflow is not one trace unless the system makes it one. Propagate trace context through message headers and emit domain-level telemetry—not just broker metrics.

Useful signals include:

• consumer lag by workflow importance;
• processing latency and retry count;
• dead-letter volume with a reason;
• the age of the oldest unprocessed business event;
• business outcomes such as orders stuck in a pending state.

Technical health and business health belong on the same operational picture.

Treat recovery as a product feature

Retries need limits, backoff, and a clear terminal state. A dead-letter queue without a replay procedure is only delayed data loss. Document how an operator investigates, corrects, and safely reprocesses a failed message.

Reliability is not the absence of failure. It is the ability to understand failure and recover without creating a second incident.

Choose complexity deliberately

Event-driven architecture is valuable when teams need independent change, asynchronous throughput, or durable integration. A synchronous call or modular monolith can be the stronger choice when those needs are absent.

The goal is not to maximize distribution. It is to place complexity where it creates durable business value—and make that complexity operable.