Messaging problems and solutions you should understand before production

Messaging is one of those architectural tools that looks cleaner on a diagram than it feels in production. At first, the promise is obvious. A service publishes an event, another service consumes it, nobody waits, teams move independently, and the system becomes more resilient. It is a beautiful story until the first slow consumer appears, the first duplicate message does something twice, or the business asks why the user still sees "processing" after the checkout screen says "done".

I keep coming back to a simple rule: asynchronous communication is not just a different transport. It changes the product, the data model, the operational model, and the contract between teams. If you treat it as "HTTP, but through a broker", it will eventually make you pay.

Start with the meaning of the message

Before choosing RabbitMQ, Kafka, Azure Service Bus, SNS, SQS, or anything else, ask what the message means. An event says that something has already happened. "Order placed" is past tense. The publisher is not asking anyone for permission. It is telling the rest of the system that the world has changed.

A command asks someone to do something. "Create shipment" or "Send invoice" is an instruction. It has a target, even if you deliver it asynchronously. Pretending that every command is an event because events look more decoupled is one of the easiest ways to create a system that lies to you.

The distinction matters because it decides where domain knowledge leaks. If an order service publishes "OrderPlaced", consumers need to understand at least the concept of an order. If the order service sends "CreateShipment", the order service now knows there is a shipping capability and that this capability should react to an order. Neither direction is free. You are choosing where the coupling lives.

That choice should come from the business process, not from what is easiest to draw. If a shipment cannot exist without an order, an event may be fine. If the order process explicitly requires a shipment to be created as a mandatory next step, a command may be more honest. The broker will not answer this for you.

Do not make every event fat

There are two common styles of event payloads. A thin event, or event notification, carries the minimum needed to say what happened. It might include an order ID, an event type, a timestamp, and correlation metadata. Consumers that need more data must ask the owning service.

A fat event, or event-carried state transfer, carries enough state for consumers to build a local projection. This can be useful when consumers need availability, autonomy, or read models that should not call the source service on every request.

Both styles are valid. The trap is treating payload size as the only question.

If the message is meant to trigger behavior, a thin event is often cleaner. The event says "this happened", and the consumer decides what to do. If the message is meant to move data into a projection, a richer payload can make sense. The consumer is not asking for behavior. It is synchronizing useful state.

The problems start when every downstream team asks for "just one more field". This is especially tempting inside one organization where teams can browse each other's repositories. Someone sees an internal field, asks for it in the event, and suddenly a private detail becomes public API. The next refactor is no longer local.

That is how event contracts become accidental data dumps.

Sensitive data changes the architecture

Fat events get uncomfortable when the payload contains personal data, tenant-specific data, or business-sensitive information.

If you publish names, addresses, phone numbers, or contract details across a broker, you also spread the responsibility for retention, deletion, access control, and audits. The publisher may still be the "owner" of the data in a domain sense, but copies now exist elsewhere. That matters.

One useful technique is tokenization. Instead of publishing full personal data, the message carries a token. A downstream service exchanges the token for the data only when it has a real reason and the right permissions. This gives up some autonomy because the consumer may need a synchronous call, but it reduces the blast radius of sensitive data.

There is no universal answer here. Sometimes the right trade-off is a richer event. Sometimes the right trade-off is a token and a lookup. Sometimes the right move is to question whether the consumer should see the data at all.

The important bit is to make the trade-off explicit. "It was easier to put the whole object on the bus" is not an architectural decision.

Asynchronous UX is a product decision

The biggest surprise for many teams is not technical. It is behavioral. When a synchronous request becomes asynchronous, the user experience changes. You may return "202 accepted" instead of "201 created". You may have to show "we are processing your order" instead of "your order is complete". You may need a status page, email confirmation, websocket updates, polling, or a correlation ID that lets the frontend track an operation. That is not a backend implementation detail. It is the product.

If the business expects the user to know immediately whether stock was reserved, payment was accepted, and shipping was scheduled, then a long asynchronous chain may be the wrong shape for that part of the process. If the business can tolerate "we accepted your request and will notify you", asynchronous processing fits much better.

The funny part is that many real checkout flows already work this way. Payment gateways, email confirmations, fulfillment systems, and external integrations are rarely instant in the pure sense. The user just gets a clear enough story. That is the job: make the system behavior honest.

Slow consumers move the cost, they do not delete it

If a consumer calls an external API that takes 30 seconds, the system still has a 30-second operation. Putting a queue in front of it may protect the publisher, but it does not remove the latency. If traffic spikes, the queue grows. If retries stack up, the queue grows faster. If your broker performs poorly with long queues or large messages, the operational pain becomes very real.

Inbox and outbox patterns help, but they are not magic spells. An inbox lets a consumer persist incoming messages quickly, acknowledge the broker, and process work from its own storage. This can keep broker queues short and make recovery easier. The price is extra moving parts and often more latency.

An outbox lets a service store a business change and a message to publish in the same transaction. This avoids the classic failure where data is saved but the event is never published. The price is another worker, another table, and another place where timing matters.

These patterns are worth learning. They are also worth respecting. They improve reliability by moving the problem into a shape you can control. They do not make the underlying business process lighter.

Eventual consistency needs a vocabulary

"Eventually consistent" is easy to say and hard to explain to a product owner. Eventually when? After one second? Thirty seconds? Two hours? After the consumer is redeployed? After someone replays a failed message from last night?

If another team depends on your event stream, they will make assumptions about timing. If users rely on a status, they will make assumptions as well. Without a shared vocabulary, asynchronous systems drift into folklore. "It usually takes two seconds" becomes an unofficial SLA. Then one incident breaks that assumption and everyone is surprised.

A healthier approach is to name the states. Accepted. Processing. Waiting for external provider. Completed. Failed permanently. Needs manual review. These states are boring, which is exactly why they are useful. They let people talk about the process without pretending it is instant.

The same applies between services. If consumers need freshness guarantees, say so. If a projection can be stale for five minutes, say so. If stale data would cause financial or legal trouble, do not bury that risk inside a queue.

Idempotency belongs in the model first

Most production messaging systems deliver at least once. That means duplicates are normal. Not embarrassing. Normal.

A message can be published twice. A consumer can process it and crash before acknowledging it. A batch outbox publisher can send ten messages and fail before marking them as sent. The broker can redeliver. The result is the same: your handler may see the same logical message more than once.

Technical deduplication helps. You can store message IDs, use a deduplication window, and make "exactly once processing" more realistic, even though "exactly once delivery" is not something you should casually assume. But the best idempotency usually starts in the domain model.

If an order is already completed, should "complete order" do anything? If a refund already exists, should another refund be created? If a ticket is closed, can it go back to "processing", or should a new process begin? These questions are not about RabbitMQ or Kafka. They are about the business rules.

Technical deduplication is still useful, especially for noisy retries. Just do not let it replace domain thinking. A message id table with a 14-day retention window will not save you from a model that allows nonsense transitions.

DLQ is not a trash can

Dead letter queues are useful because some messages should stop retrying. They are dangerous because teams start treating them as "handled".

A message in DLQ means the system did not complete something it was asked to complete. Maybe that is acceptable. Maybe it is a transient provider outage and the message can be replayed. Maybe it is a malformed payload that will never work. Maybe it is a business invariant violation that reveals a missing process. Maybe it is a contract mismatch between teams. Those are different problems. They deserve different labels.

One practical improvement is to classify failures before they hit DLQ. Transient infrastructure errors can be retried with backoff and eventually replayed. Permanent validation errors should not spin forever. Business rule failures may need support tooling, a compensating process, or a product decision. Contract failures should alert the teams that own the producer and consumer.

Monitoring "DLQ has 10,000 messages" is better than nothing, but it is not enough. You need to know what kind of messages they are, why they failed, whether replay is safe, and who owns the next action. DLQ is the beginning of a conversation, not the end of one.

What to do differently next time

When designing a messaging flow, I would start with a few uncomfortable questions.

• What behavior must be immediate from the user's point of view?
• What can be accepted and completed later?
• Which service owns the source of truth?
• Which consumers need data, and which consumers need behavior?
• What data is too sensitive to broadcast?
• What stale data is acceptable?
• What happens when a message arrives twice?
• What happens when it arrives after the business state has moved on?
• Who watches the DLQ on Monday morning?

These questions feel slower than drawing a broker between boxes. They are faster than untangling six teams, three projections, one overloaded queue, and a DLQ nobody checked for a week.

Messaging is still worth using. It can decouple availability, absorb bursts, protect user-facing requests from slow dependencies, and let teams build local models that fit their needs. It just asks for a different kind of honesty.

The future skill is not memorizing more patterns. It is knowing what problem each pattern moves, what cost it adds, and who will notice when the trade-off fails.

Happy messaging!