NestJS and Kafka: Boundaries and Retries

Where this came from

Most of my production broker work has used RabbitMQ. I wrote this note while comparing those lessons with a smaller public NestJS/Kafka configuration project. The transport changes, but ownership, idempotency, retries, and observability remain the hard parts.

My short checklist

- Choose service boundaries from ownership and failure modes.
- Treat messages as public contracts with versioning pressure.
- Keep NestJS transport adapters thin and test business logic without a broker.
- Design retries, dead-letter behavior, and observability before traffic arrives.

Kafka, RabbitMQ, HTTP, and gRPC are implementation details compared with the harder question: where should the boundary exist? A microservice boundary should match ownership, data responsibility, deployment risk, or a clear scaling need. If the boundary only mirrors a folder name, it adds network complexity without buying independence.

Start with ownership

Before introducing a message broker, I define what the service owns. Does it own the database table? Does it own the business rule? Can another team change it safely? What happens if it is down? These questions expose whether the service is a real boundary or just a remote function call.

Treat messages as contracts

A Kafka topic or queue message is a public API. Consumers may deploy at different times, retry old messages, or receive events produced by a previous version. Message names, required fields, optional fields, and versioning strategy matter. I prefer explicit event types and payloads that describe facts, not commands pretending to be facts.

Good event: OrderPaid, TicketAssigned, ProfileUpdated.
Risky event: DoEverythingForCheckout.
Required behavior: idempotent consumers and safe retries.
Required visibility: logs with message id, consumer, attempt, and outcome.

Keep NestJS modules honest

NestJS makes it easy to build clean modules, but module boundaries are only useful if dependencies stay directional. A billing module importing support internals is a design smell. In a microservice setup, I keep transport adapters thin and put business rules in services that can be tested without a broker.

Plan for failure before traffic

Distributed systems fail in ordinary ways: duplicate messages, late messages, missing dependencies, partial writes, schema drift, and poison payloads. A scalable system needs retry limits, dead-letter handling, backoff, health checks, and dashboards before it needs clever abstractions.

Before splitting a module, I want to know who will deploy it, who will investigate failed deliveries, and how the caller behaves while it is unavailable. If those answers are unclear, keeping the module in the existing service is often simpler.

Illustrative transaction contract: a unique inbox key and the business write commit atomically. A failure rolls both back; concurrent duplicate deliveries do not apply the invoice twice. External API calls require an outbox and provider idempotency.

// Pseudocode: both repositories must use the SAME database transaction.
// inbox.event_id has a UNIQUE constraint; insertion uses
// INSERT ... ON CONFLICT DO NOTHING RETURNING event_id.
async function handleInvoiceCreated(event: InvoiceCreatedEvent) {
  await db.transaction(async (tx) => {
    const inserted = await inbox.insertIfAbsent(tx, event.id);
    if (!inserted) return;
    await billing.applyInvoice(tx, event);
    // Persist any external side effects in a transactional outbox here.
  });
  // Acknowledge / commit the broker offset only after the transaction commits.
}