Event-Driven Microservices: When to Break the Monolith (and When Not To)

Meet Priya. She’s a tech lead at a fintech startup that processes 50K requests per minute. Her team’s monolith works fine — 99.9% uptime, eight engineers shipping features, customers are happy. But she can see the ceiling approaching.

Deploy coordination takes 45 minutes. Every time the billing team touches a shared module, the notification team has to coordinate a release train. The ingestion pipeline needs ten times the compute of the dashboard, but they share the same process — so scaling one means scaling everything. And the two teams? They’re stepping on each other’s toes in the same codebase, producing merge conflicts that take hours to resolve.

Priya has heard the siren song of microservices. Every conference talk, every Hacker News thread, every vendor blog post tells her to break the monolith. But she’s smart enough to ask: is this actually the right move for my team, right now?

This is her story — and it might be yours too.

The Problem

Priya’s monolith wasn’t broken. That’s the uncomfortable truth you need to sit with. A modular monolith serving 50K RPM with 99.9% uptime is a good system. But good systems can still have a ceiling, and Priya could see hers approaching from three directions:

• Deploy coordination — A change to the billing module required coordinated deploys with the notifications module. One team couldn’t ship without the other’s sign-off. That 45-minute deploy window was the symptom, not the disease.
• Scaling asymmetry — The ingestion pipeline needed 10x the compute of the dashboard, but they shared the same process. Scaling the ingestion path meant spinning up entire new instances of the monolith, wasting 90% of the resources on code that didn’t need them.
• Team coupling — Two teams working in the same codebase meant constant merge conflicts, shared release trains, and a growing sense of friction. Every sprint felt like a negotiation.

Why this matters: These three problems — deploy coupling, scaling asymmetry, and team coupling — are the classic signals that a monolith is starting to constrain your growth. But they’re also the classic wrong reasons to migrate if you haven’t done the homework first.

The Investigation

Priya did something most teams skip: she spent three months measuring before writing a single line of new code. Here’s what the investigation revealed, in plain English.

Billing was coupled to four downstream services. Every time the billing module processed a payment, it directly called the notification service, the analytics service, the audit logger, and the customer dashboard. That’s four points of failure for a single transaction. If any one of those services was slow, billing slowed down too. What this means: billing had become a traffic cop for the entire system, and traffic cops are bottlenecks.

Notification was a hidden bottleneck. Every team depended on it. When the onboarding team wanted to send a welcome email, they called notification. When the security team wanted to alert on suspicious login, they called notification. When the billing team wanted to send an invoice receipt, they called notification. The notification module had become the most-coupled piece of the system without anyone realizing it. What this means: if notification goes down, half the features in your product stop working — and nobody owns it explicitly.

Ingestion had the most distinct scaling requirements. The ingestion pipeline was I/O-bound, processing webhooks and file uploads. The dashboard was CPU-bound, running aggregations and rendering charts. They lived in the same process, which meant the ingestion pipeline couldn’t scale independently, and the dashboard couldn’t optimize independently. What this means: you’re paying for resources one side doesn’t need, and neither side can tune its performance.

Key lesson: The investigation isn’t about finding if you have problems. It’s about understanding which problems are worth solving with a migration, and which ones you can solve with better modularization inside the monolith.

The Solution: Event-Driven Architecture

Priya chose event-driven over request-driven microservices for one reason: decoupling. With events, the billing service doesn’t call notification — it emits a billing.invoice.paid event, and notification picks it up when it’s ready. The billing service doesn’t need to know if notification is up, down, or on a coffee break.

Here’s the mental model: instead of services calling each other like a phone tree, they broadcast announcements over a loudspeaker. Any service that cares about the announcement can listen. Services that don’t care simply ignore it. This is the core idea behind event-driven architecture.

The Event Schema

Every event needs a standard shape so producers and consumers can agree on what’s being communicated. Here’s the schema Priya’s team settled on:

{
  "id": "evt_01J2XYZ...",
  "type": "billing.invoice.paid",
  "source": "billing-service",
  "time": "2026-05-20T14:30:00Z",
  "data": {
    "invoice_id": "inv_20260520_001",
    "customer_id": "cus_abc123",
    "amount": 49900,
    "currency": "USD"
  },
  "specversion": "1.0"
}

They used the CloudEvents spec for interoperability. Here’s what each field does: id is a unique identifier so consumers can detect duplicates. type tells consumers what happened. source says which service produced it. data carries the payload. And specversion ensures that as the spec evolves, everyone knows which version they’re working with.

Key Implementation Details

Idempotent Consumers

The hardest lesson Priya’s team learned: Kafka guarantees at-least-once delivery. That means you will get duplicate events. It’s not a bug — it’s a fundamental property of distributed systems. When a consumer processes an event and crashes before committing the offset, Kafka will redeliver that event.

The fix is idempotency: make your consumers safe to run twice with the same input.

class IdempotentConsumer:
    def __init__(self):
        self.processed = RedisSet("processed-events", ttl=86400)

    async def process(self, event: CloudEvent):
        if await self.processed.contains(event.id):
            return
        await self.handle(event)
        await self.processed.add(event.id)

Here’s what each piece does: The processed set stores event IDs in Redis with a 24-hour TTL. Before processing any event, the consumer checks if it’s already seen that ID. If yes, it skips it. If no, it processes the event and records the ID. This way, even if Kafka delivers the same event twice, your system only acts on it once.

Production pitfall: The TTL on your dedup store matters. Set it too short, and a delayed duplicate (hours later) will slip through. Set it too long, and you’ll burn memory on stale IDs. Priya’s team settled on 24 hours — long enough to cover any realistic redelivery window, short enough to keep Redis healthy.

Dead Letter Queue

Not all events can be processed. Maybe the data is malformed. Maybe a downstream service is down. Maybe there’s a bug in the consumer. Whatever the reason, you need a way to handle failures without losing events or getting stuck in an infinite retry loop.

class DeadLetterQueue:
    def __init__(self, topic: str, max_retries: int = 3):
        self.dlq_topic = f"{topic}.dlq"
        self.retry_topic = f"{topic}.retry"
        self.max_retries = max_retries

    async def handle_failure(self, event: CloudEvent, error: Exception):
        retry_count = event.get("retry_count", 0)
        if retry_count < self.max_retries:
            event["retry_count"] = retry_count + 1
            await self.producer.send(self.retry_topic, event)
        else:
            await self.producer.send(self.dlq_topic, event)

Here’s what each piece does: When an event fails, the DLQ checks how many times it’s been retried. If under the limit (3 by default), it increments the counter and sends the event to a retry topic for another attempt. If it’s exhausted all retries, the event goes to the dead letter topic — a parking lot for events that need human investigation. This prevents a single bad event from blocking your entire pipeline.

The Results

Here’s what Priya’s team saw six months after the migration:

Metric	Before (Monolith)	After (Event-Driven)
Deploy time	45 min (coordinated)	8 min (independent)
P95 latency	320ms	280ms
Team throughput	3 features/sprint	7 features/sprint
Incidents/month	4	2
Infrastructure cost	$12K/mo	$15K/mo

What this means for you: The headline numbers are good — faster deploys, lower latency, more features, fewer incidents. But notice that infrastructure cost went up. That’s the hidden tax of microservices: you’re trading compute efficiency for organizational efficiency. For Priya’s team, the trade was worth it. For your team, it might not be. Always look at the cost column.

What Went Wrong

Priya’s migration wasn’t all smooth sailing. Here are the three things that went wrong — and what to watch out for so they don’t happen to you.

1. Event schema evolution

The billing team added a field to the event payload. The notification service, which was running an older version of the consumer, crashed. It couldn’t parse the new field, and the deserialization failed.

What to watch out for: Event schemas are contracts. When you change a contract without versioning, something breaks. The fix is to treat your event schema like you treat your API schema — with versioning, compatibility checks, and a registry.

Fix: Priya’s team adopted Avro with Schema Registry, enforcing backward compatibility. New fields must be optional or have defaults. Breaking changes require a new event type. The Schema Registry rejects incompatible schemas at publish time, so the crash never reaches production.

Production pitfall: Schema registries catch incompatibilities, but they don’t solve the rollout problem. When you add a new event type, make sure all consumers are deployed and ready before the first event is published. Otherwise, you’ll have events in the topic that nobody can process.

2. Observability debt

In the monolith, a single trace covered the entire request. Priya could open her APM tool, search for a request ID, and see every database query, every function call, every millisecond. With events, that trace was shattered across six services, three Kafka topics, and two message queues.

What to watch out for: Distributed systems are inherently harder to debug. When a customer reports an issue, you need to trace it across service boundaries. If you don’t have distributed tracing set up before the migration, you’re flying blind.

Fix: Priya’s team added OpenTelemetry instrumentation with trace context propagation through Kafka message headers. Every event carries a trace ID, and every consumer propagates it. Now a single request that spans six services shows up as one trace in their observability platform.

3. Testing complexity

Testing an event flow requires running Kafka, the producer service, and the consumer service. A unit test that used to take 50 milliseconds now takes 30 seconds and requires Docker. The feedback loop slows down, and developers start skipping tests.

What to watch out for: If testing becomes painful, your team will stop doing it. The solution isn’t to tell your team to “test harder” — it’s to make testing easy again.

Fix: Priya’s team built a test harness that uses an in-memory event bus for unit tests and reserved real Kafka for integration tests. Unit tests run in milliseconds and don’t require Docker. Integration tests run against a real Kafka instance in CI, but they’re reserved for critical flows. The rule: if you can test it with the in-memory bus, do. If you need to verify exactly-once semantics or retry behavior, use real Kafka.

Key lesson: The testing fix isn’t about technology — it’s about making the right thing the easy thing. If your test harness is slower than the monolith’s tests, developers will find ways around it. Invest in test infrastructure early.

When NOT to Break the Monolith

Priya’s story had a happy ending, but not every team’s will. If you’re considering a similar migration, ask yourself these three questions first. If the answer to any of them is “no,” pause.

1. Is the monolith actually the bottleneck? — If your deploys take 10 minutes and your team is 4 people, don’t migrate. You don’t have a scaling problem — you have a process problem. Fix your CI pipeline, adopt trunk-based development, and invest in modular boundaries. Microservices will only add complexity without giving you proportional benefit.

2. Can you modularize in-place? — Strict module boundaries, shared-nothing patterns, and clear interfaces can get you 80% of the benefit without the operational cost. Extract billing into a well-defined module with a clean API. Put a queue between ingestion and the dashboard. You might find that the monolith was never the problem — the lack of boundaries was.

3. Do you have observability? — If you can’t trace a request through your monolith, you definitely can’t trace it through 12 microservices. Before you break anything apart, invest in distributed tracing, structured logging, and centralized metrics. The migration will be hard enough without debugging in the dark.

Why this matters: The decision to break the monolith isn’t a technical decision — it’s an organizational one. Microservices solve team-scale problems, not code-scale problems. If you have one team of four people, a monolith is probably the right choice. If you have three teams of six, microservices might be. Know which camp you’re in before you start.