Microservices that don't fall over

Splitting a system into services is the easy part. The hard part is everything that happens in the gaps between them — the calls that hang, the caches that lie, the events that arrive twice. Here is what I have learned keeping those gaps closed.

The only good reason to split

Microservices are not a goal. They are a trade: you give up the simplicity of one deployable and one database, and in exchange you buy the ability to change, scale and fail in parts.

That trade is worth making when one of these is true:

  • Teams are colliding. Four teams in one codebase means release trains, merge queues and a bug in checkout blocking a copy change on the homepage.
  • One part has a different shape. Image processing needs CPU and can be slow; the API needs to answer in 50 ms. Mashed together, you size everything for the worst case.
  • One part must not take the rest down. A report that scans a year of data should never exhaust the connection pool that logins depend on.
Not a reason "It is the modern way." A well-structured monolith with clear internal modules beats a distributed system with unclear ones, every time. Distribution adds network calls, partial failure and eventual consistency to every interaction that crosses a boundary. You should be getting something concrete back for that.

My rule: start modular, split when a seam actually hurts. You will know — it shows up as a deployment you are afraid of, or a slow query that takes down something unrelated.

Drawing the boundaries

The most expensive mistake in this whole field is a boundary in the wrong place. Services split along the wrong lines do not reduce coupling; they convert a method call you could refactor in an afternoon into a network contract you cannot change without a migration plan.

Split by business capability, not by layer

A "data service", an "API service" and a "UI service" are three layers of one thing. Every feature touches all three, so every feature needs three deploys in order. That is a distributed monolith: all the pain, none of the benefit.

Split by what the business does: Orders, Payments, Shipping, Identity. Each owns its data, its rules and its vocabulary.

The test I use

For a candidate boundary, ask: can this service answer its most common question without calling anyone else? If "show me an order" needs three synchronous calls to assemble a response, the boundary is wrong, or the data needs to be copied.

The same word means different things A "customer" in Billing is a payment method and a tax status. In Shipping it is an address and a delivery window. In Support it is a contact history. Trying to build one shared Customer model that serves all three gives you a table nobody owns and everybody blocks on. Let each service keep the slice it needs, keyed by a shared id.

How services talk: calls vs events

Two services can interact in two fundamentally different ways, and choosing wrongly is the root of most "microservices are slow and fragile" complaints.

Synchronous call Asynchronous event
"Tell me the answer now" "This happened; react if you care"
Caller waits, caller is coupled to callee being up Publisher does not know or care who listens
Failure is immediate and visible Failure is delayed and must be designed for
Good for: reads the user is waiting on Good for: side effects, fan-out, slow work

Synchronous chains are where latency goes to die

Latency adds up, and availability multiplies down. If an API calls three services and each is up 99.9% of the time and answers in 80 ms, your endpoint is up about 99.7% of the time and takes 240 ms before it has done any work of its own. Add a retry and you have doubled the worst case.

// The shape to avoid: a chain the user waits on.
GET /order/42
  → Orders            (30ms)
      → Customers     (60ms)
          → Billing   (90ms)   // four hops, one slow link stalls everything
              → Tax   (120ms)

The fix is usually not "make each hop faster". It is to stop making the hop at request time: either the data is already here (copied in by an event), or the work is not needed to answer (pushed out to a queue).

When to reach for each

  • The user is waiting and needs the answer to be right now — synchronous call, with a timeout you actually chose.
  • Something else should happen because of this — publish an event. Sending the confirmation email is not the order service's job.
  • The work is slow, bursty, or allowed to be late — queue it. Thumbnails, exports, reindexing, invoices.

Event-driven, properly

"We will just publish events" is where many teams get into trouble, because a message bus moves the hard problems rather than removing them. Four things decide whether an event-driven system is calm or a nightmare.

1. The dual-write problem, and the outbox

Here is a bug that will happen to you:

// Looks fine. Is not fine.
await _db.SaveChangesAsync();        // order committed
await _bus.PublishAsync(orderPlaced); // process dies here

The order exists; the event never shipped. Nothing downstream knows. Swap the order of the two lines and you get the opposite bug: an event about an order that was rolled back.

You cannot make a database commit and a broker publish atomic. So do not try. Write the event into the same database transaction, in an outbox table, and let a separate process publish it.

using var tx = await _db.Database.BeginTransactionAsync();

_db.Orders.Add(order);
_db.Outbox.Add(new OutboxMessage {
    Type    = "order.placed",
    Payload = JsonSerializer.Serialize(evt),
    Status  = OutboxStatus.Pending
});

await _db.SaveChangesAsync();   // both rows, or neither
await tx.CommitAsync();

// A background worker reads Pending rows, publishes, marks Sent.
// Crash anywhere and it retries. At-least-once, guaranteed.
ONE DATABASE TRANSACTION orders the new row outbox the event both rows commit, or neither does relay reads pending broker consumer mark Sent — crash before this and it simply retries
The database is the only thing that has to be right. Publishing becomes a retryable step rather than a second system that can disagree.
What this buys you The database is the single source of truth about whether something happened. The broker becomes a delivery mechanism rather than a second system that can disagree with your data.

2. Everything is delivered at least once

Exactly-once delivery is not a thing you can buy. Brokers retry, consumers time out after doing the work, networks drop acknowledgements. Plan for duplicates instead of hoping.

The cure is idempotency: processing the same message twice has the same effect as once. Usually that means recording what you have already handled.

public async Task Handle(OrderPlaced evt)
{
    // The id comes from the message, not from us.
    if (await _db.Processed.AnyAsync(p => p.MessageId == evt.MessageId))
        return;

    await _invoices.CreateAsync(evt);
    _db.Processed.Add(new(evt.MessageId));
    await _db.SaveChangesAsync();  // effect + marker, one transaction
}

Note that the marker and the effect commit together. Writing the marker first gives you a message that is recorded as done but never did anything.

The other half: the inbox

The outbox makes sure an event leaves exactly in step with the data. The inbox is its mirror on the receiving side: it makes sure an event that arrives more than once is only acted on once. Together they are what people mean by "transactional messaging".

broker seen this id? select from inbox yes drop it no ONE DATABASE TRANSACTION do the work record the id
The effect and the "already handled" marker commit together. Record the id first and a crash leaves a message marked done that never did anything.

Two details decide whether an inbox works:

  • The id must come from the producer, not from the consumer. A id generated on receipt is different on every redelivery, which is exactly the case you are trying to catch.
  • Prune it. An inbox table grows forever otherwise. Keep a window comfortably longer than your broker's maximum retry age and delete beyond it.
When you can skip the inbox If the operation is naturally idempotent — setting a status to Shipped, upserting by a natural key — a duplicate changes nothing and you do not need the table. Reach for an inbox when the work accumulates: charging a card, incrementing a counter, sending an email.

3. Ordering is weaker than you think

Most brokers guarantee order only within a partition or queue. Two events about the same order can arrive out of sequence if they took different paths or one was retried.

  • Partition by entity. Key messages by order id so everything about one order goes through the same partition in sequence.
  • Carry a version. Include a sequence number and let consumers ignore anything older than what they have already applied.
  • Prefer events that are not order-sensitive. "Order total is now 40" survives reordering. "Add 10 to the total" does not.

4. Where failed messages go

A consumer that fails and requeues the same message immediately will do it forever, at whatever rate the broker allows, usually while writing a log line each time. The pattern that works:

  • Retry with backoff and jitter — a few attempts, spaced out, randomised so a thousand consumers do not all retry on the same second.
  • Then stop. After a bounded number of attempts, move the message to a dead-letter queue with the error attached.
  • Make dead letters visible and replayable. A dead-letter queue nobody looks at is just a slower way of dropping data.
Distinguish "the message is bad" from "the world is bad" A malformed payload will never succeed — dead-letter it on the first try. A 503 from a provider will succeed later — retry it. Treating these the same either burns your retry budget on hopeless messages or discards good ones during an outage.

Data: the part that hurts

Each service owning its own database is what makes independent deployment possible. It is also what makes everything else difficult, because you have given up joins and transactions across the system.

Eventual consistency, stated honestly

Once data is copied between services, there is a window where they disagree. That window is a product decision, not just a technical one. "Your order shows as placed but the invoice appears in a few seconds" is usually fine. "Your payment succeeded but your balance has not moved" usually is not.

No distributed transactions

A two-phase commit across services is a lock held over a network, which means one slow participant stalls everyone and a coordinator crash leaves locks behind. Nobody wants that in a request path. What replaces it is the saga, which is big enough to deserve its own section.

CQRS, only where it earns its place

If a screen needs data from four services, build a read model: a denormalised projection, kept up to date by events, that answers that screen in one query. This is a genuine fix for the synchronous chain problem.

It is also a second copy of your data with its own staleness and rebuild story. Add it when a read is demonstrably too slow or too chatty — not by default.

The saga pattern, in detail

A saga is how you run a workflow across services when you cannot have one transaction over all of them. Instead of one atomic unit, you get a sequence of local transactions, each with a compensating action that undoes its effect if a later step fails.

Placing an order touches three services:

1. Reserve stock      compensate: release the reservation
2. Charge the card    compensate: refund
3. Book the shipment  compensate: cancel the booking

If step 3 fails, the saga walks backwards: refund the card, release the stock. The system ends up consistent — just not instantaneously, and not by rolling anything back.

Compensation is not rollback A rollback erases history. A refund is a new fact that appears on a statement next to the charge. An email already sent cannot be unsent; the compensation is a second email. Model the business process to be honest about this, because the customer will see both halves.

Two ways to run one

Sagas come in two shapes, and the choice matters more than the name.

Choreography: everyone listens

Each service reacts to the previous service's event. There is no coordinator.

Orders order.placed Payments payment.taken Shipping shipment.failed shipment.failed — refund payment.refunded — release stock
Nothing is in charge. Each service knows only its own step and what to undo.

Good: no single point of failure, services stay decoupled, adding a listener needs no change upstream.

Bad: the workflow exists nowhere. To answer "why is this order stuck?" you read four codebases and correlate logs. Past three or four steps this becomes genuinely hard, and cyclic event chains are easy to create by accident.

Orchestration: one service holds the plan

A coordinator tells each service what to do and reacts to the reply.

Order saga state: awaiting payment Inventory reserve / release Payments charge / refund Shipping book / cancel command command command
The saga is a state machine in one place, with a row per run you can query.

Good: the workflow is one readable thing. You can query "which sagas are stuck in awaiting-payment?" and get an answer from a table. Compensation order is explicit.

Bad: the orchestrator is a dependency for everyone and can drift into holding business logic that belongs in the services.

What I reach for Choreography for two or three steps where the chain is obvious. Orchestration the moment there is a step that can fail in several ways, a timeout that matters, or anyone asks for a status screen. The cost of the orchestrator is paid back the first time you have to explain a stuck order.

The part people miss: no isolation

A saga gives up the I in ACID. Between step 1 and step 3, the system is in a state no single transaction would ever expose: stock reserved, card not yet charged. Other requests can see it.

Three ways to handle that:

  • Semantic lock. Mark the record as in-flight — an order in Pending, stock in Reserved rather than Available. Other work treats those states as "not yours to touch".
  • Pick a pivot. Order the steps so everything before a chosen point is undoable and everything after is guaranteed to succeed or retry forever. Charging the card is a natural pivot: do reversible things first.
  • Make it commutative. Where it fits, design steps whose order does not change the outcome, so a late compensation cannot corrupt a later update.

Failures a saga must survive

What happens What has to be true
A step is retried Every step is idempotent — an inbox, or a naturally repeatable operation
A compensation fails It retries forever and alerts. Never let it give up silently and leave money reserved
A reply never arrives Every step has a timeout that moves the saga on, usually to compensation
The orchestrator restarts State is in the database, not in memory, and resumes from the last recorded step
A compensation arrives late It is keyed to the saga id and step, so it cannot undo a newer, unrelated change

In practice the saga state is a row like this, and the whole pattern becomes a lot less mysterious once you see it as one:

public class OrderSaga
{
    public Guid        Id            { get; set; }
    public SagaStep    Current       { get; set; }  // Reserving, Charging, Booking, Done, Compensating
    public DateTime    StepDeadline  { get; set; }  // a sweeper moves on anything past this
    public string      FailureReason { get; set; }
    public List<string> Completed     { get; set; }  // what to undo, newest first
}
Before you build one A saga is the right answer to "these steps are in different services and all of them must happen". It is the wrong answer to "I did not want to put these two tables in the same service". If the steps always run together and always belong to one business decision, the cheaper fix is usually one service and one transaction.

Caching, and how it fails

Caching is the highest-leverage performance work available, and the easiest to get subtly wrong. A cache is a second copy of the truth, which means it can be wrong, and being wrong quietly is worse than being slow loudly.

The patterns, and when each fits

Pattern How it works Use when
Cache-aside App checks cache, misses, loads from DB, writes to cache The default. Simple, and the cache going down degrades to slow rather than broken.
Read-through Cache itself loads on a miss You want the loading logic in one place and your cache supports it.
Write-through Writes go to cache and DB together Reads immediately after writes must be correct.
Write-behind Write to cache, flush to DB later Very write-heavy and you can genuinely afford to lose the tail. Rare.

Invalidation: prefer short TTLs to clever rules

Every cached item should have a TTL, even ones you also invalidate explicitly. The TTL is your backstop for the invalidation you forgot to write, the deploy that bypassed your code path, and the manual database fix someone made at 2am.

// Expiry is not optional. "Until something clears it" is not a policy.
await _cache.SetStringAsync(key, json, new() {
    AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(10),
    SlidingExpiration               = TimeSpan.FromMinutes(2)
});

Where correctness matters more than hit rate, include a version in the key — user:42:v7. Bumping the version makes every old entry unreachable instantly, with no deletes to coordinate, and the old entries fall out on their own TTL.

The stampede, and the three fixes

A popular key expires. Two thousand requests miss simultaneously. Two thousand identical queries hit the database, which falls over, so nothing repopulates the cache, so the next two thousand do the same thing. This is the failure that turns a cache from a speed-up into an outage amplifier.

  • Lock the rebuild. First miss takes a short-lived distributed lock and loads; everyone else waits briefly or serves the stale value.
  • Refresh ahead. At 80% of the TTL, one request refreshes in the background while the rest keep being served the still-valid entry.
  • Jitter the TTL. Ten minutes plus up to sixty random seconds, so a batch of keys written together does not all expire together.
Cheap and effective Jitter costs one line and removes an entire class of synchronised-expiry incidents. If you take one thing from this section, take that.

Do not cache what you cannot afford to be stale

Permissions, balances, stock levels during checkout. Either do not cache them, or cache for seconds with an explicit invalidation on write. A stale permission check is a security bug wearing a performance costume.

Measure the hit rate or you are guessing

A cache with a 30% hit rate is usually adding a network hop and a consistency problem in exchange for very little. Instrument hits and misses per key family. The numbers are routinely surprising — the key you were sure was hot often is not.

Designing for the bad day

In a monolith, a slow function is slow. In a distributed system, a slow service becomes a queue of waiting threads in every service that calls it, and the failure spreads outward. Most big outages are this, not a crash.

Timeouts

Every network call needs a timeout you chose deliberately. The default is often infinite, or 100 seconds, which is infinite as far as a user is concerned. Budget them: if your endpoint must answer in 2 s and makes two calls, those calls do not get 2 s each.

Retries, carefully

Retries fix transient faults and amplify sustained ones. Retry only what is safe to repeat (idempotent operations), only a small number of times, with exponential backoff and jitter — and never retry at several layers at once. Three layers retrying three times each is twenty-seven requests from one click.

Circuit breakers

When a dependency is clearly down, stop calling it. Fail fast, serve a degraded response, and let one trial request decide when it is back.

// Open after 5 consecutive faults, stay open 30s, then try one.
// Crucially: a 400 is not a fault. The provider answered correctly;
// our request was wrong. Only count faults that are theirs —
// 5xx, 429, timeouts, connection errors.

Getting that distinction wrong means a wave of validation errors trips the breaker and takes down a healthy integration.

Bulkheads

Separate the resources used by different dependencies. If calls to the slow reporting service have their own bounded connection pool, they cannot consume every connection and starve the login path. This is what stops one bad dependency becoming a total outage.

Degrade rather than fail

Ask what the page can still do without each piece. Recommendations unavailable? Show the product without them. Decide this deliberately, because the default is a 500.

Seeing inside

In one process a stack trace tells you what happened. Across twelve services, a single user action is twelve stack traces in twelve places, and none of them is the whole story.

  • Correlation ids. Generate one at the edge, pass it through every call and onto every message, log it everywhere. This is the minimum, and it is cheap.
  • Structured logs. Log fields, not sentences. order_id=42 is searchable; "processing order 42" is not, reliably.
  • Distributed tracing. One timeline showing where a request actually spent its time. This is how you find out the 400 ms is a chatty loop, not the database.
  • Percentiles, never averages. An average hides the tail. p95 and p99 are what your unhappiest users experience, and tails are where distributed systems live.
Alert on symptoms, not causes "CPU is at 80%" might be fine. "Checkout error rate is above 2%" is never fine. Alert on what users feel, then use dashboards to find the cause. Teams that alert on causes end up ignoring their alerts.

The checklist I actually use

Before a new service or integration goes live:

  • Does it own its data, or is it reaching into someone else's tables?
  • Does every outbound call have a timeout, and do the timeouts fit the caller's budget?
  • Are retries bounded, jittered, and only on idempotent operations?
  • Are events written through an outbox, so the data and the event cannot disagree?
  • Can every consumer handle the same message twice?
  • Do failed messages land somewhere visible and replayable?
  • Does every cache entry have a TTL, and is the TTL jittered?
  • What does this page do when this dependency is down?
  • Does one correlation id follow the whole request, including through the broker?
  • Is there an alert on a symptom a user would notice?

None of this is exotic. It is the unglamorous half of the work, and it is the half that decides whether your architecture diagram survives contact with real traffic.

One last thing If you are not sure you need microservices yet, you probably do not. Build the modular monolith, keep the seams clean, and split the first piece that genuinely hurts. You will make a far better boundary with that evidence than with a whiteboard.

← All articles