The one property that makes it distributed
A single process has two outcomes: it worked, or it threw. Add a network and you get a third, and the third one is the whole subject.
When a call times out, you do not know what happened. The request may never have arrived. It may have arrived, done the work, and the reply got lost. It may still be running right now, about to do the work after you have given up. A timeout is not a failure. A timeout is an absence of information.
This is why "just add a retry" is a correctness decision rather than a resilience tweak. Retrying a
read is free. Retrying POST /payments can charge a card twice. The rest of this post is
mostly about making the second one as safe as the first.
Time is a lie
Two machines never agree on the time. NTP keeps them within tens of milliseconds on a good day, seconds on a bad one, and clocks can jump backwards when it corrects. If your logic compares a timestamp from one machine with a timestamp from another, your logic has a bug you have not hit yet.
Last-write-wins quietly eats data
The most common place this bites is the most innocent-looking line in the codebase:
// Two services update the same row. Newest timestamp wins.
if (incoming.UpdatedAt > existing.UpdatedAt)
existing.Apply(incoming);
Service A's clock is 40 ms ahead. A writes at a true 10:00:00.000 and stamps it 10:00:00.040. B writes at a true 10:00:00.020 — genuinely later — and stamps it 10:00:00.020. B's write is correct and more recent, and your comparison throws it away. No error, no log line. A field silently reverts, and someone reports it three weeks later as "the system forgot my change".
What to use instead
Order things by something the system controls rather than something the machine reports.
- A version number per row. The writer sends the version it read; the update is
WHERE Id = @id AND Version = @vand bumps it. Zero rows affected means somebody else got there first, and now you know, instead of guessing. - A database-assigned sequence. One clock, one authority. SQL Server's
rowversionis exactly this and costs nothing to add. - A logical clock when there is genuinely no single authority — a counter per node that travels with the message. More machinery than most systems need, but it is the honest answer when you have multiple writable regions.
Exactly-once delivery does not exist
Brokers advertise it, and the advertisement is true in a narrow sense that does not survive contact with your handler. The impossibility is the same one as above: the consumer processes a message, then acknowledges it. If it dies in between, the broker has no way to know whether the work happened, so it redelivers. If it acknowledged first instead, a crash loses the message entirely.
You get to choose which failure you prefer:
- At-most-once — acknowledge, then work. Messages can vanish. Almost never what you want.
- At-least-once — work, then acknowledge. Messages can repeat. This is what you want.
"Exactly-once processing" is real, but it is not a delivery guarantee you buy. It is at-least-once delivery plus a handler that does not care how many times it runs. The broker gives you the first half. The second half is your job, every time.
Idempotency keys, properly
An idempotent handler produces the same end state whether it runs once or five times. Some operations are naturally idempotent — setting a status, writing a value by primary key, adding to a set. Most of the interesting ones are not: charging a card, sending an email, incrementing a balance.
For those, the caller supplies a key that identifies the intent, not the attempt, and you store it.
public async Task<Result> ChargeAsync(
string idempotencyKey, Charge charge)
{
// The unique index is the lock. Two concurrent attempts race
// to insert; exactly one wins, and it is the database that
// decides, not a timestamp or a cache.
try
{
_db.Attempts.Add(new Attempt {
Key = idempotencyKey,
Status = AttemptStatus.InFlight
});
await _db.SaveChangesAsync();
}
catch (DbUpdateException e) when (e.IsUniqueViolation())
{
// Seen before. Return what we returned last time —
// never re-run, never make the caller guess.
var prior = await _db.Attempts.FindAsync(idempotencyKey);
return prior.Status switch {
AttemptStatus.Succeeded => Result.From(prior.Response),
AttemptStatus.Failed => Result.Fail(prior.Error),
_ => Result.Retry("still in flight")
};
}
var result = await _gateway.ChargeAsync(charge);
// Store the outcome before replying, or a crash here puts
// the key in InFlight forever.
await _db.RecordOutcomeAsync(idempotencyKey, result);
return result;
}
Three details decide whether this works in practice, and all three are usually missed.
- The key belongs to the caller, not to you. If you generate it, a retry generates a new one and you have learned nothing. The client creates it once, before the first attempt, and reuses it for every retry of that same intent.
- The uniqueness must be enforced by the database. A
SELECTthen anINSERTis a race with a window wide enough to drive a double charge through. Put a unique index on the key and let the insert fail. - Store the response, not just the fact. A retry should get the same answer as the original, not a bare "already done" the caller has no idea what to do with.
Consistency, in plain English
The literature is full of words that sound interchangeable and are not. Here is the version that matters when someone asks why the page is wrong.
| Model | What a user experiences | What it costs |
|---|---|---|
| Strong | Every read sees the latest write. No surprises. | Every read pays coordination. Latency, and unavailability during a partition. |
| Read-your-writes | You always see your own changes. Other people's may lag. | Route a user's reads to where their writes went. Usually cheap. Usually enough. |
| Eventual | Everyone converges, in an unspecified amount of time. | Nothing, until a user refreshes and their own edit is missing. |
The failure mode nobody plans for is the second row. A user edits their profile, you write to the primary, the page reloads from a read replica that is 300 ms behind, and the old name comes back. They edit it again. Now you have a support ticket and possibly two conflicting writes.
You almost never need strong consistency everywhere. You very often need read-your-writes for the thing the user just touched. Pin that user's reads to the primary for a few seconds after a write and the entire class of complaint disappears, at close to zero cost.
The retry that took the system down
A service gets slow. Every caller times out and retries. The retries triple the load on a service that was already struggling, so it gets slower, so more calls time out, so more retries. The outage is now caused by the clients, and it will not recover on its own even after the original problem goes away.
Three things turn retries from a hazard into a safety net.
1. Backoff, with jitter
Exponential backoff spreads the retries out. Jitter is the part people skip, and it is the part that matters: without it, every client that failed at the same instant retries at the same instant, forever, in waves.
// Without jitter: 1s, 2s, 4s — for everyone, simultaneously.
var delay = TimeSpan.FromSeconds(Math.Pow(2, attempt));
// With full jitter: the same ceiling, spread evenly underneath it.
var ceiling = TimeSpan.FromSeconds(Math.Pow(2, attempt));
var delay = ceiling * Random.Shared.NextDouble();
120 clients, one service, and a fault that clears partway through. Turn jitter off and the clients synchronise: every wave arrives together, exceeds capacity together, and re-breaks a service that had started to recover. Turn backoff off as well and they never stop. The simulation is a model, not a benchmark — but the shape is the one you will see in a real incident.
2. A budget, not a count
"Retry three times" is per-call. Under a broad failure, every call in the system retries three times and you have tripled total load exactly when you could least afford it. A retry budget caps retries as a fraction of overall traffic — say 10% — so a localised blip retries freely and a system-wide failure does not become a self-inflicted denial of service.
3. Stop retrying something that is down
This is what a circuit breaker is for, and the detail that makes it useful is what counts as a fault. A 500, a timeout, a connection refused: the provider's problem, trip the breaker. A 400 or a 422: your request is wrong, and retrying it a thousand times will not make it right. Counting both together means one malformed payload can open a circuit for everybody.
Your distributed lock is probably wrong
The usual implementation looks fine:
// Acquire with an expiry so a dead holder cannot block forever.
var ok = await _redis.StringSetAsync(
"lock:invoice:42", token,
TimeSpan.FromSeconds(30), When.NotExists);
if (ok) { await DoTheWorkAsync(); await ReleaseAsync(); }
The expiry is there so a crashed holder does not block the resource forever. That expiry is also the bug. Process A takes the lock, then stalls — a long garbage collection pause, a hypervisor freeze, a network hiccup. The lock expires. Process B takes it and starts working. Process A wakes up with no idea any time has passed, still believing it holds the lock, and writes.
Two writers, one lock, no error anywhere. You cannot fix this by making the timeout longer; you can only make it rarer and harder to reproduce.
That requires the storage layer to cooperate, and often it will not. So before building any of it, ask whether you need mutual exclusion at all. Most of the time you do not:
- A unique constraint — if the point is "only one of these may exist", say that in the schema and let the second insert fail.
- Optimistic concurrency — if the point is "do not clobber a concurrent edit", use a version column and handle the zero-rows-affected case.
- A single consumer — if the point is "only one worker at a time", partition the queue by key so the same entity always lands on the same consumer, and the exclusion comes free.
- A real lock manager — if you genuinely need distributed mutual exclusion, use something built for it. A lock on a cache is a lock on something explicitly allowed to lose data.
What I actually do
None of this requires a research background. It requires a short list of habits applied unglamorously.
- Every write endpoint takes an idempotency key, enforced by a unique index. Added on day one it is a field. Added after the first double charge it is a migration, a backfill and an apology.
- No cross-machine timestamp comparisons. Versions or sequences, enforced in the
WHEREclause. - Every retry has jitter and a cap, and a dead-letter queue at the end of it so a poison message parks instead of looping.
- Breakers distinguish a provider fault from a bad request. One trips the circuit; the other fails the one message and nothing else.
- Read-your-writes for the thing just edited, eventual for everything else.
- Correlation ids on everything, so the three-service version of "what happened" is one query rather than an afternoon.