Observability and reliability: knowing before your users do

Monitoring tells you the thing you predicted has happened. Observability lets you answer a question you had not thought of yet — at 3am, from a dashboard, without deploying a new log line.

Three signals, three jobs

Logs, metrics and traces are not three ways of doing the same thing. Each answers a different question, and a system missing one of them has a blind spot you will eventually find the hard way.

Signal Answers Shape Cost
Metrics Is something wrong, and since when? Numbers over time, aggregated Cheap — flat with traffic
Traces Where did this request spend its time? One timeline per request, across services Moderate — sample it
Logs What exactly happened in this case? Events with detail Expensive — grows with traffic

The working pattern is to move between them: a metric tells you error rate jumped at 14:10, a trace shows the time is going to one dependency, logs for that trace tell you which input broke it. Each one narrows the search for the next.

Log fields, not sentences

// Unsearchable. The id is inside prose.
_logger.LogInformation($"Processing order {order.Id} for {customer.Name}");

// Searchable, filterable, aggregatable.
_logger.LogInformation("Order processing started {OrderId} {CustomerId} {Total}",
    order.Id, customer.Id, order.Total);

The second form lets you ask "every event for order 42 across all services" and get an answer. The first lets you grep, which stops working the moment someone rewords the message.

Logs are where observability budgets go to die A debug line on a hot path is free in development and ruinous at ten thousand requests a second. Log at the boundaries — request in, dependency called, request out — plus anything genuinely unexpected. Everything in between belongs in a trace span or a metric.

The correlation id

If you do one thing from this article, do this one. It is close to free and it is the difference between an investigation and a guess.

gateway id created here API Orders broker worker every log line in every box carries the same id — one query returns the whole story
The broker hop is the one teams forget. An id that stops at the queue leaves async work uninvestigable.

Generate it at the edge if the caller did not supply one, put it in a log scope so every line picks it up automatically, forward it on outbound calls, and copy it into message headers when you publish. Return it on error responses too — then a user's screenshot of an error page is a complete bug report.

Why averages lie

An average response time of 120 ms sounds healthy. It is also perfectly consistent with one in every hundred users waiting nine seconds, because the other ninety-nine pull the mean back down.

response time average 120ms p99 — 9 seconds most requests are fine the tail is where your angriest users live
One in a hundred sounds rare until you notice a page makes thirty calls — then most page loads contain at least one slow one.

So: record latency as a histogram and alert on p95 and p99. And remember that tail latency compounds. If a page makes thirty backend calls and each independently has a 1% chance of being slow, roughly a quarter of page loads include a slow call. That is how a "1% problem" becomes the thing everyone complains about.

Watch what the percentile is over A p99 across every endpoint is dominated by whichever endpoint is busiest. Slice by endpoint, and by tenant if you are multi-tenant — one customer with ten times the data routinely has a completely different experience of the same system, and the global number hides it entirely.

SLOs and error budgets

"The site should be up" is not a target anyone can act on. An SLO makes reliability a number you can trade against delivery speed, which is what it actually is.

  • SLI — what you measure. "Percentage of checkout requests that return a success status within 500 ms."
  • SLO — the target. "99.5% over 30 days."
  • Error budget — what is left. 99.5% over 30 days allows roughly 3.6 hours of failure.

The budget is the useful part, because it turns an argument into arithmetic. Plenty of budget left means ship the risky change. Budget exhausted means the next sprint is reliability work, and nobody has to win a debate about it.

Do not set it to 99.99% because it sounds impressive Each nine costs roughly ten times the previous one, and it is paid in engineering time you are not spending on features. Pick the number where users stop noticing, not the number that looks good on a slide. For most internal and B2B systems that is considerably lower than people expect.

Alerts people do not ignore

An alert that fires and needs no action trains everyone to dismiss alerts. After a few weeks of that, your alerting system is actively harmful: it is there, it costs money, and it has taught the team not to look.

Alert on Not on
Checkout error rate above 2% for 5 minutes CPU above 80%
p99 latency above 2s for 10 minutes A single slow request
Queue depth growing for 15 minutes Queue depth above 100
Error budget burning fast enough to exhaust in 3 days Any single failed request

The pattern in the left column is that every row describes something a user would feel, sustained long enough to be real. The right column is either a cause that is often fine, or a blip.

Two rules I hold to:

  • Every alert names an action. If the response is "watch it", it is a dashboard, not a page.
  • Symptoms page, causes inform. High CPU is a clue you look up once an alert has already told you users are suffering. On its own it is noise.

Failure patterns worth recognising

Most production incidents are one of a handful of shapes. Knowing them turns a long investigation into a quick one.

  • Thread or connection pool exhaustion. One slow dependency, every worker blocked waiting on it, everything unrelated dies too. The signature is rising latency across endpoints that share nothing but a pool. Fixed by timeouts and separate pools per dependency.
  • Retry storm. A dependency wobbles, everything retries, the extra load keeps it down. Signature: request volume to one service rising while its success rate falls. Fixed by bounded retries with jitter and a circuit breaker.
  • Cache stampede. A hot key expires, every request misses at once, the database takes the full load. Signature: a latency cliff exactly on a TTL boundary. Fixed by jittered TTLs and a lock on rebuild.
  • Queue backlog. Consumers slower than producers. Signature: depth rising steadily. Alert on the trend, not a threshold — by the time a fixed number is crossed you are usually hours behind.
  • The slow leak. Memory climbs over days until a restart. Signature: a sawtooth that correlates with deploys rather than traffic.
Write the incident down, honestly and blamelessly Timeline, impact, what you believed at each point and what was actually true. The value is less in the fix than in the gap between those last two — that gap is a missing signal, and closing it is the observability work that actually matters.

Where to start tomorrow

If you have nothing, this order gets you the most for the least:

  1. Correlation ids through every service and every message. One afternoon.
  2. Structured logs with fields, at the boundaries only.
  3. Four metrics per service: request rate, error rate, latency histogram, saturation of whatever runs out first.
  4. One SLO on the one journey that matters most. Not twelve.
  5. One alert on that SLO's burn rate, that pages a human who can act.
  6. Tracing once you have more than a handful of services and the hops are the mystery.

That is a week of work, not a platform migration. The goal is modest and worth stating plainly: when something breaks, you should find out from your own system rather than from a customer, and you should be able to tell what broke without deploying anything to find out.

← All articles