Three signals, three jobs
Logs, metrics and traces are not three ways of doing the same thing. Each answers a different question, and a system missing one of them has a blind spot you will eventually find the hard way.
| Signal | Answers | Shape | Cost |
|---|---|---|---|
| Metrics | Is something wrong, and since when? | Numbers over time, aggregated | Cheap — flat with traffic |
| Traces | Where did this request spend its time? | One timeline per request, across services | Moderate — sample it |
| Logs | What exactly happened in this case? | Events with detail | Expensive — grows with traffic |
The working pattern is to move between them: a metric tells you error rate jumped at 14:10, a trace shows the time is going to one dependency, logs for that trace tell you which input broke it. Each one narrows the search for the next.
Log fields, not sentences
// Unsearchable. The id is inside prose.
_logger.LogInformation($"Processing order {order.Id} for {customer.Name}");
// Searchable, filterable, aggregatable.
_logger.LogInformation("Order processing started {OrderId} {CustomerId} {Total}",
order.Id, customer.Id, order.Total);
The second form lets you ask "every event for order 42 across all services" and get an answer. The first lets you grep, which stops working the moment someone rewords the message.
The correlation id
If you do one thing from this article, do this one. It is close to free and it is the difference between an investigation and a guess.
Generate it at the edge if the caller did not supply one, put it in a log scope so every line picks it up automatically, forward it on outbound calls, and copy it into message headers when you publish. Return it on error responses too — then a user's screenshot of an error page is a complete bug report.
Why averages lie
An average response time of 120 ms sounds healthy. It is also perfectly consistent with one in every hundred users waiting nine seconds, because the other ninety-nine pull the mean back down.
So: record latency as a histogram and alert on p95 and p99. And remember that tail latency compounds. If a page makes thirty backend calls and each independently has a 1% chance of being slow, roughly a quarter of page loads include a slow call. That is how a "1% problem" becomes the thing everyone complains about.
SLOs and error budgets
"The site should be up" is not a target anyone can act on. An SLO makes reliability a number you can trade against delivery speed, which is what it actually is.
- SLI — what you measure. "Percentage of checkout requests that return a success status within 500 ms."
- SLO — the target. "99.5% over 30 days."
- Error budget — what is left. 99.5% over 30 days allows roughly 3.6 hours of failure.
The budget is the useful part, because it turns an argument into arithmetic. Plenty of budget left means ship the risky change. Budget exhausted means the next sprint is reliability work, and nobody has to win a debate about it.
Alerts people do not ignore
An alert that fires and needs no action trains everyone to dismiss alerts. After a few weeks of that, your alerting system is actively harmful: it is there, it costs money, and it has taught the team not to look.
| Alert on | Not on |
|---|---|
| Checkout error rate above 2% for 5 minutes | CPU above 80% |
| p99 latency above 2s for 10 minutes | A single slow request |
| Queue depth growing for 15 minutes | Queue depth above 100 |
| Error budget burning fast enough to exhaust in 3 days | Any single failed request |
The pattern in the left column is that every row describes something a user would feel, sustained long enough to be real. The right column is either a cause that is often fine, or a blip.
Two rules I hold to:
- Every alert names an action. If the response is "watch it", it is a dashboard, not a page.
- Symptoms page, causes inform. High CPU is a clue you look up once an alert has already told you users are suffering. On its own it is noise.
Failure patterns worth recognising
Most production incidents are one of a handful of shapes. Knowing them turns a long investigation into a quick one.
- Thread or connection pool exhaustion. One slow dependency, every worker blocked waiting on it, everything unrelated dies too. The signature is rising latency across endpoints that share nothing but a pool. Fixed by timeouts and separate pools per dependency.
- Retry storm. A dependency wobbles, everything retries, the extra load keeps it down. Signature: request volume to one service rising while its success rate falls. Fixed by bounded retries with jitter and a circuit breaker.
- Cache stampede. A hot key expires, every request misses at once, the database takes the full load. Signature: a latency cliff exactly on a TTL boundary. Fixed by jittered TTLs and a lock on rebuild.
- Queue backlog. Consumers slower than producers. Signature: depth rising steadily. Alert on the trend, not a threshold — by the time a fixed number is crossed you are usually hours behind.
- The slow leak. Memory climbs over days until a restart. Signature: a sawtooth that correlates with deploys rather than traffic.
Where to start tomorrow
If you have nothing, this order gets you the most for the least:
- Correlation ids through every service and every message. One afternoon.
- Structured logs with fields, at the boundaries only.
- Four metrics per service: request rate, error rate, latency histogram, saturation of whatever runs out first.
- One SLO on the one journey that matters most. Not twelve.
- One alert on that SLO's burn rate, that pages a human who can act.
- Tracing once you have more than a handful of services and the hops are the mystery.
That is a week of work, not a platform migration. The goal is modest and worth stating plainly: when something breaks, you should find out from your own system rather than from a customer, and you should be able to tell what broke without deploying anything to find out.