Alert on symptoms, not on causes

Two symptoms of the same disease:

1. There are 60 alert rules and the on-call engineer mutes most of them.
2. The last three incidents were reported by a customer, not by monitoring.

These are not opposites. They are the same failure — alerting on causes rather than on symptoms — and adding more cause-alerts makes both worse.

Why CPU alerts are usually noise

CPUUtilization > 80% is the canonical example. It fires when a batch job runs and nothing is wrong. It stays silent when a deadlock makes every request hang at 2% CPU.

The rule encodes a guess — "high CPU probably means users are suffering" — and the guess is wrong in both directions. Every such rule you add trains the on-call engineer that alerts are usually noise, which is precisely the reflex that makes them miss the real one.

Alert on what a user would notice

The useful question is not "is this resource unusual?" but "is someone having a bad time right now?" For most services that reduces to a handful of signals:

• Latency — is the p99 outside what we promised?
• Errors — is the error rate above the level we treat as normal?
• Saturation — is a queue growing without bound? (Growth matters; depth alone does not.)
• Freshness — for pipelines and blockchain nodes, is the data current?

Four rules that fire on user-visible symptoms beat sixty that fire on gauges. Cause metrics stay in dashboards, where they are exactly what you want during diagnosis — they just should not page anyone.

Symptom differs by system

The mistake is copying someone else's four signals without asking what "bad" means for your system.

For an LLM gateway, latency alone is a poor symptom, because request cost varies by two orders of magnitude and a slow request may simply be a long one. Tokens per second and queue wait time are the honest signals.

For an Ethereum RPC fleet, the dangerous failure is not the node being down — it is a node that is up, healthy, answering quickly, and three blocks behind. Nothing in a standard health check catches that. Block-height lag relative to the chain head is the symptom, and it needs to be an alert.

For a batch pipeline, "did it run" is nearly useless. "Is the output fresher than N hours" is the thing anyone actually cares about.

Make every alert actionable

For each rule, answer three questions before it ships:

1. What does the person do when this fires? If the answer is "look at a dashboard and probably nothing", it is not an alert. It is a dashboard.
2. What breaks if we ignore it for an hour? If nothing, it is not a page. Make it a ticket.
3. What is the false positive rate? Anything above roughly one in ten will be muted within a month, and you will not be told.

Then link a runbook from the alert itself. Not a wiki search — the specific page, with the first three things to check.

Instrument the allowlist, not just the happy path

A failure mode worth naming because it is invisible: metrics that silently bucket unknown values.

We had routing metrics where unrecognised methods aggregated into unknown_*. A change in traffic mix pushed a meaningful share of requests into that bucket, and the dashboard kept rendering confidently while no longer describing reality. A dashboard that has quietly stopped being accurate is worse than no dashboard, because it is trusted.

If a metric has a categorical label, alert when the "other" bucket grows.

Delete rules

The strongest signal of alerting health is not how many rules you have but how many you have deleted.

Once a quarter, take every rule that fired and ask whether anyone did anything. If nobody did, it should not have paged. Delete it or demote it. The goal is a small set of alerts that the on-call engineer trusts enough to get out of bed for — and trust is spent by every false page.