Alert Fatigue: Designing Network Alerts That Matter
A NOC receiving hundreds of alerts per shift learns to ignore them — and eventually misses the one that matters. Good alerting is a design discipline, not a default configuration.
Principles of a good alert
- Actionable: someone needs to do something now. If not, it is a report, not an alert.
- Prioritised: severity reflects business impact, not just technical state.
- Specific: it says what is wrong, where, and links to the runbook.
- Deduplicated: one site outage creates one incident, not fifty device alerts.
Practical techniques
- Dependencies: if a site router is down, suppress alerts for devices behind it.
- Hold-down timers: alert only if a condition lasts longer than a short window, to ignore blips.
- Thresholds from baselines: “80% utilisation” means different things on different links.
- Maintenance windows: silence planned work automatically.
- Weekly review: tune or delete the noisiest alerts.
Measure the alert system
Track alerts per shift, the share that led to action and incidents that had no alert. Aim for fewer, better alerts.
Alert design is part of every NMS deployment we deliver. Fix your alerting.
