A network you cannot observe is a network you learn about from your users, and by then you have already lost. When I took ownership of monitoring for a fleet of 200+ routers, switches, firewalls, and wireless controllers, the goal was simple to state and hard to do well: know about every disturbance on critical equipment before the first ticket lands, and page exactly one human with exactly enough context to act.
Two pillars: state and story
I built the stack on two complementary systems, because metrics and logs answer different questions.
Zabbix owns state. It polls the entire fleet for availability (ICMP) and health over SNMP: interface status and throughput, errors and discards, CPU, memory, temperature, power supplies, and protocol state. Everything runs on SNMPv3 with authentication and privacy enabled. SNMPv2c sends community strings in clear text, which is effectively a password broadcast, and there is no excuse for it on modern gear.
Graylog owns story. Every device ships syslog to a central Graylog cluster. Metrics tell you an interface dropped; logs tell you why, in the device’s own words, with the exact timestamps and the events on either side. Centralized logging also means that when a device dies completely, its final words are already safely off the box.
Triggers that mean something
Collection is the easy half. The difference between a monitoring system and a noise generator is trigger design. I focused triggers on conditions that map to real service impact:
- Loss of reachability on critical devices and links, with dependencies modeled so one dead upstream does not page for fifty downstream children.
- BGP session state changes on transit, peering, and WAN sessions. A session that flaps at 3 a.m. and recovers still matters, because it will do it again at 3 p.m.
- Interface error rates and utilization thresholds on backbone and firewall links, which catch degrading optics and saturation before users translate them into “the internet is slow.”
- Security-relevant log patterns from Graylog: authentication failures, configuration changes outside maintenance windows, hardware faults.
Every trigger answers three questions before it exists: what breaks for the business if this fires, who needs to know, and how fast. If there is no good answer, it becomes a dashboard line, not an alert.
Routing alerts to humans
Alert delivery is tiered. Critical alerts go to PagerDuty and page the on-call engineer. Everything else lands in a Teams network monitoring channel that the team watches during business hours.
The Teams path is where a small investment paid off disproportionately. Instead of dumping raw webhook payloads, alerts pass through Microsoft Power Automate flows that parse the payload and build a clean card: severity color, device name, site, the triggered item with its current value, a deep link straight to the alert in Zabbix or Graylog, and a link to the device in NetBox for instant context on what it is and what connects to it. Ten seconds of reading tells the engineer whether this is “finish your coffee” or “drop everything,” and the deep links remove the friction of hunting through tools during an incident.
What it changed
The fleet stopped being a black box. Failures surfaced from monitoring rather than user reports, flapping links and degrading optics got caught in the “annoying” phase instead of the “outage” phase, and the on-call experience improved because pages arrived with orientation instead of dread.
This stack also became the foundation for something bigger. Once every alert flows through a structured pipeline, you can put software in front of the human. That became the AI-assisted NOC layer, where a triage agent deduplicates and correlates alerts and an investigation agent gathers evidence before the engineer even opens a terminal. I wrote that story up in the internal AI platform case study.
Start with honest fundamentals: secure polling, centralized logs, triggers that map to business impact, and alerts that carry their own context. Everything intelligent you might want to build later depends on that plumbing being trustworthy first.