An Internal AI Platform for Network Operations

Why build this in-house

By 2024 every network engineer was already pasting sanitized configs into public chatbots. That is a data-loss incident waiting to happen, and it also wastes the biggest opportunity: an assistant is only as good as its access to your real environment. So instead of blocking AI use, we built a platform where using it the right way was also the easiest way.

The goal was never a chatbot. It was three concrete outcomes: stop sensitive data from leaking into consumer tools, give the network team an assistant that actually knows our network, and reduce the investigation grind on the on-call rotation.

The platform layer

The foundation is Open WebUI, the de facto standard open-source, self-hosted AI interface. It gave us multi-user chat with role-based access control, model management, knowledge bases with built-in RAG, and custom tool integrations, all deployable on our own infrastructure.

Behind it sit two families of models:

• Local open-weight models running on GPUs in our own server rooms, served through an OpenAI-compatible inference layer. We rotated candidates as the field moved, settling on Qwen 2.5 in the 7B and 14B range for fast mechanical tasks and OpenAI’s gpt-oss-20b once it landed, which runs comfortably on a single modern GPU thanks to its mixture-of-experts design. Local models handle anything touching sensitive data and every high-volume workload where per-token API cost would sting.
• Commercial APIs (Anthropic, OpenAI, and others) connected through the same interface, under enterprise terms with an explicit no-training-on-our-data policy. These carry the genuinely hard reasoning tasks where model quality dominates cost.

Running both classes side by side taught us more than any benchmark: small local models are excellent summarizers, extractors, and formatters, and disappointing architects. Routing mechanical work to cheap local models and judgment work to frontier APIs became the platform’s core operating principle.

Purpose-built assistants, not one giant bot

Open WebUI lets you wrap a base model with a system prompt, a knowledge base, and a set of tools, then publish it to specific user groups. We used that to ship a catalog of narrow assistants instead of one generic one. Each assistant has exactly the context and access its job requires, and nothing more. Knowledge bases live in access-controlled S3 buckets, versioned alongside the documentation they are built from.

The flagship is the Network Expert. It combines:

• A RAG knowledge base built from our design documents, standards, and audit deliverables. One hard-won lesson: raw CSV exports and tables perform terribly in RAG retrieval, so we converted inventory data into short narrative chunks before indexing, which made retrieval dramatically more accurate.
• A tool that queries the NetBox API live, so answers about devices, racks, prefixes, and circuits come from the source of truth, not from a stale embedding.
• A tool that queries the GitLab API, where every device configuration is version-controlled, so it can read the actual running config of any device before answering.

The result is an assistant that can answer “which devices in Paris still terminate the legacy tunnel, and what would a maintenance window impact?” with real inventory and real configs behind the answer. The team uses it for maintenance planning, architecture questions, and best-practice sanity checks.

The unexpected win was onboarding. New engineers ask the Network Expert everything they would hesitate to ask a busy senior, at whatever pace they want, and peer time shifted from document walkthroughs to actual hands-on shadowing. One network specialist who joined after the rollout put it this way:

“Onboarding was smooth. I surprisingly really enjoyed interacting with the Network Expert AI. It went at my pace, answered my questions in detail, and was incredibly knowledgeable. It let me absorb knowledge quite easily.”

The AI-assisted NOC

The second major application builds on our monitoring stack (Zabbix for SNMP and availability, Graylog for logs, alerts fanned out to PagerDuty and a Teams channel). Monitoring tells you something is wrong. The expensive part is the hour that follows: logging into devices, running show commands, checking recent changes, forming a hypothesis. We built an agent pipeline to do that first hour automatically.

1. Aggregation and triage. Alerts from Zabbix and Graylog land in a triage agent alongside the human-facing notifications. It deduplicates repeats, collapses alert storms into a single incident (one flapping WAN link can produce dozens of downstream alerts: BGP teardown, unreachable hosts, syslog bursts), and correlates by site, device, and time window. The output is one clean, structured JSON object describing the incident.
2. Evidence collection. An enrichment agent takes that JSON and gathers context: live state from the affected devices through read-only credentials, the device’s intended role and connections from NetBox, its current configuration and recent changes from GitLab, and related log lines from Graylog. Everything is appended to the evidence document.
3. Investigation. A reasoning model receives the completed evidence package and produces an investigation report: most probable root cause, ranked alternative hypotheses, the evidence supporting each, and concrete next steps with the exact commands to run.
4. Delivery. The on-call engineer receives the report alongside the original page. The alert still pages a human, and a human still makes every change. The AI just shows up with the homework done.

The effect on incident response is exactly what you would hope. Instead of starting from a red icon and an adrenaline spike, the engineer starts from a document that has already pulled the interface counters, noticed the config change from yesterday, and suggested where to look first. Time to resolution drops because the investigation starts at hypothesis three instead of hypothesis zero, and the quality of escalations improved because every handoff carries its evidence with it.

Guardrails

A few boundaries kept this platform trustworthy enough for production use:

• Read-only, scoped credentials for every tool an agent can touch. No agent can change device state.
• Local models for sensitive payloads, external APIs under no-training contracts for the rest.
• Every AI-generated investigation is labeled as such, with its evidence attached, so engineers verify rather than believe.
• Access to assistants and knowledge bases follows team membership through role-based access control.

The platform started as a defensive move against shadow AI use. It ended up as the team’s most-used internal tool, and the pattern (one gateway, many narrow assistants, tools over static knowledge) has held up as the part worth copying.