When we built the Network Expert assistant for our operations team (part of the internal AI platform), the obvious first move was: export everything from NetBox, dump the design docs and configs into a knowledge base, wire up retrieval-augmented generation, done. An afternoon of work.

The afternoon version answered like a drunk intern. Confident, specific, and wrong in ways that were expensive to notice. The road from there to an assistant the team actually trusts taught us three lessons that I have not seen written down often enough.

Lesson one: tables are poison for RAG

RAG works by embedding chunks of text and retrieving the chunks most semantically similar to the question. That mechanic has an unstated assumption: each chunk is self-describing. A paragraph of prose is. A row from a CSV export is not.

Take a NetBox device export. Row 400 says something like sw15,site-a,rack-12,model-x,active. By the time chunking has carved the file into pieces, that row has been severed from its header and its context. Embedded alone, it is semantic gravel. A question like “what switches are in rack 12 at site A?” has almost no vector similarity to the naked row, so retrieval fetches something else plausible-sounding, and the model, given garbage context, confabulates politely around it.

The failure is quiet, which is what makes it dangerous. The assistant does not say “I cannot find that.” It answers fluently from whatever wrong chunks arrived, seasoned with its general training knowledge of what networks usually look like. Usually is doing a lot of work in a production network.

Lesson two: narrative chunking

The fix is almost embarrassingly low-tech: convert tabular data into short prose before indexing, one self-contained paragraph per entity.

Instead of a CSV row, the knowledge base holds: “Switch sw15 is a model-x access switch located in rack 12 at site A. It is active, serves the third floor, and uplinks to distribution pair dist-1/dist-2.” Every fact travels with its subject. Any chunk boundary still leaves a complete, meaningful statement, and the embedding of that paragraph sits close to the questions humans actually ask about it.

We generated these paragraphs programmatically from NetBox exports (a script templating rows into sentences), so the process stays repeatable as inventory changes. Retrieval accuracy went from coin-flip to dependable, with no model change, no fancy embedding upgrade, no reranker. The data just had to arrive shaped like language. If you take one tactical thing from this post, take that: RAG quality is mostly a data-shape problem, not a model problem.

Lesson three: know when RAG is the wrong tool

The deeper realization came later: for some questions, no amount of chunking discipline saves you, because the knowledge base is a snapshot and the network is not. “What is the current status of the link between site A and site B?” deserves an answer from now, not from export day. Stale answers delivered confidently are worse than no answer during change planning.

So the architecture split along a clean line:

  • RAG for stable knowledge: design documents, standards, audit findings, architectural rationale, procedures. Things that change on human timescales and benefit from semantic search.
  • Live tools for operational state: the assistant holds scoped, read-only API access to NetBox for inventory and to GitLab for current device configurations. When you ask about a specific device, it does not search an embedding of last month; it queries the source of truth and reads the actual config.

The mental model that stuck with the team: RAG gives the assistant memory, tools give it eyes. Long-term memory is allowed to be slightly stale; eyesight is not. Most “our chatbot lies” complaints I hear trace back to teams using memory where eyes were required.

Was it worth it?

Yes, and measurably so in behavior: the Network Expert became the first stop for maintenance planning and architecture questions, and the default answer machine for new hires. But the win only arrived after we stopped treating knowledge ingestion as a bulk upload and started treating it as data engineering: shape the stable knowledge into narrative, route the live questions to APIs, and label which is which so the model knows where to look.

Feeding an LLM your documentation is easy. Feeding it in a form it can actually retrieve, and knowing which questions should bypass documents entirely, is the actual work. Budget for the second part; it is where all the value lives.