Agentic RAG That Knows When Not to Answer

DeX started with a familiar problem: platform teams answering the same questions in Slack over and over. The answers existed, scattered across product docs and years of threads. We built an agentic RAG assistant over all of it, and it grew to 70+ platforms, resolving 80% of repeated queries. The interesting lesson wasn't how it answered. It was how it learned to decline.

Retrieval quality is the ceiling

No model can answer from context it never received. We invested heavily in ingestion: chunking that respected document structure, metadata for freshness and ownership, bge-m3 embeddings with a reranker, and coverage reports that showed which themes the knowledge base simply didn't cover.

The agentic loop

• Query rewriting turned vague Slack questions into retrievable ones.
• An answerability check asked: can the retrieved sources actually support an answer?
• A critique step reviewed the draft against the sources before anything was posted.
• A summary step kept answers short enough to be read in a thread.

When answerability failed, DeX said so and routed to a human — with the relevant context attached so the human started ahead.

A confident wrong answer in an internal tool costs you every right answer that comes after it.

Roll out in the shadows

We launched in shadow mode: DeX drafted answers privately while humans kept answering publicly. Only when confidence and accuracy were proven per channel did we push answers to the main thread. Shadow mode gave us real evaluation data without spending user trust.

Measure obsessively

We tracked thousands of metrics, but a handful mattered most: coverage, accuracy, LLM-scored answer quality, ingestion freshness, dissatisfaction rate, and time to acknowledge and resolve. Voice-of-customer themes from feedback told us which docs to write next — and DeX started generating documentation from resolved threads itself.

The takeaway

Enterprise RAG succeeds or fails on operations: ingestion pipelines, evals, rollout discipline and honest uncertainty. Get those right and the model choice becomes a cost decision, not an existential one.