A retrieval-augmented generation demo is easy to make look impressive.
Take a folder of PDFs. Split them into chunks. Create embeddings. Store them in a vector database. Ask a question. Retrieve a few chunks. Send those chunks to a model. Show the answer in a clean chat interface.
That is enough for a demo.
It is not enough for a production knowledge system.
The gap between those two things is where most RAG projects fail. The failure usually is not that the language model is too weak. It is that the system around the model has no reliable way to decide what information is authoritative, what the user is allowed to see, whether the retrieved material is current, whether the answer is grounded, and what should happen when retrieval does not find enough evidence.
Production RAG is an information system first and an LLM feature second.
That distinction changes the architecture.
The first failure: treating retrieval as a single search box
A real company does not have one clean knowledge corpus.
It has current policies and expired policies. Signed contracts and drafts. Customer-specific documents. Internal procedures. Tickets. CRM records. spreadsheets. database rows. shared drives. wiki pages. email threads. archived files. restricted folders. duplicated documents. scanned PDFs. documents with bad formatting. documents that disagree with each other.
A naive RAG system turns all of that into chunks and hopes semantic similarity will sort it out.
It will not.
The first job of a production retrieval layer is not similarity. It is scope.
Before asking, "What text looks similar to this question?" the system should often ask:
- Which source systems are relevant?
- Which tenant, customer, project, or department does this question belong to?
- Which document types are authoritative?
- Which records are active?
- Which version is current?
- What can this user access?
- Does the query contain an exact identifier that should be resolved deterministically?
If a user asks for the cancellation clause in Contract 1847-B, the best first step is usually not a global vector search. Resolve Contract 1847-B. Verify the user can access it. Identify the current signed version. Then retrieve the relevant section.
That is a very different architecture from "embed everything and search everything."
For deeper implementation patterns, see our RAG Knowledge Systems and Deterministic Retrieval pages.
The second failure: bad source boundaries
Most RAG quality problems start before retrieval.
If ingestion cannot reliably tell what a document is, where it came from, who owns it, when it changed, and whether it is still valid, the generation layer cannot fix the problem later.
A production ingestion pipeline should preserve source identity.
At minimum, useful metadata often includes:
- source system
- source record ID
- document ID
- document version
- created and modified timestamps
- ownership
- tenant or customer scope
- department or workspace
- access-control information
- document type
- effective and expiration dates where relevant
- ingestion status
This metadata is not decoration. It becomes part of retrieval and governance.
If a policy was replaced yesterday, the system should not keep citing the old version just because the old wording has a slightly better vector similarity score.
The third failure: chunking without understanding the document
"Chunk every 800 tokens with 100-token overlap" is a prototype shortcut, not a universal architecture.
Different content needs different treatment.
A contract has sections, clauses, schedules, signatures, and defined terms. A support ticket has a conversation timeline. A spreadsheet has rows and columns whose meaning depends on headers. A policy has a hierarchy. A product manual may depend on model numbers and revision dates.
Blind chunking can separate a rule from its exception, a number from its label, or a contract term from the section that defines it.
Good retrieval often uses document-aware segmentation and multiple representations of the same source. A small retrieval unit may help locate the correct passage, while a larger parent section provides enough context for the model to answer accurately.
The point is not to make chunking complicated for its own sake. The point is to preserve meaning.
The fourth failure: assuming vector search is the retrieval strategy
Vector search is useful. It is not the entire retrieval layer.
Production systems often need a mix of:
- exact identifier lookup
- metadata filtering
- structured database queries
- keyword or lexical search
- deterministic rules
- semantic retrieval
- reranking
- parent-child retrieval
- source-specific routing
If a query contains an invoice number, policy ID, case number, property address, SKU, employee ID, or contract ID, exact lookup may be more reliable than semantic similarity.
If the user asks, "What was the escalation rule for after-hours maintenance at Property 12 last year?" the system may need property scope, date filtering, source selection, and then semantic retrieval inside the remaining records.
That is why we treat deterministic retrieval and vector retrieval as complementary tools rather than competing religions.
The fifth failure: permissions added after the prototype
This one can kill an enterprise deployment.
A knowledge system that produces accurate answers from data a user was never allowed to access is still a broken system.
Permissions have to exist in retrieval, not only in the interface.
Hiding a link after generation is not sufficient. The system should avoid retrieving unauthorized material into the model context in the first place.
Depending on the environment, that may mean mapping identity and authorization from a company directory, source system, application role, tenant boundary, matter, account, department, or custom policy layer.
This is one reason enterprise RAG cannot be reduced to a chatbot wrapper.
Our Internal Knowledge and RAG systems are designed around permission-aware retrieval and source-grounded responses rather than a single shared corpus.
The sixth failure: no answer for "I do not know"
Teams spend a lot of time tuning prompts to make models answer better. They spend less time designing what should happen when the system should not answer.
That is backwards.
A production system needs uncertainty behavior.
When evidence is weak, conflicting, missing, stale, or outside the user's access scope, the system should have a defined response path. Depending on the workflow, that could mean:
- ask a clarifying question
- return the sources it found without synthesizing a conclusion
- say the approved knowledge base does not contain the answer
- escalate to a person
- create a research task
- query another authorized system
- require approval before a downstream action
The worst behavior is confident completion when the evidence does not support it.
Grounding is not a prompt phrase. It is a system behavior.
The seventh failure: no evaluation harness
A RAG system cannot be managed with a few screenshots and subjective comments like "it seems better now."
You need a test set.
The test set should represent real questions, including difficult ones. It should include common queries, ambiguous queries, missing-answer queries, permission-sensitive queries, exact-lookup queries, questions with conflicting sources, and questions where the correct response is to abstain.
Useful evaluation dimensions include:
- retrieval success
- source correctness
- citation correctness
- answer groundedness
- completeness
- refusal or abstention quality
- access-control correctness
- latency
- cost
A model upgrade can improve one category while quietly breaking another. A new chunking rule can improve broad semantic questions and hurt exact policy questions. A new source connector can increase recall while introducing stale duplicates.
Without repeatable evaluation, every change is a guess.
The eighth failure: no freshness strategy
Knowledge systems decay.
The source data changes whether the AI system is ready or not.
Policies get replaced. tickets close. CRM records change. contracts expire. new product documentation arrives. old documents are archived. permissions change.
A production RAG architecture needs an explicit synchronization model.
That means deciding whether each source is updated by event, webhook, change feed, scheduled sync, database replication, manual publishing, or another mechanism. It also means deciding how deletions propagate, how access changes propagate, and how failed ingestions are retried.
If you cannot answer "How quickly does a source update become visible in the AI system?" you do not have a freshness guarantee.
The ninth failure: no observability
When an answer is wrong, someone needs to be able to determine why.
Was the query routed to the wrong corpus? Did permission filtering remove the correct document? Did retrieval return weak passages? Did reranking choose badly? Was the correct source present but ignored by the model? Was a tool call slow? Did the source connector fail two hours ago?
Production systems need traces that make these questions answerable.
At useful boundaries, log things such as:
- normalized user query
- identity and scope information
- retrieval strategy selected
- filters applied
- candidate sources
- ranking scores where useful
- final context supplied to the model
- model and prompt version
- tool calls
- latency
- token or inference cost
- final citations
- user feedback
Be deliberate about sensitive data in logs. Observability is necessary, but uncontrolled logging can create a second data-governance problem.
The tenth failure: stopping at the answer
A knowledge answer is valuable. A system that safely moves work is often more valuable.
The strongest internal AI systems connect retrieval to business actions.
A support copilot may find the correct policy, draft the answer, attach citations, and prepare a ticket update for approval. An operations agent may retrieve a contract term, compare it with an invoice, flag a mismatch, and route the case. A property management agent may retrieve the lease and maintenance rules before drafting the next action.
That is where RAG becomes part of a production workflow rather than an isolated chat experience.
See AI Integration and Automation for how retrieval systems connect to CRMs, ticketing systems, databases, calendars, inboxes, and custom APIs.
What actually works
A reliable production RAG system usually has clear layers rather than one giant "AI" box.
A practical architecture looks more like this:
1. Source connectors collect data from approved systems. 2. Normalization and document processing preserve structure and identity. 3. Authorization metadata travels with the source. 4. Indexes support the retrieval methods the data actually needs. 5. Query routing decides which source and strategy apply. 6. Retrieval and reranking assemble evidence. 7. Generation answers only from allowed context. 8. Citations and uncertainty handling make the result inspectable. 9. Evaluation and observability measure quality over time. 10. Workflow integration turns answers into controlled action where appropriate.
Not every project needs every component. The important part is that each production risk has an owner in the architecture.
A production RAG checklist
Before launching, we want clear answers to questions like:
- What sources are authoritative?
- How are stale versions removed or deprioritized?
- How are permissions enforced before model generation?
- Which queries use deterministic lookup instead of vector search?
- How do exact identifiers work?
- What happens when evidence is missing?
- Are citations tied to the actual retrieved source?
- How is quality tested before and after changes?
- How are ingestion failures detected?
- How quickly do source updates propagate?
- Can we trace a bad answer from query to source to output?
- What data is stored in logs?
- What actions require human approval?
- Who owns the system after launch?
If those questions do not have answers, more prompt engineering is unlikely to fix the core problem.
The useful mental model
Do not ask, "Which vector database should we use?"
Ask, "What evidence does this workflow require, how do we retrieve it reliably, and what controls must be true before the system can answer or act?"
That question produces a much better system.
RAG works in production when retrieval is treated as infrastructure: scoped, permission-aware, observable, testable, and connected to the actual business process.
The language model matters. It is just not the whole architecture.
Next step: Talk to an engineer about applying this to your stack.