Five days, with a citation
The HR team forwarded a screenshot from the company's internal assistant. Someone had asked How many unused holiday days can I carry over into next year?, and the assistant answered: five days, to be used by the end of March, citing Annual leave policy 2024 (PDF). The policy had changed in February, and the current one allows ten. The answer matched its source word for word; the fault was in which source it had been handed.
Handing the model the page
A language model answers from what it learned in training, which is unlikely to include this company's internal leave policy and does not follow later changes to it. Retrieval-augmented generation (RAG) leaves the model as it is and changes what it is given: at question time, the system finds the passages most likely to answer, puts them in the prompt, and asks the model to answer from them and cite which ones it used.
It has two halves on different clocks. Ingestion runs in the background and turns the wiki, the HR portal and the IT knowledge base into an index. The query path runs once per question, in a few seconds.
From document to index
- A Document Loader pulls each document from its source with its metadata: title, source, last-modified date, and the groups allowed to read it. A Web Crawler does the same for intranet pages, an OCR Service for scanned PDFs.
- A Document Chunker splits the text into chunks of a few hundred tokens along headings and paragraphs, often with a small overlap so a sentence cut at a boundary survives whole in one of them.
- The Embedding Service turns each chunk into a vector.
- The pipeline upserts each vector into the Vector Store, keyed by document id and chunk number, with the document's metadata beside it.
Chunk size fails in both directions. One chunk for the whole leave policy gets one vector for carry-over, parental leave and sick days together, so it matches every leave question a little and none well. A chunk of two sentences can lose the line above it, This section applies to part-time staff, and carry the wrong rule into an answer. One longer than the embedding model's input limit is truncated or rejected, depending on the API.
The Vector Store is the storage; the Embedding Service is the model that produces the vectors. Both appear in a RAG diagram; they are not duplicates. If the data already lives in Postgres, pgvector can be the Vector Store: same role, no new database. Vectors only make sense for the model that made them, so questions and chunks must be embedded by the same model version, and a new model means re-embedding what the vector store holds. How embeddings, nearest-neighbour indexes and metadata filters work is Vector search: embeddings, nearest neighbours & hybrid search; running the embedding model is Model serving: inference, versions & safe rollout.
Retrieve fifty, keep eight
On the query path a RAG Retriever embeds the question with the same model and asks the vector store for the 50 nearest chunks. For questions like VPN error 809, where the exact code matters, it also runs a keyword search and merges the lists; that hybrid search, and the approximate index behind the vector half, are in Vector search: embeddings, nearest neighbours & hybrid search.
Why fifty, when the prompt will hold eight? The first stage trades precision for speed. Each chunk was compressed into one vector before any question existed, and the search compares two vectors, so it tends to find chunks about carry-over rather than the one that answers this question. Its top five might be the FAQ's general holiday paragraph, the policy's introduction and a team-calendar note, with the carry-over rule seventh.
A Reranker reads the question and each candidate together. Often it is a cross-encoder, a model that takes both texts as one input and returns a relevance score. Too slow for the whole index, it is fine over 50, and its top eight go into the prompt: fewer tokens, and less irrelevant text to mislead the model.
Ingestion fills the vector store. Per question: cache check (a hit answers at once), else 50 chunks retrieved, 8 kept, prompt sent through the gateway.
A prompt that cites
The prompt holds an instruction (answer only from these passages, say so when they lack the answer, cite each one used), the eight passages with an id, title and last-modified date each, and the question. The service then checks that every cited id was among those sent and links each to its source. Grounding narrows what the model says; it does not make it right. A model can still misread a table, blend two passages, or fill a gap from its training, and the citation is how a reader catches that. The call itself goes out through the gateway, with its screening of prompts and responses, described in Serving LLMs: gateways, fallback & token budgets.
Was it the search or the model?
Back to the five days. The assistant logs, per question, the 50 chunks retrieved, the eight kept, the prompt and the answer, and that trace puts a wrong answer in one of three places:
- The chunk that answers is not among the 50: a retrieval failure. Chunking, embedding, the query or a filter is at fault, or the document was never ingested.
- It is among the 50 but not the eight: the reranker demoted it.
- It is in the prompt and the answer is still wrong: a generation failure, in the prompt or the model.
Here both policies were retrieved, both kept, both in the prompt, and the model quoted the older one. A generation failure on paper, but the cause was upstream: when HR replaced the PDF with a page on its portal, nothing removed the PDF's chunks from the index. Telling the model to prefer newer dates would hide this case, not the next.
To keep the two apart over time, the team keeps real questions labelled with the passage that answers each, taken from the current version. Retrieval is scored by how often a chunk holding that passage reaches the 50 and the eight, answers by whether they are correct and supported by their citations, so a drop in one cannot hide behind the other.
Edits and deletions have to reach the index
Re-embedding changed text and updating its vectors is covered in Vector search: embeddings, nearest neighbours & hybrid search. What RAG adds is that one document is many chunks. Each carries its document's id and version, because a new version may split differently and overwriting chunk by chunk leaves the old tail behind, so an update indexes every chunk of the new version first, then switches the document's active version, which retrieval filters on, and only then deletes the previous version's chunks. A replaced or deleted document must lose all its chunks, which an add-only loader never does; it needs a feed of changes from the source, or a regular comparison of the source's listing with the index (Change data capture covers change feeds from databases). And an answer cached from a replaced or removed document has to go with it, so each cached answer records the documents and versions it was built from.
Context nobody was allowed to read
A team lead asks What is the salary band for a senior engineer? The salary review sits in a folder only HR can open, but its chunks share the vector store. If retrieval ignores permissions, one can reach the prompt and be paraphrased; hiding the citation does not help, and a prompt instruction not to reveal it is not access control.
The check belongs at retrieval time. Each chunk carries the groups allowed to read its document, copied at ingestion, and the retriever passes the user's groups as a metadata filter on the search, so a chunk the user cannot read never becomes a candidate. Whether that filter runs before or after the vector search changes how many results come back, which the vector search article explains. When permissions change, the edit pipeline updates the chunks' metadata, and until it does the index enforces the old ones; where that delay matters, the service also checks the final eight against the source system.
A hundred wordings of one question
Every January the carry-over question arrives in a hundred wordings, each paying for retrieval, reranking and a model call. A Semantic Cache returns a stored answer when a new question's embedding is close enough to a cached one. But How many days can I carry over? and How many days can I take unpaid? are close too. So scope it per tenant and permission set, and by whatever else the answer depends on, such as the user's country or contract type, skipping the cache when that cannot be told; use a strict similarity threshold checked against real pairs like these, and invalidate it when source documents or permissions change, or a cached five days outlives the fix.
Draw the assistant that said five
The Enterprise RAG Chatbot Challenge asks for this assistant in at most 15 nodes, with ingestion as its own asynchronous path (loader, chunker, a queue worker calling the embedding service, the vector store). Label the edge from the retriever to the reranker with how many chunks go in and how many come out, write in the vector store's title what each chunk carries (document id, version, allowed groups), and add a note saying what removes a document's chunks when it is replaced or deleted. The semantic cache goes where it knows the user's permission set.