What Is RAG, and How Does AI Find Answers in Your Documents?

In this article＋

01 / ARTIFICIAL INTELLIGENCE

Retrieve first, then generate

RAG means 'retrieve evidence, then answer'. The model does not have to memorise your internal files; it selects a few permitted passages for each question and composes a response from them.

The original research contrasts knowledge in parameters with explicit updateable memory. RAG does not make a model omniscient; it narrows the evidence available at answering time.

The classic RAG demo cites a company policy and everyone nods. The troublesome version cites last year's policy and sounds completely certain.

If a policy has three versions and retrieval picks the old one, a fluent answer becomes more dangerous, not less. Tiny chunks lose exceptions; huge chunks carry noise; weak ranking hides the passage that matters.

02 / ARTIFICIAL INTELLIGENCE

Documents are split before they can be found

PDFs, pages and manuals are extracted, divided into chunks, embedded and indexed. Chunks too large mix topics; chunks too small sever conditions from exceptions.

One of the earliest and most consequential RAG decisions is not model choice. It is deciding where one meaningful piece of evidence begins and ends.

A document library is not a hamster cage; adding more files does not make retrieval wiser. Titles, versions, permissions and chunking all matter.

One good answer proves very little. Test paraphrased questions, version changes and permissions. Ask whether the system can retire old content and keep a curious employee from seeing a document they should never access.

03 / ARTIFICIAL INTELLIGENCE

Similarity is not always relevance

Vector search finds semantic similarity, not always what a question requires. Names, dates, negations and identifiers may need keyword or structured retrieval.

The answer passage may use different words from the query. Mature systems combine methods, preserve metadata and rerank results instead of accepting the nearest vectors.

I treat 'not found' as a legitimate safe state. Forcing an answer is how old policy becomes new advice.

A citation shows that a passage was retrieved, not that every inference is supported by it. Link claims back to the source and expose page, paragraph or version when the stakes justify it.

04 / ARTIFICIAL INTELLIGENCE

The error may happen before the model speaks

If retrieval selects the wrong policy version, the model can answer faithfully from bad evidence. A citation makes the failure look grounded.

Evaluation must separate retrieval recall from answer faithfulness: first test whether necessary evidence was found, then whether the answer stayed within it.

05 / ARTIFICIAL INTELLIGENCE

Easy updates do not mean easy governance

An external index is easier to update than weights, but duplicates, permissions, deletion, effective dates and version conflicts move into retrieval. A former employee should not recover old confidential material because semantic search finds it.

Access control must filter retrieval before evidence reaches the model.

06 / ARTIFICIAL INTELLIGENCE

The best RAG system can refuse to answer

When passages are insufficient, sources conflict or the question lies outside the collection, a good system asks for clarification or declines. RAG's value is not making AI answer everything.

It restricts where an answer may come from and keeps that route inspectable.

Evaluation cannot stop at whether the answer looks right

Evaluation needs four layers: was the right document retrieved, ranked where the model could use it, answered faithfully, and cited at the claim level? An answer can look right because the model guessed from internal knowledge while retrieval failed.

The hidden failure appears when the corpus changes.

Tables, scanned PDFs and Cantonese change the retrieval problem

Enterprise content contains tables, headers, scanned pages, two-column layouts and mixed languages. Broken extraction cannot be repaired by perfect embeddings.

Hong Kong data adds written Chinese, Cantonese, English abbreviations and aliases. Much RAG engineering happens before generation: OCR, cleaning, versioning and chunk boundaries.

RAG is a data product before it is an AI feature

The visible answer is the last stage of a pipeline whose quality was often decided earlier—during ingestion, extraction, permissions, chunking and versioning. Improving the generator cannot recover a missing page or a broken table.

This is why impressive demos can deteriorate after a real organisation connects years of untidy files. The durable advantage comes from maintaining the evidence layer, not merely swapping in a newer language model.

07 / ARTIFICIAL INTELLIGENCE

RAG's secret weapon is not vector maths. It is data management.

08 / ARTIFICIAL INTELLIGENCE

RAG does not mean every cited answer is correct

A citation shows that a passage was retrieved, not that every inference is supported by it. Link claims back to the source and expose page, paragraph or version when the stakes justify it.

For a frontline colleague, 'I do not know' is more useful than a polished answer built on an old policy. Retrieval needs a safe empty result.

09 / ARTIFICIAL INTELLIGENCE

Questions people actually ask

How is RAG different from fine-tuning?
RAG changes the evidence available at answer time; fine-tuning changes behaviour. Frequently updated knowledge usually starts with RAG.

Do we need a vector database?
Not necessarily. The requirement is reliable retrieval of relevant, current and authorised evidence.

How do we know RAG works?
Use real questions and measure retrieval, evidence support, version freshness and permission boundaries—not just the final prose.

Sources

Sources support the mechanisms and limitations discussed here. Models, products and prices change; check the official page and date when a detail matters.