AI on your own documents: how RAG works, and how to keep it safe
Short answer
RAG (retrieval-augmented generation) lets a language model answer from your own documents: for every question the system first looks up the relevant passages and passes them to the model. You get answers with source references, without retraining the model. The biggest risk is not the model but access rights: the AI may only use documents the person asking is allowed to open.
How does RAG work?
A language model knows nothing about your organisation. It does not know your contracts, protocols or files. RAG solves that without retraining the model, in three steps:
- Indexing. Documents are split into chunks and converted into embeddings: series of numbers that capture meaning. These go into a vector database.
- Retrieval. For a question, the system finds the passages that best match in meaning.
- Generation. The model receives the question and the passages found, and writes an answer with a reference to the source.
The advantage: new or changed documents are usable straight away, and an employee can check where an answer comes from.
Where RAG adds value in organisations
- answering questions from policies, procedures and handbooks;
- helping the service desk with answers from earlier tickets and knowledge articles;
- finding and summarising the relevant passages in a large file;
- comparing clauses against your own model contracts;
- searching protocols and guidelines in healthcare.
Where it goes wrong: four risks
1. Access rights are bypassed. The vector database holds chunks from all indexed documents. Without controls, an employee can get information through the AI from an HR file or a board paper they cannot open themselves. This is the main risk, and OWASP names weaknesses in vector and embedding storage explicitly in its top 10 for LLM applications.
2. The index is a copy of your sensitive data. Embeddings and chunks are a new collection of data, often outside the original security. If that index sits with an outside service, a copy of your documents sits there too.
3. Outdated information. If a document is withdrawn but not removed from the index, the AI keeps using it.
4. Convincing but wrong answers. Even with sources, a model can summarise incorrectly. Source references make checking possible, but do not replace it.
How to set up RAG safely
- Permission-aware retrieval: carry the access rights over from the source system and filter on every question by what this user may see.
- Index and model inside your environment, so no copy of your documents sits with a third party. See An on-premises LLM server.
- Synchronisation with the source system, so deleted or changed documents also leave the index.
- Source references required in every answer, and human review for decisions.
- Logging: which source was used for which question, by whom.
werqly runs the models and the connections to your sources on a node of your own. In werqly AI Control you decide per role which sources a team may search through the AI, and the audit log records which source was used for which answer. The automations built on top usually follow from the TIMA Scan. See also What is private AI?
Frequently asked questions
Questions on this topic
What is the difference between RAG and training a model on your own data?
With RAG the model stays unchanged; it receives the relevant passages with every question. With training or fine-tuning, knowledge is built into the model itself. RAG is quicker to update, gives source references and makes it possible to respect rights per user.
Does RAG work with open-weight models?
Yes. RAG works with any language model, including open-weight models such as Llama, Mistral and Qwen on your own hardware. The embedding model and the vector database can run locally too, so the whole chain stays inside your environment.
How do I stop staff seeing confidential documents through the AI?
By respecting the access rights from the source system during retrieval: the AI only searches documents the person asking may open. Test this explicitly, for example with a test account that has no access to an HR folder.
Is a RAG index processing of personal data?
If the documents contain personal data, yes. The index is then a new collection of personal data, with its own requirements for security, retention and deletion. Include it in your record of processing and, where needed, in the DPIA.
Sources
- Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
- OWASP — Top 10 for LLM Applications
- vLLM — documentation
This article is general information, not legal advice. Rules and guidance change; check the sources listed or ask your lawyer or DPO if in doubt.