What is RAG in AI: using it with company documents

RAG (retrieval-augmented generation) is a technique that lets a language model answer with information from your own documents. Before the model writes an answer, the system searches your documentation for the relevant passages and gives them to the model as context. The model then answers with your company’s facts, cites the source, and you never have to retrain it.

This guide explains where the term comes from, how a RAG system works, when to choose it over the alternatives, which quality problems to expect, and what the GDPR and the EU AI Act require.

Where the term RAG comes from

The term comes from a 2020 research paper: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis and colleagues. The paper defines RAG models as models that combine two kinds of memory to generate text:

• Parametric memory: what the language model learned during training. It is general, and you cannot update it without retraining.
• Non-parametric memory: an index of external documents that the system consults for every question. You can update it whenever you like.

For a business, this split is what matters. The model brings the ability to understand and write. Your documents bring the facts. When a procedure changes, you update the document, and the next answer already uses the new version.

How a RAG system works, step by step

A RAG system over company documents has two phases. The first prepares the documents. The second answers each question.

Phase 1: preparing the documents

1. Ingestion. The system collects documents from their sources: shared drives, a document management system, the intranet, a knowledge base or resolved tickets. It turns each format (PDF, Word, HTML, email) into clean text and keeps the metadata: author, date, department and who is allowed to see it.
2. Chunking. It splits each document into manageable passages, usually by section or paragraph. Chunks that are too large add noise; chunks that are too small lose context. This step has a big effect on final quality.
3. Embeddings. An embedding model turns each chunk into a vector, a list of numbers that represents its meaning. Two texts about the same thing end up with nearby vectors, even when they use different words.
4. Vector database. The vectors are stored in a database that can search by similarity. It can be a dedicated product or an extension of a database you already run, such as pgvector for PostgreSQL.

Phase 2: answering a question

5. Retrieval. The user’s question is also turned into a vector. The system finds the nearest chunks and applies metadata filters, for example only current documents, or only documents this user may see. Many systems combine this semantic search with keyword search so that product names and reference codes are not missed.
6. Generation with citations. The language model receives the question and the retrieved chunks, with an instruction to answer only from that material. The answer references each document it used, so the user can check it. If the chunks do not contain the answer, the system should say it does not know.

RAG vs fine-tuning vs long context

There are three common ways to give a model access to company knowledge. They solve different problems rather than competing with each other.

Current models accept very large contexts; some reach one million tokens. More context is not automatically better, though. The same documentation warns that accuracy and recall degrade as the token count grows. The Lost in the Middle study (Liu et al., 2023) showed that models make worse use of information placed in the middle of a long context. For a large document collection that changes over time, RAG remains the practical choice.

A useful rule: if the question is “what do our documents say about X”, start with RAG. Fine-tuning is rarely the first step, as we explain in our guide to AI agents for business.

Business use cases for RAG

RAG fits when there is a lot of documentation, people waste time searching it, and the right answer is written down somewhere.

• Internal knowledge base. Staff ask questions in plain language about procedures, HR policies, technical manuals or internal rules. The assistant answers and links to the source document, which cuts down on the same questions landing on the same people.
• Customer support chatbot grounded in your documentation. The chatbot answers from product manuals, terms of service and FAQs. When a question needs customer data or an action, such as booking an appointment or checking an order, the chatbot also needs tools connected to your systems. That is where the agent part comes in.
• Contract and procedure search. Find clauses, deadlines or requirements across hundreds of documents and summarise them with an exact reference. A person checks the original before deciding.

What we have built at LCApps

At LCApps we have built a RAG service over documents: it processes documents, indexes them in a vector database and returns the relevant passages for each question. It powers multichannel chatbots (web, WhatsApp, SMS and Telegram) that answer from a company’s documentation and, when needed, query live business data and carry out actions.

We have also built MCP servers that connect AI assistants to business systems. A RAG service can be exposed as an MCP tool so that any compatible assistant can use it. We explain how in what is MCP (Model Context Protocol).

We design and operate all of these projects in line with the GDPR and the EU AI Act.

Quality problems to plan for

A RAG pilot usually shines in the demo and stumbles on the details. These are the most common problems:

• Hallucinations. RAG reduces them but does not remove them. The model can blend two passages or fill gaps with what it “knows” from training. Mitigate this with strict instructions, mandatory citations and an explicit option to answer “I can’t find this in the documentation”.
• Outdated or duplicate documents. If three versions of a procedure coexist, the system may cite the old one. Before indexing, decide which sources are authoritative and flag the current versions. RAG does not tidy your documentation; it exposes the mess.
• Per-user permissions. An assistant serving the whole company must not show payroll or contracts to people who should not see them. Permissions are enforced at retrieval, using each chunk’s metadata, and must match the source system. Filtering only the final answer is not enough.
• Weak retrieval. If the system cannot find the right passage, even the best model cannot answer well. Typical causes are poor chunking, badly converted tables or scanned PDFs, and internal acronyms the embedding model does not understand.
• No evaluation. Without measurement, every change is a gamble. Build a set of real questions with their correct answer and source document. Measure whether the system retrieves the right document, whether the answer is faithful to the source and whether it is useful. Re-run it after every change.

Security, GDPR and the EU AI Act

Company documents contain personal data: names in contracts, employee records, customer emails. Indexing them creates a new processing activity.

GDPR

The GDPR applies as it does to any other system. The points that matter most for a RAG project are:

• Data minimisation (Article 5). Index only the documents the use case needs. Leave out folders with sensitive data that nobody needs to query through the assistant.
• Access control and security (Article 32). Respect source permissions, encrypt the data and log who queries what.
• Data location and transfers (Chapter V). Check where the vectors are stored and where model requests are processed. If data leaves the European Economic Area, you need a legal basis for the international transfer.
• Processors (Article 28). The model provider, the embedding provider and the vector database provider all process data on your behalf. You need a contract with each, and you should check whether they use your data for training.
• Impact assessment (Article 35). If the system processes sensitive data at scale or profiles people, a DPIA may be mandatory.
• Data subject rights. If someone asks for their data to be erased, it must also be removed from the index, not only from the original document.

EU AI Act

Article 50 of Regulation (EU) 2024/1689 requires that people know they are interacting with an AI system, unless that is obvious. This obligation has applied since 2 August 2026. If your RAG chatbot serves customers or the public, say so clearly at the start of the conversation.

Regulation (EU) 2026/1744, known as the “Digital Omnibus on AI”, entered into force on 27 July 2026. It does not change that disclosure duty. It does postpone the high-risk obligations: to 2 December 2027 for Annex III systems and to 2 August 2028 for Annex I systems. A RAG assistant over internal documentation is not usually high-risk. It can be if it plays a part in decisions on employment, credit or access to essential services, so check before you start.

How to run a RAG pilot

1. Pick one business question. For example: “the support team takes too long to find fixes for known incidents”. One use case, one group of users.
2. Select and clean the sources. Decide which documents are authoritative, which are current and who may see each one.
3. Build the evaluation set. Collect 50 to 100 real questions with their correct answer and the document that contains it.
4. Review data and contracts. What personal data the documents hold, which providers are involved and where processing happens.
5. Build the pilot with ingestion, permission-filtered retrieval and cited answers. If users are external, add the notice that they are talking to an AI.
6. Measure with real users for a few weeks:
   • Share of correct answers with the correct source.
   • Share of questions where the system says “I don’t know” when it should.
   • Time saved per query.
   • Cost per query.
   • User ratings.
7. Decide with data. Extend to more sources, fix the documentation, or drop the use case.

If the pilot needs integrations with your systems, budget it like any other build; we break that down in custom software development cost.

Frequently asked questions

Does RAG train the model on my documents?

No. With RAG the model does not learn your documents; it consults them for each question. That is why you can update or delete a document and the change applies to the next answer. Still, check your model provider’s terms on how submitted data is used.

Does RAG eliminate hallucinations?

It reduces them but does not eliminate them. Citations, ongoing evaluation and the option to answer “I don’t know” are the main defences. For decisions with real impact, a person checks the source.

What is the difference between RAG and an AI agent?

RAG is a technique for retrieving information. An agent decides on steps and carries out actions through tools. Many agents use RAG as one of their tools. We cover this in AI agents for business.

How long does a pilot take?

It depends on the volume and state of the documentation. A focused pilot, with one source and one group of users, is usually measured in weeks rather than months. The most time-consuming part is often sorting out the sources and preparing the evaluation set.

Conclusion

RAG is the most practical way to make an AI answer with your company’s knowledge: it uses your current documents, cites the source and respects each user’s permissions. Quality depends more on the documentation, the chunking and the evaluation than on the model you pick.

LCApps is a Spain-based team building custom AI projects: RAG over documents, multichannel chatbots, agents and MCP servers connected to the systems you already use, always in line with the GDPR and the EU AI Act. Tell us which process you want to improve and we will help you design a realistic pilot.