Evaluate retrieval and answers separately
A RAG system finds information and passes it to a model to help generate an answer. It can fail at finding the source or at using it. If you only look at the final text, it is hard to tell whether you need to improve the search, the context sent to the model or the way the answer is generated.
Microsoft's evaluation documentation separates retrieval quality, relevance and answer groundedness. That split is useful even if you never use the same evaluation tool. It lets you ask precise questions about each stage, instead of chasing a single score that hides very different problems.
Build a question set that reflects real usage
Collect the questions users actually need answered. For each one, note which sources can support the answer, who is allowed to read them and what counts as sufficient information. Include short, ambiguous questions, outdated documents and cases where the sources contain no answer.
Say an employee asks about the travel policy. An answer based on an old version can sound convincing and still be wrong for them. The example shows why it is worth recording dates, versions and organizational context, rather than settling for a linguistic match between question and document. Avoid moving personal or confidential data into a test environment that has not been approved.
Check whether the evidence reached the model
Before judging the answer, look at which passages were retrieved. Do they include the relevant instruction? Were they pulled out of context? Does one source keep showing up and crowd out everything else? A change in document chunking or ranking can shift the outcome even without swapping the model.
- The required source was retrieved and included in the context.
- The passage contains enough information to interpret the instruction.
- The information is current for the scenario and the user.
- No user receives a source they are not permitted to read.
Check grounding, usefulness and refusal to guess
Check whether the claims in the answer are supported by the retrieved information, and whether the answer actually addresses the question. A high groundedness score does not guarantee the user got the guidance they were looking for. A link to a source is not proof that the source supports every sentence. Go through the significant claims and cross-check them.
When information is missing, check that the system states the limitation and offers a sensible next step. It can ask for clarification, surface documents for review or hand the request to a person. The goal is not a complete answer at any cost. Using another model as an evaluator can help, but it also needs its reliability checked against manually reviewed examples.
Make evaluation part of the change process
Record versions of the sources, retrieval configuration, model and instructions. After a change, rerun the same set along with any new cases you have discovered. Decide which errors block a release and which need follow-up. That way you can understand why a result changed.
A practical starting point is a small but varied question set, together with a review of sources and answers. From there you can build an evaluation process that helps the team decide what to improve. A large number of similar questions is not necessarily a substitute for covering different scenarios.