A production-shaped starter project and learning reference for evaluating Retrieval-Augmented Generation (RAG) systems with RAGAS 0.4.3 and Azure OpenAI.
The repository separates retrieval quality, generation quality, and end-to-end quality. It includes a controlled company-policy knowledge base, Azure embedding retrieval, grounded answer generation, eight current RAGAS Collections metrics, 12 intentionally varied evaluation cases, pytest comparisons, configurable quality gates, regression analysis, experiment comparison, and GitHub Actions.
API verification: Last checked on 11 August 2026 against RAGAS 0.4.3, the latest PyPI release at verification time. The project uses the non-deprecated Collections API. See RAGAS API verification notes.
RAGAS is an evaluation framework for LLM applications. For RAG, it helps answer questions such as:
- Did retrieval return the right evidence?
- Did retrieval miss evidence needed for a complete answer?
- Did the generator use the evidence faithfully?
- Did the answer address the user's question?
- Does the answer agree with a trusted reference?
- Did irrelevant context influence the response?
Normal unit tests are necessary, but they are not sufficient for RAG. Traditional assertions can verify deterministic code, schemas, ranking functions, and error handling. They cannot reliably judge whether a natural-language answer is grounded, complete, relevant, or semantically correct. RAG evaluation combines deterministic tests, controlled datasets, model-based metrics, embedding-based metrics, thresholds, regression comparison, and human review.
flowchart TD
U[User query] --> R[Retriever]
R --> C[Retrieved context]
U --> L[Generator LLM]
C --> L
L --> A[Generated answer]
U --> E[RAGAS evaluation]
C --> E
A --> E
G[Reference answer] --> E
E --> S[Metric scores]
S --> Q[Quality gate and reports]
Failures occur independently:
- Retrieval failure: the correct evidence is missing or buried in noise.
- Generation failure: the evidence is good, but the answer is unsupported, wrong, incomplete, irrelevant, or overly verbose.
- End-to-end failure: the final answer does not satisfy the task, regardless of which stage caused it.
The evaluator therefore receives the exact question, contexts, answer, and reference used for one interaction. It does not reconstruct evidence after the answer has been generated.
Measures whether relevant chunks were retrieved, ranked well, and sufficient to support the reference answer.
- Context Precision
- Context Recall
- Context Relevance
Assumes the context is already known and grades the generated answer.
- Faithfulness
- Answer Relevancy
- Answer Correctness
- Semantic Similarity
Uses retrieval evidence, the generated answer, and a human-approved reference to judge overall behavior.
- Answer Correctness
- Noise Sensitivity
- Combined quality-gate decisions across retrieval and generation metrics
| Metric | Evaluates | Main question | Reference? | Judge LLM? | Embeddings? | Better |
|---|---|---|---|---|---|---|
| Faithfulness | Generation | Is the answer supported by context? | No | Yes | No | Higher |
| Answer Relevancy | Generation | Does the answer address the question? | No | Yes | Yes | Higher |
| Context Precision | Retrieval | Are useful chunks ranked ahead of noise? | Yes | Yes | No | Higher |
| Context Recall | Retrieval | Was all necessary evidence retrieved? | Yes | Yes | No | Higher |
| Answer Correctness | End-to-end | Does the answer agree with the reference? | Yes | Yes | Yes | Higher |
| Semantic Similarity | Generation comparison | Is meaning similar to the reference? | Yes | No | Yes | Higher |
| Context Relevance | Retrieval | Is the retrieved context pertinent overall? | No | Yes | No | Higher |
| Noise Sensitivity | Robustness | Did irrelevant context contribute to errors? | Yes | Yes | No | Lower |
The complete metric guide explains every metric's purpose, inputs, interpretation, production use cases, common failure causes, and improvement options.
- A response can be relevant but unfaithful: it answers the question but invents a fact.
- It can be faithful but not relevant: it repeats true context without answering the question.
- It can be faithful but incorrect relative to a reference when the retrieved source itself is stale or wrong.
- It can be correct but unsupported by the retrieved context, revealing a retrieval or traceability problem even if the final text happens to be right.
| Combination | Meaning |
|---|---|
| High precision + high recall | Necessary evidence with little noise—the desired state. |
| High precision + low recall | Returned chunks are useful, but important evidence is missing. |
| Low precision + high recall | Required evidence is present, but retrieval returns too much noise. |
| Low precision + low recall | Retrieval misses evidence and adds distraction. |
RAGAS v0.4 changed its recommended architecture. This repository intentionally uses:
from ragas.metrics.collections import Faithfulness
metric = Faithfulness(llm=judge_llm)
result = await metric.ascore(
user_input=case.user_input,
response=case.response,
retrieved_contexts=case.retrieved_contexts,
)
score = result.valueIt does not use older tutorials based on:
- legacy metric imports from
ragas.metrics; single_turn_ascore(sample);ground_truthsinstead ofreference;- deprecated LangChain wrapper classes;
- the v0.3
evaluate(dataset=..., metrics=...)orchestration pattern.
The controlled JSON data uses the RAGAS single-turn schema fields:
{
"user_input": "What is the refund period?",
"retrieved_contexts": [
"Customers can request a full refund within 30 calendar days of purchase."
],
"response": "Customers can request a full refund within 30 days.",
"reference": "Customers may request a full refund within 30 calendar days."
}Repository-only fields such as case_id, quality_label, and tags are not sent to metrics.
The implementation keeps these roles conceptually and technically separate:
| Role | Purpose | Implementation |
|---|---|---|
| Application/generator LLM | Produces the customer-facing answer | Azure OpenAI SDK |
| Evaluation/judge LLM | Applies RAGAS rubrics | RAGAS llm_factory through LiteLLM Azure provider |
| Embedding model | Powers local retrieval and embedding-based metrics | Azure OpenAI SDK + RAGAS embedding_factory |
For a simple demo, one chat deployment can serve both generator and judge. Production systems often use separate deployments to reduce self-judging bias, isolate capacity and budgets, lock a stable evaluator version, and compare generator migrations fairly.
ragas-rag-evaluation/
├── .github/workflows/rag-evaluation.yml
├── datasets/
│ ├── knowledge_base.json
│ ├── sample_questions.json
│ ├── sample_evaluation_dataset.json
│ └── README.md
├── docs/
│ ├── api-version-notes.md
│ ├── architecture.md
│ ├── metrics-guide.md
│ └── testing-strategy.md
├── examples/
│ ├── basic_evaluation.py
│ ├── retrieval_evaluation.py
│ ├── generation_evaluation.py
│ ├── end_to_end_evaluation.py
│ ├── custom_dataset_evaluation.py
│ ├── batch_evaluation.py
│ ├── noise_sensitivity_evaluation.py
│ ├── regression_comparison.py
│ └── experiment_comparison.py
├── reports/.gitkeep
├── scripts/offline_smoke.py
├── src/rag_evaluation/
│ ├── azure_models.py
│ ├── cli.py
│ ├── config.py
│ ├── dataset.py
│ ├── evaluator.py
│ ├── metrics.py
│ ├── models.py
│ ├── rag_pipeline.py
│ ├── regression.py
│ ├── reporting.py
│ ├── retriever.py
│ └── thresholds.py
├── tests/
├── .env.example
├── pyproject.toml
├── requirements.txt
└── README.md
git clone https://github.com/ashokmanohar-ai/ragas-rag-evaluation.git
cd ragas-rag-evaluationpython -m venv .venvWindows PowerShell:
.venv\Scripts\Activate.ps1Windows Command Prompt:
.venv\Scripts\activate.batmacOS or Linux:
source .venv/bin/activateRuntime dependencies:
python -m pip install --upgrade pip
pip install -r requirements.txtDevelopment and pytest dependencies:
pip install -r requirements-dev.txtPython 3.11 or 3.12 is recommended. RAGAS 0.4.3 itself declares Python 3.9+, while this starter uses modern Python typing and supports 3.11–3.13.
Copy the template:
macOS/Linux:
cp .env.example .envWindows PowerShell:
Copy-Item .env.example .envEdit .env:
AZURE_OPENAI_API_KEY=your-key
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_VERSION=your-supported-api-version
AZURE_OPENAI_CHAT_DEPLOYMENT=your-chat-deployment
AZURE_OPENAI_EMBEDDING_DEPLOYMENT=your-embedding-deploymentOptional role separation:
AZURE_OPENAI_GENERATOR_DEPLOYMENT=your-generator-deployment
AZURE_OPENAI_JUDGE_DEPLOYMENT=your-judge-deploymentUse deployment names, not underlying model names. Select an API version supported by your Azure
resource and deployed models; the repository deliberately does not hard-code one that may become
stale. Never commit .env.
Each credentialed command makes Azure OpenAI calls and can incur usage charges.
Single controlled evaluation:
python examples/basic_evaluation.pyRetrieval evaluation:
python examples/retrieval_evaluation.pyGeneration evaluation:
python examples/generation_evaluation.pyLive end-to-end RAG evaluation:
python examples/end_to_end_evaluation.pyBatch evaluation of all 12 cases:
python examples/batch_evaluation.pyFail the process when aggregate thresholds are missed:
python examples/batch_evaluation.py --fail-on-thresholdExtended metrics, including context relevance, semantic similarity, and noise sensitivity:
python examples/batch_evaluation.py \
--metrics faithfulness,answer_relevancy,context_precision,context_recall,answer_correctness,semantic_similarity,context_relevance,noise_sensitivityCustom dataset:
python examples/custom_dataset_evaluation.py path/to/your_dataset.jsonGenerated reports:
reports/evaluation_results.jsonreports/evaluation_results.csv- one row and score set per sample;
- metric averages and overall quality-gate result.
The repository never hard-codes example output scores. It writes the values returned by RAGAS.
The fictional company knowledge base covers:
- refund policy;
- subscription plans;
- password reset;
- account cancellation;
- data retention;
- support hours;
- shipping;
- product warranty.
The demo uses Azure embeddings and in-memory cosine ranking—no Elasticsearch, Pinecone, Weaviate, Kubernetes, or database is required. A deterministic keyword retriever is included for offline tests and baseline comparison.
End-to-end execution follows this sequence:
- Load local company-policy documents.
- Embed them with the Azure embedding deployment.
- Embed the user question and rank documents by cosine similarity.
- Send the top contexts to the Azure generator deployment.
- Construct an
EvaluationCasewith the exact contexts actually used. - Score it using the Azure judge and RAGAS Collections metrics.
- Apply thresholds and save a readable report.
The 12-case dataset intentionally includes:
- Correct answer + correct context
- Hallucinated answer
- Correct answer + noisy context
- Correct but incomplete answer + incomplete context
- Incorrect answer + correct context
- Irrelevant answer
- Highly relevant answer
- Missing retrieval evidence
- Too many irrelevant retrieved chunks
- Partially correct answer
- Verbose but poorly focused answer
- Correct answer that resists irrelevant noise
These controlled contrasts make metric behavior easier to understand. A quality label is not an expected score; the judge still calculates actual values.
Example thresholds are environment-configurable:
FAITHFULNESS_THRESHOLD=0.80
ANSWER_RELEVANCY_THRESHOLD=0.75
CONTEXT_PRECISION_THRESHOLD=0.70
CONTEXT_RECALL_THRESHOLD=0.75
ANSWER_CORRECTNESS_THRESHOLD=0.75
SEMANTIC_SIMILARITY_THRESHOLD=0.70
CONTEXT_RELEVANCE_THRESHOLD=0.70
NOISE_SENSITIVITY_THRESHOLD=0.20ThresholdPolicy understands metric direction. Most metrics pass when score >= threshold;
noise sensitivity passes when score <= threshold.
These values are examples, not universal standards. Calibrate production thresholds with representative, human-reviewed application data. Regulated or safety-critical domains may need stricter thresholds, segment-specific rules, and mandatory human review.
The batch command applies thresholds to metric averages. Production gates should also inspect critical individual failures, tail percentiles, business segments, and data coverage rather than allowing averages to hide severe cases.
Deterministic tests do not make model calls:
pytest -m "not integration" -vCredentialed tests are opt-in:
macOS/Linux:
RUN_RAGAS_INTEGRATION_TESTS=1 pytest -m integration -vWindows PowerShell:
$env:RUN_RAGAS_INTEGRATION_TESTS = "1"
pytest -m integration -vThe metric tests compare clearly good and bad cases instead of depending on exact floating-point scores. This is more stable while still catching meaningful degradation.
The workflow has two jobs:
- Offline quality: runs lint, schema validation, API-contract imports, retriever tests, and
threshold/regression tests on pull requests and
main. - Credentialed RAGAS quality gate: runs Azure integration comparisons and the batch threshold
gate on
mainor manual dispatch.
Configure these GitHub Secrets:
AZURE_OPENAI_API_KEYAZURE_OPENAI_ENDPOINTAZURE_OPENAI_API_VERSIONAZURE_OPENAI_CHAT_DEPLOYMENTAZURE_OPENAI_EMBEDDING_DEPLOYMENT- optional
AZURE_OPENAI_GENERATOR_DEPLOYMENT - optional
AZURE_OPENAI_JUDGE_DEPLOYMENT
The workflow validates that required values exist without printing them. Use a protected GitHub
Environment named rag-evaluation when approvals or restricted secret access are required.
RAG quality can regress after changes to:
- chunk size or overlap;
- embedding models;
- retrieval top-k;
- metadata filtering or hybrid search;
- prompts and model deployments;
- reranking;
- knowledge-base content;
- query rewriting.
Create a baseline:
python examples/batch_evaluation.py --json-report reports/baseline.jsonRun a candidate after changing one controlled variable:
python examples/batch_evaluation.py --json-report reports/candidate.jsonCompare them:
python examples/regression_comparison.py \
reports/baseline.json \
reports/candidate.json \
--tolerance 0.03The script reads actual reports. It does not embed illustrative scores in the code. A candidate is flagged when a higher-is-better metric drops beyond tolerance or noise sensitivity rises beyond tolerance.
The experiment example evaluates the same questions using:
- Experiment A:
chunk_size=500,top_k=3 - Experiment B:
chunk_size=1000,top_k=5
Run a small comparison:
python examples/experiment_comparison.py --limit 3It creates the two RAG configurations, generates answers, executes RAGAS, saves separate reports, and prints metric deltas. This provides evidence for configuration selection rather than relying on intuition. For a rigorous experiment:
- use the same immutable dataset;
- change only the variables under investigation;
- include enough representative examples;
- repeat judge runs when variance matters;
- compare sample-level failures as well as averages;
- record model, prompt, data, and configuration versions.
The model did not hallucinate, but it failed to answer the user's actual question effectively. Review prompt focus, context relevance, and response-format instructions.
The answer sounds relevant but contains unsupported claims. Strengthen grounding and determine whether retrieval omitted needed evidence.
The retriever found the necessary evidence but returned substantial irrelevant material. Reduce top-k, improve filters, or add reranking.
The generator faithfully used the available context, but retrieval failed to supply enough information. Fix retrieval or the knowledge base before blaming the generator.
| Low metric | Investigate first |
|---|---|
| Faithfulness | Grounding prompt, unsupported claims, missing evidence, context formatting, temperature, conflicting sources |
| Answer Relevancy | Prompt focus, retrieval relevance, excessive context, verbosity, response instructions |
| Context Precision | Embedding quality, top-k, chunk size/overlap, filters, reranking, query rewriting |
| Context Recall | Missing documents, search query, top-k, chunking, embeddings, overly restrictive filters |
| Answer Correctness | Reference quality, incomplete retrieval, generator prompt/model, ambiguous questions |
| Semantic Similarity | Embedding suitability, different but valid phrasing, critical negations/numbers hidden by broad similarity |
| Context Relevance | Off-topic retrieval, broad chunks, poor query representation, missing metadata filters |
| Noise Sensitivity | Contradictory/noisy chunks, absent reranking, prompt trust assumptions, source authority and recency |
Debug from the failure stage outward. For example, low context recall plus high faithfulness is a retrieval problem; making the generator more creative can worsen it.
The same evaluation layer can support:
- Offline evaluation: run curated or sampled datasets before release.
- Continuous evaluation: evaluate governed production samples asynchronously.
- Prompt comparison: hold data and models constant while changing prompt instructions.
- Retriever comparison: compare vector, keyword, hybrid, and reranked results.
- Embedding migration: test new embedding deployments on an unchanged golden set.
- Chunk-size and top-k tuning: isolate retrieval configuration changes.
- Reranker evaluation: measure precision improvement without losing recall.
- Model migration: compare generator versions using a stable judge and references.
- Knowledge-base quality tests: detect missing, stale, conflicting, or poorly segmented content.
- Segmented analysis: track domain, language, intent, customer tier, or risk level separately.
RAGAS scores should be one part of an evaluation program. Add deterministic business rules, security and safety evaluations, latency, cost, human review, and observability appropriate to your system.
Never:
- commit API keys or
.env; - hard-code credentials in source code;
- print authentication headers or secrets;
- include secrets in datasets or reports;
- upload confidential production reports to public artifacts;
- assume model-evaluation payloads are non-sensitive.
Reports may contain user questions, retrieved documents, generated answers, and references. Apply redaction, encryption, retention, access control, and regional-processing rules. Read SECURITY.md before adapting the project to production data.
Some RAGAS metrics make multiple judge calls per sample. Answer relevancy generates multiple questions; faithfulness decomposes and verifies claims; context precision evaluates contexts; noise sensitivity performs several statement comparisons. A full 12-case, eight-metric run can be materially more expensive than a single application request.
Practical controls:
- start with a small representative smoke set;
- run expensive extended metrics on scheduled or pre-release gates;
- use stable judge deployments and conservative concurrency;
- cache evaluation results when inputs and evaluator version are unchanged;
- monitor tokens, latency, rate limits, retries, and cost;
- avoid treating a failed provider call as a quality score.
Keep your application and evaluation code separate. Replace the demo retriever or generator while preserving the evaluator contract:
case = EvaluationCase(
case_id=trace_id,
user_input=user_question,
retrieved_contexts=contexts_used_for_generation,
response=generated_answer,
reference=approved_reference,
)
result = await evaluator.evaluate_case(case)See architecture and extension guide and testing strategy.
- RAGAS 0.4.3 release on PyPI
- RAGAS v0.4 migration guide
- RAGAS available metrics
- RAGAS Azure OpenAI model configuration
- RAGAS evaluation schema
- Azure OpenAI Python client documentation
MIT. See LICENSE.