Skip to content

About

No description, website, or topics provided.

Resources

Contributing

Security policy

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

RAG Evaluation with RAGAS and Azure OpenAI

RAG evaluation Python 3.11+ RAGAS 0.4.3 License MIT

A production-shaped starter project and learning reference for evaluating Retrieval-Augmented Generation (RAG) systems with RAGAS 0.4.3 and Azure OpenAI.

The repository separates retrieval quality, generation quality, and end-to-end quality. It includes a controlled company-policy knowledge base, Azure embedding retrieval, grounded answer generation, eight current RAGAS Collections metrics, 12 intentionally varied evaluation cases, pytest comparisons, configurable quality gates, regression analysis, experiment comparison, and GitHub Actions.

API verification: Last checked on 11 August 2026 against RAGAS 0.4.3, the latest PyPI release at verification time. The project uses the non-deprecated Collections API. See RAGAS API verification notes.

What is RAGAS?

RAGAS is an evaluation framework for LLM applications. For RAG, it helps answer questions such as:

  • Did retrieval return the right evidence?
  • Did retrieval miss evidence needed for a complete answer?
  • Did the generator use the evidence faithfully?
  • Did the answer address the user's question?
  • Does the answer agree with a trusted reference?
  • Did irrelevant context influence the response?

Normal unit tests are necessary, but they are not sufficient for RAG. Traditional assertions can verify deterministic code, schemas, ranking functions, and error handling. They cannot reliably judge whether a natural-language answer is grounded, complete, relevant, or semantically correct. RAG evaluation combines deterministic tests, controlled datasets, model-based metrics, embedding-based metrics, thresholds, regression comparison, and human review.

RAG evaluation architecture

flowchart TD
    U[User query] --> R[Retriever]
    R --> C[Retrieved context]
    U --> L[Generator LLM]
    C --> L
    L --> A[Generated answer]
    U --> E[RAGAS evaluation]
    C --> E
    A --> E
    G[Reference answer] --> E
    E --> S[Metric scores]
    S --> Q[Quality gate and reports]
Loading

Failures occur independently:

  • Retrieval failure: the correct evidence is missing or buried in noise.
  • Generation failure: the evidence is good, but the answer is unsupported, wrong, incomplete, irrelevant, or overly verbose.
  • End-to-end failure: the final answer does not satisfy the task, regardless of which stage caused it.

The evaluator therefore receives the exact question, contexts, answer, and reference used for one interaction. It does not reconstruct evidence after the answer has been generated.

What the project evaluates

Retrieval evaluation

Measures whether relevant chunks were retrieved, ranked well, and sufficient to support the reference answer.

  • Context Precision
  • Context Recall
  • Context Relevance

Generation evaluation

Assumes the context is already known and grades the generated answer.

  • Faithfulness
  • Answer Relevancy
  • Answer Correctness
  • Semantic Similarity

End-to-end and robustness evaluation

Uses retrieval evidence, the generated answer, and a human-approved reference to judge overall behavior.

  • Answer Correctness
  • Noise Sensitivity
  • Combined quality-gate decisions across retrieval and generation metrics

Metric comparison

Metric Evaluates Main question Reference? Judge LLM? Embeddings? Better
Faithfulness Generation Is the answer supported by context? No Yes No Higher
Answer Relevancy Generation Does the answer address the question? No Yes Yes Higher
Context Precision Retrieval Are useful chunks ranked ahead of noise? Yes Yes No Higher
Context Recall Retrieval Was all necessary evidence retrieved? Yes Yes No Higher
Answer Correctness End-to-end Does the answer agree with the reference? Yes Yes Yes Higher
Semantic Similarity Generation comparison Is meaning similar to the reference? Yes No Yes Higher
Context Relevance Retrieval Is the retrieved context pertinent overall? No Yes No Higher
Noise Sensitivity Robustness Did irrelevant context contribute to errors? Yes Yes No Lower

The complete metric guide explains every metric's purpose, inputs, interpretation, production use cases, common failure causes, and improvement options.

Answer relevancy, correctness, and faithfulness are different

  • A response can be relevant but unfaithful: it answers the question but invents a fact.
  • It can be faithful but not relevant: it repeats true context without answering the question.
  • It can be faithful but incorrect relative to a reference when the retrieved source itself is stale or wrong.
  • It can be correct but unsupported by the retrieved context, revealing a retrieval or traceability problem even if the final text happens to be right.

Precision and recall are different

Combination Meaning
High precision + high recall Necessary evidence with little noise—the desired state.
High precision + low recall Returned chunks are useful, but important evidence is missing.
Low precision + high recall Required evidence is present, but retrieval returns too much noise.
Low precision + low recall Retrieval misses evidence and adds distraction.

Current RAGAS API design

RAGAS v0.4 changed its recommended architecture. This repository intentionally uses:

from ragas.metrics.collections import Faithfulness

metric = Faithfulness(llm=judge_llm)
result = await metric.ascore(
    user_input=case.user_input,
    response=case.response,
    retrieved_contexts=case.retrieved_contexts,
)
score = result.value

It does not use older tutorials based on:

  • legacy metric imports from ragas.metrics;
  • single_turn_ascore(sample);
  • ground_truths instead of reference;
  • deprecated LangChain wrapper classes;
  • the v0.3 evaluate(dataset=..., metrics=...) orchestration pattern.

The controlled JSON data uses the RAGAS single-turn schema fields:

{
  "user_input": "What is the refund period?",
  "retrieved_contexts": [
    "Customers can request a full refund within 30 calendar days of purchase."
  ],
  "response": "Customers can request a full refund within 30 days.",
  "reference": "Customers may request a full refund within 30 calendar days."
}

Repository-only fields such as case_id, quality_label, and tags are not sent to metrics.

Azure OpenAI model roles

The implementation keeps these roles conceptually and technically separate:

Role Purpose Implementation
Application/generator LLM Produces the customer-facing answer Azure OpenAI SDK
Evaluation/judge LLM Applies RAGAS rubrics RAGAS llm_factory through LiteLLM Azure provider
Embedding model Powers local retrieval and embedding-based metrics Azure OpenAI SDK + RAGAS embedding_factory

For a simple demo, one chat deployment can serve both generator and judge. Production systems often use separate deployments to reduce self-judging bias, isolate capacity and budgets, lock a stable evaluator version, and compare generator migrations fairly.

Repository structure

ragas-rag-evaluation/
├── .github/workflows/rag-evaluation.yml
├── datasets/
│   ├── knowledge_base.json
│   ├── sample_questions.json
│   ├── sample_evaluation_dataset.json
│   └── README.md
├── docs/
│   ├── api-version-notes.md
│   ├── architecture.md
│   ├── metrics-guide.md
│   └── testing-strategy.md
├── examples/
│   ├── basic_evaluation.py
│   ├── retrieval_evaluation.py
│   ├── generation_evaluation.py
│   ├── end_to_end_evaluation.py
│   ├── custom_dataset_evaluation.py
│   ├── batch_evaluation.py
│   ├── noise_sensitivity_evaluation.py
│   ├── regression_comparison.py
│   └── experiment_comparison.py
├── reports/.gitkeep
├── scripts/offline_smoke.py
├── src/rag_evaluation/
│   ├── azure_models.py
│   ├── cli.py
│   ├── config.py
│   ├── dataset.py
│   ├── evaluator.py
│   ├── metrics.py
│   ├── models.py
│   ├── rag_pipeline.py
│   ├── regression.py
│   ├── reporting.py
│   ├── retriever.py
│   └── thresholds.py
├── tests/
├── .env.example
├── pyproject.toml
├── requirements.txt
└── README.md

Quick start

1. Clone

git clone https://github.com/ashokmanohar-ai/ragas-rag-evaluation.git
cd ragas-rag-evaluation

2. Create a virtual environment

python -m venv .venv

Windows PowerShell:

.venv\Scripts\Activate.ps1

Windows Command Prompt:

.venv\Scripts\activate.bat

macOS or Linux:

source .venv/bin/activate

3. Install

Runtime dependencies:

python -m pip install --upgrade pip
pip install -r requirements.txt

Development and pytest dependencies:

pip install -r requirements-dev.txt

Python 3.11 or 3.12 is recommended. RAGAS 0.4.3 itself declares Python 3.9+, while this starter uses modern Python typing and supports 3.11–3.13.

Azure OpenAI configuration

Copy the template:

macOS/Linux:

cp .env.example .env

Windows PowerShell:

Copy-Item .env.example .env

Edit .env:

AZURE_OPENAI_API_KEY=your-key
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_VERSION=your-supported-api-version
AZURE_OPENAI_CHAT_DEPLOYMENT=your-chat-deployment
AZURE_OPENAI_EMBEDDING_DEPLOYMENT=your-embedding-deployment

Optional role separation:

AZURE_OPENAI_GENERATOR_DEPLOYMENT=your-generator-deployment
AZURE_OPENAI_JUDGE_DEPLOYMENT=your-judge-deployment

Use deployment names, not underlying model names. Select an API version supported by your Azure resource and deployed models; the repository deliberately does not hard-code one that may become stale. Never commit .env.

Run evaluations

Each credentialed command makes Azure OpenAI calls and can incur usage charges.

Single controlled evaluation:

python examples/basic_evaluation.py

Retrieval evaluation:

python examples/retrieval_evaluation.py

Generation evaluation:

python examples/generation_evaluation.py

Live end-to-end RAG evaluation:

python examples/end_to_end_evaluation.py

Batch evaluation of all 12 cases:

python examples/batch_evaluation.py

Fail the process when aggregate thresholds are missed:

python examples/batch_evaluation.py --fail-on-threshold

Extended metrics, including context relevance, semantic similarity, and noise sensitivity:

python examples/batch_evaluation.py \
  --metrics faithfulness,answer_relevancy,context_precision,context_recall,answer_correctness,semantic_similarity,context_relevance,noise_sensitivity

Custom dataset:

python examples/custom_dataset_evaluation.py path/to/your_dataset.json

Generated reports:

  • reports/evaluation_results.json
  • reports/evaluation_results.csv
  • one row and score set per sample;
  • metric averages and overall quality-gate result.

The repository never hard-codes example output scores. It writes the values returned by RAGAS.

Sample RAG application

The fictional company knowledge base covers:

  • refund policy;
  • subscription plans;
  • password reset;
  • account cancellation;
  • data retention;
  • support hours;
  • shipping;
  • product warranty.

The demo uses Azure embeddings and in-memory cosine ranking—no Elasticsearch, Pinecone, Weaviate, Kubernetes, or database is required. A deterministic keyword retriever is included for offline tests and baseline comparison.

End-to-end execution follows this sequence:

  1. Load local company-policy documents.
  2. Embed them with the Azure embedding deployment.
  3. Embed the user question and rank documents by cosine similarity.
  4. Send the top contexts to the Azure generator deployment.
  5. Construct an EvaluationCase with the exact contexts actually used.
  6. Score it using the Azure judge and RAGAS Collections metrics.
  7. Apply thresholds and save a readable report.

Good and bad evaluation examples

The 12-case dataset intentionally includes:

  1. Correct answer + correct context
  2. Hallucinated answer
  3. Correct answer + noisy context
  4. Correct but incomplete answer + incomplete context
  5. Incorrect answer + correct context
  6. Irrelevant answer
  7. Highly relevant answer
  8. Missing retrieval evidence
  9. Too many irrelevant retrieved chunks
  10. Partially correct answer
  11. Verbose but poorly focused answer
  12. Correct answer that resists irrelevant noise

These controlled contrasts make metric behavior easier to understand. A quality label is not an expected score; the judge still calculates actual values.

Quality thresholds

Example thresholds are environment-configurable:

FAITHFULNESS_THRESHOLD=0.80
ANSWER_RELEVANCY_THRESHOLD=0.75
CONTEXT_PRECISION_THRESHOLD=0.70
CONTEXT_RECALL_THRESHOLD=0.75
ANSWER_CORRECTNESS_THRESHOLD=0.75
SEMANTIC_SIMILARITY_THRESHOLD=0.70
CONTEXT_RELEVANCE_THRESHOLD=0.70
NOISE_SENSITIVITY_THRESHOLD=0.20

ThresholdPolicy understands metric direction. Most metrics pass when score >= threshold; noise sensitivity passes when score <= threshold.

These values are examples, not universal standards. Calibrate production thresholds with representative, human-reviewed application data. Regulated or safety-critical domains may need stricter thresholds, segment-specific rules, and mandatory human review.

The batch command applies thresholds to metric averages. Production gates should also inspect critical individual failures, tail percentiles, business segments, and data coverage rather than allowing averages to hide severe cases.

Pytest integration

Deterministic tests do not make model calls:

pytest -m "not integration" -v

Credentialed tests are opt-in:

macOS/Linux:

RUN_RAGAS_INTEGRATION_TESTS=1 pytest -m integration -v

Windows PowerShell:

$env:RUN_RAGAS_INTEGRATION_TESTS = "1"
pytest -m integration -v

The metric tests compare clearly good and bad cases instead of depending on exact floating-point scores. This is more stable while still catching meaningful degradation.

CI/CD with GitHub Actions

The workflow has two jobs:

  1. Offline quality: runs lint, schema validation, API-contract imports, retriever tests, and threshold/regression tests on pull requests and main.
  2. Credentialed RAGAS quality gate: runs Azure integration comparisons and the batch threshold gate on main or manual dispatch.

Configure these GitHub Secrets:

  • AZURE_OPENAI_API_KEY
  • AZURE_OPENAI_ENDPOINT
  • AZURE_OPENAI_API_VERSION
  • AZURE_OPENAI_CHAT_DEPLOYMENT
  • AZURE_OPENAI_EMBEDDING_DEPLOYMENT
  • optional AZURE_OPENAI_GENERATOR_DEPLOYMENT
  • optional AZURE_OPENAI_JUDGE_DEPLOYMENT

The workflow validates that required values exist without printing them. Use a protected GitHub Environment named rag-evaluation when approvals or restricted secret access are required.

Regression testing

RAG quality can regress after changes to:

  • chunk size or overlap;
  • embedding models;
  • retrieval top-k;
  • metadata filtering or hybrid search;
  • prompts and model deployments;
  • reranking;
  • knowledge-base content;
  • query rewriting.

Create a baseline:

python examples/batch_evaluation.py --json-report reports/baseline.json

Run a candidate after changing one controlled variable:

python examples/batch_evaluation.py --json-report reports/candidate.json

Compare them:

python examples/regression_comparison.py \
  reports/baseline.json \
  reports/candidate.json \
  --tolerance 0.03

The script reads actual reports. It does not embed illustrative scores in the code. A candidate is flagged when a higher-is-better metric drops beyond tolerance or noise sensitivity rises beyond tolerance.

Experiment comparison

The experiment example evaluates the same questions using:

  • Experiment A: chunk_size=500, top_k=3
  • Experiment B: chunk_size=1000, top_k=5

Run a small comparison:

python examples/experiment_comparison.py --limit 3

It creates the two RAG configurations, generates answers, executes RAGAS, saves separate reports, and prints metric deltas. This provides evidence for configuration selection rather than relying on intuition. For a rigorous experiment:

  • use the same immutable dataset;
  • change only the variables under investigation;
  • include enough representative examples;
  • repeat judge runs when variance matters;
  • compare sample-level failures as well as averages;
  • record model, prompt, data, and configuration versions.

Common diagnostic combinations

High faithfulness + low answer relevancy

The model did not hallucinate, but it failed to answer the user's actual question effectively. Review prompt focus, context relevance, and response-format instructions.

Low faithfulness + high answer relevancy

The answer sounds relevant but contains unsupported claims. Strengthen grounding and determine whether retrieval omitted needed evidence.

High context recall + low context precision

The retriever found the necessary evidence but returned substantial irrelevant material. Reduce top-k, improve filters, or add reranking.

Low context recall + high faithfulness

The generator faithfully used the available context, but retrieval failed to supply enough information. Fix retrieval or the knowledge base before blaming the generator.

RAG debugging guide

Low metric Investigate first
Faithfulness Grounding prompt, unsupported claims, missing evidence, context formatting, temperature, conflicting sources
Answer Relevancy Prompt focus, retrieval relevance, excessive context, verbosity, response instructions
Context Precision Embedding quality, top-k, chunk size/overlap, filters, reranking, query rewriting
Context Recall Missing documents, search query, top-k, chunking, embeddings, overly restrictive filters
Answer Correctness Reference quality, incomplete retrieval, generator prompt/model, ambiguous questions
Semantic Similarity Embedding suitability, different but valid phrasing, critical negations/numbers hidden by broad similarity
Context Relevance Off-topic retrieval, broad chunks, poor query representation, missing metadata filters
Noise Sensitivity Contradictory/noisy chunks, absent reranking, prompt trust assumptions, source authority and recency

Debug from the failure stage outward. For example, low context recall plus high faithfulness is a retrieval problem; making the generator more creative can worsen it.

Advanced uses

The same evaluation layer can support:

  • Offline evaluation: run curated or sampled datasets before release.
  • Continuous evaluation: evaluate governed production samples asynchronously.
  • Prompt comparison: hold data and models constant while changing prompt instructions.
  • Retriever comparison: compare vector, keyword, hybrid, and reranked results.
  • Embedding migration: test new embedding deployments on an unchanged golden set.
  • Chunk-size and top-k tuning: isolate retrieval configuration changes.
  • Reranker evaluation: measure precision improvement without losing recall.
  • Model migration: compare generator versions using a stable judge and references.
  • Knowledge-base quality tests: detect missing, stale, conflicting, or poorly segmented content.
  • Segmented analysis: track domain, language, intent, customer tier, or risk level separately.

RAGAS scores should be one part of an evaluation program. Add deterministic business rules, security and safety evaluations, latency, cost, human review, and observability appropriate to your system.

Security and privacy

Never:

  • commit API keys or .env;
  • hard-code credentials in source code;
  • print authentication headers or secrets;
  • include secrets in datasets or reports;
  • upload confidential production reports to public artifacts;
  • assume model-evaluation payloads are non-sensitive.

Reports may contain user questions, retrieved documents, generated answers, and references. Apply redaction, encryption, retention, access control, and regional-processing rules. Read SECURITY.md before adapting the project to production data.

Cost and stability considerations

Some RAGAS metrics make multiple judge calls per sample. Answer relevancy generates multiple questions; faithfulness decomposes and verifies claims; context precision evaluates contexts; noise sensitivity performs several statement comparisons. A full 12-case, eight-metric run can be materially more expensive than a single application request.

Practical controls:

  • start with a small representative smoke set;
  • run expensive extended metrics on scheduled or pre-release gates;
  • use stable judge deployments and conservative concurrency;
  • cache evaluation results when inputs and evaluator version are unchanged;
  • monitor tokens, latency, rate limits, retries, and cost;
  • avoid treating a failed provider call as a quality score.

Adapting to a real RAG system

Keep your application and evaluation code separate. Replace the demo retriever or generator while preserving the evaluator contract:

case = EvaluationCase(
    case_id=trace_id,
    user_input=user_question,
    retrieved_contexts=contexts_used_for_generation,
    response=generated_answer,
    reference=approved_reference,
)

result = await evaluator.evaluate_case(case)

See architecture and extension guide and testing strategy.

Authoritative references

License

MIT. See LICENSE.

About

No description, website, or topics provided.

Resources

Contributing

Security policy

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages