← All notes

Short answer

Ragas is a set of metrics that scores a retrieval-augmented generation (RAG) system without human-labeled reference answers. It checks the retrieval step, whether the answer stays faithful to the retrieved passages, and the quality of the answer itself. [S1]

What it measures

• Retrieval: did the retriever find relevant, focused context passages? [S1]
• Faithfulness: does the LLM use those passages faithfully rather than inventing facts? [S1]
• Generation quality: is the answer itself good? [S1]

Why reference-free

Ground-truth answers are slow and costly to collect. Skipping them allows faster evaluation cycles, which the authors argue matters given how quickly LLM apps are being adopted. [S1]

Practical use

Run the metrics on a fixed question set whenever you change the chunking, the embedding model, the retriever settings or the prompt. Spot-check with human review before trusting the numbers for a release.

Claims

• id: C1 claim: Ragas is a framework for reference-free evaluation of RAG pipelines. source: S1 evidence: "a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines"
• id: C2 claim: RAG evaluation must consider whether the LLM exploits passages faithfully. source: S1 evidence: "the ability of the LLM to exploit such passages in a faithful way"
• id: C3 claim: The Ragas metrics do not rely on ground-truth human annotations. source: S1 evidence: "without having to rely on ground truth human annotations"

Sources

• [S1] RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv. https://arxiv.org/abs/2309.15217 . License: CC BY 4.0. Accessed 2026-11-05.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_djesmvps,
  author = {{iinventedlight}},
  title = {{RAGAS: reference-free evaluation of RAG pipelines}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/2309-15217-ragas-rag-evaluation/},
  note = {Permanent ID iil:note:djesmvps. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/2309-15217-ragas-rag-evaluation.md.