What is RAG Evaluation and ragas?
RAG Evaluation is the systematic process of measuring how well your Retrieval-Augmented Generation pipeline performs. ragas (Retrieval Augmented Generation Assessment) is an open-source framework specifically built for evaluating RAG systems without requiring manual human rating of every answer.
CN: RAG评估是系统性地衡量你的检索增强生成管道表现如何的过程。ragas（检索增强生成评估）是一个专为评估RAG系统而构建的开源框架，无需人工对每个答案进行评分。
ragas Metrics System Explained
The Core Four Metrics
Detailed explanation
EN: Faithfulness – Scores from 0.0 to 1.0. Measures whether the generated answer is grounded in the retrieved context. If the LLM says something not in the context, faithfulness drops. This is your primary hallucination detector.
CN: 忠实性 – 得分0.0到1.0。衡量生成的答案是否基于检索到的上下文。 如果LLM说了上下文中没有的内容，忠实性就会下降。这是你主要的幻觉检测器。
EN: Answer Relevancy – Measures how pertinent the generated answer is to the given prompt. Low scores mean the answer is incomplete or contains redundant information.
CN: 答案相关性 – 衡量生成的答案与给定提示的相关程度。低分意味着答案不完整或包含冗余信息。
EN: Context Precision – Measures whether all relevant items retrieved are ranked higher than irrelevant ones. This evaluates your retrieval pipeline’s ranking quality.
CN: 上下文精确度 – 衡量所有检索到的相关项是否排在无关项之前。 这评估你的检索管道的排序质量。
EN: Context Recall – Estimates how many of the relevant chunks were actually retrieved. This evaluates your retrieval pipeline’s coverage.
CN: 上下文召回率 – 估算实际检索到了多少相关块。 这评估你的检索管道的覆盖率。
Key Takeaways
| 要点 (Key Point) | EN | CN |
|---|---|---|
| ragas是什么 | ragas is an open-source evaluation framework for RAG systems that uses LLM-as-judge | ragas是一个开源的RAG系统评估框架，使用LLM作为评委 |
| 四大核心指标 | The four core metrics are Faithfulness, Answer Relevancy, Context Precision, and Context Recall | 四大核心指标是：忠实性、答案相关性、上下文精确度、上下文召回率 |
| Faithfulness用途 | Faithfulness detects hallucinations by checking if answer is grounded in context | 忠实性通过检查答案是否基于上下文来检测幻觉 |
| Context Precision用途 | Context Precision measures retrieval ranking quality | 上下文精确度衡量检索排序质量 |
| Context Recall用途 | Context Recall measures retrieval coverage (requires ground_truth) | 上下文召回率衡量检索覆盖率（需要ground_truth） |
| 评估数据格式 | Evaluation data needs question, contexts (list), answer; ground_truth is optional | 评估数据需要question、contexts（列表）、answer；ground_truth可选 |
| LLM-as-judge | ragas uses an LLM (e.g., gpt-4o) as a judge to score metrics | ragas使用LLM（如gpt-4o）作为评委来给指标打分 |
| 开始简单 | Start with simple, focused metrics that directly measure your core use case | 从简单、聚焦的指标开始，直接衡量你的核心用例 |
| Eval First原则 | Evaluate before and after every change to catch regressions | 每次改动前后都要评估，以捕捉回归问题 |