What is Golden Dataset
A Golden Dataset (also called a golden set or gold standard) is a hand-curated, versioned collection of test cases — each containing a question, the correct answer, and the specific source documents/chunks that support that answer. It serves as the “answer key” or “ground truth” against which you evaluate your RAG system’s performance.
CN:
Golden Dataset（又称黄金数据集或金标准）是一个经过人工精心策划、带版本管理的测试用例集合——每个用例包含一个问题、正确答案，以及支撑该答案的具体源文档/分块。它充当评估RAG系统性能的”标准答案”或”地面真相”
Imagine you’re a teacher preparing a final exam. You don’t just write questions — you also create an answer key that tells you exactly what the correct response should be and which textbook pages the student should have used to find it. The Golden Dataset is that answer key for your RAG system. Without it, you’re grading essays by “feeling” — and that’s how hallucinations slip into production.
CN (人话比喻):
想象你是一位老师，在准备期末考试。你不仅写题目——还要准备一份标准答案，告诉你正确答案是什么，以及学生应该用教科书的哪一页来找到答案。Golden Dataset就是你的RAG系统的这份标准答案。没有它，你就是在靠”感觉”批改essay——这正是幻觉溜进生产环境的方式。
What Does a Golden Dataset Contain?
Golden Dataset包含什么？
EN:
A typical golden dataset entry (often stored as JSONL) contains:
| Field | Description | Required? |
|---|---|---|
question / query | The user’s input question | ✅ Required |
answer / expected_answer | The correct/ideal response | ✅ Required |
context / golden_chunks | The specific document chunks that contain the answer | ✅ Highly Recommended |
metadata | Difficulty, category, source, version, etc. | Optional |
CN:
一个典型的golden dataset条目（通常存储为JSONL）包含：
| 字段 | 描述 | 是否必需 |
|---|---|---|
question / query | 用户的输入问题 | ✅ 必需 |
answer / expected_answer | 正确的/理想的回答 | ✅ 必需 |
context / golden_chunks | 包含答案的具体文档分块 | ✅ 强烈推荐 |
metadata | 难度、类别、来源、版本等 | 可选 |
Key Takeaways
| 要点 | EN | CN |
|---|---|---|
| Golden Dataset是评估的地面真相 | Golden Dataset is the ground truth for evaluation | Golden Dataset是评估的地面真相 |
| 至少需要50-100个QA对 | Aim for at least 50-100 QA pairs | 目标至少50-100个QA对 |
| 从生产日志采样，不要凭空想象 | Sample from production logs, don’t invent | 从生产日志采样，不要凭空想象 |
| 人工标注是最关键也是最昂贵的步骤 | Human annotation is the most critical and most expensive step | 人工标注是最关键也是最昂贵的步骤 |
| 目标标注者间一致性 > 0.8 | Target inter-annotator agreement > 0.8 | 目标标注者间一致性 > 0.8 |
| 永远不要原地修改，始终版本管理 | Never modify in place, always version | 永远不要原地修改，始终版本管理 |
| 银级→金级：从自动化生成到人工验证 | Silver → Gold: from automated generation to human verification | 银级→金级：从自动化生成到人工验证 |
| JSONL是标准存储格式 | JSONL is the standard storage format | JSONL是标准存储格式 |
| 每个条目需要：问题、答案、支撑分块 | Each entry needs: question, answer, golden chunks | 每个条目需要：问题、答案、支撑分块 |
| 包含元数据：难度、类别、来源 | Include metadata: difficulty, category, source | 包含元数据：难度、类别、来源 |