RAG stands for Retrieval-Augmented Generation. It’s a technique that combines information retrieval with LLM generation — instead of asking the LLM to answer from memory alone, you first retrieve relevant documents from an external knowledge base, then feed those documents as context to the LLM, and let the LLM generate an answer based on that context.
Why It Matters?
Traditional LLMs have four fatal flaws that RAG solves:
Flaw
How RAG Solves It
Training Cutoff — LLM only knows data up to its training date
RAG retrieves live documents at query time
No Private Data Access — LLM doesn’t know your internal docs, Slack, Jira
RAG connects to your private knowledge base
Hallucinations — LLM makes up plausible but wrong answers
RAG grounds answers in retrieved facts
Context Window Limits — can’t paste entire company wiki
RAG retrieves only the most relevant chunks
Simple RAG Pipeline
A simple RAG pipeline has two phases and five stages:
Phase 1: Indexing (Offline / 离线阶段) Build the searchable knowledge base before users ask questions. 离线阶段（索引）：先把所有文档加载进来，切分成小块，调用 Embedding 模型转成向量，最后存进 FAISS 等向量数据库里。
Phase 2: Retrieval & Generation (Online / 在线阶段) Execute at query time for each user question. 在线阶段（检索+生成）：用户提问时，先把问题也转成向量，去库里找最相似的 Top-K 个块，把这几个块作为“参考资料”连同问题一起扔给 GPT，让它写出最终回答。
Stage-by-Stage Breakdown
Stage
EN
CN
What happens
1. Load
Load documents
加载文档
Read PDFs, TXT, HTML, DB records
2. Split
Chunk documents
切分文档
Split long docs into smaller semantic pieces
3. Embed
Convert to vectors
向量化
Turn text chunks into numerical vectors
4. Store
Store in vector DB
存储向量
Save vectors in FAISS/Chroma/Milvus
5. Retrieve + Generate
Search + Answer
检索+生成
Query → search similar vectors → LLM answers
Code Implementation
Below is a complete, self-contained simple RAG pipeline. We’ll use:
LangChain for orchestration
FAISS as vector database (local, no server needed)
OpenAI for embeddings and chat completion
Runtime Sequence
#!/usr/bin/env python3
"""
================================================================================
B02 升级版：完整集成 (Code A + Code B + 智能缓存)
================================================================================
这是一个 100% 完整的、可直接复制粘贴运行的 Python 脚本。
包含：
1. 数据加载 (Load)
2. 文本切分 (Split)
3. FAISS 专业配置 (支持 Flat / IVF 索引，带持久化)
4. RAG 检索生成链 (Retrieve + Generate)
5. 智能缓存逻辑 (首次构建，后续秒级加载)
使用方法：
1. 安装依赖：pip install faiss-cpu langchain langchain-community langchain-openai openai numpy
2. 设置环境变量：export OPENAI_API_KEY="你的Key"
3. 运行：python rag_complete.py
================================================================================
"""
# ============================================================
# 0. 依赖导入 (全部列出，一个不少)
# ============================================================
import os
import tempfile
import shutil
from typing import List, Dict, Any, Optional
import numpy as np
import faiss
# LangChain 核心
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough, RunnableLambda
# LangChain 社区
from langchain_community.document_loaders import TextLoader
from langchain_community.vectorstores import FAISS
from langchain_community.embeddings import OpenAIEmbeddings
from langchain_community.vectorstores.utils import DistanceStrategy
# LangChain 集成
from langchain_openai import ChatOpenAI
# 文本切分
from langchain_text_splitters import RecursiveCharacterTextSplitter
# ============================================================
# 第一部分：CODE A —— 原版 RAG 流程 (加载、切分、生成链)
# ============================================================
def load_documents(file_paths: List[str]) -> List[Document]:
"""
从文本文件加载文档。
"""
all_docs: List[Document] = []
for path in file_paths:
loader = TextLoader(path, encoding="utf-8")
docs = loader.load()
all_docs.extend(docs)
print(f"✅ 加载完成: {path} ({len(docs)} 个文档)")
return all_docs
def chunk_documents(documents: List[Document]) -> List[Document]:
"""
将大文档切分成语义块。
使用 RecursiveCharacterTextSplitter 保证上下文连贯。
"""
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000, # 每块最大字符数
chunk_overlap=200, # 块间重叠，保留跨块上下文
length_function=len,
separators=["\n\n", "\n", " ", ""],
add_start_index=True, # 标记在原文中的位置
)
chunks = text_splitter.split_documents(documents)
print(f"✅ 切分完成: {len(documents)} 个文档 → {len(chunks)} 个块")
return chunks
def retrieve_context(vector_store: FAISS, query: str, k: int = 4) -> List[Document]:
"""
从向量库中检索最相似的 k 个块。
"""
retrieved_docs = vector_store.similarity_search(query, k=k)
print(f"🔍 检索到 {len(retrieved_docs)} 个块")
return retrieved_docs
def format_context(documents: List[Document]) -> str:
"""
将检索到的文档格式化成上下文字符串。
"""
return "\n\n---\n\n".join([
f"[来源 {i+1}]\n{doc.page_content}"
for i, doc in enumerate(documents)
])
def create_rag_chain(vector_store: FAISS):
"""
使用 LCEL 构建完整的 RAG 生成链。
这是 Code A 的核心生成逻辑，完全不需要改动。
"""
# 提示词模板：强制 LLM 仅基于上下文回答
prompt_template = ChatPromptTemplate.from_template("""
你是一个只根据提供的上下文来回答问题的助手。
如果上下文中没有答案，请直接说"我没有足够的信息回答这个问题"。
上下文 (Context):
{context}
问题 (Question):
{question}
答案 (Answer):
""")
# 使用 gpt-4o-mini，temperature=0 保证事实性回答
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.0)
# 内部函数：检索 + 格式化
def retrieve_and_format(inputs: Dict[str, Any]) -> Dict[str, str]:
question = inputs["question"]
docs = retrieve_context(vector_store, question)
context = format_context(docs)
return {"context": context, "question": question}
# 构建 LCEL 链
rag_chain = (
RunnablePassthrough()
| RunnableLambda(retrieve_and_format)
| prompt_template
| llm
| StrOutputParser()
)
return rag_chain
# ============================================================
# 第二部分：CODE B —— FAISS 专业配置工厂 (带索引工程 + 持久化)
# ============================================================
class FAISSAdvancedConfig:
"""
FAISS 高级配置类。
控制距离度量、索引类型、检索参数。
"""
def __init__(
self,
index_factory_string: str = "Flat",
nprobe: int = 10
):
self.distance_strategy = DistanceStrategy.COSINE # 余弦相似度
self.index_factory_string = index_factory_string # 例如 "Flat", "IVF100,Flat"
self.nprobe = nprobe # IVF 查询时探测的簇数
class FAISSFactory:
"""
FAISS 工厂类。
职责：构建索引、保存到磁盘、从磁盘加载。
"""
def __init__(self, config: FAISSAdvancedConfig):
self.config = config
# 嵌入模型：text-embedding-3-small (1536维，性价比高)
self.embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
def build_from_documents(self, chunks: List[Document]) -> FAISS:
"""
从文档块构建 FAISS 索引。
支持 Flat / IVF / HNSW 等所有 FAISS 原生索引类型。
"""
print(f"🚀 正在构建 FAISS 索引 (类型: {self.config.index_factory_string})...")
# 1. 提取文本并生成向量
texts = [doc.page_content for doc in chunks]
vectors = self.embeddings.embed_documents(texts) # 返回 List[List[float]]
vectors_np = np.array(vectors).astype('float32') # FAISS 要求 float32
dim = vectors_np.shape[1] # 向量维度 (1536)
n_vectors = vectors_np.shape[0]
print(f" 向量维度: {dim}, 向量数量: {n_vectors}")
# 2. 使用工厂字符串创建原生 FAISS 索引
# "Flat" -> 暴力搜索 (精确)
# "IVF100,Flat" -> 100个聚类的倒排索引 (快速)
index = faiss.index_factory(dim, self.config.index_factory_string)
# 3. 如果使用 IVF，需要训练 (聚类) 并设置 nprobe
if "IVF" in self.config.index_factory_string:
print(f" 正在训练 IVF (nlist={self.config.index_factory_string.split('IVF')[1].split(',')[0]})...")
index.train(vectors_np)
# nprobe 越大，召回率越高，速度越慢
index.nprobe = self.config.nprobe
print(f" 设置 nprobe = {self.config.nprobe}")
# 4. 添加向量到索引
index.add(vectors_np)
print(f" 成功添加 {n_vectors} 个向量")
# 5. 包装成 LangChain 的 FAISS 对象
# 需要构建 docstore (存储原始文本) 和 id 映射
docstore = {str(i): chunk for i, chunk in enumerate(chunks)}
index_to_docstore_id = {i: str(i) for i in range(len(chunks))}
vector_store = FAISS(
embedding_function=self.embeddings.embed_query, # 查询时用的函数
index=index,
docstore=docstore,
index_to_docstore_id=index_to_docstore_id,
relevance_score_fn=None # LangChain 自动根据 distance_strategy 处理
)
print("✅ FAISS 索引构建完成")
return vector_store
def save(self, vector_store: FAISS, path: str) -> None:
"""
将 FAISS 索引保存到磁盘。
会生成两个文件: {path}.pkl 和 {path}.faiss
"""
vector_store.save_local(path)
print(f"💾 索引已保存到: {path}")
def load(self, path: str) -> FAISS:
"""
从磁盘加载 FAISS 索引。
"""
print(f"📂 正在从磁盘加载索引: {path}")
# allow_dangerous_deserialization=True 是因为本地开发环境安全
vector_store = FAISS.load_local(
folder_path=path,
embeddings=self.embeddings,
allow_dangerous_deserialization=True
)
print("✅ 索引加载成功")
return vector_store
# ============================================================
# 第三部分：集成逻辑 (智能缓存 + 自动切换)
# ============================================================
def get_vector_store_with_cache(
chunks: List[Document],
cache_path: str = "./faiss_cache",
force_rebuild: bool = False
) -> FAISS:
"""
智能获取向量存储：
- 如果 cache_path 存在且 force_rebuild=False -> 直接加载 (秒级启动)
- 否则 -> 用专业配置构建，并保存到 cache_path
自动索引选型：
- 向量数 <= 10000 -> 使用 Flat (暴力精确)
- 向量数 > 10000 -> 使用 IVF (倒排加速)
"""
# 根据数据量自动选择索引类型
if len(chunks) <= 10000:
index_type = "Flat"
else:
index_type = "IVF100,Flat" # 100个聚类，适合中等规模
print(f"⚙️ 数据量: {len(chunks)} 条，选用索引: {index_type}")
# 创建配置和工厂
config = FAISSAdvancedConfig(index_factory_string=index_type, nprobe=10)
factory = FAISSFactory(config)
# 决策：加载缓存 or 重新构建
if not force_rebuild and os.path.exists(cache_path):
# 路径存在 -> 直接加载
return factory.load(cache_path)
else:
# 路径不存在 或 强制重建 -> 构建并保存
if force_rebuild:
print("🔄 强制重建索引...")
vector_store = factory.build_from_documents(chunks)
factory.save(vector_store, cache_path)
return vector_store
# ============================================================
# 第四部分：主程序入口 (包含完整测试数据)
# ============================================================
def main():
"""
完整流程演示：
1. 生成临时测试文档
2. 加载 + 切分
3. 获取向量存储 (带缓存)
4. 执行 RAG 问答
5. 清理临时文件
"""
print("=" * 70)
print("B02 升级版 RAG Pipeline (Code A + Code B 完整集成)")
print("=" * 70)
# ----- 1. 准备测试数据 (临时文件) -----
print("\n📝 创建临时测试文档...")
with tempfile.NamedTemporaryFile(
mode="w",
suffix=".txt",
delete=False,
encoding="utf-8"
) as f:
f.write("""
Retrieval-Augmented Generation (RAG) is a technique that combines
information retrieval with large language model generation.
RAG has two main phases: indexing and retrieval-generation.
The indexing phase involves loading documents, splitting them into chunks,
converting chunks to embeddings, and storing them in a vector database.
The retrieval-generation phase involves converting a user query to an
embedding, searching the vector database for similar chunks, and using
those chunks as context for the LLM to generate an answer.
RAG solves the problem of LLM hallucinations by grounding answers in
retrieved facts. It also allows LLMs to access private or up-to-date
information that wasn't in their training data.
FAISS (Facebook AI Similarity Search) is a library for efficient
similarity search and clustering of dense vectors. It is widely used
as the vector database in RAG systems.
""")
temp_path = f.name
# ----- 2. 加载 + 切分 -----
print("\n📂 阶段 1: 加载与切分")
print("-" * 40)
docs = load_documents([temp_path])
chunks = chunk_documents(docs)
# ----- 3. 获取向量存储 (智能缓存) -----
print("\n💾 阶段 2: 向量存储初始化")
print("-" * 40)
# 首次运行会构建，第二次运行会直接加载 (秒级)
cache_dir = "./demo_faiss_cache"
vector_store = get_vector_store_with_cache(
chunks=chunks,
cache_path=cache_dir,
force_rebuild=False # 设为 True 可强制重建
)
# ----- 4. 执行 RAG 问答 -----
print("\n🤖 阶段 3: RAG 问答")
print("-" * 40)
# 创建 RAG 链 (完全使用 Code A 的逻辑)
rag_chain = create_rag_chain(vector_store)
# 测试问题列表
questions = [
"What is RAG?",
"What are the two main phases of RAG?",
"How does RAG solve the hallucination problem?",
"What is FAISS?"
]
for i, question in enumerate(questions, 1):
print(f"\nQ{i}: {question}")
answer = rag_chain.invoke({"question": question})
print(f"A{i}: {answer}")
# ----- 5. 清理临时文件 -----
print("\n🧹 清理临时文件...")
os.unlink(temp_path) # 删除临时 txt
# 保留缓存目录，方便下次测试秒启；如果想删掉，取消注释下面一行
# shutil.rmtree(cache_dir, ignore_errors=True)
print("\n" + "=" * 70)
print("✅ RAG Pipeline 运行完毕")
print(f"💡 缓存目录: {cache_dir} (保留以加速下次启动)")
print("=" * 70)
if __name__ == "__main__":
main()
Embedding is a fundamental technology in machine learning and natural language processing that transforms discrete or complex objects (such as words, sentences, or images) into numerical vector representations of a fixed dimension.
In a RAG (Retrieval-Augmented Generation) system, embedding is the foundation of retrieval. Without embeddings, you cannot perform semantic search — you would be stuck with keyword matching only
Core Concepts
Vector Space & Similarity
Embeddings live in a multi-dimensional vector space. To measure how similar two pieces of text are, we measure the distance between their embedding vectors. The most common measure is cosine similarity. 存在于一个多维向量空间中。要衡量两段文本有多相似，我们衡量它们的 Embedding 向量之间的距离。最常用的度量是余弦相似度（Cosine Similarity）.
Cosine Similarity explained:
Range: -1 (opposite) to 1 (identical) -1（相反）到 1（完全相同）
For text embeddings, values close to 1 mean high semantic similarity 对于文本 Embedding，接近 1 的值表示高语义相似度
It measures the cosine of the angle between two vectors, ignoring magnitude 它测量两个向量之间的夹角余弦值，忽略向量大小
Captures meaning 捕获含义 = 不看字，看“意思像不像” Only captures identity 仅捕获身份 = 只认“是不是这个词”，不管什么意思
Contextual vs Static Embeddings
上下文 Embedding vs 静态 Embedding
Type
Examples
Characteristics
Static
Word2Vec, GloVe, FastText
One vector per word, regardless of context 每个词一个向量，无论上下文
Contextual
BERT-based, OpenAI text-embedding-3
Vector changes based on surrounding words 向量根据周围词变化
Word2Vec: Word2Vec is a method that learns word meanings by looking at the context in which words appear. 是一种通过“上下文”来学习词语含义的模型。它通过“上下文预测”学习词向量的方法。看词和词之间的搭配关系来学习语义,让语义相似的词在向量空间中靠得更近。
GloVe = Global Vectors for Word Representation. It learns word meaning from: global word co-occurrence statistics “统计全世界词和词的关系”
FastText：FastText (Facebook) represents words as： sum of character n-grams把“词拆成字母/子词”来学向量
“bank” in “river bank” vs “bank account” ： static embedding gives the same vector; contextual embedding gives different vectors based on context. 静态 Embedding 给相同的向量；上下文 Embedding 根据上下文给不同的向量。
BERT = Bidirectional Encoder Representations from Transformers
It learns: context-aware word meaning using Transformer attention 用 Transformer（注意力机制）来理解上下文的语言模型。 “河岸”，“银行账号” “同一个词，在不同句子里有不同含义”
Mainstream Embedding Models
For modern RAG systems, the most relevant embedding models are
Model
Provider
Dim
Notes
text-embedding-3-small
OpenAI
1536
Cost-effective, good for most RAG
text-embedding-3-large
OpenAI
3072
Higher quality, higher cost
text-embedding-ada-002
OpenAI
1536
Legacy, being replaced by v3
BGE-M3
BAAI
1024
Open-source, multilingual
KaLM-Embedding-Gemma3-12B
Tencent
3840
SOTA on MMTEB
Code Implementation
OpenAI API generates Embedding：
# ============================================================
# 1. IMPORTS / 导入依赖
# ============================================================
import os
import numpy as np
from openai import OpenAI
from typing import List, Dict, Any
# ============================================================
# 2. CLIENT INITIALIZATION / 客户端初始化
# ============================================================
# Purpose: Initialize the OpenAI client with API key from environment.
# 目的：使用环境变量中的 API Key 初始化 OpenAI 客户端。
client = OpenAI(
api_key=os.environ.get("OPENAI_API_KEY"), # Get from env / 从环境变量获取
)
# ============================================================
# 3. EMBEDDING FUNCTION / Embedding 生成函数
# ============================================================
def get_embedding(
text: str,
model: str = "text-embedding-3-small"
) -> List[float]:
"""
Generate an embedding vector for a given text using OpenAI API.
使用 OpenAI API 为给定文本生成 Embedding 向量。
Purpose: Convert text into a numerical vector for semantic search.
目的：将文本转换为数值向量，用于语义搜索。
Args:
text: Input text string / 输入文本字符串
model: Embedding model name / Embedding 模型名称
Returns:
List[float]: Embedding vector of length model-specific dimension
List[float]: Embedding 向量，长度为模型指定的维度
Important: The same model must be used for both query and document embeddings
重点：查询和文档的 Embedding 必须使用同一个模型[reference:30]
"""
# Remove newlines and excess whitespace for cleaner embedding
# 移除换行和多余空白，获得更干净的 Embedding
text = text.replace("\n", " ")
# Call OpenAI Embeddings API / 调用 OpenAI Embeddings API
# Purpose: Send text to OpenAI and get back the embedding vector
# 目的：发送文本到 OpenAI，获取 Embedding 向量
response = client.embeddings.create(
model=model, # Which embedding model to use / 使用的模型
input=text, # Text to embed / 要嵌入的文本
encoding_format="float" # Return as list of floats / 以浮点数列表返回
)
# Extract the embedding from response / 从响应中提取 Embedding
# Purpose: The embedding is in response.data[0].embedding
# 目的：Embedding 位于 response.data[0].embedding 中
embedding = response.data[0].embedding
return embedding
# ============================================================
# 4. BATCH EMBEDDING / 批量 Embedding
# ============================================================
def get_embeddings_batch(
texts: List[str],
model: str = "text-embedding-3-small"
) -> List[List[float]]:
"""
Generate embeddings for multiple texts in a single API call.
在单个 API 调用中为多个文本生成 Embedding。
Purpose: More efficient than calling get_embedding() in a loop.
目的：比在循环中调用 get_embedding() 更高效。
Important: OpenAI limits total tokens to 300,000 per request[reference:31]
重点：OpenAI 限制每个请求总 token 不超过 300,000[reference:32]
"""
# Clean texts / 清理文本
cleaned_texts = [t.replace("\n", " ") for t in texts]
# Batch API call / 批量 API 调用
response = client.embeddings.create(
model=model,
input=cleaned_texts, # List of texts / 文本列表
encoding_format="float"
)
# Extract all embeddings / 提取所有 Embedding
# Purpose: Each response.data[i] corresponds to texts[i]
# 目的：每个 response.data[i] 对应 texts[i]
embeddings = [item.embedding for item in response.data]
return embeddings
# ============================================================
# 5. SIMILARITY SEARCH / 相似度搜索
# ============================================================
def cosine_similarity(vec_a: List[float], vec_b: List[float]) -> float:
"""
Calculate cosine similarity between two embedding vectors.
计算两个 Embedding 向量之间的余弦相似度。
Purpose: Measure semantic similarity between query and document.
目的：衡量查询和文档之间的语义相似度。
Important: Embeddings must be from the SAME model.
重点：Embeddings 必须来自同一个模型。
"""
# Convert to numpy arrays for efficient math / 转换为 numpy 数组以便高效计算
a = np.array(vec_a)
b = np.array(vec_b)
# Calculate cosine similarity / 计算余弦相似度
# Formula: cos(θ) = (A · B) / (||A|| * ||B||)
# 公式：cos(θ) = (A · B) / (||A|| * ||B||)
dot_product = np.dot(a, b)
norm_a = np.linalg.norm(a)
norm_b = np.linalg.norm(b)
# Avoid division by zero / 避免除以零
if norm_a == 0 or norm_b == 0:
return 0.0
return float(dot_product / (norm_a * norm_b))
def find_most_similar(
query: str,
documents: List[str],
model: str = "text-embedding-3-small",
top_k: int = 3
) -> List[Dict[str, Any]]:
"""
Find the most semantically similar documents to a query.
找到与查询语义最相似的文档。
Purpose: Core retrieval function for RAG systems.
目的：RAG 系统的核心检索函数[reference:33]。
Args:
query: User question / 用户问题
documents: List of document chunks / 文档块列表
model: Embedding model / Embedding 模型
top_k: Number of top results to return / 返回前 K 个结果
Returns:
List of dicts with document text and similarity score
包含文档文本和相似度分数的字典列表
"""
# Step 1: Embed the query / 第一步：嵌入查询
# Purpose: Convert user question to vector for comparison
# 目的：将用户问题转换为向量以便比较
query_embedding = get_embedding(query, model)
# Step 2: Embed all documents / 第二步：嵌入所有文档
# Purpose: Convert all documents to vectors
# 目的：将所有文档转换为向量
doc_embeddings = get_embeddings_batch(documents, model)
# Step 3: Calculate similarities / 第三步：计算相似度
# Purpose: Compare query vector against all document vectors
# 目的：将查询向量与所有文档向量比较
results = []
for i, doc_embedding in enumerate(doc_embeddings):
score = cosine_similarity(query_embedding, doc_embedding)
results.append({
"document": documents[i],
"score": score,
"index": i
})
# Step 4: Sort by score descending and return top_k
# 第四步：按分数降序排序，返回 top_k
# Purpose: Return the most relevant documents first
# 目的：首先返回最相关的文档
results.sort(key=lambda x: x["score"], reverse=True)
return results[:top_k]
# ============================================================
# 6. USAGE EXAMPLE / 使用示例
# ============================================================
if __name__ == "__main__":
# Example: RAG retrieval scenario / 示例：RAG 检索场景
# User question / 用户问题
query = "What is the capital of France?"
# Document chunks from a knowledge base / 来自知识库的文档块
documents = [
"France is a country in Western Europe. Its capital is Paris.",
"Germany's capital is Berlin. It is the largest city in Germany.",
"The Eiffel Tower is located in Paris, France.",
"London is the capital of the United Kingdom."
]
# Find most relevant documents / 找到最相关的文档
top_results = find_most_similar(query, documents, top_k=2)
print(f"Query: {query}")
print("\nTop Results:")
for i, result in enumerate(top_results):
print(f"{i+1}. Score: {result['score']:.4f}")
print(f" Document: {result['document']}")
print()
# Expected output: Documents about France and Paris should rank highest
# 预期输出：关于法国和巴黎的文档应该排名最高
Azure OpenAI generates Embedding（Azure AI Foundry intgerate）
# ============================================================
# AZURE OPENAI EMBEDDING / Azure OpenAI Embedding
# ============================================================
from openai import AzureOpenAI
# Initialize Azure OpenAI client / 初始化 Azure OpenAI 客户端
# Purpose: Use Azure OpenAI Service instead of OpenAI's direct API
# 目的：使用 Azure OpenAI 服务替代 OpenAI 的直接 API
azure_client = AzureOpenAI(
api_key=os.environ.get("AZURE_OPENAI_API_KEY"),
api_version="2024-02-15-preview", # Azure API version / Azure API 版本
azure_endpoint=os.environ.get("AZURE_OPENAI_ENDPOINT"),
)
def get_azure_embedding(text: str, deployment: str = "text-embedding-3-small") -> List[float]:
"""
Generate embedding using Azure OpenAI Service.
使用 Azure OpenAI 服务生成 Embedding。
Purpose: Enterprise-grade embedding with Azure's infrastructure.
目的：使用 Azure 基础设施的企业级 Embedding。
"""
response = azure_client.embeddings.create(
model=deployment, # Deployment name in Azure / Azure 中的部署名称
input=text,
encoding_format="float"
)
return response.data[0].embedding
Key Takeaways
要点
EN
CN
Embedding 是将文本转换为数值向量
Embedding converts text to numerical vectors
Embedding 将文本转换为数值向量
语义相似度通过向量距离衡量
Semantic similarity is measured by vector distance
语义相似度通过向量距离衡量
余弦相似度是最常用的度量
Cosine similarity is the most common measure
余弦相似度是最常用的度量
查询和文档必须用同一个 Embedding 模型
Query and documents must use the SAME embedding model
查询和文档必须用同一个 Embedding 模型
OpenAI 限制单请求总 token 300,000
OpenAI limits total tokens to 300,000 per request
OpenAI 限制单请求总 token 300,000
上下文 Embedding 比静态 Embedding 更精准
Contextual embeddings are more accurate than static
RAG (Retrieval-Augmented Generation) is an AI architecture that retrieves relevant information from external knowledge sources and provides it to an LLM, enabling the model to generate accurate, up-to-date, and context-aware responses.
LLM = Answer from Training while RAG = Search First + Then Answer
What is RAG 2.0? There is no official industry standard that defines “RAG 1.0”, “RAG 2.0”, or “RAG 3.0”.
The concept of “RAG 2.0”, known as Agentic RAG, represents a shift from a “simple data retrieval plugin” to a “deeply integrated, agentic system with reasoning capabilities.” The industry has indeed evolved from 1.0 and is now progressing toward 3.0.
Simplest Way to Remember
Version
Meaning
RAG 1.0
Search Once
RAG 2.0
Search + Reason
RAG 3.0 (coming ? on the way?)
Agent + Search + Tools + Workflow
RAG Foundation: Embedding
Embedding is a fundamental technology in machine learning and natural language processing that transforms discrete or complex objects (such as words, sentences, or images) into numerical vector representations of a fixed dimension
A simple RAG pipeline has two phases and five stages
Phase 1: Indexing (Offline / 离线阶段) Build the searchable knowledge base before users ask questions. 离线阶段（索引）：先把所有文档加载进来，切分成小块，调用 Embedding 模型转成向量，最后存进 FAISS 等向量数据库里。
Phase 2: Retrieval & Generation (Online / 在线阶段) Execute at query time for each user question. 在线阶段（检索+生成）：用户提问时，先把问题也转成向量，去库里找最相似的 Top-K 个块，把这几个块作为“参考资料”连同问题一起扔给 GPT，让它写出最终回答。
In AI, a chunking strategy is the method used to break down large pieces of information—like long documents, complex prompts, or even audio and video—into smaller, more manageable segments called “chunks”. These chunks are the fundamental units that AI systems, especially Large Language Models (LLMs), work with to understand, process, and retrieve information efficiently.
Common Chunking Strategies
The best strategy depends on the type of data and the specific task. Here are the most common approaches:
Strategy
Description
Best For
Fixed-Size Chunking
Splits text into chunks of a predetermined size (e.g., a specific number of words, characters, or tokens). Often uses a “sliding window” with overlap to preserve context across boundaries.
Simple implementation; works well when document structure is uniform and not critical.
Semantic Chunking
Groups text based on meaning, using algorithms to find natural topic boundaries and keep related ideas together.
Maintaining the coherence of ideas within each chunk for better understanding.
Structure-Aware Chunking
Respects the natural format of the document, splitting at points like paragraphs, sentences, or headers.
Documents with clear structure (e.g., articles, reports, code) where breaking mid-section would lose meaning.
Recursive Chunking
Starts with large chunks and recursively splits them into smaller ones until they meet a target size, trying to respect natural boundaries like paragraphs or sentences.
A balanced approach that aims to create the largest possible meaningful chunks.
Multimodal Chunking
Extends the concept to non-text data, such as segmenting audio by silences, video by scene changes, or identifying objects within images.
AI systems that process images, audio, and video, not just text
Chunking Strategy: Recursive Chunking
Recursive Chunking is a hierarchical text splitting strategy that uses a priority list of separators (e.g., ["\n\n", "\n", " ", """]), starting with the highest-level (semantically strongest) separator. If a chunk still exceeds the chunk_size limit after splitting, it recursively applies the next-level separator to that chunk, continuing until all chunks meet the size requirement.
Semantic Chunking is a strategy for splitting documents into smaller pieces (chunks) based on meaning, rather than on fixed character counts or simple separators. It tries to keep sentences or paragraphs that are about the same topic together, and split where the topic changes. Think of it as “a smart editor who knows where one idea ends and the next begins
Hybrid Search refers to a search technique that combines multiple search algorithms simultaneously to retrieve the most relevant results. It most commonly merges Lexical (Keyword) Search with Semantic (Vector/Dense) Search. 基于关键词的搜索（词汇搜索）与基于语义的搜索（向量/稠密搜索）
Lexical Search (Sparse Retrieval): Typically powered by algorithms like BM25 or TF-IDF. It relies on exact keyword matching and statistical term frequency. It excels at finding specific proper nouns, IDs, or rare terminology (e.g., “error code 404”). 词汇搜索（稀疏检索）： 通常由 BM25 或 TF-IDF 等算法驱动。它依赖于精确的关键词匹配和统计词频。它在查找特定的专有名词、ID 或罕见术语时表现出色（例如：“错误代码 404”）。
Semantic Search (Dense Retrieval): Powered by embedding models (e.g., Sentence-BERT). It converts text into high-dimensional vectors and retrieves documents based on “meaning” rather than exact words. It excels at understanding synonyms, context, and natural language queries (e.g., “How to fix a broken internet connection”). 语义搜索（稠密检索）： 由嵌入模型（如 Sentence-BERT）驱动。它将文本转换为高维向量，并根据“含义”而非精确词汇来检索文档。它在理解同义词、上下文和自然语言查询（例如：“如何修复断开的网络连接”）方面表现出色。
BM25 (keyword-based search)
BM25 (Best Matching 25) is a keyword-based ranking algorithm used in information retrieval to score and rank documents based on their relevance to a search query. It’s called “Best Matching 25” because it was the 25th variant in a series of scoring functions proposed by its creators. BM25 is the default ranking algorithm in Elasticsearch and most production search engines
RRF stands for Reciprocal Rank Fusion. It is an algorithm that merges multiple ranked result lists from different search systems into a single unified ranking. 全称是 Reciprocal Rank Fusion（倒数排名融合）。它是一种将来自不同检索系统的多个排名结果列表合并成一个统一排名的算法.
Compare normal DB, such as SQL DB, MySQL, PostgreSQL, They fundamental difference is what they search for and how they find it. A traditional SQL database (like MySQL, PostgreSQL without pgvector) is built for exact matching and structured queries. It answers questions like: “Find the customer with ID = 12345” or “Give me all orders over $100.” A Vector Database is built for semantic similarity and unstructured data. It answers questions like: “Find all documents that talk about the same topic as this paragraph” or “Show me products that look visually similar to this image.”
Normal DB vs Vector DB
要点
EN
CN
核心：精确匹配 vs. 语义相似度
Core: Exact matching vs. Semantic similarity
核心：精确匹配 vs. 语义相似度
SQL存结构化数据（数字/字符串），Vector存浮点数数组（含义）
SQL stores structured data (numbers/strings); Vector stores float arrays (meaning)
SQL存结构化数据，Vector存浮点数数组（含义）
SQL查询用WHERE精确条件；Vector查询用ORDER BY距离
SQL queries use WHERE exact conditions; Vector queries use ORDER BY distance
SQL results are binary (match/no match); Vector results are ranked (similarity scores)
SQL结果是二元的；Vector结果是排序的
SQL适合事务、财务报表；Vector适合RAG、推荐、AI
SQL suits transactions, ledgers; Vector suits RAG, recommendations, AI
SQL适合事务、报表；Vector适合RAG、推荐、AI
专用向量库不能做JOIN和ACID；但pgvector可以在PostgreSQL中兼得
Dedicated vector DBs can’t do JOINs/ACID; pgvector lets you have both in PostgreSQL
专用向量库不能做JOIN和ACID；pgvector可以让两者兼得
在实际RAG中，两者是互补的，不是替代关系
In real-world RAG, they are complementary, not replacements
在实际RAG中，两者是互补的，不是替代关系
Vector database selection is the process of choosing the right vector database technology for your AI application from dozens of available options — Milvus, Qdrant, Weaviate, Pinecone, pgvector, Chroma, and more.
Mainstream Vector Database Landscape
Category
Examples
CN
Fully Managed (PaaS)
Pinecone, Zilliz Cloud, Weaviate Cloud
全托管云服务
Self-Hosted Open Source
Qdrant, Milvus, Weaviate, Chroma
自托管开源
Database Extensions
pgvector, MongoDB Vector Search, Elasticsearch
数据库扩展
Cloud Provider Services
Azure AI Search, AWS S3 Vectors, Tencent Cloud VDB
云厂商服务
Embedded/Specialized
SQLite (vector), LanceDB
嵌入式/专用
Detailed Comparison
Database
Avg Query Time
Cost (1M @ 1536-dim)
Best For
CN 最适合
Milvus/Zilliz
50.7ms
$115/mo
Fastest queries + good flexibility
最快查询+灵活性好
Weaviate
51.7ms
$160/mo
Native datetime/geo + hybrid search
原生时间/地理+混合检索
Qdrant
73.1ms
$103/mo
Best balance (speed + flexibility + cost)
最佳平衡（速度+灵活性+成本）
Pinecone
106.3ms
$30/mo
Cheapest (⚠️ poor schema flexibility)
最便宜（⚠️ Schema灵活性差）
Chroma
275.4ms
$139/mo
Easiest setup + prototyping
最简单设置+原型开发
7 Types of Data Stored in VectorDB in AI Projects
类型
EN
CN
更新频率
过滤器
主要用途
RAG 文档块
RAG Document Chunks
RAG 文档块
低
source, page
问答
用户记忆 (Mem0)
User Memory (Mem0)
用户记忆 (Mem0)
高
user_id
个性化
工具 Schema
Tool Schemas
工具 Schema
中
tool_category
工具选择
Agent 轨迹
Agent Trajectories
Agent 轨迹
中
user_id, outcome
经验复用
黄金数据集
Golden Dataset
黄金数据集
低
category
评估
语义缓存
Semantic Cache
语义缓存
中
无
降本提速
代码索引
Code Index
代码索引
低
language, path
代码生成
Reranker is a sophisticated machine learning model designed to refine and reorder a list of candidate items—such as search results, document passages—to maximize their relevance to a specific query or context.
Advanced RAG (Advanced Retrieval-Augmented Generation) is an evolutionary upgrade over the basic “Naive RAG” pipeline. It adds a suite of optimization techniques at every stage of the RAG workflow — pre-retrieval, retrieval, post-retrieval, and evaluation — to systematically improve retrieval precision, recall, and generation quality.
Azure AI Foundry is a Microsoft’s unified Azure platform-as-a-service offering for enterprise AI operations, model builders, and application development. 微软新的企业级 AI 平台，主要用于开发。
AI apps / AI 应用
Copilots / Copilot
AI agents / AI Agent (智能系统)
RAG systems / RAG 系统
enterprise AI workflows / 企业智能工作流
It is becoming Microsoft’s main AI engineering platform. Think of it as 它正在变成微软主要的AI工程平台，本质上可以理解成
Enterprise companies care about this heavily / 企业极其重视这个.
An Agent = LLM + Tools + Memory + Planning
Agent can:
decide steps / 自动拆解任务
call tools (search, DB, API, code)
store memory
execute workflows
👉 Think:
“You give goal → agent figures out how to achieve it”
What is BPE?
BPE (Byte-Pair Encoding) is an algorithm that splits text into tokens by repeatedly merging the most frequent adjacent pairs of characters. 是一种将文本拆分成 token 的算法，它通过反复合并最常出现的相邻字符对来构建词汇表。
BPE starts with a base vocabulary of bytes/characters and iteratively merges the most frequent adjacent pairs across a large text corpus.
Three key points
CN
EN
1. 输入是文本
1. Input is text
2. 输出是一套合并规则 + token 序列
2. Output is a set of merge rules + a token sequence
3. 核心操作：找最频繁的相邻对，合并，重复
3. Core operation: find the most frequent adjacent pair, merge, repeat
Highest frequency is (a, b) and (b, c), both 2 times. Pick (a, b) to merge.
(a, b) → ab
Result：ab c ab c c
Step 2: Count adjacent pairs again：
Pair
Count
(ab, c)
2
(c, ab)
1
(c, c)
1
Highest frequency is (ab, c) with 2 occurrences. Merge.
Result: abc abc c
Step 3 (optional)
Count adjacent pairs again: does (abc, abc) appear?
Check: abc abc c → adjacent pairs:
(abc, abc): 1 occurrence
(abc, c): 1 occurrence
(abc, abc) merge into abcabc
Final result comparison
CN：步骤
EN：Step
CN：结果
EN：Result
开始
Start
a b c a b c c (7 个单位)
a b c a b c c (7 units)
第 1 步后
After step 1
ab c ab c c (5 个单位)
ab c ab c c (5 units)
第 2 步后
After step 2
abc abc c (3 个单位)
abc abc c (3 units)
Core summary
CN
EN
每一步只合并相邻的两个单位
Each step merges only two adjacent units
合并后单位变少
Units decrease after each merge
新单位可以参与下一步的合并
New units can participate in next step’s merges
停止条件：达到目标词表大小
Stop condition: target vocabulary size reached
What is Completion in AI/LLM?
Completion is the fundamental, raw operation of an LLM where the model takes an input text prompt and generates the most likely continuation of that text, token by token, in an autoregressive manner. It has no concept of roles or conversation history — just text in, text out.
Prompt (提示词): "The capital of France is"
Completion (补全): " Paris."
Prompt (提示词): "def fibonacci(n):"
Completion (补全): "\n if n <= 1:\n return n\n else:\n return fibonacci(n-1) + fibonacci(n-2)"
What is Chat in AI/LLM?
Chat is a structured, turn-based interaction paradigm built on top of completion. It adds role awareness (system, user, assistant) and automatic conversation history management. Each chat interaction is internally converted into a completion with special formatting tokens.
e.g.
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
{"role": "assistant", "content": "The capital of France is Paris."}
]
A Context Window is the amount of information an LLM can “see” or “remember” during a conversation or request. Think of it as: The AI model’s working memory. Everything inside the context window can influence the AI’s response. 模型只能基于“Context Window 内的信息”来回答问题， LLM 一次能“看到/记住”的信息量就是Context windows。可以理解为AI的”内存容量“
Important Understanding
The context window includes BOTH:
Included in Context Window
Examples
Input tokens
prompts, chat history, RAG docs
Output tokens
model response / AI 输出，回答
Total tokens = Input + Output
Context Engineering
Meaning:
deciding WHAT information goes into the context window
After deployment, Azure gives endpoint + API access
VERY important
Deployment ≠ Agent
A deployment is: an exposed model service 可以简单理解为：将一个Model, 例如将 GPT 或 Deep Seek，调进我的系统，并激活它，让这个model 在我的系统里变为“可使用了”。
What are Embeddings in AI / LLM?
Embeddings in AI / LLM are numerical representations of text (or other data like images, audio) in a high‑dimensional vector space. Simply put, they turn words, sentences, or documents into lists of numbers so that computers can “understand” their meaning mathematically.
在 AI / 大语言模型中，Embedding 是把文本（或图像、音频等）转换成数字列表（向量） 的技术。简单说，就是让计算机通过一串数字来“理解”文字的含义。
Key points:
What it looks like: A word like "king" might be represented as a vector: [0.25, -0.78, 0.43, …, 0.12] (e.g., 300–4096 dimensions).
How it works: Words or phrases with similar meanings are placed close together in this vector space.
"king" and "queen" are close.
"apple" (fruit) and "apple" (company) have different vectors depending on context.
Why embeddings matter:
They capture semantic meaning – relationships like king − man + woman ≈ queen.
They enable search (find similar texts), clustering (group topics), and recommendation.
LLMs use embeddings internally to process every token you feed into the model.
Vector Databases
A Vector Database is a database designed to store and search embeddings (vectors). Vector DB stores semantic meaning vectors
Example, Suppose company documents contain: “Employees may work remotely twice weekly.”
User asks: “What is the work from home policy?”. Traditional keyword search may fail because “remote” ≠ “work from home”. But embedding vectors capture semantic similarity.
Semantic Search
Similarity search is a technique that finds items in a dataset that are most similar to a given query vector, based on distance metrics in a high-dimensional embedding space — enabling semantic matching rather than exact keyword matching. 相似性搜索(Similarity search) 是一种技术，基于高维嵌入空间中的距离度量，在数据集中找到与给定查询向量最相似的项目 — 实现语义匹配而非精确关键词匹配。
Grounding = making the AI answer based on real external evidence, not memory. 让 AI 的回答“有依据”，不是靠记忆乱猜。或者说“给 AI 看资料”， 不是让它自己想答案
What is Hallucination in AI / LLM?
Hallucination in AI / LLM refers to the phenomenon where the model generates content that is factually incorrect, nonsensical, or completely unrelated to the real world or the provided source, while presenting it with high confidence as if it were true. This is one of the BIGGEST concerns in enterprise AI systems.
Common examples include:
Inventing non‑existent references, laws, or historical events.
Incorrectly calculating simple arithmetic.
Misinterpreting the user’s input and fabricating plausible‑sounding but false information.
虚假生成, 模型编造 AI 生成了错误的，编造的，不真实的，没依据的的信息。但AI却“很自信”地说出来。即： 模型自信地输出错误或凭空捏造的信息
Why Hallucinations Happen？
LLMs Predict Language, Not Truth；
2) Missing Context. If the model lacks:
sufficient information
enterprise data
current data
it may “fill in the gaps.”
3) Ambiguous Prompts. Poor prompts can cause:
assumptions
invented details
unstable outputs
4) Outdated Training Data. Models have training cutoffs. They may:
not know recent events
generate outdated answers
guess newer information
5) Weak RAG / Retrieval. In enterprise AI:
bad retrieval
irrelevant documents
incomplete grounding
can produce hallucinated answers.
Types of Hallucinations
A. Factual Hallucination: Wrong facts. /事实幻觉, 事实错误, 编造公司政策
B. Citation Hallucination: Fake sources or references. / 引用幻觉, 假论文、假来源。
C. Logical Hallucination: Reasoning errors / 推理幻觉, 逻辑推理错误
D. Tool/API Hallucination: Inventing APIs, functions, parameters, libraries / 编造API等
How Enterprises Reduce Hallucinations
1) RAG (Retrieval-Augmented Generation)
RAG: Most important technique. Instead of relying only on model memory / 最核心
What is LLM? Large language models, also known as LLMs, are very large deep learning models that are pre-trained on vast amounts of data. The underlying transformer is a set of neural networks that consist of an encoder and a decoder with self-attention capabilities. The encoder and decoder extract meanings from a sequence of text and understand the relationships between words and phrases in it.
大型语言模型（英语：large language model，LLM），也称大语言模型，简称大模型，是一种基于人工神经网络的已经训练过的语言模型。大语言模型专为自然语言处理任务而设计，尤其适用于语言生成。
Memory = system that stores user/context over time
Types:
🔹 Short-term memory
current conversation context
🔹 Long-term memory
user preferences
past interactions
profile data
Why important?
every chat is “reset” / 没有memory,每次都是新用户
personalized AI experience / 有memory AI 变成“个人助理”
What is Prompt? A Prompt is the instruction, question, context, or input you give to an AI model (LLM) to tell it what you want it to do. 就是你给 AI 的“指令/输入”， 告诉AI： 要做什么，用什么方法做，输出什么。
e.g.
Summarize this document in 5 bullet points.
That sentence is a prompt.
Another example:
You are a senior Azure architect. Explain Medallion Architecture for a banking platform.
The prompt tells the AI:
its role
the task
the expected output
sometimes the tone/style
Basic Prompt Structure
A prompt often contains:
Part
Purpose
Instruction
What to do / 做什么
Context
Background information / 背景信息
Constraints
Rules/limits/ 限制条件
Examples
Demonstrations / 示例
Output format
Expected response structure / 要求的输出格式
Example
You are a data architect.
Context: The company uses Azure Databricks and Synapse.
Task: Design a metadata-driven ingestion framework.
Output: Provide architecture, components, and best practices.
This is a more structured prompt.
Core components of Good prompt
A good prompt usually includs:
EN
CN
Goal
目标
Context
背景
Constraints
限制条件
Input
输入数据
Output Format
输出格式
Examples
示例
What is Prompt Engineering?
Prompt Engineering = the practice of designing prompts to get better AI outputs. 提示词工程就是：设计 Prompt 来获得更好 AI 输出”的技术。 其实简单说就是 “会问 AI 问题”
It is:
writing prompts strategically / 更聪明地写 Prompt
structuring context correctly / 更合理地组织 Context
controlling AI behavior / 更稳定地控制 AI 行为
improving reliability and quality / 提高可靠性和质量
Think of it as:
Programming with language instead of code.
可理解成：用自然语言“编程”，而不是用代码编程
Why Prompt Engineering Matters / 为什么重要
LLMs are highly sensitive to / LLM对这些高度敏感:
wording / 措辞
context / 背景信息
instructions / 要求
examples / 示例
formatting / 输出要求
Small prompt changes can dramatically affect / 对 prompt上述这些哪怕是小的改动都会影响到结果:
accuracy / 准确率
reasoning / 推理能力
hallucination / 幻觉, 无根据的结论
consistency / 稳定性
output quality / 输出质量
Common Prompt Engineering Techniques
common prompt technical include:
Tech
CN
Role Prompting
指定 AI 身份
Few-shot
给多个例子
Chain of Thought
引导 AI 一步一步思考
Output Control
控制输出格式
Constraints
加限制条件
Context Injection
注入业务背景
1) Role Prompting
Tell the AI who it is.
Example:
You are a senior enterprise architect.
This changes response style and depth.
2) Context Injection
Provide necessary information / 提供/注入必要的背景信息，以提高结果的准确性
Example:
The environment uses: - Azure Databricks - Delta Lake - Unity Catalog
Without context, AI guesses / 不提供这些背景资料，AI会去乱猜。影响结果的准确性.
Give examples of desired behavior. Few-Shot Learning / (少样本学习) 是一种人工智能技术。它指的是在给模型的提示词（Prompt）中提供少量（通常 2 到 5 个）示例，帮助模型理解任务要求，从而生成更准确的回复。
Example:
I want to classify sentiment.
Example 1: "I love this food!" -> Positive
Example 2: "This is the worst day ever." -> Negative
Example 3: "The movie was okay." -> Neutral
Now classify this: "The weather is quite nice today." ->
output: Positive
AI learns pattern/style from examples.
5) Chain-of-Thought Prompting
Chain-of-Thought (CoT) is a prompting technique that forces the LLM to show its reasoning steps before giving a final answer — like asking a SQL analyst to explain their logic before writing the query.
# Question：
"Roger has 5 marbles. He buys 2 bags with 4 marbles each. Then he loses 3 marbles. Think step by step. How many does he have left?"
# the answer with all steps:
Roger starts with 5 marbles.
He buys 2 bags with 4 marbles each:
2 × 4 = 8 marbles
Now he has:
5 + 8 = 13 marbles
Then he loses 3 marbles:
13 − 3 = 10 marbles
Answer: 10 marbles. 🟢
Prompt Injection is an attack where malicious user input tries to override or hijack the system prompt, making the AI behave in unintended ways. In production, prompt injection is one of the Top 5 LLM security risks (OWASP LLM Top 10). Any customer-facing AI must implement these defenses.
提示词注入是一种攻击方式，恶意用户输入试图覆盖或劫持 system prompt，让AI做出非预期的行为。
Schema enforcement = forcing data to follow a fixed structure (schema), not free-form text.
强制 AI 或 API 输出“符合格式的数据”，不能乱写。例如 John is 30 years old and lives in Toronto
AI 可能输出：John is 30 years old and lives in Toronto 也可能是： name: John age: thirty location: Toronto Canada maybe
StreamingIn the context of Large Language Models (LLMs), streaming refers to the technique of returning generated tokens one by one (or in small chunks) as soon as they are produced by the model, rather than waiting for the entire response to be completed. The underlying transport is typically Server-Sent Events (SSE) or chunked HTTP responses, where the server pushes incremental updates to the client.
Temperature is a hyperparameter that controls the randomness or creativity of an LLM’s output. It scales the logits (raw prediction scores) before the softmax function that converts them into probabilities — lower temperatures make the model more deterministic and focused, while higher temperatures make it more diverse and exploratory. Temperature 是一个超参数，用于控制大语言模型输出的随机性或创造性。它在 softmax 函数（将原始预测分数转换为概率）之前对这些 logits 进行缩放 — 较低的温度使模型更确定、更专注，而较高的温度使其更多样化、更具探索性。
Imagine you’re at a restaurant with a menu of 10 dishes. Temperature controls how likely you are to pick your absolute favorite vs. trying something new. 想象你在一个有 10 道菜的餐厅里。温度控制着你选择最爱的菜 vs. 尝试新菜的可能性。
Temperature
Analogy (English)
Analogy (中文)
Low (0.1 ~ 0.3)
You always order your #1 favorite dish. Very predictable.
你总是点你最爱的第一道菜。非常可预测。
Medium (0.7 ~ 1.0)
You usually pick your top dish, but sometimes try #2 or #3. Balanced.
你通常选最爱的菜，但有时尝试第二或第三喜欢的。平衡。
High (1.5+)
You randomly pick any dish, even ones you don’t know. Very unpredictable.
你随机选任何菜，甚至你不认识的菜。非常不可预测。
Top-K – Sample only from the K most probable tokens
The model looks at all possible next tokens and their probabilities.
It keeps only the K tokens with the highest probabilities and discards the rest.
Then it randomly selects one token from these K tokens (using their relative probabilities).
Effect:
Smaller K (e.g., 10) → Fewer choices → More deterministic, predictable, and safe outputs.
Larger K (e.g., 100) → More choices → More random and diverse outputs.
Example (K=3): Probabilities: “cat” (50%), “dog” (30%), “bird” (12%), “car” (5%), “tree” (3%) → Keep only {cat, dog, bird} → “car” and “tree” can never be chosen.
Top-P (Nucleus Sampling) – Choose the smallest set of tokens whose cumulative probability ≥ P
Instead of a fixed number of tokens (K), Top-P dynamically selects tokens from the most probable downward until the sum of their probabilities reaches or exceeds P.
This selected set is called the nucleus.
Effect:
Smaller P (e.g., 0.9) → Keeps only the top few high-probability tokens → More stable.
Larger P (e.g., 0.95–1.0) → Keeps more tokens (sometimes all) → More random.
Example (P=0.9): Probabilities: “cat” (50%, cumulative 50%), “dog” (30%, cumulative 80%), “bird” (12%, cumulative 92% ≥ 90%) → Nucleus = {cat, dog, bird} → “car” and “tree” are excluded.
Key difference from Top-K: If the probability distribution is very flat, P=0.9 might keep 20+ tokens. If very sharp, it might keep only 1 token. Top-K always keeps exactly K tokens.
Tokens in AI / LLM are the basic units of text that the model reads and generates. Instead of processing raw text character‑by‑character or word‑by‑word, the model breaks text into smaller, meaningful pieces called tokens.
在 AI / 大语言模型中，Token 是模型处理文本时的最基本单元。模型不会一个字符一个字符地读，也不会按完整单词读，而是把文本切分成有意义的片段，每个片段就是一个 Token。
LLM Reasoning is the ability of a Large Language Model to understand a user’s input, interpret meaning, and generate logical outputs based on patterns learned during training. It does not directly access external data during reasoning (unless tools are used). It mainly relies on internal parameters learned from training data.
RAG = search new knowledge and append to LLM Retrieval-Augmented Generation (RAG) is a method where the system first searches external knowledge sources (such as databases, documents, or enterprise knowledge bases) and then provides the retrieved information to the LLM to generate a grounded answer.
RAG retrieves external knowledge and injects it into the prompt context at runtime.
RAG 在运行时从外部检索信息，并把结果“临时放进上下文”，让 LLM 使用。
🔗 3. Relationship between LLM Reasoning and RAG
🧩 Core relationship
LLM Reasoning is the thinking engine, while RAG is the information supply system. RAG provides external factual knowledge, and LLM reasoning interprets and synthesizes that information into a final answer.
LLM provides reasoning based on pre-trained knowledge, while RAG supplies external, up-to-date information at inference time; together they enable grounded and accurate responses.
