Recursive Chunking is a hierarchical text splitting strategy that uses a priority list of separators (e.g., ["\n\n", "\n", " ", """]), starting with the highest-level (semantically strongest) separator. If a chunk still exceeds the chunk_size limit after splitting, it recursively applies the next-level separator to that chunk, continuing until all chunks meet the size requirement.
递归分块是一种层次化的文本分块策略，它使用一个分隔符优先级列表（如 ["\n\n", "\n", " ", ""]），从最高级（语义最强的分隔符）开始尝试分割文本。如果分割后的某个块仍然超过 chunk_size 限制，则对该块递归地使用下一级分隔符继续分割，直到所有块都符合大小要求
Core Content
Separator Priority
The separator list is the “brain” of the Recursive Chunking algorithm. It is not a random list—it follows a strict semantic hierarchy. Higher-priority separators represent stronger semantic boundaries.
默认分隔符: ["\n\n", "\n", " ", ""]
| 优先级 | 分隔符 | 含义 | 语义强度 |
|---|---|---|---|
| 1 (最高) | \n\n | 双换行 → 段落边界 | 最强 |
| 2 | \n | 单换行 → 行边界 | 中等 |
| 3 | | 空格 → 单词边界 | 较弱 |
| 4 (最低) | "" | 空字符串 → 字符级别 | 最弱（保底） |
Core Parameters
Chinese Language Support
中文分隔符列表
chinese_separators = [
“\n\n”, # 段落
“\n”, # 行
“。”, # 句号
“？”, # 问号
“！”, # 感叹号
“，”, # 逗号
“、”, # 顿号
” “, # 空格（如果有）
“” # 保底
]
4. Code Implementation
4.1 Basic Usage
</>bash
pip install langchain-text-splitters
%pip install langchain-text-splitters
# ============================================================
# 导入 RecursiveCharacterTextSplitter
# Import RecursiveCharacterTextSplitter
# ============================================================
from langchain_text_splitters import RecursiveCharacterTextSplitter
# ============================================================
# 准备示例文本
# Prepare example text
# ============================================================
sample_text = """
第一章：人工智能概述
人工智能是计算机科学的一个分支，致力于创建能够执行通常需要人类智能的任务的系统。
这些任务包括视觉感知、语音识别、决策制定和语言翻译等。
第二章：机器学习基础
机器学习是人工智能的核心子领域。它通过算法让计算机从数据中学习规律。
机器学习主要分为监督学习、无监督学习和强化学习三大类。
"""
# ============================================================
# 初始化 RecursiveCharacterTextSplitter
# Initialize RecursiveCharacterTextSplitter
#
# 参数说明 (Parameter Description):
# - chunk_size: 每个块的最大字符数 / Maximum characters per chunk
# - chunk_overlap: 块之间的重叠字符数 / Overlap characters between chunks
# - separators: 分隔符优先级列表 / Separator priority list
# - length_function: 计算长度的函数 / Function to measure length
# ============================================================
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=100, # 每块最多100字符 / Max 100 chars per chunk
chunk_overlap=20, # 块间重叠20字符 / 20 chars overlap
length_function=len, # 用 len() 计算长度 / Use len() for length
separators=["\n\n", "\n", "。", "，", " ", ""], # 中文分隔符 / Chinese separators
is_separator_regex=False, # 是否按正则解析 / Whether to parse as regex
)
# ============================================================
# 方式1: split_text() - 返回字符串列表
# Method 1: split_text() - Returns list of strings
# ============================================================
chunks = text_splitter.split_text(sample_text)
print(f"共生成 {len(chunks)} 个块")
print("-" * 50)
for i, chunk in enumerate(chunks):
print(f"块 {i+1} (长度: {len(chunk)}):")
print(f" {chunk[:50]}...") # 只显示前50字符
# debug
print ('-'*50)
print (chunks)
--------------------------------------------------
['第一章：人工智能概述\n\n人工智能是计算机科学的一个分支，致力于创建能够 ....、无监督学习和强化学习三大类。']
# ============================================================
# 方式2: create_documents() - 返回 Document 对象列表
# Method 2: create_documents() - Returns list of Document objects
#
# Document 对象包含 page_content 和 metadata，适合下游 RAG 流程
# Document objects contain page_content and metadata, suitable for downstream RAG
# ============================================================
from langchain_core.documents import Document
# 创建 Document 对象 / Create Document objects
documents = text_splitter.create_documents([sample_text])
# 可以附加元数据 / Can attach metadata
documents_with_meta = text_splitter.create_documents(
[sample_text],
metadatas=[{"source": "sample.txt", "chapter": "AI基础"}]
)
#debug
for doc in documents_with_meta:
print(f"内容: {doc.page_content[:50]}...")
print(f"元数据: {doc.metadata}")
print("-" * 50)
内容: 第一章：人工智能概述
人工智能是计算机科学的一个分支，致力于创建能够执行通常需要人类智能的任务的系...
元数据: {'source': 'sample.txt', 'chapter': 'AI基础'}
内容: 第二章：机器学习基础
机器学习是人工智能的核心子领域。它通过算法让计算机从数据中学习规律。
机器学...
元数据: {'source': 'sample.txt', 'chapter': 'AI基础'}4.2 从文件加载并分块（Load from File and Chunk）
<Bash>pip install langchain_community
<python>%pip install langchain_community
# ============================================================
# 从文件加载文档并分块
# Load document from file and chunk
# ============================================================
from langchain_community.document_loaders import TextLoader
# ============================================================
# 1. 加载文档 / Load document
# ============================================================
loader = TextLoader("my_document.txt")
documents = loader.load() # 返回 List[Document]
# ============================================================
# 2. 配置分块器 / Configure splitter
# ============================================================
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=500, # 技术文档用较大块 / Larger chunks for tech docs
chunk_overlap=80, # 重叠约16% / ~16% overlap
separators=[
"\n\n", # 段落 / Paragraphs
"\n", # 行 / Lines
"。", "？", "！", # 中文句末标点 / Chinese sentence endings
"，", "、", # 中文句中标点 / Chinese mid-sentence punctuation
" ", # 空格 / Spaces
"" # 保底 / Fallback
]
)
# ============================================================
# 3. 执行分块 / Execute chunking
# ============================================================
chunked_docs = text_splitter.split_documents(documents)
print(f"原始文档数: {len(documents)}")
print(f"分块后文档数: {len(chunked_docs)}")
# ============================================================
# 4. 查看分块统计 / View chunk statistics
# ============================================================
chunk_lengths = [len(doc.page_content) for doc in chunked_docs]
print(f"块长度统计: 最小={min(chunk_lengths)}, 最大={max(chunk_lengths)}, 平均={sum(chunk_lengths)/len(chunk_lengths):.0f}")4.3 按 Token 数分块（Token-based Chunking）
%pip install langchain_openai
%pip install tiktoken
# ============================================================
# 按 Token 数分块（更精确控制 LLM 上下文）
# Token-based chunking (more precise control for LLM context)
# ============================================================
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
# ============================================================
# 使用 tiktoken 编码器计算 token 数
# Use tiktoken encoder to count tokens
# ============================================================
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
encoding_name="cl100k_base", # OpenAI 的编码器 / OpenAI's encoder
chunk_size=512, # 每块 512 tokens
chunk_overlap=50, # 重叠 50 tokens
)
chunks = text_splitter.split_text(sample_text)
#debug
print(f"共生成 {len(chunks)} 个块")
print(chunks)
共生成 1 个块
['第一章：人工智能概述\n\n人......无监督学习和强化学习三大类。']
# 验证 token 数 / Verify token count
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
for i, chunk in enumerate(chunks):
token_count = len(enc.encode(chunk))
print(f"块 {i+1}: {token_count} tokens")
块 1: 172 tokens4.4 自定义长度函数（Custom Length Function）
# ============================================================
# 自定义长度函数：按单词数而非字符数
# Custom length function: count words instead of characters
# ============================================================
def word_count(text: str) -> int:
"""计算文本中的单词数 / Count words in text"""
return len(text.split())
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=100, # 每块100个单词 / 100 words per chunk
chunk_overlap=20, # 重叠20个单词 / 20 words overlap
length_function=word_count, # 使用自定义长度函数 / Use custom length function
separators=["\n\n", "\n", "。", "，", " ", ""]
)
chunks = text_splitter.split_text(sample_text)
for i, chunk in enumerate(chunks):
print(f"块 {i+1}: {word_count(chunk)} 个单词")Complete RAG Chunking Pipeline
# ============================================================
# 完整 RAG 分块流程：加载 → 分块 → 向量化 → 存入向量库
# Complete RAG chunking pipeline: Load → Chunk → Embed → Vector Store
# ============================================================
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
# ============================================================
# Step 1: 加载文档 / Load documents
# ============================================================
loader = TextLoader("knowledge_base.txt",encoding="utf-8")
docs = loader.load()
print(f"✅ 加载了 {len(docs)} 个文档")
# ============================================================
# Step 2: 递归分块 / Recursive chunking
# ============================================================
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=400, # 适中大小 / Moderate size
chunk_overlap=80, # 20% 重叠 / 20% overlap
separators=[
"\n\n", "\n",
"。", "？", "！",
"，", "、",
" ", ""
]
)
chunked_docs = text_splitter.split_documents(docs)
print(f"✅ 分块完成: {len(chunked_docs)} 个块")
✅ 分块完成: 1 个块
%pip install chromadb
%pip install langchain-chroma
# ============================================================
# Step 3: 向量化并存入向量数据库 / Embed and store in vector DB
# ============================================================
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain_community.embeddings import OllamaEmbeddings # <- I installed Ollama,runslocal
import os
#embeddings = OpenAIEmbeddings(model="text-embedding-3-small", api_key=your_API_KEY )
# I installed Ollama， running local
embeddings = OllamaEmbeddings(
model="nomic-embed-text",
base_url="http://localhost:11434",
show_progress=True # 显示下载进度
)
vectorstore = Chroma.from_documents(
documents=chunked_docs,
embedding=embeddings,
persist_directory="./chroma_db" # 持久化目录 / Persist directory
)
print(f"✅ 向量库创建完成，共 {vectorstore._collection.count()} 个向量")
OllamaEmbeddings: 100%|██████████| 1/1 [00:43<00:00, 43.13s/it]
✅ 向量库创建完成，共 1 个向量
# ============================================================
# Step 4: 测试检索 / Test retrieval
# ============================================================
query = "什么是机器学习？"
results = vectorstore.similarity_search(query, k=3)
print(f"\n🔍 查询: {query}")
print("-" * 50)
for i, doc in enumerate(results):
print(f"结果 {i+1}: {doc.page_content[:100]}...")Key Takeaways
| 要点 (Key Point) | EN | CN |
|---|---|---|
| 分隔符顺序决定语义完整性 | Separator order determines semantic integrity. Always go from coarse (paragraph) to fine (character). | 分隔符顺序决定语义完整性。永远从粗（段落）到细（字符）。 |
chunk_overlap 防止边界信息丢失 | chunk_overlap prevents information loss at boundaries. Set to 10-20% of chunk_size. | chunk_overlap 防止边界信息丢失。设置为 chunk_size 的 10-20%。 |
| 中文必须自定义分隔符 | Chinese text must use custom separators (。, ？, ！, ，, 、). Never use the default English list. | 中文文本必须使用自定义分隔符（。、？、！、，、、）。切勿使用默认的英文列表。 |
| 元数据重复是隐藏的陷阱 | Metadata duplication is a hidden trap. When using split_documents(), be careful with unique ID fields. | 元数据重复是隐藏的陷阱。使用 split_documents() 时，注意唯一 ID 字段。 |
| 大文件要用流式处理 | For large files (> 1 GB), implement streaming chunking to avoid OOM errors. | 对于大文件（> 1 GB），实现流式分块以避免 OOM 错误。 |
| 字符数 vs Token 数 | Character count is fast but inaccurate. Token count is accurate but slower. Choose based on your use case. | 字符数快但不准确。Token 数准确但较慢。根据您的用例选择。 |