What is BM25?
BM25 (Best Matching 25) is a keyword-based ranking algorithm used in information retrieval to score and rank documents based on their relevance to a search query. It’s called “Best Matching 25” because it was the 25th variant in a series of scoring functions proposed by its creators. BM25 is the default ranking algorithm in Elasticsearch and most production search engines
In production RAG systems, BM25 is the standard keyword search algorithm. It has “withstood the test of time for decades since its invention”. Here’s why it matters:
What is it used for?
BM25 is used to find documents that contain the exact words or phrases a user typed into the search box. Its core job is lexical (term-based) retrieval โ matching literal text strings between the query and the documents.
BM25 ็จไบๆฅๆพๅ
ๅซ็จๆท่พๅ
ฅๆ็ดขๆกไธญ็็กฎๅๅ่ฏๆ็ญ่ฏญ็ๆๆกฃใๅฎ็ๆ ธๅฟๅทฅไฝๆฏ่ฏๆณ๏ผๅบไบ่ฏ้กน็๏ผๆฃ็ดขโโๅจๆฅ่ฏขๅๆๆกฃไน้ดๅน้
ๅญ้ขไธ็ๆๆฌๅญ็ฌฆไธฒใ
ๆณ่ฑกไฝ ๆไธไธช่ฃ ็ๆๅไธไธไปฝๅๅ็ๅทจๅคงๆไปถๆใBM25ๅฐฑๆฏ้ฃไธชๅธฆๆ ็ญพ็็ดขๅผ็ณป็ปโโๅฝไฝ ๆ็ดข”ไธๅฏๆๅๆกๆฌพ็ฌฌ4.2่”ๆถ๏ผๅฎ็ดๆฅ็ฟปๅฐๅ ๅซ่ฟไบ็กฎๅๆๅญ็้กต้ขใๅฎไธ่ฏๅพ็่งฃ”ไธๅฏๆๅ”ๆฏไปไนๆๆ๏ผๅฎๅชๆฏๆพๅฐ้ฃไบๅญๆฏๅบ็ฐ็้กต้ขใ่ฟๅฐฑๆฏๅฎ็ๅทฅไฝ๏ผๅญ้ขใ็ฒพ็กฎใๅฟซ้็ๆๆฌๅฎไฝใ
Specific Use Cases example
| ๅบๆฏ (Scenario) | BM25ๅฆไฝๅๆฅไฝ็จ (How BM25 helps) |
|---|---|
| ้่ฏฏไปฃ็ ๆฅ่ฏข (Error code lookup) | User searches “HTTP 503” โ BM25 finds the exact doc containing “503” |
| ไบงๅSKU/ๅบๅๅท (Product SKU/Serial #) | User searches “ABC-123-XYZ” โ BM25 precisely matches the alphanumeric string |
| ไผไธๅ ้จๆฏ่ฏญ (Enterprise jargon) | User searches “Databricks Unity Catalog” โ BM25 retrieves docs with those specific terms |
| ๆณๅพ/ๅ่งๆๆกฃ (Legal/Compliance) | User searches “GDPR Article 17” โ BM25 matches the exact legal reference |
| ๆฅๅฟๅๆ (Log analysis) | User searches “TimeoutException at line 42” โ BM25 finds the exact log entry |
Why use it?
You use BM25 because vector search (dense embeddings) has a fatal weakness: it understands meaning but ignores exact form. Here’s the hard truth:
ไฝ ไฝฟ็จBM25ๆฏๅ ไธบๅ้ๆฃ็ดข๏ผ็จ ๅฏๅตๅ
ฅ๏ผๆไธไธช่ดๅฝๅผฑ็น๏ผๅฎ็่งฃๅซไน๏ผไฝๅฟฝ็ฅ็ฒพ็กฎๅฝขๅผใไปฅไธๆฏ็กฌๆ ธ็็ธ๏ผ
Hard Keyword Matching (Serial Numbers, IDs, and Codes)
Vector embeddings compress text into semantic space, which often washes out the exact identity of unique strings like product IDs, error codes (e.g., ERR_404_AUTH), or specific numbers. If a user searches for a specific part number, vector search might return a “similar” product. BM25 treats text as discrete tokens, ensuring exact matches are surfaced instantly.
ๅ้ๅตๅ
ฅๅฐๆๆฌๅ็ผฉๅฐ่ฏญไน็ฉบ้ดไธญ๏ผ่ฟๅพๅพไผๆจก็ณๆๅฏไธๅญ็ฌฆไธฒ๏ผๅฆไบงๅ IDใ้่ฏฏไปฃ็ ๅฆ ERR_404_AUTH ๆ็นๅฎๆฐๅญ๏ผ็็ฒพ็กฎ็นๅพใๅฆๆ็จๆทๆ็ดข็นๅฎ็้ถไปถๅท๏ผๅ้ๆ็ดขๅฏ่ฝไผ่ฟๅไธไธชโ็ธไผผโ็ไบงๅใ่ BM25 ๅฐๆๆฌ่งไธบ็ฆปๆฃ็ Token๏ผ่ฝ็กฎไฟ็ซๅณๅน้
ๅฐ็ฒพ็กฎ็็ฎๆ ใ
The example Scenario
Our database has two documents:
- Document A: “Troubleshooting guide for pump model P-400. If you encounter error FX-992, it means the pressure valve is jammed. Clear the debris.” (P-400 ๅๆฐดๆณตๆ ้ๆ้คๆๅใๅฆๆ้ๅฐ FX-992 ้่ฏฏ๏ผ่ฏดๆๅๅ้ๅกไฝใ่ฏทๆธ ็็ข็ใ)
- Document B: “Troubleshooting guide for pump model P-401. If you encounter error FX-993, it means the pressure valve is jammed. Clear the debris.” (P-401 ๅๆฐดๆณตๆ ้ๆ้คๆๅใๅฆๆ้ๅฐ FX-993 ้่ฏฏ๏ผ่ฏดๆๅๅ้ๅกไฝใ่ฏทๆธ ็็ข็ใ)
ๅ้ๆ็ดขๆฏๅฆไฝ็ๅพ ่ฟไธช้ฎ้ข็๏ผไธบไปไนๅฎไผๅคฑ่ดฅ๏ผ
- EN: Vector search converts the query and documents into lists of numbers (embeddings) based on their meaning.
- To the vector model, Document A and Document B mean almost the exact same thing: “A troubleshooting guide for a water pump experiencing a jammed pressure valve.”
- Because the models compress the text,
P-400vsP-401andFX-992vsFX-993look 99% similar in the mathematical “semantic space”. - The Result: Vector search might score Document B higher than Document A just by random mathematical variance. If the RAG system feeds Document B to the LLM, the technician gets the wrong repair instructions for a completely different pump model.
- CN: ๅ้ๆ็ดขๆ นๆฎๅซไนๅฐๆฅ่ฏข่ฏๅๆๆกฃ่ฝฌๆขไธบไธไธฒๆฐๅญ๏ผๅตๅ
ฅๅ้๏ผใ
- ๅฏนไบๅ้ๆจกๅๆฅ่ฏด๏ผๆๆกฃ A ๅๆๆกฃ B ็ๅซไนๅ ไนๅฎๅ จ็ธๅ๏ผโๅ ณไบๆฐดๆณตๅๅ้ๅกไฝ็ๆ ้ๆ้คๆๅใโ
- ๅ ไธบๆจกๅๅ็ผฉไบๆๆฌ๏ผ
P-400ไธP-401ใFX-992ไธFX-993ๅจๆฐๅญฆ็โ่ฏญไน็ฉบ้ดโไธญ็่ตทๆฅๆ 99% ็็ธไผผๅบฆใ - ็ปๆ๏ผ ๅ้ๆ็ดขๅฏ่ฝไผๅ ไธบ้ๆบ็ๆฐๅญฆๅๅทฎ๏ผ็ปๆๆกฃ B ๆๅบๆฏๆๆกฃ A ๆด้ซ็ๅๆฐใๅฆๆ RAG ็ณป็ปๆๆๆกฃ B ๅ็ปไบๅคงๆจกๅ๏ผLLM๏ผ๏ผๆๆฏไบบๅๅฐฑไผๅพๅฐๅฎๅ จไธๅ็ๆฐดๆณตๅๅท็้่ฏฏ็ปดไฟฎๆไปคใ
Out-of-Vocabulary (OOV) & Domain-Specific Jargon
ๆช็ปๅฝ่ฏ๏ผOOV๏ผไธ่กไธไธไธๆฏ่ฏญ
Pre-trained embedding models have a fixed vocabulary. When your production data contains highly specialized enterprise jargon, internal acronyms, or brand-new product names, the vector model won’t understand the semantics and will guess poorly. BM25 doesn’t need to “understand” the word; it calculates frequency ($TF$) and rarity ($IDF$), making it incredibly robust for proprietary data.
้ข่ฎญ็ป็ๅตๅ
ฅๆจกๅ่ฏ่กจๆฏๅบๅฎ็ใๅฝๆจ็็ไบงๆฐๆฎๅ
ๅซ้ซๅบฆไธไธๅ็ไผไธๆฏ่ฏญใๅ
้จ็ผฉๅๆๅ
จๆฐๅๅธ็ไบงๅๅ็งฐๆถ๏ผๅ้ๆจกๅๆ ๆณ็่งฃๅ
ถ่ฏญไน๏ผๅช่ฝ่ฟ่ก็ณ็ณ็็ฒ็ใBM25 ไธ้่ฆโ็่งฃโ่ฟไธช่ฏ๏ผๅฎ็ดๆฅ่ฎก็ฎ่ฏ้ข๏ผ$TF$๏ผๅ็จ็ผบๅบฆ๏ผ$IDF$๏ผ๏ผ่ฟไฝฟๅพๅฎๅจๅค็็งๆไธๅฑๆฐๆฎๆถ่กจ็ฐๅพๅผๅธธ้ฒๆฃใ
Outperforming on Short Queries
When users type short, concise queries (e.g., “SQL deadlock fix”), vector models sometimes lack enough context to generate a high-quality embedding vector, leading to diluted results. BM25 shines here because it treats those exact terms as heavy anchors, instantly pulling documents containing those exact keywords.
ๅฝ็จๆท่พๅ
ฅ็ฎ็ญใ็ฒพ็ผ็ๆฅ่ฏข๏ผไพๅฆ โSQL ๆญป้ไฟฎๅคโ๏ผๆถ๏ผๅ้ๆจกๅๆๆถไผๅ ไธบ็ผบไน่ถณๅค็ไธไธๆ่ๆ ๆณ็ๆ้ซ่ดจ้็ๅตๅ
ฅๅ้๏ผๅฏผ่ดๆฃ็ดข็ปๆ่ขซ็จ้ใBM25 ๅจ่ฟ็งๅบๆฏไธๅคงๆพๅผๅฝฉ๏ผๅ ไธบๅฎๅฐ่ฟไบๅ
ทไฝ็่ฏ่งไธบๆ ธๅฟ้็น๏ผ็ฌ้ดๆๅๅ
ๅซ่ฟไบ็ฒพ็กฎๅ
ณ้ฎ่ฏ็ๆๆกฃใ
Cost, Speed, and Scale
Vector databases require specialized, memory-heavy hardware (RAM/GPUs) to perform Approximate Nearest Neighbor (ANN) searches at scale. BM25 runs on highly optimized, inverted indices (via OpenSearch, Elasticsearch, etc.) that are computationally cheap, lightning-fast, and can handle billions of documents with standard CPU architecture.
ๅ้ๆฐๆฎๅบ้่ฆไธ้จ็้ซๅ
ๅญ็กฌไปถ๏ผRAM/GPU๏ผๆฅๅจๅคง่งๆจกๆฐๆฎไธๆง่ก่ฟไผผๆ่ฟ้ป๏ผANN๏ผๆ็ดขใ่ BM25 ่ฟ่กๅจ้ซๅบฆไผๅ็ๅๆ็ดขๅผไธ๏ผ้่ฟ OpenSearchใElasticsearch ็ญ๏ผ๏ผ่ฎก็ฎๆๆฌๆไฝ๏ผ้ๅบฆๆๅฟซ๏ผไฝฟ็จๆ ๅ็ CPU ๆถๆๅณๅฏ่ฝปๆพๅค็ๆฐ็พไบฟๆกๆๆกฃใ
Decision Matrix example
| ๅบๆฏ (Scenario) | ๅช็จๅ้ (Vector Only) | ๅช็จBM25 (BM25 Only) | ๆททๅ (Hybrid) |
|---|---|---|---|
็จๆทๆ็ฝ่ง้่ฏฏ็ 0xDEADBEEF | โ ๅคฑ่ดฅ (้ถๅฌๅ) | โ ๅฎ็พ | โ ๅฎ็พ |
| ็จๆทๆ้็จๆฆๅฟต “cloud data integration” | โ ๅฅฝ (ๅไน่ฏๆณๅ) | โ ๏ธ ไธ่ฌ (ไป ๅญ้ข) | โ ๆๅฅฝ |
| ๅปถ่ฟๆๆๅ/้ซQPSๅบๆฏ | โ GPUๆ่ดต | โ CPUๆๅฟซ | โ ๏ธ ๅฏไผๅ(ไธค้ถๆฎต) |
| ๅ่งๅฎก่ฎก้่ฆ่งฃ้ๆฃ็ดขๅๅ | โ ไธๅฏ่งฃ้ | โ ๅฎๅ จๅฏ่งฃ้ | โ ๅฏ่งฃ้(BM25้จๅ) |
| ๆฐไบงๅไปฃๅท (Project Athena) ๅๅๅธ | โ ไธ่ฎค่ฏ | โ ๅณๆถๆฏๆ | โ ๅณๆถๆฏๆ |
One-Sentence Summary
ou use BM25 not because it’s “better” than vector search, but because it covers the exact-match failure cases that vector search inherently cannot, while simultaneously saving cost, enabling auditability, and serving as the mandatory lexical leg for Hybrid Search (RRF).
What does BM25 include?
Formula Overview: BM25 generates a relevance score for each document-query pair. The total score is the sum of scores for each query term.
BM25ไธบๆฏไธชๆๆกฃ-ๆฅ่ฏขๅฏน็ๆไธไธช็ธๅ
ณๆงๅๆฐใๆปๅๆฏๆฏไธชๆฅ่ฏข่ฏ้กนๅๆฐ็ๆปๅ
Three Core Improvements over TF-IDF:
Parameter k1
k1 ( 1.2-2.0) : Controls term frequency saturation. Higher k1 means term frequency continues to matter more; lower k1 means saturation happens faster.
โๅ
ณ้ฎ่ฏๅบ็ฐๅคๅฐๆฌกไนๅ๏ผๅๅ ๅๅฐฑๆฒกๅฅ็จไบโ
ๆไฝๅผ๏ผๅฆ k1=0๏ผ๏ผๅช่ฆๆๆกฃๅบ็ฐ่ฟโๆซ่จโ๏ผๅๆฐๅฐฑๅบๅฎไบ๏ผๅ้ขๅบ็ฐ 100 ๆฌกไนไธๅ ๅ๏ผ่ฟไบๆญปๆฟ๏ผใ
ไธญ้ดๅผ๏ผ้ป่ฎค k1=1.2~2.0๏ผ๏ผๅบ็ฐ็ฌฌ 1 ๆฌกๅ 10 ๅ๏ผๅบ็ฐ็ฌฌ 2 ๆฌกๅ 5 ๅ๏ผๅบ็ฐ็ฌฌ 3 ๆฌกๅ 2 ๅโฆโฆๅบ็ฐ็ฌฌ 10 ๆฌกๆถ๏ผๅ ๅๅ ไนไธบ 0๏ผ่ฟๅฐฑๆฏไฝ ็ฌฌไธ่ฝฎ้ฎ็โๆถ็้ๅโ๏ผใ
ๆ้ซๅผ๏ผๅฆ k1=10๏ผ๏ผๅบ็ฐ็ฌฌ 1 ๆฌกๅ 1 ๅ๏ผๅบ็ฐ็ฌฌ 10 ๆฌกๅ 9 ๅ๏ผๅ ไนๅ็บฟๆงๅข้ฟ๏ผ่ฟๅฐฑ้ๅๆ่ๆง็ TF-IDF๏ผๅฎนๆ่ขซๅ
ณ้ฎ่ฏๅ ็ ไฝๅผ๏ผใ
e.g.
k1 ๅณๅฎไฝ ็โ้ฅญ้โใๆญฃๅธธไบบ๏ผk1=1.2๏ผๅ 3 ๅๆซ่จๅฐฑ้ฅฑไบ๏ผๅไธ็ฌฌ 10 ๅไนๅไธไธไบ๏ผๅๆฐๅฐ้กถ๏ผใk1 ่ฐๅพ่ถไฝ๏ผไบบ่ถๅฎนๆ้ฅฑ๏ผ้ฅฑๅ่ถๅฟซ๏ผ๏ผ่ฐๅพ่ถ้ซ๏ผไบบ่ถ่ฝๅ๏ผ้ฅฑๅ่ถๆ
ข๏ผใ
Parameter “b”
parameter “b” (้ๅธธ 0.0-1.0) : Controls document length normalization. b=0 means no length normalization; b=1 means full normalization. Default is typically 0.75.
็ฎกโๅ ไธบๆ็ซ ๅคช้ฟ่ๆฃๅๆถ๏ผๆฃๅพๆๅค็ โ๏ผๅณไฝ ็ฌฌไบ่ฝฎ้ฎ็โ้ฟๆๆกฃๆฉ็ฝโ๏ผใ
ๆไฝๅผ๏ผb=0๏ผ๏ผๅฎๅ
จไธ็ๆ็ซ ้ฟๅบฆใไธ็ฏ 10000 ๅญ็็พ็งๅ
จไนฆๅไธ็ฏ 100 ๅญ็ๅพฎๅ็ญๆ๏ผๅๅฐ็ๅพ
้ๅฎๅ
จไธๆ ทใๅช่ฆ้ฟๆ้โๆซ่จโๅบ็ฐๅพๅค๏ผๅฎๅฐฑๆฐธ่ฟๆ็ฌฌไธ๏ผ่ฟไธๅ
ฌๅนณ๏ผใ
ๆ้ซๅผ๏ผb=1๏ผ๏ผไธฅๆ ผๆ็
ง้ฟๅบฆๆฏไพๆฃๅใ1000 ๅญ็ๆ็ซ ๏ผๆฉ็ฝๅฐฑๆฏ 100 ๅญ็ 10 ๅ๏ผ่ฟไบๆฟ่ฟ๏ผ่ฟๅฐฑๆฏ TF-IDF ็ๅผ็ซฏ๏ผใ
ไธญ้ดๅผ๏ผ้ป่ฎค b=0.75๏ผ๏ผๆไธญๆนๆกใ้ฟๆ็ซ ไผ่ขซๆฃไธ็นๅ๏ผไฝๆฃๅพๅพโๆธฉๆโใๅชๆๆ็ซ ๆ 10000 ๅญ๏ผๅช่ฆๆ ธๅฟๅๅ ๆฎตๅๅคๆๅฐไบโๆซ่จโ๏ผ็ฎๆณๅฐฑ็ฅ้ไฝ ็กฎๅฎๆฏ่ฎฒๆซ่จ็๏ผไธไผๅ ไธบๅ้ขๅบ่ฏๅคๅฐฑๆไฝ ๅฝปๅบๅๆฒกใ
็ฎๅ็ฒๆด็็่งฃ๏ผb ๅณๅฎโ่ฟ่ดน้ฉ็ๆฃ่ดนๆ ๅโใb=0 ๆๅณ็ไนฐ 1 ๆคๅไนฐ 100 ๆค่ฟ่ดนไธๆ ท๏ผๅฏน้ฟๆๅคชๅฎฝๅฎน๏ผ๏ผb=1 ๆๅณ็ไนฐ 100 ๆค่ฟ่ดนๆฏ 1 ๆค็ 100 ๅ๏ผๅฏน้ฟๆๅคช่ๅป๏ผ๏ผb=0.75 ๆๅณ็ไนฐ 100 ๆคๅชๆถ 1.5 ๅ็่ฟ่ดน๏ผ่ถ้ๅ ไปท่ถๅฐ๏ผ้ๅๆฉ็ฝ๏ผใ
Code Implementation
Step 0๏ผEnvironment Setup
pip install rank_bm25
# ๅฆๆไฝ ๅค็ไธญๆ๏ผๅฎ่ฃ
jieba๏ผๅค็่ฑๆๆจ่nltk๏ผไฝไปฅไธไปฃ็ ่ชๅธฆๆญฃๅ๏ผๅฏไธ่ฃ
๏ผ
# If handling Chinese, install jieba; for English, nltk is optional (our regex works fine)
pip install jieba numpyStep1๏ผDefine Tokenizer โ The Most Critical Step
ๅฎไนๅ่ฏๅจ
The tokenizer splits text into “terms”. BM25’s effectiveness depends entirely on this. Never use raw characters. Always: lowercase, strip punctuation, handle mixed languages.
ๅ่ฏๅจๅฐๆๆฌๆๅๆ”่ฏ้กน”ใBM25็ๆๆๅฎๅ
จๅๅณไบๆญคใๆฐธ่ฟไธ่ฆไฝฟ็จๅๅงๅญ็ฌฆใๆปๆฏ๏ผๅฐๅๅใๅป้คๆ ็นใๅค็ๆททๅ่ฏญ่จใ
# tokenizer_factory.py
# ็ฎ็: ๅๅปบ้็จไบBM25็ๅฅๅฃฎๅ่ฏๅจ
# Purpose: Create a robust tokenizer for BM25
import re
from typing import List
def create_bm25_tokenizer(language: str = "mixed"):
"""
ๅๅปบBM25ไธ็จๅ่ฏๅจ
Create a tokenizer specifically for BM25
BM25ๅ่ฏๅจ็ไธๅคงๅๅ (Three principles for BM25 tokenizer):
1. ็ปไธๅฐๅ (Unified lowercase) โ ็กฎไฟ "Timeout" ๅ "timeout" ่ขซ่ฏๅซไธบๅไธไธช่ฏ
2. ๅป้คๆ ็น (Strip punctuation) โ "timeout!" ๅๆ "timeout"
3. ไฟ็ๅญๆฏๆฐๅญ (Keep alphanumeric) โ "ABC-123" ไธญ็ "ABC" ๅ "123" ่ขซไฟ็
"""
if language == "zh" or language == "mixed":
try:
import jieba
def tokenizer(text: str) -> List[str]:
# 1. ็ปไธๅฐๅ (Unify case)
text_lower = text.lower()
# 2. ไฝฟ็จjiebaๅ่ฏ (Use jieba tokenization)
# jieba่ฝๅคๆบ่ฝๅค็ไธญ่ฑๆททๅๆๆฌ
# jieba intelligently handles mixed Chinese-English text
tokens = list(jieba.cut(text_lower))
# 3. ่ฟๆปคๆ็บฏ็ฉบ็ฝๅๅๅญ็ฌฆๆ ็น (Filter out pure whitespace and single-char punctuation)
# ้็น: ไฟ็ๆๆไน็่ฏ้กน๏ผๅป้คๅชๅฃฐ
# Key point: Keep meaningful tokens, remove noise
filtered = [t for t in tokens if t.strip() and not re.match(r'^[\W_]+$', t)]
# ๅฆๆๆฒกๆ่ฏ้กน๏ผ่ฟๅ็ฉบๅ่กจ๏ผๅ็ปญไผๅค็๏ผ
return filtered
return tokenizer
except ImportError:
print("โ ๏ธ jiebaๆชๅฎ่ฃ
๏ผไฝฟ็จๅ้็้็จๅ่ฏๅจ")
# ๅ้: ๆญฃๅๅน้
ๆๆๅญๆฏๆฐๅญๅบๅ (Fallback: regex match all alphanumeric sequences)
return _fallback_tokenizer
else:
# ็บฏ่ฑๆๅ่ฏๅจ (Pure English tokenizer)
return _english_tokenizer
def _english_tokenizer(text: str) -> List[str]:
"""่ฑๆๅ่ฏๅจ: ๅฐๅ + ๆญฃๅๆๅๅ่ฏ (Lowercase + regex extract words)"""
text_lower = text.lower()
# \b[a-zA-Z0-9_]+\b ๅน้
ๆๆๅ่ฏ๏ผๅ
ๆฌๅธฆไธๅ็บฟ็
# ้็น: ่ฟไผๆ "error_code" ไฝไธบไธไธชๆดไฝไฟ็๏ผ่ไธๆฏๆๆ "error" ๅ "code"
# Key point: This keeps "error_code" as one token, not split into "error" and "code"
return re.findall(r'\b[a-zA-Z0-9_]+\b', text_lower)
def _fallback_tokenizer(text: str) -> List[str]:
"""ๅ้ๅ่ฏๅจ: ็บฏๅญ็ฌฆ็บงๅซๆ่งฃ + ่ฟๆปค (Fallback: character-level split + filter)"""
text_lower = text.lower()
# ๅชไฟ็ไธญ่ฑๆๅๆฐๅญๅญ็ฌฆ (Keep only Chinese, English letters, and digits)
chars = [ch for ch in text_lower if ch.isalnum() or ('\u4e00' <= ch <= '\u9fff')]
# ๆ็ฉบๆ ผๆ่ฟ็ปญ่ฑๆ/ไธญๆๅ็ป๏ผ็ฎๅ็๏ผ
# ่ฟ้็ฎๅ่ฟๅๅญ็ฌฆๅ่กจ๏ผไฝ็ไบง็ฏๅขไธๆจ่
return chars if chars else ["[EMPTY]"]Step 2๏ผIndex Building โ “Feed” documents to BM25
# build_bm25_index.py
# ็ฎ็: ๅฐ่ฏญๆๅบๅ่ฏๅนถๆๅปบBM25็ดขๅผ
# Purpose: Tokenize corpus and build BM25 index
from rank_bm25 import BM25Okapi
from typing import List
def build_bm25_index(
corpus: List[str],
tokenizer,
k1: float = 1.5,
b: float = 0.75
) -> tuple[BM25Okapi, List[List[str]]]:
"""
ๆๅปบBM25็ดขๅผ
Build BM25 index
Args:
corpus: ๅๅงๆๆกฃๅ่กจ (Raw document list)
tokenizer: ๅ่ฏๅจๅฝๆฐ (Tokenizer function)
k1: ่ฏ้ข้ฅฑๅๅบฆๅๆฐ (Term frequency saturation)
b: ้ฟๅบฆๅฝไธๅๅๆฐ (Length normalization)
Returns:
(bm25_object, tokenized_corpus)
"""
print("๐ ๅผๅงๆๅปบBM25็ดขๅผ (Building BM25 index)...")
# ============================================================
# ๆญฅ้ชค2.1: ๅฏน่ฏญๆๅบ้็ฏๅ่ฏ (Tokenize each document)
# ============================================================
tokenized_corpus = []
empty_doc_count = 0
for idx, doc in enumerate(corpus):
tokens = tokenizer(doc)
# ๅ
ณ้ฎๆฃๆฅ: ๅฆๆไธ็ฏๆๆกฃๅ่ฏๅๆฒกๆ่ฏ้กน๏ผBM25ไผๆฅ้ๆๅฟฝ็ฅๅฎ
# Critical check: If a document has no tokens after tokenization, BM25 errors or ignores it
if not tokens:
empty_doc_count += 1
# ๆๅ
ฅไธไธชๅ ไฝ็ฌฆ๏ผ็กฎไฟ็ดขๅผๅฏไปฅๆญฃๅธธๅทฅไฝ
# Insert a placeholder to keep the index working
tokens = ["[EMPTY_DOC]"]
tokenized_corpus.append(tokens)
# ่ฐ่ฏ: ๆๅฐๅ3็ฏๆๆกฃ็ๅ่ฏ็ปๆ (Debug: print first 3)
if idx < 3:
print(f" Doc {idx} tokens: {tokens[:10]}{'...' if len(tokens) > 10 else ''}")
if empty_doc_count > 0:
print(f"โ ๏ธ ๆ {empty_doc_count} ็ฏ็ฉบๆๆกฃ๏ผๅทฒๆๅ
ฅๅ ไฝ็ฌฆ")
# ============================================================
# ๆญฅ้ชค2.2: ๅๅงๅBM25Okapi (Initialize BM25Okapi)
# ============================================================
# ้็น: BM25Okapi็ๆ้ ๅฝๆฐๆฅๅ tokenized corpus, k1, b
# Key point: BM25Okapi constructor accepts tokenized corpus, k1, b
bm25 = BM25Okapi(tokenized_corpus, k1=k1, b=b)
print(f"โ
็ดขๅผๆๅปบๅฎๆ (Index built):")
print(f" - ๆๆกฃๆฐ (Docs): {len(corpus)}")
print(f" - ๅนณๅๆๆกฃ้ฟๅบฆ (AvgDL): {bm25.avgdl:.2f}")
print(f" - ๅๆฐ k1: {k1}, b: {b}")
return bm25, tokenized_corpusStep3๏ผExecute Query โ Core Retrieval Logic
# query_bm25.py
# ็ฎ็: ๅฏนๆฅ่ฏข่ฟ่กๅ่ฏๅนถ่ทๅTop-K็ปๆ
# Purpose: Tokenize query and retrieve Top-K results
import numpy as np
from typing import List, Tuple
def search_bm25(
bm25: BM25Okapi,
tokenizer,
query: str,
corpus: List[str],
top_k: int = 5,
normalize: bool = True
) -> List[Tuple[str, float]]:
"""
ๆง่กBM25ๆฃ็ดข
Execute BM25 search
ๅฎๆดๆต็จ:
1. ๅ่ฏๆฅ่ฏข (Tokenize query)
2. ่ฎก็ฎๆๆๆๆกฃ็BM25ๅๆฐ (Compute BM25 scores for all docs)
3. ๆๅๆฐ้ๅบๆๅบๅtop_k (Sort descending and take top_k)
4. (ๅฏ้) ๅฝไธๅๅฐ0-1 (Optionally normalize to 0-1)
"""
# ============================================================
# ๆญฅ้ชค3.1: ๅ่ฏๆฅ่ฏข (Tokenize query)
# ============================================================
# ้็น: ๆฅ่ฏขๅฟ
้กปไฝฟ็จไธๆๆกฃๅฎๅ
จ็ธๅ็ๅ่ฏๅจ
# Key point: Query MUST use the exact same tokenizer as documents
query_tokens = tokenizer(query)
# ๅฆๆๆฅ่ฏขๅ่ฏๅไธบ็ฉบ๏ผ่ฟๅ็ฉบ็ปๆ (If query has no tokens, return empty)
if not query_tokens:
print("โ ๏ธ ๆฅ่ฏขๆ ๆๆ่ฏ้กน๏ผ่ฟๅ็ฉบ็ปๆ")
return []
print(f"๐ ๆฅ่ฏข่ฏ้กน (Query tokens): {query_tokens}")
# ============================================================
# ๆญฅ้ชค3.2: ่ทๅๆๆๆๆกฃ็ๅๅงBM25ๅๆฐ (Get raw BM25 scores)
# ============================================================
# get_scores() ่ฟๅไธไธชlist๏ผ็ดขๅผๅฏนๅบๆๆกฃ้กบๅบ
# get_scores() returns a list, index aligns with document order
raw_scores = bm25.get_scores(query_tokens)
# ============================================================
# ๆญฅ้ชค3.3: ๆๅบๅนถๆๅTop-K (Sort and extract Top-K)
# ============================================================
# ๅฐๅๆฐๅ็ดขๅผ้
ๅฏน (Pair scores with indices)
scored_docs = [(idx, score) for idx, score in enumerate(raw_scores)]
# ๆๅๆฐ้ๅบๆๅบ (Sort by score descending)
# ้็น: ไฝฟ็จkey=lambda x: x[1] ๆๅๆฐๆๅบ
# Key point: Use key=lambda x: x[1] to sort by score
sorted_docs = sorted(scored_docs, key=lambda x: x[1], reverse=True)
# ่ฟๆปคๆๅๆฐไธบ0็็ปๆ๏ผๅฏ้๏ผ (Filter out zero-score results โ optional)
# ๆไน: ๅๆฐไธบ0ๆๅณ็ๆฒกๆไปปไฝๆฅ่ฏข่ฏ้กนๅบ็ฐๅจ่ฏฅๆๆกฃไธญ๏ผๅฎๅ
จๆ ๅ
ณ
# Meaning: Score 0 means no query term appears in the document; completely irrelevant
relevant_docs = [(idx, score) for idx, score in sorted_docs if score > 0]
# ๅๅtop_kไธช (Take top_k)
top_k_docs = relevant_docs[:top_k]
# ============================================================
# ๆญฅ้ชค3.4: ๅฝไธๅๅๆฐ (Normalize scores โ optional)
# ============================================================
# ็ฎ็: ๅฐๅๆฐๆ ๅฐๅฐ0-1ไน้ด๏ผไพฟไบไบบ็ฑป้
่ฏปๅๅ็ปญ่ๅ(RRFไธ้่ฆ)
# Purpose: Map scores to 0-1 for readability (RRF doesn't need this)
if normalize and top_k_docs:
max_score = top_k_docs[0][1] # ๆ้ซๅ (Highest score)
if max_score > 0:
normalized_results = [
(corpus[idx], score / max_score)
for idx, score in top_k_docs
]
return normalized_results
# ่ฟๅๅๅงๅๆฐ (Return raw scores)
return [(corpus[idx], score) for idx, score in top_k_docs]End-to-End Complete Example โ Put It All Together
# bm25_complete_pipeline.py
# ็ฎ็: ๅฎๆด็BM25ๆฃ็ดขๆตๆฐด็บฟ๏ผไปๅๅงๆๆฌๅฐๆฃ็ดข็ปๆ๏ผ
# Purpose: Complete BM25 retrieval pipeline (from raw text to retrieval results)
from rank_bm25 import BM25Okapi
import numpy as np
import re
from typing import List, Tuple
# ============================================================
# 1. ๅฎไนๅ่ฏๅจ (Define Tokenizer)
# ============================================================
def simple_mixed_tokenizer(text: str) -> List[str]:
"""
ๆททๅ่ฏญ่จๅ่ฏๅจ๏ผๆ ้้ขๅคๅบ๏ผ
Mixed language tokenizer (no extra libraries required)
ๅค็ๆนๆณ:
1. ็ปไธๅฐๅ (Lowercase)
2. ็จๆญฃๅๆๅๆๆๅญๆฏๆฐๅญๅบๅ (Extract all alphanumeric sequences)
3. ๅๆถไฟ็ไธญๆๅญ็ฌฆ (Keep Chinese characters too)
"""
text_lower = text.lower()
# ๅน้
: ่ฑๆๅญๆฏ+ๆฐๅญ+ไธๅ็บฟ๏ผไปฅๅไธญๆๅญ็ฌฆ
# Match: English letters + digits + underscores, and Chinese characters
# ้็น: ่ฟ่ฝๅค็ "Azureๆฐๆฎๅทฅๅ" ่ฟๆ ท็ๆททๅๆๆฌ
# Key point: This handles mixed text like "Azureๆฐๆฎๅทฅๅ"
pattern = r'[a-zA-Z0-9_]+|[\u4e00-\u9fa5]+'
tokens = re.findall(pattern, text_lower)
# ่ฟๆปคๆ็บฏๆฐๅญ๏ผๅฏ้๏ผ่งๆ
ๅต่ๅฎ๏ผ
# ๅฆๆไฝ ็ๅบๆฏ้่ฆๅน้
ๆฐๅญ๏ผๅฆ้่ฏฏ็ 500๏ผ๏ผไธ่ฆ่ฟๆปค๏ผ
# ่ฟ้ไฟ็ๆๆtoken
# Filter out pure numbers (optional). If you need error codes like 500, DON'T filter!
# We'll keep all tokens here.
return tokens if tokens else ["[EMPTY]"]
# ============================================================
# 2. ๅๅคๆฐๆฎ (Prepare Data)
# ============================================================
corpus = [
"Azure Data Factory is a cloud-based data integration service",
"Databricks is a unified analytics platform for data engineering and ML",
"BM25 is a probabilistic ranking algorithm used in information retrieval",
"Azure AI Foundry provides tools for building enterprise AI applications",
"When Azure Function App times out, error code 500 is returned",
"Databricks Photon engine accelerates query performance on large datasets",
"The data pipeline uses Event Hubs to ingest streaming data",
"Error 500: Internal Server Error - check the application logs",
]
# ============================================================
# 3. ๆๅปบ็ดขๅผ (Build Index)
# ============================================================
print("="*70)
print("ๆญฅ้ชค 1: ๅ่ฏๅนถๆๅปบ็ดขๅผ (Tokenize & Build Index)")
print("="*70)
tokenized_corpus = []
for doc in corpus:
tokens = simple_mixed_tokenizer(doc)
tokenized_corpus.append(tokens)
# ๆๅฐๅ่ฏ็ปๆ้ข่ง (Print tokenization preview)
for i, tokens in enumerate(tokenized_corpus[:3]):
print(f"Doc {i}: {tokens}")
# ๅๅงๅBM25 (Initialize BM25)
# ไฝฟ็จ้ป่ฎคๅๆฐ k1=1.5, b=0.75
bm25 = BM25Okapi(tokenized_corpus, k1=1.5, b=0.75)
print(f"\nโ
็ดขๅผๆๅปบๅฎๆ (Index built)")
print(f" ๆๆกฃๆฐ (Docs): {len(corpus)}")
print(f" ๅนณๅๆๆกฃ้ฟๅบฆ (AvgDL): {bm25.avgdl:.2f}")
# ============================================================
# 4. ๆง่กๆฅ่ฏข (Execute Query)
# ============================================================
print("\n" + "="*70)
print("ๆญฅ้ชค 2: ๆง่กๆฅ่ฏข (Execute Query)")
print("="*70)
# ๆฅ่ฏข1: ็ฒพ็กฎ้่ฏฏ็ (Exact error code)
query1 = "error code 500"
query1_tokens = simple_mixed_tokenizer(query1)
print(f"ๆฅ่ฏข1่ฏ้กน: {query1_tokens}")
scores1 = bm25.get_scores(query1_tokens)
# ่ทๅๅ3ไธช็ปๆ (Get top 3)
top_indices1 = np.argsort(scores1)[::-1][:3]
print(f"\n๐ ๆฅ่ฏข (Query): '{query1}'")
print("็ปๆ (Results):")
for rank, idx in enumerate(top_indices1):
if scores1[idx] > 0:
print(f" {rank+1}. ๅๆฐ {scores1[idx]:.4f} -> {corpus[idx]}")
# ๆฅ่ฏข2: ไบงๅๅ็งฐ + ๅ่ฝ (Product name + feature)
query2 = "Databricks Photon acceleration"
query2_tokens = simple_mixed_tokenizer(query2)
print(f"\nๆฅ่ฏข2่ฏ้กน: {query2_tokens}")
scores2 = bm25.get_scores(query2_tokens)
top_indices2 = np.argsort(scores2)[::-1][:3]
print(f"\n๐ ๆฅ่ฏข (Query): '{query2}'")
print("็ปๆ (Results):")
for rank, idx in enumerate(top_indices2):
if scores2[idx] > 0:
print(f" {rank+1}. ๅๆฐ {scores2[idx]:.4f} -> {corpus[idx]}")
# ๆฅ่ฏข3: ๆททๅไธญ่ฑ (Mixed CN-EN)
query3 = "ๆฐๆฎๅทฅ็จ pipeline ่ถ
ๆถ"
query3_tokens = simple_mixed_tokenizer(query3)
print(f"\nๆฅ่ฏข3่ฏ้กน: {query3_tokens}")
scores3 = bm25.get_scores(query3_tokens)
top_indices3 = np.argsort(scores3)[::-1][:3]
print(f"\n๐ ๆฅ่ฏข (Query): '{query3}'")
print("็ปๆ (Results):")
for rank, idx in enumerate(top_indices3):
if scores3[idx] > 0:
print(f" {rank+1}. ๅๆฐ {scores3[idx]:.4f} -> {corpus[idx]}")
# ============================================================
# 5. ๅๆฐ่ฐไผๆผ็คบ (Parameter Tuning Demo)
# ============================================================
print("\n" + "="*70)
print("ๆญฅ้ชค 3: ๅๆฐ่ฐไผ (Parameter Tuning)")
print("="*70)
# ๅฐ่ฏไธๅ็k1ๅผ (Try different k1 values)
test_k1s = [1.0, 1.5, 2.0]
test_query = "Azure Function App timeout"
print(f"ๆต่ฏๆฅ่ฏข: '{test_query}'")
print("ไธๅk1ๅผๅฏน็ปๆ็ๅฝฑๅ (Impact of different k1 values):")
for k1 in test_k1s:
# ้ๆฐๅๅปบBM25ๅฏน่ฑก (Re-create BM25 object)
temp_bm25 = BM25Okapi(tokenized_corpus, k1=k1, b=0.75)
tokens = simple_mixed_tokenizer(test_query)
scores = temp_bm25.get_scores(tokens)
top_idx = np.argmax(scores) # ๅๆ้ซๅๆๆกฃ (Take highest scoring doc)
print(f" k1={k1:.1f}: ๆไฝณๆๆกฃ็ดขๅผ {top_idx} -> '{corpus[top_idx][:50]}...' (score: {scores[top_idx]:.4f})")Key Takeaways
| ่ฆ็น | EN | CN |
|---|---|---|
| BM25่งฃๅณๅ้ๆฃ็ดข็”็ฒพ็กฎๅน้ ็ฒๅบ”้ฎ้ข | BM25 solves the “exact match blind spot” of vector retrieval | BM25่งฃๅณๅ้ๆฃ็ดข็”็ฒพ็กฎๅน้ ็ฒๅบ”้ฎ้ข |
| BM25ๆฏๆททๅๆฃ็ดข๏ผHybrid Search๏ผ็ๅ ณ้ฎ”ๅ ณ้ฎ่ฏๅๆฏ” | BM25 is the essential “keyword leg” of Hybrid Search | BM25ๆฏๆททๅๆฃ็ดข๏ผHybrid Search๏ผ็ๅ ณ้ฎ”ๅ ณ้ฎ่ฏๅๆฏ” |
k1ๆงๅถ่ฏ้ข้ฅฑๅๅบฆ๏ผๅผ่ถ้ซ๏ผ้ซ้ข่ฏ่ดก็ฎ่ถๅคง | k1 controls term frequency saturation: higher value = more contribution from high-frequency terms | k1ๆงๅถ่ฏ้ข้ฅฑๅๅบฆ๏ผๅผ่ถ้ซ๏ผ้ซ้ข่ฏ่ดก็ฎ่ถๅคง |
bๆงๅถๆๆกฃ้ฟๅบฆๅฝไธๅ๏ผb=1ๅฎๅ
จๆฉ็ฝ้ฟๆๆกฃ๏ผb=0ไธๆฉ็ฝ | b controls document length normalization: b=1 fully penalizes long docs, b=0 doesn’t | bๆงๅถๆๆกฃ้ฟๅบฆๅฝไธๅ๏ผb=1ๅฎๅ
จๆฉ็ฝ้ฟๆๆกฃ๏ผb=0ไธๆฉ็ฝ |
| ๆฅ่ฏขๅๆๆกฃๅฟ ้กปไฝฟ็จๅฎๅ จ็ธๅ็ๅ่ฏๅ้ขๅค็ๆต็จ | Query and documents must use exactly the same tokenization and preprocessing pipeline | ๆฅ่ฏขๅๆๆกฃๅฟ ้กปไฝฟ็จๅฎๅ จ็ธๅ็ๅ่ฏๅ้ขๅค็ๆต็จ |
| BM25่ฟ่กๅจCPUไธ๏ผๅปถ่ฟไฝ๏ผ~50ms๏ผ๏ผๆๆฌๆไฝ | BM25 runs on CPU with low latency (~50ms) and extremely low cost | BM25่ฟ่กๅจCPUไธ๏ผๅปถ่ฟไฝ๏ผ~50ms๏ผ๏ผๆๆฌๆไฝ |
็ไบง้จ็ฝฒๆถ๏ผ็จ้ช่ฏ้ๅฏนk1ๅbๅ็ฝๆ ผๆ็ดข่ฐไผ | In production, tune k1 and b via grid search on a validation set | ็ไบง้จ็ฝฒๆถ๏ผ็จ้ช่ฏ้ๅฏนk1ๅbๅ็ฝๆ ผๆ็ดข่ฐไผ |
| ไธไธไธชๅๅ ๏ผB07๏ผๅฐ็จRRF่ๅBM25ๅๅ้ๆฃ็ดข | The next unit (B07) will fuse BM25 with vector retrieval using RRF | ไธไธไธชๅๅ ๏ผB07๏ผๅฐ็จRRF่ๅBM25ๅๅ้ๆฃ็ดข |
