Tag: FastText

Embedding is a fundamental technology in machine learning and natural language processing that transforms discrete or complex objects (such as words, sentences, or images) into numerical vector representations of a fixed dimension.

In a RAG (Retrieval-Augmented Generation) system, embedding is the foundation of retrieval. Without embeddings, you cannot perform semantic search — you would be stuck with keyword matching only

Core Concepts

Vector Space & Similarity

Embeddings live in a multi-dimensional vector space. To measure how similar two pieces of text are, we measure the distance between their embedding vectors. The most common measure is cosine similarity.
存在于一个多维向量空间中。要衡量两段文本有多相似，我们衡量它们的 Embedding 向量之间的距离。最常用的度量是余弦相似度（Cosine Similarity）.

Cosine Similarity explained:

• Range: -1 (opposite) to 1 (identical)
  -1（相反）到 1（完全相同）
• For text embeddings, values close to 1 mean high semantic similarity
  对于文本 Embedding，接近 1 的值表示高语义相似度
• It measures the cosine of the angle between two vectors, ignoring magnitude
  它测量两个向量之间的夹角余弦值，忽略向量大小

import numpy as np

def cosine_similarity(vec_a: np.ndarray, vec_b: np.ndarray) -> float:
    """
    Calculate cosine similarity between two vectors.
    计算两个向量之间的余弦相似度。
    
    Purpose: Measures semantic similarity between two embeddings.
    目的：衡量两个 Embedding 之间的语义相似度。
    """
    # Normalize vectors / 归一化向量
    norm_a = np.linalg.norm(vec_a)
    norm_b = np.linalg.norm(vec_b)
    
    # Dot product / 点积
    dot_product = np.dot(vec_a, vec_b)
    
    # Cosine similarity = dot product / (norm_a * norm_b)
    # 余弦相似度 = 点积 / (norm_a * norm_b)
    return dot_product / (norm_a * norm_b)

Dense vs Sparse Vectors

稠密向量 vs 稀疏向量

Aspect	Dense Embedding	Sparse (One-Hot)
Dimension	Fixed, low (e.g., 384, 768, 1536)	Very high (vocabulary size)
Values	Continuous floats	0 or 1
Semantic info 语义信息	Captures meaning 捕获含义	Only captures identity 仅捕获身份
Efficiency	Computationally efficient	Wastes memory

Sparse (One-Hot) 中，每个词是一个独立的维度，”king”是 [0,0,0,…,1,…,0]，”queen”是另一个位置。它们没有任何语义关系。而 Embedding 中，”king”和”queen”的向量是相似的。

Captures meaning 捕获含义 = 不看字，看“意思像不像”
Only captures identity 仅捕获身份 = 只认“是不是这个词”，不管什么意思

Contextual vs Static Embeddings

上下文 Embedding vs 静态 Embedding

Type	Examples	Characteristics
Static	Word2Vec, GloVe, FastText	One vector per word, regardless of context 每个词一个向量，无论上下文
Contextual	BERT-based, OpenAI text-embedding-3	Vector changes based on surrounding words 向量根据周围词变化

• Word2Vec: Word2Vec is a method that learns word meanings by looking at the context in which words appear.
  是一种通过“上下文”来学习词语含义的模型。它通过“上下文预测”学习词向量的方法。看词和词之间的搭配关系来学习语义,让语义相似的词在向量空间中靠得更近。
• GloVe = Global Vectors for Word Representation. It learns word meaning from: global word co-occurrence statistics
  “统计全世界词和词的关系”
• FastText：FastText (Facebook) represents words as： sum of character n-grams把“词拆成字母/子词”来学向量

“bank” in “river bank” vs “bank account” ： static embedding gives the same vector; contextual embedding gives different vectors based on context.
静态 Embedding 给相同的向量；上下文 Embedding 根据上下文给不同的向量。

BERT = Bidirectional Encoder Representations from Transformers

It learns: context-aware word meaning using Transformer attention
用 Transformer（注意力机制）来理解上下文的语言模型。
“河岸”，“银行账号” “同一个词，在不同句子里有不同含义”

Mainstream Embedding Models

For modern RAG systems, the most relevant embedding models are

Model	Provider	Dim	Notes
text-embedding-3-small	OpenAI	1536	Cost-effective, good for most RAG
text-embedding-3-large	OpenAI	3072	Higher quality, higher cost
text-embedding-ada-002	OpenAI	1536	Legacy, being replaced by v3
BGE-M3	BAAI	1024	Open-source, multilingual
KaLM-Embedding-Gemma3-12B	Tencent	3840	SOTA on MMTEB

Code Implementation

OpenAI API generates Embedding：

# ============================================================
# 1. IMPORTS / 导入依赖
# ============================================================
import os
import numpy as np
from openai import OpenAI
from typing import List, Dict, Any

# ============================================================
# 2. CLIENT INITIALIZATION / 客户端初始化
# ============================================================
# Purpose: Initialize the OpenAI client with API key from environment.
# 目的：使用环境变量中的 API Key 初始化 OpenAI 客户端。
client = OpenAI(
    api_key=os.environ.get("OPENAI_API_KEY"),  # Get from env / 从环境变量获取
)

# ============================================================
# 3. EMBEDDING FUNCTION / Embedding 生成函数
# ============================================================
def get_embedding(
    text: str, 
    model: str = "text-embedding-3-small"
) -> List[float]:
    """
    Generate an embedding vector for a given text using OpenAI API.
    使用 OpenAI API 为给定文本生成 Embedding 向量。
    
    Purpose: Convert text into a numerical vector for semantic search.
    目的：将文本转换为数值向量，用于语义搜索。
    
    Args:
        text: Input text string / 输入文本字符串
        model: Embedding model name / Embedding 模型名称
        
    Returns:
        List[float]: Embedding vector of length model-specific dimension
        List[float]: Embedding 向量，长度为模型指定的维度
        
    Important: The same model must be used for both query and document embeddings
    重点：查询和文档的 Embedding 必须使用同一个模型[reference:30]
    """
    # Remove newlines and excess whitespace for cleaner embedding
    # 移除换行和多余空白，获得更干净的 Embedding
    text = text.replace("\n", " ")
    
    # Call OpenAI Embeddings API / 调用 OpenAI Embeddings API
    # Purpose: Send text to OpenAI and get back the embedding vector
    # 目的：发送文本到 OpenAI，获取 Embedding 向量
    response = client.embeddings.create(
        model=model,           # Which embedding model to use / 使用的模型
        input=text,            # Text to embed / 要嵌入的文本
        encoding_format="float"  # Return as list of floats / 以浮点数列表返回
    )
    
    # Extract the embedding from response / 从响应中提取 Embedding
    # Purpose: The embedding is in response.data[0].embedding
    # 目的：Embedding 位于 response.data[0].embedding 中
    embedding = response.data[0].embedding
    
    return embedding

# ============================================================
# 4. BATCH EMBEDDING / 批量 Embedding
# ============================================================
def get_embeddings_batch(
    texts: List[str],
    model: str = "text-embedding-3-small"
) -> List[List[float]]:
    """
    Generate embeddings for multiple texts in a single API call.
    在单个 API 调用中为多个文本生成 Embedding。
    
    Purpose: More efficient than calling get_embedding() in a loop.
    目的：比在循环中调用 get_embedding() 更高效。
    
    Important: OpenAI limits total tokens to 300,000 per request[reference:31]
    重点：OpenAI 限制每个请求总 token 不超过 300,000[reference:32]
    """
    # Clean texts / 清理文本
    cleaned_texts = [t.replace("\n", " ") for t in texts]
    
    # Batch API call / 批量 API 调用
    response = client.embeddings.create(
        model=model,
        input=cleaned_texts,   # List of texts / 文本列表
        encoding_format="float"
    )
    
    # Extract all embeddings / 提取所有 Embedding
    # Purpose: Each response.data[i] corresponds to texts[i]
    # 目的：每个 response.data[i] 对应 texts[i]
    embeddings = [item.embedding for item in response.data]
    
    return embeddings

# ============================================================
# 5. SIMILARITY SEARCH / 相似度搜索
# ============================================================
def cosine_similarity(vec_a: List[float], vec_b: List[float]) -> float:
    """
    Calculate cosine similarity between two embedding vectors.
    计算两个 Embedding 向量之间的余弦相似度。
    
    Purpose: Measure semantic similarity between query and document.
    目的：衡量查询和文档之间的语义相似度。
    
    Important: Embeddings must be from the SAME model.
    重点：Embeddings 必须来自同一个模型。
    """
    # Convert to numpy arrays for efficient math / 转换为 numpy 数组以便高效计算
    a = np.array(vec_a)
    b = np.array(vec_b)
    
    # Calculate cosine similarity / 计算余弦相似度
    # Formula: cos(θ) = (A · B) / (||A|| * ||B||)
    # 公式：cos(θ) = (A · B) / (||A|| * ||B||)
    dot_product = np.dot(a, b)
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    
    # Avoid division by zero / 避免除以零
    if norm_a == 0 or norm_b == 0:
        return 0.0
    
    return float(dot_product / (norm_a * norm_b))

def find_most_similar(
    query: str,
    documents: List[str],
    model: str = "text-embedding-3-small",
    top_k: int = 3
) -> List[Dict[str, Any]]:
    """
    Find the most semantically similar documents to a query.
    找到与查询语义最相似的文档。
    
    Purpose: Core retrieval function for RAG systems.
    目的：RAG 系统的核心检索函数[reference:33]。
    
    Args:
        query: User question / 用户问题
        documents: List of document chunks / 文档块列表
        model: Embedding model / Embedding 模型
        top_k: Number of top results to return / 返回前 K 个结果
    
    Returns:
        List of dicts with document text and similarity score
        包含文档文本和相似度分数的字典列表
    """
    # Step 1: Embed the query / 第一步：嵌入查询
    # Purpose: Convert user question to vector for comparison
    # 目的：将用户问题转换为向量以便比较
    query_embedding = get_embedding(query, model)
    
    # Step 2: Embed all documents / 第二步：嵌入所有文档
    # Purpose: Convert all documents to vectors
    # 目的：将所有文档转换为向量
    doc_embeddings = get_embeddings_batch(documents, model)
    
    # Step 3: Calculate similarities / 第三步：计算相似度
    # Purpose: Compare query vector against all document vectors
    # 目的：将查询向量与所有文档向量比较
    results = []
    for i, doc_embedding in enumerate(doc_embeddings):
        score = cosine_similarity(query_embedding, doc_embedding)
        results.append({
            "document": documents[i],
            "score": score,
            "index": i
        })
    
    # Step 4: Sort by score descending and return top_k
    # 第四步：按分数降序排序，返回 top_k
    # Purpose: Return the most relevant documents first
    # 目的：首先返回最相关的文档
    results.sort(key=lambda x: x["score"], reverse=True)
    return results[:top_k]

# ============================================================
# 6. USAGE EXAMPLE / 使用示例
# ============================================================
if __name__ == "__main__":
    # Example: RAG retrieval scenario / 示例：RAG 检索场景
    
    # User question / 用户问题
    query = "What is the capital of France?"
    
    # Document chunks from a knowledge base / 来自知识库的文档块
    documents = [
        "France is a country in Western Europe. Its capital is Paris.",
        "Germany's capital is Berlin. It is the largest city in Germany.",
        "The Eiffel Tower is located in Paris, France.",
        "London is the capital of the United Kingdom."
    ]
    
    # Find most relevant documents / 找到最相关的文档
    top_results = find_most_similar(query, documents, top_k=2)
    
    print(f"Query: {query}")
    print("\nTop Results:")
    for i, result in enumerate(top_results):
        print(f"{i+1}. Score: {result['score']:.4f}")
        print(f"   Document: {result['document']}")
        print()
    
    # Expected output: Documents about France and Paris should rank highest
    # 预期输出：关于法国和巴黎的文档应该排名最高

Azure OpenAI generates Embedding（Azure AI Foundry intgerate）

# ============================================================
# AZURE OPENAI EMBEDDING / Azure OpenAI Embedding
# ============================================================
from openai import AzureOpenAI

# Initialize Azure OpenAI client / 初始化 Azure OpenAI 客户端
# Purpose: Use Azure OpenAI Service instead of OpenAI's direct API
# 目的：使用 Azure OpenAI 服务替代 OpenAI 的直接 API
azure_client = AzureOpenAI(
    api_key=os.environ.get("AZURE_OPENAI_API_KEY"),
    api_version="2024-02-15-preview",  # Azure API version / Azure API 版本
    azure_endpoint=os.environ.get("AZURE_OPENAI_ENDPOINT"),
)

def get_azure_embedding(text: str, deployment: str = "text-embedding-3-small") -> List[float]:
    """
    Generate embedding using Azure OpenAI Service.
    使用 Azure OpenAI 服务生成 Embedding。
    
    Purpose: Enterprise-grade embedding with Azure's infrastructure.
    目的：使用 Azure 基础设施的企业级 Embedding。
    """
    response = azure_client.embeddings.create(
        model=deployment,  # Deployment name in Azure / Azure 中的部署名称
        input=text,
        encoding_format="float"
    )
    return response.data[0].embedding

Key Takeaways

要点	EN	CN
Embedding 是将文本转换为数值向量	Embedding converts text to numerical vectors	Embedding 将文本转换为数值向量
语义相似度通过向量距离衡量	Semantic similarity is measured by vector distance	语义相似度通过向量距离衡量
余弦相似度是最常用的度量	Cosine similarity is the most common measure	余弦相似度是最常用的度量
查询和文档必须用同一个 Embedding 模型	Query and documents must use the SAME embedding model	查询和文档必须用同一个 Embedding 模型
OpenAI 限制单请求总 token 300,000	OpenAI limits total tokens to 300,000 per request	OpenAI 限制单请求总 token 300,000
上下文 Embedding 比静态 Embedding 更精准	Contextual embeddings are more accurate than static	上下文 Embedding 比静态 Embedding 更精准
Embedding 是 RAG 检索质量的决定因素	Embedding quality determines RAG retrieval quality	Embedding 质量决定 RAG 检索质量