Tag: Temperature

Azure AI Foundry is a Microsoft’s unified Azure platform-as-a-service offering for enterprise AI operations, model builders, and application development. 
微软新的企业级 AI 平台，主要用于开发。

• AI apps / AI 应用
• Copilots / Copilot
• AI agents / AI Agent (智能系统)
• RAG systems / RAG 系统
• enterprise AI workflows / 企业智能工作流

It is becoming Microsoft’s main AI engineering platform. Think of it as
它正在变成微软主要的AI工程平台，本质上可以理解成

Azure AI Foundry = 
  Azure OpenAI
    + Prompt 管理
    + AI Orchestration
    + Agent Framework
    + RAG
    + Evaluation
    + Deployment
    + Monitoring

What does it do? It helps companies

• build GenAI apps / 构建 AI 系统
• connect enterprise data / 连接企业数据
• orchestrate AI workflows
• RAG / 做 RAG
• manage prompts / 管理 Prompt
• Mange Agent / 管理智能系统
• evaluate AI quality / 监控 AI 质量
• deploy AI safely / 部署 AI

Key Components / 核心组成

A. Model Access / 模型管理

via / 通过:

• Azure OpenAI
• model catalog

Use models like / 调用模型:

• GPT-4
• GPT-4o
• open-source models

B. Prompt Flow

Visual orchestration for:

• prompts / 链接Prompt
• workflows / 组织工作流
• chaining / 调试
• testing / 测试

C. RAG

Connect AI to:

• SharePoint
• PDFs / 文档
• databases / 企业数据库
• enterprise documents / 企业文档

D. AI Agents

Build agents that can /构建可自动执行任务的智能系统（Agent）:

• use tools / Tool calling
• call APIs / 调用API
• automate workflows / 自动工作流
• reason across tasks / 推理，自动分析

E. Evaluation & Monitoring
监控

Measure:

• hallucination
• safety
• quality
• groundedness

Enterprise companies care about this heavily / 企业极其重视这个.

---

An Agent = LLM + Tools + Memory + Planning

Agent can:

• decide steps / 自动拆解任务
• call tools (search, DB, API, code)
• store memory
• execute workflows

👉 Think:

“You give goal → agent figures out how to achieve it”

---

What is BPE?

BPE (Byte-Pair Encoding) is an algorithm that splits text into tokens by repeatedly merging the most frequent adjacent pairs of characters.
是一种将文本拆分成 token 的算法，它通过反复合并最常出现的相邻字符对来构建词汇表。

BPE starts with a base vocabulary of bytes/characters and iteratively merges the most frequent adjacent pairs across a large text corpus.

Three key points

CN	EN
1. 输入是文本	1. Input is text
2. 输出是一套合并规则 + token 序列	2. Output is a set of merge rules + a token sequence
3. 核心操作：找最频繁的相邻对，合并，重复	3. Core operation: find the most frequent adjacent pair, merge, repeat

BPE 每一步只看相邻的两个字符（或两个 token）。这就是为什么叫 Byte-Pair（字节对）—— 每次只合并一对。

See BPE clearly with an example

e.g. “a b c a b c c”

 Units: a, b, c, a, b, c, c

Step 1: Count all adjacent pairs

Adjacent Pair	EN：Frequency
(a, b)	2
(b, c)	2
(c, a)	1
(c, c)	1

Highest frequency is (a, b) and (b, c), both 2 times. Pick (a, b) to merge.

(a, b) → ab

Result：ab c ab c c

Step 2: Count adjacent pairs again：

Pair	Count
(ab, c)	2
(c, ab)	1
(c, c)	1

 Highest frequency is (ab, c) with 2 occurrences. Merge.

Result: abc abc c

Step 3 (optional)

Count adjacent pairs again: does (abc, abc) appear?

Check: abc abc c → adjacent pairs:

• (abc, abc): 1 occurrence
• (abc, c): 1 occurrence

(abc, abc) merge into abcabc

Final result comparison

CN：步骤	EN：Step	CN：结果	EN：Result
开始	Start	a b c a b c c (7 个单位)	a b c a b c c (7 units)
第 1 步后	After step 1	ab c ab c c (5 个单位)	ab c ab c c (5 units)
第 2 步后	After step 2	abc abc c (3 个单位)	abc abc c (3 units)

Core summary

CN	EN
每一步只合并相邻的两个单位	Each step merges only two adjacent units
合并后单位变少	Units decrease after each merge
新单位可以参与下一步的合并	New units can participate in next step’s merges
停止条件：达到目标词表大小	Stop condition: target vocabulary size reached

---

What is Completion in AI/LLM?

Completion is the fundamental, raw operation of an LLM where the model takes an input text prompt and generates the most likely continuation of that text, token by token, in an autoregressive manner. It has no concept of roles or conversation history — just text in, text out.

Completion 是 LLM 最基础、最原始的操作：模型接收一段输入文本提示，然后以自回归的方式逐 token 生成该文本最可能的延续内容。它没有角色或对话历史的概念 — 仅仅是文本输入、文本输出。

Key Characteristics (关键特征)

Aspect	English	Chinese
Input	Single string prompt	单个字符串提示词
Output	Raw text continuation	原始文本延续
Roles	None	无
History	Must be manually managed	必须手动管理
Underlying mechanism	Autoregressive token prediction	自回归 token 预测
Modern status	Legacy (GPT-3, Davinci era)	遗留模式（GPT-3、Davinci 时代）

Simple Example

Prompt (提示词):     "The capital of France is"
Completion (补全):   " Paris."

Prompt (提示词):     "def fibonacci(n):"
Completion (补全):   "\n    if n <= 1:\n        return n\n    else:\n        return fibonacci(n-1) + fibonacci(n-2)"

What is Chat in AI/LLM?

Chat is a structured, turn-based interaction paradigm built on top of completion. It adds role awareness (system, user, assistant) and automatic conversation history management. Each chat interaction is internally converted into a completion with special formatting tokens.

Chat是构建在Completion之上的结构化、基于轮次的交互范式。它增加了角色感知（系统、用户、助手）和自动对话历史管理。每次对话交互在内部都被转换为带有特殊格式标记的补全。

Key Characteristics

Aspect	English	Chinese
Input	Array of messages with roles	带角色的消息数组
Output	Role-labeled assistant response	带角色标签的助手回复
Roles	System, User, Assistant	系统、用户、助手
History	Automatically managed in message array	在消息数组中自动管理
Underlying mechanism	Still completion (with special tokens)	仍然是补全（带特殊标记）
Modern status	Standard (GPT-4, Claude, DeepSeek)	标准模式（GPT-4、Claude、DeepSeek）

e.g.
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
    {"role": "assistant", "content": "The capital of France is Paris."}
]

Completion vs. Chat

Context – understanding the word in Chinese

Context 的核心意思其实是：模型在生成回答时所依据的、对话或任务中已经存在的全部有效信息（包括历史对话、当前问题、隐含条件、用户偏好等）。这个词作为 “语境”（最推荐）,“背景信息” , “前文背景”， “关联信息” , “依托信息”, “对话记忆”（针对对话系统） 比较好对应中文。

我个人觉得“语境”比较好。

What is Context window？

A Context Window is the amount of information an LLM can “see” or “remember” during a conversation or request. Think of it as: The AI model’s working memory. Everything inside the context window can influence the AI’s response.
模型只能基于“Context Window 内的信息”来回答问题， LLM 一次能“看到/记住”的信息量就是Context windows。可以理解为AI的”内存容量“

Important Understanding

The context window includes BOTH:

Included in Context Window	Examples
Input tokens	prompts, chat history, RAG docs
Output tokens	model response / AI 输出，回答

Total tokens = Input + Output

Context Engineering

Meaning:

• deciding WHAT information goes into the context window
• optimizing token usage
• ranking retrieved documents
• summarizing history
• removing irrelevant content

Context Engineering 也就是：“决定什么信息进入 Context Window”。包括：

• 哪些文档最重要
• 如何节省 token
• 如何压缩历史
• 如何排序 RAG 结果
• 如何去掉无关信息

这是 Enterprise AI 非常核心的能力。

---

Deployment = making the model callable/useable.

Without deployment:

• model exists in catalog
• but your app cannot use it

After deployment, Azure gives endpoint + API access

VERY important

Deployment ≠ Agent

A deployment is: an exposed model service
可以简单理解为：将一个Model, 例如将 GPT 或 Deep Seek，调进我的系统，并激活它，让这个model 在我的系统里变为“可使用了”。

What are Embeddings in AI / LLM?

Embeddings in AI / LLM are numerical representations of text (or other data like images, audio) in a high‑dimensional vector space. Simply put, they turn words, sentences, or documents into lists of numbers so that computers can “understand” their meaning mathematically.

在 AI / 大语言模型中，Embedding 是把文本（或图像、音频等）转换成数字列表（向量） 的技术。简单说，就是让计算机通过一串数字来“理解”文字的含义。

Key points:

• What it looks like:
  A word like "king" might be represented as a vector:
  [0.25, -0.78, 0.43, …, 0.12] (e.g., 300–4096 dimensions).
• How it works:
  Words or phrases with similar meanings are placed close together in this vector space.
  • "king" and "queen" are close.
  • "apple" (fruit) and "apple" (company) have different vectors depending on context.

• Why embeddings matter:
  • They capture semantic meaning – relationships like king − man + woman ≈ queen.
  • They enable search (find similar texts), clustering (group topics), and recommendation.
  • LLMs use embeddings internally to process every token you feed into the model.

Vector Databases

A Vector Database is a database designed to store and search embeddings (vectors). Vector DB stores semantic meaning vectors

Common Vector Databases

• Pinecone / 全托管、无服务器、低延迟
• Weaviate / 内置混合搜索 + 模块化
• FAISS / 库（非数据库），高度优化的ANN
• Azure AI Search
• Databricks Vector Search
• Milvus / 云原生、GPU加速、十亿级规模
• Chroma / 轻量级、嵌入式、原生Python

These databases optimize:
nearest neighbor search
semantic retrieval
high-dimensional vector operations

Traditional Database vs Vector Database

Traditional Database	Vector Database
Stores rows/columns	Stores vectors
SQL queries	Similarity search
Exact matching	Semantic matching
Keyword search	Meaning search
Structured data	Embeddings

Example, Suppose company documents contain: “Employees may work remotely twice weekly.”

User asks: “What is the work from home policy?”. Traditional keyword search may fail because “remote” ≠ “work from home”. But embedding vectors capture semantic similarity.

Semantic Search

Similarity search is a technique that finds items in a dataset that are most similar to a given query vector, based on distance metrics in a high-dimensional embedding space — enabling semantic matching rather than exact keyword matching.
相似性搜索(Similarity search) 是一种技术，基于高维嵌入空间中的距离度量，在数据集中找到与给定查询向量最相似的项目 — 实现语义匹配而非精确关键词匹配。

---

Grounding = making the AI answer based on real external evidence, not memory.
让 AI 的回答“有依据”，不是靠记忆乱猜。或者说“给 AI 看资料”， 不是让它自己想答案

What is Hallucination in AI / LLM?

Hallucination in AI / LLM refers to the phenomenon where the model generates content that is factually incorrect, nonsensical, or completely unrelated to the real world or the provided source, while presenting it with high confidence as if it were true. This is one of the BIGGEST concerns in enterprise AI systems.

Common examples include:

• Inventing non‑existent references, laws, or historical events.
• Incorrectly calculating simple arithmetic.
• Misinterpreting the user’s input and fabricating plausible‑sounding but false information.

虚假生成, 模型编造 AI 生成了错误的，编造的，不真实的，没依据的的信息。但AI却“很自信”地说出来。即： 模型自信地输出错误或凭空捏造的信息

Why Hallucinations Happen？

LLMs Predict Language, Not Truth；

2) Missing Context. If the model lacks:

• sufficient information
• enterprise data
• current data

it may “fill in the gaps.”

3) Ambiguous Prompts. Poor prompts can cause:

• assumptions
• invented details
• unstable outputs

4) Outdated Training Data. Models have training cutoffs. They may:

• not know recent events
• generate outdated answers
• guess newer information

5) Weak RAG / Retrieval. In enterprise AI:

• bad retrieval
• irrelevant documents
• incomplete grounding

can produce hallucinated answers.

Types of Hallucinations

• A. Factual Hallucination: Wrong facts. /事实幻觉, 事实错误, 编造公司政策
• B. Citation Hallucination: Fake sources or references. / 引用幻觉, 假论文、假来源。
• C. Logical Hallucination: Reasoning errors / 推理幻觉, 逻辑推理错误
• D. Tool/API Hallucination: Inventing APIs, functions, parameters, libraries / 编造API等

How Enterprises Reduce Hallucinations

1) RAG (Retrieval-Augmented Generation)

• RAG: Most important technique. Instead of relying only on model memory / 最核心
• Better Prompt Engineering, Clear prompts reduce ambiguity.
• Context Engineering Control: what information enters context, retrieval quality, ranking, chunking, summarization.
• Evaluation Systems: AI outputs are tested for: factual accuracy, roundedness, consistency, safety.
• Human-in-the-Loop, Humans validate :sensitive outputs, approvals, critical decisions.

---

What is LLM?
Large language models, also known as LLMs, are very large deep learning models that are pre-trained on vast amounts of data. The underlying transformer is a set of neural networks that consist of an encoder and a decoder with self-attention capabilities. The encoder and decoder extract meanings from a sequence of text and understand the relationships between words and phrases in it.

大型语言模型（英语：large language model，LLM），也称大语言模型，简称大模型，是一种基于人工神经网络的已经训练过的语言模型。大语言模型专为自然语言处理任务而设计，尤其适用于语言生成。

他们关系基本如这个层级结构/包含关系：
人工智能 (AI) > 模型 (Model) > 生成式 AI (Generative AI) > 大语言模型 (LLM)

---

 LCEL & Runnable

Memory = system that stores user/context over time

Types:

🔹 Short-term memory

• current conversation context

🔹 Long-term memory

• user preferences
• past interactions
• profile data

---

Why important?

• every chat is “reset” / 没有memory,每次都是新用户
• personalized AI experience / 有memory AI 变成“个人助理”

---

What is Prompt? A Prompt is the instruction, question, context, or input you give to an AI model (LLM) to tell it what you want it to do.
就是你给 AI 的“指令/输入”， 告诉AI： 要做什么，用什么方法做，输出什么。

e.g.

Summarize this document in 5 bullet points.

That sentence is a prompt.

Another example:

You are a senior Azure architect.
Explain Medallion Architecture for a banking platform.

The prompt tells the AI:

• its role
• the task
• the expected output
• sometimes the tone/style

Basic Prompt Structure

A prompt often contains:

Part	Purpose
Instruction	What to do / 做什么
Context	Background information / 背景信息
Constraints	Rules/limits/ 限制条件
Examples	Demonstrations / 示例
Output format	Expected response structure / 要求的输出格式

Example

You are a data architect.

Context:
The company uses Azure Databricks and Synapse.

Task:
Design a metadata-driven ingestion framework.

Output:
Provide architecture, components, and best practices.

This is a more structured prompt.

Core components of Good prompt

A good prompt usually includs:

EN	CN
Goal	目标
Context	背景
Constraints	限制条件
Input	输入数据
Output Format	输出格式
Examples	示例

What is Prompt Engineering?

Prompt Engineering = the practice of designing prompts to get better AI outputs.
提示词工程就是：设计 Prompt 来获得更好 AI 输出”的技术。
其实简单说就是 “会问 AI 问题”

It is:

• writing prompts strategically / 更聪明地写 Prompt
• structuring context correctly / 更合理地组织 Context
• controlling AI behavior / 更稳定地控制 AI 行为
• improving reliability and quality / 提高可靠性和质量

Think of it as:

Programming with language instead of code.
可理解成：用自然语言“编程”，而不是用代码编程

Why Prompt Engineering Matters / 为什么重要

LLMs are highly sensitive to / LLM对这些高度敏感:

• wording / 措辞
• context / 背景信息
• instructions / 要求
• examples / 示例
• formatting / 输出要求

Small prompt changes can dramatically affect / 对 prompt上述这些哪怕是小的改动都会影响到结果:

• accuracy / 准确率
• reasoning / 推理能力
• hallucination / 幻觉, 无根据的结论
• consistency / 稳定性
• output quality / 输出质量

Common Prompt Engineering Techniques

common prompt technical include:

Tech	CN
Role Prompting	指定 AI 身份
Few-shot	给多个例子
Chain of Thought	引导 AI 一步一步思考
Output Control	控制输出格式
Constraints	加限制条件
Context Injection	注入业务背景

1) Role Prompting

Tell the AI who it is.

Example:

You are a senior enterprise architect.

This changes response style and depth.

---

2) Context Injection

Provide necessary information / 提供/注入必要的背景信息，以提高结果的准确性

Example:

The environment uses:
- Azure Databricks
- Delta Lake
- Unity Catalog

Without context, AI guesses / 不提供这些背景资料，AI会去乱猜。影响结果的准确性.

---

3) Output Formatting

Specify desired structure / 输出格式控制， 给AI提出输出的格式要求， 可以帮助提高结果的准确性

Example:

Return the answer as:
- architecture diagram
- bullet points
- implementation steps

---

4) Few-Shot Prompting

Give examples of desired behavior.
Few-Shot Learning / (少样本学习) 是一种人工智能技术。它指的是在给模型的提示词（Prompt）中提供少量（通常 2 到 5 个）示例，帮助模型理解任务要求，从而生成更准确的回复。

Example:

I want to classify sentiment.
Example 1: "I love this food!" -> Positive
Example 2: "This is the worst day ever." -> Negative
Example 3: "The movie was okay." -> Neutral

Now classify this: "The weather is quite nice today." ->

output:  Positive

AI learns pattern/style from examples.

---

5) Chain-of-Thought Prompting

Chain-of-Thought (CoT) is a prompting technique that forces the LLM to show its reasoning steps before giving a final answer — like asking a SQL analyst to explain their logic before writing the query.

It transforms input → answer into input → step1 → step2 → … → answer.

CoT 是一种提示技术，强迫 LLM 在给出最终答案之前展示推理步骤 —— 就像让数据分析师在写 SQL 之前先解释逻辑一样。
它将 输入 → 答案  转变为 输入 → 步骤1 → 步骤2 → … → 答案

Example:

# Question： 
"Roger has 5 marbles. He buys 2 bags with 4 marbles each. Then he loses 3 marbles. Think step by step. How many does he have left?"



# the answer with all steps: 

Roger starts with 5 marbles.
He buys 2 bags with 4 marbles each:

2 × 4 = 8 marbles

Now he has:

5 + 8 = 13 marbles

Then he loses 3 marbles:

13 − 3 = 10 marbles

Answer: 10 marbles. 🟢

e.g.

# Question
一个农夫有15只鸡。一只狐狸每晚吃掉3只鸡，连续2晚。然后农夫又买了5只鸡。第二天晚上，狐狸吃掉2只鸡。还剩多少只鸡？


# result 
我们一步一步算：

初始数量
农夫有 15 只鸡

第1晚狐狸吃掉 3 只
15 − 3 = 12

第2晚狐狸再吃掉 3 只
12 − 3 = 9

农夫又买了 5 只鸡
9 + 5 = 14

第二天晚上狐狸又吃掉 2 只
14 − 2 = 12



Final Answer:
12 chickens remain

Useful for:

• logic
• architecture
• math
• troubleshooting

Prompt Injection

Prompt Injection is an attack where malicious user input tries to override or hijack the system prompt, making the AI behave in unintended ways. In production, prompt injection is one of the Top 5 LLM security risks (OWASP LLM Top 10). Any customer-facing AI must implement these defenses.

提示词注入是一种攻击方式，恶意用户输入试图覆盖或劫持 system prompt，让AI做出非预期的行为。

---

Schema enforcement = forcing data to follow a fixed structure (schema), not free-form text.

强制 AI 或 API 输出“符合格式的数据”，不能乱写。例如
John is 30 years old and lives in Toronto

AI 可能输出：John is 30 years old and lives in Toronto
也可能是：
name: John
age: thirty
location: Toronto Canada maybe

不稳定、不可机器处理。 用Schema enforcement 强制它输出这样
{
“name”: “John”,
“age”: 30,
“location”: “Toronto”
}

StreamingIn the context of Large Language Models (LLMs), streaming refers to the technique of returning generated tokens one by one (or in small chunks) as soon as they are produced by the model, rather than waiting for the entire response to be completed. The underlying transport is typically Server-Sent Events (SSE) or chunked HTTP responses, where the server pushes incremental updates to the client.

Streaming

async/await LLM Call

What is Temperature?

Temperature is a hyperparameter that controls the randomness or creativity of an LLM’s output. It scales the logits (raw prediction scores) before the softmax function that converts them into probabilities — lower temperatures make the model more deterministic and focused, while higher temperatures make it more diverse and exploratory.
Temperature 是一个超参数，用于控制大语言模型输出的随机性或创造性。它在 softmax 函数（将原始预测分数转换为概率）之前对这些 logits 进行缩放 — 较低的温度使模型更确定、更专注，而较高的温度使其更多样化、更具探索性。

 Imagine you’re at a restaurant with a menu of 10 dishes. Temperature controls how likely you are to pick your absolute favorite vs. trying something new.
 想象你在一个有 10 道菜的餐厅里。温度控制着你选择最爱的菜 vs. 尝试新菜的可能性。

Temperature	Analogy (English)	Analogy (中文)
Low (0.1 ~ 0.3)	You always order your #1 favorite dish. Very predictable.	你总是点你最爱的第一道菜。非常可预测。
Medium (0.7 ~ 1.0)	You usually pick your top dish, but sometimes try #2 or #3. Balanced.	你通常选最爱的菜，但有时尝试第二或第三喜欢的。平衡。
High (1.5+)	You randomly pick any dish, even ones you don’t know. Very unpredictable.	你随机选任何菜，甚至你不认识的菜。非常不可预测。

Top-K – Sample only from the K most probable tokens

• The model looks at all possible next tokens and their probabilities.
• It keeps only the K tokens with the highest probabilities and discards the rest.
• Then it randomly selects one token from these K tokens (using their relative probabilities).

Effect:

• Smaller K (e.g., 10) → Fewer choices → More deterministic, predictable, and safe outputs.
• Larger K (e.g., 100) → More choices → More random and diverse outputs.

Example (K=3):
Probabilities: “cat” (50%), “dog” (30%), “bird” (12%), “car” (5%), “tree” (3%)
→ Keep only {cat, dog, bird} → “car” and “tree” can never be chosen.

只从概率最高的 K 个 token 中采样

• 模型先算出所有可能的下一个 token 及其概率。
• 只保留 概率最高的前 K 个 token，扔掉其余 token。
• 然后在这 K 个 token 中按概率随机选一个。

效果：

• K 越小（如 10） → 可选词越少 → 输出越 确定、安全、可预测。
• K 越大（如 100） → 可选词越多 → 输出越 随机、多样化。

例子（K=3）：
概率：猫(50%)、狗(30%)、鸟(12%)、车(5%)、树(3%)
→ 只保留 {猫, 狗, 鸟} → “车”“树” 永远不可能被选中。

Top-P (Nucleus Sampling) – Choose the smallest set of tokens whose cumulative probability ≥ P

• Instead of a fixed number of tokens (K), Top-P dynamically selects tokens from the most probable downward until the sum of their probabilities reaches or exceeds P.
• This selected set is called the nucleus.

Effect:

• Smaller P (e.g., 0.9) → Keeps only the top few high-probability tokens → More stable.
• Larger P (e.g., 0.95–1.0) → Keeps more tokens (sometimes all) → More random.

Example (P=0.9):
Probabilities: “cat” (50%, cumulative 50%), “dog” (30%, cumulative 80%), “bird” (12%, cumulative 92% ≥ 90%)
→ Nucleus = {cat, dog, bird} → “car” and “tree” are excluded.

Key difference from Top-K:
If the probability distribution is very flat, P=0.9 might keep 20+ tokens. If very sharp, it might keep only 1 token. Top-K always keeps exactly K tokens.

Top‑P（核采样） – 选择累计概率 ≥ P 的最小 token 集合

• 不固定 token 个数，而是 从概率最高的 token 开始往下加，直到 累计概率 ≥ P。
• 这个动态选出来的 token 集合叫做 “核（nucleus）”。

效果：

• P 越小（如 0.9） → 只保留少数几个高概率 token → 输出越 稳定。
• P 越大（如 0.95～1.0） → 保留更多 token（甚至全部） → 输出越 随机。

例子（P=0.9）：
概率：猫(50%、累计50%)、狗(30%、累计80%)、鸟(12%、累计 92% ≥ 90%)
→ 候选集 = {猫, 狗, 鸟} → “车”“树” 被排除。

与 Top-K 的关键区别：
如果概率分布很平坦，P=0.9 可能保留 20+ 个 token；如果很尖锐，可能只保留 1 个 token。
而 Top-K 永远固定保留 K 个 token。

Why use them together?

• Top-K alone: Can still include unlikely tokens if K is large.
• Top-P alone: Works well alone, but combined with Top-K (e.g., top_k=50, top_p=0.95) → First limit to K tokens, then apply nucleus → Best balance.

为什么要组合使用？

• 只用 Top-K：K 较大时仍可能保留不合理的 token。
• 只用 Top-P：已经很不错，但和 Top-K 组合（如 top_k=50, top_p=0.95）→ 先限制最多 50 个 token，再从中挑核 → 既排除尾部垃圾词，又保持灵活性。

---

What are Tokens in AI / LLM?

Tokens in AI / LLM are the basic units of text that the model reads and generates. Instead of processing raw text character‑by‑character or word‑by‑word, the model breaks text into smaller, meaningful pieces called tokens.

在 AI / 大语言模型中，Token 是模型处理文本时的最基本单元。模型不会一个字符一个字符地读，也不会按完整单词读，而是把文本切分成有意义的片段，每个片段就是一个 Token。

Token 就是把一句话切成模型能“消化”的最小碎片，每个碎片有相对独立的意义。切的方式取决于分词器，不同模型切法可能不一样。

Key points:

• A token is not always a whole word, nor a single character. It can be:
  • A short common word: "cat" → 1 token
  • Part of a longer word: "unhappiness" → "un" + "happiness" (2 tokens)
  • A single character: "a" → 1 token
  • A punctuation mark: "." → 1 token
  • A space or part of a space (depending on the tokenizer)
• Examples (using OpenAI’s tokenizer):
  • "Hello, world!" → ["Hello", ",", " world", "!"] (4 tokens)
  • "I love you" → ["I", " love", " you"] (3 tokens)
  • A long Chinese sentence → often 1 Chinese character = 1–2 tokens (less efficient than English)

Why Tokens Matter

• Context length is measured in tokens (e.g., “this model has an 8K token context”).
• Cost is usually based on tokens (input tokens + output tokens).
• Speed depends on how many tokens the model processes.

---

A Tool is a function you give to an LLM so it can take actions beyond just generating text — it lets the model interact with the real world.

工具 (Tool) 是你给 LLM 的一个函数，让它不只是生成文字，而是能真正与外界交互、执行操作。

Examples:

• 🔍 Search (Bing / web) / 搜索引擎
• 🗄️ Database query (SQL)
• 📊 Data processing (Python)
• 🔗 APIs (CRM, ERP)
• 📁 File reading

Why tools matter?

Because LLM alone:

• cannot access real-time data
• cannot query enterprise systems
• cannot execute actions

👉 Tools = “hands of the model” / model 的“手”

Tool Schema

Tool Calling — LLM Decision & Tool Selection

Tool Execution And Result Return

Multi-turn Tool Loop

ReAct Mode Implementation

---

Workflow Rules = logic that controls how an agent behaves
控制 Agent 行为的“流程规则”

Examples:

• Step ordering
• Tool selection rules
• Approval conditions
• Safety constraints

Example

1. Understand intent / 理解问题
2. Check memory / 查看记忆
3. Decide if tool is needed / 判断是否要工具
4. Call tool (if needed) / 调用工具
5. Combine results / 汇总结果
6. Generate final answer / 输出答案

---