Token Fundamental

What is a Token?

A token is the smallest unit that the LLM reads and generates.

It is NOT exactly:

a word ❌
a character ❌
a sentence ❌

It is something in between, decided by the model tokenizer.

e.g.
“hello”: usually, [“hello”] → 1 token (or sometimes 1)

“unbelievable”
→ [“un”, “believ”, “able”]
→ 3 tokens

Why Tokens matter

(1) Cost = Tokens

You are charged based on: input tokens + output tokens

(2) Context Window

Context Window = the maximum amount of text the model can “see and remember” in one request.

It includes:

System prompt
User messages
Assistant replies
Your input text
Retrieved documents (RAG)

👉 Every model has a limit. If it exceeds the limit → older content gets cut off.

(3) RAG Chunking

RAG systems split documents into chunks, like 500 tokens per chunk, before storing in a vector database.

e.g. Original text: Azure Synapse is a cloud analytics service. It integrates big data and data warehousing. It supports Spark and SQL engines.

After chunking
# Chunk 1
Azure Synapse is a cloud analytics service.
# Chunk 2
It integrates big data and data warehousing.
# Chunk 3
It supports Spark and SQL engines.

Chunk Size

Type	Size
Small chunk	200–300 tokens
Standard	300–800 tokens
Large chunk	1000+ tokens

Overlap Strategy (Professional RAG Design)

We often use overlapping chunks: Example:

Chunk 1: A → B → C
Chunk 2: C → D → E
Chunk 3: E → F → G

Chunking Strategies

Fixed-size chunking
e.g. every 500 tokens, simple, but may break sentences
Sentence-based chunking
e.g. Split by: paragraphs, sentences. better meaning, but uneven size
Semantic chunking (advanced)
e.g. Use AI to detect meaning boundaries. best quality, but more expensive

Tag: Token Fundamental