tiktoken: Count LLM Tokens in Python (2026)

Llm And Generative Ai Oct 05, 2026

Watch on YouTube

tiktoken: Count LLM Tokens in Python (2026) shows how to measure prompts the way OpenAI models actually see them: pick the right encoding, count tokens, inspect how text is split, truncate to a budget, chunk documents with overlap for RAG, and stop special-token surprises before they reach the API. Every number below comes from a real run.

Chunked text feeds straight into a vector store like ChromaDB or LanceDB hybrid search, and token budgets matter even more when you stream LLM tokens with FastAPI SSE.

TL;DR

  • tiktoken.encoding_for_model("gpt-4o") returns the o200k_base encoding used by current OpenAI models.
  • len(enc.encode(text)) is your token count; enc.decode(ids) round-trips back to the exact text.
  • Truncate and chunk on token boundaries, not characters, so you never overflow a context window.
  • Strings like <|endoftext|> raise ValueError by default; pass disallowed_special=() to treat user text as plain text.
  • enc.encode_batch() counts many prompts in one call. We tested tiktoken 0.14.0 on Python 3.13.5.

Why count tokens in 2026?

LLM APIs bill per token, cap requests by a context window measured in tokens, and rate-limit by tokens per minute. Character counts are a poor guess: English averages roughly four characters per token, but code, numbers, and non-Latin scripts behave very differently. Counting locally with tiktoken is fast, free, and works offline after the first download of the encoding file.

APIWhat it doesTypical use
encoding_for_model()Model name to encodingMatch the tokenizer to your model
get_encoding("o200k_base")Load an encoding by namePin the tokenizer explicitly
encode() / decode()Text to token ids and backCount, truncate, chunk
encode_batch()Encode many strings at oncePre-flight budget checks

Versions tested (2026-10-05)

  • Python 3.13.5
  • tiktoken 0.14.0
  • Encodings: o200k_base (gpt-4o and newer) and cl100k_base (older gpt-4 / gpt-3.5 era)
python -m venv .venv && source .venv/bin/activate
pip install tiktoken==0.14.0
python token_demo.py

The first call downloads the encoding file and caches it. Set TIKTOKEN_CACHE_DIR to a folder you ship with your app if production servers have no internet access.

1. Count tokens for a model

import sys
import tiktoken

print("python", sys.version.split()[0], "| tiktoken", tiktoken.__version__)

# 1) Pick the encoding for a model, then count tokens
enc = tiktoken.encoding_for_model("gpt-4o")
text = "Python makes LLM apps easy. Count tokens before you call the API!"
ids = enc.encode(text)
print("encoding:", enc.name)
print("chars:", len(text), "| tokens:", len(ids))
python 3.13.5 | tiktoken 0.14.0
encoding: o200k_base
chars: 65 | tokens: 15

65 characters became 15 tokens. Always choose the encoding from the model name so your counts match what the API bills.

2. See how text is split

# 2) See how the text is split
pieces = [enc.decode([t]) for t in ids[:8]]
print("first pieces:", pieces)
print("round trip ok:", enc.decode(ids) == text)
first pieces: ['Python', ' makes', ' L', 'LM', ' apps', ' easy', '.', ' Count']
round trip ok: True

Notice that LLM splits into ' L' and 'LM', and leading spaces belong to the next token. That is why slicing text by characters can cut a word in half while slicing token ids never breaks the round trip.

3. Compare encodings (English vs Urdu)

# 3) Same text, two encodings
old = tiktoken.get_encoding("cl100k_base")
urdu = "پائتھون سیکھنا آسان ہے"
for label, s in [("english", text), ("urdu", urdu)]:
    print(f"{label}: cl100k={len(old.encode(s))} o200k={len(enc.encode(s))}")
english: cl100k=15 o200k=15
urdu: cl100k=24 o200k=10

English costs the same in both encodings, but the Urdu sentence drops from 24 tokens to 10 with o200k_base. If your users write in Urdu, Hindi, Arabic, or other non-Latin scripts, the newer encoding means cheaper calls and more room in the context window.

4. Truncate a prompt to a token budget

# 4) Truncate a long prompt to a token budget
def truncate(s: str, max_tokens: int) -> str:
    toks = enc.encode(s)
    return s if len(toks) <= max_tokens else enc.decode(toks[:max_tokens])

doc = " ".join(f"Line {i}: tiktoken counts tokens fast." for i in range(1, 201))
short = truncate(doc, 50)
print("doc tokens:", len(enc.encode(doc)), "| truncated:", len(enc.encode(short)))
doc tokens: 2200 | truncated: 50

5. Chunk documents with overlap for RAG

# 5) Chunk by tokens with overlap (for RAG / embeddings)
def chunk(s: str, size: int = 200, overlap: int = 20) -> list[str]:
    toks = enc.encode(s)
    step = size - overlap
    return [enc.decode(toks[i:i + size]) for i in range(0, len(toks), step)]

chunks = chunk(doc)
print("chunks:", len(chunks), "| sizes:", [len(enc.encode(c)) for c in chunks])
chunks: 13 | sizes: [200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 40]

Each chunk is exactly 200 tokens with a 20-token overlap so a sentence cut at a boundary still appears whole in the next chunk. Feed these chunks to your embedding model and vector store.

6. Handle special tokens safely

# 6) Special tokens are blocked by default
try:
    enc.encode("user text <|endoftext|>")
except ValueError:
    print("special token blocked: ValueError")
print("allowed as text:", len(enc.encode("user text <|endoftext|>", disallowed_special=())))
special token blocked: ValueError
allowed as text: 9

By default tiktoken refuses to encode control strings like <|endoftext|>. That protects you from prompt-injection tricks, but it also crashes when a user pastes that text. Passing disallowed_special=() encodes it as ordinary characters.

7. Budget check a batch before sending

# 7) Budget check before you send a batch
budget = 1000
prompts = [text, doc, short]
counts = enc.encode_batch(prompts)
for p, c in zip(["question", "doc", "short"], counts):
    print(f"{p:9} {len(c):5} tokens  fits={len(c) <= budget}")
question     15 tokens  fits=True
doc        2200 tokens  fits=False
short        50 tokens  fits=True

Full real output

python 3.13.5 | tiktoken 0.14.0
encoding: o200k_base
chars: 65 | tokens: 15
first pieces: ['Python', ' makes', ' L', 'LM', ' apps', ' easy', '.', ' Count']
round trip ok: True
english: cl100k=15 o200k=15
urdu: cl100k=24 o200k=10
doc tokens: 2200 | truncated: 50
chunks: 13 | sizes: [200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 40]
special token blocked: ValueError
allowed as text: 9
question     15 tokens  fits=True
doc        2200 tokens  fits=False
short        50 tokens  fits=True

Common mistakes

  • Using the wrong encoding. cl100k_base counts differ from o200k_base, especially for non-English text.
  • Counting characters. Use len(enc.encode(text)), never len(text) / 4, when the limit matters.
  • Forgetting the reply. The context window covers input and output, so leave room for the model's answer.
  • Assuming one tokenizer fits all. Claude, Gemini, and Llama use their own tokenizers; treat tiktoken counts as an estimate for them.

FAQ

Does tiktoken call the OpenAI API? No. It runs locally; it only downloads the encoding file once and caches it.

Which encoding should I use for GPT-4o and newer models? o200k_base, which encoding_for_model("gpt-4o") returns in our test.

How do I estimate cost? Multiply the token count by your provider's current per-million-token price from its pricing page, and add the expected output tokens.

Next steps

Wrap truncate() and chunk() in a small utility module, call it before every LLM request, and log the token count next to each response so you can track cost and latency over time.

Share:
Last updated: October 2026
Built-in Functions in Python 2026 str() in Python 2026: String Conversion + Modern Formatting & Best Practices memoryview with JAX in Python 2026: Zero-Copy NumPy → JAX Array Interop + Efficient ML Examples memoryview with TensorFlow in Python 2026: Zero-Copy NumPy → Tensor Interop + GPU Pinning & ML Examples memoryview with TensorFlow in Python 2026: Zero-Copy NumPy → Tensor Interop + ML Examples memoryview with NumPy & PyTorch in Python 2026: Zero-Copy Views, Efficient Slicing & ML Interop Examples memoryview with NumPy in Python 2026: Zero-Copy Views, Efficient Slicing & Real ML Examples memoryview() in Python 2026: Zero-Copy Magic for Large Binary Data + Real Examples import() in Python 2026: How It Works, Security Risks & Modern Alternatives (importlib) zip() in Python 2026: How to Use It, strict=True Behavior & Real-World Examples vars() in Python 2026: Accessing Object Namespace + Modern Introspection Patterns type() in Python 2026: Dynamic Type Inspection & Object Creation + Modern Patterns tuple() in Python 2026: Immutable Sequences + Modern Patterns & Best Practices super() in Python 2026: Method Resolution & Modern Inheritance Patterns sum() in Python 2026: Summing Iterables + Modern Numeric Patterns & Best Practices staticmethod() in Python 2026: Static Methods, Utility Functions & Modern Best Practices sorted() in Python 2026: Sorting Iterables + Modern Patterns & Best Practices slice() in Python 2026: Creating Slice Objects + Modern Patterns & Best Practices setattr() in Python 2026: Dynamic Attribute Setting + Modern Patterns & Safety set() in Python 2026: Mutable Sets Creation + Modern Patterns & Best Practices