How Tokenization Works: BPE and the Algorithm Behind Your LLM
Learn how Byte Pair Encoding (BPE) actually works — the algorithm that powers GPT, Claude, and LLaMA tokenizers. Step-by-step with examples.
Feb 3, 20269 min read5
Command Palette
Search for a command to run...
#tokenization
Articles tagged with #tokenization
What Are Tokens and Why Your LLM Bill Depends on Them
Learn what tokens really are, why they're not words, and how understanding tokenization saves you money on LLM API costs.
Feb 1, 20269 min read28
OpenAI Prompt Caching: Undocumented Cross-Model Behavior and Production Cost Implications
I'm building an AI agent from scratch—no frameworks, no abstractions—specifically to understand where every token goes and how much it costs. This is Phase 3 of my token economics research. Phase 1 covered basic tool calling mechanics. Phase 2 reveal...
Dec 19, 202512 min read4
Model Selection for AI Agents: Measuring Token Costs Across OpenAI's Model Family
I've been building an AI agent from scratch. No frameworks, no abstractions, to understand where every token goes and what drives cost at scale. In previous post/phase, I measured how tool definitions and conversation depth impact token usage. The fi...
Dec 19, 202521 min read8
Token Explosion in AI Agents: Why Your Costs Scale Exponentially
I built an AI agent from scratch. Not because frameworks aren't good. They are(and I suggest you use them). But because I needed to see where every token goes. When you're building production systems that could cost $150K+/year in LLM tokens alone, y...
Dec 10, 202515 min read8