Build a BPE Tokenizer from Scratch in Python, Step-by-Step Guide
Build a working BPE tokenizer in Python step by step. Learn how LLMs split text into tokens, implement byte pair encoding, and count tokens with...
tiktoken tutorials
5 tutorials tagged tiktoken on Machine Learning Plus, newest first. Most of them belong to the Gen AI category.
tiktoken vs HuggingFace Tokenizers: Benchmark Guide
Benchmark tiktoken vs HuggingFace Tokenizers on speed, vocabulary, and encoding. Runnable Python code, migration guide, and decision framework for...
LLM API Pricing Guide: Compare & Optimize Costs
Learn to count LLM tokens with tiktoken, compare API pricing across OpenAI, Claude, and Gemini (March 2026), and cut costs by 90% with 10 proven...
LLM Context Windows Explained: Token Budget Guide
Learn how LLM context windows work, count tokens with tiktoken, estimate API costs, and build a Python token budget manager that allocates context...
How LLM Tokenization Works: Build a BPE Tokenizer
Build a working BPE tokenizer in Python, learn how text becomes tokens, and use tiktoken to count tokens and estimate LLM API costs across providers.