Index · tokenizer tax
The tokenizer tax
Two models at the same $/Mtok do not cost the same: each tokenizer turns the same text into a different number of tokens. We encode a fixed corpus with every public tokenizer in the catalogue and rank endpoints by$ per million input characters — the number the sticker price hides.
Endpoints by effective input cost
| # | Model | Provider | $ / Mchar input | tok/char | vs cheapest |
|---|
† tokenizer inherited from the model family (no model-specific tokenizer published).
Tokens per character, by tokenizer family
Lower is denser (fewer tokens for the same text). The spread is real money: on French prose the widest gap below means the same prompt costs ~40% more in tokens before prices even enter the picture.
| Tokenizer | English prose | French prose | Python code | JSON data | Source |
|---|---|---|---|---|---|
| minimax-m1 | 0.1903 | 0.2182 | 0.2738 | 0.3634 | MiniMaxAI/MiniMax-M1-80k |
| minimax-m2 | 0.1903 | 0.2182 | 0.2738 | 0.3634 | MiniMaxAI/MiniMax-M2 |
| minimax-m3 | 0.1903 | 0.2182 | 0.2738 | 0.3634 | MiniMaxAI/MiniMax-M3 |
| o200k_base | 0.1914 | 0.2328 | 0.2634 | 0.3621 | tiktoken o200k_base |
| o200k_harmony | 0.1914 | 0.2328 | 0.2634 | 0.3621 | tiktoken o200k_harmony |
| kimi-k2 | 0.1914 | 0.2909 | 0.2622 | 0.3621 | hf-tiktoken moonshotai/Kimi-K2-Instruct/tiktoken.model |
| kimi-k3 | 0.1914 | 0.2909 | 0.2622 | 0.3621 | hf-tiktoken moonshotai/Kimi-K3/tiktoken.model |
| llama-4 | 0.1920 | 0.2318 | 0.2640 | 0.3621 | unsloth/Llama-4-Scout-17B-16E-Instruct |
| glm-4 | 0.1936 | 0.2534 | 0.2597 | 0.3621 | zai-org/GLM-4.5 |
| glm-5 | 0.1936 | 0.2534 | 0.2597 | 0.3621 | zai-org/GLM-5.3 |
| llama-3 | 0.1936 | 0.2633 | 0.2597 | 0.3602 | NousResearch/Meta-Llama-3.1-8B |
| granite | 0.1936 | 0.2647 | 0.2597 | 0.3608 | ibm-granite/granite-4.0-h-small |
| qwen3 | 0.1936 | 0.2618 | 0.2628 | 0.4005 | Qwen/Qwen3-8B |
| gemma-3 | 0.1959 | 0.2403 | 0.3095 | 0.4511 | unsloth/gemma-3-27b-it |
| qwen3.5 | 0.1959 | 0.2379 | 0.2837 | 0.4255 | Qwen/Qwen3.5-9B |
| qwen3.8 | 0.1959 | 0.2379 | 0.2837 | 0.4255 | Qwen/Qwen3.8-27B |
| deepseek-v3 | 0.1970 | 0.2600 | 0.2868 | 0.3794 | deepseek-ai/DeepSeek-V3.2-Exp |
| mistral-tekken | 0.1970 | 0.2309 | 0.2905 | 0.4127 | mistralai/Mistral-Nemo-Instruct-2407 |
| nemotron | 0.1970 | 0.2309 | 0.2905 | 0.4127 | nvidia/NVIDIA-Nemotron-Nano-9B-v2 |
| mistral-v3 | 0.2182 | 0.3055 | 0.3471 | 0.4709 | mistralai/Mixtral-8x22B-Instruct-v0.1 |
| mistral-large | 0.2182 | 0.3050 | 0.3471 | 0.4709 | mistralai/Mistral-Large-Instruct-2411 |
What's not here — and why
96% of priced endpoints are covered. The rest publish no tokenizer, so no number is shown — an estimate dressed up as a measurement would be worse than a gap: claude-fable-5 (closed — no public tokenizer (Anthropic)); claude-fable-5-1 (closed — no public tokenizer (Anthropic)); claude-opus-5 (closed — no public tokenizer (Anthropic)); claude-sonnet-5 (closed — no public tokenizer (Anthropic)); command-a (gated on Hugging Face (Cohere)); command-r (gated on Hugging Face (Cohere)); gemini-3.5-flash-lite (closed — no public tokenizer (Google)); gemini-3.6-flash (closed — no public tokenizer (Google)); gemini-3.7-flash (closed — no public tokenizer (Google)); gemini-3.8-flash (closed — no public tokenizer (Google)); grok-4.5 (closed — no public tokenizer (xAI)); grok-4.6 (closed — no public tokenizer (xAI)). Closed-model counts will come from real API usage fields when we start making metered calls, not from guesses.
Method
Fixed corpus v1 (four segments, self-authored, immutable — ratios are only comparable at constant corpus; sha256 fingerprints recorded at each run): english prose (1792 chars), french prose (2131 chars), python code (1625 chars), json data (1563 chars). Each family's public tokenizer encodes each segment; tokens ÷ characters gives the ratio; ratio × the endpoint's input $/Mtok gives $ per million input characters. Input side only: output token counts depend on what the model generates, and we don't invent data. Cached-input pricing is a separate lever — see the caching guideand the cost calculator for full bills in $/Mtok. Measured 2026-09-05.