Tokens explained
What you'll learn
Estimated time: 9 minutes
A model does not read words. It reads tokens: chunks of text, roughly four characters or three-quarters of a word in English, that a tokeniser splits the input into before anything else happens. Pricing, context limits and even a model’s odd blind spots all trace back to this one detail.
- Explain what a token is and how a tokeniser splits text.
- Estimate a token count from characters or words without a tool.
- Recognise that code, non-English text and images tokenise differently from plain English prose.
Video lesson
The animation splits a sentence, a code snippet and an image into tokens side by side, showing why each counts differently.
Text becomes pieces, not words
A tokeniser breaks input into a fixed vocabulary of chunks before a model sees any of it. Common words often become a single token; longer or rarer words split into two or three pieces, and punctuation usually gets a token of its own. The model then predicts one token at a time, always working from that broken-up version of your text, never from the words you actually typed.
As a working estimate for English, one token is about four characters or three-quarters of a word: a 750-word report runs to roughly 1,000 tokens. That is only a starting point. The real count depends on the tokeniser, the language and the content, which is why the next section counts a real sentence properly.
A worked count
Prompt: "Roughly how many tokens is this sentence: ‘Our Q3 renewal rate improved to 94%, driven mainly by the onboarding changes we shipped in April.’?"
Response (excerpt): "About 24 to 26 tokens. Common words like ‘our’, ‘rate’ and ‘mainly’ are single tokens. ‘94%’ typically splits into a few pieces because of the digits and the percent sign, and ‘onboarding’ often splits into two pieces. This is an estimate; the exact count depends on the model’s tokeniser." The sentence itself has seventeen words; at roughly 1.3 tokens per word that is about 22, a touch under the model’s estimate. That gap is the point: a rule of thumb gets you close, not exact, and landing in the right neighbourhood is usually all a quick estimate needs to do.
Why code and other languages cost more tokens
Programming languages tokenise less efficiently than English prose. Indentation, brackets, camelCase and repeated symbols each consume tokens that carry little of the meaning a person reads from them, so a 200-line JSON file can use more tokens than a 200-line email a person would read far faster.
Most tokenisers are trained mostly on English, so other languages usually tokenise less efficiently too. Anthropic’s own pricing notes flag this directly: as of mid-2026, the tokeniser behind Claude Opus 4.7 and later, and Claude Sonnet 5, produces around 30% more tokens for the same text than the previous tokeniser, a trade-off for better answers rather than a defect.
Every major vendor exposes a token counter, either as a developer API endpoint or a usage panel inside the chat interface. Use the rule of thumb above for a one-off estimate; count properly before anything that affects a budget or a context-window limit.
Put it into practice
1. Prepare your practice material
Write three to five sentences describing a real or fictional situation in which you need to explain what tokens are and estimate how many a piece of text or code uses. Include the intended user, one constraint and how you will check the result.
2. Choose your AI tool
Choose one tool for this exercise. The remaining steps will adapt to it.
3. Copy the prompt
Prompt you will use
Take a paragraph of your own writing and a short code snippet of similar length. Ask an AI to estimate the token count of each and explain why they differ. Then check the estimate against your tool’s own token counter or usage panel, and note how far off the four-characters-per-token rule was.
4. Open ChatGPT · Paste the prompt into a new chat
Open a new chat in ChatGPT and paste the copied prompt into the message box.
Open ChatGPT5. Add your practice material
Paste your prepared scenario beneath the prompt, separated by a heading such as “Practice scenario”.
6. Send, then review the result
Answer any clarifying questions. Before using the result, check it against your source and the completion checks below.
Before you continue
You can explain a token in one sentence to a non-technical colleague.
You can estimate a token count from a character or word count.
You know why a code file often costs more tokens than prose of the same length.
