Tokens are not words or characters
Language models operate on tokens, produced by a subword tokeniser that splits text into pieces drawn from a fixed vocabulary. Common words are usually a single token; rarer words break into several. For ordinary English prose, a useful approximation is about 0.75 words per token, or roughly four characters per token.
That ratio is a property of English specifically. Languages written in other scripts fare considerably worse, because the vocabulary was built predominantly from English text — the same sentence in Japanese, Thai, Hindi or Arabic can cost two or three times as many tokens, which makes the same request proportionally more expensive and consumes context faster. This is a real and under-discussed inequity in how these systems are priced.
Structure costs more than prose. JSON, code and markup spend tokens on braces, quotes and indentation, and whitespace is not free — repeated indentation in a long file can account for a surprising share of the total. Numbers tokenise poorly too, often splitting into several pieces per number, so a large table of figures is denser in tokens than its character count suggests.
Why counts differ between models
Each model family uses its own tokeniser and vocabulary, so the same text yields different counts depending on which model you are targeting. A count produced for one family is an estimate rather than an exact figure for another, and the gap widens for non-English text and for code.
Vocabularies have also grown over time, and larger vocabularies generally mean fewer tokens for the same text — newer models often encode identical input more efficiently than their predecessors. That is worth knowing when comparing costs, since a lower per-token price on one model does not automatically mean a lower cost per request.
Treat any count as an estimate with a margin, and leave headroom in your calculations rather than budgeting to the exact limit.
Budgeting a context window
The context window covers everything in the exchange: the system prompt, the conversation history, any retrieved documents, the current message, and the space the response will occupy. Requests fail when the total exceeds the window, and the most common cause is forgetting that the output needs room too — reserve it explicitly.
In a multi-turn conversation the history grows with every exchange, so cost per turn rises even if messages stay the same length, and eventually the earliest turns must be dropped or summarised. Retrieval-augmented systems face the sharper version: attaching several documents to a prompt can consume the window rapidly, so retrieving fewer, better-matched passages usually beats retrieving more.
Input and output are typically priced differently, with output the more expensive of the two, so an application generating long responses costs more than its input volume implies. Prompt caching, where a long unchanging prefix is billed at a reduced rate on subsequent calls, changes this arithmetic substantially for applications with a large fixed system prompt — worth checking against your provider's current pricing rather than assuming.
How many words is a token? Conversion table
For ordinary English prose one token averages about 0.75 words, or roughly 4 characters. That ratio is a planning estimate, not a rule — the counter above tokenises your actual text, which is the only exact answer. Code, markup, non-Latin scripts and unusual names all use more tokens per word than the table implies.
| Tokens | Approx. words | Approx. characters | Roughly |
|---|
| 100 tokens | ~75 words | ~400 | A short paragraph |
| 500 tokens | ~375 words | ~2,000 | A one-page memo |
| 1,000 tokens | ~750 words | ~4,000 | Two pages, or a long email |
| 2,048 tokens | ~1,500 words | ~8,200 | A short blog post |
| 4,096 tokens | ~3,000 words | ~16,000 | A long article |
| 8,000 tokens | ~6,000 words | ~32,000 | A short book chapter |
| 16,000 tokens | ~12,000 words | ~64,000 | A long chapter |
| 32,000 tokens | ~24,000 words | ~128,000 | A short thesis |
| 128,000 tokens | ~96,000 words | ~512,000 | A full-length novel |
| 200,000 tokens | ~150,000 words | ~800,000 | A long novel |
| 1,000,000 tokens | ~750,000 words | ~4,000,000 | Around eight novels |
Going the other way: 1,000 words is roughly 1,333 tokens, and 1,000 characters is roughly 250 tokens. A useful sanity check is that a token is about three-quarters of a word, so any word count multiplied by 1.33 lands close.