What Is a Token, and Why Can the Same Sentence Use a Different Amount?

01 / ARTIFICIAL INTELLIGENCE / DEEP EXPLAINER

A token is neither a word nor a character

A common English word may fit one token while an unusual name breaks into several.Equal visible length does not imply equal model input.

02 / ARTIFICIAL INTELLIGENCE / DEEP EXPLAINER

The split is part of model design

The vocabulary comes from a design and training process.Frequent fragments are worth reserving as units; rare strings are decomposed.

Token count therefore reveals a hidden preference about which languages, scripts and spellings are familiar building blocks.It is not a neutral word counter attached after the model is complete.

03 / ARTIFICIAL INTELLIGENCE / DEEP EXPLAINER

The same meaning can have a different price

Multilingual research finds that equivalent information can require several times as many tokens in one language as another, depending on the tokenizer.Extra tokens consume context, add processing and may hit limits sooner.

A million-token window is not a million words and does not provide equal content capacity across languages.

04 / ARTIFICIAL INTELLIGENCE / DEEP EXPLAINER

Spaces, punctuation and formatting also count

JSON brackets and quotes, code indentation, table separators and chat-role markers all count.A short visible question may be accompanied by system policy, tool definitions, conversation history and retrieved documents.

The user's typed characters are only the visible corner of the token budget.

05 / ARTIFICIAL INTELLIGENCE / DEEP EXPLAINER

Output is often the expensive side

Services often price input, cached input and output separately.Output is decoded token by token, so latency and opportunity for error grow with length.

Be more detailed is not only a stylistic request.Useful savings usually come from removing repeated context and unnecessary output, not compressing a question into unreadable shorthand.

06 / ARTIFICIAL INTELLIGENCE / DEEP EXPLAINER

The only reliable answer is measurement

The familiar estimate of four characters per token is only a rough English guide.Run the exact prompt through the tokenizer associated with the chosen model, including system messages, history, tool schemas and expected output.

The hidden truth is that you pay not only for the question, but for the entire conversational stage around it.

07 / ARTIFICIAL INTELLIGENCE / DEEP EXPLAINER

Token efficiency is not writing quality

A shorter token sequence is not automatically a clearer prompt, and the language using fewer tokens is not automatically understood better.Compression, training coverage and task performance interact.

Optimising only the count may remove examples or structure that prevent an expensive mistake.Measure cost per successful task rather than cost per request: a prompt that uses twenty percent more context but avoids a second run may be the cheaper system.