Context Windows and Tokens
7 min read
What tokens are, why the context window matters, and how to budget it deliberately.
Language models read and write text as tokens — sub-word chunks. The context window is the maximum number of tokens the model can attend to at once, covering the prompt, retrieved context, and the generated output together.
Why it matters
Every system prompt, retrieved passage, and tool result competes for the same budget. When you overflow the window, the model truncates or degrades, so treating the context window as a scarce resource is a core engineering skill.
warning
Relevant information placed in the middle of a very long context is often recalled worse than content at the start or end — the 'lost in the middle' effect.
- Trim boilerplate from system prompts
- Retrieve fewer, higher-quality passages
- Summarize long histories instead of appending them verbatim