Context Budgeting
Also known as: context allocation, window partitioning, token budget
Give every component of the prompt an explicit token allocation and enforce it, instead of letting whatever arrives first fill the window.
The problem
Context is a fixed resource that most systems spend implicitly. Tool definitions grow as integrations are added, retrieval returns whatever the retriever felt like, conversation history accumulates without limit, and one day a long tool result pushes the system prompt's safety rules out of the window — or the request simply fails at 3am under a load pattern nobody tested. Nobody decided this; it emerged from components that each assumed they could have what they needed.
The pattern
Partition the window explicitly. Assign each component a token budget in priority order — system instructions, current task, tools, retrieved context, history, scratch — and enforce each budget at assembly time with a component-appropriate reduction strategy. Assembly becomes a single function that either produces a prompt within budget or reports precisely what it had to drop.
Use when
- Any system where context content varies at runtime: RAG, agents, long conversations
- You have hit context-length errors in production, or truncation is happening silently
- Cost is dominated by input tokens and you need to know where they go
- Multiple teams contribute to one prompt and no one owns the total
Avoid when
- Fixed, short prompts with no runtime variability
- Very early prototyping, where premature budgeting slows iteration
- The model's window is so much larger than your worst case that spending is genuinely irrelevant
Tradeoffs
GainContext-overflow failures become impossible rather than intermittent
CostAn assembly layer to build and maintain, plus per-component reduction strategies
GainToken spend becomes attributable — you can see which component costs what
CostBudgets need retuning whenever a component's typical size changes
GainPriority ordering guarantees critical instructions are never the thing dropped
CostSomething must be dropped, and the drop policy is a real product decision
A context window is a budget, and budgets that are not allocated get spent by whoever grabs first. The fix is unglamorous and effective: decide the allocation, encode it, and enforce it in one place.
Allocate in priority order
Order components by what must never be dropped. System instructions and the current user turn are inviolable. Tools are usually next, because a missing tool silently removes a capability. Retrieved context and history compete for what remains, and both have graceful reduction strategies — retrieval can return fewer chunks, history can be summarized.
@dataclass
class Allocation:
name: str
max_tokens: int
priority: int # 1 = never drop
reduce: Callable[[str, int], str]
BUDGET = [
Allocation("system", 2_000, 1, reduce=never),
Allocation("user_turn", 4_000, 1, reduce=truncate_middle),
Allocation("tools", 6_000, 2, reduce=drop_least_relevant_tools),
Allocation("retrieved", 24_000, 3, reduce=drop_lowest_ranked_chunks),
Allocation("history", 20_000, 4, reduce=summarize_oldest),
Allocation("scratch", 4_000, 5, reduce=truncate_head),
]
WINDOW = 128_000
RESERVED_FOR_OUTPUT = 8_000
SAFETY_MARGIN = 4_000 # tokenizer disagreement, provider overheadEnforce at assembly, report what was dropped
One function assembles the prompt. It reduces any component over its own allocation, then — if the total still exceeds the window — reclaims from the lowest-priority components until it fits. It returns both the prompt and a report of what was reduced, because silent truncation is how systems degrade invisibly for weeks.
def assemble(parts: dict[str, str]) -> tuple[list[Message], AssemblyReport]:
report, sized = AssemblyReport(), {}
for alloc in BUDGET:
text = parts.get(alloc.name, "")
n = count_tokens(text)
if n > alloc.max_tokens:
text = alloc.reduce(text, alloc.max_tokens)
report.reduced[alloc.name] = (n, count_tokens(text))
sized[alloc.name] = text
ceiling = WINDOW - RESERVED_FOR_OUTPUT - SAFETY_MARGIN
total = sum(count_tokens(t) for t in sized.values())
# Still over: reclaim from the least important components first.
for alloc in sorted(BUDGET, key=lambda a: -a.priority):
if total <= ceiling or alloc.priority == 1:
break
before = count_tokens(sized[alloc.name])
sized[alloc.name] = alloc.reduce(sized[alloc.name], max(0, before - (total - ceiling)))
after = count_tokens(sized[alloc.name])
report.reclaimed[alloc.name] = before - after
total -= before - after
if total > ceiling:
raise ContextOverflow(report) # loud, not silent
metrics.gauge_many({f"context.{k}": count_tokens(v) for k, v in sized.items()})
return render(sized), reportReserve tokens for output and keep a safety margin. Tokenizer counts differ between your library and the provider, and system overhead is not always visible to you. A prompt sized to exactly the window will fail intermittently in ways that are miserable to debug.
Position matters as much as size
Models attend unevenly across a long context: material at the very beginning and very end is used more reliably than material buried in the middle. Put stable instructions at the top — which also makes them cacheable — and the current task and most relevant retrieved context at the bottom, closest to where generation begins. Reordering costs nothing and measurably changes behaviour.
- Allocate by priority; system instructions and the current turn are never dropped
- Reduce per component with a strategy that suits it, not with blanket truncation
- Reserve output tokens and a safety margin below the nominal window
- Emit per-component token gauges — this is where cost regressions become visible
- Stable content first for cache hits; the most decision-relevant content last
Alert when any component exceeds its allocation regularly. It is a leading indicator: tools creeping over budget means someone added integrations without pruning, and history overrunning means your summarization threshold is stale.