Structure Beats Magic
← All concepts
System architecture

The Attention Budget

Context is not free space. Too little and the model guesses; too much and it loses what mattered — the budget has two failure modes, not one.

The Attention Budget

A large context window reads like an invitation to fill it, and that instinct is wrong in a specific way. Attention is a budget with two failure modes, not one. Spend too little — omit the policy, the prior decision, the definition — and the model fills the gap by guessing, fluently. Spend too much and the signal you needed sits somewhere in fifty pages of padding, competing with everything else for the same finite attention. Recall degrades in the middle of long contexts; relevant facts get outvoted by volume.

This makes "just paste everything" a strategy with a ceiling, and the ceiling arrives sooner than the token limit suggests. The discipline is not compression for its own sake but selection: what does this specific decision need, in what order, and what can be left out without changing the answer? A short, precisely assembled context routinely beats a long, complete one — not because less is virtuous, but because attention spent on irrelevant material is attention withdrawn from relevant material.

The economics point the same way. Long contexts are re-read on every turn, so a conversation that grows unbounded pays for its whole history at each step. Rebuilding context per prompt rather than reusing a prepared, indexed slice is one of the largest hidden costs in production AI — and one of the easiest to remove, because the fix is structural rather than clever: assemble once, reuse, cut per task.

Which is why the budget is an argument for structure rather than against context. You cannot cut a precise slice from an undifferentiated pile; you can cut one from a graph with types, domains and dates. The vault exists so that the working set can be small — big structure at rest, small context in motion. Context engineering is the discipline of spending an attention budget well, and it only becomes possible once the material has shape.

The neighbourhood

How this connects