Bigger Isn’t Always Better When It Comes to Context Windows

If one model supports a 30,000-token context window and another supports a million, the second one probably sounds better to you. More context means access to more information, so it must be smarter? Well, not always.

What Is a Context Window?

Think of a context window as your working desk. You can fill it with books, previous notes, basic stationery, and maybe even tools like a laptop. Basically, everything you might need while working or studying.

The same goes for large language models. They operate inside a limited working area, usually called the context window, which may contain:

  • Human prompt
  • Previous user messages
  • Instructions
  • Documents
  • Source code
  • Tool outputs

Tokens are the units the model uses to represent this information. Just as a bigger desk gives you room for more books and tools, a larger context window allows a model to work with more information at once.

The Problem with Bigger Desks

Imagine your desk covered with hundreds of books, papers, notes, and tools. The note you need might be somewhere on that desk, but finding it among everything else becomes harder. At some point it’s not efficient anymore.

Imagine giving a thousand-page manual to an AI just to answer a few questions, sharing an entire source-code repository just to generate code that fits the project, or keeping every message from a very long conversation in its context.

Modern models may have a large enough context window to fit all of this data, but does that mean the model can use all of it in an equal, efficient way? Are they still reliable?

There is a well-known paper by Liu et al. (2024) that explores this question. Researchers tested models using tasks where the necessary information was placed at different positions inside a long context. Models often performed better when the relevant information appeared near the beginning or end of the prompt. But performance often started to drop when that same information was placed anywhere in the middle.

This phenomenon became known as “lost in the middle.” It shows that information being inside the context does not guarantee that the model will use it equally well. Later, a broader evaluation called Context Rot examined how increasing context length affects model performance. It found that performance generally became less reliable as the context grew. Also, the degradation wasn’t always smooth or predictable.

Maximum Context vs. Effective Context

There can be a substantial gap between what many token context windows a model advertises and the point at which the model starts to make mistakes, or an effective context window.

And the effective window isn’t necessarily a fixed number either. It depends on the task; for example, finding one unusual sentence inside 100,000 tokens may be relatively easy. But finding five related facts, connecting them, resolving contradictions, and reasoning about them can be much harder.

More Context Can Actually Introduce Problems

The main idea behind increasing the context window is to access more information, which can be useful. In some cases it does, but adding to the context window endlessly can introduce even more new problems.

More context may contain irrelevant or outdated information, noisy data, duplicate data, contradictory content, and basically more opportunities to mess up. More chances to end up with a wrong answer.

Also don’t forget that tokens aren’t free. Larger prompts require more processing and obviously more tokens, more usage memory, and more latency. Imagine processing a million tokens every time someone asks a question that could have been answered using a fraction of that. Even if the final answer were identical, one approach is doing considerably more work and has more engineering cost.

What Is the Solution?

Maybe the easiest way to think about context is not as storage, but as working memory. We already have databases, filesystems, search engines, vector databases, and object storage for keeping large amounts of information.

The context window doesn’t need to replace them, but instead, it can contain the information the model needs right now. Just like the desk example, you might own hundreds of books in your library. But that doesn’t mean you need to put all of the books on your desk every time you sit down to work or study. You can just cherry-pick a few books relevant to your current topic. We call this Context Engineering.

So, Is Bigger Better?

Well, sometimes. Large context windows are a major improvement cause they allow models to analyze longer documents, work across larger codebases, compare multiple sources, process long conversations, and solve problems that would have been impossible with very small context windows.

The mistake is assuming that context window size is itself a measure of intelligence. A model supporting one million tokens isn’t automatically smarter than one supporting 300,000, and a model accepting an entire repository isn’t necessarily going to understand code better than a system that carefully selects the relevant files.