What is Large Language Model (LLM) Context Windows?

What is Large Language Model (LLM) Context Windows?

Introduction to Context Windows

When interacting with a Large Language Model (LLM), it is easy to anthropomorphize the system and assume it has a continuous, human-like memory. In reality, the model itself is stateless. Every time you send a message to an AI, the application sends the conversation history again and the model processes it (providers may cache repeated prompt prefixes to save work).

The limit on how much text an LLM can process in a single request is known as the Context Window.

If the LLM is the brain, the context window is its short-term working memory. It determines how large a document you can ask the model to summarize, how much historical chat it can remember, and how much external data can be injected via Retrieval-Augmented Generation (RAG). Understanding the mechanics, limitations, and costs associated with context windows is a basic requirement for anyone engineering AI applications.

Tokens: The Currency of Context

To understand context windows, we must first understand how LLMs read text. LLMs do not read words; they read Tokens.

Before text is fed into a neural network, a “Tokenizer” chops the string into sub-word pieces.

• A short, common word like apple is usually one token.
• A longer or less common word like anthropomorphize might be split into several tokens (e.g., anthro, pomorph, ize).
• As a general rule of thumb in English, 100 tokens equates to roughly 75 words.

When an AI provider advertises a “128,000 token context window,” they mean the model can ingest approximately 96,000 words (the length of a short novel) in a single prompt.

Input vs. Output Tokens

The context window is a shared space. It includes the system instructions, the user’s prompt, the injected RAG data (the Input Tokens), plus the space required for the model to generate its response (the Output Tokens).

If a model has an 8,000 token limit, and you provide a 7,500 token prompt, the model will stop generating its answer after at most 500 tokens, resulting in a cut-off, incomplete response.

The Engineering Challenge: The Attention Mechanism

Why can’t we simply build models with unlimited context windows? The limitation lies in the core architecture of the Transformer neural network: the Self-Attention Mechanism.

When a Transformer processes text, it compares every token in the sequence to every other token in the sequence to understand context and relationships. This mathematical operation scales quadratically ($O(N^2)$).

• If you double the number of tokens in the context window, the compute required for the attention step roughly quadruples. (Memory-efficient kernels like FlashAttention avoid storing the full attention matrix, but the compute still grows quadratically.)
• A 1-million token context window requires substantial GPU memory, much of it for the key-value cache, and often multiple GPUs for a single request.

Techniques such as Ring Attention, along with alternative architectures like state-space models (Mamba), have pushed these limits, and models like Gemini 1.5 Pro offered context windows of 1 to 2 million tokens. Very long context windows still introduce secondary challenges.

The “Lost in the Middle” Phenomenon

Research (notably the 2023 “Lost in the Middle” paper by Liu et al.) has shown that LLMs do not use all parts of a long context equally well. Retrieval accuracy follows a U-shaped curve.

• The model reliably uses facts located at the beginning of the prompt (the system instructions).
• The model reliably uses facts at the end of the prompt (the user’s final question).
• However, accuracy drops when the model must retrieve facts buried in the “middle” of a long document.

Therefore, simply stuffing a 500-page PDF into a large context window is often an anti-pattern. The model may miss or misstate important details buried on page 250. Newer models have improved on this, but careful context selection still helps.

Strategies for Managing Context Limits

Because context is computationally expensive and prone to degradation, AI engineers use strategies to manage it.

1. Retrieval-Augmented Generation (RAG)

Instead of feeding an entire database into the context window, RAG uses semantic search to find only the few passages most relevant to the user’s question. This keeps the prompt short, cheap, and focused on relevant facts, reducing the “Lost in the Middle” effect.

2. Conversation Summarization (Memory Rolling)

In a long-running chatbot application, the conversation history will eventually breach the context limit. To keep requests within the limit, engineers implement “Memory Rolling.” When the chat approaches the context limit, a background process takes the oldest messages and asks a cheaper, faster LLM to summarize them. The raw messages are deleted from the array and replaced with a short summary (e.g., “User and Assistant discussed setting up a Python environment”). This frees up room in the context window while preserving the gist of the historical conversation.

3. Prompt Compression

Some tools (such as Microsoft’s LLMLingua) compress prompts before sending them to the LLM. They remove low-information tokens, unnecessary whitespace, and redundant phrasing, keeping the meaning in fewer tokens to save costs and space.

Conclusion

The context window is a defining operational constraint of generative AI. While providers keep pushing toward million-token capacities, compute cost and attention degradation remain. Mastery of Prompt Engineering and AI architecture ultimately comes down to treating the context window not as an unlimited dumping ground, but as valuable, constrained space where only the most relevant, compressed information should reside.

Learn More

To learn more about the Data Lakehouse, read The Lakehouse Built for Everyone by Alex Merced. You can find this and other books by Alex Merced at books.alexmerced.com.