Context Window

Context Window


The context window is the token budget a model can take into account during a single request. Prompt, conversation history, retrieved documents and the response the model is about to write all share that budget. Once the limit is passed the oldest parts drop out, and the model behaves as though it was never given that information.

Size varies by model and is counted in tokens rather than words. Text in some languages splits into more tokens than English, so the same window holds less content. A flow that wants to pass a long PDF in one go hits the ceiling earlier than expected for that reason.

The window has a cost. Attention in a transformer grows with the square of sequence length, which makes long context both expensive and slow. Even with a large window, the weight a model gives to information in the middle can drop, an effect measurable on long documents.

A concrete case: a support assistant runs a 60-message conversation with a customer. After around the thirtieth message the window fills and the account number from the opening turn falls out of context. When the customer then writes "on the same account", the model has no idea what that refers to. The fix is to summarise older turns and re-inject the critical fields on every request.

It should not be confused with maximum response length. The context window is the total budget, maximum response length is the share of it reserved for output.

From generative AI strategy to custom agent development and retrieval architectures, we help you scale AI responsibly.
Discuss your AI project