1. The Context Window: All the Text an AI Can Refer to at Once
The context window is like an AI’s work desk. According to Anthropic, it is all the text a language model can refer to when making a response, including the response itself. It is separate from knowledge learned during training. [1]
As a conversation continues, user messages and AI replies stack up in this window. When nearing the limit, older parts may be summarized to continue. [1]
The size varies by model and time. As of October 2026, some newer models are said to have windows of 1 million tokens, while Claude Sonnet 4.5 has a window of 200,000 tokens. The maximum output for one response is listed as 128,000 tokens. These values may change. [1] Tokens are the units AI uses to count text.
So, is a bigger window always better?
2. Why Doubling Sentence Length Quadruples the Attention Pairs in a Transformer
The foundation of current AI is the Transformer design, published in 2017. Instead of reading sentences one by one, it processes relationships across the whole sentence using "attention." This allows for parallel processing and faster learning. [2]
In attention, every word (token) calculates its relationship with every other token. The paper shows that computational cost is proportional to n², where n is the length of the sequence. It is like everyone in a class greeting everyone else: if the number of people doubles, the number of combinations becomes about 4 times larger. [2]
If n doubles, attention combinations become 4 times larger. For example, going from a 200,000-token window to 1 million tokens (5 times larger) means the number of attention combinations becomes about 25 times larger in simple calculation. This applies only to the attention part, not the entire AI computation. [2]
The paper states that for normal-length sentences, this attention mechanism is faster than previous methods. As sentences get longer, this advantage fades. [2]
Beyond speed, what else limits the window size?
3. How the KV Cache Used About 30% of GPU Memory in One Model
To generate text, AI saves calculation results (keys and values) for past tokens to use when creating the next word. This is called KV cache. Saving it avoids recalculating every time.
A 2023 paper on vLLM reported that for a model named OPT with 13 billion parameters, the KV cache per token was 800 KB. For one request of 2,048 tokens, this is about 1.6 GB. [3] The same paper noted that in 40 GB of GPU (A100) memory, model weights took about 65%, while dynamic state per request (mostly KV cache) took about 30%. [3]
KV cache grows proportionally with tokens. If we apply the OPT value (800 KB) to 1 million tokens, it would be 800 GB. This is 20 times the memory of a 40 GB GPU. This is a simple multiplication estimate; actual models vary by structure.
The paper reports that PagedAttention, which uses virtual memory concepts from operating systems, reduced KV cache waste and achieved 2 to 4 times more processing than existing systems. [3] Improvements are possible, but storage for long text is still needed.
4. Why AI Struggles With Inputs Longer Than the Lengths It Learned
AI receives word order as position information. The 2017 paper chose wave-like representations partly because they might extend well to sequences longer than those seen during training. "Outside the learned length" was a challenge from the start. [2]
In 2021, RoPE (Rotary Position Embedding) was proposed. It represents word order with numbers and uses distance between words in calculations, said to be flexible for length. [5]
However, models using RoPE often cannot handle sentences longer than their training length without adjustment. [6] In 2023, YaRN proposed a method to expand windows with less additional training. It reported needing 1/10th the tokens and 1/2.5th the training rounds compared to previous methods. These numbers are reported by the authors under specific conditions. [6]
If we expand the window this way, can AI use long texts well?
5. The 2023 Experiment: AI Performed Worse on Answers in the Middle of Long Inputs
Liu et al. tested AI performance by changing the position of documents containing answers within long inputs. In question-answering with 10, 20, or 30 documents and key-value retrieval, performance was highest when the answer was at the beginning or end. It dropped in the middle, showing a U-shaped trend. [7] This paper was published in 2023 and later appeared in an academic journal (TACL).
The same paper found that when input fits within both normal and extended windows, performance is nearly the same. This suggests window size and the ability to use contents are different things. [7]
These results are from 2023 models. Latest models may differ. The "context rot" mentioned earlier also varies by source.
Safety Note: It is safe to read the constraints behind limits (attention computation, KV cache, learned length) and the trend of lower performance for information in the middle as separate problems.
6. Try It Yourself: Do People Also Forget the Middle of a List?
Write a list of about 20 unrelated object names. Have an adult read them one by one. After listening, write down what you remember. Check which positions (start, middle, end) were remembered most.
If results are similar or different, it does not mean AI and human memory work the same way. Trying with different lists helps notice effects of position and word memorability.
To read more, start with Anthropic’s official docs [1] and Liu et al.’s paper [7], looking at the figures first.