Chroma Research tested 18 frontier language models on long documents in July 2025 and found something that cuts against a comforting assumption: organizing the text into something readable made every single model perform worse than leaving it scrambled. Anthropic's Claude Opus 4 was in the batch. So was OpenAI's o3 and Google's Gemini 2.5 Pro. All eighteen showed the same pattern. If your product ships an agent that reads a full ticket thread, a full contract, or a full codebase before it answers, the number printed on the context window was never the one that mattered.
What "context rot" actually is
"Context rot" is Chroma Research's name for a model's accuracy degrading as its input gets longer, even when the context window is nowhere near full and the correct answer is still sitting in the text. Kelly Hong, Anton Troynikov and Jeff Huber published the finding on July 14, 2025, after running three kinds of tests, needle-in-haystack retrieval with distractors, a conversational-memory benchmark called LongMemEval, and a repeated-words task, across 18 models: Anthropic's Claude Opus 4, Sonnet 4, Sonnet 3.7, Sonnet 3.5 and Haiku 3.5; OpenAI's o3, GPT-4.1, GPT-4.1 mini, GPT-4.1 nano, GPT-4o, GPT-4 Turbo and GPT-3.5 Turbo; Google's Gemini 2.5 Pro, 2.5 Flash and 2.0 Flash; and Alibaba's Qwen3 in three sizes. The needle-in-haystack runs alone covered 1,090 haystack configurations across 11 needle positions, at input lengths from 25 words up to 10,000.
A bigger window doesn't buy what the number implies
The instinct, when a long prompt starts failing, is to reach for a model with a bigger advertised window: 128,000 tokens, then 200,000, then a million. Chroma's report won't hand you one clean percentage to quote here, its accuracy curves bend downward well before the documented limit and do so unevenly across models and task types, and that unevenness is itself the finding worth taking seriously. A window rated for 200,000 tokens describes how much text a model can accept, not how reliably it uses all of it. The report states plainly that performance "degrades as input length increases, often in surprising and non-uniform ways." Non-uniform is the part that makes this hard to plan around. You can't assume a fixed safety margin and call the problem solved.
The result that should worry you more than the window size
The counterintuitive part sits in how the input is arranged, not just how much of it there is. Across all 18 models and every haystack configuration tested, Chroma found that "models perform better on shuffled haystacks than on logically structured ones." A well-organized document, sections in order, a coherent narrative, is not the easy case here. It's the harder one. Chroma's own reading is that structure hands a model a plausible path to follow that isn't necessarily the right one, while a shuffled set of sentences offers no such shortcut and forces something closer to an actual search. That's an opinion worth defending on a call: the tidy, well-formatted context your team is proud of building may be quietly working against the model reading it.
Distractors compound it. "Even a single distractor reduces performance relative to the baseline (needle only), and adding four distractors compounds this degradation further," the report states. On the LongMemEval benchmark, the gap between a focused prompt (around 300 tokens) and a full prompt (around 113,000 tokens) was large enough that Claude models showed the most pronounced drop of the families tested. In one example the report includes, Claude Sonnet 4 responds that it "cannot determine the number of days" between two dates that are both sitting in the chat history it was handed.
What this costs a team that ships agents on long context
None of this stays abstract once a product routes a full support thread or a full contract into one model call and trusts what comes back. A wrong answer from context rot doesn't look like an error message. It looks like a support agent missing a return-window date a customer already stated twice, or a code-review bot skipping a caveat buried on page four of a design doc it was handed whole. The failure is quiet and confident, and it shows up in production, not in a test suite built on the short, tidy prompts that made the demo look clean.
A worked example
Picture a 30-person AI-native startup that built its support agent on a single long-context call: dump the full ticket thread, the full account history and the product docs into one prompt, and let a frontier model answer. It passes every demo, because demos use short, clean threads. Three months after launch, tickets that reopen after a bad first answer start climbing, concentrated on the customers with the longest, messiest histories, exactly the shape of input Chroma's report says degrades first and least predictably. Nobody touched the model. The input got longer as real customers accumulated real history, and reliability slid with it.
What actually helps
Chroma's own conclusion is specific: "whether relevant information is present in a model's context is not all that matters; what matters more is how that information is presented," and it names the fix "effective context engineering." In practice that means retrieving and ranking only the passages a given question needs instead of pasting in everything available, the same discipline behind the paper that introduced RAG; summarizing older turns instead of replaying them verbatim; and testing at the input lengths your product will actually see in production rather than the short prompts a demo uses. It doesn't mean a long context window is worthless, some tasks genuinely need one, and it doesn't mean the fix is always a smaller model either. A rising time horizon for agents is a real trend, but it describes how long a model can work, not how much of what you hand it will actually get read correctly. A bigger window is not, by itself, an engineering plan.
If your team is choosing between a bigger-context model and an actual retrieval layer for a feature that reads long documents, that's a design call worth making before it ships, not after a support queue tells you it shipped wrong. Our Silicon Valley team works through that trade-off with engineering leaders directly, and we'll scope what your feature actually needs within 48 hours of a call, or start with a conversation about the feature itself.