You retrieved the right documents. You put them all in the prompt. The model still missed the answer. This article explains the mechanism behind that failure “Lost in the Middle” and what to do about it.

The short version: transformers do not treat all input positions equally. Recall follows a U-shaped curve. Information at the start and end of a context is used well; information in the middle is used badly. The effect was named in 2023, it is architectural rather than cosmetic, and bigger context windows have not removed it.

What the effect actually is

Liu et al. tested this directly. They built multi-document question answering prompts where exactly one document contained the answer, then moved that document to different positions and measured accuracy. They also ran a key-value retrieval task over long JSON inputs. Accuracy was highest when the answer sat at the beginning or the end of the context and dropped significantly when it sat in the middle. The pattern held even for models explicitly built for long contexts, and overall accuracy fell as the input grew longer.

That gives you the shape of the problem: a U, not a flat line.

Two failure modes are routinely conflated, and separating them makes debugging much easier:

Failure mode	What varies	What you observe
Positional degradation (“lost in the middle”)	Where the evidence sits	Accuracy depends on position, following a U-curve
Length degradation (“context rot”)	How much text surrounds the evidence	Accuracy falls as input grows, even with evidence well placed

They compound. A long prompt has both a bigger middle and more distractors.

Why it happens

Three mechanisms in a standard decoder-only transformer push in different directions. The U-shape is what you get when they combine.

1. Causal masking creates a primacy bias

A decoder-only model uses a causal mask: each token attends only to itself and to earlier tokens. That makes information flow asymmetric. Early tokens are visible to everything downstream, so their representations get re-read and re-mixed at every subsequent position and every layer. Late tokens are visible to almost nothing.

Recent theoretical work models attention masks as directed graphs and shows this formally: causal masking inherently biases attention toward earlier positions, because tokens in deeper layers end up attending to increasingly contextualized representations of earlier tokens. The first part of your prompt is structurally privileged before any training happens.

2. Attention sinks park probability mass on the first tokens

Softmax attention weights must sum to 1. When a head has nothing relevant to attend to, it still has to put that mass somewhere. Empirically, models dump it on the first few tokens — the beginning-of-sequence token in particular — regardless of whether those tokens carry any meaning. Xiao et al. named this the attention sink and showed it is strong enough that dropping the first tokens from a sliding-window KV cache destroys generation quality, while keeping just four of them restores it.

The practical consequence: a measurable share of every head’s attention budget is spent on tokens that are not about your task. Follow-up work has also found sink behaviour on low-information tokens like punctuation and newlines in the middle and later parts of an input, where the effect is not benign.

3. Rotary position embeddings decay with distance

RoPE encodes relative position by rotating query and key vectors. As the distance between two positions grows, the phase shifts accumulate and the rotated components fall out of alignment, so attention magnitude decays. This is a useful inductive bias for language — nearby tokens usually matter more — but on a long prompt it becomes a recency bias that suppresses anything far from the current position.

The three combined

Causal masking pulls attention backwards toward the start. RoPE decay pulls it forward toward the recent. Content sitting in the middle gets neither anchor. Analysis by Hsieh et al. confirms the resulting attention pattern is U-shaped with respect to position and largely independent of relevance — documents in the middle lose attention even when they hold the answer, and even when document order is shuffled.

One more factor sits on top of the architecture rather than inside it. Longer inputs mean more distractors competing for the same fixed attention budget, and the more semantically similar those distractors are to the query, the worse it gets.

The evidence is not just from 2023

Three lines of work matter here.

NoLiMa (ICML 2025) removed the shortcut that flatters most needle-in-a-haystack tests. In standard NIAH, the question and the needle share vocabulary, so the model can succeed by literal matching. NoLiMa builds needles with minimal lexical overlap, forcing real semantic association. Across models claiming 128K or more, short-context performance was strong but degraded sharply with length: at 32K, most tested models fell below half their short-context baseline, and GPT-4o dropped from 99.3% to 69.7%. Chain-of-thought and reasoning modes did not rescue it.

Chroma’s Context Rot report (2025) evaluated 18 frontier models on tasks where task difficulty was deliberately held constant while input length varied. Every model degraded. Degradation was non-uniform across families, and it depended on needle-question similarity, distractor presence, and even how the surrounding haystack was structured.

HELMET (2024) extended the position analysis up to 128K tokens across six evenly spaced needle depths and found models generally favouring the most recent context while struggling with the middle and earlier portions.

Read together: the U-curve is a positional effect, context rot is a length effect, and shipping a 1M-token window addresses neither. A larger window mostly gives you more middle to lose things in.

What to do about it

Ordered roughly by how much leverage you get per unit of effort.

1. Send less. The most reliable intervention is a smaller, cleaner prompt. Treat your effective context as a fraction of the advertised window, not as a target to fill. If retrieval returns 40 chunks and 6 are relevant, send 6.
2. Put critical content at the edges. Task instructions, the question, and output constraints belong at the very start or the very end — ideally both, with the instruction restated after the documents. Reserve the middle for material the model only needs to skim.
3. Order retrieved documents deliberately. Do not paste your vector store’s output in raw similarity order. Rerank, then place the strongest candidates at the head and tail of the block and the weaker ones inside. This costs nothing and directly exploits the curve.
4. Make boundaries explicit. Wrap each document in a delimiter with an identifier — XML-style tags work well — so the model can address a specific source rather than a position. Then ask it to cite or quote the passage it used before answering. Grounding the answer in an explicit retrieval step reduces reliance on whatever the attention pattern happened to favour.
5. Compact aggressively in agent loops. Agents accumulate tool output, failed attempts, and exploration noise. That noise is input length, and input length is the second failure mode. Summarise finished sub-tasks, drop dead branches, and re-inject only the working set.
6. Measure your own effective context. Take a real task from your system, embed the answer at 0%, 25%, 50%, 75% and 100% depth, and run each position 20 times at several context lengths. You will get your own U-curve for your model and your data, which is worth more than any published benchmark number.

Research-side mitigations exist but are mostly not exposed in production APIs. Attention calibration — estimating the positional component of attention and subtracting it — recovered up to 15 percentage points on downstream RAG tasks in the Found-in-the-Middle work. Positional-encoding rescaling and hybrid RoPE/NoPE layer schemes attack the decay term directly. If you serve your own weights, these are on the table. If you call an API, structure and retrieval are your levers.

What does not fix it

• A bigger context window. Capacity is not the constraint; signal-to-noise is.
• Asking the model to pay attention. Position bias is structural, not a matter of instruction following.
• A single needle-in-a-haystack score. If the needle shares vocabulary with the question, you are measuring literal matching, not retrieval under realistic conditions.

The practitioner’s summary

Attention is a budget, not a spotlight. Causal masking spends part of it on the beginning, RoPE decay spends part on the end, attention sinks spend part on nothing at all, and whatever is left is what your middle documents are competing over. Design the prompt around that budget rather than around the window size on the pricing page.

Glossary

Grouped by theme, alphabetical within each group.

Model architecture and mechanics

• Attention — The operation that lets a token pull information from other tokens. Each token emits a query, every token exposes a key and a value; the query-key similarities become weights, and the output is the weighted average of the values. “Attending to” a token means assigning it a non-trivial weight.
• Attention budget — Informal but useful framing: because attention weights sum to a fixed total per head, attention is a finite quantity to be divided, not a spotlight that can be pointed everywhere at once. Anything spent on sink tokens or on distractors is unavailable to the content you care about.
• Attention head — One independent attention computation inside a layer. A layer runs many heads in parallel, each with its own learned projections, so different heads specialise in different relationships (syntax, coreference, position, and so on).
• Attention weights — The normalised scores produced by softmax over query-key similarities. They quantify how much each source token contributes to a given target token’s updated representation.
• BOS token (beginning-of-sequence) — A special token prepended to the input to mark where the sequence starts. It carries no task content, which makes its outsized attention share in most models notable rather than expected.
• Causal mask / causal masking — The constraint in a decoder-only model that a token may attend only to itself and to earlier positions. It is what makes autoregressive generation possible, and it is also what makes information flow through the sequence asymmetric.
• Contextualised representation — The vector a layer produces for a token after that token has mixed in information from the tokens it attended to. Deeper layers hold representations that have absorbed more surrounding context, which is why early tokens gain influence with depth: later positions attend to increasingly rich summaries of them.
• Decoder-only transformer — The architecture behind essentially all current generative language models: a stack of layers, each combining causally masked self-attention with a feed-forward network. “Decoder-only” distinguishes it from encoder-decoder designs used in translation.
• Head / tail of a block — In prompt-construction advice, the first and last items of a list of documents. These are the positions the U-curve rewards.
• Inductive bias — A built-in preference that makes a model favour some hypotheses over others before it sees data. Distance decay in a positional encoding is an inductive bias toward local dependencies: helpful for ordinary language, harmful when the dependency you need spans 40,000 tokens.
• Layer — One block in the stack. Text passes through every layer in order, so “at every layer” means an effect compounds dozens of times in a large model.
• Query, key, value (QKV) — Three learned linear projections of each token’s representation. Queries ask, keys advertise, values carry the payload that gets mixed.
• Softmax — The function that turns raw attention scores into a probability distribution: exponentiate, then divide by the sum. The crucial property here is the normalisation — the weights are forced to sum to 1, so a head cannot decline to attend to anything.
• Token — The unit a model actually reads: a word, word fragment, or piece of punctuation produced by the tokenizer. Context lengths and prices are quoted in tokens, not words; for English, roughly 0.75 words per token is a workable estimate.
• Weights (model weights) — The learned parameters. “If you serve your own weights” means you run the model yourself and can modify inference internals such as attention scores — impossible through a hosted API.

Position and positional encoding

• Effective context length — The longest input at which a model still performs acceptably, as opposed to the longest input it will accept. NoLiMa operationalises it as the maximum length where the score stays above 85% of the model’s short-context baseline; measured that way, effective lengths land far below advertised windows.
• NoPE (no positional encoding) — Layers with no explicit position signal at all, relying on the causal mask to convey order implicitly. Hybrid schemes alternate RoPE layers (for local resolution) with NoPE layers (to avoid distance decay).
• Phase shift — In RoPE, position is applied as a rotation by an angle proportional to the index. The phase shift is that angle. Over long distances, the accumulated shifts across frequency components interfere and cancel, which is the mechanism behind long-term decay.
• Position bias / positional bias — Any systematic dependence of model behaviour on where content sits rather than what it says. Lost-in-the-middle and attention sinks are both instances.
• Positional encoding — The mechanism that tells attention about order, since the attention operation itself is permutation-invariant. Approaches include absolute learned embeddings, additive relative biases such as ALiBi, and rotary embeddings.
• Positional-encoding rescaling — Adjusting the frequency or scaling parameters of an existing positional encoding, usually to extend a model’s usable context beyond its training length or to flatten distance decay. YaRN and LongRoPE-style methods belong here.
• Primacy bias — Better use of information near the beginning of the input.
• Recency bias — Better use of information near the end of the input, closest to the position currently being generated.
• RoPE (rotary position embedding) — The dominant positional encoding in current open models. It rotates query and key vectors by a position-dependent angle so that their dot product depends on relative distance. Elegant and extrapolation-friendly, but it carries an intrinsic decay term.
• Long-term decay — The property, provable for standard RoPE, that the upper bound on attention magnitude shrinks as relative distance grows. It is the formal statement of “far-away tokens get less attention regardless of relevance”.
• U-shaped curve — The characteristic plot of accuracy (or of average attention) against the position of the relevant information: high at both ends, low in the middle.

The phenomena

• Attention sink — A token that absorbs a large, persistent share of attention without being semantically relevant, typically the first few tokens of the sequence. Named by Xiao et al., who showed that discarding sink tokens from a sliding-window cache collapses generation quality while retaining about four of them restores it. Later work found sink behaviour on low-information tokens elsewhere in the input too.
• Context rot — Length degradation: accuracy falling as input grows even when the relevant content is fixed and well placed. Coined by Chroma in 2025. Distinct from window overflow, which is a hard limit; rot begins well before it.
• Distractor — A passage in the input that is plausibly related to the query but does not contain the answer. Distractors are what make realistic retrieval hard, and their damage grows with their semantic similarity to the query.
• Lost in the middle — Positional degradation: relevant information is used well at the input’s edges and badly in its interior. Named by Liu et al. (2023; TACL 2024).
• Positional degradation vs length degradation — The two-way split this article insists on. The first varies where the evidence is; the second varies how much text surrounds it. Different causes, different fixes, and they compound.
• Signal-to-noise ratio — The proportion of the prompt that is actually relevant to the task. The argument of this article is that this ratio, not raw window capacity, is what governs long-context reliability.

Evaluation

• Baseline score — A model’s accuracy on the same task at short context, used as the reference point against which longer-context scores are expressed. Without it, absolute long-context numbers say little, because task difficulty and model strength are confounded.
• Chain-of-thought (CoT) — Prompting or training a model to produce intermediate reasoning before its answer. Effective on many reasoning tasks; NoLiMa found it does not rescue long-context retrieval.
• Depth (needle depth) — Where the needle is placed, expressed as a fraction of total context: 0% is the very start, 100% the very end. Sweeping depth is how you recover a U-curve for your own setup.
• Haystack — The bulk of irrelevant filler text surrounding the needle.
• HELMET — A long-context evaluation suite that tests multiple task families and analyses needle position up to 128K tokens.
• Lexical overlap / literal matching — Shared surface vocabulary between the question and the target passage. When it exists, a model can succeed by pattern-matching strings rather than understanding, which inflates benchmark scores relative to real use.
• Needle-in-a-haystack (NIAH) — The standard long-context probe: insert a distinctive fact (the needle) into long filler text (the haystack) and ask the model to retrieve it. Cheap to run, and easy to pass for the wrong reason.
• NoLiMa — A benchmark that rebuilds NIAH with needles deliberately sharing minimal vocabulary with the question, forcing genuine semantic association. Modarressi et al., ICML 2025.
• RULER — A synthetic long-context benchmark that varies task type and length systematically to estimate the length at which each model’s performance actually breaks down.
• Semantic / latent association — Recognising that two passages refer to the same thing despite sharing no words. What NoLiMa tests, and what real queries usually require.

Systems and mitigation

• Agent loop — An iterative pattern where a model calls tools, reads results, and decides what to do next. Each iteration appends to the context, so accumulated tool output, failed attempts, and dead-end exploration turn directly into length degradation.
• Attention calibration — Estimating the purely positional component of attention and subtracting it, so remaining scores track relevance rather than location. The Found-in-the-Middle method does this and reported gains of up to 15 percentage points on downstream RAG tasks. It requires access to attention internals, so it is generally unavailable through hosted APIs.
• Chunk / chunking — Splitting source documents into passages small enough to embed and retrieve individually. Chunk size and boundary placement determine how much irrelevant text rides along with each hit.
• Compaction — Replacing accumulated context with a shorter summary so the working set stays small. The standard defence against length degradation in long-running agents.
• Embedding — A dense vector representation of a passage or query, positioned so that semantically similar text lands nearby. The basis of vector search.
• Grounding — Tying a generated claim to an identified source passage, usually by requiring a citation or quotation before the answer. It reduces reliance on whichever positions the attention pattern happened to favour.
• RAG (retrieval-augmented generation) — Retrieving relevant passages from an external store at query time and placing them in the prompt, instead of relying on parametric knowledge. Everything in this article applies to how those retrieved passages are ordered and sized.
• Reranking — A second-stage scoring pass over retrieved candidates, typically with a cross-encoder that reads query and passage together and is therefore more accurate than vector similarity. Reranking gives you a trustworthy ordering, which is what makes deliberate placement worthwhile.
• Vector store — The database holding embeddings and serving nearest-neighbour lookups. Its default output order reflects similarity, not the placement your prompt should use.
• Working set — The subset of accumulated context that is still needed for the current step. Keeping the prompt close to the working set is the practical goal of compaction.
• XML-style tags — Explicit delimiters such as <doc id=”7″>…</doc> wrapped around each source. They give the model a stable handle for referring to a document by identity rather than by position, and they make source boundaries unambiguous.

Sources

• Lost in the Middle: How Language Models Use Long Contexts — Liu et al., TACL 2024 (ACL Anthology, code and data)
• Efficient Streaming Language Models with Attention Sinks — Xiao et al., ICLR 2024 (MIT Han Lab explainer)
• Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization — Hsieh et al., Findings of ACL 2024
• On the Emergence of Position Bias in Transformers — graph-theoretic analysis of causal masking and positional encodings
• NoLiMa: Long-Context Evaluation Beyond Literal Matching — Modarressi et al., ICML 2025 (benchmark repository)
• Context Rot: How Increasing Input Tokens Impacts LLM Performance — Hong, Troynikov and Huber, Chroma, 2025
• HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly — position analysis up to 128K tokens