Vanilla attention was never wrong. It was expensive. Everything on the timeline below is somebody looking at that bill and trying to pay less of it β and every one of them pays for the saving with something. This page puts them in launch order, gives each one the problem it was answering, and states honestly what it gives up.
Dates are the part that is easiest to get wrong and easiest to check, so every one here was verified against the primary source rather than recalled. Turn on Show date provenance in the timeline to see the receipts.
Nothing after this makes sense without it. Six tokens, four dimensions per head, every number below computed live in your browser β not drawn. Step through the five stages.
Every token is compared with every other token. Six tokens give 36 scores. Six hundred give 360,000. Ten thousand give one hundred million. This exact all-to-all comparison is simultaneously the strength of attention and its entire cost.
Attention itself does not know order. If token 0 and token 4 hold the same word, their queries, keys and values are identical. Something else has to say which came first β which is why a third of the timeline below is about position.
Turn the causal mask off above and watch weight move onto tokens that have not happened yet. That is precisely the problem the mask solves.
They grow differently, they hurt at different moments, and almost every mechanism below attacks one of them and not the other. Confusing the two is the fastest way to pick the wrong technique.
Queryβkey comparisons at this context length: β. Double the context and this becomes roughly four times larger. It hurts when you process a long input.
Saved keys and values for one conversation: β (48 layers, 8 KV heads, head_dim 128, bf16). It hurts when you serve, and it is private per user β model weights are shared, conversation history is not.
So the real question behind every row below is: does attention become too expensive to compute, or does the saved history become too large to store? Different answers lead to different mechanisms.
Not the order they were taught, and not grouped by family. Launch order β because that is the only arrangement in which you can watch the field change its mind. Click any mechanism for its full trade-off.
Five claims from the timeline, each one recomputed live so you can move the inputs and watch the trade appear.
One query, three old keyβvalue pairs. The left route visits every old key. The right route reads one pre-built state S. With softmax off they agree exactly β which is the whole licence for linear attention.
Key A currently returns 40. It should now return 55. There is only ever one state matrix β the question is what you write into it.
2 Γ layers Γ kv_heads Γ head_dim Γ T Γ batch Γ bytes β every factor answers a simple
question. Model weights load once and are shared; this does not.
Eight query heads. Change how many K/V heads they share. Sharing lowers the line; it does not stop it rising.
Two different savings. Compression reduces how much is stored; top-k reduces how much is read.
A fixed-state layer (D) is cheap to serve but compresses the past. A sparse-attention layer (G) restores exact token access but carries a KV cache. The ordering is itself part of the architecture.
The compute panic ended in May 2022, and almost nobody noticed. From 2019 to 2021 the field produced Sparse Transformer, Reformer, Longformer, Linformer, linear attention, BigBird and Performer β seven attacks on TΒ² in under two years. Then FlashAttention made exact attention 2β4Γ faster with O(T) memory, and that entire line went quiet for three years. It was never refuted. It was made unnecessary by a kernel. A list of mechanisms cannot show you that; only the gap between 2022 and 2025 can.
The bill people care about changed, and you can date the switch. Everything before 2023 is about FLOPs. Almost everything in 2023 β GQA, sliding window, attention sinks β is about the KV cache. Nothing about attention changed in between; deployment changed. Once models served millions of concurrent conversations, the per-user private cache became the binding constraint, and the field pivoted within about twelve months.
Position is a repair cycle, not a solved problem. Learned table (2017) β sinusoidal (2017) β relative (2018) β RoPE (2021) β PI, NTK-aware, YaRN (2023) β DroPE (2025). Every entry after RoPE is a patch on RoPE. Then DroPE arrives and says the patches were the wrong idea because the positional embedding itself was the obstacle β which is what a field does right before it abandons an assumption.
Good ideas wait for their engineering. The delta rule enters linear attention in February 2021 and then nothing happens for three years and four months, until someone finds a way to parallelize it over sequence length in June 2024. Gated DeltaNet follows six months after that, and production deployments six months after that. The idea was never the bottleneck. Trainability was. Sitting in a list, the delta rule looks like a 2024 invention.
February 2025 is the tell. NSA and MoBA β two labs, the same core idea of trained-in block sparsity, published two days apart. Simultaneous invention like that means the problem was fully ripe and the pieces were lying in the open. It is the single best predictor on the whole timeline that this direction was about to become the mainstream, which it did.
The oscillation the lecture predicted is real, and it is visible. Exactness (2014β17) β compute (2018β20) β position, then exactness gets cheap again (2021β22) β decode memory (2023) β recurrent state returns (2024) β sparsity returns, now trained in (2025) β compression as the primary lever (2026). Twice the field has left fixed-size state behind and twice it has come back for it, each time with the missing piece added β first gating, then the delta rule.
Extrapolating the slope rather than claiming knowledge: the 2026 cluster disagrees about the primary lever but agrees the era of one uniform mechanism per network is over. Three things the shape suggests β (1) the fixed-size state gives way to a growing-but-sublinear one, which is exactly what Log-Linear Attention is already prototyping; (2) the schedule ratio stops being inherited and becomes something you learn or search per layer; and (3) compression moves earlier still, from the KV cache into the tokenizer and embedding, since the cheapest token remains the one never spent. The honest caveat: the timeline's own record is that the next move often comes from a kernel, not an equation, and nobody predicts those.
A longer prose version of this section is in the write-up.
Every date was checked against the primary paper or official release. These two findings emerged from that verification work.
Searching for "DroPE arXiv" returns arXiv:2503.15029 first. That paper is real, and it is not this. It is about trajectory generation for self-driving cars.
"Directional Rotary Position Embedding for Efficient Agent Interaction Modeling", 19 Mar 2025. Adapts RoPE to encode agent headings for autonomous-driving trajectory models, evaluated on Waymo Open Motion and Argoverse 2. Nothing to do with LLM context extension.
"Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings", Sakana AI, 13 Dec 2025. Removes RoPE after pretraining and recalibrates for under 1% of the pretraining budget. This is the relevant LLM method.
Nine months apart, different field, different mechanism, near-identical name. This is exactly the verification failure to guard against: an agent that half-remembers a technique and confidently attaches the wrong paper to it.
Section 9 says: "What the available record does not establish: the exact DroPE algorithm or which rotary dimensions it changes." The public record does establish it. The Sakana AI paper (arXiv:2512.12167, v1 13 Dec 2025) gives the full method, and the code is open at github.com/SakanaAI/DroPE.
The answer to "which rotary dimensions does it change" turns out to be all of them β the method's whole claim is that positional embeddings are a training scaffold rather than a permanent requirement, so they are removed from every layer and the model is recalibrated. That is fully consistent with the V4 cookbook line "positional recalibration: DroPE, applied before annealing", and with the section's correct insistence that this is a training-time procedure and not an inference switch. The epistemic caution in that section was right; it was just aimed at a gap that a public paper had already closed.
Not every mechanism has a paper. These four are the weakest links on the page, and pretending otherwise would defeat the point.