Attention, in order
← The interactive version

Attention, in order

I put 46 attention mechanisms on a single timeline by launch date — not by family, not by how they’re usually taught. Six things became visible that a list completely hides. One of them is that the great sparse-attention wave of 2019–2021 was killed by a CUDA kernel.

If you learned attention from a survey paper, you learned it as a taxonomy: here are the sparse ones, here are the linear ones, here are the positional encodings. That arrangement is tidy and it throws away the single most informative dimension in the data — when each one showed up.

So I built the other version. Every mechanism from Bahdanau’s 2014 additive attention to DeepSeek-V4’s compressed sparse attention in April 2026, on one chronological axis, each one framed as an answer to a problem that existed at that moment.

The app is here: attention-in-order.netlify.app
The code and the full source table: github.com/dattatreyamanjunath/ERA-V5

Start with the thing everything else is modifying

Standard scaled dot-product attention is softmax(QKᵀ/√dₖ + M)V. Every token projects into a query, a key and a value; all pairs get scored in one matmul; a causal mask sets future scores to −∞; softmax turns each row into weights summing to one; the weighted sum of values is the output.

It was never wrong. It was expensive, in two completely different ways, and confusing the two is the fastest route to picking the wrong technique:

Almost every mechanism that follows attacks one of these and not the other. GQA does nothing for compute. FlashAttention does nothing for cache size. Reading them as competing “efficient attention” methods — which a taxonomy actively encourages — gets both of them wrong.

1. The compute panic ended in May 2022, and a kernel ended it

Between April 2019 and September 2020 the field shipped Sparse Transformer, Reformer, Longformer, Linformer, linear attention, BigBird and Performer. Seven separate assaults on T² in eighteen months.

Then FlashAttention landed on 27 May 2022 and made exact attention 2–4× faster with O(T) memory instead of O(T²), by never materialising the score matrix at all. And that entire research line went quiet for roughly three years.

None of those methods was refuted. They were made unnecessary. In a taxonomy FlashAttention looks like a footnote, because it changes precisely zero about the mathematics — it returns bit-for-bit identical results. On a timeline it is the event that explains a three-year hole.

It also has a second-order effect that is easy to miss. A fused attention kernel cannot cheaply express an arbitrary per-pair attention bias. Shaw-style relative position tables are exactly that. RoPE and ALiBi are not — they fuse trivially. So a hardware constraint quietly settled a modelling argument, and the field standardised on the two schemes that happened to be kernel-friendly.

2. The bill people cared about switched in 2023, and you can date the switch

Everything before 2023 attacks FLOPs. Almost everything in 2023 attacks the KV cache: GQA in May, attention sinks in September, Mistral’s sliding window in October.

Nothing about attention itself changed in between. Deployment changed. Once models were serving millions of concurrent conversations, the per-user private cache became the binding constraint, and the whole field pivoted inside about twelve months.

3. Position is a repair cycle, and DroPE is what the end of one looks like

Learned lookup table (May 2017) → sinusoidal (June 2017) → relative (March 2018) → RoPE (April 2021) → Position Interpolation (June 2023) → NTK-aware scaling (June 2023) → YaRN (August 2023) → DroPE (December 2025).

Every entry from Position Interpolation onward is a patch on RoPE. Notice they arrive in a cluster — PI, NTK-aware and YaRN all land inside ten weeks in mid-2023, which is what a field looks like when everybody hits the same wall at the same time.

Then DroPE arrives and argues the patches were the wrong move, because the positional embedding itself was the obstacle: drop it from every layer after pretraining and recalibrate for under 1% of the pretraining budget. That is the shape of a field about to abandon an assumption rather than patch it again.

4. Good ideas sit on a shelf waiting for their engineering

The delta rule — read what memory currently says, compute the difference, write only that — enters linear attention in February 2021. Then nothing happens for three years and four months.

The reason is not that the idea was bad. It is that the update is sequential in its natural form, so it lost the parallel-scan trick that made linear attention trainable at scale. In June 2024 someone recognised the update as a generalised Householder transform and reparameterised it via the WY representation, which turns the chunk recurrence into matmuls. Gated DeltaNet follows six months later. Production deployment six months after that.

In a list, the delta rule looks like a 2024 invention. On a timeline it is a 2021 idea that waited on a kernel — which reframes “can this be trained in parallel” from an implementation detail into a first-class architectural constraint. Linear attention itself has the same shape: proposed 2020, genuinely deployed 2025.

5. February 2025 shows simultaneous invention, which is the strongest signal you get

NSA (DeepSeek, 16 Feb 2025) and MoBA (Moonshot, 18 Feb 2025) are two labs arriving at the same core idea — block-level sparsity trained into the model rather than bolted on at inference — and publishing two days apart.

Independent simultaneous invention means the problem was fully ripe and the pieces were lying in the open. Nothing about an alphabetical list would tell you those two are the same moment. It is the single best predictor on the whole timeline that the direction was about to become mainstream — which it did: DSA that September, then DeepSeek-V4’s compressed sparse attention in April 2026.

6. The field has left recurrent state twice and come back for it twice

Laid out by date, the priorities read: exactness (2014–17) → compute (2018–20) → position gets solved and exact attention gets cheap again (2021–22) → decode memory (2023) → recurrent state returns (2024) → sparsity returns, now trained in (2025) → compression as the primary lever (2026).

Techniques don’t die on this timeline. They wait for the piece they were missing. Fixed-size state was abandoned and reclaimed twice — first with gating added, so it could forget; then with the delta rule added, so it could correct. Sparsity was abandoned after FlashAttention and returned in 2025 as learned, trained-in selection rather than a hand-drawn pattern.

The part I got most wrong before checking

Dates are the easiest thing to get confidently wrong, so I didn’t trust my own recall for any of them. There’s a script in the repo that reads every arXiv ID out of the data file, queries the arXiv API, and asserts the stored date equals the published field — the v1 submission date. 42 of the 46 are machine-verified with zero mismatches; the other four have no paper behind them and are labelled individually rather than smoothed over.

Three traps that caught me:

And one trap that is one character wide

Searching “DroPE arXiv” returns arXiv:2503.15029 as the first hit. That paper is real, and it is not the one you want.

Nine months apart, different field, different mechanism, near-identical name. If you ask a model about DroPE and it answers fluently, this is the failure you should expect: a half-remembered technique with a confidently attached wrong paper.

So what comes next?

Extrapolating the slope rather than claiming knowledge. The 2026 cluster is the interesting datum: inside three weeks, Qwen3.5, GLM-5, Ling 2.5 and MiniMax M2.5 shipped four different attention stacks and all four were competitive. That disagreement is evidence the choice is workload-dependent rather than solved.

Three things the shape suggests. The fixed-size state gives way to a growing-but-sublinear one — Log-Linear Attention is already prototyping O(log T) hierarchical state. The schedule ratio stops being inherited; 3:1 and D-D-D-G are currently copied rather than measured, and nobody has run the ablation. And compression moves earlier still, out of the KV cache and into tokenisation, because the cheapest token remains the one you never spend.

The honest caveat is that this timeline’s own record says the next move often arrives as a kernel rather than an equation, and nobody predicts those.

The interactive version — where the attention demo computes real attention in your browser rather than illustrating it, and every date carries its source — is at attention-in-order.netlify.app. Code, the verification script and the full 46-row source table are on GitHub.