- Published on
The Transformer is turning into a component
- Authors

- Name
- AbnAsia.org
- @steven_n_t
Listen to this article · The Transformer is turning into a component
0:00 / 0:00
Watch this article · The Transformer is turning into a component
Spend time with the people building frontier models and a pattern shows up that the leaderboards hide: an unusual number of them are working on something that is not a Transformer.
Not replacing it this quarter. Working on it as a live research question, in parallel, with real money. JEPA-style predictive architectures that never generate a token or a pixel. Diffusion language models that denoise a whole paragraph at once. Mamba and the state-space family, which replace attention's quadratic scan with a fixed-size recurrent state. Each is a different answer to the same question: what should the machine actually compute?
What actually shipped
The state-space argument has run since Mamba in 2023. The honest scorecard: linear-time mixing won throughput and memory, lost recall on its own, and the industry responded by keeping a minority of attention layers.
Nobody shipped a pure recurrent model above mid-size. What shipped is a ratio. A matched study of hybrid linear attention put the useful band at 3:1 to 6:1 linear-to-full, where hybrids approach or exceed a full-attention baseline on recall benchmarks at four to seven times less KV cache. Kimi K3's report lands its own ablation at 3:1 — 5.65 validation perplexity against 5.66 at 1:1 — across 93 text layers, 24 of them full attention. NVIDIA's Nemotron-H reports up to roughly three times faster inference at matched accuracy; IBM's Granite 4.0-H reports more than 70 percent lower memory. Both are vendor numbers on vendor stacks, and both are structurally unsurprising: you removed the thing that grows.
State space · Mamba, RWKV, KDA
Fixed-size recurrent state, linear compute in sequence length. Ships as a hybrid at roughly 3:1.
Diffusion · Mercury, LLaDA
Denoise many tokens at once instead of one at a time. Advertised 5–10× faster inference.
Joint embedding · JEPA
Predict in a latent space rather than generating pixels or tokens. LeCun left Meta and founded AMI Labs in 2026.
The unchanged default
The Transformer, still the base of every model above — just no longer the whole of it.
The counter-case matters. MiniMax ran the linear-hybrid play at 456B and reversed in public, shipping full attention with a postmortem: parity at development scale, multi-hop deficits at larger scale. A June 2026 paper found chain-of-thought fine-tuning degrades long-range recall in hybrids. Recall is not a property an architecture has once; it is a property a checkpoint has, after everything you did to it.
Diffusion stopped being a curiosity
Diffusion language models invert the usual contract. Instead of emitting token after token, they start from noise and denoise a whole block in parallel, iterating until the text resolves. Inception's Mercury was the first commercial-scale version, advertised at five to ten times the inference speed of comparable autoregressive models; by 2026 the hosted generation is passing a thousand tokens per second, with LLaDA 2 at 70B and Google's Gemini Diffusion in the same family.
The efficiency claim is about inference, not training, and that distinction gets lost. Faster decoding changes the economics of serving and of agent loops. It does not by itself make a model smarter per unit of training compute.


The compute question, which is the real question
Here is where the argument gets concrete enough to test. HRM-Text, a 1B model from a group including Yasin Abbasi Yadkori, replaces the standard Transformer with a hierarchical recurrent model that splits computation into a slow strategic layer and a fast execution layer. Trained from scratch on 40 billion tokens for a $1,500 budget, it reports 60.7 on MMLU, 84.5 on GSM8K and 56.2 on MATH — competitive with open models two to seven times larger, at roughly 96 to 432 times less estimated compute than the standard baselines.
Read that carefully. It is a 1B model, not a frontier model, and "estimated" is doing work in the sentence. But it is an existence proof of the specific thing that matters: a different architecture plus a different training objective produced a radically better ratio of capability to compute.
times less training compute than standard baselines
times less, per the paper’s upper bound
The frontier-scale version of the same direction is quieter. Kimi K3's report includes a scaling fit where its attention variant reaches equal loss at roughly 1.16 times less compute — a 14 percent saving, not a revolution, and extrapolated from a research testbed rather than measured at 2.8 trillion parameters. That is what progress usually looks like before it looks like a breakthrough.
Why a breakthrough is the reasonable bet
The case is not that any single architecture is about to win. It is that the search space has widened while the measurement got sharper.
For a decade there was one architecture and one recipe, so capability came from more compute and more data. Now there are four families in production or serious research, each with a different failure mode: hybrids lose exact recall, diffusion loses the left-to-right contract, JEPA loses the interface of generated text, the Transformer loses the memory argument. Different failure modes mean different niches, and niches mean parallel funding.
Nothing here suggests total training compute falls — the industry's own plans point the other way. The claim worth making is narrower and more useful: the amount of intelligence returned per unit of compute is no longer improving only because the models are bigger. It is improving because the shape of the computation is changing — and that vector has barely been explored compared with how long it has been since anyone questioned the architecture itself.
None of this has a date on it. What is checkable today: hybrids ship at frontier scale, diffusion decodes at a thousand tokens a second, latent-prediction research has a billion-dollar lab behind it, and a 1B model undercut standard training compute by two orders of magnitude. If the axis keeps moving at the rate of the last eighteen months, the question stops being whether a breakthrough arrives and becomes which family delivers it.
Source: arXiv 2605.20613 (HRM-Text); Kimi K3 technical report (July 2026); "State Space Models in 2026: The Recall Gap" (August 2026), citing arXiv 2507.06457, 2603.15569 and 2606.11052; Inception Labs' Mercury material; state-space and JEPA literature to November 2026.
Contact: [email protected]