Thesis
Thesis: Superposition linearity may be an architectural property of Transformers that can be measured, altered by training, and exploited for multi-stream decoding.
Observed facts
Observed facts: the authors report a superposition of individual next-token distributions after combining inputs, and they report that the effect diminishes during pretraining.
1The authors report that lightweight fine-tuning can substantially restore linearity and introduce guided decoding to separate the superposed output into two continuations.
1
Mechanism
Mechanism: Linear mixing of two input representations is reported to produce a mixed next-token distribution, which guided decoding then attempts to disentangle into two continuations.
Why now
Why now: The work connects an architectural observation to a proposed single-forward-pass decoding procedure at a time when inference efficiency remains a central constraint.
Implications
The first implication is that representation-level multiplexing could become an additional design space for inference research, rather than merely a mechanistic-interpretability curiosity.
If reproducible at useful quality, the finding could motivate new serving methods that trade decoding complexity for fewer forward passes.
Evidence boundary
Evidence boundary: These are claims from one paper’s reported experiments, not evidence that production LLM serving can safely double throughput.
Watch conditions
The first watch condition is independent evaluations showing that guided decoding preserves task quality while delivering net latency or cost gains.
Replication across model families, sequence lengths, and practical serving stacks would establish whether the effect generalizes.