The authors argue that this behavior is intrinsic to the Transformer architecture rather than solely a learned effect. They report that pretraining tends to reduce the effect, while lightweight fine-tuning can restore it. The paper also introduces guided decoding intended to disentangle the combined output and generate two coherent continuations from one forward pass.
Raw SignalSep 25, 20261 min read1 source
Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs
In one minute
A paper published on September 24 reports evidence for “Superposition Linearity” in LLMs: linearly combining representations from unrelated text streams can yield a next-token distribution that superposes the streams’ individual predictions.