Yifan Zhang, Jichen Feng and Shihan Qin released a report and Apache-2.0 repository for the Recurrent Looped Transformer, dated September 12, 2026 and updated September 17. The decoder reads encoder-derived global KV memory and keeps a sliding-window attention cache at every layer.
The Recurrent Looped Transformer adds depth through recurrence across tokens rather than by stacking more layers per token. RLT-1 passes the decoder's final hidden state to the next token together with that token's causal encoder representation, so the recurrent path grows with sequence length while the blocks evaluated per token stay fixed. Looped and recurrent designs are being revisited as a way to get more computation per parameter without widening or deepening the model.
The September 17 snapshot covers 108 runs of 2,000 optimizer steps across six algorithmic tasks and three seeds. At step 500, RLT-1 6+2 reached 99.44 ± 0.98% parity accuracy, compared with 48.48 ± 0.53% for an eight-layer Transformer. On standard S5, every model stayed below 3%.
The results come from models of roughly 25 to 29 million parameters trained on six algorithmic tasks, so they do not establish gains for language modeling or at larger scale. The chunked RLT-2 variant is only proposed, with no training or throughput results. Architecture researchers can use the matched Transformer baseline and three-seed reporting to check whether the parity advantage survives replication. Replications on language modeling tasks, and any throughput results for the proposed chunked RLT-2 variant, would show whether the design holds up.