---
type: "post"
slug: "full-bandwidth-transformers"
title: "Full-bandwidth Transformers"
description: "A preprint widens the narrow vertical channel between decoding steps by feeding the previous top-layer hidden state back in, letting non-verbalized computation re-enter the stack."
retrieval_nugget: "A preprint widens the narrow vertical channel between decoding steps by feeding the previous top-layer hidden state back in, letting non-verbalized computation re-enter the stack."
published_at: "2026-09-01"
updated_at: "2026-09-12"
record_date: "2026-08-25"
date_kind: "discovered_at"
topics: ["agent-security","agents","ai"]
entities: ["arXiv"]
source_url: "https://arxiv.org/abs/2608.08888"
source_title: "Full-bandwidth transformer"
source_domain: "arxiv.org"
source_terms: ["Full-bandwidth","Transformers","feedback","transformer","latent"]
summary_word_count: 183
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:full-bandwidth-transformers"
status: "published"
source_urls: ["https://arxiv.org/abs/2608.08888"]
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:8c465a1ca74cb36a643c8e486bdb0a5bc78fba374fc530b7fc9b2c4577127215","reviewed_at":"2026-09-12T10:00:00Z","source_evidence_count":1,"verified_claim_count":2,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":2,"implication_count":1,"watch_condition_count":1,"related_record_count":2}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"A preprint submitted on 9 August points at a structural asymmetry in autoregressive transformers.","observed_facts":[{"text":"The paper describes the vertical feedback channel between decoding steps as narrow because only the sampled token returns to the bottom of the stack.","source_urls":["https://arxiv.org/abs/2608.08888"]},{"text":"Latent feedback fuses the previous top-layer hidden state with the sampled token embedding through a gated linear unit.","source_urls":["https://arxiv.org/abs/2608.08888"]}],"mechanism":"at each decoding step the previous top-layer hidden state is fused with the sampled token embedding through a gated linear unit and fed back as the next input","why_now":"Only the sampled token returns to the bottom of the stack; the top-layer hidden state is discarded.","implications":["In other words, whatever the model worked out but did not put into a token stops being thrown away at each step."],"evidence_boundary":"This is an abstract: the mechanism is described without the evaluation results, scale, or training cost that would establish whether the wider channel pays for itself.","watch_conditions":["The condition to watch is whether latent feedback survives at frontier scale under a published ablation, because a decoding-time change that helps small models often disappears when depth is already abundant."],"related_records":[{"url":"https://newruntime.com/posts/liquid-ai-releases-cpu-friendly-long-context-encoders","relation":"Prior coverage of carrying more usable state per unit of compute."},{"url":"https://newruntime.com/topics/ai","relation":"Topic hub for model architecture results."}]}
routes: {"html":"https://newruntime.com/posts/full-bandwidth-transformers/","markdown":"https://newruntime.com/posts/full-bandwidth-transformers.md","json":"https://newruntime.com/posts/full-bandwidth-transformers.json"}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
---

# Full-bandwidth Transformers

## Retrieval answer

A preprint widens the narrow vertical channel between decoding steps by feeding the previous top-layer hidden state back in, letting non-verbalized computation re-enter the stack.

A preprint submitted on 9 August points at a structural asymmetry in autoregressive transformers. They compute along two axes: horizontally across generated tokens, where dense attention gives each token broad access to the past, and vertically through model depth, where the feedback channel between decoding steps is narrow. Only the sampled token returns to the bottom of the stack; the top-layer hidden state is discarded.

The proposed full-bandwidth transformer widens that channel with latent feedback: at each decoding step the previous top-layer hidden state is fused with the sampled token embedding through a gated linear unit and fed back as the next input. The consequence the authors emphasise is that non-verbalized computation can re-enter the stack with a renewed depth budget, while the standard transformer shape is preserved. In other words, whatever the model worked out but did not put into a token stops being thrown away at each step.

That is a mechanism-level version of the question this site has tracked at the systems level in [Liquid AI's long-context encoders designed for CPUs](https://newruntime.com/posts/liquid-ai-releases-cpu-friendly-long-context-encoders/): how much useful state a model can carry per unit of compute, and where it is allowed to keep it. The [AI hub](https://newruntime.com/topics/ai/) follows the architecture side.

This is an abstract: the mechanism is described without the evaluation results, scale, or training cost that would establish whether the wider channel pays for itself. The condition to watch is whether latent feedback survives at frontier scale under a published ablation, because a decoding-time change that helps small models often disappears when depth is already abundant.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Agent Security](https://newruntime.com/topics/agent-security.md): Explore the agent-security topic hub.
- topic hub: [Agents](https://newruntime.com/topics/agents.md): Explore the agents topic hub.
- topic hub: [Ai](https://newruntime.com/topics/ai.md): Explore the ai topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)