What happened
Wuschel Schulz and tvb describe a research project conducted as part of MARS 5 and the Meridian Visiting Researcher Programme in a LessWrong research update dated November 11, 2026. They investigate “filler-token reasoning”: a model uses extra token positions to process a task, but the words at those positions do not convey the intermediate reasoning. The authors distinguish this from open reasoning, where intermediate results are written and reused, and steganographic reasoning, where reasoning is present in text but concealed in innocuous-looking wording.
Their central change is in the training setup. Earlier work trained models to emit dots after a problem and then produce an answer, mixing this data with examples of open reasoning. Schulz and tvb instead use dots as the input and train the model to predict an open-reasoning step at each dot position. At inference, the model is prefixed with dots; without that setup, it would reason openly. They train rank-32 LoRAs on two existing models, Qwen3.6-35B-A3B and gpt-oss-20b, rather than training models from scratch.
The researchers use synthetic tasks with generated prompts, step-by-step solutions, and answers. These include modular sums, tracking a ball through cup shuffles, state changes in a three-lamp system, modular arithmetic, category inference from rules, Horn-clause derivations, and function composition. They compare their approach with the earlier dot-training method and an answer-only baseline. In the reported curriculum experiments, each stage uses 2,000 training examples; the curriculum increases task length from 10 to 20 to 40 steps after accuracy reaches 0.9. They use at least three random seeds for each model/task combination in this comparison.
Their method enables filler-token reasoning on three model-task combinations: modular sum and GL(3,2) on Qwen3.6-35B-A3B, and modular sum on gpt-oss-20b. Across the broader set of seven tasks, the authors report that learnability varies substantially. Some tasks can be learned for many steps, while modular arithmetic and function composition encounter empirical limits at one to three steps in their tests. They say large information packets, such as lookup tables or premises, appear to make learning harder. In some cases models learned up to 60 steps, but this is not a general capability claim for arbitrary tasks.
The post also offers a theoretical analysis of the layer depth needed to solve each task with filler tokens. The authors construct algorithms under a bounded-precision transformer model and treat the resulting depths as upper bounds, not minimum requirements. They report no fully parallel algorithm for any studied task; for most tasks, their constructed depth grows more slowly than the number of reasoning steps. They caution that assumptions in the analysis, including a 32-bit lookup-table limit for an MLP, affect the curves. They make no claim that the training procedure is a standard part of an LLM pipeline.
What the archive adds
New Runtime’s November 5, 2026 note, “Language models can spend too much effort on easy questions and too little on steps that need checking”, considered how models allocate reasoning time. This LessWrong update is a new fact on the broader reasoning story: it shows that some multi-step computation can occur without visible intermediate words, rather than addressing only how much time a model spends deliberating.
Trend signal
This does not yet fit a pattern New Runtime tracks. The result raises questions about interpretability and evaluation, but it is a small study of synthetic tasks and non-standard fine-tuning, not evidence of a deployed agent architecture or a broad shift in how production systems are built.
Why now
The authors connect the work to research on steganographic reasoning and to concern that models may answer reasoning questions without visibly reasoning about them. Their experiments are intended partly to separate hidden-in-text reasoning from computation that uses filler positions. That distinction matters when evaluating whether a transcript reveals the process behind an answer.
Why it matters
For teams using language models, a chain-of-thought transcript is an observable output, not a guaranteed trace of every computation that produced the final answer. A model that writes irrelevant text before a correct answer may be using hidden state across token positions; conversely, a plausible explanation does not establish that the explanation caused or faithfully records the answer. The study does not show that commercial systems commonly use this behavior, but it weakens a simple assumption that no meaningful computation happens when the visible reasoning is absent.
This has practical implications for evaluation and oversight. If a workflow depends on checking intermediate steps, teams should verify the actual task outcome and use tests that expose errors, rather than treating a verbose rationale as proof. Where hidden intermediate computation would be consequential, evaluations should test behavior under varied task lengths and formats, and compare answers against independently checkable results. The paper’s task-specific differences also argue against extrapolating from one benchmark: success on modular sums does not establish success on rule chains or real-world planning.
What happens next
- Follow-up studies will test filler-token reasoning on less synthetic and more varied tasks. (12 months, 60%). Watch for: comparisons with standard training pipelines and independently reproduced results.
Limits and uncertainty
The authors characterize the project as a demonstration using small models and training runs, and the procedure is non-standard. Its tasks are synthetic; the reported findings do not establish how often filler-token reasoning occurs in deployed systems, whether it transfers to naturalistic work, or how robust it is outside the tested fine-tuning setup. The update reports results for selected experiments, not a general survey of model behavior. The theoretical analysis depends partly on choices made to bound transformer operations, and its algorithms provide upper bounds rather than exact minimum depth. The work also does not establish that visible chain-of-thought is useless: it shows only that visible words need not capture all multi-step computation in these experiments.
Reader takeaway
A model can learn some multi-step computations while emitting tokens that do not expose its intermediate reasoning. Treat visible chain of thought as an output to evaluate, not as a complete audit trail: check answers against independent criteria, and avoid generalizing from success on one synthetic task to reasoning ability overall.