New branch reason: No prior New Runtime record was supplied; this starts a distinct record for self-play as a data-generation and curriculum-search mechanism in pretraining.
Thesis
This paper reframes pretraining as an adaptive search for useful synthetic experience rather than consumption of a fixed curated corpus.
Observed facts
The authors train a generator and learner from random initialization; the generator proposes programs interpreted by a universal Turing machine, and the resulting byte sequences become the learner’s autoregressive prediction task.
1They report predictable improvement in zero-shot loss on several natural datasets as self-play compute rises, even though neither model is trained on natural data.
1The paper reports in-context learning and recognizable mathematical sequences emerging during training.
1
Mechanism
The learner is optimized with standard cross-entropy, while the generator is trained with reinforcement learning to produce sequences at the frontier of the learner’s current capabilities. This creates an adaptive curriculum: generator reward is tied to finding structure that is neither trivial nor currently out of reach.
Why now
As curated high-quality data becomes a visible constraint in scaling discussions, a measurable transfer result from a zero-natural-data setup turns synthetic-data generation from a static augmentation tactic into a trainable search problem.
Implications
If this behavior survives larger scales, training budgets may increasingly be allocated to curriculum-search systems and compute rather than only to corpus acquisition.
Benchmark transfer becomes the crucial test, because internally generated progress alone would not establish usefulness outside the generator’s distribution.
Evidence boundary
These are author-reported results from an initial arXiv proof-of-concept; the abstract does not establish broad capability gains, competitive performance against conventional pretraining, or independent replication.
Watch conditions
Whether larger experiments retain predictable natural-data transfer and whether comparisons against equally sized conventional or synthetic-data baselines show a practical advantage will determine whether this is a new pretraining path rather than an elegant toy regime.
Whether independent groups reproduce the reported compute scaling and transfer to natural data.