New Runtime synthesis ·
Useful short-term context can be written into model parameters, then used to generate further training.
The week has one shift, and it is about where memory lives. Useful short-term context is moved into model parameters through a consolidation phase, so material that mattered in a session can become part of the weights instead of remaining only as tokens that must be shown again. The same approach then has the model generate a synthetic curriculum and continue improving. Consolidation is not a one-time save; it is the start of further training supplied by the model itself. Anyone still treating context as the only place new useful material can live, and weights as a frozen store, is looking at a method that does the reverse.
The method contains a tension of its own. Consolidation keeps what is called useful, which means some short-term context is written into parameters and some is not. After that write, improvement continues on a curriculum the model generates, so later training depends on what the consolidated model produces, not only on the original context. Holding useful short-term material in the prompt gets less necessary. The model as a stable object gets riskier: the parameter set changes after consolidation, and further gains come from self-generated lessons. Teams that can run and inspect that loop gain a way to keep improving from session material. Buyers who freeze a checkpoint and assume knowledge lives only in prompts lose that assumption when this approach is in use.
A builder or buyer should therefore judge the model after consolidation, not only the model that received the short-term context, because the useful material is meant to be in the parameters by then. The synthetic curriculum is part of what is being bought or shipped, and it should be reviewed with the same care as any other training data. Watch whether the systems in use expose the consolidation phase and the generated curriculum, because those are the steps where memory is written in and where further improvement is supposed to happen.
Key stories
- Language Models Need SleepShows that useful short-term context can leave the prompt and enter model parameters, after which the model generates its own curriculum and keeps improving from it.
Memory
1
- Language Models Need Sleep
A new approach moves useful short-term context into model parameters through a consolidation phase, then has the model generate a synthetic curriculum and continue improving.