---
schema_version: "newruntime-agent-readable-v0.1"
type: "post"
slug: "language-models-need-sleep"
title: "Language Models Need Sleep"
description: "A new approach moves useful short-term context into model parameters through a consolidation phase, then has the model generate a synthetic curriculum and continue improving."
status: "published"
published_at: "2026-07-16"
topics: ["memory","loop-engineering"]
source_urls: ["https://arxiv.org/abs/2606.03979","https://arxiv.org/abs/2605.12978"]
routes: {"html":"https://newruntime.com/posts/language-models-need-sleep/","markdown":"https://newruntime.com/posts/language-models-need-sleep.md","json":"https://newruntime.com/posts/language-models-need-sleep.json"}
source_format: "markdown"
---

# Language Models Need Sleep

A new approach moves useful short-term context into model parameters through a separate consolidation phase. After that, the model generates a synthetic curriculum and continues improving without additional human labeling.

## What changed

The authors of `Language Models Need Sleep` propose treating model learning as a loop with separate wake, consolidation, and dreaming phases. The idea is not to endlessly grow prompt or raw context, but to move selected useful experience from short-term memory into a more durable internal representation.

For agentic systems this matters because a model constantly exposed to new experience needs a controlled mechanism for selection, rechecking, and consolidation. Otherwise useful decisions remain inside dialogue history and never become transferable skill.

## Core mechanism

| Phase | What the model does | Why it matters |
| --- | --- | --- |
| Wake | Solves tasks while new experience remains in short-term context. | Captures episodes, failures, and useful moves. |
| Memory Consolidation | `Knowledge Seeding` moves accumulated knowledge into a larger network through distillation and RL-based imitation. | Preserves useful patterns outside one prompt. |
| Dreaming | Generates synthetic tasks around new knowledge and rehearses it. | Creates a self-improvement curriculum without new human labels. |

### Wake

In the `Wake` phase, the model encounters tasks and gathers new experience. That experience is not long-term memory yet. It lives in context, solution traces, successful attempts, and failures.

### Memory Consolidation

In `Memory Consolidation`, `Knowledge Seeding` moves accumulated knowledge into a larger-capacity network. Distillation and RL-based imitation teach the model to reproduce useful actions and intermediate conclusions that appeared in short-term experience.

```text
episodes -> selected traces -> consolidation -> updated model behavior
```

### Dreaming

In `Dreaming`, the model creates synthetic tasks around the new knowledge. This is rehearsal: the system does not merely store a fact, but applies it repeatedly in new phrasings and nearby situations.

## Important limitation

Automatic consolidation can make memory worse. In the related `Useful Memories Become Faulty When Continuously Updated by LLMs` work, GPT-5.4 lost the solution in 54 percent of previously solved ARC tasks. Agents that kept raw episodes performed twice as well as the forced-consolidation mode.

The problem is not the idea of memory itself. The problem is loss of verifiability. If the system stores only an abstraction and drops the original episodes, it becomes harder to distinguish a correct generalization from a confident mistake.

## New Runtime editorial read

> Memory should store not only the extracted lesson, but also the original episodes that let the lesson be rechecked. Memory without source episodes can turn a bad abstraction into permanent knowledge.

Practical rule for AI products: consolidation should be reversible and checkable. A production system should store not only the final `lesson`, but also provenance: which episodes, metrics, and counterarguments led to it.

## What to carry into engineering

- Separate short-term context, long-term memory, and training artifacts.
- Do not delete raw episodes immediately after extracting a summary.
- Test consolidated memory against old tasks, not only new synthetic ones.
- Keep sources and provenance next to each durable conclusion.
- Make memory rollback as normal as code rollback.

## Primary Sources

1. `Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories` — https://arxiv.org/abs/2606.03979
2. `Useful Memories Become Faulty When Continuously Updated by LLMs` — https://arxiv.org/abs/2605.12978