---
type: "post"
id: "knowledge-distillation-lets-smaller-language-models-inherit-capabilities-d1d19a"
slug: "knowledge-distillation-lets-smaller-language-models-inherit-capabilities-d1d19a"
title: "Knowledge distillation lets smaller language models inherit capabilities from larger systems, but the standard recipe creates a train-test gap."
description: "Mingyang Song and Mao Zheng have published “A Survey of On-Policy Distillation for Large Language Models” on arXiv."
retrieval_nugget: "Mingyang Song and Mao Zheng have published “A Survey of On-Policy Distillation for Large Language Models” on arXiv."
published_at: "2026-10-24"
updated_at: "2026-10-24"
record_date: "2026-10-24"
date_kind: "published_at"
topics: ["models"]
entities: ["A Survey of On-Policy Distillation for Large Language Models","Mingyang Song","Mao Zheng","arXiv"]
related_patterns: []
novelty_label: "new_fact"
what_changed: "Adds a broad survey framing on-policy distillation, unifying its supervision and optimization approaches and discussing links to capability transfer, efficiency, and deployment."
prior_refs: ["memory:49f320a4-f4d6-8ed5-ac85-f92fc96ac665","memory:f0fcda2e-5803-81e4-ac7c-e7c5ba1bf49a","memory:189005de-dc84-82cb-adcb-9ec9ceb8f2b6","memory:e06ef06f-4516-84b6-adf7-d31e5ca098b3"]
source_url: "https://arxiv.org/abs/2604.00626"
source_urls: ["https://arxiv.org/abs/2604.00626"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:knowledge-distillation-lets-smaller-language-models-inherit-capabilities-d1d19a"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/knowledge-distillation-lets-smaller-language-models-inherit-capabilities-d1d19a/","markdown":"https://newruntime.com/posts/knowledge-distillation-lets-smaller-language-models-inherit-capabilities-d1d19a.md","json":"https://newruntime.com/posts/knowledge-distillation-lets-smaller-language-models-inherit-capabilities-d1d19a.json"}
---

# Knowledge distillation lets smaller language models inherit capabilities from larger systems, but the standard recipe creates a train-test gap.

## Retrieval answer

Mingyang Song and Mao Zheng have published “A Survey of On-Policy Distillation for Large Language Models” on arXiv.

Mingyang Song and Mao Zheng have published “A Survey of On-Policy Distillation for Large Language Models” on arXiv. The paper was first submitted on April 1, 2026 and the listed page identifies the current version as v5, revised October 21, 2026.

The survey examines on-policy distillation: training a student language model with guidance on trajectories generated by the student itself. Its stated target is the mismatch in conventional distillation, where a student trains on fixed teacher demonstrations but, at deployment, must continue from its own earlier outputs and errors.

Song and Zheng organize the area around student-generated states, teacher information, and policy updates. They connect the framing to self-distillation, multiple teachers, reinforcement learning, and interactive agents, and discuss supervision reliability, learnability, capability transfer, training efficiency, and computation allocation.

This is a survey and describes itself as work in progress. The source does not establish that one on-policy method is best, report a new model or a common benchmark result, or provide evidence that the approach improves a particular deployed system.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Models](https://newruntime.com/topics/models.md): Explore the models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)