Across three heterogeneous teacher–student pairs on mathematical reasoning and code generation, strict one-to-one groups covered most student-generated tokens. The paper reports that shared vocabulary retained nearly all teacher and student probability mass at strictly aligned positions on average. A student-selected top-16 subset at those positions achieved accuracy comparable to full shared-vocabulary OPD and better than the evaluated cross-tokenizer baselines.
Adding span log-probability supervision in mismatch groups produced gradients with weak or negative directional agreement with strict-loss gradients, potentially introducing conflicting training signals. The paper makes an empirical case against treating coverage as a sufficient proxy for useful training information in cross-tokenizer OPD. In the evaluated settings, compact supervision at strict positions can perform as well as or better than broader coverage. These are results reported in one arXiv paper across three heterogeneous pairs and math and code-generation tasks; they do not establish that strict-only supervision is preferable across all architectures, datasets, or training regimes. Replication across additional model pairs and tasks would show whether the reported gradient conflict and accuracy trade-off generalize.
The paper is a distinct analytical record because no prior New Runtime record was supplied for this specific cross-tokenizer distillation result.