What happened
In a November 17, 2026 blog post, Sebastian Raschka describes what he sees as a possible new direction in LLM pretraining: dropping the learning-rate decay phase. The alternative he outlines is to train at a constant learning rate and average the last N checkpoints, or use a running average of model weights. He says this may make practical planning easier because a team does not have to commit to the total training duration in advance.
A learning-rate schedule controls how aggressively a model's parameters are updated over a training run. In the conventional approaches Raschka groups together, the rate eventually declines. In the no-decay formulation he describes, the rate remains constant, while the eventual model is produced by averaging checkpoints near the end of training or by maintaining a running weight average. The post presents these as distinct recipe choices, rather than as interchangeable names for the same training procedure.
Raschka frames the post as a “little meta analysis” of training details shared publicly. He groups the recipes into three categories. The first is classic cosine decay, which he lists for Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, and Phi-4. This category is the clearest reminder that a decaying schedule remains common among the systems he surveyed.
The second category is warmup-stable-decay (WSD), which he lists for DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, and MiniMax M2. The label describes a schedule with a warmup phase, a stable period, and a decay phase. Raschka does not claim that the systems in either of these first two groups used identical data, architecture, batch size, token budget, or evaluation procedure.
The third category is “no decay”: a constant learning rate combined with checkpoint averaging. Raschka names Kolibri-1 and DatologyAI’s models as examples. The post points readers to architecture entries in his LLM gallery for Kolibri-1 and DatologyAI, and says the website version is lightly edited from an X post quoting DatologyAI’s note on weight averaging.
The distinction is about both the training schedule and how a final model is formed from checkpoints or weights. In Raschka’s account, the constant-rate recipe avoids committing to a fixed endpoint in advance. That flexibility is the operational argument in the post: a team may be able to continue training without having designed a decay phase around a predetermined finish. His post identifies examples and a potential trend; it does not report a controlled head-to-head experiment, quantitative comparison, or performance result for the approaches. It also does not make a claim that cosine decay or WSD is obsolete.
What the archive adds
New Runtime’s earlier coverage on September 24, 2026, noted that MiMo-V2.6 Pro’s training highlighted agent-oriented post-training. The November 16 note on PAN 2’s public training code concerned which training stages were disclosed. Raschka adds a distinct pretraining-schedule signal, not a restatement of either story.
Trend signal
This does not yet fit a pattern New Runtime tracks. Raschka’s examples point to variation in pretraining schedules, but a small observational grouping is not enough to establish a durable shift in how frontier training is designed or evaluated.
Why it matters
Learning-rate schedules are a consequential training choice, and the stated operational appeal here is straightforward: constant learning rates with averaging may reduce the need to decide the full training duration in advance. That could matter to teams whose training runs are costly or whose stopping point is difficult to predict, although Raschka does not quantify savings or describe a particular team’s operational results.
The practical implication is not simply that a team can omit a final decay phase. It must also define how the final parameters will be selected or averaged. Raschka mentions either the last N checkpoints or a running average, and those are separate implementation choices with potentially different operational behavior. A training team considering the recipe should make the window, checkpoint cadence, and weight-averaging method explicit before comparing results. Otherwise, a claimed comparison would mix a schedule decision with an unclear model-selection procedure.
The post is best read as a prompt for an experiment, not a recommendation to replace a proven recipe. A team can compare a constant-rate run with averaging against its current schedule while keeping the data, architecture, compute budget, evaluation set, and checkpoint-selection procedure comparable. It should also decide before the run what result would justify a change: final quality, stability, flexibility to extend training, or a combination of those measures. Without that controlled comparison, the recipe’s practical value for a particular model remains unknown.
Limits and uncertainty
Raschka’s post is an informal analysis of training details that others have shared, not a controlled study. It does not provide comparable benchmark results, training budgets, model sizes, or enough information to determine whether the listed systems used otherwise similar procedures. It also does not quantify how checkpoint averaging affects stability, final quality, or compute cost.
The claimed planning benefit is presented as a practical possibility, not measured evidence that teams can shorten runs, extend them safely, or improve results. The post does not state the number of checkpoints averaged for each example, the running-average configuration where one was used, or the criteria by which a training run should stop. The examples show that the recipe has been used; they do not establish its prevalence, reproducibility, or superiority over cosine decay or WSD.
Skill to build now
Write a small training-run comparison plan before changing a production recipe. Hold the dataset, model configuration, token budget, and evaluation suite fixed; run the current schedule alongside a constant-rate alternative; and record the checkpoint interval plus either the final-N averaging window or running-average rule. Use the approach only where the team genuinely needs flexibility about the stopping point. The result will separate the appeal of a more open-ended run from any measured effect on the final model.
Reader takeaway
Constant learning rate plus checkpoint or weight averaging is a recipe worth evaluating, not yet a demonstrated replacement for cosine decay or WSD. Its potential advantage is planning flexibility when the endpoint of a training run is uncertain. The useful next step for a training team is a controlled comparison under its own compute and evaluation constraints, with the averaging method and checkpoint window recorded explicitly.