What happened
Habr’s author says the project needed a 141 GB-or-larger GPU for full fine-tuning of a language model intended to write and answer in Crimean Tatar. The team rented cloud machines rather than buying hardware for a few runs a year. Over about six weeks, GPU rental cost slightly more than $300 across a series of iterations. The first H200 run, at about $4.60 an hour, was budgeted at $33–40 for two stages. After 11.5 hours and $53, the author stopped it with only 15% of the second stage complete.
The largest source of waste was padding. Training dialogues had a median length of about 20 tokens, but each example was padded to 1,024. Although attention and loss masks kept padding from affecting the training result, the GPU still processed those positions. The estimate counted 59 million actual tokens, while the GPU processed 1.17 billion positions, a 20-fold discrepancy. The team changed to packing short dialogues into dense blocks with an attention mask that prevents neighboring examples from attending to one another. Useful-token share rose from about 5% to 99%. The author estimates that the second stage would have cost $331–413 with padding, compared with $14–17 after packing; the padded figure is a calculation, not a completed run.
Checkpointing and orchestration added further costs. The training state was synchronously written to a network volume every 400 steps, at 46 GB per checkpoint. Across the run, that meant roughly 600 GB of writes, and the GPU was occupied only 58–68% of the time even before accounting for padding. The team decided to checkpoint by elapsed time rather than step count, write to local storage first, and move checkpoints to the network volume later. On a 128-core machine, only two data-loader processes prepared batches; the GPU could finish a batch before the next one arrived. The author’s rule of thumb became roughly one loader process per four CPU cores, with utilization checked early.
A later attempt exposed failures in the surrounding scripts rather than in training itself. Five rented machines failed before finishing, including a run stopped by a stale failure marker and a check that needed about 160 GB of memory on a 141 GB GPU. The team began rehearsing the entire launch sequence—including watchdogs and stage transitions—at home with a small model. One 4.7-hour rehearsal found six defects. They also changed the response to errors: repair and restart the process on the same machine instead of destroying it and paying to reinstall the environment and download weights again.
The watchdog, which monitors training, saves results, and shuts down rented machines, also needed changes. The team now tests it with a simulated alert before each run, checks near completion manually, and runs the watchdog from outside the rented machine. Its hang condition combines low average GPU utilization with a step counter that has stopped advancing, so a legitimate checkpoint pause alone does not trigger shutdown. The author reports a subsequent two-stage run costing about $41, compared with the first run’s $53 for only 15% of stage two. The latest training cycles were costing about $10–12.
What the archive adds
New Runtime’s earlier coverage noted that domain-specific speech adaptation can improve technical-term transcription without improving every dimension of general transcription quality (Glas reports fine-tuning GigaAM-v3 for Russian IT dictation, October 22, 2026). This Habr account is a new, related event: it adds operational detail about the cost and reliability of cloud fine-tuning, rather than reporting the same model or result.
Trend signal
This supports the idea that agent and model economics should be judged by useful completed work, not nominal resource prices alone. The author’s estimate shifted when it accounted for actual GPU positions, idle time, setup, storage, and run failures. That is consistent with Agent economics moves to completed work, though the case concerns training-run economics rather than an agent completing tasks.
Why it matters
A cloud GPU’s hourly price is not the cost of a successful training run. Padding can make the hardware perform far more computation than the dataset’s token count suggests; synchronous network writes and slow data loading can leave expensive accelerators waiting; setup, downloads, and disks continue to accrue charges outside the measured training interval. A run can also be destroyed by stale files or a watchdog that mistakes a checkpoint pause for a hang.
The transferable lesson is to instrument the whole operation before scaling it. Sample sequence lengths and estimate useful-token share, rehearse the complete launch chain on cheap hardware, compare early measured throughput with the estimate, and preserve logs and outputs before shutdown. These controls do not guarantee a better model, but they make each experiment cheaper to interpret and less likely to lose work to avoidable orchestration errors.
What happens next
- The team will continue cloud training with lower-cost cycles and a broader data collection effort. (6 months, 75%). Watch for: reported follow-up runs or a new model version, as the author says cloud training will continue.
Limits and uncertainty
This is one team’s retrospective, not an independent audit or a controlled comparison. The author does not name the base model, fine-tuning technique, checkpoints, or data sources, limiting replication and comparison across setups. The $331–413 padded-run cost is an estimate; the team did not rerun the inefficient configuration to verify it. Reported costs and efficiency changes are specific to the machines, provider conditions, workloads, and period described. The article does not establish that packing, checkpoint schedules, or watchdog thresholds will perform identically in other training systems.
Skill to build now
Before renting a GPU for a dataset of short examples, sample tokenized sequence lengths and calculate useful tokens divided by padded positions at the planned maximum length. If the ratio is poor, test packed sequences with an attention mask that prevents examples from mixing. Then rehearse the full launch, checkpoint, watchdog, and shutdown path on a smaller model. Compare the first minutes of actual throughput with the estimate before committing to a long run.
Reader takeaway
Budget for the work the machine will actually do and the full time it will exist—not just the dataset’s nominal token count or the training loop. Packing, end-to-end rehearsal, and cautious watchdog behavior turned avoidable waste into manageable experiments in this account. Treat its figures as a case study, then measure your own pipeline.