---
type: analysis
slug: research-blockmax-fits-a-7b-full-fine-tune-in-19-3-gb-but-quality-gains-are-tiny-and
title: "BlockMax fits a 7B full fine-tune in 19.3 GB, but quality gains are tiny and unreproduced"
description: "A one-author preprint and MIT-licensed repo fully fine-tune Qwen2.5-Coder-7B on one 24 GB A10G with a momentum-free optimizer that stores 2.25 bytes per parameter. The 7B run has no baseline, the evidence covers one Python SFT task, and as of 27 October 2026 no one else has reproduced it."
published_at: 2026-11-09T01:00:00.000Z
topics: ["memory","research"]
source_urls: ["https://github.com/marchfederico/blockmax-finetune","https://github.com/marchfederico/blockmax-finetune/blob/main/PACKED_BLOCKMAX_RECIPE.md","https://arxiv.org/abs/2310.10195","https://arxiv.org/abs/2010.06192","https://zenodo.org/doi/10.5281/zenodo.23679887"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
BlockMax is a small open-source project by Marcello Federico. It comes with a preprint titled "Stochastic Rounding, Not Momentum: Full Fine-Tuning at 2.25 Bytes per Parameter." The method drops Adam's momentum and stores only bf16 weights plus one fp32 scale per 16-weight block. It updates each tensor as soon as that tensor's gradient is complete, then frees the gradient, and writes weights back with stochastic rounding. The headline claim is a full-parameter fine-tune of Qwen2.5-Coder-7B on a single 24 GB A10G at 19.3 GB peak. Standard mixed-precision AdamW needs roughly 16 bytes per parameter before activations.

Builders care because full fine-tuning on one consumer- or cloud-class 24 GB card is normally the job of LoRA/QLoRA. A cheap full-parameter alternative would change cost calculations for small-team SFT. The idea is not new, though. It builds directly on LOMO/AdaLomo, which fuse updates into the backward pass and argued that an adaptive learning rate matters more than momentum. It also builds on earlier work showing that nearest rounding cancels small bf16 updates and that stochastic rounding fixes this. The new part is a specific packing and a specific claim: stochastic rounding, not momentum, is the key ingredient.

## What we checked

- **Can independent teams reproduce the 7B memory and held-out-loss results across tasks, models, and practical training workloads?** unresolved. I found no independent reproduction, issue thread or third-party benchmark. The Zenodo preprint was published on 26 October 2026 and listed as v1. The repo shows 4 commits and 0 stars/forks when fetched. All results come from the author, on one Python code SFT task, one GPU type (A10G) and one epoch. [1][5]
- **(Added) Is the 2.25 bytes/parameter and 19.3 GB peak for 7B internally consistent?** partly confirmed. The arithmetic holds: bf16 weights (2 bytes) plus one fp32 scale per 16 weights (4/16 = 0.25 bytes) gives 2.25 bytes/param. For about 7B parameters that is roughly 16 GB before activations, which fits under the reported 19.3 GB peak with gradient checkpointing. The peak itself is self-reported, and sequence length and batch size for the 7B run are not stated in the recipe. [1][2]
- **(Added) Does BlockMax clearly beat existing low-memory full fine-tuning optimizers on quality or memory?** partly confirmed. BlockMax beats or ties AdaLomo and FlashAdamW on held-out loss, but by margins of about 0.0001 to 0.004. The recipe says the gain over FlashAdamW is not significant at every seed and that pass rates did not separate any setup. At 1.5B and 1.7B, AdaLomo used slightly less peak memory (8.3 GB vs 8.7/8.6 GB). The 7B run has no baseline at all. [1][2]

## What we found

- The optimizer state is minimal by design: bf16 weights plus one fp32 scale per 16 weights, or 2.25 bytes/param. Each tensor is updated as soon as its gradient is complete, and the gradient is then freed. The repo is MIT-licensed, includes an AdaLomo reference derived from OpenLMLab's lomo-optim, and pins torch 2.14, triton 3.8 and transformers 5.17 on CUDA. [1]
- The 7B evidence is a feasibility run only. Qwen2.5-Coder-7B with gradient checkpointing peaked at 19.3 GB and took about 25 minutes per run on an A10G at learning rate 8e-6. Held-out loss fell from 0.2759 to a 3-seed mean of 0.1526 (seeds: 0.1512/0.1532/0.1533). The recipe states that no other optimizer was run at 7B, so the run shows that training works on the card, not that BlockMax beats an alternative. [2]
- The comparison benchmark is narrow. It uses Python SFT on 1,865 verified programs from DeepMind CodeContests for one epoch, with held-out loss measured on 288 programs from 50 unseen problems, 3 seeds and per-method tuned learning rates. On Qwen2.5-Coder-1.5B, losses were 0.1191 (BlockMax), 0.1192 (AdaLomo+SR), 0.1195 (AdaLomo) and 0.1194 (FlashAdamW). On SmolLM2-1.7B they were 0.1848, 0.1855, 0.1875 and 0.1884. The recipe says pass rates (300 problems × 8 samples) did not separate any of the setups. [1][2]
- BlockMax does not win on memory at small scale. At 1.5B/1.7B it peaked at 8.7/8.6 GB, versus 8.3 GB for AdaLomo and about 15.3 to 15.7 GB for FlashAdamW. At 3B with checkpointing it used 14.9 GB versus 16.7 GB for AdaLomo+SR. Its clear memory advantage is over AdamW-style optimizers, not over AdaLomo. [1]
- The ablations back the paper's title more than its method name. Adding stochastic rounding to AdaLomo closed most of its gap (Qwen 0.1195 to 0.1192; SmolLM2 0.1875 to 0.1855). Deterministic rounding in the Flash-BlockMax variant gave 0.1274 versus about 0.119 with stochastic rounding. Removing momentum barely changed results, and Adam-mini's mean-of-squares statistic matched the block max once tuned. The best learning rate differed by about 20x between Qwen and SmolLM2. [2][1]
- The core ingredients have precedents. AdaLomo (ACL Findings 2024) argued that an adaptive learning rate, not momentum, is what closes the gap between LOMO and AdamW. Zamirai et al.'s 'Revisiting BFloat16 Training' showed that nearest rounding cancels small weight updates and that stochastic rounding or Kahan summation restores 32-bit-level accuracy in pure 16-bit training. BlockMax looks like a careful combination and packing of these ideas rather than a new mechanism. [3][4]
- The paper is a v1 preprint on Zenodo, published 26 October 2026 under CC BY 4.0. Its only file is a 70.1 kB PDF, and the page shows zero views or downloads, with no sign of peer review. Searches turned up no independent coverage, reproduction or discussion of the repo or paper title. [5][1]

## What it means

Treat BlockMax as a credible, cheap experiment, not a proven replacement for AdamW or LoRA. The memory claim follows from the state layout and is easy to check: on a 24 GB card, run the repo's 7B script with your own sequence length and batch size and watch the peak. Sequence length and batch size are the variables the 7B write-up leaves out, and activations, not optimizer state, will dominate there. The quality claim is weaker. Every comparison sits on one small Python SFT set scored by held-out loss, where the margins are in the third or fourth decimal place and pass rates show no difference. The 7B run has no comparison at all.

For practitioners, the most portable takeaway is probably the ablation result, not the BlockMax packing itself. If you train with bf16 weights and no fp32 master copy, as in LOMO-style fused updates, use stochastic rounding. Without it, small updates vanish. If you only need around 24 GB at 7B, AdaLomo with stochastic rounding is a nearby, already published alternative worth benchmarking side by side. Before committing, test on your real data mix and on downstream metrics beyond loss: instruction following, math, and multi-epoch runs. Budget for a learning-rate sweep, since the optimal rate shifted about 20x between model families.

## Still open

- Will independent teams reproduce the 19.3 GB peak and loss curve for 7B, and at what sequence length and batch size?
- How does BlockMax compare at 7B against AdaLomo+SR, 8-bit AdamW with CPU offload, or QLoRA/LoRA on downstream task accuracy, not just held-out loss?
- Does momentum-free streaming hold up for longer, multi-epoch runs, non-code domains (chat, math, multilingual) and larger models?
- Is the method compatible with gradient accumulation, multi-GPU sharding and common trainers, given that each tensor is updated as soon as its gradient is ready?

## Sources

1. [marchfederico/blockmax-finetune (GitHub repository)](https://github.com/marchfederico/blockmax-finetune) (primary)
2. [PACKED_BLOCKMAX_RECIPE.md: full tables and ablations](https://github.com/marchfederico/blockmax-finetune/blob/main/PACKED_BLOCKMAX_RECIPE.md) (primary)
3. [AdaLomo: Low-memory Optimization with Adaptive Learning Rate](https://arxiv.org/abs/2310.10195)
4. [Revisiting BFloat16 Training](https://arxiv.org/abs/2010.06192)
5. [Stochastic Rounding, Not Momentum: Full Fine-Tuning at 2.25 Bytes per Parameter (Zenodo preprint)](https://zenodo.org/doi/10.5281/zenodo.23679887) (primary)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Memory](https://newruntime.com/topics/memory.md): Explore the memory topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)