Field noteNov 16, 20265 min read1 source

Quark: our whole training stack, end to end, for $4,200

In one minute

Empero says it trained and released Quark, a roughly 165-million-parameter language model, through a full training pipeline for about $4,200 in compute. The four Apache-2.0 checkpoints and detailed account make this useful to teams assessing what a small-budget model experiment can test—and what its results do not establish.

What changed: Empero released Quark, a 165M-parameter model trained end to end for about $4,200, with 128K context, an RL-only version, and four Apache-2.0 checkpoints.

What happened

Empero describes Quark as a proof that its training stack works before the lab spends more on a larger model. The model has about 165 million parameters, or 140 million without embeddings, and supports contexts up to 128,000 tokens. Empero says the end-to-end effort cost about $4,200 of compute, equivalent to roughly 1,300 H100-hours. That figure includes a discarded first design, failed attempts, data generation, and evaluation; the company says some compute was sponsored by lium.io. All four checkpoints are released under Apache-2.0.

The checkpoints mark successive stages. Quark-150M-32K-Raw is the pre-anneal checkpoint after 472 billion tokens. Quark-150M-32K-Base follows a 28-billion-token anneal at 32K context. Quark-150M-128K-Base follows another 10 billion tokens of long-context training. Quark-150M-128K-Zero starts from that 128K base and receives reinforcement learning. The pretraining run used a single node with eight H100 GPUs and 500 billion tokens: a short warmup at 1K context, 470 billion tokens at 4K context, then the anneal at 32K. The data was mostly English web text, math, and code, alongside multilingual text, reasoning traces with verified answers, and synthetic long-context documents.

The architecture uses 16 layers. Twelve attend locally over the previous 256 tokens and include a short convolution; every fourth layer attends globally. Empero says this means only four global layers retain a full key-value cache, about 0.5 GiB in bf16 at 128K context, versus 2 GiB if every layer were global. For context extension, the team trained on another 10 billion tokens of 128K sequences, including books, papers, long documents, instruction-following examples, and synthetic tool-use sessions.

Empero trained the Zero checkpoint with GRPO, without supervised fine-tuning or distillation. About 95% of rewards came from code tasks, using exact answers, format rules, unit tests, and file checks in real sandboxes; LLM judges scored the remainder. The run took under 1,300 steps and about 32 hours, and Empero says it stopped early to stay within budget while scores were still rising. In training rollouts on simple terminal tasks across 15 emulated coding-agent harnesses, the share solved rose from 3% before RL to 65% after it. This is not a held-out agent benchmark. The release includes modeling code, tokenizer, chat template, and model cards; using the model requires Transformers 5.23 or newer.

What the archive adds

New Runtime's earlier coverage of training Qwen3 on AWS Trainium with torchtitan (October 22, 2026) also examined a concrete model-training workflow, but on different hardware and with a different training stack. Quark is a new fact on that broader training-cost thread, not a continuation of the Qwen3 project.

Trend signal

This supports the broader economics question behind agent economics moves to completed work, but only indirectly: Empero reports the compute cost and task-solving results of a small RL experiment rather than a cost per verified production task. The account is useful for separating training-run economics from deployed-agent economics; it does not yet establish the latter.

Why it matters

The important contribution is not that a 165M model is ready to replace a larger system. Empero’s account gives research and platform teams a relatively legible unit of experimentation: a stated compute budget, named training stages, multiple published checkpoints, and failures alongside scores. That can help a team decide whether it can test a training hypothesis on modest resources before committing to a large run.

The RL result is also a reminder to keep evaluation claims tied to their setting. A rise from 3% to 65% solved in training rollouts across emulated harnesses is evidence that this run changed behavior on those tasks; it is not evidence of robust performance on real coding work. Teams considering similar experiments should distinguish training rollouts, held-out benchmarks, and live workflows, and should budget separately for safety and generalization checks. Empero explicitly says Quark has not had a safety evaluation and remains a research artifact.

Limits and uncertainty

Empero’s benchmark results are from its own runs using the same harness and settings for Quark and comparison models; they have not been independently checked here. Its agentic figures are training rollouts on simple tasks, not a held-out benchmark. The long-context checks use teacher-forced token accuracy, which tests retrieval of expected values rather than whether free generation reaches them. Empero reports failures in quote attribution and one of 15 harness formats, and says long-context scores fell during RL. The model is mostly English-trained, makes factual and reasoning errors, and has only limited refusal training; it has not been safety-evaluated. The $4,200 figure is compute cost, not total labor or project cost.

Reader takeaway

Quark is best read as a transparent experiment in whether a small team can exercise an entire training pipeline on a constrained compute budget. The reported costs, checkpoints, and failure modes are informative; the training-rollout result should not be mistaken for evidence of a reliable production agent.

Evidence / sources

  1. 1empero.org/writing/quark
All Field Notes

Next reads

Read next

Open the graph
  1. 01topicComputeExplore the compute topic hub.
  2. 02archiveField NotesOpen the latest editorial analysis.
  3. 03sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract