This is the exact thesis: Shopify’s task-expert loop makes a narrow agent workload—not a general model leaderboard—the unit of optimization. The company says Sidekick delegates focused work to post-trained open-model subagents, and its GraphQL example reads API schemas and documentation, writes and runs queries, then returns results. Shopify reports 74.5 on its internal judge for Nemotron 3.5 Lightning after six loops, versus 73 for the frontier-model version.
The mechanism is an evaluation-and-retraining flywheel: define a task rubric, calibrate an LLM judge against human labels, first improve the existing frontier agent, then select and post-train an open model, measure serving performance, and feed repaired failures back into training. A calibrated task judge scores candidate changes and checkpoints, while a self-healing process converts some failed conversations into training examples. This is the exact mechanism: A calibrated task judge scores candidate changes and checkpoints, while a self-healing process converts some failed conversations into training examples. Shopify reports 59,535 repaired GraphQL conversations across 86 nightly runs, 69% of attempted failures. In replay tests it reports a 0.87-second median response versus 6.50 seconds for the frontier API, and about 90% lower serving cost. For bounded, high-volume subagent jobs, a continually evaluated open model can be a cheaper and faster production component than routing every step to a frontier API. This is the exact first implication: For bounded, high-volume subagent jobs, a continually evaluated open model can be a cheaper and faster production component than routing every step to a frontier API.
Why now: Shopify is describing a deployed workflow in which open-model post-training, evaluation, and self-hosting are linked to its commerce assistant rather than presented as isolated experiments. This is the exact why_now: Shopify is describing a deployed workflow in which open-model post-training, evaluation, and self-hosting are linked to its commerce assistant rather than presented as isolated experiments. These are company-reported results on Shopify workloads; they do not establish independent replication, generalize to other tasks, or show the full engineering and GPU costs of operating the loop. This is the exact evidence_boundary: these are company-reported results on Shopify workloads; they do not establish independent replication, generalize to other tasks, or show the full engineering and GPU costs of operating the loop. The first watch condition is whether Shopify reports sustained quality and cost across additional task experts and workloads. This is the exact first watch condition: Whether Shopify reports sustained quality and cost across additional task experts and workloads.