---
type: "post"
id: "how-shopify-turns-open-models-into-task-experts-edc9e2"
slug: "how-shopify-turns-open-models-into-task-experts-edc9e2"
title: "How Shopify turns open models into task experts"
description: "Shopify’s task-expert loop makes a narrow agent workload—not a general model leaderboard—the unit of optimization."
retrieval_nugget: "Shopify’s task-expert loop makes a narrow agent workload—not a general model leaderboard—the unit of optimization."
published_at: "2026-10-21"
updated_at: "2026-10-21"
record_date: "2026-10-21"
date_kind: "published_at"
topics: ["open-models"]
source_urls: ["https://shopify.engineering/open-model-task-experts"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:how-shopify-turns-open-models-into-task-experts-edc9e2"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:2303abe9f87f12450824a7fc09e5d01f5eaab3975d0c55ec9c64c4c43082560e","reviewed_at":"2026-10-21T12:05:20.638Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":1,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Shopify’s task-expert loop makes a narrow agent workload—not a general model leaderboard—the unit of optimization.","observed_facts":[{"text":"Shopify reports Nemotron 3.5 Lightning scored 74.5 on its GraphQL judge after six loops, compared with 73 for the frontier-model baseline.","source_urls":["https://shopify.engineering/open-model-task-experts"]},{"text":"Shopify reports self-healing repaired 59,535 GraphQL conversations over 86 nightly runs, or 69% of attempted failures.","source_urls":["https://shopify.engineering/open-model-task-experts"]},{"text":"Shopify reports a 0.87-second median response in replay testing versus 6.50 seconds for the frontier API, with about 90% lower serving cost.","source_urls":["https://shopify.engineering/open-model-task-experts"]}],"mechanism":"A calibrated task judge scores candidate changes and checkpoints, while a self-healing process converts some failed conversations into training examples.","why_now":"Shopify is describing a deployed workflow in which open-model post-training, evaluation, and self-hosting are linked to its commerce assistant rather than presented as isolated experiments.","implications":["For bounded, high-volume subagent jobs, a continually evaluated open model can be a cheaper and faster production component than routing every step to a frontier API."],"evidence_boundary":"These are company-reported results on Shopify workloads; they do not establish independent replication, generalize to other tasks, or show the full engineering and GPU costs of operating the loop.","watch_conditions":["Whether Shopify reports sustained quality and cost across additional task experts and workloads."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied; this creates a distinct analytical record on Shopify’s production task-expert training and evaluation loop."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/how-shopify-turns-open-models-into-task-experts-edc9e2/","markdown":"https://newruntime.com/posts/how-shopify-turns-open-models-into-task-experts-edc9e2.md","json":"https://newruntime.com/posts/how-shopify-turns-open-models-into-task-experts-edc9e2.json"}
related_patterns: ["ai-engineering-evaluations-are-splitting-into-task-specific"]
---

# How Shopify turns open models into task experts

## Retrieval answer

Shopify’s task-expert loop makes a narrow agent workload—not a general model leaderboard—the unit of optimization.

Shopify’s task-expert loop makes a narrow agent workload—not a general model leaderboard—the unit of optimization. This is the exact thesis: Shopify’s task-expert loop makes a narrow agent workload—not a general model leaderboard—the unit of optimization. The company says Sidekick delegates focused work to post-trained open-model subagents, and its GraphQL example reads API schemas and documentation, writes and runs queries, then returns results. Shopify reports 74.5 on its internal judge for Nemotron 3.5 Lightning after six loops, versus 73 for the frontier-model version.

The mechanism is an evaluation-and-retraining flywheel: define a task rubric, calibrate an LLM judge against human labels, first improve the existing frontier agent, then select and post-train an open model, measure serving performance, and feed repaired failures back into training. A calibrated task judge scores candidate changes and checkpoints, while a self-healing process converts some failed conversations into training examples. This is the exact mechanism: A calibrated task judge scores candidate changes and checkpoints, while a self-healing process converts some failed conversations into training examples. Shopify reports 59,535 repaired GraphQL conversations across 86 nightly runs, 69% of attempted failures. In replay tests it reports a 0.87-second median response versus 6.50 seconds for the frontier API, and about 90% lower serving cost. For bounded, high-volume subagent jobs, a continually evaluated open model can be a cheaper and faster production component than routing every step to a frontier API. This is the exact first implication: For bounded, high-volume subagent jobs, a continually evaluated open model can be a cheaper and faster production component than routing every step to a frontier API.

Why now: Shopify is describing a deployed workflow in which open-model post-training, evaluation, and self-hosting are linked to its commerce assistant rather than presented as isolated experiments. This is the exact why_now: Shopify is describing a deployed workflow in which open-model post-training, evaluation, and self-hosting are linked to its commerce assistant rather than presented as isolated experiments. These are company-reported results on Shopify workloads; they do not establish independent replication, generalize to other tasks, or show the full engineering and GPU costs of operating the loop. This is the exact evidence_boundary: these are company-reported results on Shopify workloads; they do not establish independent replication, generalize to other tasks, or show the full engineering and GPU costs of operating the loop. The first watch condition is whether Shopify reports sustained quality and cost across additional task experts and workloads. This is the exact first watch condition: Whether Shopify reports sustained quality and cost across additional task experts and workloads.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Open Models](https://newruntime.com/topics/open-models.md): Explore the open-models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)