---
type: analysis
slug: research-fine-tuning-a-small-model-on-expert-trajectories-can-undo-the-gains-from-an
title: "Fine-tuning a small model on expert trajectories can undo the gains from an evolved harness. Correcting single turns in its own rollouts avo"
description: "A Salesforce AI preprint finds that LoRA imitation of Gemini 3.1 Pro under a harness evolved for Qwen3-Coder cut success by 4–30 points on all seven tasks. Correcting single turns preserved the harness gains and added only about 1.7 points on average. The negative result is the part builders should act on."
published_at: 2026-10-22T23:00:00.000Z
topics: ["gemini","tools","research"]
source_urls: ["https://arxiv.org/abs/2607.08938","https://arxiv.org/html/2609.09134","https://arxiv.org/abs/2609.09134","https://arxiv.org/abs/2607.22688","https://arxiv.org/abs/2609.24974","https://arxiv.org/abs/2604.25850"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
C15 is the arXiv preprint "Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails" (2609.09134, submitted 8 September 2026, Salesforce AI). It builds on recent work showing that automatically evolving an agent's harness (system prompt, tools, hooks, context scaffolding) around a cheap small model can recover most of the performance of a frontier model. The natural next step is to fine-tune the small model on a stronger expert's trajectories as well. The paper asks whether the two kinds of gain stack.

This matters to anyone running small open-weight models in production agents. Many teams already tune harnesses and distill from frontier models, often in that order. The paper's main claim is that the order and the type of supervision matter. Expert demonstrations recorded under a harness tuned for the student can erase the harness gains.

## What we checked

- **If the C15 mechanism (imitation breaks model–harness fit, on-policy correction preserves it) generalizes, do comparable primary studies show the same direction under different models or tasks?** unresolved. Inside the paper, the imitation regression shows up on all seven tasks with Qwen3-Coder-30B-A3B and is reproduced for Gemma-4-26B-A4B on only one task (website management). The correction method was not tested on Gemma. No outside study tests expert imitation under a student-evolved harness. Co-Harness reports that fine-tuning on improved-harness trajectories helps, but in a different setup. Harness-Zero's gains from correcting student turns are consistent with the correction idea but do not test it directly. [2][4][5]
- **Does full-trajectory imitation under the evolved harness regress performance by 4–30 points on all seven tasks, as the abstract claims?** confirmed. For Qwen3-Coder, the evolved harness alone scored 78.0% on average, and imitation SFT under it scored 63.1%. Per-task drops run from 4.2 points (anomaly detection) to 29.8 points (payroll auditing). The same imitation under the unevolved harness raised the average from 29.2% to 35.5%. [2][3]
- **Does on-policy expert correction meaningfully close the gap to the expert model?** partly confirmed. Correction SFT reached 79.7% on average, against 78.0% for the evolved harness alone and 93.6% for Gemini under the evolved harness. Two tasks came in slightly below harness-only. The paper reports no significance tests and states the method 'does not fully close the gap'. [2]

## What we found

- The setup reuses the task suite and harness optimizer from Yang et al. (CMU). That work reported optimized harnesses significantly improving 16 of 21 task–SLM pairs, with the best small-model agent recovering 89.7% of LLM performance at 4% of the cost. C15 evolves harnesses GEPA-style with gemini-3.1-pro-preview as the reflection model, a $20 budget per task, and three seeds. The students are qwen3-coder-30b-a3b-instruct and gemma-4-26b-a4b-it. [1][2]
- Harness evolution alone did most of the work for Qwen3-Coder, raising average test success from 29.2% to 78.0% across seven enterprise tasks: payroll, budget, stock, IoT anomaly, Playwright, website management, and refactoring. The expert also got more out of the evolved harness than the student did, going from 84.4% to 93.6%. That gap is what motivated distillation. [2]
- LoRA imitation of the expert's successful trajectories under the evolved harness cut the student's average to 63.1%, with drops on every task. Under the original harness, the same procedure helped (29.2% to 35.5%). Gemma on the website task fell from 55.6% (evolved harness) to 41.1% after imitation. The authors' explanation is that the student copies the expert's planning style without being able to execute it, so it no longer fits a harness built around its own planning. [2][3]
- The fix works like this. A failure-locus step finds the single failing turn in each of the student's own failed rollouts. The expert rewrites only that turn, as one sentence of strategy plus a corrected tool call. Three candidates are sampled and judged, and LoRA training runs on roughly 500 rows. This reached 79.7% on average, only 1.7 points above harness-only, and was slightly below it on payroll and budget. No significance tests are reported, results are mean ± SEM over three runs, the number of test instances per task is not given, and no code release is stated. [2]
- Neighbouring work points in mixed directions. Co-Harness (July 2026) alternates harness edits with fine-tuning on improved-harness trajectories and presents this as beneficial, though its abstract gives no comparative numbers. Harness-Zero (September 2026) has an agent correct student responses in the target harness's action space. It reports macro task success rising from 23.3% to 44.3% with the specialized harness removed, a method close in spirit to C15's turn-level correction. [4][5]
- Separate evidence suggests that harness components differ in how well they transfer. Agentic Harness Engineering reports that a frozen evolved harness transferred to three other model families on Terminal-Bench 2 with +5.1 to +10.1 point gains. Its ablations attribute most gains to tools, middleware and memory rather than prompt prose. That fits C15's view that strategy-level behaviour is the model-specific part. [6]

## What it means

If you have already tuned a harness around a small model, don't assume that fine-tuning on frontier-model demonstrations will add to it. In this study, imitation under the student-tuned harness lost about 15 points on average, while the same data under a generic harness helped. Before shipping a fine-tune, re-evaluate it under the exact production harness, and compare it against harness-only rather than against the base model. When you do distill, prefer supervision that starts from the student's own rollouts: correct the failing step and keep the rest of its trajectory. That at least kept the harness gains here.

Be modest about expectations. The positive method added only about 1.7 points over harness-only, with no significance testing. The full study used one student model, and the second student was checked on only one task. The robust takeaways are that harness evolution was by far the biggest lever and that naive imitation can hurt. "Co-evolution closes the gap" is not established. Treat the model and its harness as one unit you version and evaluate together.

## Still open

- Does the imitation regression hold for other student families and sizes, and for full fine-tuning or RL rather than LoRA?
- Is the 1.7-point gain from on-policy correction statistically distinguishable from noise, given that the number of test instances per task is unreported?
- Does on-policy correction help Gemma 4 or any second model? It was not tested.
- Would re-evolving the harness after fine-tuning (true alternating co-evolution) recover or exceed expert performance?

## Sources

1. [Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation](https://arxiv.org/abs/2607.08938)
2. [Co-Evolving Harnesses and Models (full HTML)](https://arxiv.org/html/2609.09134) (primary)
3. [Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails (abstract)](https://arxiv.org/abs/2609.09134) (primary)
4. [Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents](https://arxiv.org/abs/2607.22688)
5. [Harness-Zero: Harness Distillation via Agent-as-Harness](https://arxiv.org/abs/2609.24974)
6. [Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses](https://arxiv.org/abs/2604.25850)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Gemini](https://newruntime.com/topics/gemini.md): Explore the gemini topic hub.
- topic hub: [Tools](https://newruntime.com/topics/tools.md): Explore the tools topic hub.
- topic hub: [Research](https://newruntime.com/topics/research.md): Explore the research topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)