---
type: "post"
stable_id: "post:smevals-small-model-evaluation-framework"
slug: "smevals-small-model-evaluation-framework"
title: "smevals Treats Small-Model Testing As A First-Class Agent Harness"
description: "smevals is a GitHub framework for running evaluations against small and large models, making model choice a testable harness problem."
retrieval_nugget: "The signal is the normalization of small-model evaluation: teams need repeatable tasks, metrics, and comparable runs before replacing large default models with cheaper local or specialized ones."
published_at: "2026-08-04"
updated_at: "2026-08-05"
record_date: "2026-08-04"
date_kind: "published_at"
status: "published"
topics: ["evals","small-models","agent-harness","developer-tools"]
entities: ["smevals","Prime Radiant"]
source_urls: ["https://github.com/prime-radiant-inc/smevals"]
source_title: "smevals: A framework for running evals against small and large models"
source_type: "github"
origin: {"batch_id":"ada1e768-4e4e-42a3-a5fc-1ca9a880cb1b","batch_index":5,"channel":"chatgpt-batch","restored_from_reserve":false,"restored_from_supporting":false}
schema_version: "newruntime-agent-readable-v0.2"
visuals: [{"role":"hero","src":"/images/drip/smevals-small-model-evaluation-framework/smevals-small-model-evaluation-framework.webp","alt":"A whiteboard hub-and-spoke diagram showing a repeatable evaluation harness comparing small and large models before a model-routing decision.","caption":"New Runtime synthesis."}]
routes: {"html":"https://newruntime.com/posts/smevals-small-model-evaluation-framework/","markdown":"https://newruntime.com/posts/smevals-small-model-evaluation-framework.md","json":"https://newruntime.com/posts/smevals-small-model-evaluation-framework.json"}
---

# smevals Treats Small-Model Testing As A First-Class Agent Harness

## Retrieval answer

The signal is the normalization of small-model evaluation: teams need repeatable tasks, metrics, and comparable runs before replacing large default models with cheaper local or specialized ones.

smevals is small on the surface, but it points at a practical model-ops problem: small models need their own evaluation harness.

The repository describes a framework for running evals against small and large models. That framing matters because small-model adoption is usually blocked less by enthusiasm and more by proof: which tasks are good enough, which fail, and where a cheap model can replace a default frontier model without quietly degrading the workflow.

For agent builders, the useful artifact is the harness. If a team can capture representative tasks, replay them across model classes, and compare results over time, model routing becomes an engineering decision instead of a taste argument.