---
type: "post"
id: "nvidia-skillevaluator-0-9-0-adds-evaluation-budgets-76807e"
slug: "nvidia-skillevaluator-0-9-0-adds-evaluation-budgets-76807e"
title: "NVIDIA SkillEvaluator 0.9.0 adds evaluation budgets, baselines, and optional CI"
description: "NVIDIA SkillEvaluator 0.9.0 adds evaluation budgets, reusable no-skill baselines, skill-version comparisons, richer reports and optional CI checks."
retrieval_nugget: "NVIDIA SkillEvaluator 0.9.0 adds evaluation budgets, reusable no-skill baselines, skill-version comparisons, richer reports and optional CI checks. For teams building agent skills, the release makes it easier to measure whether a skill helps, control evaluation spending and inspect why a run passed or failed."
published_at: "2026-11-15"
updated_at: "2026-11-15"
record_date: "2026-11-15"
date_kind: "published_at"
topics: ["github"]
entities: ["NVIDIA","SkillEvaluator","GitHub","GitLab"]
related_patterns: ["skills-become-portable-capability-layer","ai-engineering-evaluations-are-splitting-into-task-specific"]
novelty_label: "new_fact"
what_changed: "Version 0.9.0 adds evaluation budgets, baselines for comparing skill versions, and optional CI support."
prior_refs: ["https://newruntime.com/posts/nvidia-open-sourced-skillevaluator-an-apache-2-0-7de78b/"]
source_url: "https://github.com/NVIDIA/SkillEvaluator/releases/tag/v0.9.0"
source_urls: ["https://github.com/NVIDIA/SkillEvaluator/releases/tag/v0.9.0"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:nvidia-skillevaluator-0-9-0-adds-evaluation-budgets-76807e"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/nvidia-skillevaluator-0-9-0-adds-evaluation-budgets-76807e/","markdown":"https://newruntime.com/posts/nvidia-skillevaluator-0-9-0-adds-evaluation-budgets-76807e.md","json":"https://newruntime.com/posts/nvidia-skillevaluator-0-9-0-adds-evaluation-budgets-76807e.json"}
markdown_url: "https://newruntime.com/posts/nvidia-skillevaluator-0-9-0-adds-evaluation-budgets-76807e.md"
---

# NVIDIA SkillEvaluator 0.9.0 adds evaluation budgets, baselines, and optional CI

## Retrieval answer

NVIDIA SkillEvaluator 0.9.0 adds evaluation budgets, reusable no-skill baselines, skill-version comparisons, richer reports and optional CI checks. For teams building agent skills, the release makes it easier to measure whether a skill helps, control evaluation spending and inspect why a run passed or failed.

## Summary
NVIDIA SkillEvaluator 0.9.0 adds evaluation budgets, reusable no-skill baselines, skill-version comparisons, richer reports and optional CI checks. For teams building agent skills, the release makes it easier to measure whether a skill helps, control evaluation spending and inspect why a run passed or failed.

## What happened
NVIDIA’s GitHub release notes say SkillEvaluator 0.9.0 expands dataset construction, comparison of skill versions and evidence about evaluation results. Teams can declare deterministic checks for script execution, workspace files and final responses. Datasets can be generated from skill instructions, author intent or traces; production traces can be imported as redacted review candidates, with expectations promoted only after explicit human review. The release also adds previews for legacy dataset migrations before writing a new file.

For comparisons, users can retain skill versions and compare them with `--compare-to` or `compare-versions`. A compatible baseline without the skill can be reused with `--reuse-baseline`. The release notes distinguish version comparisons from Skill Lift, which compares against a no-skill baseline, and describe pass@k as supporting reliability information. Evaluation spending can be bounded by token, judge-call, attempt and priced-cost budgets. Reports preserve partial results when a limit stops a run and show token use per case and agent, including missing counters and usage from reused baselines.

The reports identify met, unmet, unchecked and inapplicable expectations, as well as dataset coverage and sufficiency, skill activation, failed-arm coverage and next steps. HTML charts work offline. The release also adds grader qualification with `tier3 validate-grader`, inspection of retained judge requests and replies, opt-in judge retries and fallback policies, and questions about completed runs through `tier3 feedback`. Externally computed custom metrics can be imported with provenance.

CI integration is opt-in for GitHub and GitLab and adds advisory Tier 2 and Tier 3 evaluations alongside required Tier 1 checks. The release notes say Tier 3 remains advisory even for all-tier CI; deprecated settings that made it a required CI gate no longer do so. SkillEvaluator 0.9.0 is available as a Python package, installable with `python -m pip install --upgrade skillevaluator`; the notes list Python 3.12, 3.13 and 3.14. Optional Tier 2, Tier 3 and Python security dependencies are available through the `all` extra. Live evaluation requires provider credentials and a selected agent and sandbox runtime.

## What the archive adds
New Runtime covered NVIDIA’s initial open-source release of SkillEvaluator on 2026-11-12 in [an earlier Field Note](https://newruntime.com/posts/nvidia-open-sourced-skillevaluator-an-apache-2-0-7de78b/). Version 0.9.0 is a new fact on that story: it adds spending controls, version comparisons, more explicit evaluation evidence and opt-in CI support to the framework.

## Trend signal
This release supports New Runtime’s verification-bandwidth-is-the-scarce-engineering-resource pattern. Its reports expose more of the evidence behind agent-skill results, while budgets and CI checks help teams repeat evaluations without treating a passing run as self-explanatory. The release does not establish that these features reduce review effort or improve delivery outcomes; those effects remain to be measured in use.

## Why it matters
A skill that appears useful in one successful run may fail on other cases, consume more tokens than expected or create a misleading comparison if the baseline changes. The new controls give teams a way to set evaluation limits, preserve a no-skill point of comparison and inspect coverage and unchecked expectations before deciding whether a skill is ready to rely on.

CI support can make evaluation part of the development loop rather than a one-off experiment. Keeping Tier 2 and Tier 3 advisory, however, means teams still need to decide what evidence is sufficient for their own release gates. The release notes also flag changed scoring behavior: accuracy now evaluates task outcomes without penalizing a no-skill baseline for not identifying the skill, and some scored attempts or dimensions can fail under revised thresholds. Teams should re-evaluate before comparing scores with older releases.

## Limits and uncertainty
The release notes describe capabilities, but do not report independent measurements of evaluation accuracy, cost savings or the effect of CI integration on skill quality. Results depend on the team's datasets, checks, judges, agent and sandbox setup; production traces are review candidates, and expectations require human review before promotion. Tier 2 and Tier 3 checks are advisory in CI, not required gates. NVIDIA also labels Hermes support experimental. The release advises teams to re-evaluate before comparing scores with older versions because grading outcomes changed.

## Skill to build now
Pick one skill that matters to a real workflow and define a small set of deterministic checks for its expected outputs, files or script behavior. Run the same cases with and without the skill, set a token or cost budget, and inspect unchecked expectations and dataset coverage before trusting a higher Skill Lift. If the evaluation is useful, add the generated CI checks as advisory first and review their findings before turning any local policy into a required gate.

## Reader takeaway
SkillEvaluator 0.9.0 turns skill evaluation into a more inspectable and budgeted workflow, with optional CI integration. The practical shift is not simply that teams can generate more scores: they can compare against a reusable no-skill baseline and see more of the evidence behind each result. Re-run evaluations before treating scores from this release as comparable with earlier versions.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Github](https://newruntime.com/topics/github.md): Explore the github topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
