---
type: "post"
id: "nvidia-open-sourced-skillevaluator-an-apache-2-0-7de78b"
slug: "nvidia-open-sourced-skillevaluator-an-apache-2-0-7de78b"
title: "NVIDIA open-sourced SkillEvaluator, an Apache-2.0 framework for evaluating agent skills end to end."
description: "NVIDIA has released SkillEvaluator, an Apache-2.0 framework for validating agent skills, checking overlap between them, and measuring their effect in agent runs."
retrieval_nugget: "NVIDIA has released SkillEvaluator, an Apache-2.0 framework for validating agent skills, checking overlap between them, and measuring their effect in agent runs. Its three-tier design gives teams a path from deterministic checks to sandboxed evaluation, making skill changes easier to test in a repository and CI pipeline."
published_at: "2026-11-12"
updated_at: "2026-11-12"
record_date: "2026-11-12"
date_kind: "published_at"
topics: ["skills"]
entities: ["NVIDIA","SkillEvaluator","NVIDIA Verified Skills","SkillSpector","Harbor"]
related_patterns: ["skills-become-portable-capability-layer"]
novelty_label: "new_fact"
what_changed: "NVIDIA has released SkillEvaluator, an Apache-2.0 framework for evaluating agent skills end to end."
prior_refs: []
source_url: "https://github.com/NVIDIA/SkillEvaluator"
source_urls: ["https://github.com/NVIDIA/SkillEvaluator"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:nvidia-open-sourced-skillevaluator-an-apache-2-0-7de78b"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/nvidia-open-sourced-skillevaluator-an-apache-2-0-7de78b/","markdown":"https://newruntime.com/posts/nvidia-open-sourced-skillevaluator-an-apache-2-0-7de78b.md","json":"https://newruntime.com/posts/nvidia-open-sourced-skillevaluator-an-apache-2-0-7de78b.json"}
markdown_url: "https://newruntime.com/posts/nvidia-open-sourced-skillevaluator-an-apache-2-0-7de78b.md"
---

# NVIDIA open-sourced SkillEvaluator, an Apache-2.0 framework for evaluating agent skills end to end.

## Retrieval answer

NVIDIA has released SkillEvaluator, an Apache-2.0 framework for validating agent skills, checking overlap between them, and measuring their effect in agent runs. Its three-tier design gives teams a path from deterministic checks to sandboxed evaluation, making skill changes easier to test in a repository and CI pipeline.

## Summary

NVIDIA has released SkillEvaluator, an Apache-2.0 framework for validating agent skills, checking overlap between them, and measuring their effect in agent runs. Its three-tier design gives teams a path from deterministic checks to sandboxed evaluation, making skill changes easier to test in a repository and CI pipeline.

## What happened

NVIDIA describes SkillEvaluator as an open-source tool for evaluating agent skills: folders of instructions and supporting files defined by the Agent Skills specification. The project is part of NVIDIA’s Verified Skills pipeline. Its repository offers a command-line workflow and ready-to-use CI templates for GitHub and GitLab. The repository shows an Apache License 2.0, Python 3.13 support, 774 stars, and experimental support status; NVIDIA says support is community-based and best-effort, without an SLA or enterprise support entitlement.

The framework separates evaluation into three independent tiers. Tier 1 validates whether a skill is safe and well-formed. It can run without an LLM, but full static coverage requires external scanners; LLM checks require a provider and model clients. NVIDIA’s quickstart uses a bundled text-analyzer skill and says the no-LLM example scores 84.5/100 (grade B), exits successfully, and saves a JSON report. The default passing threshold is 70. The repository cautions that this static result is not publication evidence and that scanner failures can produce an incomplete result and a nonzero exit.

Tier 2 checks whether guidance repeats or overlaps. It uses embeddings and chat, and can compare a skill against a catalog when one is supplied. Tier 3 runs live evaluations to ask whether a skill helps an agent. That requires provider access, a supported agent, and a running sandbox; Docker is the default. NVIDIA says the tier may reuse evaluation sources or create one starter case with limited coverage, and advises generating and reviewing more cases for broader coverage. SkillEvaluator also offers commands to create an evaluation dataset.

Teams can set up repository checks through a guided CLI or copy a template into GitHub Actions or GitLab CI. The checks can include quality validation, SkillSpector security scanning, and reports; GitHub PR comments are an optional publisher workflow. The combined pipeline runs all applicable tiers, but the gates are configurable: Tier 1 always gates, Tier 2 gates unless disabled, and Tier 3 gates only when explicitly configured to block. NVIDIA says model and managed-sandbox usage can incur charges. Eligible NVIDIA Developer Program members can prototype with NVIDIA-hosted endpoints subject to access terms and rate limits; the framework does not require that route for its no-LLM quickstart.

## What the archive adds

New Runtime’s earlier coverage described NVIDIA’s OpenShell reference architecture as an available approach to safe agents ([earlier coverage](https://newruntime.com/posts/nvidia-launches-openshell-reference-architecture-for-safe-ai-bbb154/), October 24, 2026). SkillEvaluator is a new release, not another runtime architecture: it adds a way to evaluate skills and their effects, alongside that earlier focus on agent controls.

## Trend signal

This supports the pattern that [skills are becoming a portable capability layer](https://newruntime.com/patterns/skills-become-portable-capability-layer/). SkillEvaluator makes validation, overlap checks, and behavioral evaluation part of the skill lifecycle. It does not establish that the framework catches every unsafe skill or that teams will adopt its checks broadly; those outcomes require evidence from use beyond the release itself.

## Why now

The repository positions SkillEvaluator as a way to bring skill evaluation into existing repositories and CI workflows. That makes a change to a skill something a team can check before merging, rather than only inspect manually after an agent behaves differently. NVIDIA does not explain the timing of the release, so the practical significance is in the workflow it offers, not a claimed launch rationale.

## Connections to other events

NVIDIA’s earlier OpenShell coverage concerned an architecture for safe agents. The two projects touch different control points: OpenShell is about runtime boundaries, while SkillEvaluator checks skill files and can evaluate their effects in a sandbox. The repository also places SkillEvaluator in NVIDIA’s Verified Skills pipeline, connecting it directly to NVIDIA’s broader work on agent skills without implying that the two tools provide equivalent safeguards.

## Why it matters

Skills are executable product inputs in the practical sense: they shape what an agent attempts, and changes to instructions can alter task outcomes. A team that stores skills in a repository can use SkillEvaluator to make at least some of that change reviewable. The static tier can provide a repeatable baseline without an API key, while overlap and live evaluation address different failure modes that a formatting or lint check would not capture.

The distinctions between tiers matter. A passing static score is not proof that an agent performs better, and a live run is only as useful as its cases, supported runtime, and sandbox setup. Teams need to choose which tiers block a merge, budget for model or sandbox costs, and review incomplete reports rather than treating a green check as a universal safety guarantee. The useful contribution is a configurable path toward evidence in CI, not an automatic certification of a skill.

## What happens next

- **Teams using the framework will add skill checks to GitHub or GitLab CI.** (6 months, 55%). Watch for: public examples or documentation showing recurring CI use beyond the bundled templates.

## Limits and uncertainty

NVIDIA labels the project experimental and offers no SLA or enterprise support entitlement. The repository’s example score measures a bundled skill; it does not show that the framework improves outcomes across teams, tasks, or agent runtimes. Tier 3 coverage depends on the evaluation cases and supported agents, and NVIDIA warns that a generated starter case offers limited coverage. The release does not independently establish how often the scanners detect harmful skills, how reliable model-backed rubric scores are, or what typical evaluation costs will be. Those questions require testing in real workflows.

## Skill to build now

Pick one skill that your team changes regularly and run Tier 1 against it in a branch. Review the findings, then add the provided GitHub Actions or GitLab template and tune its threshold against examples of acceptable and unacceptable skills. Before making the check a merge gate, add representative evaluation cases and run Tier 3 in a sandbox; treat a small starter set as a baseline, not proof of broad coverage.

## Reader takeaway

SkillEvaluator turns skill checks into a repository workflow, from static validation through live agent evaluation. The practical move is to start with inexpensive, repeatable checks, then add behavioral tests and blocking gates only when your own cases show that the results are useful.

## Open question

How reliably will SkillEvaluator’s configurable tiers catch harmful or ineffective skills in real teams’ varied agent workflows?

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Skills](https://newruntime.com/topics/skills.md): Explore the skills topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)