What happened
Google's Hugging Face dataset page describes ASIMOV Agentic as an evaluation harness for agentic AI systems on safety-critical robotics tasks. The repository combines Parquet datasets with Python evaluation scripts, and its quick-start instructions use Inspect AI. The page lists a CC-BY-4.0 dataset license and an MIT project license. Although the repository is publicly accessible, users must accept Hugging Face's conditions to access its files.
The evaluation scripts cover several distinct task families. Single-turn agentic tasks test decisions under physical constraints such as payload and reach limits, feasibility over extended dialogue with tool invocation, safety-monitor decisions, and uncertainty quantification. Separate scripts test gauge reading under glare, blur, or occlusion and whether an agent expresses uncertainty when a gauge cannot be read. Human-safety monitoring evaluates proximity detection from stereo-camera images, with protocols for calling stop_robot() or continue() and for returning a JSON distance estimate.
The closed-loop VLA estimator emulator evaluates subtask planning and execution in simulation, including tool calls, success-probability estimation, and safety compliance. The physical-safety script tests paired harmful and benign tabletop manipulation tasks, both with tool declarations alone and with an explicit safety policy. It also includes a multi-turn jailbreak evaluation: 845 samples, up to 10 steps, and 15 adversarial strategy groups. Scoring combines deterministic checks of tool calls with an LLM judge; the listed measures include attack success rate, benign over-refusal, and a 41-criterion Physical Safety Rubric v2.
Google's page distinguishes the two physical-safety configurations by their instructions. The default configuration uses implicit safety in its tool declarations, while the global-safety configuration adds an explicit policy. It lists 100 harmful and 100 benign examples for the combined default evaluation, and 100 harmful plus 55 policy-compatible benign examples for the combined global-safety evaluation. This structure lets an evaluator examine unsafe compliance and excessive refusal separately rather than treating a refusal rate as a complete safety result.
The page's quick start calls for Git LFS, a Python virtual environment, and dependencies including inspect_ai, pandas, pyarrow, and model-provider libraries. Users set an API key for the model they want to evaluate, then run a task-specific script. The examples name Gemini, GPT-5.5, and Claude Opus models, and show an option to run multiple judge models. The scripts expose limits for sample counts and concurrent connections; the page does not present a single aggregate score for the suite.
What the archive adds
New Runtime covered Gemini 4 Argon's security benchmark results in Cursor on 2026-10-23, including a tie at 29% with Fable 5 and two uniquely solved tasks (earlier coverage). That note concerns coding-agent security, not physical robotics. ASIMOV Agentic is a new event: it adds a robotics-specific evaluation suite rather than another result on the earlier benchmark.
Trend signal
This supports the view that agent completion needs external checks rather than relying on the model's own judgment: the suite uses tool-call checks, task scores, and rubric-based judging. New Runtime tracks that assumption in Goal-scoped loops replace manual continuation. The robotics setting makes the distinction consequential, but the benchmark does not establish that these checks are sufficient for deployment safety.
Why it matters
Robotics teams need ways to test more than whether an agent can produce a plausible answer. ASIMOV Agentic provides concrete probes for recognizing physical constraints, handling uncertain or degraded sensor readings, making stop-or-continue decisions, and resisting adversarial instructions over multiple turns. Its paired benign and harmful tasks can also help surface the cost of overly broad refusal alongside unsafe compliance.
That makes the suite potentially useful during model and policy evaluation, especially where a team can adapt the tasks to its own robot, tools, and operating envelope. It also offers a common script-and-data structure for rerunning a test after changing a model, system policy, tool interface, or safety threshold. But test results are only one input to a safety case. Simulation, test-set performance, and an LLM judge cannot substitute for runtime controls, hardware safeguards, or validation against the conditions in which a robot will actually operate.
Limits and uncertainty
The Hugging Face page describes datasets, scripts, sample counts, and scoring methods, but it does not provide a single benchmark result establishing how current models perform across the suite. Results depend on the selected model, task, sample limit, judge model, and configuration. Some safety-monitor scoring uses an LLM autorater, while other checks are deterministic; those methods measure different things. The page does not independently validate the rubric's relationship to real-world harm or establish that passing these tasks predicts safe behavior on physical robots. Access to the dataset files also requires accepting Hugging Face's conditions.
Skill to build now
Pick one failure mode that matters to your robot, such as a safety monitor missing a nearby person or an agent following a harmful manipulation instruction. Run the corresponding evaluation against your current model and policy, then inspect individual traces rather than relying only on an aggregate score. Record the model, judge, task settings, and failure criteria so a later run can be compared meaningfully.
Reader takeaway
ASIMOV Agentic turns several robotics safety concerns into runnable evaluation tasks, including a sizable multi-turn jailbreak set. Treat it as a way to find and compare failure modes—not as certification that an agent or robot is safe.