Observed facts: Vals AI describes 24 tasks: 13 on a 7-DoF Franka Emika Panda arm and 11 on a 29-DoF Unitree G1 humanoid. Each episode lasts three minutes of wall-clock time, and the simulation continues while the model deliberates. The page reports that GPT-6.1 Sol averages 51/100 on arm tasks and solves four; no evaluated model solves a humanoid task.
Mechanism: The benchmark gives the agent camera images and robot state, then expects it to issue motor commands within a live simulation. The physics continues during deliberation, so a useful system must do more than produce plausible instructions: it must create a controller that can act repeatedly as conditions change. Vals AI also holds 21 tasks out of the 24, limiting direct tuning to the public examples.
Why now: Vals AI has published both a benchmark methodology and initial model results, offering robotics builders a concrete way to compare embodied control rather than infer capability from language or static task scores.
First implication: The arm-versus-humanoid split suggests that success on simpler manipulation tasks should not be read as evidence of robust whole-body control.
Evidence boundary: These results are from simulated MuJoCo tasks, not deployments on physical robots; the supplied benchmark page does not establish how scores transfer to hardware or to tasks outside its evaluation set.
First watch condition: Watch whether later evaluations on the held-out tasks improve humanoid scores without weakening the real-time constraints.
The useful signal is therefore not only the leaderboard order. The benchmark makes latency and control-loop construction part of the task itself, while exposing enough setup detail for readers to understand what the reported scores measure. It is an evaluation design and a set of reported results, not proof that any model can safely operate a physical robot. Future comparisons will matter most if they preserve these constraints and publish evidence about performance on tasks the systems were not tuned against.