What happened
A public GitHub repository by AlexWortega presents an independent baseline for ShadowBench, a benchmark in which a system makes hand poses to match target silhouettes. Its “dumb rigid-hands IoU baseline” fixes all 46 joint angles of two ordinary open hands at zero. The algorithm only moves and rotates the hands as rigid objects, seeking to maximize pixel intersection-over-union (IoU) between the resulting silhouette and the target mask. The README says this method uses no animal names, CLIP, AdamW, or book images. Its point is that a high silhouette score can come from a pose that plainly does not resemble the pictured figure.
The repository reports results on 20 certified targets. A 300-proposal offline static-pose search had a mean official score of 41.76, compared with 36.19 for the published blind CMA search and 53.06 for the published best model per target. The baseline beat blind CMA on 19 of 20 matched targets, with a paired mean difference of +5.58 and a bootstrap 95% interval of [3.46, 7.83]; it beat the published best model on none. At 1,000 proposals, the reported mean was 47.02. That version beat blind CMA on all 20 targets, with a paired mean difference of +10.84 and interval [9.40, 12.36], and beat the per-target best model on 3 of 20.
These are not all like-for-like agent runs. The repository labels the static-pose searches offline and says their budgets differ from the benchmark's published comparison. It reports a separate tool-agent run using only the initial observation, tool acceptance text, and “look” images to decide moves. Within 100 calls and 32 look images, that run averaged 29.22. The published rest-pose score is 12.11, and a published target-convex-hull baseline scores 56.91, though the latter uses the target mask. The README calls the comparisons diagnostic, not a new leaderboard ranking, and explicitly says the offline result does not prove the contest is solved.
The project provides scripts for downloading the public dataset at pinned commit 795a17b86e4f24095c0551849909c32482cfe183, running the searches, comparing published scores, and auditing or validating saved results. It says the work was tested on Linux with Python 3.12. The authors report independently rerendering saved final poses and passing the static anatomical gate with identical official scores; the tool-agent logs passed a budget audit. The repository attributes the simulator, target drawings, task metadata, and Ultraleap rig to Perpetual Research, and describes itself as an independent baseline rather than an official ShadowBench submission.
What the archive adds
New Runtime's earlier coverage noted that a voice model’s headline quality can differ from the constraints that shape a live interaction in a note on voice-agent benchmarking. This is a new event, but it similarly asks whether a headline measurement reflects the task a system must actually perform.
Trend signal
This does not yet fit a pattern New Runtime tracks: it is a narrow, independent diagnostic of one benchmark’s scoring behavior rather than evidence about a durable product, protocol, organizational, or deployment shift.
Why it matters
For teams choosing or designing AI evaluations, a metric is part of the system specification: models and search procedures will optimize what it rewards. Here, matching a silhouette can be separated from making a plausible hand figure, so the score alone may overstate task competence. The distinction between 1,000 offline proposals and a tool-agent run limited to 100 calls and 32 looks is equally important. Combining those rows into one leaderboard conclusion would hide the operational constraint that matters in deployment.
A useful response is not to discard IoU, but to pair it with checks that test the intended outcome: human review of examples, independent validity gates, and comparisons under matched budgets. The repository's saved-pose validation and tool-log budget audit are relevant practices, even though its small public footprint and independent status mean its findings should be treated as a diagnostic result rather than a settled evaluation of ShadowBench.
Limits and uncertainty
The reported comparisons cover 20 certified targets and come from the repository authors; they are not an official ShadowBench submission or an independent replication. The README says the dataset, task table, and rig are downloaded from upstream rather than vendored, and does not claim that the benchmark as a whole is solved. Offline searches use budgets different from the agent protocol, so their scores cannot establish how a constrained agent would perform. The results show a possible metric shortcut on these tasks, not how often it occurs across other benchmarks or whether the benchmark maintainers will change scoring or rules.
Reader takeaway
A high benchmark score is evidence about the metric first, and about the intended capability only when the evaluation rules make shortcuts difficult. ShadowBench's rigid-hands baseline is a concrete reminder to check outputs and budgets before interpreting a leaderboard gap as progress.