What happened
RETROGRADE is presented as a procedurally generated, behaviorally graded benchmark for reverse-engineering agents. The repository’s methods paper describes how targets are built, how answers are evaluated, and how results are intended to be reported. The paper does not report model results; the repository says those will be published separately after Round 0 completes.
The benchmark gives an agent a stripped binary and, in the default blind setting, withholds the workload, seed, language and, for re-hosted targets, instruction set. The agent operates in a network-isolated sandbox with tools including disassemblers, a debugger, emulators and Python, subject to a fixed budget of tool-calling rounds. It submits structured claims with a confidence estimate and an address that supports each claim. The listed tasks include recovering structure, predicting outputs, testing counterfactual interventions, discovering triggers, reconstructing behavior and providing provenance.
Targets are synthesized from a compositional behavior grammar rather than selected from a static corpus. They are compiled from one intermediate representation to C, C++, Rust or Go as native aarch64 Linux ELF binaries. The source lists 15 logic archetypes, including protocols, parsers, interpreters, an automotive ECU and a benign malware-behavior surrogate. Targets may also be re-hosted on Cortex-M, RISC-V RV32, PowerPC e500, MSP430 or the Hexagon DSP. A lattice of ten protection passes can be applied while preserving behavior; the listed examples include control-flow flattening, string encryption, virtualization, packing and anti-debugging.
RETROGRADE grades claims through execution rather than string similarity. The described checks include differential testing against a reference oracle, output prediction on adversarial inputs, and causal interventions verified in an emulator. Confidence is scored using a Brier-style proper scoring rule with asymmetric loss, so a confident wrong claim about a safety-relevant fact costs more than an honest uncertainty. Provenance also matters: correct answers with bogus justification receive only a fraction of the credit. The repository says the benchmark uses fresh seeds in each round, commits to the seed draw in advance, and plans full transcripts and key disclosure with results.
At publication, Round 0 is a reduced preview: 13 frontier models, 24 episodes per model, across eight workloads and three protection tiers, with one seed per cell. Each model is served through its maker’s API. The repository states that the full pre-registered round will use five seeds and two samples per cell. It publishes a SHA-256 commitment for the preview seed set; the key is to be revealed when results are released. No price or public service access terms are specified in the supplied material.
What the archive adds
We have not covered this before; it opens a new thread.
Trend signal
No supplied SITE_PATTERNS thesis directly addresses procedurally generated, execution-graded reverse-engineering benchmarks, so this item does not map cleanly to a listed trend.
Why it matters
Agent benchmarks are useful only if their scores measure the capability teams care about. A fixed set of binaries can become familiar through repeated publication or training exposure, while text similarity may reward plausible descriptions that fail when tested. RETROGRADE makes a different choice: generate new cases and check claims against executable behavior. If the approach works in practice, it offers evaluators a way to test whether an agent can infer behavior on unseen targets, not just reproduce patterns from a known dataset.
The design also makes uncertainty and evidence part of the score. In reverse engineering, a confident but unsupported statement can be more dangerous than an incomplete answer. Penalizing bad confidence and provenance could make benchmark results more relevant to workflows where engineers need to decide which findings to trust and inspect. For teams building agents, the method is a useful prompt to make evaluation claims auditable: specify the target-generation process, the behavioral oracle, and how evidence quality affects scoring.
However, a methods design is not itself evidence that the benchmark is contamination-proof in deployment or that its scores predict real-world performance. Round 0’s results, transcripts and later full-round evaluations will determine whether the protocol produces informative distinctions among models and protection tiers.
Limits and uncertainty
The repository has not yet published Round 0 results or transcripts, so the supplied material cannot establish how the 13 models perform, how much protection degrades their capabilities, or whether rankings are stable. The preview is explicitly reduced: it has one seed per workload/protection cell, while the full planned round uses five seeds and two samples per cell. The description and design results come from the benchmark creators; no independent replication or external audit is supplied here. The repository’s stated commitment and future key disclosure provide a planned check on the seed draw, not evidence yet that targets cannot leak through other routes. Nor does this material show that benchmark performance predicts success on real-world binaries.
Reader takeaway
RETROGRADE’s notable contribution is an evaluation design, not a leaderboard: generate fresh reverse-engineering targets, test claims by executing them, and score confidence and provenance as well as correctness. The approach is worth watching, but its practical value remains unproven until Round 0 results and transcripts are available and the findings can be scrutinized.