---
type: analysis
slug: research-decepticon-s-102-104-xbow-score-is-self-reported-ran-with-hints-and-uses-a
title: "Decepticon's 102/104 XBOW score is self-reported, ran with hints, and uses a benchmark its creators now call saturated"
description: "PurpleAILAB's agent publishes per-tier results and run traces, but not the model it used or what its \"cycles\" mean, and no one has reproduced the run. XBOW itself says top systems now score about 100% on these benchmarks."
published_at: 2026-10-22T01:00:00.000Z
topics: ["benchmarks","agents","research"]
source_urls: ["https://github.com/PurpleAILAB/Decepticon/blob/main/benchmark/results/README.md","https://github.com/PurpleAILAB/xbow-validation-benchmarks","https://github.com/PurpleAILAB/Decepticon/tree/main/benchmark","https://github.com/PurpleAILAB/Decepticon/blob/main/benchmark/README.md","https://github.com/PurpleAILAB/Decepticon","https://github.com/PurpleAILAB/Decepticon/blob/main/docs/benchmark-comparison.md","https://github.com/KeygraphHQ/xbow-validation-benchmarks","https://github.com/xbow-engineering/validation-benchmarks"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
The newsroom flagged a claim of 98.08% (102/104) on the XBOW validation benchmarks. That set has 104 Jeopardy-style web CTF challenges, each hiding one flag, and XBOW published it to evaluate offensive security tools. The PurpleAILAB repo given as the source is a fork of xbow-engineering/validation-benchmarks (it now shows under BitterSecurity). It contains the challenges only, with no score. The score comes from Decepticon, PurpleAILAB's open-source LangGraph-based red-team agent, which publishes the result in its own benchmark results page.

This matters to builders because XBOW scores are widely used as a headline number for AI pentesting agents. Agents now sit in the 85–98% range, so small differences depend on hint level, black-box versus white-box access, number of attempts, and whether the benchmark is still uncontaminated. A self-reported top-of-leaderboard result needs those details before it means anything.

## What we checked

- **Can the 102/104 result be independently reproduced with a documented agent, harness, and per-tier scoring protocol?** unresolved. The agent (Decepticon), the harness code (benchmark/runner.py, harness.py, scorer.py), binary flag scoring and the per-tier split (45/45, 50/51, 7/8) are all public. The model, the run configuration and the meaning of 'cycle-4 final' are not stated. I found no independent reproduction. [1][3][4]
- **Does a near-perfect score on the XBOW validation set still tell you much about an agent's capability?** confirmed. The upstream XBOW repo now says the benchmarks are outdated and saturated as of mid-2026. It says the vulnerabilities 'have been trained into the models', industry-standard performance is 'approximately 100%', and the set 'is no longer useful for discriminating between models or attack frameworks.' [8]
- **Is the 98.08% directly comparable with other published scores on the same 104 challenges?** refuted. Decepticon ran black-box but was given each challenge's description and vulnerability tags. Its own comparison page says Strix got only the description, and Shannon Lite's 100/104 was white-box on a version with the hints removed. The setups differ too much to rank the systems against each other. [6][7]

## What we found

- The 102/104 figure is Decepticon's own result, labeled 'cycle-4 final'. By tier it is L1 45/45 (100%), L2 50/51 (98.0%) and L3 7/8 (87.5%). Each solved challenge links to an evidence folder and a public LangSmith trace of 'the winning run'. The two failed challenges are not named, and no run dates are given. [1]
- The source repo does not contain the claim. Its README describes the 104 benchmarks, says they were confidential until release, and gives the build instructions. It has no pass rate, agent, harness or list of failures. It is a fork of xbow-engineering/validation-benchmarks. [2]
- The harness is public but missing key details. Decepticon's benchmark directory includes a runner CLI, an orchestrator, a scorer and a provenance module. Scoring is pass/fail on finding FLAG{sha256(CHALLENGE_ID.upper())}, with a default timeout of 1800 seconds. With BENCHMARK_MODE=1, a middleware adds the target, the tags, the flag format and a mission brief to every model call. The docs name no model or sampling settings, and the main README only lists multi-provider 'eco/max/test' profiles. [3][4][5]
- The attempt protocol is unclear. Re-runs keep earlier timestamped result directories so that 'cycles' can be compared, and the results page links 'the winning run'. That points to several attempts and possibly tuning across cycles, not a single pass. Separately, the results page lists one run of about 5,549 seconds (XBEN-040-24), well over the 1800-second default, so that run used a different configuration from the one documented. [4][1]
- The agent was given hints. Decepticon's comparison page says it ran black-box but received each challenge's one-line description, vulnerability tags and flag format, while Strix got only the description. The same page places Decepticon first in an 11-system table, with Shannon Lite and Strix at 100/104, PentestGPT at 90/104 and XBOW's commercial agent at about 85%. It admits Shannon's white-box, hint-removed number is 'not directly comparable'. [6]
- KeygraphHQ's cleaned fork says the original challenges leak hints through descriptive variable names, comments, file paths, titles and Dockerfile settings, and removes them. Shannon Lite scored 100/104 on that version in white-box mode. Hint-level choices therefore change what a score means on this suite. [7]
- XBOW's own repo now carries a warning. It says the benchmarks are outdated and saturated as of mid-2026, the vulnerabilities have been trained into models, typical performance is about 100%, and the set no longer separates models or frameworks. The repo is kept 'for historical purposes only.' [8]

## What it means

Read 102/104 as a vendor's claim on a solved benchmark, not as evidence that Decepticon leads the field. The publication is better than most: the harness code is open, the tier split is given, and per-challenge traces are linked. It still leaves out the facts you need to compare results. Those are the model and provider used, how many runs or 'cycles' went into the best result, whether runs were tuned between cycles, which two challenges failed, and whether exposing the flag format and challenge ID could let an agent compute a deterministic sha256 flag instead of capturing it. Add the tag hints and XBOW's own statement that the set is saturated and contaminated, and the gap between 98% and 96% says almost nothing about real-world capability.

If you are choosing or building an offensive agent, rerun it yourself. Pin the model, allow one attempt per challenge, withhold tags and descriptions (or use KeygraphHQ's hint-removed set), make sure flags are randomized at build time rather than derived from the challenge ID, and report pass@1. Better still, move to benchmarks that are not yet saturated or held out, and to your own targets. XBOW scores are now a minimum bar for competence, not a way to tell agents apart.

## Still open

- Which LLM or LLMs and which profile (eco, max or test) produced the cycle-4 run?
- What is a 'cycle', and how many attempts per challenge went into the winning runs (pass@1 or best-of-n)?
- Which L2 and L3 challenges failed, and why?
- Could the agent see the challenge ID and the sha256 flag formula, which would make flags derivable without exploitation?

## Sources

1. [Decepticon benchmark results README](https://github.com/PurpleAILAB/Decepticon/blob/main/benchmark/results/README.md) (primary)
2. [PurpleAILAB/xbow-validation-benchmarks (fork of XBOW validation benchmarks)](https://github.com/PurpleAILAB/xbow-validation-benchmarks) (primary)
3. [Decepticon benchmark harness directory](https://github.com/PurpleAILAB/Decepticon/tree/main/benchmark) (primary)
4. [Decepticon benchmark harness README](https://github.com/PurpleAILAB/Decepticon/blob/main/benchmark/README.md) (primary)
5. [Decepticon repository README](https://github.com/PurpleAILAB/Decepticon) (primary)
6. [Decepticon benchmark comparison](https://github.com/PurpleAILAB/Decepticon/blob/main/docs/benchmark-comparison.md) (primary)
7. [KeygraphHQ hint-removed XBOW benchmarks](https://github.com/KeygraphHQ/xbow-validation-benchmarks)
8. [XBOW validation-benchmarks (upstream, with saturation notice)](https://github.com/xbow-engineering/validation-benchmarks)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Benchmarks](https://newruntime.com/topics/benchmarks.md): Explore the benchmarks topic hub.
- topic hub: [Agents](https://newruntime.com/topics/agents.md): Explore the agents topic hub.
- topic hub: [Research](https://newruntime.com/topics/research.md): Explore the research topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)