Flint's 89.7% is on Scale's real SWE-bench Pro v2, but likely sits at the bottom of that board

What we checked

• Can Recursive publish the SWE-bench Pro v2 split, harness configuration, attempt protocol, checkpoint precision, and comparable baseline runs? unresolved. As of October 30 Recursive's model card and launch post still give only mini-swe-agent, no internet, at least five attempts and up to 5-hour timeouts; they name no split (642 or HARD-51), no clean-container re-grade, no checkpoint precision and no baseline runs. Nothing prevents publication, since Scale ships the tasks, harness flags and re-grader, but Recursive has not done it. [3][4][1]
• Is SWE-bench Pro v2 an undefined or private benchmark, as the S5 Labs review implies? refuted. Scale released SWE-Bench Pro V2 on September 22, 2026 as a 642-task public split with a 51-task Hard subset, a locked offline protocol and clean-container re-grading, with a paper on arXiv. The name in Recursive's materials matches this public benchmark, though Recursive never links to it. [5][2][1]
• Does 89.7% place Flint near frontier coding agents? refuted. If the score is on the full 642 tasks, it sits just below Inkling (89.9%) and above only Haiku 5.5 (87.2%) among the systems Scale reports, while frontier agents reach 96.9 to 99.4%. The 15.5-point gap over Flint's Verified score fits a benchmark where public scores run near ceiling, which Scale partly attributes to training-time exposure. [1][2]

What we found

• On the full 642-task set, scores cluster near the ceiling. Scale's paper reports frontier agents up to 99.4%, and its blog gives Inkling 577 of 642 (89.9%) and notes near-ceiling public scores can reflect recall as well as capability. If Flint's 89.7% is on the full set, it lands at Inkling's level and below every frontier system Scale lists, which also explains why it sits far above Flint's Verified score. [1][2]
• Recursive's disclosure is thin and partly diverges from Scale's locked protocol. The card and blog give mini-swe-agent, no internet, average pass@1 over at least five attempts and timeouts of up to five hours; Scale's protocol uses a 50-minute budget and clean-container re-grading, which Recursive does not mention. Neither document says whether the 642 set or HARD-51 was used. [3][4][1]
• Checkpoint precision for the score is not stated. The model card warns results vary with precision and runtime, and the blog gives only an average claim that NVFP4 is within 0.7 points of 16-bit across benchmarks, with no per-benchmark breakdown. [3][4]
• No outside party has scored Flint on Pro v2. Flint is absent from Scale's leaderboard and paper table, and independent coverage says the company has given outsiders no way to check its weight-class lead. [5][1][6]

What it means

Treat 89.7% as a self-reported score on a real, public, near-saturated benchmark, not as evidence of frontier coding ability. On the full split it would put Flint in the same band as Inkling and Haiku 5.5, which is still notable for a 2.5B-active local model, but far from the 96.9 to 99.4% frontier range. The Hard split, where systems spread from 52.9% to 98.0%, would say far more, and Recursive has not reported it.

Builders can close the gap themselves: Scale publishes the tasks, the locked-protocol flags and the clean-container re-grader. Running Flint's quantized checkpoint through that protocol on HARD-51, which is only 51 tasks, is a cheap way to know what the local model can do before replacing a hosted coding API.

Competing readings

• The score is on Scale's full 642-task public split under a setup close to the locked protocol, so Flint is a mid-pack, Inkling-level system on a near-saturated benchmark (likely, about 65%). What would tell it apart: The V2 public split has 642 tasks and Inkling resolves 577 of them, roughly level with Flint's 89.7% [5][2]; Recursive reports the Pro v2 score next to its Verified score [3].
• Recursive's run departs from the locked protocol (longer timeouts, possibly no clean re-grade), so 89.7% overstates what a protocol-compliant run would show (unlikely, about 25%). What would tell it apart: Recursive allowed up to 5-hour timeouts and at least 5 attempts, against Scale's 50-minute per-task cap and clean-container re-grade [1][4].
• The score is on the 51-task Hard split, which would make Flint near-frontier (very unlikely, about 8%). What would tell it apart: Frontier systems top out at 99.4% on the full set, and S5 Labs notes Recursive names no split [7][1].

Base rate

The New Runtime archive holds only 4 earlier research stories about companies, models, organisations, products or projects from the last 12 months that are at least 3 months old: too few (minimum 5) for a base rate. The archive holds only 4 comparable earlier research stories, of which 3 were followed by a release; 4 is below the minimum of 5, so there are too few cases and this analysis does not lean on it.

What would change this view

• Recursive publishes Flint trajectories or per-task grades showing the run was on HARD-51 (model-listing signal): 89.7% would be a near-frontier result rather than a mid-pack one.
• Scale or an independent group adds Flint to the V2 leaderboard under the locked protocol with a much lower score (model-listing signal): Recursive's harness settings inflated the self-reported number.
• Recursive releases its eval configs and baseline runs for the competitors it claims to lead (open-source signal): The weight-class leadership claim becomes checkable and the comparability gap closes.

Still open

• Did Recursive run the full 642-task split or HARD-51?
• Were Flint's patches re-graded in a clean container, and with what budget per task?
• Which checkpoint (BF16, NVFP4 or int4) produced 89.7%?
• Which comparison models did Recursive run itself on Pro v2, and under what settings?