Saluki's tool-calling lead over Qwen3.8-27B is Conway's own claim, and outsiders can't yet check it

What we checked

• Can a public source verify Conway's 120-task Underdog Bench protocol and reported Saluki scores against the stated Qwen baseline? partly confirmed. Conway's model card publicly states the protocol and scores: 120 BFCL v4 tasks frozen before testing, thinking off, temperature 0, Saluki 88, full-size Qwen3.8-27B 84, Bonsai 2 70. That confirms what Conway claims. It does not verify the results, because the card names no task IDs, selection method or harness, and the repo holds only weights, license, notice and README. Berkeley's BFCL leaderboard says nothing about Underdog Bench or Saluki. [1][3][4]
• Has any independent party reproduced the Saluki vs. Qwen3.8-27B tool-calling gap? unresolved. Every independent write-up I opened repeats the model card's numbers and runs no tests of its own. ThreatFrontier says outright that the figures are vendor-run and have not been independently reproduced. [6][7]
• Is the 4-task gap (88 vs 84) bigger than the noise Conway itself acknowledges? unresolved. The card says 120 tasks is a modest sample and that a few tasks of any gap may be run-to-run variation. That describes a 4-task difference. The card reports no repeated runs or confidence intervals. [1]

What we found

• The model card reports Underdog Bench scores of 88/120 for Saluki, 84 for full-size Qwen3.8-27B and 70 for PrismML's Bonsai 2 (5.95 GB). It says the tasks were drawn from BFCL v4 and "frozen before any model was tested," and that runs used thinking off and temperature 0. The full model was run "with the same harness and settings," so this particular comparison is like-for-like as far as the card describes. [1][2]
• Several things are missing from the card and the repo: how the 120 tasks were chosen, which BFCL categories they cover, the task IDs, a freeze date, and the harness code. The repo contains only the main GGUF, two mmproj vision files, LICENSE, NOTICE and README. Some secondary sites describe 24 tasks in each of five categories and a 27 September freeze, but I couldn't find that in the primary README, so it remains unverified. [3][2]
• A second tool-calling result uses 100 BFCL v4 parallel-call tasks scored with the official checker. Saluki gets 42 and the full model 35. The card also says about a fifth of Saluki's parallel-call replies have small formatting slips, which matters for any production parser. [1]
• Saluki loses ground outside tool calling. Conway reports AIME 2025 at 79.2 against a public 96.7, MuSR at 67.5 against 79.6, MBPP+ at 78.0 against 83.9, and SWE-bench Verified at 30 against 33 issues out of a 50-issue sample. The card warns that the "(public)" baselines come from a different harness and aren't directly comparable. It claims 96% average retention across nine benchmarks but shows only seven rows. [1]
• Berkeley's BFCL page describes V4 as holistic agentic evaluation and defines Overall Accuracy as an unweighted average of sub-categories. Models there were evaluated at commit f7cf735 (bfcl-eval==2025.12.17), and the page was last updated April 12, 2026. It doesn't mention Saluki, Conway or any 120-task subset, so a 120-task score can't be placed on the official leaderboard. [4]
• The upstream ISTA-DASLab repo exists and offers GSQ/RCO quants of Qwen3.8-27B from IQ2_XS (2.50 bpw, 8.4 GB) to IQ3_S (3.50 bpw, 11.8 GB). Its own benchmark claims are self-reported, and it doesn't mention Conway. That fits the card's account of where the model came from, but it says nothing about the tool-calling claim. [5]
• Independent coverage repeats Conway's numbers without testing them. ThreatFrontier calls them vendor-run and not independently reproduced, and MindStudio attributes them to the model card. A TokenPost piece on Conway's Apple-silicon benchmarks also describes self-run results with no third-party check. [6][7][8]

What it means

The defensible reading is that Saluki looks roughly on par with full-precision Qwen3.8-27B at single tool calls, on Conway's own unpublished 120-task subset. "Beats the original" goes further than the evidence. The 4-task margin falls within the run-to-run variation the card admits to, and nobody outside Conway can rerun the subset. The parallel-call gain (42 vs 35, scored with the official checker) is the stronger signal, but a fifth of those replies have formatting slips. The trade-off is real and visible: math and multi-step reasoning drop noticeably.

If you're considering Saluki for a local agent, run your own evaluation. Use the public bfcl-eval package on the full BFCL v4 categories you care about, plus your real tool schemas. Use thinking off and temperature 0 to match Conway's setup, and do several seeds. Test your output parser against malformed parallel calls. Compare against ISTA-DASLab's IQ2_XS/IQ2_S files at similar size, since Saluki derives from them. Treat any claim of a quant "beating" its parent as unproven until the subset and harness are published.

Still open

• Which 120 BFCL v4 task IDs make up Underdog Bench, which categories they cover, and how they were selected?
• Will Conway publish the harness, prompts and per-task outputs so others can reproduce the 88 vs 84 result?
• How much do scores vary across repeated runs or seeds, given the 4-task margin?
• How does Saluki score on the full official BFCL v4 suite with the public bfcl-eval package?