The suite contains 50 programming and reasoning tasks across eight categories: 33 code tasks and 17 exact-answer tasks.
A full session runs every task three times, producing 150 independently graded attempts. Code is extracted from responses and run in a separate Node process against tests; answer tasks are normalized and compared with an expected result. The model receives no tools or repository, and no model is used as a grader. Unanswered requests are retried up to five times and leave a session incomplete rather than counting as failures.
Each model’s first complete session is frozen as its 100% reference. NerfBench’s “Power” score combines correctness (60%), output tokens (20%), and cost (20%), relative to that reference. Results between 90% and 110% are considered normal; an out-of-band result requires two consecutive comparable sessions before publication. The source does not disclose the exact prompts, hidden tests, expected answers, or reference implementations, so its results cannot be independently reproduced from the published material alone.