The review examined the quality of the tests themselves—including topical and linguistic coverage, diversity of formats, difficulty calibration, and methodological soundness—rather than assessing only model outputs. Amazon says AVERI conducted the review without input from Amazon’s evaluation team.
According to Amazon’s account, AVERI validated broad coverage of documented risk categories, varied testing approaches, and low contamination risk. It also identified gaps: multilingual prompts were not evenly distributed across risk categories; the suite needed more multistep adversarial prompts; some time-sensitive references had become stale; and some grading criteria could be clearer. Amazon says it responded by refreshing the dataset with harder scenarios, introducing tiered difficulty levels, expanding multilingual coverage across risk categories, and setting a quarterly review cadence.
The report is a company-authored account of the collaboration. It does not provide the sampled prompts, an independently published AVERI report, quantitative results, or enough detail to assess how representative the sample was. The announcement therefore documents a process change and the company’s description of external feedback, not independently verifiable evidence that the revised evaluations now predict safety in deployment.