Amazon Science’s PatientAgentBench is a useful benchmark because it stops treating health AI as a single answer.
The framework generates a synthetic patient health record, a realistic clinical vignette, and a patient agent that talks with the health AI system being evaluated. The evaluated system is also an agent: a base model plus a harness that reasons over patient context and uses stateful simulated healthcare tools.
That matters because patient-facing healthcare agents do not only answer medical trivia. They schedule, triage, manage prescriptions, gather missing information, and decide when escalation is required. Static medical exams and short clinician-facing exchanges miss that workflow shape.
PatientAgentBench scores conversations across six clinician-vetted dimensions: clinical safety, triage quality, workflow accuracy, task completion, clinical helpfulness, and conversational quality. An LLM-as-a-jury panel returns scores and explanations, and Amazon says licensed clinicians validated a shared sample of conversations.
The failure patterns are the part to watch. More capable models narrow gaps but do not close them. The benchmark found recurring clinical shortcomings such as omitting crisis resources in an emergency, fabricating clinical information, citing fake providers, or claiming tool actions that never ran. The hardest cases were not always obvious emergencies; routine administrative requests from clinically complex patients could hide real risk.
For New Runtime, this is the same verification problem in a higher-stakes domain. The agent’s final answer is not enough. The workflow needs state, tools, escalation rules, and an audit trail that proves what the system actually did.