METR measures task difficulty, not autonomous runtime
METR’s Task-Completion Time Horizons page defines an agent’s time horizon as the human-expert duration of tasks at which the agent is predicted to succeed at a given reliability. The 50% horizon is the task duration at which the fitted estimate predicts success half the time. METR reports a predicted 50% horizon of 17 hours for GPT-4, while warning that measurements above 16 hours are unreliable with its current task suite.
The evaluation draws on software engineering, machine learning and cybersecurity tasks, primarily self-contained tasks with clear success criteria that can be automatically evaluated. METR estimates human completion time mostly through contracted experts’ successful completion times. The page explicitly distinguishes task difficulty from how long an agent can act autonomously: the horizon is not a claim that an agent runs continuously for that duration.
The results are evidence about performance on this task suite under its measurement assumptions. They do not establish that agents can independently conduct end-to-end AI research, nor that benchmark tasks capture the context and tacit knowledge of ordinary professional work.