---
type: "post"
id: "scale-labs-leaderboard-remote-labor-index-rli-4e7b47"
slug: "scale-labs-leaderboard-remote-labor-index-rli-4e7b47"
title: "Scale Labs Leaderboard: Remote Labor Index (RLI)"
description: "Scale Labs’ Remote Labor Index leaderboard lists Fable 5.2 at a 41.30% automation rate, the highest score shown, against 21.25% for Opus 5.5 and 20.83% for GPT 6 Astra."
retrieval_nugget: "Scale Labs’ Remote Labor Index leaderboard lists Fable 5.2 at a 41.30% automation rate, the highest score shown, against 21.25% for Opus 5.5 and 20.83% for GPT 6 Astra. The result matters because it measures whether agents can deliver work judged acceptable against professional human outputs—not just solve isolated benchmark tasks."
published_at: "2026-11-11"
updated_at: "2026-11-11"
record_date: "2026-11-11"
date_kind: "published_at"
topics: ["evals","automation","knowledge-work"]
entities: ["Scale AI","Scale Labs","Remote Labor Index","Fable 5.2","Opus 5.5","GPT 6 Astra","Upwork"]
related_patterns: ["agent-economics-moves-to-completed-work"]
novelty_label: "new_fact"
what_changed: "Fable 5.2 completed 41% of commercial projects at a freelancer-level standard on the Remote Labor Index, adding a new result on real-world end-to-end work."
prior_refs: ["https://newruntime.com/posts/claude-fable-5-2-is-now-available-08970e/"]
source_url: "https://labs.scale.com/leaderboard/rli"
source_urls: ["https://labs.scale.com/leaderboard/rli"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:scale-labs-leaderboard-remote-labor-index-rli-4e7b47"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/scale-labs-leaderboard-remote-labor-index-rli-4e7b47/","markdown":"https://newruntime.com/posts/scale-labs-leaderboard-remote-labor-index-rli-4e7b47.md","json":"https://newruntime.com/posts/scale-labs-leaderboard-remote-labor-index-rli-4e7b47.json"}
markdown_url: "https://newruntime.com/posts/scale-labs-leaderboard-remote-labor-index-rli-4e7b47.md"
---

# Scale Labs Leaderboard: Remote Labor Index (RLI)

## Retrieval answer

Scale Labs’ Remote Labor Index leaderboard lists Fable 5.2 at a 41.30% automation rate, the highest score shown, against 21.25% for Opus 5.5 and 20.83% for GPT 6 Astra. The result matters because it measures whether agents can deliver work judged acceptable against professional human outputs—not just solve isolated benchmark tasks.

## Summary
Scale Labs’ Remote Labor Index leaderboard lists Fable 5.2 at a 41.30% automation rate, the highest score shown, against 21.25% for Opus 5.5 and 20.83% for GPT 6 Astra. The result matters because it measures whether agents can deliver work judged acceptable against professional human outputs—not just solve isolated benchmark tasks.

## What happened
Scale Labs describes the Remote Labor Index (RLI) as a benchmark for AI agents performing economically valuable remote work. Its 240 projects were sourced from experienced freelancers on Upwork and span 23 domains. Scale says it began with 550 candidate projects, then filtered them for completeness, reproducibility, and professional quality. The projects were anonymized, with personal information removed or replaced by synthetic alternatives.

Each project includes a brief, input files, a professionally accepted human deliverable, and economic data on human completion time and cost. Across the dataset, Scale reports $143,991 in total economic value. Average human completion time is 28.9 hours, with a median of 11.5 hours; average project value is $632.60, with a median of $200. Deliverables can involve multiple files and formats, including documents, audio, video, 3D models, and CAD.

The leaderboard’s primary measure, Automation Rate, is the percentage of projects for which an agent’s deliverable is judged at least as good as the human reference—work a “reasonable client” would accept. Reviewers compare the agent’s output with the professional deliverable and score it as failing, meeting, or exceeding the standard. Scale says three independent evaluators contribute to a majority vote and reports 94.4% inter-annotator agreement for this metric. The leaderboard excerpt lists Fable 5.2 at 41.30%, Opus 5.5 at 21.25%, GPT 6 Astra at 20.83%, Fable 5.1 at 17.92%, and Gemini Argon 4 at 17.08%.

Scale says the quantitative leaderboard uses a private set of 230 projects; 10 projects and an open-source evaluation platform are publicly released for qualitative analysis. Agents were tested in CLI and computer-use environments, with the best result for each agent reported regardless of environment. The common setup included standardized prompts and tooling for multimedia tasks. Each task had a maximum generation budget of $30. Scale also reports relative Elo scores, with a human baseline fixed at 1,000, and measures dollars earned and cost savings. The leaderboard excerpt does not show those figures for the listed models.

## What the archive adds
New Runtime’s earlier coverage on [Fable 5.2’s availability in Devin](https://newruntime.com/posts/claude-fable-5-2-is-now-available-in-0cc74d/) reported Cognition’s FrontierCode 2.0 result of 49.1%. The RLI result is a new fact about the same model, on a different benchmark: it evaluates project deliverables against freelance work rather than coding performance. This is an update to a known model story, not a restatement.

## Trend signal
The result supports the shift toward measuring agent progress by completed work rather than token or model-call cost. The relevant pattern is [agent economics moves to completed work](https://newruntime.com/patterns/agent-economics-moves-to-completed-work/): RLI’s automation rate asks whether a deliverable meets a client-facing standard, while its dollars-earned and cost-savings measures connect performance to economic value. Those additional measures matter; a high acceptance rate alone does not establish that a system is cheaper or profitable to deploy.

## Why now
Scale presents RLI as a way to track progress that isolated research benchmarks may miss: agents’ ability to handle end-to-end, multi-file projects. The leaderboard says the result is a snapshot of its current agent evaluations, and it reports that newer frontier models tend to rank higher on relative Elo even while absolute automation remains limited.

## Why it matters
For teams evaluating agents, the headline number is evidence of benchmark performance, not a hiring plan or a forecast that a similar share of jobs can be automated. A project counts only when reviewers judge its deliverable at least as good as a professional reference. The distinction matters because a system can produce something plausible while still missing files, breaking a format, or failing the brief.

The benchmark’s breakdown of failure modes reinforces that point. Scale reports poor quality, incomplete deliverables, technical or file issues, and inconsistencies across files among common problems. These are precisely the failures that can turn a promising demo into costly review and rework. The RLI also shows why evaluation should be tied to real outputs: projects include varied media and domain-specific work that cannot be reduced to one automatically graded answer.

Builders can use the leaderboard to form hypotheses about which agents merit hands-on testing, but not to select a production system on rank alone. Compare tools on representative tasks from your own workflow, inspect complete deliverables, and include human review time, retries, and failure costs. Scale’s own cost metric treats a failed project as yielding zero savings, a useful reminder that cheap attempts do not imply cheap completed work.

## Limits and uncertainty
The leaderboard reports no confidence interval for Automation Rate, so the precision implied by decimal scores should not be mistaken for a quantified margin of certainty. Scale says its 230-project quantitative set is private, which limits independent reproduction of the official rankings. The evaluation depends on trained human reviewers; Scale says this makes assessment of complex work possible but costly and slow to scale. Its dataset also excludes physical labor, direct client interaction, and long-term work such as SEO. The listed automation rate does not establish deployment cost, time saved in a live workflow, or performance on work outside the benchmark. Scale’s page does not provide a breakdown of Fable 5.2’s own results by project category in the leaderboard excerpt.

## Reader takeaway
Fable 5.2’s 41.30% RLI score is a notable result on a benchmark built around professional deliverables, not proof that the model can replace a freelancer across the board. Treat it as a reason to run controlled trials on your own multi-step work—and judge complete, reviewable outputs, not the leaderboard rank alone.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- topic hub: [Automation](https://newruntime.com/topics/automation.md): Explore the automation topic hub.
- topic hub: [Knowledge Work](https://newruntime.com/topics/knowledge-work.md): Explore the knowledge-work topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)