What happened
The paper, “Remote Labor Index: Measuring AI Automation of Remote Work,” introduces a benchmark built from end-to-end projects sourced from professionals and freelance platforms. Each project includes a brief, input files and a human-produced deliverable. The authors say the final dataset contains 240 projects spanning 23 categories in Upwork’s taxonomy, with a range of artifact types and work including design, operations, marketing, administration, data and business intelligence, audio-video production, software, and research.
The benchmark is intended to measure whether an AI agent can complete economically valuable work, rather than whether a model can perform an isolated skill. The paper reports that RLI projects took human professionals a mean of 28.9 hours and a median of 11.5 hours. Their average reported cost was $632.60, with a median of $200; in total, the work represents more than 6,000 hours and over $140,000 in value. The authors say the average completion time matches a random sample of Upwork jobs and exceeds comparable prior benchmarks by more than two times.
For quantitative evaluation, the researchers use a private test set of 230 projects and publish 10 projects with evaluation-platform code for qualitative evaluation. Agents receive a project brief, input files and an evaluation compatibility prompt describing supported output formats; the paper does not mandate a particular execution environment or agent architecture. Human evaluators compare agent output with the professional deliverable and judge whether a reasonable client would accept it as commissioned work. Three independent evaluations determine the majority judgment. The paper reports inter-annotator agreement above 94% and explains that the project outputs require manual review because automated evaluation is not currently feasible at this level of complexity.
The paper defines the automation rate as the share of projects on which an agent’s deliverable is judged at least as good as the human deliverable. It reports these results: Manus, 2.5%; Grok 4, 2.1%; Sonnet 4.5, 2.1%; GPT-5, 1.7%; ChatGPT agent, 1.3%; and Gemini 2.5 Pro, 0.8%. The authors also use Elo comparisons to measure relative performance between AI agents, and define a “dollars earned” measure based on the human project value for jobs the agent completes. They describe the benchmark as a basis for tracking automation of remote work, not as a claim that these systems cannot assist people with portions of such work.
What the archive adds
New Runtime’s earlier coverage noted Claude Fable 5.2’s availability in GitHub Copilot and Devin: Copilot coverage and Devin coverage, both dated November 10, 2026. The paper here is dated October 30, 2025, and does not establish a new Fable 5.2 launch or provide results for that model. This is a separate research thread, not a new fact about the availability stories.
Trend signal
This supports the New Runtime pattern “verification bandwidth is the scarce engineering resource”: evaluating finished artifacts across many project types required trained human reviewers, and the paper says automated evaluation was not feasible for this work. The benchmark also makes completion quality, rather than a model’s ability to produce output, the decisive measure.
Why it matters
For teams deciding where to deploy agents, the distinction between assistance and end-to-end automation is operationally important. A system may accelerate research, drafting, coding or file manipulation without consistently producing work that can be handed to a client without substantial repair. The RLI result cautions against translating gains on short, standardized tasks directly into estimates of jobs or projects that can be automated.
The benchmark’s broad project format suggests a practical evaluation discipline: define a real deliverable, give the system the actual inputs, and judge whether a competent customer would accept the result. Teams should still measure their own workflows; the RLI sample is not a universal forecast, and it does not quantify how much time human-AI collaboration can save. But it offers a useful baseline for resisting inflated claims based solely on model scores or intermediate task success.
Limits and uncertainty
The reported rates apply to the agents, project sample and evaluation setup used in the paper; they do not establish performance for every current model, workflow or human-agent collaboration. The benchmark covers 23 categories rather than all 64 in Upwork’s taxonomy, and only 10 projects are public while 230 form a private test set. The results rely on human judgments of client acceptability, although the paper reports above-94% inter-annotator agreement. The paper is dated October 2025, so these figures should not be read as a measurement of later model releases.
Reader takeaway
The paper’s strongest signal is not that AI agents cannot help with remote work, but that full-project automation is a much higher bar than performing component tasks. Measure whether a system delivers work a reasonable customer would accept, using representative projects and explicit review, before turning benchmark progress into an automation forecast.