---
type: analysis
slug: research-qwen-s-e-commerce-bench-shows-the-top-earning-agents-are-not-the-safest
title: "Qwen's E-Commerce Bench shows the top-earning agents are not the safest operators"
description: "In Alibaba Qwen's year-long store simulation, GPT-5.6 Sol turned ¥100,000 into about ¥1.43M but sent 18% of its purchasing cash to scam suppliers. Most models never learned to use the prices they had already won. The benchmark belongs to a growing group of year-long business simulations, and it is better read as a change in how agents are evaluated than as a market shift."
published_at: 2026-10-24T05:00:00.000Z
topics: ["agents","claude","memory","research"]
source_urls: ["https://github.com/QwenLM/E-CommerceBench","https://arxiv.org/html/2608.30730v1","https://ecbench.github.io/","https://arxiv.org/abs/2607.28956","https://andonlabs.com/evals/vending-bench-2","https://arxiv.org/abs/2608.30730v1"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
E-Commerce Bench (arXiv 2608.30730, submitted 31 August 2026) comes from Alibaba's Qwen Team and Taobao & Tmall Group, with co-authors at HKUST. Each LLM agent plays a merchant who starts with ¥100,000 and runs up to four online stores for 365 simulated days. The market is built from anonymized real Taobao/Tmall data: 6,886 products and 576 suppliers, 152 of them fraudulent. The agent negotiates with suppliers, sets prices, manages stock, handles returns and keeps cash flowing. The original source was a Telegram post, but the benchmark itself is well documented: there is a paper, Apache-2.0 code and a project site with a leaderboard.

It matters to builders because it tests what single-session agent benchmarks can't: whether an agent stays coherent, solvent and resistant to fraud over thousands of turns, after its context window has overflowed many times. It also follows Vending-Bench, MerchantBench and others, so it should be read as part of a trend in evaluation rather than a one-off launch.

## What we checked

- **The cluster may reveal a broader market or architecture transition that cannot be responsibly reduced to a single release note.** partly confirmed. There is a real pattern, but it is in evaluation methods, not in the market. E-Commerce Bench is one of several 365-day business simulations; the paper itself compares against Vending-Bench 2, YC-Bench, MerchantBench and RetailBench. The shared lesson is about agent architecture: memory, context eviction and cash timing decide outcomes. Nothing here shows agent commerce moving into production. [2][4][5]
- **(Added) Does the top final-assets score identify the best overall operator?** refuted. No. GPT-5.6 Sol leads on assets but ranks 16th of 18 on fraud avoidance, with 18.48% of procurement spend going to fraudulent suppliers. Claude Opus 4.7 has the lowest bad spend (0.12%) and the best negotiation capture, yet ends with only about ¥259k. The authors say no model leads on all seven dimensions. [6][1]
- **(Added) Do model rankings carry over between long-horizon business benchmarks?** partly confirmed. Not reliably. On Andon Labs' Vending-Bench 2, Claude Opus 4.7 ($10,937) ranks above GPT-5.6 Sol ($9,619). On E-Commerce Bench, GPT-5.6 Sol ends with more than five times Opus 4.7's assets. The two benchmarks differ in design, so this shows rankings depend on the environment, not that either benchmark is wrong. [5][1]

## What we found

- The simulation is built to be reproducible. A deterministic negotiation kernel sets every price, and the LLM supplier only writes the dialogue; it cannot change outcomes. Demand comes from a fixed multi-factor model. The paper presents this as its main difference from Vending-Bench, which it calls stochastic and open to manipulation because LLM-driven suppliers there determine outcomes. In the open-source release, the supplier NPC defaults to gpt-4o-mini. [1][2]
- Mean year-end assets over five runs: GPT-5.6 Sol (max) ¥1,431k, Fable5 ¥805k, GPT-5.5 ¥702k (bankrupt in 2 of 5 runs), Claude Opus 4.8 ¥498k, Qwen3.8-Max ¥472k, Claude Opus 4.7 ¥259k. Qwen3.5-Plus went bankrupt in 4 of 5 runs. The project site reports 10 insolvent episodes out of 105 and a 1,264× gap between the best and worst configurations. [1][3]
- The leaderboard has changed since the paper. The paper evaluates 18 models, while the site and README list 21. Qwen3.8-Max and GLM 5.3 were run after publication 'under the same protocol'. The paper's runs used an internal gateway; the public config points to provider endpoints. The best open-weight result is attributed to Qwen3.8-Max-Preview at ¥416k, so builders should note that the authors' own model family gets the open-weight crown. [3][1]
- The weakest area is learning from experience. AnchorRatio compares what an agent pays on repeat orders with the cheapest price it has already won from that supplier, against a shuffled-order baseline. The paper finds 16 of 18 models show no clear downward trend in purchase prices. The site says 18 of 21 models do worse than their own shuffled baseline by more than two standard deviations, and only Qwen3.8-Max-Preview clearly beats it. [2][3]
- The harness explains much of this. All models share a 128k-token budget. When usage reaches 120k, the harness evicts whole tool-call groups, oldest first, and the agent is told about this policy. A persistent memory store holds 20 entries. Across 90 episodes there were 1,495 evictions. The site blames weak learning partly on eviction throwing away tool results that held won prices, and partly on agents barely using memory. Cash failures arise because revenue settles about nine days after a sale. [2][3][1]
- Similar benchmarks give different pictures. MerchantBench (arXiv 2607.28956) runs a 365-day e-commerce simulation on 98,843 real products and finds the best LLM setup reached only 27.3% of human participants' mean final net assets. Vending-Bench 2 scores only the final bank balance from a $500 start, and there Claude Opus 4.7 ranks above GPT-5.6 Sol. The authors themselves list limitations: five episodes per model, fraud detection that depends on generated narrative text, and metrics that report outcomes rather than causes. [4][5][2]

## What it means

If you are building agents that handle money, procurement or operations, the main lesson is that a single "profit" number hides risk. The model that grew assets 14× also gave the largest share of its spend to scammers. A model that barely grew assets was the safest negotiator. So score agents on several axes: fraud exposure (share of spend going to bad counterparties), worst drawdown or solvency, and whether repeat decisions get better over time. Treat these as separate deployment requirements, not as tie-breakers. Ranking differences between E-Commerce Bench and Vending-Bench 2 also suggest testing candidate models in an environment shaped like your own workload, instead of borrowing a leaderboard.

The harness results apply to your systems too. If your context manager evicts old tool output, agents lose the facts they need to improve, such as prices they have already negotiated. Unless you give agents a structured store for those facts and prompt them to use it, they mostly won't. Cash-timing failures likewise argue for hard-coded financial guardrails outside the model. Because the code is Apache-2.0 and the environment is deterministic, you can run it with your own scaffolding, memory design or fraud filters and measure whether they help. That is more useful than the headline rankings, which come from five runs per model and include post-publication rows.

## Still open

- Would the rankings hold with more than five episodes per model, or with a different supplier-dialogue model (the default is gpt-4o-mini)?
- How much of the learning gap comes from the shared 128k eviction harness rather than the models? Would native long-context or memory-native scaffolds change the results?
- Were the post-publication Qwen3.8-Max and GLM 5.3 runs made on the internal gateway or on public endpoints, and has anyone outside the team reproduced any results?
- Does performance in the simulation predict real merchant outcomes on Taobao/Tmall or elsewhere? No real-world deployment data was found.

## Sources

1. [QwenLM/E-CommerceBench GitHub repository](https://github.com/QwenLM/E-CommerceBench) (primary)
2. [E-Commerce Bench (arXiv HTML full text)](https://arxiv.org/html/2608.30730v1) (primary)
3. [E-Commerce Bench project site and leaderboard](https://ecbench.github.io/) (primary)
4. [MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations](https://arxiv.org/abs/2607.28956)
5. [Andon Labs: Vending-Bench 2](https://andonlabs.com/evals/vending-bench-2)
6. [E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation (arXiv abstract)](https://arxiv.org/abs/2608.30730v1) (primary)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Agents](https://newruntime.com/topics/agents.md): Explore the agents topic hub.
- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- topic hub: [Memory](https://newruntime.com/topics/memory.md): Explore the memory topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)