It evaluates generated memory by whether it improves an agent’s solve rate on downstream tasks, rather than by recall of facts from prior context.
Agents receive transcripts from earlier task attempts, including successful and failed ones, and generate memory from them. FrontierMemory then tests that memory on coding and knowledge-work tasks drawn from public benchmarks including HarborIndex, DeepSWE, and SkillsBench. Some tasks are modified so that required knowledge exists only in the supplied transcripts. Web browsing and online solution lookup are not allowed; memory generation is capped at six hours.
The current leaderboard lists Opus 5.5 and GLM-5.3 at 73.3%, with average task costs of $2.80 and $3.58. GPT-6.1 Sol is listed at 60.0% and $0.77. For harnesses, Letta Dream is first at 73.3% and $2.80; Prime Agent is second at 60.0% and $3.74.
The published ranking uses the higher downstream solve rate from Opus 5.5 and GPT-6 Sol at maximum reasoning. The source does not establish that these results generalize to arbitrary production workloads, nor does it provide an independent validation of the benchmark design.