What the benchmarks make visible
Thesis: Geospatial agent evaluation needs to measure whether a system can execute a defensible workflow, not only whether it can produce a plausible answer. The report introduces two complementary benchmarks: GeoExpertQueries for expert-grounded open-ended tasks, and GeoKTools for controlled variation in tool requirements.
Observed facts: GeoExpertQueries pairs each problem with a consensus workflow from three independent experts and uses a human-validated, tool-agnostic autorater. GeoKTools uses templated queries and deterministic reference solvers, varying the number of required tool types. The authors compare four configurations, from a standalone model through tool and skill variants to full GRA. Mechanism: The benchmarks separate task structure from the agent’s execution stack: one tests alignment with expert workflows, while the other provides controlled tool-use tests. The reported comparisons then distinguish gains from tools and skills from gains due to planning and execution guidance.
Why now: As geospatial systems combine foundation models, data discovery and specialized tools, answer-only scores can conceal whether a usable sequence of operations was followed. First implication: For teams building tool-using systems, benchmark design should make intermediate workflow quality observable alongside final answers. The report says skills and tools yield the largest step up, while planning and execution guidance add further gains for Flash on open-ended tasks, where it matches or exceeds Pro at lower cost. Evidence boundary: These are results reported by the paper’s authors; this source does not establish independent replication or generalize the findings beyond the tested benchmarks and configurations.
First watch condition: Watch for independent replication and for whether the released development splits support reproducible evaluation of workflow quality beyond the authors’ own setup. The report also notes UnivEARTH saturation, underscoring that benchmark coverage can lose discriminatory value as systems improve. This starts a distinct analytical record because no prior New Runtime record on these specific geospatial benchmarks or their workflow-level evaluation is supplied.