{"type":"post","id":"technical-report-introduces-two-new-benchmarks-geoexpertqueries-and-e1a6b1","slug":"technical-report-introduces-two-new-benchmarks-geoexpertqueries-and-e1a6b1","title":"Technical report introduces two new benchmarks, GeoExpertQueries and GeoKTools, with dev splits on GitHub .","description":"What the benchmarks make visible Thesis Geospatial agent evaluation needs to measure whether a system can execute a defensible workflow not only whether it can produce a plausible answer. The report introduces two complementary benchmarks GeoExpertQueries for expert-grounded open-ended tasks","retrieval_nugget":"What the benchmarks make visible Thesis Geospatial agent evaluation needs to measure whether a system can execute a defensible workflow not only whether it can produce a plausible answer. The report introduces two complementary benchmarks GeoExpertQueries for expert-grounded open-ended tasks","published_at":"2026-10-15","updated_at":"2026-10-15","record_date":"2026-10-15","date_kind":"published_at","topics":["github"],"source_urls":["https://arxiv.org/abs/2610.15750"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:technical-report-introduces-two-new-benchmarks-geoexpertqueries-and-e1a6b1","status":"published","visuals":[],"editorial_provenance":{"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:53dc97220ef09d6742220d7f2365f6ad17931c49865fb5050b471e2404ca1cce","reviewed_at":"2026-10-15T10:05:19.345Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":2,"related_record_count":0},"analysis":{"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Geospatial agent evaluation needs to measure whether a system can execute a defensible workflow, not only whether it can produce a plausible answer.","observed_facts":[{"text":"GeoExpertQueries pairs each problem with a consensus workflow from three independent experts and uses a human-validated, tool-agnostic autorater.","source_urls":["https://arxiv.org/abs/2610.15750"]},{"text":"GeoKTools uses templated queries and deterministic reference solvers, varying the number of required tool types.","source_urls":["https://arxiv.org/abs/2610.15750"]},{"text":"The authors compare four configurations, from a standalone model through tool and skill variants to full GRA.","source_urls":["https://arxiv.org/abs/2610.15750"]}],"mechanism":"The benchmarks separate task structure from the agent’s execution stack: one tests alignment with expert workflows, while the other provides controlled tool-use tests. The reported comparisons then distinguish gains from tools and skills from gains due to planning and execution guidance.","why_now":"As geospatial systems combine foundation models, data discovery and specialized tools, answer-only scores can conceal whether a usable sequence of operations was followed.","implications":["For teams building tool-using systems, benchmark design should make intermediate workflow quality observable alongside final answers.","The report says skills and tools yield the largest step up, while planning and execution guidance add further gains for Flash on open-ended tasks, where it matches or exceeds Pro at lower cost."],"evidence_boundary":"These are results reported by the paper’s authors; this source does not establish independent replication or generalize the findings beyond the tested benchmarks and configurations.","watch_conditions":["Watch for independent replication and for whether the released development splits support reproducible evaluation of workflow quality beyond the authors’ own setup.","The report also notes UnivEARTH saturation, underscoring that benchmark coverage can lose discriminatory value as systems improve."],"related_records":[],"new_branch_reason":"This starts a distinct analytical record because no prior New Runtime record on these specific geospatial benchmarks or their workflow-level evaluation is supplied."},"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/technical-report-introduces-two-new-benchmarks-geoexpertqueries-and-e1a6b1/","markdown":"https://newruntime.com/posts/technical-report-introduces-two-new-benchmarks-geoexpertqueries-and-e1a6b1.md","json":"https://newruntime.com/posts/technical-report-introduces-two-new-benchmarks-geoexpertqueries-and-e1a6b1.json"},"related_patterns":["ai-engineering-evaluations-are-splitting-into-task-specific"],"next_reads":[{"type":"topic","path":"/topics/github/","url":"https://newruntime.com/topics/github/","title":"Github","reason":"Explore the github topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/github.md","json_url":"https://newruntime.com/topics/github.json"},{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}