GeoAgentbench: a dynamic execution benchmark for tool-augmented agents in spatial analysis

Complementing this, a vision-language model (VLM)-based verification protocol is proposed to assess data and spatial accuracy and cartographic style adherence. To support diagnostic comparisons across reasoning strategies, we instantiate four representative agent baselines, including direct tool invocation, ReAct, Plan-and-Solve, and a hybrid Plan-and-React baseline that combines global workflow decomposition with stepwise reactive execution.

Extensive experiments with seven representative LLMs show that GABench can reveal distinct capability boundaries in long-chain orchestration, parameter-level execution, multimodal map-product generation, and runtime error recovery. The results further indicate that the hybrid planning–reactive strategy provides a strong reference baseline under dynamic GIS execution, while also exposing persistent limitations of current LLM agents in complex spatial analysis.

Overall, GABench establishes a reproducible benchmark for assessing and advancing tool-augmented agents in autonomous GeoAI. The source code and dataset are available at: <a target="_blank" href="github.com/GeoX-Lab/GABench">https://github.com/GeoX-Lab/GABench</a>.</p>