VLMs Playing StarCraft II: A Multimodal Decision Benchmark

The provided research introduces VLM-Attention, a novel StarCraft II environment designed to better reflect human perception and decision-making by incorporating RGB visuals and natural language. This framework utilizes vision-language models with specialized mechanisms for unit targeting, knowledge retrieval for tactical decisions, and dynamic role assignment for coordinated multi-agent behavior. Experiments demonstrate that agents powered by these models can perform complex maneuvers without explicit training, rivaling traditional reinforcement learning methods. The work aims to advance human-aligned game AI and provides a new benchmark for multimodal AI in strategic games.