GUI Agents: From Environment Interaction to Autonomous Planning
Date:
Large language models are moving artificial intelligence from language generation toward goal-oriented task execution. Graphical user interfaces are the common interaction entry point for mobile apps, web systems, and desktop software, and they also concentrate the core capabilities agents need in real environments: perceiving state, choosing actions, using feedback, and handling exceptions. This talk introduces the perception, decision-making, action, and feedback loop of GUI agents. Drawing on Magic-GUI, ToolCUA, and MagicAgent, it discusses interactive environments, GUI–tool collaboration, autonomous planning, and reinforcement learning methods, and analyzes the roles of harness engineering, action alignment, and multi-agent collaboration in improving system reliability, safety, and controllability.