Evaluating LLMs for multimodal GUI test generation in Android applications

Abstract

Graphical User Interface (GUI) testing is a fundamental task in mobile application development, as it ensures that the user interface of any mobile application behaves cor rectly and meets user expectations. However, when performed manually, GUI testing remains time-consuming. With the rise of Large Language Models (LLMs), there is in creasing interest in exploring their potential to automate software development tasks, including the generation of GUI tests. This study investigates how LLMs can generate GUI test intentions and scripts for Android applications using multimodal inputs, such as screenshots and structured UI data, which provide both visual and semantic informa tion about the interface. This work present an approach that combines these inputs from open-source Android apps and evaluate the performance of four LLMs, including three proprietary models and one open-source model. The results show significant differences among the models, where the Claude 3 Sonnet model produced the most complete results, GPT-4o generated smaller tests focusing on essential flows, while Gemini 2.5 Pro and the open-source Gemma 3 model presented similar results, limiting themselves to basic in teractions. Overall, the results demonstrate that LLMs models show potential to reduce manual effort and increase productivity in GUI test creation, offering distinct benefits according to the model used.