History-based prioritization in the context of manual testing : a study in a real industrial setting

Abstract

Many test case prioritization techniques have been proposed with the ultimate goal of speeding up fault detection. History-based prioritization, in particular, has been shown to be an effective strategy. However, most empirical studies on this topic have focused on the context of auto- mated testing. Investigating the effectiveness of history-based prioritization in the context of manual testing is important because, despite the popularity of automated approaches, manual testing is still largely adopted in the industry. In this work, we propose two history-based prioritization heuristics and evaluate them in the context of manual testing in a real industrial setting. We compared our proposed approaches against alternative prioritization strategies, including a state-of-the-art history-based approach, an optimal prioritization, the real ordering followed by the testers, the ordering suggested by a test management tool, and a random or-dering. For our evaluation, we collected historical test execution information from 35 products, spanning over seven years of historical information, accounting for a total of 3,196 unique test cases and 5,859,989 test results. The results of our experiments using historical test execution data from real subjects and with real faults showed that the effectiveness of the proposed approaches is not far from a theoretical optimal prioritization and that they are significantly better than alternative orderings of the test suite, including the state-of-the-art history-based approach, and the execution order followed by the testers during the real execution of the test suites evaluated as part of our study. With respect to efficiency, our proposed approaches yield similar results and they are both better (faster) than the state-of-the-art history-based competitor.