A formal approach to test automation based on requirements, domain model, and test cases written in natural language

Abstract

Software testing is a costly and time-consuming activity. For this reason, there is a substantial effort both in the academy and industry to automate it as much as possible. There are several test generation strategies and theories based on formal specifications. Formalisms such as process algebra, transition systems, and so forth, allow a precise defi- nition of the semantics of system requirements, which one can leverage to verify essential properties, such as soundness, and derive test cases automatically. In an industrial con- text, however, ad-hoc/manual strategies are far more common due to their convenience since natural language descriptions are, more likely, easier to understand. Still, the lack of formal rigor can generate inaccurate tests, as there is no verification mechanism to ensure relevant properties. Also, requirements specified by product owners tend to be abstract by design, and should not be changed by other stakeholders, such as test engineers down the line. Then, it is a challenge to generate concrete test cases while still guaranteeing that the original behavior is preserved. Thus, in this work, we promote the use of natural language descriptions with rigorously defined underlying semantics (transparent to the user). Re- garding the scope of this work, we cover the entire traditional (direct engineering) testing process and artifacts, from requirements to automation scripts generated automatically. Requirements written in a controlled natural language are parsed, and their semantics are automatically modeled using the CSP process algebra. To deal with different abstraction levels, from requirements to concrete tests, we formalize the concept of a domain model, in which additional information (such as dependencies, compositions, etc.), also written in natural language, can be combined with the requirements while preserving the original behavior. Then, by considering the domain model, sound and consistent test cases are generated from the discovered scenarios using the cspio conformance relation. These test cases can then be linearized back to natural language to allow manual execution or di- rectly translated into test scripts for automated execution. On the other hand, we should not ignore a recurring scenario in which there is a large number of test cases already gen- erated by hand, potentially created with ad-hoc techniques, without using requirements as input to the generation process. Hence, we also provide assistance to automate and make legacy test cases consistent. Custom tools for test generation, consistency analysis, and automation, were developed to mechanize the entire process, and evaluated in the real-world setting of a partnership with Motorola, a Lenovo Company. As a result, we not only generated and automated the same test cases that were described by hand in retro- spect from old requirements (with more than 90% precision and 80% text size reduction) but also discovered new scenarios and created, from new features, test cases approved to be used in production. The efficiency of the consistency analysis, carried out through the Alloy Analyzer, was also evaluated for real test cases. Although most analyses took less than 10 (ten) seconds, we developed an alternative implementation that exhibited a massive decrease in the analysis time.