Leveraging diversity to find bugs in JavaScript engines

Abstract

JavaScript is a very popular programming language today with several implementations competing for market dominance. Although a specification document and a conformance test suite exist to guide engine development, bugs occur and have important practical consequences. This work evaluates the importance of different techniques to find functional bugs in JavaScript engines. For that, we explored two existing techniques—test transplantation and cross-engine differential testing. The first technique runs test suites of a given engine in another engine. The second technique fuzzes existing inputs and then compares the output produced by different engines with a differential oracle. We considered engines from four major players in our experiments–V8, SpiderMonkey, ChakraCore, and JavaScriptCore. We present a tool capable of running tests on any javascript engine and obtaining reports based on the test output. It was possible to run the four engines in a test suite extracted from open-source projects, using the two techniques mentioned and we analyzed the behavior of each engine, classifying the output as a bug or not. The results indicate that both techniques revealed several bugs, many of which confirmed by developers. Overall, we reported 50 bugs in this study. Of which, 36 were confirmed by developers and 29 were fixed. To sum, our results show that the techniques are easy to apply and very effective in finding bugs in complex software, such as JavaScript engines.