Choose the middle way: supporting coding experiments according to their particular context characteristics

Abstract

Experiments play an essential role in evaluating solutions in software engineering. A field of software engineering where experiments are frequently used is software development. In this field, many solutions are proposed to foster coding activities, such as different programming languages, developing techniques, tools, and other solutions. In this context, this research has two primary goals. The first goal is to investigate experiments performing coding activities (Coding Experiments). This investigation shall raise the most common characteristics of such experiments, and how current solutions supporting experiments address such coding context. The second goal is to propose a solution to support coding experiments according to their particular context characterization. This research was divided in many sub-steps. Each sub-step adopted distinct method. In the first step, we conducted a systematic analysis of coding experiments published in seven renowned venues in software engineering from 2003 to 2016. In the next step, we systematically evaluated the current solutions to support experiments in SE according to previous findings. Based on the results from previous steps, we proposed our solution. Finally, in the final step, we carried out a case study on replicating coding experiments with the proposed solution. The first study revealed many issues in coding experiments that can be addressed to aid its execution. The second study brought to light which aspects of coding experiments are covered by current solutions. In fact, the majority of general characteristics of coding experiments are adequately addressed by current solutions. However, some context-specific characteristics are not satisfactorily undertaken. Based on found lacks, a metamodel was proposed to specify context-specific coding experiments characteristics. This metamodel can be seen as a specialization of current solution focusing only on coding context characterization. Also, a set of tools were developed to (i) specify models according to the proposed metamodel and (ii) support the experiment execution according to its specification. The last study assessed the proposed solution usage to help researchers carrying out coding activities in experiments. From experiment planner’s perspective, the effort to conduct and collect data was reduced, even considering the extra effort to specify the coding experiment. From participants’ perspective, the proposed solution seemed reasonable to support experiments. However, some issues were identified. Finally, although positive results, performing more assessments including different settings is required to generalize these results. This research focuses only on supporting coding experiments, more precisely planning and execution phases. This work does not deal with other aspects such as data analysis, and we believe current solutions can deal with them. By specifying coding context characteristics, many activities when carrying out a coding experiment can be (semi-)automated, thus contributing to reduce effort to experiment. Moreover, the proposed solution proved adequate for supporting coding experiments, and it is available to support researches around the world through our repository.