Design recommendations for building teacher-facing learning analytics dashboards for process-oriented feedback in online learning

Abstract

In online learning, feedback is critical in both directions: teachers need as much feedback from students as students need from teachers. In this scenario, feedback can be provided by automatically analyzing the large amounts of interaction data stored in Virtual Learning Envi- ronments (VLEs) databases and communicating the findings through user-friendly computer interfaces, usually called Learning Analytics Dashboards (LADs). LADs can offer different types of feedback, depending on the designer’s goals and theoretical assumptions about how learning occurs; in this work, we focus on behavioral processes-oriented feedback for teachers in online learning. However, few previous works have proposed and validated teacher-facing LADs for this type of feedback, and none of them provides a detailed characterization of how teachers interact and respond to this class of LAD. As a result, there is lack of detailed, empirically validated recommendations to guide researchers and practitioners when building these tools. To fill this research gap, this thesis posed the following research questions: 1) What are the data/task requirements and appropriate visual encodings for building teacher-facing LADs for behavioral process-oriented feedback in online learning? and 2) what types of insights would teachers get from a LAD that provides this kind of feedback, and what pedagogical actions would they take based on these insights? To answer these questions, we used a design-based method to iteratively build and evaluate an instance of such type of LAD. We ran three design iterations and, based on the results of this process, we identified: a set of data and task re- quirements, the strengths and weaknesses of several visual encodings, and the types of insights teachers can get from such dashboards and what pedagogical actions would they take based on these insights. Finally, based on our empirical results, we propose a set of recommendations that can be reused by other researchers and practitioners interested in building similar types of LADs.