Multimodal Data-Driven Approaches for Inferring Drowsiness from Machine Learning Models in Industrial Environments with Critical Safety Systems

Abstract

Catastrophic accidents have been an issue in complex industries like oil and gas (O&G), chemical, and nuclear sectors, despite ongoing efforts to improve safety. While physical systems have advanced, human factors such as fatigue, drowsiness, and inattention remain significant risks, leading to reduced performance, errors in judgment, and an increased likelihood of accidents. Fatigue-related factors—poor rest, sleep deprivation, night shifts, stress, and prolonged monotony—are common in safety-critical environments and frequently result in drowsiness and lapses in attention. However, the subjective nature of self-reported drowsiness presents a challenge in detecting early signs to reduce potential risks and prevent accidents in organizations with high safety and environmental demands. Thus, this thesis presents an all-encompassing framework addressing operator performance and attention-related challenges in safety-critical industrial systems through several key contributions. First, it explores the application of machine learning (ML) and quantum machine learning (QML) for electroencephalogram (EEG) signal analysis, leveraging ensemble models and advanced neural network architectures to improve accuracy in detecting drowsiness. The introduction of variational quantum algorithms applied to EEG data analysis, which highlights quantum computing’s potential to process large, complex datasets in industrial safety contexts, emerges as one of novel contribution of this work. Second, the thesis proposes a data fusion approach that combines physiological and visual (EEG and facial) data to enhance the robustness of drowsiness detection systems. This fusion is implemented at both the decision and feature levels, with experimental results showing significant improvements in recall and accuracy compared to single-modality approaches. Third, the development of a real-time web-based application, DrowsinessNET, integrates the detection model into a practical tool for monitoring drowsiness in high-risk environments. This application highlights the feasibility of applying advanced detection models in real-world scenarios. Finally, a simulator-based experiment was conducted to assess operator performance in automated O&G operations, particularly focusing on the impact of automation-related factors such as overconfidence, boredom, and inattention. The experiment reveals that automation can induce human errors and reduce attentiveness in monotonous tasks, further emphasizing the critical need for integrating human reliability technologies in safety-critical systems. Thus, this thesis pushes the boundaries of research field in human performance and operational safety by introducing multimodal data-driven models (ML/QML/DL), data fusion techniques, and practical applications to prevent accidents and enhance safety in high-risk industries.