SensAI develops methods and driver-monitoring systems for measuring cognitive distraction

Detecting cognitive distraction in vehicles

Cognitive distraction poses a particular challenge to traffic safety: Drivers may be looking at the road but still fail to consciously perceive relevant situations. Conventional driver monitoring systems primarily detect individual visible characteristics or short-term driver states. However, this information is insufficient for detecting complex, multidimensional states such as cognitive distraction.

In the “SensAI” research project, the Fraunhofer Institutes IOSB, IMS, and IDMT are therefore developing a multimodal AI approach that combines various types of information about the driver, the vehicle, and the environment. The goal is to reliably detect cognitive distraction and thereby create new possibilities for assessing driver states and developing future driver monitoring systems.

New approaches to driver state detection

To detect complex driver states such as cognitive distraction, SensAI relies on multimodal AI. Camera, audio, and vehicle data are processed via specialized encoders and merged for joint analysis in a multimodal AI system. Rather than examining individual characteristics—such as gaze behavior—in isolation, this approach combines various types of information about the driver and the driving situation.

Fraunhofer IOSB contributes its expertise in computer vision and activity recognition. Fraunhofer IMS supplements the analysis with vital signs—such as heart rate and respiratory rate—captured non-invasively from camera data. Fraunhofer IDMT expands the multimodal system to include acoustic information.

Contribution by Fraunhofer IDMT: acoustic event and anomaly detection

In this project, Fraunhofer IDMT is developing methods for detecting and classifying relevant sound events and acoustic anomalies. These include warning signals as well as spontaneous, distracting noises inside and outside the vehicle. Spoken information and the acoustic context of the driving situation are also taken into account.

The information obtained is fed into the multimodal AI system along with visual data, vital signs, and vehicle data. In this way, acoustic analysis helps to map various factors influencing the driver’s condition and to capture cognitive distraction more comprehensively.

Project goal: demonstrator and validation dataset

SensAI is currently in the development phase. The participating institutes are establishing the sensor technology and data processing for multimodal driver state detection and preparing for integration into test vehicles. The key project deliverables are an integrated demonstrator, a multimodal validation dataset, and an empirically supported measurement definition for cognitive distraction.

This will enable the systematic development, comparison, and testing of driver state detection systems. For vehicle manufacturers, suppliers, and testing organizations, this provides opportunities for collaboration in areas such as sensor integration, AI models, data collection, and validation.