The advancement of autonomous driving and the acceleration of vehicle electrification have transformed vehicles from basic transportation means into personal spaces. This indicates that the in-vehicle customer experience could undergo significant changes, requiring a deep understanding of the in-cabin context.

This paper introduces an integrated cabin LMM(Large Multimodal Model) designed to enhance customer experience by understanding and gaining insights into the overall in-cabin context. By utilizing various in-cabin sensors (Camera, Microphone, IMU, Gas Sensor, Radar, etc.), we were able to comprehend complex patterns and interactions that single-modality models cannot achieve. Additionally, our model improves the reliability of existing Driver Monitoring System(DMS) and provides advanced functionalities such as alcohol impaired driving detection and health monitoring.

Also, we discuss how generative AI is utilized in our model’s training pipeline and share a brief overview of the user experience.