From detecting distraction to predicting driver intent, hand-tracking is becoming an increasingly valuable part of the intelligent cockpit.
As in-cabin monitoring evolves beyond traditional gaze and facial analysis, advances in computer vision are unlocking new ways to understand driver behaviour and create safer, more intuitive human-machine interfaces. In this interview, Agnes Jernström, Senior Software Engineer at Neonode, explains how hand-tracking is helping to advance occupant monitoring and the next generation of in-cabin AI.
1. Which driver intentions can be reliably inferred through hand-tracking alone?
At Neonode we believe that the best systems are built on several sources of information, rather than looking at just one behavioural signal in isolation. Hand-tracking is one more piece of this puzzle. With that said, hand-tracking alone is very powerful in intercepting sources of distraction.
Crashes due to driver distraction often occurs because the driver was on their phone or engaging with some other object not related to driving. These types of behaviours, where the driver is often mentally disengages from the driving task, can be efficiently detected by studying the driver’s hands, even when their eye gaze behaviour cannot be detected.
For automated driving systems, analysis of the driver’s hands can also help in anticipating an overriding steering input from the driver and provide the context for understanding if the input from the driver was intended or an unintentional effect of the driver’s engagement in a non driving related task. Beyond intentions, we also see indications of hand-tracking providing useful insights to understanding driver states such as cognitive load and drowsiness.
2. How does hand-tracking compare with gaze and facial monitoring systems?
Hands have a larger variability in their appearance than eyes because of the different tasks and gestures that people can perform with them, and also have a wider span of possible positions than the driver’s head. This makes the analysis of hand behavior a more complex task, but using modern computer vision solutions like Neonode’s MultiSensing platform, hand-tracking can be performed with a single occupant monitoring camera.
3. What challenges arise when interpreting natural driver behaviour?
All behavioural patterns have a degree of individual and cultural differences which the interpreting systems have to account for. When designing the systems you will always be faced with the challenge of increasing the detection rate while simultaneously minimizing false positives. To detect and counteract potential corner cases, we continuously collect real world driving data for the verification of new functions.
4. How could intent prediction improve future HMI systems?
For HMI development, knowing that the driver is about to use the infotainment screen can be used to develop smarter, less distracting interfaces. Selection can be made faster and easier by presenting the most relevant information for the situation or enlarging the most commonly used buttons as soon as the driver reaches for it, before any touch screen input is given.
Understanding whether it’s the driver or the passenger who are performing the interaction is also crucial information for tailoring the infotainment experience. Passengers can be allowed to perform more tasks on the touch screen than driver while the car is in motion, ensuring both safety and convenience.
Interested in exterior vehicle sensing technology?
With a pass to InCabin Europe, you’ll also get full access to our
co-located sister event, AutoSens. Find out more here >>
