Quantifying sensor simulation quality with Prof Dr Alexander Braun, University of Applied Sciences, Düsseldorf

1. How can the fidelity of in-cabin simulations for passenger vehicles be quantitatively assessed in terms of replicating real-world environmental conditions such as lighting, sound, and temperature variations? 

Quantitatively measuring the fidelity of simulations is in general a hard problem, and very application specific. For sound you’ll have completely different metrics then for light, and temperature is an independent dimension that influences the physical sensor that in reality measures light (like a camera) or sound (like a microphone). Nonetheless in my research group we developed a guideline we believe to be quite practical, and broadly applicable at that. We call it ’simulating tests to test simulation’, and the gist is simple: reproduce in simulation a typical laboratory setup to measure a camera or a microphone, and then evaluate the simulation results just like you would a real product. Thus we use all the knowledge the community — both industry and academia — has developed over the decades in measuring and quantifying products, and apply it to simulation quality. In practice this has of course many caveats, as the ’syn-to-real’ gap — the always remaining rest difference between simulation and reality — does not simply go away. But using our approach we can turn this into an advantage: now we can quantify the syn-to-real gap using established metrics, and know exactly where our simulation model is lacking, which then immediately tells us how to improve those simulation models. 

 2. Why is it important that we standardise metrics for evaluating the accuracy and reliability of human-machine interaction simulations within the cabin, particularly concerning driver and passenger behaviour under various scenarios? 

While measuring the quality of a — say — camera system (like we do in the IEEE P2020 standard on automotive image quality) is already hard, trying to standardize human interactions is even harder. The camera is at least definable in technical terms, where voltages and currents can be measured. Human behavior in the car, for the driver and for passengers, obviously reflects the variety of all us individuals that make up humanity, that’s a tall order. So we do what has been shown to work in the past, by defining and standardizing scenarios that cover base functionality and typical human reactions. Two current examples are the EuroNCAP test suite which now comprises pedestrians and bicyclists crossing a street, or the UN R157 which defines some dozen test scenarios for an Lane-Keeping Assistance System (LKAS). In all cases a standardized behavior of the traffic participants is given. So really, without standardizing a sort of ‚mean behavior‘ and typical scenarios that define relevant tests to be performed for in-cabin applications, there is no chance of reaching robust and repeatable performance of those functions. In my view these standards are an absolute necessity going forward, as exactly these functions become more and more safety relevant, from SAE Level L2 and the first implementations of L3 to even higher autonomy.  

3. Why are forums like InCabin important for engineers working in these fields, and what do you hope participants will take away from your session? 

Both AutoSens and InCabin are special conferences, situated half-way between an industry fair and an academic conference. This is where those engineers meet and interact that actually implement all the technical products that lead to novel applications and solutions, helping everyone look outside their own four walls and sometimes comfort zones. This is how we learn and grow. And combining my academic teaching with my industry experience that is what I hope the participants can take away from this session: a better knowledge of the underlying theory, grounded in exactly those tasks that need to be solved today.