We were joined at InCabin in September by the HELIAUS Project, as a Signature Sponsor. The project was funded to develop smart thermal perception systems that will detect Long Wave Infra-Red light (LWIR) both for in-cabin passengers monitoring and for the car surrounding. Now coming to its completion, we catch up to see the results.Â
Tell us about the HELIAUS project, what is the goal and how did it start?
The future of smart mobility is emerging as a more effective, cleaner, and safer solution for intelligent traffic management. However, it requires adaptable and reliable perceptual systems monitoring inside and outside the cabin. The improvement of perceptual systems in all weather conditions is instrumental for safety. Thermal sensing offers additional information vital for autonomous driving.
The EU-funded HELIAUS project aims to develop smart thermal perceptual systems to effectively control the inside and outside of the cabin. It will create low-cost and innovative technology and will validate the prototypes in perceptual systems applications. It will measure the added value of thermal sensing and promote the advantages that such systems can offer for autonomous driving.
HELIAUS stands for tHErmaL vIsion Augmented awarenesS. It formally started in April 2019. Before kick-off it required one year of collective work with the consortium to submit a proposal and being awarded by the funding authority, ECSEL JU.
Who is involved and how were participating companies selected?
The HELIAUS partners cover the full competences chain, from technology development, over manufacturing and assembly, to use-case evaluation. It also covers different entity profiles as universities: National University of Ireland Galway and Université de Rennes- Centre National de la Recherche Scientifique, Research institutes as Commissariat à l’Energie Atomique TECH, Fraunhofer Institute for Production Technology and Office National d’Etudes et de Recherches Aerospatiales, start-ups as Another Brain and Next2U and companies as Umicore, Denso, Xperi and Lynred.
Lynred were seeking partners in 2018, and brought together a consortium. Thanks to networking they created a shared project on thermal imaging, and have written a project proposal and has submitted it to a project call. A jury from ECSEL JU awarded it at the end of 2018.
Is there an advantage to LWIR over SWIR?
SWIR and LWIR are different electromagnetic wave bands, based on different physics (considering ADAS use-cases like street scenarios).
SWIR is adjacent to visible spectrum and so rely mainly on external illumination and reflection. It can be a passive system thanks to sun or streetlight but with a low reliability. For ADAS scenarios, an active illumination is mandatory to guarantee system performance but raise eye safety concerns. Hence, system cost will be important, due to illumination and sensor cost. In this band, it is possible to make LiDAR sensor with point clouds or active camera with image sensor array.
On the other hand, LWIR band relies mainly on self-emitted light from the scene. It means that it is purely passive and saves the cost of illumination. It is also immune to conditions like lighting and in some cases weather conditions. In this band, passive camera with image sensor array is the norm.
Cameras of both of these bands will generate images that are possible to fuse with RGB images or RADAR.
We know that the benefit to IR is that it doesn’t require lighting. What information does IR give us about the inside of the vehicle?
What applications does IR enable?
Correct, the scene emits naturally thermal information. The intensity of the signal depends on temperature and emissivity. The thermal resolution of the camera is often below 0.1°C, meaning that it can detect a small temperature difference.
For In-cabin application, to enable Thermal comfort estimation or improve Driver and passenger Monitoring features like drowsiness, emotion detection and drunkenness, this performance is used. Improvement is coming from the fact that the body is emitting specific thermal information related to those applications earlier than in the RGB or NIR band. Then the system is more accurate and reactive.
What are the unique challenges for computer vision algorithms when it comes to LWIR?
Basically, computer vision methods and algorithms designed for visible images like YOLO or ResNet work well on LWIR images. The biggest issue is the availability of training and test datasets. RGB imaging is the most used imaging technology in the world. It is the human eye band, where the markets demand is high. High volume industries such as consumer and smartphone contribute to spread it. Historically, small markets were using LWIR imaging, leading to a lack of data. This is why HELIAUS project has created its own datasets thanks to Denso, NUIG, Xperi and Next2U to get enough data to train and test AI and demonstrate the added value of thermal.
What is the biggest obstacle in getting LWIR to market?
For civil use – thermography, firefighting and leisure are using LWIR. These require a volume lower than automotive markets, that leading to a cost considered as prohibitive for ADAS. The goal of HELIAUS project is also to develop new technologies and supply chains that will lower the cost.
Another point is the consideration of LWIR by vehicle sensors design teams as a viable and reliable option for ADAS. Projects like HELIAUS help to demonstrate the added value of LWIR. The technology is also improving quickly to reach automotive expectations.
What are the next steps on the roadmap for the HELIAUS project?
HELIAUS is a research project that is reaching its end (unfortunately). Thanks to the 3.5 years of research, we have results to present to the market. Autosens and InCabin are have matched our timeline perfectly and we were pleased to present them to the community at the events. Partners are also preparing the future and seeking for new consortiums to bring the technology and results of the project to the market.
What does success look like for this project?
Despite the disruption of COVID-19, the project stayed on track and reached the goals fixed at the beginning. LWIR demonstrated a benefit in most of the applications targeted. It was a great source of satisfaction for everyone. From a human point of view, even if we did a lot of meeting remotely it was enriching to learn and work with different people across Europe and beyond.
