The Hearing Car

Modern vehicles are equipped with numerous driver assistance systems, such as cameras, lidar and radar, which help with parking or keeping in lane. They act as eyes, as it were, and identify relevant objects in the environment. What cars still lack so far is a sense of hearing. However, to monitor closely the traffic situation all around, being able to perceive and categorize external sounds is very important. This is because many events in road traffic announce themselves acoustically, such as an approaching emergency vehicle with its siren.

Essential for autonomous driving: Acoustic event detection

Fraunhofer IDMT in Oldenburg develops AI-based technologies for acoustic event detection as well as methods and concepts for optimal signal capture on the vehicle. Applied in practice, these developments can enable faster reaction times, smart decision-making for drivers and predictive maintenance intervals. The »car’s ears« become even more important in autonomous vehicles. These are expected not only to be able to see at least as well as humans, but also hear.

Danilo Hollosi is our automotive expert and heads the »Hearing Car« team. He explains: »As yet, no systems exist that allow autonomous vehicles to perceive external sounds, despite their high application potential. For example, they signal in a fraction of a second when a vehicle is approaching with its siren going. In this way, the autonomous vehicle knows that it must move aside to make way for the emergency vehicle.«

The car of the future listens

Insights into the research and development around The Hearing Car at Fraunhofer IDMT in Oldenburg

Privacy warning

With the click on the play button an external video from www.youtube.com is loaded and started. Your data is possible transferred and stored to third party. Do not start the video if you disagree. Find more about the youtube privacy statement under the following link: https://policies.google.com/privacy

Danilo Hollosi, group leader Acoustic Event Detection and head of the project The Hearing Car, explains in this video clip why the car's sense of hearing will play an important role in road traffic in the future. In doing so, he gives an insight into the test vehicle of Fraunhofer IDMT in Oldenburg.

The Hearing Car portfolio

  • Use of our »Hearing Car« development environment for client-specific development of hardware and software
  • Acoustic end-of-line testing on short test tracks at the end of vehicle production
  • Assistance with quality assurance through acoustic event detection on the vehicle
  • Siren/alarm signal recognition in various driving scenarios and other early warning systems
  • Platform and component monitoring, e.g. of powertrain, chassis or tyres
  • Recording and interpretation of environmental and vehicle data for dynamic vehicle adaptation (adaptronics)

     

The Hearing Car on the proving ground

Privacy warning

With the click on the play button an external video from www.youtube.com is loaded and started. Your data is possible transferred and stored to third party. Do not start the video if you disagree. Find more about the youtube privacy statement under the following link: https://policies.google.com/privacy

We collected lots of data with The Hearing Car to test our sensor and analysis technologies for various use cases. The scenario of speech recognition outside the car in particular is highly requested among automotive manufacturers and suppliers. We also tested up to which speed we can robustly detect and localize a siren.

Intelligent technologies for specific market requirements

© Fraunhofer IDMT/Anika Bödecker
A rolling prototype at the Oldenburg Branch for Hearing, Speech and Audio Technology HSA enables the collection of test data and the client-specific development of various »Hearing Car« functions.

At the Oldenburg Branch for Hearing, Speech and Audio Technology HSA, experts are working in close cooperation with car manufacturers and suppliers on developing the necessary sensor and analysis technologies for the »Hearing Car«. For this purpose, we are using our own vehicle, which acts as a rolling prototype in the project and enables us to record important training data. In this way, our event detectors can be trained using the acoustic signatures of the relevant sounds and by means of machine learning. The result is intelligent sensor platforms with an efficient detector performance.

In order to make the »Hearing Car« reality, our developers are contributing their own special project experience in the automotive sector as well as cross-group expertise. Apart from the development of AI algorithms for acoustic event detection, the researchers at Fraunhofer IDMT are also getting to grips with other challenges. These include optimal signal capture through sensor positioning, signal pre-processing and enhancement, and suppression of background noise. To minimize the influence of wind noise, the »Hearing Car« team is also, for example, developing and testing suitable housings and shields for sensors to pick up airborne sound. 

The »Hearing Car’s« potential

© iStock/Bruskov
New possibilities in adaptronics: The acoustic signal of a wet road can trigger the automatic adjustment of chassis components.

The contribution made by »hearing for the car« does not end with the recognition of a siren. It is far more intended to ensure greater safety and reliability in road traffic, especially in autonomous driving. Acoustic environment analysis has the potential in future not only to recognize an ambulance but also other acoustic scenarios, such as the sound of playing children when the vehicle turns into a traffic calming zone. Such important information could reach the on-board system even before cameras and radar sensors have recorded the situation. This enables automated driving systems to respond with greater caution according to the situation and thus in a comparable way to the person at the wheel when they hear children playing.

Another objective of The Hearing Car is improve vehicle safety and comfort in general. Acoustic sensors at various points in and on the car supply data to an analysis unit. Recordings from the wheel arch, for example, can indicate road conditions and trigger automatic adjustments to the chassis, e.g. in the area of the suspension or the braking system. In addition, continuous acoustic monitoring of the car’s operating noises can detect damage or wear, enabling an »out-of-round« engine, worn brakes or even a nail in the tyre to be predicted and identified. This enables smart maintenance intervals and repairs before breakdowns or even accidents occur. In addition, communication solutions for voice interaction outside the vehicle provide more comfort and safety, e.g. with a voice-based opening of doors or when an emergency call is made.

Further Information

 

New solutions for the mobility of tomorrow

With our “Hearing for the car”, our "YourSound", checking drivers' attention or monitoring in production, we offer a wide-ranging portfolio for the automotive sector.

 

Press Release / 1.9.2023

Intelligent acoustic sensor systems support The Hearing Car

From September 5 to 8 at IAA MOBILITY 2023 in Munich, researchers from Fraunhofer IDMT in Oldenburg will showcase their latest developments in smart acoustic sensor technology on and in vehicles in Hall B1, Booth D11.

 

Press information / 5.2.2020

The hearing car of the future

Anyone buying a new car today does not have forego features such as remote park assist, lane assist or driver fatigue detection. 

Funding

The Oldenburg Branch Hearing, Speech and Audio Technology HSA is funded in the program »Vorab« by the Lower Saxony Ministry of Science and Culture (MWK) and the Volkswagen Foundation for its further
development.

Data Privacy

Data processing in connection with our hearing car