BASS – Evaluation and analysis of speech interaction with smart speakers

Development of test methods and standard protocols for evaluating speech interaction with smart speakers

 

In the »BASS« project, the technological basis for test methods and standard protocols for evaluating speech interaction with smart speakers is to be created. This should make it possible to offer a measurement system on the market that follows standardized test specifications for smart speakers. For the project, the company HEAD acoustics and the Fraunhofer IDMT in Oldenburg are collaborating to develop the new measurement methods. For this purpose, the Oldenburg Branch Hearing, Speech and Audio Technology HSA subproject draws on a machine-learning-based model developed at the University of Oldenburg for the automatic evaluation of the listening effort of speech, which Fraunhofer IDMT has already further developed and extensively validated. The subproject of the Oldenburg Branch comprises the (further) development of reference-free perceptual models for the instrumental evaluation of user-defined input speech and synthetic output speech in smart speakers.

The partner HEAD acoustics will mainly focus on the simulation of room acoustics, the implementation of the measurement methodology and the system integration in order to build up a unique selling point with »BASS« in the growing smart speaker market in the long run.

An initial test specification for smart speakers has already been standardized, but there are critical gaps here. For example, this specification does not describe how the quality of speech inputs and outputs from smart speakers can be tested. In addition, a simulation of room acoustics for the acoustic test scenario is missing.

With the help of »BASS«, the technological foundations are to be created to be able to offer a measurement system on the market in the future that closes the aforementioned gaps. More and more manufacturers of »classic« sound systems and consumer electronics want to enter the rapidly growing smart speaker market and, to this end, equip their products with voice assistants from well-known service providers such as Google, Amazon or Apple. These in turn naturally attach importance to a certain minimum quality of the components used. Otherwise, any problems that might occur could be perceived by the user as a shortcoming of the service provider used. As is often the case in industry, open standards help here to define clear minimum requirements between manufacturer and supplier in advance of product developments of smart speakers. In addition, the aim is for the new measurement methods developed to find their way into a successor standard so that the measurement system developed is standard-compliant, which would increase its acceptance and demand.