Human Perception vs Quantitative Metrics

This is a follow-up study to the work presented in EUSIPCO 2016 [1]. In this study, we compare the quality of harmonic signals extracted with 2 harmonic-percussive separation algorithms, using listening tests and quality metrics obtained with the PEASS.2.0.1 toolkit (both BSS and PEASS).

Dataset

The same data set as the one used in our previous work [1] was used. It is composed of 10 music signals obtained from multi-track recordings.

Algorithms

In this study, the quality of two separation algorithms was evaluated under the common dataset: alg1 is a variation of [2] with an additional post-processing step based on median filtering, alg2 as presented in [3].

Listening Tests

Similar to [1], a MUSHRA listening test was conducted to evaluate perceptual quality of the harmonic signals extracted with the two algorithms. Participants were asked to rate the quality of the signals based on four dimensions: overall quality, interference, artifact distortions, and target distortions. The perceptual ratings and audio files can be downloaded in the following link:

Results

The following figures illustrate the results obtained from the listening tests and the quantitative metrics. All quality metrics, including BSS and PEASS, were obtained with the implementation included in PEASS.2.0.1.

1. Overall Quality

(a) Listening Tests
(b) BSS
(c) PEASS

Figure 1: Overall quality scores obtained with the three methods: (a) Listening Tests, (b) BSS, (c) PEASS

2. Interference Distortions

(a) Listening Tests
(b) BSS
(c) PEASS

Figure 2: Interference scores obtained with the three methods: (a) Listening Tests, (b) BSS, (c) PEASS

3. Artifact Distortions

(a) Listening Tests
(b) BSS
(c) PEASS

Figure 3: Artifacts scores obtained with the three methods: (a) Listening Tests, (b) BSS, (c) PEASS

4. Target Distortions

(a) Listening Tests
(b) BSS
(c) PEASS

Figure 4: Target scores obtained with the three methods: (a) Listening Tests, (b) BSS, (c) PEASS

References

[1] Estefanía Cano, Derry Fitzgerald, and Karlheinz Brandenburg, Evaluation of quality of sound source separation algorithms: Human perception vs quantitative metrics, in 24th European Signal Processing Conference (EUSIPCO).

[2] Estefanía Cano, Mark Plumbley, and Christian Dittmar, Phase-based harmonic/percussive separation, in 15th Annual Conference of the International Speech Communication Association (2014), Interspeech, 2014

[3] Derry FitzGerald, Antoine Liutkus, Zafar Rafii, Bryan Pardo, and Laurent Daudet, “Harmonic/percussive separation using kernel additive modelling,” in Proceedings of the Irish Signals and Systems Conference, 2014.