Subject: Re: perceptual audio evaluation tests From: James Johnston <audioskeptic@xxxxxxxx> Date: Wed, 17 Dec 2014 01:32:18 -0800 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>--90e6ba3fd739ddaaa6050a66262c Content-Type: text/plain; charset=UTF-8 The problem I have with MUSHRA is that quality is not a 1-dimensional quantity, and quality is not necessarily transitive, either. On Wed, Dec 17, 2014 at 12:13 AM, Christian Uhle < christian.uhle@xxxxxxxx> wrote: > > Dear Mark, > > I'm not aware of a standardized method for subjective testing (i.e. > listening tests) for source separation. > I recommend a testing similar to MUSHRA, i.e. a multiple stimuli test with > hidden reference and anchor. > > For the reference (and hidden reference) the best option IMO is to start > with signals where the desired signal and the interfering signals are > separately available. The reference signal is then either the desried > signal or as a mixture of the desired and interfering signal where the > interferer is attenuated (in case your separation does not aim at complete > separation but at an enhancement of the desired signal w.r.t the > interferer). > > For the anchor: the standard 3.5 kHz low-pass filtered signal is one > option. Of course, the other conditions (aka processed signals) should not > sound much worse than the anchor. So, the processing for deriving the > anchor signals depends a bit on the conditions under test. Starting with an > oracle mask, introducing degradations to it and computing an output signal > is one option when testing BSS methods that are based on spectral weighting. > Also, having more interefer in the anchor thn in the conditions under test > might be good. > > The main problem IMO is this: > when using a MUSHRA test for accessing the quality of let's say an audio > codec, we often ask for transparency. This is a one-dimensional quantity. > Evaluation of BSS is about a multi-dimensional quantity: 1) reducing the > interference and 2) sound quality are the most important dimensions here. > You can either ask the test listeners for a combined rating (in a > preference test) or you ask for ratings regarding each of the > characteristics separately. > This depends a bit on the aim of the test (e.g. an aim could be comparing > different methods in order to decide which one to buy, or testing during > development for the purpose of tuning). > > Good luck, > Christian > > Mark Cartwright schrieb: > >> Hello, >> >> I'm looking for the list's opinions on perceptual audio evaluation >> listening tests for signals that have large impairments. In particular, I'm >> primarily interested in the evaluation of the output of source separation >> algorithms. What standardized tests do people recommend (e.g. ITU-R >> BS.1534-2 / MUSRHA, ITU-T P.800, etc.) and what are their pros and cons? >> Also, are there other tests that are preferred over these but have not yet >> been standardized? >> >> Thanks! >> >> Mark >> >> -- >> Mark Cartwright <mcartwright@xxxxxxxx <mailto:mcartwright@xxxxxxxx >> northwestern.edu>> >> PhD Candidate in Computer Science >> Interactive Audio Lab <http://music.cs.northwestern.edu/>, Northwestern >> University >> www.markcartwright.com <http://www.markcartwright.com/> >> >> > > -- > Dr.-Ing. Christian Uhle > Senior Scientist > AudioLabs-IIS > Fraunhofer Institute for Integrated Circuits IIS > > Am Wolfsmantel 33 > 91058 Erlangen > Germany > E-mail: christian.uhle@xxxxxxxx > Phone: +49(0) 9131 / 776 - 6230 > Fax: +49(0) 9131 / 776 - 6099 > > http://www.iis.fraunhofer.de/amm/ > http://www.audioblog.iis.fraunhofer.com > > Meet us at these events: > > CES Las Vegas, January 6-9, 2015 > MWC Barcelona, March 2-5, 2015 > NAB Las Vegas, April 13-16, 2015 > ... > -- James D. (jj) Johnston Independent Audio and Electroacoustics Consultant --90e6ba3fd739ddaaa6050a66262c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">The problem I have with MUSHRA is that quality is not a 1-= dimensional quantity, and quality is not necessarily transitive, either.<br= ></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Wed, De= c 17, 2014 at 12:13 AM, Christian Uhle <span dir=3D"ltr"><<a href=3D"mai= lto:christian.uhle@xxxxxxxx" target=3D"_blank">christian.uhle@xxxxxxxx= fraunhofer.de</a>></span> wrote:<blockquote class=3D"gmail_quote" style= =3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear Mar= k,<br> <br> I'm not aware of a standardized method for subjective testing (i.e. lis= tening tests) for source separation.<br> I recommend a testing similar to MUSHRA, i.e. a multiple stimuli test with = hidden reference and anchor.<br> <br> For the reference (and hidden reference) the best option IMO is to start wi= th signals where the desired signal and the interfering signals are separat= ely available. The reference signal is then either the desried signal or as= a mixture of the desired and interfering signal where the interferer is at= tenuated (in case your separation does not aim at complete separation but a= t an enhancement of the desired signal w.r.t the interferer).<br> <br> For the anchor: the standard 3.5 kHz low-pass filtered signal is one option= . Of course, the other conditions (aka processed signals) should not sound = much worse than the anchor. So, the processing for deriving the anchor sign= als depends a bit on the conditions under test. Starting with an oracle mas= k,=C2=A0 introducing degradations to it and computing an output signal is o= ne option when testing BSS methods that are based on spectral weighting.<br= > Also, having more interefer in the anchor thn in the conditions under test = might be good.<br> <br> The main problem IMO is this:<br> when using a MUSHRA test for accessing the quality of let's say an audi= o codec, we often ask for transparency. This is a one-dimensional quantity.= <br> Evaluation of BSS is about a multi-dimensional quantity: 1) reducing the in= terference and 2) sound quality are the most important dimensions here.<br> You can either ask the test listeners for a combined rating (in a preferenc= e test) or you ask for ratings regarding each of the characteristics separa= tely.<br> This depends a bit on the aim of the test (e.g. an aim could be comparing d= ifferent methods in order to decide which one to buy, or testing during dev= elopment for the purpose of tuning).<br> <br> Good luck,<br> Christian<br> <br> Mark Cartwright schrieb:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Hello,<br> <br> I'm looking for the list's opinions on perceptual audio evaluation = listening tests for signals that have large impairments. In particular, I&#= 39;m primarily interested in the evaluation of the output of source separat= ion algorithms. What standardized tests do people recommend (e.g. ITU-R BS.= 1534-2 / MUSRHA,=C2=A0 ITU-T P.800, etc.) and what are their pros and cons?= Also, are there other tests that are preferred over these but have not yet= been standardized?<br> <br> Thanks!<br> <br> Mark<br> <br> -- <br> Mark Cartwright <<a href=3D"mailto:mcartwright@xxxxxxxx" targe= t=3D"_blank">mcartwright@xxxxxxxx<u></u>edu</a> <mailto:<a href= =3D"mailto:mcartwright@xxxxxxxx" target=3D"_blank">mcartwright@xxxxxxxx= <u></u>northwestern.edu</a>>><br> PhD Candidate in Computer Science<br> Interactive Audio Lab <<a href=3D"http://music.cs.northwestern.edu/" tar= get=3D"_blank">http://music.cs.northwestern.<u></u>edu/</a>>, Northweste= rn University<br> <a href=3D"http://www.markcartwright.com" target=3D"_blank">www.markcartwri= ght.com</a> <<a href=3D"http://www.markcartwright.com/" target=3D"_blank= ">http://www.markcartwright.<u></u>com/</a>><br> <br> </blockquote> <br> <br> -- <br> Dr.-Ing. Christian Uhle<br> Senior Scientist<br> AudioLabs-IIS<br> Fraunhofer Institute for Integrated Circuits IIS<br> <br> Am Wolfsmantel 33<br> 91058 Erlangen<br> Germany <br> E-mail: <a href=3D"mailto:christian.uhle@xxxxxxxx" target=3D"_blan= k">christian.uhle@xxxxxxxx<u></u>de</a><br> Phone:=C2=A0 +49(0) 9131 / 776 - 6230<br> Fax:=C2=A0 =C2=A0 +49(0) 9131 / 776 - 6099<br> <br> <a href=3D"http://www.iis.fraunhofer.de/amm/" target=3D"_blank">http://www.= iis.fraunhofer.de/<u></u>amm/</a><br> <a href=3D"http://www.audioblog.iis.fraunhofer.com" target=3D"_blank">http:= //www.audioblog.iis.<u></u>fraunhofer.com</a><br> <br> Meet us at these events:<br> <br> CES Las Vegas, January 6-9, 2015<br> MWC Barcelona, March 2-5, 2015<br> NAB Las Vegas, April 13-16, 2015<br> ...<br> </blockquote></div><br clear=3D"all"><br>-- <br><div class=3D"gmail_signatu= re"><div>James D. (jj) Johnston</div><div>Independent Audio and Electroacou= stics Consultant</div></div> </div> --90e6ba3fd739ddaaa6050a66262c--