[no subject]

[John Lazzaro <lazzaro@xxxxxxxxxxxxxxxxxxxx>]

>David Huron writes

>In my opinion, we will need to develop a similar set of benchmark stimuli
>for acoustic and auditory scene analysis.

I think its important to take a hard look at the effect of benchmarks
on research fields before choosing this path.  When you allow research
to be cast into a set of performance numbers, the evaluation of
research ideas and programs can degrade into searching for the
performance table and checking it against the current leader.  This
can quickly quench the exploration of truly new ideas, ideas that
would need several years to mature to the point of being able to
survive comparison with established methods. And it encourages
research which can improve performance in some benchmark by a few
percent, without adding any new ideas along the way. In the worst
case, the field can be caught in a "local minimum" of paradigms which
provide mediocre performance, with the specter of competitive
benchmarks blocking the path towards the global minimum indefinitely

That said, I think a collection of carefully chosen "hard problems"
would be a useful resource for the field. I just caution about letting
the solution of these problems dominate the research agenda.

                                                --john lazzaro