Subject: From: John Lazzaro <lazzaro(at)BOOM.CS.BERKELEY.EDU> Date: Sat, 19 Dec 1992 10:59:35 PST>David Huron writes >In my opinion, we will need to develop a similar set of benchmark stimuli >for acoustic and auditory scene analysis. I think its important to take a hard look at the effect of benchmarks on research fields before choosing this path. When you allow research to be cast into a set of performance numbers, the evaluation of research ideas and programs can degrade into searching for the performance table and checking it against the current leader. This can quickly quench the exploration of truly new ideas, ideas that would need several years to mature to the point of being able to survive comparison with established methods. And it encourages research which can improve performance in some benchmark by a few percent, without adding any new ideas along the way. In the worst case, the field can be caught in a "local minimum" of paradigms which provide mediocre performance, with the specter of competitive benchmarks blocking the path towards the global minimum indefinitely That said, I think a collection of carefully chosen "hard problems" would be a useful resource for the field. I just caution about letting the solution of these problems dominate the research agenda. --john lazzaro