[AUDITORY] Registered reports

[Nilesh Madhu <000000405df1884c-dmarc-request@xxxxxxxxxxxxxxx>]

Dear Tim,

I appreciate your initiative towards reproducible research. However I 
fear that registered reports would just add another layer of overhead to 
academics and students already under the pressure to publish. If I 
understand correctly, this involves two rounds of review: a first review 
based on the methodology and evaluation and a second based on the 
results of the research. For each stage, probably at least two review 
rounds would be needed (going by the current publishing cycle). I fear, 
as Gaston does, this might stifle creativity and lead to overwork also 
for reviewers and editors. Of course, this is assuming you want to make 
registered reports compulsory...

Furthermore, such an approach may not be equally applicable to all 
research. For research into algorithms, for example, the value of the 
research lies, usually, in the core idea. There are myriad accepted 
forms of evaluation and to force a strict evaluation pattern/methodology 
would be counterproductive. Reproducible research in this case is 
targeted by encouraging authors to make their code and test data public.

What I would support are (voluntary) guidelines on reporting results of 
experiments. This is often to be found in in the engineering field, when 
one participates in an open challenge.

Lastly, the main reason for this initiative is to avoid 'mis-reporting' 
the results in favour of a hypothesis. Surely, this calls for self 
policing? Aren't we, as researchers, possessed of sufficient integrity 
and ethics to present our research in the correct light? If this core 
value is missing, I fear no external policing is going to help.

Best regards
Nilesh Madhu