Subject: Re: [AUDITORY] stats (mis)use in psychology and hearing science From: Pierre Divenyi <pdivenyi@xxxxxxxx> Date: Fri, 20 Sep 2013 17:34:56 -0700 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>--Apple-Mail-D32972FD-01FC-4620-AD93-61CCA4A9C79C Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Well said. Also, why is the use of nonpametric statistics so rare? They do n= ot require meeting the stiff criteria that e.g. ANOVA does.=20 Pardon the typos, it's Apple's fault. On Jun 24, 2013, at 0:04, Brian Gygi <bgygi@xxxxxxxx> wrote: > =20 > The problem is not just with higher education. As long as one needs a sig= nificant effect to get published, people are going to continue using the eas= iest and most powerfult stats tests out there. Reviewers are complicit in t= his as well - I have rarely seen a paper turned down because of inappropriat= e statistical tests (even though many publications specifically ask about th= is). We as scientists could start by cleaning up our own shop a bit. >=20 > Brian Gygi, Ph.D. > =20 > -----Original Message----- > From: Iftikhar Nizami [mailto:nizamii2@xxxxxxxx > Sent: Sunday, June 23, 2013 11:21 AM > To: AUDITORY@xxxxxxxx > Subject: Re: stats use in psychology and hearing science >=20 > Dear List - My thanks to Holger Mitterer for pointing out the paper by Sim= mons et al in Psych Sci, which promises to be an interesting read. It is jus= t one of a long string of papers in recent years which point out just how li= ttle of value can arise through statistical testing of experimental results (= see also the numerous papers of John Ioannidis at Stanford on this topic in m= edicine). > =20 > Unfortunately, this problem of designing experiments for the data analysis= - and the wider problem of inappropriate experimental design and inappropri= ate data analysis - is only going to get worse, especially in departments of= education, psychology, and hearing research. There, the older generation of= researchers, who might have had at least an undergrad freshman calculus cou= rse, has been replaced by a new generation of workers who do not have math b= eyond the 10th grade of high school and who barely passed their weak undergr= aduate mandatory course in practical stats. Too many people now seem to thin= k of stats testing (ANOVA in particular) as an act of magic that tells them w= hat's "significant". It is exceedingly rare, for example, to find any mentio= n of whether the assumptions underlying the statistical tests are actually o= beyed, as no-one seems to realize that statistical tests are derived from ma= thematical models that involve assumptions. There is a solution to this prob= lem: stricter math requirements at the undergraduate and graduate levels, in= cluding introductory theoretical statistics, not just basic stats testing. I= f we're going to use stats, let's do it properly. - Lance Nizami PhD, Palo A= lto, Cal. >=20 > From: Holger Mitterer <holgermitterer@xxxxxxxx> > To: AUDITORY@xxxxxxxx=20 > Sent: Saturday, June 22, 2013 7:40 AM > Subject: [AUDITORY] Reminder: Speed Sound Finding Experiment >=20 > probably many readers caught this, but just to make sure: > The reminder for the speech sound finding experiment contained a somewhat q= uestionable phrase: >=20 > > We are missing a few participants to reach statistical significance so > > please consider giving it a try: >=20 > Stopping data collection when an effect becomes significant is a very prob= lematic research strategy, > see the paper by Simmons et al. in PsychScience (http://pss.sagepub.com/co= ntent/22/11/1359.abstract). >=20 >=20 >=20 >=20 --Apple-Mail-D32972FD-01FC-4620-AD93-61CCA4A9C79C Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D= utf-8"></head><body dir=3D"auto"><div><meta http-equiv=3D"content-type" cont= ent=3D"text/html; charset=3Dutf-8"><div>Well said. Also, why is the use of n= onpametric statistics so rare? They do not require meeting the stiff criteri= a that e.g. ANOVA does. </div><div><br><br>Pardon the typos, it's Apple= 's fault.</div><div><br>On Jun 24, 2013, at 0:04, Brian Gygi <<a href=3D"= mailto:bgygi@xxxxxxxx">bgygi@xxxxxxxx</a>> wrote:<br><br></div><blockqu= ote type=3D"cite"><div><div><font face=3D"Verdana" size=3D"2"> </font><= /div> The problem is not just with higher education. As long as one needs a s= ignificant effect to get published, people are going to continue using the e= asiest and most powerfult stats tests out there. Reviewers are complic= it in this as well - I have rarely seen a paper turned down because of inapp= ropriate statistical tests (even though many publications specifically ask a= bout this). We as scientists could start by cleaning up our own shop a= bit.<br><br>Brian Gygi, Ph.D.<div><font color=3D"#0000ff" face=3D"Verdana" s= ize=3D"2"></font> </div> <blockquote style=3D"PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000= ff 2px solid; MARGIN-RIGHT: 0px"><font face=3D"Tahoma" size=3D"2">-----Origi= nal Message-----<br><b>From:</b> Iftikhar Nizami [<a href=3D"mailto:nizamii2= @xxxxxxxx">mailto:nizamii2@xxxxxxxx</a>]<br><b>Sent:</b> Sunday, June 23, 2013= 11:21 AM<br><b>To:</b> <a href=3D"mailto:AUDITORY@xxxxxxxx">AUDITORY= @xxxxxxxx</a><br><b>Subject:</b> Re: stats use in psychology and hear= ing science<br><br></font><table id=3D"201179" border=3D"0" height=3D"400" w= idth=3D"100%"><tbody><tr><td valign=3D"top" width=3D"100%"><div style=3D"col= or:#000; background-color:#fff; font-family:times new roman, new york, times= , serif;font-size:12pt"><div style=3D"RIGHT: auto"><span style=3D"RIGHT: aut= o">Dear List - My thanks to Holger Mitterer for pointing out the paper by Si= mmons et al in Psych Sci, which promises to be an interesting read. It is ju= st one of a long string of papers in recent years which point out just how l= ittle of value can arise through statistical testing of experimental results= (see also the numerous pap<var id=3D"yui-ie-cursor"></var>ers of John Ioann= idis at Stanford on this topic in medicine).</span></div> <div style=3D"RIGHT: auto"><span style=3D"RIGHT: auto"></span> </div> <div style=3D"RIGHT: auto"><span style=3D"RIGHT: auto">Unfortunately, this p= roblem of designing experiments for the data analysis - and the wider proble= m of inappropriate experimental design and inappropriate data analysis - is o= nly going to get worse, especially in departments of education, psychology, a= nd hearing research. There, the older generation of researchers, who might h= ave had at least an undergrad freshman calculus course, has been replaced by= a new generation of workers who do not have math beyond the 10th grade of h= igh school and who barely passed their weak undergraduate mandatory course i= n practical stats. Too many people now seem to think of stats testing (ANOVA= in particular) as an act of magic that tells them what's "significant". It i= s exceedingly rare, for example, to find any mention of whether the assumpti= ons underlying the statistical tests are actually obeyed, as no-one seems to= realize that statistical tests are derived from mathematical models that involve <i>assumptions</i>. There is a soluti= on to this problem: stricter math requirements at the undergraduate and grad= uate levels, including introductory <i style=3D"RIGHT: auto">theoretical </i= >statistics, not just basic stats testing. If we're going to use stats, let'= s do it properly. - Lance Nizami PhD, Palo Alto, Cal.</span></div> <div style=3D"RIGHT: auto"><br style=3D"RIGHT: auto"> <blockquote style=3D"BORDER-LEFT: rgb(16,16,255) 2px solid; MARGIN-TOP: 5px;= PADDING-LEFT: 5px; MARGIN-LEFT: 5px; RIGHT: auto"> <div style=3D"FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZ= E: 12pt"> <div style=3D"FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZ= E: 12pt"> <div style=3D"RIGHT: auto" dir=3D"ltr"> <div style=3D"BORDER-BOTTOM: #ccc 1px solid; BORDER-LEFT: #ccc 1px solid; PA= DDING-BOTTOM: 0px; LINE-HEIGHT: 0; MARGIN: 5px 0px; PADDING-LEFT: 0px; PADDI= NG-RIGHT: 0px; HEIGHT: 0px; FONT-SIZE: 0px; BORDER-TOP: #ccc 1px solid; BORD= ER-RIGHT: #ccc 1px solid; PADDING-TOP: 0px" class=3D"hr" readonly=3D"true" c= ontenteditable=3D"false"></div><font face=3D"Arial" size=3D"2"><b><span styl= e=3D"FONT-WEIGHT: bold">From:</span></b> Holger Mitterer <<a href=3D"mail= to:holgermitterer@xxxxxxxx">holgermitterer@xxxxxxxx</a>><br><b><spa= n style=3D"FONT-WEIGHT: bold">To:</span></b> <a href=3D"mailto:AUDITORY@xxxxxxxx= S.MCGILL.CA">AUDITORY@xxxxxxxx</a> <br><b><span style=3D"FONT-WEIGHT:= bold">Sent:</span></b> Saturday, June 22, 2013 7:40 AM<br><b><span style=3D= "FONT-WEIGHT: bold">Subject:</span></b> [AUDITORY] Reminder: Speed Sound Fin= ding Experiment<br></font></div> <div style=3D"RIGHT: auto" class=3D"y_msg_container"><br> <div style=3D"RIGHT: auto" id=3D"yiv1008380456"> <div style=3D"RIGHT: auto"> <div style=3D"BACKGROUND-COLOR: #fff; FONT-FAMILY: tahoma, new york, times, s= erif; COLOR: #000; FONT-SIZE: 12pt"> <div>probably many readers caught this, but just to make sure:</div> <div>The reminder for the speech sound finding experiment contained a somewh= at questionable phrase:</div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAMILY= : tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px"><br></= div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAMILY= : tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px">> W= e are missing a few participants to reach statistical significance so<br>>= ; please consider giving it a try:</div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAMILY= : tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px"><br></= div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAMILY= : tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px">Stoppi= ng data collection when an effect becomes significant is a very problematic r= esearch strategy,</div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAMILY= : tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px">see th= e paper by Simmons et al. in PsychScience (<a href=3D"http://pss.sagepub.com= /content/22/11/1359.abstract">http://pss.sagepub.com/content/22/11/1359.abst= ract</a>).</div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAMILY= : tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px"><br></= div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAMILY= : tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px"><br></= div></div></div></div><br><br></div></div></div></blockquote></div></div></t= d></tr></tbody></table></blockquote> </div></blockquote></div></body></html>= --Apple-Mail-D32972FD-01FC-4620-AD93-61CCA4A9C79C--