Re: [AUDITORY] stats (mis)use in psychology and hearing science ("Oberfeld-Twistel, Daniel" )

```Subject: Re: [AUDITORY] stats (mis)use in psychology and hearing science
From:    "Oberfeld-Twistel, Daniel"  <oberfeld@xxxxxxxx>
Date:    Thu, 26 Sep 2013 12:13:52 +0000
List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>

Dear list,

> From:    Pierre Divenyi <pdivenyi@xxxxxxxx>
> Subject: Re: stats (mis)use in psychology and hearing science
>=20
> That may be correct under certain circumstances but the real problem is
> ascertaining that ANOVA is, indeed, appropriate. And it is truly a "REAL"
> problem!
>=20
> On 9/22/13 11:15 PM, "Kyle Nakamoto" <knakamoto@xxxxxxxx> wrote:
>=20
> >Nonparametrics are not automatically better. If you use a nonparametric
> >statistic when an ANOVA is appropriate the chance of missing a real effe=
ct
> >increases (False Negative).

It is of course true that we should all pay attention to the assumptions of=
statistical tests. For example, one should not use tests assuming independ=
ent observations when analyzing data from a repeated-measures design. Anoth=
er example, as I mentioned in a previous posting, is that repeated-measures=
ANOVAs are sensitive to departures from normality, and many additional pro=
blems arise when the design is unbalanced (i.e., unequal group sizes, see (=
Keselman, Algina, & Kowalchuk, 2001)).
However, it is a sort of "magical" thinking that nonparametric methods are =
*free* of assumptions - this is of course not the case.

Just to give you two simple examples:

1) The nonparametric Friedman rank test that can be used for analyzing data=
from a one-factorial repeated-measures design assumes equal variances and =
covariances of the measures. In real data sets, this assumption is almost a=
lways violated, known as a deviation from sphericity. That's why we (hopefu=
lly) use for example the Huynh-Feldt correction for the degrees-of-freedom =
when computing a repeated-measures ANOVA with the univariate approach. Simu=
lation studies showed that when the (co-)variances are heterogeneous, Fried=
man's test does *not* control the Type I error rate (i.e., probability to p=
roduce a significant p-value (e.g., p < .05) when the population means are =
*identical*), especially when the distribution of the response measure is s=
kewed (Harwell & Serlin, 1994; St. Laurent & Turk, 2013).

2) The (nonparametric) Mann-Whitney U-test that can be used to compare the =
means of two independent samples does not control the Type I error rate in =
the case of variance heterogeneity (i.e., the two groups have different var=
iances), especially when the distribution of the response measure is asymme=
tric (skewed). In fact, the U-test was shown to be *more* sensitive to viol=
ations of these assumption than the classical t-test in several conditions =
(Stonehouse & Forrester, 1998).

Thus, it is not a given that "nonparametric" methods are more "robust" (or =
even "free of assumptions") than parametric methods like the GLM. Instead, =
if you have reason to believe that the assumptions for a parametric test ar=
e violated, then it will be a good idea to consult the (sometimes very exte=
nsive, sometimes very sparse) literature on simulation studies concerning t=
he effects of such violations - sometimes it might turn out that using the =
"bad" good old t-test or another parametric method is in fact superior to a=
pplying a nonparametric procedure. But sometimes the answer will likely nei=
ther be "parametric better" or "nonparametric better", but "it's complicate=
d" (or: "we don't know yet")...

I would be interested to learn which data analysis problems you are facing =
in your research -- maybe we could use this list to identify the best solut=
ions to these problems?

Best,

Daniel

Harwell, M. R., & Serlin, R. C. (1994). A Monte-Carlo study of the Friedman=
test and some competitors in the single factor, repeated-measures design w=
ith unequal covariances. Computational Statistics & Data Analysis, 17(1), 3=
5-49.
Keselman, H. J., Algina, J., & Kowalchuk, R. K. (2001). The analysis of rep=
eated measures designs: A review. British Journal of Mathematical and Stati=
stical Psychology, 54, 1-20.
St. Laurent, R., & Turk, P. (2013). The effects of misconceptions on the pr=
operties of Friedman's test. Communications in Statistics-Simulation and Co=
mputation, 42(7), 1596-1615.
Stonehouse, J. M., & Forrester, G. J. (1998). Robustness of the t and U tes=
ts under combined assumption violations. Journal of Applied Statistics, 25(=
1), 63-74.

PD Dr. Daniel Oberfeld-Twistel
Johannes Gutenberg - Universitaet Mainz
Department of Psychology
Experimental Psychology
Wallstrasse 3
55122 Mainz
Germany

Phone ++49 (0) 6131 39 39274=20
Fax   ++49 (0) 6131 39 39268
http://www.staff.uni-mainz.de/oberfeld/