Subject: Re: within subject comparisons From: Chris Chambers <Chris.Chambers(at)SCI.MONASH.EDU.AU> Date: Mon, 30 Oct 2000 10:58:11 +1100Hello, Firstly, thankyou to all respondents for your comments on within subject comparisons. There have certainly been a wide range of opinions, and that's not even including the statisticians I have talked to! Wherever you happen to stand on it, it seems clear that if you are doing within subject comparisons the independence assumption is maintained provided you accept that the nature of the inference is different (as Pallier put it). Can we assume that we are sampling *randomly* (and thus independently) from a particular subject's population of responses? Michael Kubovy believes you can, provided you implement certain counterbalancing techniques to prevent autocorrelations. Can we ever assume absolute independence between the responses of a single subject, even if stringent counterbalancing is employed? I doubt it, but maybe we don't need it to be absolute (although some statistical purists will argue otherwise). Al Bregman has questioned the usefulness of such comparisons because the subject's own population of responses is the only justifiable statistical generalization. Al points out that if you provide the reader with error bars and descriptive statistics then common sense should do the rest. I think this may a reasonable approach, but I think it could only be reasonable if the estimates of the variance that are used to generate standard error are accurate (which requires independence). If the data are highly autocorrelated then the variance will be underestimated, the error bars will be misleading, and any conclusions that the reader might draw from them based on common sense may be incorrect. A good example of this is can be found in adaptive staircase techniques. A threshold point is often taken as the mean of a prescribed number of reversals. The variance of this mean is not very useful as an estimate of the standard error of the threshold because of the high autocorrelation between the stimulus level at each turnaround. Typically, the variance within a staircase will underestimate the standard error of the threshold across staircases. If, for some reason, we were unaware of this correlation and reported the standard error of the threshold based on this variance we would be overstating the magnitude of the effect and the reader, equipped with only the descriptive statistics, will be misled. I think it is important to show statistical significance of within-subject comparisons because it should force us to examine the level of autocorrelation or other dependencies that may be lurking in the data. Even if no violations are present then the p value and particularly estimates of effect size and power are still useful in telling us about the size of the effect, its replicability, and its likelihood of being real. Thanks again for your many contributions. Best, Chris -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Chris Chambers Department of Psychology Monash University Clayton, Victoria 3168 AUSTRALIA Tel. +61 3 9905 3978 Fax. +61 3 9905 3948 EMAIL: chris.chambers(at)sci.monash.edu.au -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-