Our research in this area was enhanced by a visit from Bruce Thompson (Texas A&M University, editor of Educational and Psychological Measurement), Nov 2000 – Jan 2001. For leads to many useful resources see Bruce’s website.

Current projects include:

• Our primer on confidence intervals based on central and noncentral distributions (Cumming & Finch, 2001) describes how confidence intervals for Cohen’s delta (also known as Cohen’s d) require use of noncentral-t distributions. We use the ESCI software to provide illustrative demonstrations, and a facility for calculating confidence intervals for delta.


• Our study of reporting practices in psychological journals (over 60 years) found little evidence of change in response to ongoing calls for reform (Finch, Cumming, & Thomason, 2001).


• Our historical and critical analysis of publication guidelines and practices in psychology has been accepted for publication subject to amendments (Finch, Thomason, & Cumming, 2001).


• Our case study of editorial policy changes and publication practices in Memory & Cognition (1992-2000) has given excellent results and is currently being prepared for publication.


• Our first web study of published researchers’ attitudes towards and understanding of various statistical practices was revealing and instructive. It is being prepared for publication, and is guiding the design of a follow-on study on researcher cognition.


• We are currently analysing and writing up a study of statistical practice in medical publication, which has given very interesting findings.


• Development of the ESCI software <4>, and writing of papers to help students and researchers use it.

Cumming, G., & Finch, S. (2001). A primer on the understanding, use and calculation of confidence intervals based on central and noncentral distributions. Educational and Psychological Measurement, 61 (August).

Abstract:
Reform of statistical practice in the social and behavioural sciences requires wider use of confidence intervals (CIs), and of effect size measures and meta-analysis. In this context we discuss four reasons for promoting use of CIs: (i) they give useful, interpretable information; (ii) they are linked to statistical significance tests with which researchers are already familiar; (iii) they can encourage meta-analytic thinking that focuses on estimation; and (iv) CI width gives information about precision that may be more useful than a statistical power value. We focus on a basic standardised effect size measure, Cohen’s delta (also referred to as Cohen’s d). We give methods and examples for the calculation of CIs for delta, which require use of noncentral t distributions, and contrast these with the familiar CIs for original score means. We discuss noncentral t distributions, unfamiliar to many social scientists, and apply these also to statistical power and to simple meta-analysis of standardised effect sizes. We provide the ESCI graphical software, which runs under Microsoft Excel, to illustrate the discussion. Wider use of CIs for delta and other effect size measures should help promote highly desirable reform of statistical practice in the social sciences.

Finch, S., Thomason, N., & Cumming, G. (accepted subject to amendments, 2001) Past and future APA guidelines for statistical practice. Theory & Psychology.

Abstract:
We review the publication guidelines of the American Psychological Association (APA) since 1929 and document their advice for authors about statistical practice. Although the advice has been extended with each revision of the guidelines, it has largely focussed on Null Hypothesis Significance Testing (NHST) to the exclusion of other statistical methods. In parallel, we review over 40 years of critiques of NHST in psychology. The critiques have had little impact on the APA guidelines. The guidelines are influential in broadly shaping statistical practice, although in some cases they are not closely followed. They have an important role to play in reform of statistical practice in psychology. Following the report of the APA’s Task Force on Statistical Inference, we propose that revisions of the guidelines reflect a broader philosophy of analysis and inference, provide detailed statistical requirements for reporting research, and directly address concerns about NHST. In addition the APA needs to develop ways to ensure that its editors succeed in their leadership role in achieving essential reform.

Finch, S., Cumming, G., & Thomason, N. (2001). Reporting of statistical inference in the Journal of Applied Psychology: Little evidence of reform. Educational and Psychological Measurement, 61, 181-210

Abstract:
Reformers have long argued that misuse of Null Hypothesis Significance Testing (NHST) is widespread and damaging. We analyzed 150 papers from the Journal of Applied Psychology (JAP) covering 1940 to 1999. We examined statistical reporting practices related to misconceptions about NHST, APA guidelines, and reform recommendations. Our analysis reveals (a) inconsistency in reporting alpha and p-values, (b) use of ambiguous language in describing NHST, (c) frequent acceptance of null hypotheses without consideration of power, (d) that power estimates are rarely reported, (e) virtually no confidence intervals. APA guidelines have been followed only selectively. Research methodology reported in JAP has increased greatly in sophistication over 60 years, but inference practices have shown remarkable stability. There is little sign that decades of cogent critiques by reformers had by 1999 led to changes in statistical reporting practices in JAP.