Sharpshooter Fallacy/Quotes

In this article, we discuss an uncomfortable fact that threatens the core of psychology’s academic enterprise: almost without exception, psychologists do not commit themselves to a method of data analysis before they see the actual data. It then becomes tempting to fine tune the analysis to the data in order to obtain a desired result — a procedure that invalidates the interpretation of the common statistical tests. The extent of the fine tuning varies widely across experiments and experimenters but is almost impossible for reviewers and readers to gauge.

— Wagenmakers, E-J, Wetzels, R, Borsboom, D, van der Maas, HLJ, Kievit, RA, An Agenda for Purely Confirmatory Research (abstract)