I have been thinking about the issue of how a field knows what they know. In a previous post, I wrote about how the field of social psychology is working through the implications of fraudulent research, and is closely examining the cozy interactions between journals, reviewers, and famous researchers. And any empirical field based on statistical analysis has got to live with the fact that if there 1000 results in the field, some number (50 perhaps, if p=.05 is a normal cutoff and lots of results are just under that value) are going to be wrong just because the statistical test created a false positive. Of course, replication can help determine what is real and what is not, but how often do you see a paper “Confirming Prof. X’s result”? Definitely not a smooth path to tenure.
This is worse if malevolent forces are at work. Suppose a pharmaceutical company has bet the firm on drug X, and they want to show that drug X works. And suppose drug X doesn’t work. No problem! Simply find 20 researchers, sign them to a non-disclosure, and ask them to see if drug X works. Chances are one or more researchers will come back with a statistically significant result (in fact, there is about a 65% chance that one or more will, given a p=.05). Publish the result, and voila! The company is saved! Hurray for statistics and capitalism!
Fortunately, I am not the first to see this issue: way back in 1997, the US Congress passed a bill requiring the registration of clinical trials, before the trials get underway.
The first U.S. Federal law to require trial registration was the Food and Drug Administration Modernization Act of 1997 (FDAMA) (PDF).
Section 113 of FDAMA required that the National Institutes of Health (NIH) create a public information resource on certain clinical trials regulated by the Food and Drug Administration (FDA). Specifically, FDAMA 113 required that the registry include information about federally or privately funded clinical trials conducted under investigational new drug applications (INDs) to test the effectiveness of experimental drugs for patients with serious or life-threatening diseases or conditions.
This led to the creation of clinicaltrials.gov (where I am getting this history and the quotes) in 2000. This was followed by major journals requiring registration before papers could be considered for publication:
In 2005 the International Committee of Medical Journal Editors (ICMJE) began to require trial registration as a condition of publication.
- ICMJE Uniform Requirements for Manuscripts Submitted to Biomedical Journals – Obligation to Register Clinical Trials
- ICMJE Frequently Asked Questions about Clinical Trials Registration
I think what you have in mind could perhaps be handled by an edited wiki, provided that (a) you could find someone to host it (hello, INFORMS) and (b) you could find a cadre of engaged moderators/editors. This might help with the errors-in-print problem, but I’m not sure how it would help with bias. Using a wiki would allow for comments along the lines of “I confirmed this” or “I got different results”, but the value of those comments would depend on documentation of the method of confirmation/refutation. In any case, there’s still the fundamental problem that the academic reward system only rewards journal publications, and journals in OR and related fields are for the most part not too excited about confirmatory studies, to put it quite mildly. (Blasting a hole in a published paper has a reasonable chance of being published.)