{"id":1735,"date":"2012-12-03T16:19:31","date_gmt":"2012-12-03T20:19:31","guid":{"rendered":"http:\/\/mat.tepper.cmu.edu\/blog\/?p=1735"},"modified":"2012-12-03T16:19:31","modified_gmt":"2012-12-03T20:19:31","slug":"registries-to-avoid-publication-bias","status":"publish","type":"post","link":"https:\/\/mat.tepper.cmu.edu\/blog\/index.php\/2012\/12\/03\/registries-to-avoid-publication-bias\/","title":{"rendered":"Registries To Avoid Publication Bias"},"content":{"rendered":"

I have been thinking about the issue of how a field knows what they know. \u00a0In a previous post<\/a>, I wrote about how the field of social psychology is working through the implications of\u00a0fraudulent\u00a0research, and is closely examining the cozy interactions between journals, reviewers, and famous researchers. \u00a0 And any empirical field based on statistical analysis has got to live with the fact that if there 1000 results in the field, some number (50 perhaps, if p=.05 is a normal cutoff and lots of results are just under that value) are going to be wrong just because the statistical test created a false positive. \u00a0Of course, replication can help determine what is real and what is not, but how often do you see a paper “Confirming Prof. X’s result”? \u00a0Definitely not a smooth path to tenure.<\/p>\n

This is worse if\u00a0malevolent forces are at work. \u00a0Suppose a pharmaceutical company has bet the firm on drug X, and they want to show that drug X works. \u00a0And suppose drug X doesn’t work. \u00a0No problem! \u00a0Simply find 20 researchers, sign them to a non-disclosure, and ask them to see if drug X works. \u00a0Chances are one or more researchers will come back with a statistically significant result (in fact, there is about a 65% chance that one or more will, given a p=.05). \u00a0Publish the result, and voila! \u00a0The company is saved! \u00a0Hurray for\u00a0statistics and capitalism!<\/p>\n

Fortunately, I am not the first to see this issue: \u00a0way back in 1997, the US Congress passed a bill requiring the registration of clinical trials, before the trials get underway.<\/p>\n

The first U.S. Federal law to require trial registration was the\u00a0Food and Drug Administration Modernization Act of 1997 (FDAMA)<\/a>\u00a0(PDF).<\/p>\n

Section 113 of FDAMA required that the National Institutes of Health (NIH) create a public information resource on certain clinical trials regulated by the Food and Drug Administration (FDA). Specifically, FDAMA 113 required that the registry include information about federally or privately funded clinical trials conducted under investigational new drug applications (INDs) to test the effectiveness of experimental drugs for patients with serious or life-threatening diseases or conditions.<\/p><\/blockquote>\n

This led to the creation of clinicaltrials.gov<\/a> (where I am getting this history and the quotes) in 2000. \u00a0This was followed by major journals requiring registration before papers could be considered for publication:<\/p>\n

In 2005 the\u00a0International Committee of Medical Journal Editors<\/a>\u00a0(ICMJE) began to require trial registration as a condition of publication.<\/p>\n