{"id":1732,"date":"2012-11-30T12:59:54","date_gmt":"2012-11-30T16:59:54","guid":{"rendered":"http:\/\/mat.tepper.cmu.edu\/blog\/?p=1732"},"modified":"2012-11-30T12:59:54","modified_gmt":"2012-11-30T16:59:54","slug":"referees-considered-harmful","status":"publish","type":"post","link":"https:\/\/mat.tepper.cmu.edu\/blog\/index.php\/2012\/11\/30\/referees-considered-harmful\/","title":{"rendered":"Referees considered harmful"},"content":{"rendered":"

When doing empirical work, researchers often mess up either in the design of the experiment or in the analysis of data. \u00a0In operations research, much of our “empirical work” is in computational testing of algorithms. \u00a0Is algorithm A faster than algorithm B? \u00a0“It depends” is generally the only honest answer. \u00a0It depends on the instance selection, it depends on the computing environment, it depends on the settings, etc. etc. \u00a0 If we are careful enough, we can say things that are (within the limits of the disclaimers) true. \u00a0 But even a a careful experiment can fall prey to issues. \u00a0For instance, throwing away “easy” instances can bias the results against whatever algorithm is used to determine easiness<\/a>. \u00a0And don’t get me started on empirical approaches that test dozens of possibilities and\u00a0miraculously find something “statistically significant”<\/a>, to be duly marked with an\u00a0asterisk in the table of results. \u00a0 \u00a0It is very difficult to truly do trustworthy empirical work. \u00a0And it is even harder to do such work when researchers cheat or reviewers don’t do their job.<\/p>\n

For some fields, these issues are even more critical. \u00a0Operations research generally has some theoretical grounding: \u00a0we know about polytopes and complexity, and so on, and can prove theorems that help guide our empirical work. \u00a0In fields like Social Psychology<\/a> (the study of people in their interactions with others), practically all that is known is due to the results of experiments. \u00a0 The fundamental structure in this field is a mental state, something that can only be\u00a0imprecisely observed.<\/p>\n

Social psychology is in a bit of a crisis. \u00a0In a very real sense, the field no longer knows what is true. \u00a0Some of that crisis is due to academic\u00a0malfeasance, particularly that of an influential researcher Diederik Stapel<\/a>. \u00a0Stapel has been found inventing data for dozens of papers, as described by a\u00a0 “Science Insider” column<\/a>.<\/p>\n

Due to data fraud by Stapel and others, the field has to reexamine much of what it thought was true. \u00a0Are meat eaters more selfish than vegetarians? \u00a0We thought so for a while<\/a>, but now we don’t know. \u00a0A Dutch report on this<\/a> goes into\u00a0great\u00a0detail on this affair.<\/p>\n

But overt fraud is not the only issue, as outlined in the report. \u00a0I was particularly struck by the role paper reviewers played in this deceit:<\/p>\n

It is almost inconceivable that co-authors who analysed the data intensively, or reviewers of the international “leading journals”, who are deemed to be experts in their field, could have failed to see that a reported experiment would have been almost infeasible in practice, did not notice the reporting of impossible statistical results, \u2026 and did not spot values identical to many decimal places in entire series of means in the published tables. Virtually nothing of all the impossibilities, peculiarities and sloppiness mentioned in this report was observed by all these local, national and international members of the field, and no suspicion of fraud whatsoever arose.<\/p><\/blockquote>\n

And the role of reviewers goes beyond that of negligence:<\/p>\n

Reviewers have also requested that not all executed analyses be reported, for example by simply leaving unmentioned any conditions for which no effects had been found, although effects were originally expected. Sometimes reviewers insisted on retrospective pilot studies, which were then reported as having been performed in advance. In this way the experiments and choices of items are justified with the benefit of hindsight.<\/p>\n

Not infrequently reviews were strongly in favour of telling an interesting, elegant, concise and compelling story, possibly at the expense of the necessary scientific diligence.<\/p><\/blockquote>\n

I think it is safe to say that these issues are not unique to social psychology. \u00a0I think that I too have, as a reviewer, pushed toward telling an interesting story, although I hope not at the expense of scientific diligence. \u00a0 And perhaps I could have worked harder to replicate some results during the reviewing process.<\/p>\n

I don’t think we in operations research are in crisis over empirical issues. \u00a0I am pretty confident that CPLEX 12.4<\/a> is faster than CPLEX 4.0 for practically any instance you can throw at it. \u00a0And some journals, like\u00a0\u00a0Mathematical Programming Computation<\/a>\u00a0have attempted to seriously address these issues. \u00a0But I am also pretty sure that some things I think are true are not true, either due to fraud by the author or negligence by reviewers.<\/p>\n

One important role of a reviewer is to be on the lookout for\u00a0malfeasance or bias and to avoid allowing (or, worse, forcing) authors to present data in an untruthful way. \u00a0And I think many of us are not doing a great job in this regard. \u00a0I would hate to have to rethink the foundations of my field due to these issues.<\/p>\n","protected":false},"excerpt":{"rendered":"

When doing empirical work, researchers often mess up either in the design of the experiment or in the analysis of data. \u00a0In operations research, much of our “empirical work” is in computational testing of algorithms. \u00a0Is algorithm A faster than algorithm B? \u00a0“It depends” is generally the only honest answer. \u00a0It depends on the instance … Continue reading “Referees considered harmful”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[32,46],"tags":[],"class_list":["post-1732","post","type-post","status-publish","format-standard","hentry","category-journals","category-research"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/mat.tepper.cmu.edu\/blog\/index.php\/wp-json\/wp\/v2\/posts\/1732","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mat.tepper.cmu.edu\/blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mat.tepper.cmu.edu\/blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mat.tepper.cmu.edu\/blog\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mat.tepper.cmu.edu\/blog\/index.php\/wp-json\/wp\/v2\/comments?post=1732"}],"version-history":[{"count":0,"href":"https:\/\/mat.tepper.cmu.edu\/blog\/index.php\/wp-json\/wp\/v2\/posts\/1732\/revisions"}],"wp:attachment":[{"href":"https:\/\/mat.tepper.cmu.edu\/blog\/index.php\/wp-json\/wp\/v2\/media?parent=1732"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mat.tepper.cmu.edu\/blog\/index.php\/wp-json\/wp\/v2\/categories?post=1732"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mat.tepper.cmu.edu\/blog\/index.php\/wp-json\/wp\/v2\/tags?post=1732"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}