I have previously written on how decision makers (and journalists) need to know some elementary probability and statistics to prevent them from making horrendously terrible decisions. Coincidentally, Twitter’s @ORatWork (John Poppelaars) has provided a pointer to an excellent example of how easily organizations can get messed up on some very simple things.
As reported by the blog Bad Science, Stonewall (a UK-based lesbian, gay and bisexual charity) released a report stating that the average coming out age has been dropping. This was deemed a good enough source to get coverage in the Guardian. Let’s check out the evidence:
The average coming out age has fallen by over 20 years in Britain, according to Stonewall’s latest online poll.
The poll, which had 1,536 respondents, found that lesbian, gay and bisexual people aged 60 and over came out at 37 on average. People aged 18 and under are coming out at 15 on average.
Oh dear! I guess the most obvious statement is that it would be truly astounding if people aged 18 and under had come out at age 37! Such a survey does not, and can not (based just on that question), answer any questions about the average coming out age. There is an obvious sampling bias: asking people who are 18 when they come out ignores gays, lesbians, and bisexuals who are 18 but come out at age 40! This survey question is practically guaranteed to show a decrease in coming out age, whether it is truly decreasing, staying the same, or even increasing. How both the charity and news organizations who report on this can’t see this immediately baffles me.
But people collect statistics without considering whether the data address the question they have. They get nice reports, they pick out a few numbers, and the press release practically writes itself. And they publish and report nonsense. Bad Science discusses how the question “Are people coming out earlier” might be addressed.
I spent this morning discussing the MBA curriculum for the Tepper School, with an emphasis on the content and timing of our quantitative, business analytics skills. This example goes to the top of my “If a student doesn’t get this without prodding, then the student should not get a Tepper MBA” list.
Added December 2. Best tweet I saw on this issue (from @sarumbear):
#Stonewall ‘s survey has found that on average, as people get older, they get older. http://bit.ly/gT731O #fail
I wonder if anyone has compiled a list of dopey abuses of statistics in the press. It would probably make a good BLOSSOMS (http://blossoms.mit.edu/default.htm) video — in each segment, quote the news report and let the class figure out why it fails the sniff test.
Coincidentally, I just ran across this TED talk on statistical misunderstandings, with a particular reference to a serious miscarriage of justice in a jury trial.
And it’s not just statistics either. The widespread misconceptions about the economy, the federal budget, and other matters of public policy–particularly among the policymakers themselves–is positively frightening.
And that’s the main problem, Mike. As Matthew states, what media publishes is at the end of the day ‘harmless’ (besides making people dumber); the most of the harm is done when this kind of surveys and/or reports are used for policy-making purposes.
It’s a pity, but it seems that only quarter a politician (0.25) ON AVERAGE knows something about statistics… Ò__Ó | In our dreams!
I’m going to disagree with Francisco. In a democratic society where policymakers are beholden to voters for their jobs, the harm associated with “making people dumber” is twofold: (1) voters vote their misunderstandings, and (2) policymakers have an incentive to pander to those voters by turning those misconceptions into policy.. Witness what has passed for political discourse in our most recent elections and their aftermath.
I’ll go with Matt on this one. In addition (and this is a lesser concern), I think statistical abuses in the press reinforce a sense among readers that they understand statistics better than they really do, which can affect their personal lives (choices of medical treatments, for instance).
@Mike: Thanks for the quote from Bad Science — I used it in class today to illustrate how the press botches statistics just like the rest of us.
Great example Mike. But, it’s not only the media — exactly this bias tainted early estimates of the AIDS incubation time (time from infection with the HIV virus until symptomatic manifestation of AIDS) that were published in the medical literature. For example, a study of blood transfusion recipients who subsequently developed AIDS estimated the mean incubation time by averaging the time from transfusion to manifestation of AIDS symptoms. Of course, those who had been infected via transfusion before the study cutoff date but had yet to progress to AIDS at the time of the study failed to make it into the sample, as only people identified as having AIDS by the cutoff date met the inclusion criteria.