Pop quiz! What’s the fruit proven to prevent ulcers, diabetes, and obesity, that also decreases your blood pressure? If you guessed blueberries, grapefruit, acai berries, or bananas, boy, are you behind the times. The new super fruit is the humble pear.
Skeptical? You should be. How many times have you heard about the newly discovered cancer-fighting powers of one superfood, only to be told nearly the same spiel about a different one days later? I, for one, was super excited when I learned that I could prevent heart disease with dark chocolate. “Thank God,” I said to myself, munching a second Snickers bar, “My heart is going to be so healthy in 50 years.” Sadly, top scientists later decided thatkale is the true preventer of heart disease. Have they made kale chocolate yet?*
As a fun exercise, try hunting down the research cited in the above superfood links. Can you find the relevant papers? Do you think the results were reported accurately? How much did these reporters really look into the ability of pear phenols to fight Type-II Diabetes?
Sadly, many reporters don’t have the best track record when it comes to doing that kind of critical research. It might seem obvious to us that reporters aren’t doing their homework when we see the huge amount of articles about superfoods, miracle fruits, and cancer-fighting grains that get reported about in questionable magazines. However, like a true scientist, molecular biologist and science journalist Johannes Bohannon was interested in testing for himself just how much research goes on behind the scenes before reporters tell us that science has discovered the next miracle food.
Bohannon has a PhD in the molecular biology of bacteria and decided to prove that chocolate is good for us. (If his mismatched qualifications don’t raise your eyebrows, then the fact that he began with a result in mind should.) He conducted a legitimate and documented nutritional study, created a website, and paid an obscure journal to publish a paper about chocolate as a “weight-loss accelerator.” After a carefully phrased press release, his paper was being referred to in multiple articles touting chocolate as a diet food.
Ready for another fun task? Bohannon wrote an amazing description of conducting his nutritional study from start to finish, which is your new reading assignment. It’s a glimpse into the art of experimental design and analysis with the aim of achieving a publishable p-value, the accepted measure of statistical significance. This process can also be called “p-hacking.” But after his study was published, he could sit back and watch the journalists dig their own way into a hole.
He reports: “When reporters contacted me at all, they asked perfunctory questions. ‘Why do you think chocolate accelerates weight loss? Do you have any advice for our readers?’ Almost no one asked how many subjects we tested, and no one reported that number. Not a single reporter seems to have contacted an outside researcher. None is quoted.”
For the record, Bohannon’s study included just 15 participants and measured 18 variables. Because all he needed was any of those variables to “improve” in the group of participants who were eating chocolate (the variable he controlled), his study was designed to give him what he wanted. With such a small amount of people it’s statistically very likely that something will change for the better in the group eating chocolate.
In this case it happened to be weight loss, but it could just as easily have been blood pressure or the squishiest of measurements: “well-being.” If the reporters had contacted an outside researcher or a statistician, they might have realized Bohannon’s conclusions are junk.
That website Bohannon created? Nobody looked more closely than the home page. The dismal number of participants in his study? Not questioned. Sadly, if the state of diet science reporting is anything to go by, this is the norm. Everyone wants to grab your attention while you’re browsing the checkout line offerings, and the easiest way to do that is to tell you the latest, greatest, magical cure for all ailments. Today, it appears to be pears.
Alright, I’ve given you a pop quiz and some exciting in-class assignments. Time to do some homework. The next time you see some “reporting” about a “superfood” in your favorite “newspaper”, look for a citation. If there is one, great! Now hunt down the primary literature and decide if you think it’s believable. If this sounds like a lot of work and extra research on your part, it should! Now remember—this is what a reporter is paid to do. Our job as consumers is to remind reporters that they can’t fool us (or some of us, anyway). If you get the chance, call them out on blind reporting of flimsy research or sensationalist representations of good science.
*Funny story: yes.
By Alison Gilchrist
Science Buffs 