Science Buffs is covering an upcoming Keystone Symposia live streaming event called “Rigor in Science: The Challenge of Reproducibility in Biomedical Research.” A panel discussion (filmed previously this year) will be streamed for free on November 8th, 2017 and will be followed with a live Q&A where anyone from the public can pose questions. The next three posts on our blog will be about reproducibility in science, with an emphasis on biological science, in order to aid the discussion. Find the first post here.
Sometimes Biology Works Against You: Biological Complexity and Reproducible Science
In our first post in this series on reproducibility in science, I talked about attempting to replicate an experiment another group of scientists had done. After months of banging my head against the wall, I finally gave up and moved on. But the question has hung over my head ever since: what was I doing differently?
Unfortunately for me, the list of variables is long. I’m a biologist, and biological systems are almost unfathomably complicated—we just don’t understand all the different ways that two animals or even two cell lines can be distinct from each other.
A recently recorded Keystone Symposia panel about reproducibility in science, to be streamed on November 8th, focused primarily on reproducibility in biology. Although reproducibility is an issue in all areas of science, the big question specific to biology is: what are the biological complexities that make experiments in animals, cells, and clinical trials hard to replicate?
For example, if an experiment is done in mice the researchers may have to consider age, sex, and breed of the mice, what time of day the experiments were done, what kind of chow the mice were eating, what kind of cages they were in, and more. Imagine the impossibility of getting that spreadsheet exactly right for each iteration of an experiment. But it can be an absolute necessity in some cases, because all of these factors can determine the outcome of an experiment. Sometimes they may be the difference between an experiment replicating or not.
If an experiment replicates when all of the variables are just so, the original researchers may breathe a sigh of relief. After all, it shows that they published something that is true. But there’s a deeper question here: how important is the outcome if you can only do it on a Tuesday under the light of a full moon? What’s the greater importance of an experiment that only has an exciting conclusion in one species of mice, only when they’re female, only when they’re eating a particular brand of mice chow?
Luckily, researchers are unearthing more and more variables important to biological research. The microbiome of animals, or the bacteria resident in animal guts and on animal skin, fur, feathers, and scales, is only recently becoming appreciated as a critical variable in research outcomes. Similarly, circadian rhythms, or when an animal’s body clock is telling it to sleep, eat, and poop, has an unexpectedly large role in biological processes.
Probably one of the most important variables in biomedical research is age, as Dr. Michael Holtzman, Director of Pulmonary and Critical Care Medicine at Washington University School of Medicine, knows well. Holtzman has studied asthma in an animal model—laboratory mice.
“We routinely give the mice viral infections when they’re four to six weeks old, and then we see ‘asthma’ weeks later,” says Holtzman. “But when we started to do experiments where we’re trying to do cell transfers and so on, those take a lot of time, we ended up infecting the animals some weeks to months later, and we lost the response. We didn’t see anything.”
In this case, the age of the mice was incredibly important for Holtzman’s work to be reproducible.
When does age matter?
As the importance of such variables is being defined, people are paying more attention to recording and publishing that information along with results and conclusions. The more information you have about how another experiment was done, and with what kind of reagents, the better chance you have of replicating the results. Along those lines, institutions and journals are trying to change the expectations about reporting data and techniques.
The National Institutes of Health (NIH), for example, now requires researchers to describe how they will ensure the identity and legitimacy of key reagents and cell lines. Holtzman thinks that this is a useful exercise.
“It does make you stop and think about the issue, at least momentarily,” said Holtzman. “You think about these possible differences and the way you’re doing things…introducing artifacts and so on.”
Journals, too, are trying to improve reproducibility. They are requiring more and more information about how experiments were done. Alison Farrell, an editor at Nature Research, says her journal has been drastically changing the requirements for publication.
“We do now have a much more extensive and more detailed information that hopefully, moving forward, will allow other researchers to reproduce another group’s work,” said Farrell. “That includes reporting on all the reagents used, the methods used in a more detailed manner than perhaps has been done in the past, and ensuring deposition of data in public databases.”
Farrell went on to make the point that sometimes an experiment won’t repeat for good reasons—reasons that might tell us important information about basic biological questions. But that means that the reader of any paper should keep in mind that results from an experiment done in a single system can’t necessarily be translated to all systems, in all cases.
“I think that it’s important to remember that there is variability in science. Variability isn’t inherently bad,” said Farrell. “But it is important to be able to recognize when there are limitations in an experimental design that preclude grand conclusions. I think that it’s important as a researcher and important as a reader.”
In these first two posts about reproducibility, we’ve talked about how scientists can improve the reproducibility of their work by designing good experiments and considering the biological variables that might make an experiment hard to replicate. But why are we saying that science is experiencing a reproducibility “crisis” now, instead of fifty or a hundred years ago? And what contributed to the increase in papers that wouldn’t replicate? In the third post in this series, we’ll talk about the publishing incentives that many scientists say led to our current state.
By Alison Gilchrist (@AlisonAbridged)
Science Buffs 