Kathryn Baker looks into the role of DNA mutations in the development of colorectal cancer in patients with ulcerative colitis. By studying both mitochondrial DNA mutations and clonal expansions, she hopes to better understand cancer development in an inflammatory setting and to help develop biomarkers for early cancer detection in UC patients.
Most scientists take comfort in the idea that science is essentially self-correcting. Faith in research hinges on the collection and publication of observable data and facts, the peer-review process, and the possibility of recreating our experiments to verify the findings. The question is then, what happens when study results are held up for scrutiny? Cancer researchers are discovering that their discoveries may not be as iron-clad as previously thought.
At a time when research funding continues to be tight, convincing either researchers or granting agencies to move forward with reproducibility studies is difficult. Because resources are scarce, the pressure to fund and produce exciting, novel, “sexy” science can seem insurmountable. Scientists want to believe in their results, but they know that many times, the effort will not be put forth to prove that they’re reliable. In 2012 the trend was broken by Amgen, a powerhouse biotechnology firm located in Thousand Oaks, California. (1) Citing the fact that oncology clinical trials have the highest failure rate versus other therapeutic areas, Amgen set out to validate 53 “landmark” studies. They shocked the scientific community when they reported that they could only reproduce six publications, or 11%, of the 53 they attempted. While some difficulty in reproducing results is expected, this failure rate was unprecedented. (1) These results were not made public, however, so the rest of the scientific community was in the dark about which studies were actually recreated.
Reproducibility studies for a wider audience arrived when The Reproducibility Project: Cancer Biology began in 2013. The large-scale collaborative effort set out to recreate findings from 50 major cancer studies published in high-impact journals like Science and Nature. In contrast to the Amgen studies, the project’s work will be openly available online. (2) In January of 2017, results for the first five studies were released. Two studies were “substantially reproduced” but faced some minor problems, like problems with statistical calculations. Two had strange, unexpected results and were deemed “uninterpretable” because they could not be reasonably compared to the original study results. The last was unable to be reproduced. (3) With such mixed findings, no one is quite sure what this means for the field yet.
It might be easy to be unsettled by these results, but it’s important to keep in mind that just because these results were not replicated immediately, does not mean that the findings are not true. There are a number of factors that might make the process less than straightforward. Some studies’ methods and protocols are vague or not specific enough to accurately recreate, and could thus create unforeseen areas of variability. Reproducibility Project scientists are working with the original study authors to understand where those differences could arise. While some authors worry that a single failure to replicate will reflect poorly on them and their work, the goal is not to cast doubt on good work, but to encourage better reporting practices in the future.
Despite concerns, The Reproducibility Project will likely help the scientific community in several key ways. First, it could help researchers and oncologists understand why some preclinical studies fail to become viable cancer treatments. It might help implement new research checks and practices or prevent time wasted on clinical studies that can’t be well reproduced and won’t help patients in the long run. Second, it will encourage scientists to communicate their findings better by using more specific and clear language in their publications and eliminate confusion surrounding possibly important medical findings. And finally, but perhaps most importantly, scientists may become more skeptical of even their own work, and will then work harder to be sure that they are putting out solid, reproducible science.
For more information on The Reproducibility Project: https://osf.io/e81xl/wiki/home/
- Begly CG and Ellis CG, Drug development: Raise standards for preclinical cancer research. Nature 483, 531-533 (29 March 2012).
- Baker, Monya and Dolgin, Elie. “Cancer reproducibility project release first results.” 18 January 2017. http://www.nature.com/news/cancer-reproducibility-project-releases-first-results-1.21304. Accessed 10 February 2017.
- “Reproducibility Project: Cancer Biology.” eLife Sciences Publications, Ltd. Web. Accessed 10 February 2017.