Reproducibility Project only Partially Able to Validate Findings of Prominent Cancer Studies

The Reproducibility Project: Cancer Biology has been working to confirm the results in five prominent cancer biology experiments, and just released the results of its preliminary findings on Jan. 19, 2017. The researchers were only able to validate the findings in two of the five initial studies. The researchers could not replicate the findings in one study, and deemed another two studies “uninterpretable” because the tumor cells under investigation either grew too quickly or too slowly to measure whether the cancer drug had the predicted effect.

Erkki Ruoslahti, a cancer biologist at the Sanford Burnham Prebys Medical Discovery Institute, is an author of one of the original papers that was included in the Reproducibility Study. The replication authors could not reproduce the findings from Ruoslahti’s original study. Ruoslahti told Nature News that a least a dozen other labs have successfully replicated his findings, and there are more studies in the pipeline confirming his original conclusions. He told the publication that he worries that the findings released on Jan. 19, 2017 could influence the future funding of his research.

An editorial accompanying the research findings asserted that it would be virtually impossible to directly replicate all of a study’s findings. Thus, rather than strive for statistical significance in their replication studies (citing a less than 50% chance that all three experiments would yield a significant p value), the authors focused on whether or not they could see effects that were similar to those demonstrated in the original studies following the methodology in the original studies. Although 100% reproducibility is an unrealistic expectation, the authors were seeking to come to some of the same conclusions as the original study authors.

The replication team interviewed original study authors in an effort to avoid errors due to discrepancies in samples and materials. The team “undertook authentication of key biological materials (such as STR profiling of cell lines)” and asked the authors of the original studies to “share any original reagents, protocols, and data in order to maximize the quality and fidelity of the replication designs.”

Methodology sections need work

The takeaway? Small changes in experimental design, such as buffer conditions, pH, slight differences in cell line, reagents used in studies, cell culture changes, and even differences in tubing and labware suppliers could theoretically change the outcomes of experiments. Although it was difficult to draw conclusions from some of the results in the five featured reproducibility studies, replication study authors deduced while conducting these experiments that too little information is typically included in the “materials and methods” sections of most scientific studies. And, the fact that the Reproducibility Project authors could not produce many of the same results in their experiments highlights the inherently complex nature of living cells and how they react to different manipulations and variables.

The accompanying editorial acknowledged that replication efforts could be affected by inadequate expertise of the contract labs performing the replication studies, and this could be a disadvantage: “the contract research laboratories performing the replications may not have the same level of expertise or motivation as the original laboratories.” Elizabeth Iorns of the Reproducibility Project, however, told Ed Yong at The Atlantic that the contract labs used for the replication experiments were proficient at running these types of experiments.

The Reproducibility Project has 24 more experiments that it hopes to try to reproduce, and will publish its findings throughout 2017.

Leonard P. Freedman, PhD, who has written on the crisis of reproducibility and is president of the Global Biological Standards Institute (GBSI), tells BioPharm International that he is not surprised with the results from the first five replication studies, adding that irreproducibility in biomedical research is already a well-established concept. He notes that transparency on methodologies, reagents, data analysis, and other key variables has not been common in science. "The significance of the RP:CB project [Reproducibility Project] is how transparency is a central feature of the effort. While reproducibility in biology is difficult because of the inherent variabilities at play in biological systems, the RP:CB project hopes that transparency in how experiments are carried out-precisely so that the work can be effectively scrutinized and then repeated-will become the norm."

Freedman says these studies prove that reagent variability "is a huge contributor to variability in published results." He adds, "GBSI’s work in firming up standards and validation guidelines for, in particular, cell lines and research antibodies, can go a long way to addressing irreproducibility of experiments, including several of the problems highlighted in the five RP:CB papers."

Incomplete disclosure affects all aspects of scientific research

Irreproducibility is also a problem for the patents that are meant to support and protect innovation in pharma, as Jacob S. Sherkow recently wrote in Duke Law Journal. He wrote that patents, which disclose the details of an invention or method, are supposed to describe a scientific result that can be verified and replicated by scientists. "To the contrary, the availability of patents for the products of clinical research appears to hamper or even actively dissuade reproducibility,” Sherkow wrote. “Since pharmaceutical manufacturers often structure drug development and clinical research around patent protection, patent law’s deficiencies with respect to disclosure encourage manufacturers to engage in research-often, irreproducible research-that satisfies the bare minimum needed to obtain protection. This failure of patent law exacerbates the current real-world reproducibility crisis.”

Sources: Nature, eLife, The Atlantic, Duke Law Journal

*Editorial note: The article has been updated to include comments from Leonard P. Freedman.