Complying With the National Institutes of Health Guidelines and Principles for Rigor and Reproducibility
In 2014, the National Institutes of Health (NIH) delivered a document that described the principles and guidelines for reporting preclinical research (https://www.nih.gov/research-training/rigor-reproducibility/principles-guidelines-reporting-preclinical-research). These principles and guidelines were developed in a workshop that addressed the concern of reproducibility of preclinical research and were built on previous NIH recommendations for transparency in reporting data.1 A major driving force behind this effort stemmed from concerns voiced by the pharmaceutical industry that described their failure to replicate studies performed in academic laboratories2,3 These issues have also attracted attention through commentaries in both scientific journals4 and the lay press.5
As guardians of a large investment of public funds, the NIH aspires to promote confidence in science through enhanced reporting of protocols and data in journals, focusing in particular on reproducibility of results of similar experiments performed in different laboratories. A wide spectrum of scientific leaders, including many journal editors, attended the meeting at which this document was developed. The consequent report emphasized the need for journals to modify their editorial processes in several respects. These included requirement that authors document a detailed description of statistical analyses and data reporting with the aim of increasing transparency, sharing of data and materials, and establishing best practices for reagent verification. An additional requirement was for journals to establish a standard mechanism for the reporting of refutations of published work. All American Heart Association journals, including ATVB, have endorsed these NIH guidelines.
The NIH guidelines state specifically that “the journal assumes responsibility to consider refutations of the publication, in accord with the usual standards of quality.” The guidelines do not define a standardized mode of handling refutations; rather, they suggest that each journal should develop its own policy. This issue of ATVB presents the policy that the editors have developed to deal with the issue of refutations.
The ATVB editors emphasize that reporting of a refutation does not imply any fraudulent aspect of the original publication. As scientists, we are well aware that not all scientific findings are precisely reproducible in different laboratories. Indeed, the editors regard scientific fraud as only 1 contributor to failure to reproduce results among various others. Instead, we think that the lack of reproducibility is more often a reflection of the complexity of biomedical research, in which minor changes in protocols or reagents could yield unintentional differences in data generation. Reproducibility can be enhanced by providing a detailed Materials and Methods section, as already requested by the ATVB Instructions for Authors. This section is typically published online, thus eliminating any potential barriers attributable to length of text or concerns over extra costs incurred when providing comprehensive details. But even with extensive and granular description of methodology, some unforeseen variables may affect experimental outcome. For example, the current understanding of the profound and manifold effects of intestinal microbiota or of the specific vivaria conditions are but 2 aspects of unmeasured background that could potentially influence results and could vary between laboratories in a manner that is not easily monitored or documented.
So how should our journal respond when we are faced with new data that contradicts the findings of a previous ATVB publication? One approach would be to permit the authors to describe the conflicting data along with differences in interpretation and conclusions in a letter. Letters are published on the journal’s website and not in the print version. Depending on the extent of disagreement, this may be an appropriate path allowing the journal to fulfill its commitment to the NIH guidelines. However, a shortcoming of this approach is the relatively poor dissemination of the opposing information compared with the original publication. For example, a letter to ATVB will not be cited in PubMed, which remains the primary source by which many scientists access and retrieve data. Therefore, although publication of the refutation as a letter allows for some dissemination, the inherently limited access seems to be at odds with the spirit of the NIH guidelines, which emphasize wide disbursement to the scientific community. On the other hand, if the conflicting data do not negate or override the major conclusions or interpretation of the original article, a letter may still represent an appropriate outlet for dissemination.
The editors consider that presenting data that question the major premise of a previous publication needs to be handled in a distinct manner. To be consistent with the content and spirit of the NIH guidelines, data that fundamentally overturn a conclusion or interpretation of a previous publication should be submitted in article format with clear description of the conflicting data. It is the responsibility of the ATVB editors to ensure that an article that presents conflicting data is reviewed in a fair and balanced manner. At the ATVB journal, all articles undergo careful, but expeditious external review, acquiring detailed critiques from 3 expert reviewers that are subsequently adjudicated by at least 2 editors, with the entire editorial board having access to all decisions. In the case of a refutation, all individuals involved in the review of the previously published article—editors and external reviewers—are alerted to that fact that a refutation has entered the peer-review process. In any event, refutations are dealt with in a manner consistent with the standards of the journal for any article reporting original findings. Specifically, the peer-review process focuses on the quality of the data and interpretation in a manner that is equivalent to all other submissions to the journal. It should be noted that despite its intrinsic rigor, peer review cannot be expected to anticipate the possibility that other laboratories may generate conflicting data. Therefore, the publication of a refutation is not a reflection of any possible deficiencies of peer review of the original article.
As noted at the outset, journals have been requested to develop a policy for handling refutations. In response, the editors of ATVB have conceived and developed a policy in which each request for refutations will be assessed by subject experts from the editorial team. Depending on the extent of contradiction or scientific divergence, the editors can guide the authors toward the appropriate article format. Possibilities range from either a letter that contradicts some aspects of the published study to, at the other end of the spectrum, a full-scale publication describing in detail a refutation that contradicts the core conclusions of the original publication; such a article will, itself, be submitted for full peer review, as detailed above. We anticipate that the application of a fair and uniform policy for handling refutations will ultimately prove to be an asset to the scientific community.
The opinions expressed in this article are not necessarily those of the editors or of the American Heart Association.
- © 2016 American Heart Association, Inc.