Credible evidence

Avebury henge. Photo courtesy H. Colt

Here we go again. Just when the general public needs credible scientific evidence regarding COVID-19, another leading journal publishes controversial data, this time from a Noble Prize recipient. After the Proceedings of the National Academy of Sciences (PNAS) published the paper Identifying airborne transmission as the dominant route for the spread of COVID-191, the paper quickly rose to the top 5% in Altmetric’s list of papers being shared and discussed globally2. Meanwhile, a group of more than 50 leading researchers wrote the journal, describing “serious methodological errors that undermine any confidence in its findings”3,4 and requesting that PNAS immediately retract the publication.

Other leading journals, including the New England Journal of Medicine, The Lancet, and the Annals of Internal Medicine are also guilty of publishing papers with conclusions that could not be justified by the evidence5.

Health care professionals and the general public look to the scientific community for leadership and expert advice. It is in the nature of scientific inquiry to bear controversy and generate debate in the search for truth. Therein lies an assumption of responsibility and accountability that is not always equally borne by authors, editorial board members, and reviewers.

Those of us who have published widely know and understand the politics of peer-review. We know the fragility of the process, and how sometimes personal vendettas or reviews done in poor faith may prompt rejections. We recognize the unwillingness of many editors to publish studies with negative findings or papers with conclusions that might justify a contrarian position. We may not always accept the often stern and sometimes unfounded critiques of reviewers who recommend rejection. We revise papers when told that our conclusions are not justified by the results, that results are not addressed by our methods, or when our discussion overstates the study’s objectives.

The purpose of scientific peer review is, among others, to question the validity as well as style of the science presented. It is also to find errors, suggest corrections, and recommend revisions that might improve a paper’s readability. It is not always easy for reviewers to accept as valid, findings that run contrary to one’s predetermined biases, or to accept as valid a well-laid argument that puts in doubt a lifetime of one’s own work. That is, as I mentioned earlier, in the nature of scientific inquiry, and it is partly the responsibility of a diligent peer-review.

This is also a responsibility that ultimately resides with the reader. In my own field of Bronchology and Interventional Pulmonology, I wonder if Train-the-Trainer workshops should include sessions on critical thinking. There could be frank discussions about how to teach students to formulate hypotheses, justify scientific findings without going beyond what an honest analysis of the data provides, and credibly argue opinions. 

Well-informed readers do not need to rely on where an article is published to establish the paper’s credibility or scientific value. They are able to reject poorly designed studies, papers reporting questionable evidence, and authors who overstate their positions. It is one thing to rely on credible evidence, but it is quite another to know whether the evidence is credible.



Please subscribe to Colt’s Corner to automatically receive email notification of future blog posts.