Cochrane Reviews controversy: Are the concerns valid?

In the annals of medicine, Cochrane Reviews occupy rarefied space—they are widely considered the gold standard of systematic reviews. Nevertheless in recent years, criticisms have been leveled at their validity, quality of evidence, and timeliness. Rather than discouraging such criticism, the Cochrane Collaboration actively embraces, encourages, and even incentivizes it.

For physicians striving to adhere to evidence-based medicine and standard of care, primary sources such as Cochrane reviews are exceedingly important. As with any human research endeavor, bias, topicality, and validity are ubiquitous concerns. So, how do clinicians interpret and utilize findings from these reviews in light of certain concerns raised by other researchers? In this article, we review the background and importance of Cochrane Reviews, outline the controversy, and provide expert perspective through exclusive interviews with MDLinx. 

Cochrane process defined

Systematic reviews are a form of research that involves rigorous and explicit methodology, with an aim to limit bias, provide reliable results, and enlighten decision-making.

Cochrane Reviews are systematic reviews of healthcare research and policy that are published in the Cochrane Database of Systematic Reviews. These reviews are developed with reliable results of studies that meet quality criteria and are composed with the limitations of bias in mind.

Cochrane authors adhere to the following process:

1. Identification of relevant studies from a gamut of sources, which can be unpublished

2. Study selection based on strengths/limitations per clear, predefined criteria

3. Systematic data collection

4. Applicable data synthesis

Criticism of outcomes

In recent years, some researchers have raised concerns over Cochrane Reviews in peer-reviewed studies—specifically the veracity of results, bias, and topicality. 

In the Journal of Clinical Epidemiology, one group of investigators analyzed the number of recent Cochrane Reviews that included outcomes in the literature search strategy. They also looked at how often this step was described as a limitation, as well as the qualitative difference of results included/not included in the search strategy.  

The researchers found that of 523 Cochrane Reviews from 2020, nearly 10% employed outcomes in their search strategy (ie, outcome terms linked with “AND”), with only one study acknowledging this step as a limitation. Furthermore, 92% analyzed outcomes not included in the search strategy, noting that outcomes that were part of search strategies tended to favor the intervention.

The authors stated that the practice of including outcome terms in the search strategy results in overlooking evidence. Still, they strike a conciliatory tone. “It should also be noted that we do not intend to criticize researchers merely for including terms related to outcomes in their search strategies,” they wrote. “Nevertheless, the practice of including outcomes may still overlook relevant studies, thus warranting acknowledgement and discussion.”

Criticism of timeliness

In another study published in the Journal of Clinical Epidemiology, investigators assessed the time lags from publication of a protocol for a Cochrane Review to publication of the review itself. They mined the entire Cochrane Database of Systematic Reviews (CDSR) for 8,201 reviews published between 1995 and 2019.

The authors included 6,764 reviews, which had a median publication time of 2 years (range: 0 days to 21.7 years). Of the total, 11% were published more than 5 years out, and 19% were published within 1 year. The median time ranged from 15 to 39 months.

In an exclusive interview with MDLinx, Mikkel Zola Andersen, MD, of the University of Copenhagen, and the study’s primary author, commented on the implications of these findings.

“I think delays in and of themselves do not necessarily affect the quality of the evidence negatively,” he said. “They may, however, affect how quickly the evidence becomes available, which may in turn affect the quality of treatment, as new knowledge surrounding the treatments are available later. This may result in using a potentially suboptimal treatment for a longer time.”

Andersen plans future research on the nature of delays and possible resolution of these issues. 

“My recommendation is to start by examining which processes may be agreed from all sides are long-drawn and attempting to optimize these processes,” he said.

Criticism of evidence

In a study published in Wiener Medizinische Wochenschrift, authors undertook five systematic literature reviews in neonatology (1996-2010), pediatric neurology (1996-2010), pediatric gastroenterology (1993-2012), pediatric cardiology (2001-2015), and complementary/alternative medicine (1996-2012).

They found that in all five pediatric domains, inconclusive data was abundant, ranging from 26.8% in neurology to 66.9% in complementary/alternative medicine.

“Our updated systematic synopsis reiterates the need for high-quality, sophisticated research to reduce the number of inconclusive meta-analyses in the field of pediatrics—most importantly in the field of complementary and alternative medicine (CAM), neonatology, and pediatric cardiology,” they concluded. “The realization of high-quality, clinically driven research will in turn yield more systematic reviews with a clear conclusion (eg, in favor or against a certain intervention, or treatment modality), thus, substantively decreasing the proportion of inconclusive reviews.”

Cochrane responds

In an exclusive interview with MDLinx, Cochrane responded to issues raised by the aforementioned studies. They readily conceded that such concerns are legitimate.

“Cochrane has a long history of taking seriously the implications of research into how methods are applied in our reviews,” a spokesperson said. “We recognize the value that criticism can have as a way to improve the credibility of our output through the annual award of the Bill Silverman prize.”

The Silverman prize is awarded annually for a published paper that offers constructive recommendations regarding any aspect of Cochrane Reviews or of the work of Cochrane in general. 

One recent winner shed light on measures of outcomes of interest that are underreported or missed in the review. 

“This underlines the importance of involving an information specialist in the development of the review, as well as the importance of peer review of search strategies,” the spokesperson said.

Cochrane also admits that some reviews take too long to publish. 

“Reasons for that are many, and include complexity and level of details that we provide, but most importantly this is a challenge that we have been dealing with by centralizing the editorial process and embarking on a transformation in the way we produce and publish our reviews,” the spokesperson said.

Cited factors that contribute to time lag include question complexity, breadth of scope, number of studies included, and analysis approach.

“The role of automation in the review process is increasingly being used by review teams to make the search process more efficient, although that will reduce only one of the many complexities of the review process. The role of technology in review production has also become more widely acknowledged as an important step in helping teams to collect and manage data,” the spokesperson said.

Cochrane pointed to a number of resources they have developed, including guidance/policies on problematic studies. Although not universally defined, problematic studies can refer to the trustworthiness of data or findings with respect to retracted or unretracted studies. Such studies can result from scientific misconduct, poor research practices, or unintentional errors.

The road ahead

In an editorial, Cochrane editors reflected on the road ahead. 

“The challenge ahead is to establish consensus around reliable, feasible approaches for identifying whether a study is problematic in the first place,” they wrote. “As a starting point we need a standard, agreed definition of a ‘problematic’ study. Research is then needed to develop and validate methods to identify such problematic studies, in order to preserve the status of systematic reviews as a source of definitive health information.”

In an article published in Nature, Lisa Bero, PhD, a Cochrane reviewer who specializes in bias in research design, conduct, and publication, discussed her role in weeding out fraudulent studies from Cochrane Reviews. 

She avers that her work is intensive and includes many steps, such as checking registration, consulting post-publication peer review platforms, and consideration of the plausibility of claims such as study sample size, as well as sites/investigators involved. Reviewers like her must look for inconsistencies, overlapping text, and unlikely outcomes.

“This is tough work,” she wrote. “Some reviewers have suggested that, instead, broad classes of study should be excluded, such as those from certain countries or those that have not been prospectively registered. But that’s problematic: Global representation is important, and registration is no guarantee that a study was done appropriately. Furthermore, observational studies, common in public health, are often unregistered.”

Sources

1. About Cochrane reviews. Cochrane Library.

2. Bero L. Stamp out fake clinical data by working together. Nature.

3. History of systematic reviews. Eppi Center. 

4. Meyer S. The role and limitations of Cochrane reviews at the bedside: a systematic synopsis of five pediatric subspecialties. Wiener Medizinische Wochenschrift.

5. Tsujimoto Y. Around ten percent of most recent Cochrane reviews included outcomes in their literature search strategy and were associated with potentially exaggerated results: A research-on-research study. Journal of Clinical Epidemiology.

   Read Next: The top retractions of 2021