Comparison of Effect Sizes Associated With Biomarkers Reported in Highly Cited Individual Articles and in Subsequent Meta-Analyses

Study Questions:

How reliable are the effect sizes of biomarkers reported in highly cited studies?


Eligible biomarker studies were those that had received more than 400 citations in the ISI Web of Science. A MEDLINE search for subsequent meta-analyses on the same associations (same biomarker and same outcome) was also performed. In the highly cited studies, data extraction was focused on the disease/outcome, biomarker under study, and first reported relative risk in the abstract. From each meta-analysis, the overall relative risk and the relative risk in the largest study was extracted.


Thirty-five highly cited associations were analyzed. For 30 of the 35 (86%), the highly cited studies had a stronger effect estimate than the largest study; for three, the largest study was also the highly cited study; and only twice was the effect size estimate stronger in the largest than in the highly cited study. For 29 of the 35 (83%) highly cited studies, the corresponding meta-analysis found a smaller effect estimate. Only 15 of the associations were nominally statistically significant based on the largest studies, and of those, only seven had a relative risk point estimate greater than 1.37.


The authors concluded that highly cited biomarker studies often report larger effect estimates for postulated associations than are reported in subsequent meta-analyses evaluating the same associations.


Citation bias for extreme results and failure to replicate initial exciting observations is common in all areas of research. This has been studied and shown for clinical interventions/outcomes, genome-wide association studies, and now biomarkers. There are multiple explanations for this, and a healthy skepticism of the medical literature is needed, especially when the results of a study may impact patient management. Progress in the medical field is much slower than the literature would indicate.

Keywords: Risk, Biological Markers, Cardiology, Bias (Epidemiology), Genome-Wide Association Study, Biomedical Research

< Back to Listings