Concordance, efficiency and regional application of epidemiological methodologies versus randomized trials: a systematic review and meta-epidemiological meta-analysis
DOI:
https://doi.org/10.71068/hs000106Keywords:
epidemiology, research methodology, meta-epidemiology, target trial emulation, real-world evidence, ratio of ratios, health equityAbstract
Introduction: the choice of study design conditions the validity, timeliness and usefulness of epidemiological evidence. We aimed to estimate the concordance of non-randomized and emerging designs with the randomized controlled trial (RCT) and to describe their efficiency and regional application. Method: systematic review and meta-epidemiological meta-analysis following PRISMA 2020. We searched MEDLINE, Embase, Scopus, Web of Science and Cochrane (January 2020 to May 2026). The primary comparison was restricted to five non-randomized versus randomized-trial estimates; a six-estimate mixed-design pool was retained as exploratory. Random-effects analyses included REML, Hartung-Knapp-Sidik-Jonkman and a joint exclusion of potentially overlapping sources. Risk of bias, adapted GRADE certainty and ordinal regional and efficiency matrices were assessed. Results: Twenty-four studies were included. Five non-randomized versus randomized-trial comparisons yielded a RoR of 1.00 (95% CI 0.92–1.10); the exploratory six-estimate pool yielded 0.98 (0.89–1.06; I² = 77.9%). Joint exclusion of Toews and Kimachi yielded approximately 0.95 (0.88–1.02) among the three remaining non-randomized comparisons, reconstructed from published rounded estimates. Subgroup RoRs were 1.05 for observational designs and 0.91 for trial emulation/real-world evidence. Certainty was low and very low, respectively. Of 28 highest regional ratings, 16 relied on indirect evidence. Conclusions: A pooled RoR close to 1 does not establish equivalence with randomized trials. Between-design comparisons and regional scoring require cautious, family-specific interpretation; the proposed combination of methods remains a hypothesis for future evaluation.
References
1. Kim DH, et al. The impact of study size on COVID-19 treatment outcomes: a meta-epidemiological study comparing large and small randomized controlled trials. Rev Med Virol. 2026;36(1):e70125. https://doi.org/10.1002/rmv.70125.
2. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. https://doi.org/10.1136/bmj.n71.
3. Hernán MA, Wang W, Leaf DE. Target trial emulation: a framework for causal inference from observational data. JAMA. 2022;328(24):2446-7. https://doi.org/10.1001/jama.2022.21383.
4. Hemming K, Taljaard M. Reflection on modern methods: when is a stepped-wedge cluster randomized trial a good study design choice? Int J Epidemiol. 2020;49(3):1043-52. https://doi.org/10.1093/ije/dyaa077
5. RECOVERY Collaborative Group; Horby P, Lim WS, Emberson JR, Mafham M, Bell JL, et al. Dexamethasone in hospitalized patients with Covid-19. N Engl J Med. 2021;384(8):693-704. https://doi.org/10.1056/NEJMoa2021436.
6. Toews I, Anglemyer A, Nyirenda JLZ, Alsaid D, Balduzzi S, Grummich K, et al. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials: a meta-epidemiological study. Cochrane Database Syst Rev. 2024;1(1):MR000034. https://doi.org/10.1002/14651858.MR000034.pub3.
7. Concato J, Corrigan-Curay J. Real-world evidence: where are we now? N Engl J Med. 2022;386(18):1680-2. https://doi.org/10.1056/NEJMp2200089
8. Dean NE, Hogan JW, Schnitzer ME. Covid-19 vaccine effectiveness and the test-negative design. N Engl J Med. 2021;385(15):1431-3. https://doi.org/10.1056/NEJMe2113151.
9. Sanderson E, Glymour MM, Holmes MV, Kang H, Morrison J, Munafò MR, et al. Mendelian randomization. Nat Rev Methods Primers. 2022;2:6. https://doi.org/10.1038/s43586-021-00092-5.
10. Bergeri I, Whelan MG, Ware H, Subissi L, Nardone A, Lewis HC, et al. Global SARS-CoV-2 seroprevalence from January 2020 to April 2022: a systematic review and meta-analysis of standardized population-based studies. PLoS Med. 2022;19(11):e1004107. https://doi.org/10.1371/journal.pmed.1004107.
11. Brito AF, Semenova E, Dudas G, Hassler GW, Kalinich CC, Kraemer MUG, et al. Global disparities in SARS-CoV-2 genomic surveillance. Nat Commun. 2022;13(1):7003. https://doi.org/10.1038/s41467-022-33713-y.
12. Diamond MB, Keshaviah A, Bento AI, Conroy-Ben O, Driver EM, Halden RU, et al. Wastewater surveillance of pathogens can inform public health responses. Nat Med. 2022;28(10):1992-5. https://doi.org/10.1038/s41591-022-01940-x.
13. Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021;374:n2061. https://doi.org/10.1136/bmj.n2061.
14. Curran GM, Landes SJ, McBain SA, Pyne JM, Smith JD, Fernandez ME, et al. Reflections on 10 years of effectiveness-implementation hybrid studies. Front Health Serv. 2022;2:1053496. https://doi.org/10.3389/frhs.2022.1053496.
15. Ortiz K, Nash J, Shea L, Oetzel J, Garoutte J, Sanchez-Youngman S, et al. Partnerships, processes, and outcomes: a health equity-focused scoping meta-review of community-engaged scholarship. Annu Rev Public Health. 2020;41:177-99. https://doi.org/10.1146/annurev-publhealth-040119-094220.
16. Kimachi M, Onishi A, Tajika A, Kimachi K, Furukawa TA. Systematic differences in effect estimates between observational studies and randomized control trials in meta-analyses in nephrology. Sci Rep. 2021;11(1):6088. https://doi.org/10.1038/s41598-021-85519-5.
17. Stadelmaier J, Bantle G, Gorenflo L, Kiesswetter E, Nikolakopoulou A, Schwingshackl L. Evaluating agreement between individual nutrition randomised controlled trials and cohort studies: a meta-epidemiological study. BMC Med. 2025;23. https://doi.org/10.1186/s12916-025-03860-2
18. Schwingshackl L, Balduzzi S, Beyerbach J, Bröckelmann N, Werner SS, Zähringer J, et al. Evaluating agreement between bodies of evidence from randomised controlled trials and cohort studies in nutrition research: meta-epidemiological study. BMJ. 2021;374:n1864. https://doi.org/10.1136/bmj.n1864.
19. Bröckelmann N, Balduzzi S, Harms L, Beyerbach J, Petropoulou M, Kubiak C, et al. Evaluating agreement between bodies of evidence from randomized controlled trials and cohort studies in medical research: a meta-epidemiological study. BMC Med. 2022;20(1):174. https://doi.org/10.1186/s12916-022-02369-2.
20. Wang SV, Schneeweiss S; RCT-DUPLICATE Initiative. Emulation of randomized clinical trials with nonrandomized database analyses: results of 32 clinical trials. JAMA. 2023;329(16):1376-85. https://doi.org/10.1001/jama.2023.4221.
21. Heyard R, Held L, Schneeweiss S, Wang SV. Design differences and variation in results between randomised trials and non-randomised emulations: meta-analysis of RCT-DUPLICATE data. BMJ Med. 2024;3(1):e000709. https://doi.org/10.1136/bmjmed-2023-000709.
22. Franklin JM, Patorno E, Desai RJ, Glynn RJ, Martin D, Quinto K, et al. Emulating randomized clinical trials with nonrandomized real-world evidence studies: first results from the RCT DUPLICATE initiative. Circulation. 2021;143(10):1002-13. https://doi.org/10.1161/CIRCULATIONAHA.120.051718.
23. WHO Solidarity Trial Consortium; Pan H, Peto R, Henao-Restrepo AM, Preziosi MP, Sathiyamoorthy V, et al. Repurposed antiviral drugs for Covid-19: interim WHO Solidarity trial results. N Engl J Med. 2021;384(6):497-511. https://doi.org/10.1056/NEJMoa2023184.
24. Andaur Navarro CL, Damen JAA, Takada T, Nijman SWJ, Dhiman P, Ma J, et al. Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ. 2021;375:n2281. https://doi.org/10.1136/bmj.n2281.
25. Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22(4):139-42. https://doi.org/10.1136/ebmed-2017-110713.
26. Means AR, Kemp CG, Gwayi-Chore MC, Gimbel S, Soi C, Sherr K, et al. Evaluating and optimizing the consolidated framework for implementation research (CFIR) for use in low- and middle-income countries: a systematic review. Implement Sci. 2020;15(1):17. https://doi.org/10.1186/s13012-020-0977-0.
27. Breilh J. Critical epidemiology and the people's health. New York: Oxford University Press; 2021. ISBN: 9780190492793.
28. Ministerio de Salud Pública y Asistencia Social de Guatemala, Organización Panamericana de la Salud. Guía de diálogos interculturales en salud. Guatemala: MSPAS, OPS; 2024. Disponible en: https://www.paho.org/es/documentos/guia-dialogos-interculturales-salud
29. Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160. https://doi.org/10.1136/bmj.n160.
30. Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225-34. https://doi.org/10.1016/j.jclinepi.2015.06.005.
31. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383-94. https://doi.org/10.1016/j.jclinepi.2010.04.026.
32. IntHout J, Ioannidis JPA, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol. 2014;14:25. https://doi.org/10.1186/1471-2288-14-25.
33. Langan D, Higgins JPT, Jackson D, Bowden J, Veroniki AA, Kontopantelis E, et al. A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses. Res Synth Methods. 2019;10(1):83-98. https://doi.org/10.1002/jrsm.1316.
34. Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995;311(7003):485. https://doi.org/10.1136/bmj.311.7003.485.
35. Huang Bartlett C, Mardekian J, Cotter MJ, Huang X, Zhang Z, Parrinello CM, et al. Concordance of real-world versus conventional progression-free survival from a phase 3 trial of endocrine therapy as first-line treatment for metastatic breast cancer. PLoS One. 2020;15(4):e0227256. https://doi.org/10.1371/journal.pone.0227256.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Marco Antonio Ramos Espinal
This work is licensed under a Creative Commons Attribution 4.0 International License.
Articles published in this journal are available under the Creative Commons Attribution 4.0 International (CC BY 4.0) licence. This licence authorises third parties to download, copy, distribute, adapt and reuse the contents, including for commercial purposes, provided that credit is given to the original author of the work.
