MacAma: Multi-AI Agent as a Co-Scientist for Automated Meta-Analysis

Abstract

Meta-analysis is fundamental to evidence-based medicine, yet traditional workflows remain labor-intensive and susceptible to bias. Although LLM-based research agents offer opportunities for workflow automation, they often lack the data fidelity and methodological traceability required for rigorous quantitative evidence synthesis, particularly when parsing multimodal scientific charts. To address this challenge, we introduce MacAma, a semi-automated multi-agent framework for protocol-constrained and human-verifiable meta-analysis. MacAma operationalizes selected PRISMA 2020 reporting items, PICOS-based eligibility logic, and SYRCLE risk-of-bias domains as structured prompts, decision rules, output fields, and audit records. Critically, MacAma adopts a risk-aware automation strategy: Lower risk, repetitive, and protocol-driven tasks, such as literature screening and drafting, are delegated to AI agents, whereas high-impact steps that directly affect effect-size estimation and statistical conclusions, such as quantitative chart-data extraction, remain subject to expert verification. In a preclinical radiotherapy case study evaluating tumor-related immune outcomes and metastatic potential mediated by circulating tumor cells, MacAma achieved competitive screening performance in the evaluated benchmark and reduced the manual screening burden by over 80% within the current workflow. The case study further demonstrates how structured agent outputs, predefined criteria, and audit records can support transparent screening, data extraction, statistical synthesis, and manuscript drafting. These results suggest that MacAma may provide a scalable and auditable framework for AI-assisted meta-analysis, although important limitations remain in full-text access, quantitative chart data extraction, and expert interpretation of heterogeneity. MacAma is open-source and available at https://github.com/YilinYuan/MacAma.

Keywords

large language models / literature screening / meta-analysis / prompt engineering / research automation

References

[1] L. S. Uman, “Systematic Reviews and Meta-Analyses,” Journal of the Canadian Academy of Child and Adolescent Psychiatry 20 (2011): 57–59.

[2] M. Pai, M. McCulloch, J. D. Gorman, et al., “Systematic Reviews and Meta-Analyses: An Illustrated, Step-by-Step Guide,” National Medical Journal of India 17 (2004): 86–95.

[3] M. J. Bown and A. J. Sutton, “Quality Control in Systematic Reviews and Meta-Analyses,” European Journal of Vascular and Endovascular Surgery 40 (2010): 669–677.

[4] M. J. Page, J. E. McKenzie, P. M. Bossuyt, et al., “The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews,” BMJ 372 (2021): n71.

[5] S. Roth and A. Wermer-Colan, “Machine Learning Methods for Systematic Reviews: A Rapid Scoping Review,” Delaware Journal of Public Health 9 (2023): 40–47.

[6] S. Van der Mierden, K. Tsaioun, A. Bleich, and C. H. C. Leenaars, “Software Tools for Literature Screening in Systematic Reviews in Biomedical Research,” Altex 36 (2019): 508–517.

[7] Q. Chen, M. Yang, L. Qin, et al., “AI4Research: A Survey of Artificial Intelligence for Scientific Research,” preprint, arXiv, July 2, 2025, https://doi.org/10.48550/arXiv.2507.01903.

[8] Y. He, G. Huang, P. Feng, et al., “PaSa: An LLM Agent for Comprehensive Academic Paper Search,” preprint, arXiv, January 17, 2025, https://doi.org/10.48550/arXiv.2501.10120.

[9] Y. Wang, Q. Guo, W. Yao, et al., “AutoSurvey: Large Language Models Can Automatically Write Surveys,” Advances in Neural Information Processing Systems 37 (2024): 115119–115145.

[10] X. Yan, S. Feng, J. Yuan, et al., “Surveyforge: On the Outline Heuristics, Memory-Driven Generation, and Multi-Dimensional Evaluation for Automated Survey Writing,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, Vol. 1, ed. W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Association for Computational Linguistics, 2025), 12444–12465.

[11] C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha, “The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery,” preprint, arXiv, August 12, 2024, https://doi.org/10.48550/arXiv.2408.06292.

[12] S. Schmidgall, Y. Su, Z. Wang, et al., “Agent Laboratory: Using LLM Agents as Research Assistants,” preprint, arXiv, January 8, 2025, https://doi.org/10.48550/arXiv.2501.04227.

[13] T. W. Lin, R. Yang, Z. ul Abdeen, A. Sangiovanni-Vincentelli, H. Huang, and M. Jin, “LLMs Tackle Meta-Analysis: Automating Scientific Hypothesis Generation With Statistical Rigor,” in AI for Research and Scalable, Efficient Systems, ed. Q. Wang, W. Yin, A. Aich, Y. Suh, and K. C. Peng (Springer, 2025), 38–58.

[14] Q. Wu, G. Bansal, J. Zhang, et al., “AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation,” preprint, arXiv, August 16, 2023, https://doi.org/10.48550/arXiv.2308.08155.

[15] S. Hong, M. Zhuge, J. Chen, et al., “MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework,”preprint, arXiv, August 1, 2023, https://doi.org/10.48550/arXiv.2308.00352.

[16] L. Gao, X. Ma, J. Lin, and J. Callan, “Precise Zero-Shot Dense Retrieval Without Relevance Labels,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Vol. 1, ed. A. Rogers, J. Boyd-Graber, and N. Okazaki (Association for Computational Linguistics, 2023), 1762–1777.

[17] S. Robertson and H. Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond,” Foundations and Trends in Information Retrieval 3 (2009): 333–389.

[18] P. Lewis, E. Perez, A. Piktus, et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” in Advances in Neural Information Processing Systems 33, ed. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (NeurIPS, 2020), 9459–9474.

[19] F. M. Delgado-Chaves, M. J. Jennings, A. Atalaia, et al., “Transforming Literature Screening: The Emerging Role of Large Language Models in Systematic Reviews,” Proceedings of the National Academy of Sciences 122 (2025): e2411962122.

[20] M. Zheng, X. Feng, Q. Si, et al., “Multimodal Table Understanding,” preprint, arXiv, June 12, 2024, https://doi.org/10.48550/arXiv.2406.08100.

[21] M. Huang, L. Zhang, J. Ma, et al., “ChartSketcher: Reasoning With Multimodal Feedback and Reflection for Chart Understanding,” preprint, arXiv, May 25, 2025, https://doi.org/10.48550/arXiv.2505.19076.

[22] F. Zhu, Q. Niu, X. Li, Q. Zhao, H. Su, and J. Shuai, “FM-FCN: A Neural Network With Filtering Modules for Accurate Vital Signs Extraction,” Research 7 (2024): 0361.

[23] Q. C. Li, H. Lin, P. Wang, et al., “Analysis of Spatiotemporal Dynamic Patterns of Gene Expression During Mouse Embryonic Development Based on Moran’s I and Spatial Transcriptomics,” Chinese Physics B 34 (2025): 088703.

[24] D. Guo, D. Yang, H. Zhang, et al., “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” preprint, arXiv, January 22, 2025, https://doi.org/10.48550/arXiv.2501.12948.

[25] A. Yang, B. Yang, B. Hui, et al., “Qwen2 Technical Report,” preprint, arXiv, July 15, 2024, https://doi.org/10.48550/arXiv.2407.10671.

[26] Q. He, X. Li, J. Zhong, G. Yang, J. Han, and J. Shuai, “Dear-PSM: A Deep Learning-Based Peptide Search Engine Enables Full Database Search for Proteomics,” Smart Medicine 3 (2024): e20240014.

[27] Z. Shi, R. Deng, Q. Yuan, et al., “Enzyme Commission Number Prediction and Benchmarking With Hierarchical Dual-Core Multitask Learning Framework,” Research 6 (2023): 0153.

[28] C. R. Hooijmans, M. M. Rovers, R. B. De Vries, M. Leenaars, M. Ritskes-Hoitinga, and M. W. Langendam, “SYRCLE’S Risk of Bias Tool for Animal Studies,” BMC Medical Research Methodology 14 (2014): 43.

[29] L. Fan, L. Wang, X. Wang, M. Li, H. Gu, and H. Zhang, “Multifunctional Silk and Gelatin Composed Microneedle Patches for Enhanced Wound Healing,” Smart Medicine 4 (2025): e137.

[30] R. Wang, F. Chen, H. Chen, et al., “Deep Learning in Digital Breast Tomosynthesis: Current Status, Challenges, and Future Trends,” MedComm 6 (2025): e70247.

[31] X. Yang, Y. Wang, Y. Lin, et al., “A Multi-Task Self-Supervised Strategy for Predicting Molecular Properties and FGFR1 Inhibitors,” Advanced Science 12 (2025): 2412987.

[32] Y. Lu, Y. Ying, C. Huang, et al., “STORM Image Denoising and Information Extraction,” Biomedical Physics & Engineering Express 10 (2024): 065028.

[33] A. Masry, X. Do, J. Tan, S. Joty, and E. Hoque, “ChartQA: A Benchmark for Question Answering About Charts With Visual and Logical Reasoning,” in Findings of the Association for Computational Linguistics, ed. S. Muresan, P. Nakov, and A. Villavicencio (Association for Computational Linguistics, 2022), 2263–2279.

[34] R. Xia, H. Ye, X. Yan, et al., “Chartx & Chartvlm: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning,” IEEE Transactions on Image Processing 34 (2025): 7436–7447.

[35] I. J. Marshall and B. C. Wallace, “Toward Systematic Review Automation: A Practical Guide to Using Machine Learning Tools in Research Synthesis,” Systematic Reviews 8 (2019): 163.

[36] G. Tsafnat, P. Glasziou, M. K. Choong, A. Dunn, F. Galgani, and E. Coiera, “Systematic Review Automation Technologies,” Systematic Reviews 3 (2014): 74.

[37] N. Temur, E. E. Eren, I. Celik, et al., “The Recent Development in the Diagnosis of Mycobacterium Tuberculosis,” Smart Medicine 4 (2025): e70007.

[38] H. Lin, H. Hu, Z. Feng, et al., “SCTC: Inference of Developmental Potential From Single-Cell Transcriptional Complexity,” Nucleic Acids Research 52 (2024): 6114–6128.

[39] B. Zheng, Z. Fang, Z. Xu, et al., “Agent4S: The Transformation of Research Paradigms From the Perspective of Large Language Models,” preprint, arXiv, June 30, 2025, https://doi.org/10.48550/arXiv.2506.23692.

[40] M. Frasca, D. La Torre, G. Pravettoni, and I. Cutica, “Explainable and Interpretable Artificial Intelligence in Medicine: A Systematic Bibliometric Review,” Discover Artificial Intelligence 4 (2024): 15.

[41] U. Ehsan and M. Riedl, “Explainable AI Reloaded: Challenging the XAI Status Quo in the Era of Large Language Models,” in Proceedings of the Halfway to the Future Symposium (ACM, 2024), 1–8.

[42] M. Guan, X. Zhang, X. Li, et al., “Research Progress of Osteoarthritis Treatment by Low Intensity Pulsed Ultrasound,” Smart Medicine 4 (2025): e70003.

[43] M. Schwab, N. Karrenbach, and J. Claerbout, “Making Scientific Computations Reproducible,” Computing in Science & Engineering 2 (2000): 61–67.

Rights & permissions

2026 The Author(s). Smart Medicine published by Wiley-VCH GmbH on behalf of Wenzhou Institute, University of Chinese Academy of Sciences.