The replication crisis and its relevance to Earth Science studies: Case studies and recommendations

Abstract

Numerous scientific fields are facing a replication crisis, where the results of a study often cannot be replicated when a new study uses independent data. This issue has been particularly emphasized in psychology, health, and medicine, as incorrect results in these fields could have serious consequences, where lives might be at stake. While other fields have also highlighted significant replication problems, the Earth Sciences seem to be an exception. The paucity of Earth Science research aimed at understanding the replication crisis prompted this study. Specifically, this work aims to fill that gap by seeking to replicate geological results involving various types of time-series. We identify and discuss 11 key variables for replicating U-Pb age distributions: independent data, global sampling, proxy data, data quality, disproportionate non-random sampling, stratigraphic bias, potential filtering bias, accuracy and precision, correlating time-series segments, testing assumptions and divergent analytical methods, and analytical transparency. Even while this work primarily focuses on U-Pb age distributions, most of these factors (or variations of them) also apply to other geoscience disciplines. Thus, some of the discussions involve time-series consisting of εHf, δ¹⁸O-zircon, ¹⁴C, ¹⁰Be, marine δ¹³C, and marine δ¹⁸O. We then provide specific recommendations for minimizing adverse effects related to these factors, and in the process enhancing prospects for replicating geological results.

Keywords

Replication crisis / Replicability / Independent data / Global time-series / Globality index / Filtered data

CRediT authorship contribution statement

Stephen J. Puetz: Conceptualization, Methodology, Project administration, Software, Validation, Formal analysis, Data curation, Writing – original draft. Kent C. Condie: Writing – review & editing. Kurt Sundell: Validation, Formal analysis, Writing – review & editing. Nick M.W. Roberts: Writing – review & editing. Christopher J. Spencer: Writing – review & editing. Slah Boulila: Writing – review & editing. Qiuming Cheng: Writing – review & editing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

The authors thank Peter Voice, Malgorzata Lagisz, and an anonymous reviewer, as well as editor Damian Nance for considerable suggestions and guidance that significantly improved the manuscript.

References

[1] J.D. Aber. Why don’t we believe the models?. Bull. Ecol. Soc. Am., 78 (1997), pp. 232-233

[2] M. Baker. 1,500 scientists lift the lid on reproducibility. Nature, 533 (2016), pp. 452-454,

[3] C. Balica, M.N. Ducea, G.E. Gehrels, J. Kirk, R.D. Roban. A zircon petrochronologic view on granitoids and continental evolution. Earth Planet. Sci. Lett., 531 (2020), Article 116005,

[4] A.P. Barth, J.L. Wooden, C.E. Jacobson, R.C. Economos. Detrital zircon as a proxy for tracking the magmatic arc system: the California arc example. Geology, 41 (2013), pp. 223-226,

[5] A.C. Chang, P. Li. Is economics research replicable? Sixty published papers from thirteen journals say “often not”. Critical Finance Rev., 11 (2022), pp. 185-206,

[6] Committee on Reproducibility and Replicability in Science, 2019. Reproducibility and Replicability in Science. National Academies Press, Washington DC, USA. https://www.ncbi.nlm.nih.gov/books/NBK547546/.

[7] K.C. Condie, S.J. Puetz, C.J. Spencer, N.M.W. Roberts. Four billion years of secular compositional change in granitoids. Chem. Geol., 644 (2023), Article 121868,

[8] D.J. Condon, B. Schoene, N.M. McLean, S.A. Bowring, R.R. Parrish. Metrology and traceability of U-PB isotope dilution geochronology (EARTHTIME tracer calibration part I). Geochim. Cosmochim Acc., 164 (2015), pp. 464-480,

[9] G.M. Cox, T.W. Lyons, R.N. Mitchell, D. Hasterok, M. Gard. Linking the rise of atmospheric oxygen to growth in the continental phosphorus inventory. Earth Planet. Sci. Lett., 489 (2018), pp. 28-36,

[10] M.D. Crisp, S.A. Trewick, L.G. Cook. Hypothesis testing in biogeography. Trends Ecol. Evol., 26 (2011), pp. 66-72,

[11] S.C. Dobbs, M.A. Malkowski, T.M. Schwartz, Z.T. Sickmann, S.A. Graham. Depositional controls on detrital zircon provenance: an example from upper cretaceous strata, southern Patagonia. Front. Earth Sci., 2022 (2022), p. 70232516,

[12] M.H. Dodson, W. Compston, I.S. Williams, J.F. Wilson. A search for ancient detrital zircons in Zimbabwean sediments. Geol Soc. London J., 145 (1988), pp. 977-983,

[13] S. Doucet, H. Gamaleldien, Z.X. Li. Pitfalls in using the geochronological information from the EarthChem portal for Precambrian time-series analysis. Precambrian Res., 369 (2022), Article 106514,

[14] U.C. Farrell, R. Samawi, S. Anjanappa, R. Klykov, O.O. Adeboye. The sedimentary geochemistry and paleoenvironments project. Geobiology, 19 (2021), pp. 545-556,

[15] Gehrels, G., Sundell, K., George, S., 2019. Short Course modules on U-Pb Geochronology of Detrital Zircons: Best Practices for U-Pb Data Acquisition, Reduction, Analysis, and Archiving. GSA 2019 Meeting, Sept 22-25; Phoenix, Arizona. https://sites.google.com/a/laserchron.org/laserchron/.

[16] D. Hasterok, M. Gard, G. Cox, M. Hand. A 4 Ga record of granitic heat production: implications for geodynamic evolution and crustal composition of the early earth. Precambrian Res., 331 (2019), Article 105375,

[17] F.J. Hilgen, L.A. Hinnov, H.A. Aziz, H.A. Abels, S. Batenburg. Stratigraphic continuity and fragmentary sedimentation: the success of cyclostratigraphy as part of integrated stratigraphy. Geol. Soc. London Spec. Pubs., 404 (2015), pp. 157-197,

[18] L.A. Hinnov. Cyclostratigraphy and its revolutionizing applications in the earth and planetary sciences. GSA Bull., 125 (2013), pp. 1703-1734,

[19] M.S.A. Horstwood, J. Košler, G. Gehrels, S.E. Jackson, N.M. McLean, C. Paton. Community-derived standards for LA-ICP-MS U-(th-)pb geochronology – uncertainty propagation, age interpretation and data reporting. Geostand. Geoanal. Res., 40 (2016), pp. 311-332,

[20] J.P.A. Ioannidis. Why Most published research findings are false. PLoS Med., 2 (2005), p. e124

[21] T.E. Johnson, C.L. Kirkland, N.J. Gardiner, M. Brown, R.H. Smithies, M. Santosh. Secular change in TTG compositions: implications for the evolution of Archaean geodynamics. Earth Planet. Sci. Lett., 505 (2019), pp. 65-75,

[22] T.D. Jones, D.J. Lunt, D.N. Schmidt, A. Ridgwell, A. Sluijs, P.J. Valdes, M. Maslin. Climate model and proxy data constraints on ocean warming across the Paleocene-Eocene Thermal Maximum. Earth-Sci. Rev., 125 (2013), pp. 123-145,

[23] C. Keller, B. Schoene. Statistical geochemistry reveals disruption in secular lithospheric evolution about 2.5 Gyr ago. Nature, 485 (2012), pp. 490-493,

[24] J. Košler, J. Sláma, E. Belousova, F. Corfu, G.E. Gehrels. U-pb detrital zircon analysis – results of an inter-laboratory comparison. Geostand. Geoanal. Res., 37 (2013), pp. 243-259,

[25] M. Kuhn, K. Johnson. Over-fitting and model tuning. Applied Predictive Modeling, Springer, New York (2013),

[26] B.E. Kushner, G.S. Soreghan, M.J. Soreghan. Late Paleozoic cratonal sink: distally sourced sediment filled the Anadarko Basin (USA) from multiple source regions. Geosphere, 18 (2022), pp. 1831-1850,

[27] J. Laskar, P. Robutel, F. Joutel, M. Gastineau, A.C.M. Correia, B. Levard. A long-term numerical solution for the insolation quantities of the Earth. Astron. Astrophys., 428 (2004), pp. 261-285,

[28] H. Liu, R.E. Zartman, T.R. Ireland, W.D. Sun. Global atmospheric oxygen variations recorded by Th/U systematics of igneous rocks. Proc. Natl. Acad. Sci., 116 (2019), pp. 18854-18859,

[29] G.M. Lu, W. Wang, R.E. Ernst, H. El Bilali, C.J. Spencer, Y.G. Xu, A. Bekker. Evolutionary stasis during the Mesoproterozoic Columbia-Rodinia supercontinent transition. Precambrian Res., 391 (2023), Article 107057,

[30] R.M. McDowall. What biogeography is: a place for process. J. Biogeogr., 31 (2004), pp. 345-351,

[31] S.R. Meyers. The evaluation of eccentricity-related amplitude modulation and bundling in paleoclimate data: an inverse approach for Astro-chronologic testing and time scale optimization. Paleoceanogr. Paleocl., 30 (2015), pp. 1625-1640,

[32] R. Moonesinghe, M.J. Khoury, A.C.J.W. Janssens. Most published research findings are false—But a little replication goes a long way. PLoS Med., 4 (2007), p. e28

[33] D.B. Owen. The power of student's t-test. J. Am. Stat. Assoc., 60 (1965), pp. 320-333,

[34] S.J. Puetz. A relational database of global U-Pb ages. Geosci. Front., 9 (2018), pp. 877-891,

[35] S.J. Puetz, A. Prokoph, G. Borchardt. Evaluating alternatives to the Milankovitch theory. J. Stat. Planning Infer., 170 (2016), pp. 158-165,

[36] S.J. Puetz, K.C. Condie, S. Pisarevsky, A. Davaille, C.J. Schwarz, C.E. Ganade. Quantifying the evolution of the continental and oceanic crust. Earth-Sci. Rev., 164 (2017), pp. 63-83,

[37] S.J. Puetz, K.C. Condie. Time series analysis of mantle cycles Part I: Periodicities and correlations among seven global isotopic databases. Geosci. Front., 10 (2019), pp. 1305-1326,

[38] S.J. Puetz, K.C. Condie. Applying Popperian falsifiability to geodynamic hypotheses: empirical testing of the episodic crustal/zircon production hypothesis and selective preservation hypothesis. Int. Geol. Rev., 63 (2021), pp. 1920-1950,

[39] S.J. Puetz, K.C. Condie. A review of methods used to test periodicity of natural processes with a special focus on harmonic periodicities found in global U-Pb detrital zircon age distributions. Earth-Sci. Rev., 224 (2022), Article 103885,

[40] S.J. Puetz, C.E. Ganade, U. Zimmermann, G. Borchardt. Statistical analyses of global U-Pb database 2017. Geosci. Front., 9 (2018), pp. 121-145,

[41] S.J. Puetz, C.J. Spencer, C.E. Ganade. Analyses from a validated global U-Pb detrital zircon database: enhanced methods for filtering discordant U-Pb zircon analyses and optimizing crystallization age estimates. Earth-Sci. Rev., 220 (2021), Article 103745,

[42] S.J. Puetz, C.J. Spencer. Evaluating U-Pb accuracy and precision by comparing zircon ages from 12 standards using TIMS and LA-ICP-MS methods. Geosyst. Geoenviron., 2 (2023), Article 100177,

[43] S.J. Puetz, C.J. Spencer, K.C. Condie, N.M.W. Roberts. Enhanced U-Pb detrital zircon, Lu-Hf zircon, δ18O zircon, and Sm-Nd whole rock global databases. Sci. Data, 11 (2024), p. 56

[44] D.T. Redden, D.B. Allison. Nonreplication in genetic association studies of obesity and diabetes. J. Nutr., 133 (2003), pp. 3323-3326,

[45] P. Reimer, M. Baillie, E. Bard, A. Bayliss, J. Beck, P. Blackwell. IntCal09 and Marine09 radiocarbon age calibration curves, 0–50,000 years cal BP. Radiocarbon, 51 (2009), pp. 1111-1150,

[46] J.R. Reimink, J.H.F.L. Davies, A. Ielpi. Global zircon analysis records a gradual rise of continental crust throughout the Neoarchean. Earth Planet. Sci. Lett., 554 (2021), Article 116654,

[47] B. Schoene, D.J. Condon, L. Morgan. Precision and accuracy in geochronology. Elements, 9 (2013), pp. 19-24,

[48] T.M. Smith, J.E. Saylor, T.J. Lapen, K. Hatfield, K.E. Sundell. Identifying sources of non-unique detrital age distributions through integrated provenance analysis: an example from the paleozoic Central Colorado trough. Geosphere, 19 (2023), pp. 471-492,

[49] S.V. Stehman, D.J. Selkowitz. A spatially stratified, multi-stage cluster sampling design for assessing accuracy of the Alaska (USA) National Land Cover Database (NLCD). Int. J. Remote Sens., 31 (2010), pp. 1877-1896,

[50] R. Tamblyn, D. Hasterok, M. Hand, M. Gard. Mantle heating at ca. 2 Ga by continental insulation: evidence from granites and eclogites. Geology, 50 (2021), pp. 91-95,

[51] I.G. Usoskin, G.A. Kovaltsov. Occurrence of extreme solar particle events: assessment from historical proxy data. Astrophys. J., 757 (2012), p. 92,

[52] S. Vaughan, R.J. Bailey, D.G. Smith. Detecting cycles in stratigraphic data: spectral analysis in the presence of red noise. Paleoceanography, 26 (2011), p. PA2195,

[53] S. Vaughan, R.J. Bailey, D.G. Smith. Cyclostratigraphy: data filtering as a source of spurious spectral peaks. Geol. Soc. London Spec. Publ, 404 (2014), pp. 151-156,

[54] P. Vermeesch. How many grains are needed for a provenance study?. Earth Planet Sci. Lett., 224 (2004), pp. 441-451,

[55] P. Vermeesch. On the visualisation of detrital age distributions. Chem. Geol., 312–313 (2012), pp. 190-194,

[56] P. Vermeesch. IsoplotR: a free and open toolbox for geochronology. Geosci. Front., 9 (2018), pp. 1479-1493,

[57] A.M. Wahbi, M.D. Blum, C.N. Doerger. Early cretaceous continental-scale sediment routing, the McMurray formation, Western Canada Sedimentary Basin, Alberta, Canada. GSA Bull., 135 (2023), pp. 2088-2106,

[58] J.M. Waters, D. Craw. Goodbye Gondwana? New Zealand biogeography, geology, and the problem of circularity. Syst. Biol., 55 (2006), pp. 351-356,

[59] S.G. West, A.B. Taylor, W. Wu. Chapter 13: model fit and model selection in structural equation modeling. R.H. Hoyle (Ed.), Handbook of Structural Equation Modeling, Guilford Press, New York (2012), pp. 209-231

[60] Wilson, C., 2022. The replication crisis has spread through science – can it be fixed? New Scientist: Humans, April 6, 2022.

[61] H. Wu, Q. Fang, L.A. Hinnov, S. Zhang, T. Yang, M. Shi, H. Li. Astronomical time scale for the Paleozoic Era. Earth-Sci. Rev., 244 (2023), Article 104510,

[62] J. Zachos, M. Pagani, L. Sloan, E. Thomas, K. Billups. Trends, rhythms, and aberrations in global climate 65 Ma to present. Science, 292 (2001), pp. 686-693,

[63] A.L. Zerkle. Biogeodynamics: bridging the gap between surface and deep Earth processes. Philos. Tran. Royal Soc. A, 376 (2018), Article 20170401,