A case study on the detailed reproducibility of a Human Cell Atlas project

Abstract

Background: Reproducibility is a defining feature of a scientific discovery. Reproducibility can be at different levels for different types of study. The purpose of the Human Cell Atlas (HCA) project is to build maps of molecular signatures of all human cell types and states to serve as references for future discoveries. Constructing such a complex reference atlas must involve the assembly and aggregation of data from multiple labs, probably generated with different technologies. It has much higher requirements on reproducibility than individual research projects. To add another layer of complexity, the bioinformatics procedures involved for single-cell data have high flexibility and diversity. There are many factors in the processing and analysis of single-cell RNA-seq data that can shape the final results in different ways.

Methods: To study what levels of reproducibility can be reached in current practices, we conducted a detailed reproduction study for a well-documented recent publication on the atlas of human blood dendritic cells as an example to break down the bioinformatics steps and factors that are crucial for the reproducibility at different levels.

Results: We found that the major scientific discovery can be well reproduced after some efforts, but there are also some differences in some details that may cause uncertainty in the future reference. This study provides a detailed case observation on the on-going discussions of the type of standards the HCA community should take when releasing data and publications to guarantee the reproducibility and reliability of the future atlas.

Conclusion: Current practices of releasing data and publications may not be adequate to guarantee the reproducibility of HCA. We propose building more stringent guidelines and standards on the information that needs to be provided along with publications for projects that evolved in the HCA program.

Graphical abstract

Keywords

Human Cell Atlas / reproducibility / single cell / bioinformatics

References

[1] Watson, J. D. (1990) The human genome project: past, present, and future. Science, 248, 44–49

[2] Collins, F. S., Morgan, M. and Patrinos, A. (2003) The Human Genome Project: lessons from large-scale biology. Science, 300, 286–290

[3] Gibbs, R. A., Belmont, J. W., Hardenbol, P., Willis, T. D., Yu, F., Zhang, H., Zeng, C., Matsuda, I., Fukushima, Y., Macer, D. R., (2003) The International HapMap Project. Nature, 426, 789–796

[4] Feingold, E. A., Good, P. J., Guyer, M. S., Kamholz, S., Liefer, L., Wetterstrand, K., Collins, F. S., Gingeras, T. R., Kampa, D., Sekinger, E. A. (2004) The ENCODE (ENCyclopedia of DNA Elements) project. Science, 306, 636–640

[5] Haines, J. L., Hauser, M. A., Schmidt, S., Scott, W. K., Olson, L. M., Gallins, P., Spencer, K. L., Kwan, S. Y., Noureddine, M., Gilbert, J. R., (2005) Complement factor H variant increases the risk of age-related macular degeneration. Science, 308, 419–421

[6] The 1000 Genomes Project Consortium. (2010) A map of human genome variation from population-scale sequencing. Nature, 467, 1061–1073

[7] The 1000 Genomes Project Consortium. (2012) An integrated map of genetic variation from 1092 human genomes. Nature, 491, 56–65

[8] Kellis, M., Wold, B., Snyder, M. P., Bernstein, B. E., Kundaje, A., Marinov, G. K., Ward, L. D., Birney, E., Crawford, G. E., Dekker, J., (2014) Defining functional DNA elements in the human genome. Proc. Natl. Acad. Sci. USA, 111, 6131–6138

[9] The Human Cell Atlas Meeting Participants. (2017) The Human Cell Atlas. eLife, 6, e27041

[10] Rozenblatt-Rosen, O., Stubbington, M. J. T., Regev, A. and Teichmann, S. A. (2017) The Human Cell Atlas: from vision to reality. Nature, 550, 451–453

[11] Svensson, V., Vento-Tormo, R. and Teichmann, S. A. (2018) Exponential scaling of single-cell RNA-seq in the past decade. Nat. Protoc., 13, 599–604

[12] Cusanovich, D. A., Daza, R., Adey, A., Pliner, H. A., Christiansen, L., Gunderson, K. L., Steemers, F. J., Trapnell, C. and Shendure, J. (2015) Multiplex single cell profiling of chromatin accessibility by combinatorial cellular indexing. Science, 348, 910–914

[13] Nagano, T., Lubling, Y., Stevens, T. J., Schoenfelder, S., Yaffe, E., Dean, W., Laue, E. D., Tanay, A. and Fraser, P. (2013) Single-cell Hi-C reveals cell-to-cell variability in chromosome structure. Nature, 502, 59–64

[14] Zenobi, R. (2013) Single-cell metabolomics: analytical and biological perspectives. Science, 342, 1243259

[15] Crosetto, N., Bienko, M. and van Oudenaarden, A. (2015) Spatially resolved transcriptomics and beyond. Nat. Rev. Genet., 16, 57–66

[16] Lein, E., Borm, L. E. and Linnarsson, S. (2017) The promise of spatial transcriptomics for neuroscience in the era of molecular cell typing. Science, 358, 64–69

[17] Zhong, S., Zhang, S., Fan, X., Wu, Q., Yan, L., Dong, J., Zhang, H., Li, L., Sun, L., Pan, N., (2018) A single-cell RNA-seq survey of the developmental landscape of the human prefrontal cortex. Nature, 555, 524–528

[18] Muraro, M. J., Dharmadhikari G., Grün, D., Groen, N., Dielen, T., Jansen, E., van Gurp, L., Engelse, M. A., Carlotti, F., de Koning, E. J. P. (2016) A single-cell transcriptome atlas of the human pancreas. Cell Syst., 3, 385–394 e3

[19] Darmanis, S., Sloan, S. A., Zhang, Y., Enge, M., Caneda, C., Shuer, L. M., Hayden Gephart, M. G., Barres, B. A. and Quake, S. R. (2015) A survey of human brain transcriptome diversity at the single cell level. Proc. Natl. Acad. Sci. USA, 112, 7285–7290

[20] Macosko, E. Z., Basu, A., Satija, R., Nemesh, J., Shekhar, K., Goldman, M., Tirosh, I., Bialas, A. R., Kamitaki, N., Martersteck, E. M., (2015) Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets. Cell, 161, 1202–1214

[21] Stubbington, M. J. T., Rozenblatt-Rosen, O., Regev, A. and Teichmann, S. A. (2017) Single-cell transcriptomics to explore the immune system in health and disease. Science, 358, 58–63

[22] Jaitin, D. A., Kenigsberg, E., Keren-Shaul, H., Elefant, N., Paul, F., Zaretsky, I., Mildner, A., Cohen, N., Jung, S., Tanay, A., (2014) Massively parallel single-cell RNA-seq for marker-free decomposition of tissues into cell types. Science, 343, 776–779

[23] Villani, A. C., Satija, R., Reynolds, G., Sarkizova, S., Shekhar, K., Fletcher, J., Griesbeck, M., Butler, A., Zheng, S., Lazo, S., (2017) Single-cell RNA-seq reveals new types of human blood dendritic cells, monocytes, and progenitors. Science, 356, eaah4573

[24] Data Coordination – Human Cell Atlas (https://www.humancellatlas.org/data-sharing)

[25] Butler, A., Hoffman, P., Smibert, P., Papalexi, E. and Satija, R. (2018) Integrating single-cell transcriptomic data across different conditions, technologies, and species. Nat. Biotechnol., 36, 411–420

[26] van der Maaten, L. and Hinton, G. (2008) Visualizing data using t-SNE. J. Mach. Learn. Res., 9, 2579–2605.

[27] Wattenberg, F. V. M. and Johnson, I. (2016) How to use t-SNE effectively. Distillhttp://doi.org/10.23915/distill.00002

[28] Yuansheng Zhou, T. O. S. (2018) Using global t-SNE to preserve inter-cluster data structure. bioRxiv, Doi: https://doi.org/10.1101/331611

[29] Kobak, D. and Berens, P. (2018) The art of using t-SNE for single-cell transcriptomics. bioRxiv, Doi: https://doi.org/10.1101/453449

[30] Baker, M. (2016) Is there a reproducibility crisis? Nature. 533, 452–454

[31] Berg, J. (2018) Progress on reproducibility. Science, 359, 9

[32] Stark, P. B. (2018) Before reproducibility must come preproducibility. Nature, 557, 613

SUPPLEMENTARY MATERIALS

The supplementary materials can be found online with this article at https://doi.org/10.1007/s40484-018-0164-3.

AUTHORIZATION STATEMENT

Figure 1A, Figure 1C, Figure 2A, Figure 3A, Figure 3C and Figure 4A in this paper are reprinted from Ref. [23]. The use of these figures is authorized with the license number 4502220162373.

ACKNOWLEDGEMENTS

We thank Nir Hacohen, Alexandra-Chloé Villani and Orit Rozenblatt-Rosen (authors of the original paper) for their helpful discussion. This work is supported by CZI Human Cell Atlas Pilot Project and the National Natural Science Foundation of China (Nos. 61673231 and 61721003).

COMPLIANCE WITH ETHICS GUIDELINES

The authors Kui Hua and Xuegong Zhang declare that they have no conflict of interests.ƒAll procedures performed in studies were in accordance with the ethical standards of the institution or practice at which the studies were conducted, and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

RIGHTS & PERMISSIONS

2018 Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature