Current opinions on large cellular models

Keywords

large cellular models / large language models / scBERT / Geneformer / scGPT / scFoundation / GeneCompass / single‐cell transcriptomics

References

[1] Brown T, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020; 33: 1877- 901.

[2] Touvron H, Lavril T, Izacard G, Martinet X, Lachaux M‐A, Lacroix T, et al. LLaMA: open and efficient foundation language models. 2023. Preprint at arXiv.2302.13971

[3] Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, et al. Llama 2: open foundation and fine‐tuned chat models. 2023. Preprint at arXiv.2307.09288

[4] Srivastava A, Rastogi A, Rao A, Shoeb AAM, Abid A, Fisch A, et al. Beyond the imitation game: quantifying and extrapolating the capabilities of language models. Trans Mach Learn Res. 2023.

[5] Yang F, Wang W, Wang F, Fang Y, Tang D, Huang J, et al. scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data. Nat Mach Intell. 2022; 4 (10): 852- 66.

[6] Theodoris CV, Xiao L, Chopra A, Chaffin MD, Al Sayed ZR, Hill MC, et al. Transfer learning enables predictions in network biology. Nature. 2023; 618 (7965): 1- 9.

[7] Cui H, Wang C, Maan H, Pang K, Luo F, Duan N, et al. scGPT: toward building a foundation model for single-cell multi-omics using generative AI. Nat Methods. 2024.

[8] Hao M, Gong J, Zeng X, Liu C, Guo Y, Cheng X, et al. Large-scale foundation model on single-cell transcriptomics. Nat Methods. 2024.

[9] Yang, X, Liu, G, Feng, G, Bu, D, Wang, P, Jiang, J, et al. GeneCompass: deciphering universal gene regulatory mechanisms with knowledge‐informed cross‐species foundation model. Preprint at bioRxiv 2023.09.26.559542

[10] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. 2017. Preprint at arXiv.1706.03762

[11] Devlin J, Chang M‐W, Lee K, Toutanova K. BERT: pre‐training of deep bidirectional transformers for language understanding. 2019. Preprint at arXiv.1810.04805

[12] Radford A, Narasimhan K. Improving language understanding by generative pre‐training. Preprint. Work in progress at Open AI. 2018.

[13] Shen R, Liu L, Wu Z, Zhang Y, Yuan Z, Guo J, et al. Spatial-ID: a cell typing method for spatially resolved transcriptomics via transfer learning and spatial embedding. Nat Commun. 2022; 13 (1): 7640.

[14] Yuan Z, Li Y, Shi M, Yang F, Gao J, Yao J, et al. SOTIP is a versatile method for microenvironment modeling with spatial omics data. Nat Commun. 2022; 13 (1): 7330.

[15] Li W, Yang F, Wang F, Rong Y, Liu L, Wu B, et al. scPROTEIN: a versatile deep graph contrastive learning framework for single-cell proteomics embedding. Nat Methods. 2024; 21 (4): 623- 34.

[16] Wang F, Yang F, Huang L, Li W, Song J, Gasser RB, et al. Deep domain adversarial neural network for the deconvolution of cell type mixtures in tissue proteome profiling. Nat Mach Intell. 2023; 5 (11): 1236- 49.

[17] Program, C.S.‐C. B., Aevermann B, Assis P, Badajoz S, Bell SM, Bezzi E, et al. CZ CELL×GENE Discover: a single‐cell data platform for scalable exploration, analysis and modeling of aggregated data. Preprint at bioRxiv 2023.10.30.563174v1

[18] Gong J, Hao M, Cheng X, Zeng X, Liu C, Ma J, et al. xTrimoGene: an efficient and scalable representation learner for single-cell RNA-seq data. Adv Neural Inf Process Syst. 2024; 36: 1- 13.

[19] Roohani Y, Huang K, Leskovec J. Predicting transcriptional outcomes of novel multigene perturbations with GEARS. Nat Biotechnol. 2024; 42: 927- 35.

[20] Bian H, Chen Y, Dong X, Li C, Hao M, Chen S, et al. scMulan: a multitask generative pre‐trained language model for single‐cell analysis. In: Ma J, editor. Research in computational molecular biology. Cham: Springer Nature Switzerland; 2024. p. 479- 82.

[21] Christina VT. Perspectives on benchmarking foundation models for network biology. Quantitative Biology. 2024.

RIGHTS & PERMISSIONS

2024 The Author(s). Quantitative Biology published by John Wiley & Sons Australia, Ltd on behalf of Higher Education Press.