Predicting drug-perturbed transcriptional responses using multi-conditional diffusion transformer

Abstract

Drug-perturbed transcriptomes are important for personalized medicine and drug discovery. Nevertheless, the existing high-throughput screening and sequencing techniques for drug-perturbed transcriptomes remain expensive and time-consuming. In this study, we propose a novel multi-condition diffusion transformer model, designated as perturbation diffusion transformer (PertDiT), which is tailored for conditionally generating the perturbed transcriptomes based on drug text information. PertDiT combines the potent transformer architecture with the text representation of pre-trained large language models and utilizes a novel perturbation and transcriptome fusion modules. We have designed two network structures, namely, CrossDiT and CatCrossDiT, applicable to drug discovery and personalized medicine scenarios, respectively. Through a comprehensive set of metrics and an effective data splitting strategy, our model outperforms existing methods, demonstrating a superior ability in post-perturbation transcriptome reconstruction and the prediction of perturbation-induced transcriptional changes. The rationality and effectiveness of the model structure have also been meticulously validated.

Keywords

diffusion model / perturbation / transcriptome

References

[1] Kornauth C , Pemovska T , Vladimer GI , Bayer G , Bergmann M , Eder S , et al. Functional precision medicine provides clinical benefit in advanced aggressive hematologic cancers and identifies exceptional responders. Cancer Discov. 2022; 12 (2): 372- 87.

[2] Snijder B , Vladimer GI , Krall N , Miura K , Schmolke AS , Kornauth C , et al. Image-based ex-vivo drug screening for patients with aggressive haematological malignancies:interim results from a single-arm, open-label, pilot study. Lancet Haematol. 2017; 4 (12): e595- 606.

[3] Zhu J , Wang J , Wang X , Gao M , Guo B , Gao M , et al. Prediction of drug efficacy from transcriptional profiles with deep learning. Nat Biotechnol. 2021; 39 (11): 1444- 52.

[4] Srivatsan SR , McFaline-Figueroa JL , Ramani V , Saunders L , Cao J , Packer J , et al. Massively multiplex chemical transcriptomics at single-cell resolution. Science. 2020; 367 (6473): 45- 51.

[5] Subramanian A , Narayan R , Corsello SM , Peck DD , Natoli TE , Lu X , et al. A next generation connectivity map:L1000 platform and the first 1, 000, 000 profiles. Cell. 2017; 171 (6): 1437- 52.e17.

[6] Lotfollahi M , Wolf FA , Theis FJ . scGen predicts single-cell perturbation responses. Nat Methods. 2019; 16 (8): 715- 21.

[7] Piran Z , Cohen N , Hoshen Y , Nitzan M . Disentanglement of single-cell data with biolord. Nat Biotechnol. 2024; 42 (11): 1- 6.

[8] Hetzel L , Boehm S , Kilbertus N , Günnemann S , Theis F . Predicting cellular responses to novel drug perturbations at a single-cell resolution. Adv Neural Inf Process Syst. 2022; 35: 26711- 22.

[9] Qi X , Zhao L , Tian C , Li Y , Chen Z-L , Huo P , et al. Predicting transcriptional responses to novel chemical perturbations using deep generative model for drug discovery. Nat Commun. 2024; 15 (1): 1- 19.

[10] Yang L , Zhang Z , Song Y , Hong S , Xu R , Zhao Y , et al. Diffusion models: a comprehensive survey of methods and applications. ACM Comput Surv. 2023; 56 (4): 1- 39.

[11] Ho J , Jain A , Abbeel P . Denoising diffusion probabilistic models. Adv Neural Inf Process Syst. 2020; 33: 6840- 51.

[12] Rombach R , Blattmann A , Lorenz D , Esser P , Ommer B . High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition; 2022.

[13] Peebles W , Xie S . Scalable diffusion models with transformers. Proceedings of the IEEE/CVF international conference on computer vision; 2023;

[14] Vaswani A , Shazeer N , Parmar N , Uszkoreit J , Jones L , Gomez AN , et al. Attention is all you need. Adv Neural Inf Process Syst 30 (NeurIPS 2017). Nat Biotechnol. 2012;

[15] Wan X , Xiao J , Tam SST , Cai M , Sugimura R , Wang Y , et al. Integrating spatial and single-cell transcriptomics data using deep generative models with SpatialScope. Nat Commun. 2023; 14 (1): 7848.

[16] Luo E , Hao M , Wei L , Zhang X . scDiffusion:conditional generation of high-quality single-cell data using diffusion model. Bioinformatics. 2024; 40 (9): btae518.

[17] Xu H , Woicik A , Poon H , Altman RB , Wang S . Multilingual translation for zero-shot biomedical classification using BioTranslator. Nat Commun. 2023; 14 (1): 738.

[18] Weininger D . SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. J Chem Inf Comput Sci. 1988; 28 (1): 31- 6.

[19] Edwards C , Lai T , Ros K , Honke G , Cho K , Ji H . Translation between molecules and natural language. 2022. Preprint at arXiv: 220411817.

[20] Yasunaga M , Leskovec J , Liang P . Linkbert:pretraining language models with document links. 2022. Preprint at arXiv: 220315827.

[21] Bento AP , Hersey A , Félix E , Landrum G , Gaulton A , Atkinson F , et al. An open source chemical structure curation pipeline using RDKit. J Cheminf. 2020; 12: 1- 16.

[22] Karras T , Aittala M , Aila T , Laine S . Elucidating the design space of diffusion-based generative models. Adv Neural Inf Process Syst. 2022; 35: 26565- 77.

[23] Song Y , Dhariwal P , Chen M , Sutskever I . Consistency models. 2023. Preprint at arXiv: 230301469.

[24] Dao T . Flashattention-2:faster attention with better parallelism and work partitioning. 2023. Preprint at arXiv: 230708691.

[25] Dao T , Gu A . Transformers are SSMs:generalized models and efficient algorithms through structured state space duality. 2024. Preprint at arXiv: 240521060.

[26] Wolf FA , Angerer P , Theis FJ . SCANPY:large-scale single-cell gene expression data analysis. Genome Biol. 2018; 19: 1- 5.

[27]

[28] Xiong R , Yang Y , He D , Zheng K , Zheng S , Xing C , et al. On layer normalization in the transformer architecture. Proceedings of the International conference on machine learning. PMLR; 2020.

RIGHTS & PERMISSIONS

The Author(s). Quantitative Biology published by John Wiley & Sons Australia, Ltd on behalf of Higher Education Press.