Secformer: Privacy-preserving atomic-level componentized transformer-like model with MPC✩

Chi Zhang; Tao Shen; Fenhua Bai; Kai Zeng; Xiaohui Zhang; Bin Cao

doi:10.1016/j.dcan.2025.04.009

›› 2026, Vol. 12 ›› Issue (1) :86 -100. DOI: 10.1016/j.dcan.2025.04.009

Regular Papers

research-article

Secformer: Privacy-preserving atomic-level componentized transformer-like model with MPC^✩

Author information +

^a Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China

^b The State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China

^* E-mail addresses: laxzhang@outlook.com (C. Zhang), shentao@kust.edu.cn (T. Shen), baifenhua@kust.edu.cn (F. Bai), zengkai@kust.edu.cn (K. Zeng), xh.zhang@stu.kust.edu.cn (X. Zhang), caobin@bupt.edu.cn (B. Cao).

Chi Zhang received his Ph.D. degree from the Faculty of Information and Automation, Kunming University of Science and Technology, Kunming, China, in 2025. He is currently a lecturer with the Yunnan Police College, Kunming, China. He has authored or coau-thored more than 10 papers in first-class SCI/EI. His current research interests include data sharing, cryptography, privacy-preserved computing and the IoT.

Tao Shen received the Ph.D. degree in electrical engineering from the Illinois Institute of Technology, Chicago, IL, USA, in 2013. He is currently the President with the Yunnan Institute of Mechanical and Electrical Vocational Technology, Kunming, China. He has au-thored or coauthored more than 100 papers in first-class SCI/EI and other internationally famous journals and top international conferences in relevant research fields. His research interests include artificial intelligence, blockchain technology, and the Industrial Internet.

Fenhua Bai received the Ph.D. degree from the Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China, in 2023. She is currently a lecturer with Kunming University of Science and Technology, Kunming, China. Her research interests include blockchains, edge computing, network security and games application.

Kai Zeng serves as an Associate Professor at Kunming University of Science and Technology. His research encompasses distributed computing architectures, granular com-puting methods, and deep learning applications.

Xiaohui Zhang received the B.Eng. (2017) and M.Eng. (2022) degrees from the faculty of information engineering and automation, Kunming University of Science and Technology, China, where he is currently pursuing a Ph.D. degree in Computer Science and Technology. His research focuses on distributed systems and privacy-preserving computation.

Bin Cao received the Ph.D. degree (Hons.) in communication and information systems from the National Key Laboratory of Science and Technology on Communications, University of Electronic Science and Technology of China, Chengdu, China, in 2014. He is currently a Full Professor with the State Key Laboratory of Network and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China. He was a Research Fellow with the National University of Singapore, Singapore, from July 2015 to July 2016. His research interests include blockchain systems, the Internet of Things, and mobile edge computing. He serves/served as an Associate Editor for IEEE Transactions on Mobile Computing and Digital Communications and Networks, and a Lead Guest Editor for IEEE Internet of Things Journal, IEEE Communications Magazine, IEEE Network, and IEEE Wireless Communications.

Show less

History +

PDF

Abstract

The global surge in Artificial Intelligence (AI) has been triggered by the impressive performance of deep-learning models based on the Transformer architecture. However, the eﬃcacy of such models is increasingly dependent on the volume and quality of data. Data are often distributed across institutions and companies, making cross-organizational data transfer vulnerable to privacy breaches and subject to privacy laws and trade secret regulations. These privacy and security concerns continue to pose major challenges to collaborative training and inference in multi-source data environments. These challenges are particularly significant for Transformer models, where the complex internal encryption computations drastically reduce computational eﬃciency, ultimately threatening the model’s practical applicability. We hence introduce Secformer, an innovative architecture specifically designed to protect the privacy of Transformer-like models. Secformer separates the encoder and decoder modules, enabling the decomposition of computation flows in Transformer-like models and their eﬃcient mapping to Multi-Party Computation (MPC) protocols. This design eﬀectively addresses privacy leakage issues during the collaborative computation process of Transformer models. To prevent performance degradation caused by encrypted attention modules, we propose a modular design strategy that optimizes high-level components by reconstructing low-level operators. We further analyze the security of Secformer’s core components, presenting security definitions and formal proofs. We construct a library of fundamental operators and core modules using atomic-level component designs as the basic building blocks for encoders and decoders. Moreover, these components can serve as foundational operators for other Transformer-like models. Extensive experimental evaluations demonstrate Secformer’s excellent performance while preserving privacy and oﬀering universal adaptability for Transformer-like models.

Keywords

Privacy-preserving computation / Deep learning / Multi-party computation / Data sharing

Cite this article

Download citation ▾

Chi Zhang, Tao Shen, Fenhua Bai, Kai Zeng, Xiaohui Zhang, Bin Cao. Secformer: Privacy-preserving atomic-level componentized transformer-like model with MPC^✩. , 2026, 12(1): 86-100 DOI:10.1016/j.dcan.2025.04.009

登录浏览全文

4963

注册一个新账户忘记密码

CRediT authorship contribution statement

Chi Zhang: Writing-original draft, Methodology, Formal analysis, Conceptualization. Tao Shen: Writing-review & editing, Methodology, Conceptualization. Fenhua Bai: Writing-review & editing, Validation. Kai Zeng: Writing-review & editing. Xiaohui Zhang: Writing-review & editing, Validation. Bin Cao: Writing-review & editing.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Bin Cao is an associate editor for Digital Communications and Networks and was not involved in the editorial review or the decision to publish this article. All authors declare that there are no competing interests. If there are other authors, they declare that they have no known competing fi-nancial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant 62471205; and in part by the Yunnan Fundamental Research Projects under Grant 202301AV070003; and in part by the Major Science and Technology Projects in Yunnan Province under Grant 202302AG050009.

References

Publishing order | Descend order by publishing year | Descend order by cited within

[1]	G.E. Hinton, S. Osindero, Y.-W. Teh, A fast learning algorithm for deep belief nets, Neural Comput. 18 (7) (2006) 1527-1554.

[2]	A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Proceedings of the 2017 Advances in Neural Information Processing Systems 30, MIT Press, 2017, pp. 5998-6008.

[3]	Uzma F. Al-Obeidat A. Tubaishat B. Shah Z. Halim, Gene encoder: a feature selec-tion technique through unsupervised deep learning-based clustering for large gene expression data, Neural Comput. Appl. 34 (11) (2022) 8309-8331.

[4]	S. Ullah, Z. Halim, Imagined character recognition through eeg signals using deep convolutional neural network, Med. Biol. Eng. Comput. 59 (5) (2021) 1167-1183.

[5]	E. Elahi, Z. Halim, Graph attention-based collaborative filtering for user-specific rec-ommender system using knowledge graph and deep neural networks, Knowl. Inf. Syst. 64 (9) (2022) 2457-2480.

[6]	J.D.M.-W. Chang, L.K. Toutanova, Bert: pre-training of deep bidirectional transform-ers for language understanding,in:Proceedings of the 2019 of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, 2019, pp. 4171-4186.

[7]	T. Brown, B. Mann, N. Ryder, et al., Language models are few-shot learners,in: Proceedings of the 2020 Advances in Neural Information Processing Systems 33, MIT Press, 2020, pp. 1877-1901.

[8]	B. Cao, Z. Wang, L. Zhang, D. Feng, M. Peng, L. Zhang, Z. Han, Blockchain systems, technologies, and applications: a methodology perspective, IEEE Commun. Surv. Tu-tor. 25 (1) (2023) 353-385.

[9]	W. Liu, B. Cao, M. Peng, Web 3 technologies: challenges and opportunities, IEEE Netw. 38 (3) (2024) 187-193.

[10]	W. Liu, B. Cao, M. Peng, B. Li, Distributed and parallel blockchain: towards a multi-chain system with enhanced security, IEEE Trans. Dependable Secure Comput. 22 (1) (2025) 723-739.

[11]	H. Yang, M. Ge, D. Xue, K. Xiang, H. Li, R. Lu, Gradient leakage attacks in feder-ated learning: research frontiers, taxonomy and future directions, IEEE Netw. 38 (2) (2024) 247-254.

[12]	X. Zhang, T. Shen, F. Bai, C. Zhang, Collusion-based poisoning attacks against blockchained federated learning, IEEE Netw. 37 (6) (2023) 50-57.

[13]	Y. Akimoto, K. Fukuchi, Y. Akimoto, J. Sakuma, Privformer: privacy-preserving transformer with mpc,in:Proceedings of the 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P), IEEE, 2023, pp. 392-410.

[14]	D. Li, R. Shao, H. Wang, H. Guo, E.P. Xing, H. Zhang, Mpcformer: fast, performant and private transformer inference with mpc, in:Proceedings of the 2023 Interna-tional Conference on Learning Representations (ICLR), 2022, pp. 1-16.

[15]	T. Chen, H. Bao, S. Huang, L. Dong, B. Jiao, D. Jiang, H. Zhou, J. Li, F. Wei, The-x: privacy-preserving transformer inference with homomorphic encryption, in: Pro-ceedings of the 2022 Findings of the Association for Computational Linguistics, ACL, 2022, pp. 3510-3520.

[16]	M. Zheng, Q. Lou, L. Jiang, Primer: fast private transformer inference on encrypted data,in:Proceedings of the 2023 60th ACM/IEEE Design Automation Conference (DAC), IEEE, 2023, pp. 1-6.

[17]	H. Li, Z. Cai, J. Wang, J. Tang, W. Ding, C.-T. Lin, Y. Shi, Fedtp: federated learning by transformer personalization, IEEE Trans. Neural Netw. Learn. Syst. 35 (10) (2024) 13426-13440.

[18]	J. Lang, L. Li, W. Chen, D. Zeng, Privacy protection in transformer-based neural network, in: Proceedings of the 2019 IEEE International Conference on Intelligence and Security Informatics (ISI), IEEE, 2019, pp. 182-184.

[19]	Q. Feng, D. He, Z. Liu, H. Wang, K.-K.R. Choo, Securenlp: a system for multi-party privacy-preserving natural language processing, IEEE Trans. Inf. Forensics Secur. 15 (1) (2020) 3709-3721.

[20]	J. Liu, X. Li, X. Liu, J. Tang, Y. Wang, Q. Tong, J. Ma, Privacy-preserving and verifi-able outsourcing linear inference computing framework, IEEE Trans. Serv. Comput. 16 (6) (2023) 4591-4604.

[21]	M. Keller, Mp-spdz: a versatile framework for multi-party computation, in: Pro-ceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, ACM, 2020, pp. 1575-1590.

[22]	V. Goyal, H. Li, R. Ostrovsky, A. Polychroniadou, Y. Song, Atlas: eﬃcient and scal-able mpc in the honest majority setting,in: Proceedings of the 2021 Advances in Cryptology-CRYPTO, Springer, 2021, pp. 244-274.

[23]	A.C. Yao, Protocols for secure computations, in: Proceedings of the 1982 23rd Annual Symposium on Foundations of Computer Science, IEEE, 1982, pp. 160-164.

[24]	O. Goldreich, S. Micali, A. Wigderson, How to Play Any Mental Game, or a Com-pleteness Theorem for Protocols with Honest Majority, ACM, New York, 2019.

[25]	D. Beaver, S. Micali, P. Rogaway, The round complexity of secure protocols, in: Proceedings of the 1990 Twenty-Second Annual ACM Symposium on Theory of Com-puting, ACM, 1990, pp. 503-513.

[26]	D. Demmler, T. Schneider, M. Zohner,Aby-a framework for eﬃcient mixed-protocol secure two-party computation, in:Proceedings of the 2015 Network and Distributed System Security (NDSS) Symposium, 2015, pp. 1-15.

[27]	E.M. Songhori, S.U. Hussain, A.-R. Sadeghi, T. Schneider, F. Koushanfar, Tinygarble: highly compressed and scalable sequential garbled circuits,in:Proceedings of the 2015 IEEE Symposium on Security and Privacy, IEEE, 2015, pp. 411-428.

[28]	M. Hastings, B. Hemenway, D. Noble, S. Zdancewic, Sok: general purpose compilers for secure multi-party computation,in:Proceedings of the 2019 IEEE Symposium on Security and Privacy (SP), IEEE, 2019, pp. 1220-1237.

[29]	W. Zheng, R.A. Popa, J.E. Gonzalez, I. Stoica, Helen: maliciously secure coopetitive learning for linear models,in:Proceedings of the 2019 IEEE Symposium on Security and Privacy, IEEE, 2019, pp. 724-738.

[30]	R. Cleve, Limits on the security of coin flips when half the processors are faulty, in: Proceedings of the 1986 18th Annual ACM Symposium on Theory of Computing, ACM, 1986, pp. 364-369.

[31]	M. Ben-Or, S. Goldwasser, A. Wigderson, Completeness Theorems for Non-cryptographic Fault-Tolerant Distributed Computation, ACM, New York, 2019.

[32]	Y. Li, B. Cao, M. Peng, L. Zhang, L. Zhang, D. Feng, J. Yu, Direct acyclic graph-based ledger for Internet of things: performance and security analysis, IEEE/ACM Trans. Netw. 28 (4) (2020) 1643-1656.

[33]	M. Cao, L. Zhang, B. Cao, Toward on-device federated learning: a direct acyclic graph-based blockchain approach, IEEE Trans. Neural Netw. Learn. Syst. 34 (4) (2023) 2028-2042.

[34]	H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, W. Zhang, Informer: beyond eﬃcient transformer for long sequence time-series forecasting,in: Proceedings of the 2021 AAAI Conference on Artificial Intelligence, 2021, pp. 11106-11115.

[35]	OpenAI,Gpt-4 technical report, https://cdn.openai.com/papers/gpt-4.pdf, 2023. (Accessed 15 March 2025).

[36]	Q. Zhang, C. Xin, H. Wu, Privacy-preserving deep learning based on multiparty se-cure computation: a survey, IEEE Internet Things J. 8 (13) (2021) 10412-10429.

[37]	B. Yan, K. Li, M. Xu, Y. Dong, Y. Zhang, Z. Ren, X. Cheng, On protecting the data privacy of large language models (llms) and llm agents: a literature review, High-Confid. Comput. (2025) 10030.

[38]	Y. Wang, E. Suh, W. Xiong, B. Knott, B. Lefaudeux, M. Annavaram, H.-H. Lee, Char-acterizing and improving mpc-based private inference for transformer-based models, in: Proceedings of the 2021 NeurIPS Workshop Privacy in Machine Learning, MIT Press, 2021, pp. 1-6.

[39]	Y. Xiong, Z. Zeng, R. Chakraborty, M. Tan, G. Fung, Y. Li, V. Singh, Nyströmformer: a Nyström-based algorithm for approximating self-attention,in:Proceedings of the 2021 AAAI Conference on Artificial Intelligence, AAAI, 2021, pp. 14138-14148.

[40]	A. Nguyen, K. Pham, D. Ngo, T. Ngo, L. Pham, An analysis of state-of-the-art activa-tion functions for supervised deep neural network, in: Proceedings of the 2021 Inter-national Conference on System Science and Engineering, IEEE, 2021, pp. 215-220.

[41]	C. Zhao, S. Zhao, M. Zhao, Z. Chen, C.-Z. Gao, H. Li, Y.-a. Tan, Secure multi-party computation: theory, practice and applications, Inf. Sci. 476 (1) (2019) 357-372.

[42]	A. Shamir, How to share a secret, Commun. ACM 22 (11) (1979) 612-613.

[43]	J. Furukawa, Y. Lindell, A. Nof, O. Weinstein, High-throughput secure three-party computation for malicious adversaries and an honest majority, in: Proceedings of the 2017 Annual International Conference on the Theory and Applications of Cryp-tographic Techniques, Springer, 2017, pp. 225-255.