VMask: tunable label privacy protection for vertical federated learning via layer masking

Abstract

Though vertical federated learning (VFL) is generally considered to be privacy-preserving, recent studies have shown that VFL system is vulnerable to label inference attacks originating from various attack surfaces. Among these attacks, the model completion (MC) attack is currently the most powerful one. Existing defense methods against it either sacrifice model accuracy or incur impractical computational overhead. In this paper, we propose VMask, a novel label privacy protection framework designed to defend against MC attack from the perspective of layer masking. Our key insight is to disrupt the strong correlation between input data and intermediate outputs by applying the secret sharing (SS) technique to mask layer parameters in the attacker’s model. We devise a strategy for selecting critical layers to mask, reducing the overhead that would arise from naively applying SS to the entire model. Moreover, VMask is the first framework to offer a tunable privacy budget to defenders, allowing for flexible control over the levels of label privacy according to actual requirements. We built a VFL system, implemented VMask on it, and extensively evaluated it using five model architectures and 13 datasets with different modalities, comparing it to 12 other defense methods. The results demonstrate that VMask achieves the best privacy-utility trade-off, successfully thwarting the MC attack (reducing the label inference accuracy to a random guessing level) while preserving model performance (e.g., in Transformer-based model, the averaged drop of VFL model accuracy is only 0.09%). VMask’s runtime is up to 60,846 times faster than cryptography-based methods, and it only marginally exceeds that of standard VFL by 1.8 times in a large Transformer-based model, which is generally acceptable.

Graphical abstract

Keywords

vertical federated learning / label inference attack / privacy protection

References

[1] Yang Q, Liu Y, Chen T, Tong Y . Federated machine learning: concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 2019, 10( 2): 12

[2] Liu Y, Kang Y, Zou T, Pu Y, He Y, Ye X, Ouyang Y, Zhang Y Q, Yang Q . Vertical federated learning: concepts, advances, and challenges. IEEE Transactions on Knowledge and Data Engineering, 2024, 36( 7): 3615–3634

[3] Li Q, Thapa C, Ong L, Zheng Y, Ma H, Camtepe S A, Fu A, Gao Y. Vertical federated learning: taxonomies, threats, and prospects. 2023, arXiv preprint arXiv: 2302.01550

[4] Yang L, Chai D, Zhang J, Jin Y, Wang L, Liu H, Tian H, Xu Q, Chen K. A survey on vertical federated learning: from a layered perspective. 2023, arXiv preprint arXiv: 2304.01829

[5] Wei K, Li J, Ma C, Ding M, Wei S, Wu F, Chen G, Ranbaduge T. Vertical federated learning: challenges, methodologies and experiments. 2022, arXiv preprint arXiv: 2202.04309

[6] Khan A, ten Thij M, Wilbik A . Vertical federated learning: a structured literature review. Knowledge and Information Systems, 2025, 67( 4): 3205–3243

[7] Li O, Sun J, Yang X, Gao W, Zhang H, Xie J, Smith V, Wang C. Label leakage and protection in two-party split learning. In: Proceedings of the 10th International Conference on Learning Representations. 2022

[8] Sun J, Yang X, Yao Y, Wang C. Label leakage and protection from forward embedding in vertical federated learning. 2022, arXiv preprint arXiv: 2203.01451

[9] Fu C, Zhang X, Ji S, Chen J, Wu J, Guo S, Zhou J, Liu A X, Wang T. Label inference attacks against vertical federated learning. In: Proceedings of the 31st USENIX Security Symposium (USENIX Security 22). 2022, 1397−1414

[10] Zou T, Liu Y, Kang Y, Liu W, He Y, Yi Z, Yang Q, Zhang Y Q . Defending batch-level label inference and replacement attacks in vertical federated learning. IEEE Transactions on Big Data, 2024, 10( 6): 1016–1027

[11] Kang Y, Luo J, He Y, Zhang X, Fan L, Yang Q. A framework for evaluating privacy-utility trade-off in vertical federated learning. 2022, arXiv preprint arXiv: 2209.03885

[12] Ghazi B, Golowich N, Kumar R, Manurangsi P, Zhang C. Deep learning with label differential privacy. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 2078

[13] Zou T, Liu Y, Zhang Y Q. Mutual information regularization for vertical federated learning. 2023, arXiv preprint arXiv: 2301.01142

[14] Zheng F, Chen C, Yao B, Zheng X. Making split learning resilient to label leakage by potential energy loss. 2022, arXiv preprint arXiv: 2210.09617

[15] Zhang Y, Zhu H. Additively homomorphical encryption based deep neural network for asymmetrically collaborative machine learning. 2020, arXiv preprint arXiv: 2007.06849

[16] Wang Y, Lv Q, Zhang H, Zhao M, Sun Y, Ran L, Li T . Beyond model splitting: preventing label inference attacks in vertical federated learning with dispersed training. World Wide Web, 2023, 26( 5): 2691–2707

[17] Zhou J, Zheng L, Chen C, Wang Y, Zheng X, Wu B, Chen C, Wang L, Yin J . Toward scalable and privacy-preserving deep neural network via algorithmic-cryptographic co-design. ACM Transactions on Intelligent Systems and Technology, 2022, 13( 4): 53

[18] Cai S, Chai D, Yang L, Zhang J, Jin Y, Wang L, Guo K, Chen K. Secure forward aggregation for vertical federated neural networks. In: Proceedings of the 1st International Workshop on Trustworthy Federated Learning. 2022, 115−129

[19] Fu F, Xue H, Cheng Y, Tao Y, Cui B. BlindFL: vertical federated machine learning without peeking into your data. In: Proceedings of 2022 International Conference on Management of Data. 2022, 1316−1330

[20] Raghuraman S, Rindal P. Blazing fast PSI from improved OKVS and subfield VOLE. In: Proceedings of 2022 ACM SIGSAC Conference on Computer and Communications Security. 2022, 2505−2517

[21] Rindal P, Schoppmann P. VOLE-PSI: fast OPRF and circuit-PSI from vector-OLE. In: Proceedings of the 40th Annual International Conference on the Theory and Applications of Cryptographic Techniques on Advances in Cryptology. 2021, 901−930

[22] Kolesnikov V, Kumaresan R, Rosulek M, Trieu N. Efficient batched oblivious PRF with applications to private set intersection. In: Proceedings of 2016 ACM SIGSAC Conference on Computer and Communications Security. 2016, 818−829

[23] Berthelot D, Carlini N, Goodfellow I, Oliver A, Papernot N, Raffel C. MixMatch: a holistic approach to semi-supervised learning. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 454

[24] Beaver D. Efficient multiparty protocols using circuit randomization. In: Feigenbaum J, ed. Advances in Cryptology—CRYPTO’91. Berlin, Heidelberg: Springer, 1992, 420−432

[25] Ishai Y, Kilian J, Nissim K, Petrank E. Extending oblivious transfers efficiently. In: Proceedings of the 23rd Annual International Cryptology Conference on Advances in Cryptology. 2003, 145−161

[26] LeCun Y, Bottou L, Bengio Y, Haffner P . Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86( 11): 2278–2324

[27] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd International Conference on Learning Representations. 2015

[28] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 770−778

[29] LeCun Y, Cortes C, Burges C. MNIST handwritten digit database. ATT Labs., 2010

[30] Krizhevsky A, Nair V, Hinton G. The CIFAR-10 dataset. 2014

[31] Gao X, Zhang L. PCAT: functionality and data stealing from split learning by Pseudo-Client attack. In: Proceedings of the 32nd USENIX Security Symposium (USENIX Security 23). 2023, 295

[32] Pasquini D, Ateniese G, Bernaschi M. Unleashing the tiger: inference attacks on split learning. In: Proceedings of 2021 ACM SIGSAC Conference on Computer and Communications Security, CCS ’21. 2021, 2113−2129

[33] Chen P, Yang J, Lin J, Lu Z, Duan Q, Chai H. A practical clean-label backdoor attack with limited information in vertical federated learning. In: Proceedings of 2023 IEEE International Conference on Data Mining (ICDM). 2023, 41−50

[34] China Academy of Information and Communications Technology. Research report on practical applications of federated learning. 2022

[35] Ye M, Shen W, Du B, Snezhko E, Kovalev V, Yuen P C . Vertical federated learning for effectiveness, security, applicability: a survey. ACM Computing Surveys, 2025, 57( 9): 223

[36] Naseri M, Han Y, De Cristofaro E. BadVFL: backdoor attacks in vertical federated learning. In: Proceedings of 2024 IEEE Symposium on Security and Privacy (SP). 2024, 2013−2028

[37] Pang Q, Yuan Y, Wang S, Zheng W. ADI: adversarial dominating inputs in vertical federated learning systems. In: Proceedings of 2023 IEEE Symposium on Security and Privacy (SP). 2023, 1875−1892

[38] He Y, Shen Z, Hua J, Dong Q, Niu J, Tong W, Huang X, Li C, Zhong S . Backdoor attack against split neural network-based vertical federated learning. IEEE Transactions on Information Forensics and Security, 2024, 19: 748–763

[39] European Union. Regulation (EU) 2016/679 of the European parliament and of the council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC. Luxembourg: European Union, 2016

[40] BrownBatman T. Calculate the output size in convolution layer. 2018

[41] Sanh V, Debut L, Chaumond J, Wolf T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. 2019, arXiv preprint arXiv: 1910.01108

[42] Xiao H, Rasul K, Vollgraf R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. 2017, arXiv preprint arXiv: 1708.07747

[43] Netzer Y, Wang T, Coates A, Bissacco A, Wu B, Ng A Y. Reading digits in natural images with unsupervised feature learning. In: Proceedings of NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011

[44] Darlow L N, Crowley E J, Antoniou A, Storkey A J. CINIC-10 is not ImageNet or CIFAR-10. 2018, arXiv preprint arXiv: 1810.03505

[45] Le Y, Yang X. Tiny ImageNet visual recognition challenge. CS 231N, 2015

[46] CriteoLabs . The Criteo dataset. 2014

[47] Voorhees E M, Tice D M. The TREC-8 question answering track evaluation. In: Proceedings of the 8th Text REtrieval Conference. 1999

[48] Zhang X, Zhao J, LeCun Y. Character-level convolutional networks for text classification. In: Proceedings of the 29th International Conference on Neural Information Processing Systems. 2015, 649−657

[49] Maas A, Daly R E, Pham P T, Huang D, Ng A Y, Potts C. Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 2011, 142−150

[50] Gu H, Luo J, Kang Y, Fan L, Yang Q. FedPass: Privacy-preserving vertical federated deep learning with adaptive obfuscation. In: Proceedings of the 32nd International Joint Conference on Artificial Intelligence, IJCAI-23. 2023, 418

[51] Arazzi M, Nicolazzo S, Nocera A. KDk: a defense mechanism against label inference attacks in vertical federated learning. 2024, arXiv preprint arXiv: 2404.12369

[52] Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Köpf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S. PyTorch: an imperative style, high-performance deep learning library. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 721

[53] Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, Davison J, Shleifer S, von Platen P, Ma C, Jernite Y, Plu J, Xu C, Le Scao T, Gugger S, Drame M, Lhoest Q, Rush A. Transformers: state-of-the-art natural language processing. In: Proceedings of 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2020, 38−45

[54] Python Software Foundation. Python socket library. 2025

[55] Data61 . Python-Paillier library. 2013

[56] van der Maaten L, Hinton G . Visualizing data using t-SNE. Journal of Machine Learning Research, 2008, 9( 86): 2579–2605

[57] Zhu H, Xu J, Liu S, Jin Y . Federated learning on non-IID data: a survey. Neurocomputing, 2021, 465: 371–390

[58] Li Q, Diao Y, Chen Q, He B. Federated learning on non-IID data silos: an experimental study. In: Proceedings of the 38th IEEE International Conference on Data Engineering (ICDE). 2022, 965−978

[59] Wan X, Sun J, Wang S, Chen L, Zheng Z, Wu F, Chen G. PSLF: defending against label leakage in split learning. In: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 2023, 2492−2501

[60] He Y, Niu M, Hua J, Mao Y, Huang X, Li C, Zhong S. LabObf: a label protection scheme for vertical federated learning through label obfuscation. 2024, arXiv preprint arXiv: 2405.17042

[61] Duan L, Sun J, Chen Y, Gorlatova M. PrivaScissors: enhance the privacy of collaborative inference through the lens of mutual information. 2023, arXiv preprint arXiv: 2306.07973

[62] Qiu X, Pan H, Zhao W, Ma C, Gusmao P P B, Lane N D. vFedSec: efficient secure aggregation for vertical federated learning via secure layer. 2023, arXiv preprint arXiv: 2305.16794

[63] Dwork C. Differential privacy. In: Proceedings of the 33rd International Colloquium on Automata, Languages and Programming, ICALP. 2006, 1−12

[64] Fan L, Ng K W, Chan C S, Yang Q . DeepIPR: deep neural network ownership verification with passports. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44( 10): 6122–6139

[65] Székely G J, Rizzo M L, Bakirov N K . Measuring and testing dependence by correlation of distances. Annals of Statistics, 2007, 35( 6): 2769–2794

[66] Mohassel P, Rindal P. ABY³: a mixed protocol framework for machine learning. In: Proceedings of 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS ’18. 2018, 35−52

[67] Wagh S, Tople S, Benhamouda F, Kushilevitz E, Mittal P, Rabin T . Falcon: honest-majority maliciously secure framework for private deep learning. Proceedings on Privacy Enhancing Technologies, 2021, 2021( 1): 188–208

[68] Abbaszadeh K, Pappas C, Katz J, Papadopoulos D. Zero-knowledge proofs of training for deep neural networks. In: Proceedings of 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24. 2024, 4316−4330

RIGHTS & PERMISSIONS

Higher Education Press