PEPFL: A framework for a practical and efficient privacy-preserving federated learning

Abstract

As an emerging joint learning model, federated learning is a promising way to combine model parameters of different users for training and inference without collecting users’ original data. However, a practical and efficient solution has not been established in previous work due to the absence of efficient matrix computation and cryptography schemes in the privacy-preserving federated learning model, especially in partially homomorphic cryptosystems. In this paper, we propose a Practical and Efficient Privacy-preserving Federated Learning (PEPFL) framework. First, we present a lifted distributed ElGamal cryptosystem for federated learning, which can solve the multi-key problem in federated learning. Secondly, we develop a Practical Partially Single Instruction Multiple Data (PSIMD) parallelism scheme that can encode a plaintext matrix into single plaintext for encryption, improving the encryption efficiency and reducing the communication cost in partially homomorphic cryptosystem. In addition, based on the Convolutional Neural Network (CNN) and the designed cryptosystem, a novel privacy-preserving federated learning framework is designed by using Momentum Gradient Descent (MGD). Finally, we evaluate the security and performance of PEPFL. The experiment results demonstrate that the scheme is practicable, effective, and secure with low communication and computation costs.

Keywords

Federated learning / Partially single instruction multiple data / Momentum gradient descent / ElGamal / Multi-key / Homomorphic encryption

References

[1] S. Sharma, K. Chen, Image disguising for privacy-preserving deep learning, in: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS 2018), ACM, 2018, pp. 2291-2293.

[2] Y. Chen, Y. Ping, Z. Zhang, B. Wang, S. He, Privacy-preserving image multi-classification deep learning model in robot system of industrial iot, Neural Comput, Appl 33 (10) (2021) 4677-4694.

[3] G. Kaissis, M.R. Makowski, D. Rueckert, R. Braren, Secure, privacy-preserving and federated machine learning in medical imaging, Nat. Mach. Intell. 2 (6) (2020) 305-311.

[4] Y. Wang, H. Tan, Y. Wu, J. Peng, Hybrid electric vehicle energy management with computer vision and deep reinforcement learning, IEEE Trans. Ind. Inf. 17 (6) (2021) 3857-3868.

[5] Y. Qu, L. Gao, T.H. Luan, Y. Xiang, S. Yu, B. Li, G. Zheng, Decentralized privacy using blockchain-enabled federated learning in fog computing, IEEE Internet Things J. 7 (6) (2020) 5171-5183.

[6] S. Li, P. Zheng, L. Zheng, An ar-assisted deep learning-based approach for automatic inspection of aviation connectors, IEEE Trans. Ind. Inf. 17 (3) (2021) 1721-1731.

[7] W. Tang, B. Li, M. Barni, J. Li, J. Huang, An automatic cost learning framework for image steganography using deep reinforcement learning, IEEE Trans. Inf. Forensics Secur. 16 (2021) 952-967.

[8] J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh,D. Bacon, Federated Learning: Strategies for Improving Communication Efficiency, arXiv preprint arXiv: 1610.05492.

[9] B. Hitaj, G. Ateniese, F. Pérez-Cruz, Deep models under the GAN: information leakage from collaborative deep learning,in:Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS 2017), ACM, 2017, pp. 603-618.

[10] B. McMahan, E. Moore, D. Ramage, S. Hampson, B.A. y Arcas, Communication-efficient learning of deep networks from decentralized data, in: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS 2017), vol. 54, PMLR, 2017, pp. 1273-1282.

[11] L.T. Phong, Y. Aono, T. Hayashi, L. Wang, S. Moriai, Privacy-preserving deep learning via additively homomorphic encryption, IEEE Trans. Inf. Forensics Secur. 13 (5) (2018) 1333-1345.

[12] G. Xu, H. Li, Y. Zhang, S. Xu, J. Ning, R.H. Deng, Privacy-preserving federated deep learning with irregular users, IEEE Trans. Dependable Secure Comput. 19 (2) (2022) 1364-1381.

[13] G. Xu, H. Li, S. Liu, K. Yang, X. Lin, Verifynet: secure and verifiable federated learning, IEEE Trans. Inf. Forensics Secur. 15 (2020) 911-926.

[14] T. Li, J. Li, X. Chen, Z. Liu, W. Lou, Y.T. Hou, Npmml: a framework for non-interactive privacy-preserving multi-party machine learning, IEEE Trans. Dependable Secure Comput. (99) (2020) 1-14.

[15] P. Xie, B. Wu, G. Sun, BAYHENN: combining bayesian deep learning and homomorphic encryption for secure DNN inference,in:Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, ijcai. org, 2019, pp. 4831-4837.

[16] J. Liu, M. Juuti, Y. Lu, N. Asokan, Oblivious neural network predictions via minionn transformations, in: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS 2017), ACM, 2017, pp. 619-631.

[17] C. Juvekar, V. Vaikuntanathan, A. Chandrakasan, GAZELLE: a low latency framework for secure neural network inference,in:Proceedings of 27th USENIX Security Symposium (USENIX Security 2018), USENIX Association, 2018, pp. 1651-1669.

[18] E. Ghadimi, H.R. Feyzmahdavian, M. Johansson, Global convergence of the heavy-ball method for convex optimization, in: Proceedings of the 14th European Control Conference (ECC 2015), IEEE, 2015, pp. 310-315.

[19] J. Wang, V. Tantia, N. Ballas, M.G. Rabbat, Slowmo: improving communication-efficient distributed SGD with slow momentum,in:Proceedings of the 8th International Conference on Learning Representations (ICLR 2020), OpenReview.net, 2020, pp. 1-25.

[20] W. Liu, L. Chen, Y. Chen, W. Zhang, Accelerating federated learning via momentum gradient descent, IEEE Trans. Parallel Distr. Syst. 31 (8) (2020) 1754-1766.

[21] L. Zhao, Q. Wang, Q. Zou, Y. Zhang, Y. Chen, Privacy-preserving collaborative deep learning with unreliable participants, IEEE Trans. Inf. Forensics Secur. 15 (2020) 1486-1500.

[22] L. Cui, Y. Qu, G. Xie, D. Zeng, R. Li, S. Shen, S. Yu, Security and privacy-enhanced federated learning for anomaly detection in iot infrastructures, IEEE Trans. Ind. Inf. 18 (5) (2022) 3492-3500.

[23] Y. Chen, B. Wang, Z. Zhang, Pdlhr: privacy-preserving deep learning model with homomorphic re-encryption in robot system, IEEE Syst. J. (2021) 1-12.

[24] S. Truex, N. Baracaldo, A. Anwar, T. Steinke, H. Ludwig, R. Zhang, Y. Zhou, A hybrid approach to privacy-preserving federated learning, in: Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security (AISec@CCS 2019), ACM, 2019, pp. 1-11.

[25] N. Agrawal, A.S. Shamsabadi, M.J. Kusner, A. Gascón, QUOTIENT: two-party secure neural network training and prediction,in:Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (CCS 2019), 2019, pp. 1231-1247.

[26] R. Xu, N. Baracaldo, Y. Zhou, A. Anwar, H. Ludwig, Hybridalpha: an efficient approach for privacy-preserving federated learning,in:Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security (AISec@CCS 2019), ACM, 2019, pp. 13-23.

[27] G. Xu, H. Li, Y. Zhang, S. Xu, J. Ning, R.H. Deng, Privacy-preserving federated deep learning with irregular users, IEEE Trans. Dependable Secure Comput. 19 (2) (2022) 1364-1381.

[28] X. Liu, R.H. Deng, K.R. Choo, J. Weng, An efficient privacy-preserving outsourced calculation toolkit with multiple keys, IEEE Trans. Inf. Forensics Secur. 11 (11)(2016) 2401-2414.

[29] C. Li, W. Ma, Comments on “an efficient privacy-preserving outsourced calculation toolkit with multiple keys”, IEEE Trans. Inf. Forensics Secur. 13 (10) (2018) 2668-2669.

[30] P. Li, J. Li, Z. Huang, T. Li, C. Gao, S. Yiu, K. Chen, Multi-key privacy-preserving deep learning in cloud computing, Future Generat. Comput. Syst. 74 (2017) 76-85.

[31] X. Ma, J. Ma, H. Li, Q. Jiang, S. Gao, PDLM: privacy-preserving deep learning model on cloud with multiple keys, IEEE Trans. Serv. Comput. 14 (4) (2021) 1251-1263.

[32] H. Chen, W. Dai, M. Kim, Y. Song, Efficient multi-key homomorphic encryption with packed ciphertexts with application to oblivious neural network inference, in: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (CCS 2019), ACM, 2019, pp. 395-412.

[33] S. Zhang, A. Choromanska, Y. LeCun,Deep learning with elastic averaging SGD, in:Proceedings of Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, 2015, pp. 685-693.

[34] A. Fu, X. Zhang, N. Xiong, Y. Gao, H. Wang, J. Zhang, Vfl, A verifiable federated learning with privacy-preserving for big data in industrial iot, IEEE Trans. Ind. Inf. 18 (5) (2022) 3316-3326.

[35] A.C. Yao, Protocols for secure computations (extended abstract),in:Proceedings of 23rd Annual Symposium on Foundations of Computer Science, IEEE Computer Society, 1982, pp. 160-164.

[36] T. ElGamal, A public key cryptosystem and a signature scheme based on discrete logarithms, in:Proceedings of CRYPTO ’84, vol. 196, Springer, 1984, pp. 10-18.

[37] J. Pollard, Monte Carlo method for index computation (mod p), Math. Comput. 32 (143) (1978) 918-924.

[38] R. Gennaro, S. Jarecki, H. Krawczyk, T. Rabin, Secure distributed key generation for discrete-log based cryptosystems, J. Cryptol. 20 (1) (2007) 51-83.

[39] X. Yi, F. Rao, E. Bertino, A. Bouguettaya, Privacy-preserving association rule mining in cloud computing, in: Proceedings of the 10th ACM Symposium on Information, Computer and Communications Security (ASIA CCS ’15), ACM, 2015, pp. 439-450.

[40] D. Chaum, C. Crépeau, I. Damgård, Multiparty unconditionally secure protocols (abstract),in:Proceedings of the Conference on the Theory and Applications of Cryptographic Techniques (CRYPTO ’87), vol. 293, Springer, 1987, p. 462.

[41] X. Yang, Y. Feng, W. Fang, J. Shao, X. Tang, S.-T. Xia, R. Lu,An Accuracy-Lossless Perturbation Method for Defending Privacy Attacks in Federated Learning, 2021 09843 arXiv:2002.

[42] N. Qian, On the momentum term in gradient descent learning algorithms, Neural Network. 12 (1) (1999) 145-151.

[43] T. Ogilvie, R. Player, J. Rowell, Improved privacy-preserving training using fixed-hessian minimisation, IACR Cryptol. ePrint Arch. 2020 (2020) 1514.

[44] D. Chai, L. Wang, K. Chen, Q. Yang, Secure federated matrix factorization, IEEE Intell. Syst. 36 (5) (2021) 11-20.

[45] D. Chaum, C. Crépeau, I. Damgård, Multiparty unconditionally secure protocols (extended abstract),in:Proceedings of the 20th Annual ACM Symposium on Theory of Computing, ACM, 1988, pp. 11-19.

[46] E. Hesamifard, H. Takabi,M. Ghasemi, Cryptodl: Deep Neural Networks over Encrypted Data, CoRR abs/1711.05189.

[47] D. Rathee, M. Rathee, N. Kumar, N. Chandran, D. Gupta, A. Rastogi, R. Sharma, Cryptflow2: practical 2-party secure inference,in:Proceedings of 2020 ACM SIGSAC Conference on Computer and Communications Security (CCS ’20), ACM, 2020, pp. 325-342.

[48] R. Canetti, M. Varia, Decisional diffie-hellman problem, in: Encyclopedia of Cryptography and Security, second ed., Springer, 2011, pp. 316-319.

[49] Y. Desmedt, Y. Frankel, Threshold cryptosystems, in: Proceedings of the 9th Annual International Cryptology Conference (CRYPTO ’89), vol. 435, Springer, 1989, pp. 307-315.

[50] R. Canetti, Universally composable security: a new paradigm for cryptographic protocols,in:Proceedings of the 42nd Annual Symposium on Foundations of Computer Science (FOCS 2001), IEEE Computer Society, 2001, pp. 136-145.

[51] P. Paillier, Public-key cryptosystems based on composite degree residuosity classes, in: Proceedings of International Conference on the Theory and Application of Cryptographic Techniques (EUROCRYPT ’99), vol. 1592, Springer, 1999, pp. 223-238.

[52] E. Bresson, D. Catalano, D. Pointcheval, A simple public-key cryptosystem with a double trapdoor decryption mechanism and its applications, in: Proceedings of the 9th International Conference on the Theory and Application of Cryptology and Information Security (ASIACRYPT 2003), vol. 2894, Springer, 2003, pp. 37-54.

[53] V. Smith, C. Chiang, M. Sanjabi, A.S. Talwalkar, Federated multi-task learning, in: Advances in Neural Information Processing Systems, vol. 30, 2017, pp. 4424-4434. NeurIPS 2017.