XMAM:X-raying models with a matrix to reveal backdoor attacks for federated learning^☆

Abstract

Federated Learning (FL), a burgeoning technology, has received increasing attention due to its privacy protection capability. However, the base algorithm FedAvg is vulnerable when it suffers from so-called backdoor attacks. Former researchers proposed several robust aggregation methods. Unfortunately, due to the hidden characteristic of backdoor attacks, many of these aggregation methods are unable to defend against backdoor attacks. What's more, the attackers recently have proposed some hiding methods that further improve backdoor attacks' stealthiness, making all the existing robust aggregation methods fail.
To tackle the threat of backdoor attacks, we propose a new aggregation method, X-raying Models with A Matrix (XMAM), to reveal the malicious local model updates submitted by the backdoor attackers. Since we observe that the output of the Softmax layer exhibits distinguishable patterns between malicious and benign updates, unlike the existing aggregation algorithms, we focus on the Softmax layer's output in which the backdoor attackers are difficult to hide their malicious behavior. Specifically, like medical X-ray examinations, we investigate the collected local model updates by using a matrix as an input to get their Softmax layer's outputs. Then, we preclude updates whose outputs are abnormal by clustering. Without any training dataset in the server, the extensive evaluations show that our XMAM can effectively distinguish malicious local model updates from benign ones. For instance, when other methods fail to defend against the backdoor attacks at no more than 20% malicious clients, our method can tolerate 45% malicious clients in the black-box mode and about 30% in Projected Gradient Descent (PGD) mode. Besides, under adaptive attacks, the results demonstrate that XMAM can still complete the global model training task even when there are 40% malicious clients. Finally, we analyze our method's screening complexity and compare the real screening time with other methods. The results show that XMAM is about 10-10000 times faster than the existing methods.

Keywords

Federated learning / Backdoor attacks / Aggregation methods

References

[1] J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh,D. Bacon, Federated Learning: Strategies for Improving Communication Efficiency, arXiv preprint arXiv: 1610.05492.

[2] B. McMahan, E. Moore, D. Ramage, S. Hampson, B. A. y Arcas, Communication-efficient learning of deep networks from decentralized data, in: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS, vol. 54, PMLR, 2017, pp. 1273-1282.

[3] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, V. Shmatikov, How to backdoor federated learning, in: Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, AISTATS, vol. 108, PMLR, 2020, pp. 2938-2948.

[4] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al., Advances and Open Problems in Federated Learning, arXiv preprint arXiv:1912.04977.

[5] T. Gu, B. Dolan-Gavitt,S. Garg, Badnets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain, arXiv preprint arXiv:1708.06733.

[6] H. Wang, K. Sreenivasan, S. Rajput, H. Vishwakarma, S. Agarwal, J.-y. Sohn, K. Lee, D. Papailiopoulos, Attack of the Tails: Yes, You Really Can Backdoor Federated Learning, arXiv preprint arXiv:2007.05084.

[7] P. Blanchard, E.M.E. Mhamdi, R. Guerraoui, J. Stainer, Machine learning with adversaries: byzantine tolerant gradient descent,in: Proceedings of Annual Conference on Neural Information Processing Systems, 2017, pp. 119-129.

[8] A.N. Bhagoji, S. Chakraborty, P. Mittal, S.B. Calo, Analyzing federated learning through an adversarial lens, in: Proceedings of the 36th International Conference on Machine Learning, ICML, vol. 97, PMLR, 2019, pp. 634-643.

[9] X. Cao, M. Fang, J. Liu, N.Z. Gong, Fltrust: Byzantine-Robust Federated Learning via Trust Bootstrapping, arXiv preprint arXiv, 2012, 13995.

[10] N. Rieke, J. Hancox, W. Li, F. Milletari, H.R. Roth, S. Albarqouni, S. Bakas, M. N. Galtier, B.A. Landman, K. Maier-Hein, et al., The future of digital health with federated learning, NPJ digit. med. 3 (1) (2020) 1-7.

[11] Z. Sun, P. Kairouz, A. T. Suresh,H. B. McMahan, Can You Really Backdoor Federated Learning?, arXiv preprint arXiv:1911.07963.

[12] L. Li, W. Xu, T. Chen, G.B. Giannakis, Q. Ling, RSA: byzantine-robust stochastic aggregation methods for distributed learning from heterogeneous datasets,in:Proceedings of the 33rd Conference on Artificial Intelligence, AAAI Press, 2019, pp. 1544-1551, https://doi.org/10.1609/aaai.v33i01.33011544.

[13] K. Pillutla, S. M. Kakade,Z. Harchaoui, Robust Aggregation for Federated Learning, arXiv preprint arXiv:1912.13445.

[14] M. Fang, X. Cao, J. Jia, N. Gong, Local model poisoning attacks to byzantine-robust federated learning, in: Proceedings of the 29th {USENIX} Security Symposium ({USENIX} Security 20), 2020, pp. 1605-1622.

[15] R.J. Campello, D. Moulavi, J. Sander, Density-based clustering based on hierarchical density estimates, in: Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, Springer, 2013, pp. 160-172.

[16] T.-M. H. Hsu, H. Qi,M. Brown, Measuring the Effects of Non-identical Data Distribution for Federated Visual Classification, arXiv preprint arXiv:1909.06335.

[17] A. Krizhevsky, G. Hinton, et al., Learning multiple layers of features from tiny images, Tech. rep. (2009).

[18] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Proc. IEEE 86 (11) (1998) 2278-2324.

[19] X. Huang, M. Alzantot,M. Srivastava, Neuroninspect: Detecting Backdoors in Neural Networks via Output Explanations, arXiv preprint arXiv:1911.07399.

[20] S. Kolouri, A. Saha, H. Pirsiavash, H. Hoffmann, Universal litmus patterns: revealing backdoor attacks in cnns,in:Proceedings of the 2020 Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2020, pp. 298-307, https://doi.org/10.1109/CVPR42600.2020.00038.

[21] S. Huang, W. Peng, Z. Jia, Z. Tu, One-pixel signature: characterizing cnn models for backdoor detection,in:Proceedings of the European Conference on Computer Vision (ECCV), Springer, 2020, pp. 326-341.

[22] B. Biggio, B. Nelson, P. Laskov, Poisoning attacks against support vector machines, in: Proceedings of the 29th International Conference on Machine Learning, ICML, Omnipress, 2012.

[23] X. Chen, C. Liu, B. Li, K. Lu,D. Song, Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning, arXiv preprint arXiv:1712.05526.

[24] M. Fang, G. Yang, N.Z. Gong, J. Liu,Poisoning attacks to graph-based recommender systems, in:Proceedings of the 34th Annual Computer Security Applications Conference, 2018, pp. 381-392.

[25] M. Jagielski, A. Oprea, B. Biggio, C. Liu, C. Nita-Rotaru, B. Li, Manipulating machine learning: poisoning attacks and countermeasures for regression learning,in:Proceedings of the 2018 IEEE Symposium on Security and Privacy (SP), IEEE, 2018, pp. 19-35.

[26] B. Li, Y. Wang, A. Singh, Y. Vorobeychik,Data poisoning attacks on factorization-based collaborative filtering, in:Annual Conference on Neural Information Processing Systems (NIPS), 2016, pp. 1885-1893.

[27] L. Muñoz-González, B. Biggio, A. Demontis, A. Paudice, V. Wongrassamee, E. C. Lupu, F. Roli,Towards poisoning of deep learning algorithms with back-gradient optimization, in:Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, 2017, pp. 27-38.

[28] B. Nelson, M. Barreno, F.J. Chi, A.D. Joseph, B.I. Rubinstein, U. Saini, C.A. Sutton, J. D. Tygar, K. Xia, Exploiting machine learning to subvert your spam filter, LEET 8 (2008) 1-9.

[29] B.I. Rubinstein, B. Nelson, L. Huang, A.D. Joseph, S.-h. Lau, S. Rao, N. Taft, J. D. Tygar, Antidote: understanding and defending against poisoning of anomaly detectors,in: Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement, 2009, pp. 1-14.

[30] A. Shafahi, W.R. Huang, M. Najibi, O. Suciu, C. Studer, T. Dumitras, T. Goldstein,Poison frogs! targeted clean-label poisoning attacks on neural networks, in:Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2018, pp. 6106-6116.

[31] O. Suciu, R. Marginean, Y. Kaya, H. Daume III, T. Dumitras, When does machine learning {FAIL}? generalized transferability for evasion and poisoning attacks, in:Proceedings of the 27th {USENIX} Security Symposium, 2018, pp. 1299-1316.

[32] B. Wang, N.Z. Gong,Attacking graph-based classification via manipulating the graph structure, in:Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, 2019, pp. 2023-2040.

[33] H. Xiao, B. Biggio, G. Brown, G. Fumera, C. Eckert, F. Roli, Is feature selection secure against training data poisoning?, in: Proceedings of the 32nd International Conference on Machine Learning, ICML, vol. 37 JMLR.org, 2015, pp. 1689-1698.

[34] G. Yang, N.Z. Gong, Y. Cai,Fake co-visitation injection attacks to recommender systems, in:Proceedings of NDSS, 2017.

[35] M. Fang, M. Sun, Q. Li, N.Z. Gong, J. Tian, J. Liu,Data poisoning attacks and defenses to crowdsourcing systems, in:Proceedings of the Web Conference 2021, 2021, pp. 969-980.

[36] C. Xie, K. Huang, P. Chen, B. Li, DBA: distributed backdoor attacks against federated learning,in:Proceedings of the 8th International Conference on Learning Representations (ICLR), OpenReview.net, 2020.

[37] G. Baruch, M. Baruch, Y. Goldberg, A little is enough: circumventing defenses for distributed learning,in: Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2019, pp. 8632-8642.

[38] C. Xie, O. Koyejo, I. Gupta, Fall of empires: breaking byzantine-tolerant SGD by inner product manipulation,in: Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI vol. 115, AUAI Press, 2019, pp. 261-270.

[39] L. He, S.P. Karimireddy, M. Jaggi, Byzantine-robust Learning on Heterogeneous Datasets via Resampling, arXiv preprint arXiv, 2006, 09365.

[40] C. Liao, H. Zhong, A. Squicciarini, S. Zhu,D. Miller, Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbation, arXiv preprint arXiv:1808.10307.

[41] Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, X. Zhang,Trojaning attack on neural networks, in:Proceedings of NDSS, 2018.

[42] S. Zhao, X. Ma, X. Zheng, J. Bailey, J. Chen, Y.-G. Jiang,Clean-label backdoor attacks on video recognition models, in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 14443-14452.

[43] J. Dumford, W. Scheirer, Backdooring convolutional neural networks via targeted weight perturbations, in: Proceedings of the 2020 IEEE International Joint Conference on Biometrics (IJCB), IEEE, 2020, pp. 1-9.

[44] Y. Li, B. Wu, Y. Jiang, Z. Li, S.-T. Xia, Backdoor Learning: A Survey, arXiv preprint arXiv, 2007, 08745.

[45] Y. Liu, Y. Xie, A. Srivastava, Neural trojans, in: Proceedings of the 2017 IEEE International Conference on Computer Design (ICCD), IEEE, 2017, pp. 45-48.

[46] B.G. Doan, E. Abbasnejad, D.C. Ranasinghe, Februus: input purification defense against trojan attacks on deep neural network systems,in: Proceedings of the Annual Computer Security Applications Conference, 2020, pp. 897-912.

[47] S. Udeshi, S. Peng, G. Woo, L. Loh, L. Rawshan, S. Chattopadhyay,Model Agnostic Defence against Backdoor Attacks in Machine Learning, arXiv preprint arXiv: 1908.02203.

[48] H. Qiu, Y. Zeng, S. Guo, T. Zhang, M. Qiu, B. Thuraisingham, Deepsweep: an evaluation framework for mitigating dnn backdoor attacks using data augmentation,in: Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security, 2021, pp. 363-377.

[49] Y. Gao, C. Xu, D. Wang, S. Chen, D.C. Ranasinghe, S. Nepal, Strip: a defence against trojan attacks on deep neural networks,in: Proceedings of the 35th Annual Computer Security Applications Conference, 2019, pp. 113-125.

[50] M. Subedar, N. Ahuja, R. Krishnan, I. J. Ndiour,O. Tickoo, Deep Probabilistic Models to Detect Data Poisoning Attacks, arXiv preprint arXiv:1912.01206.

[51] M. Du, R. Jia,D. Song, Robust Anomaly Detection and Backdoor Attack Detection via Differential Privacy, arXiv preprint arXiv:1911.07116.

[52] M. Javaheripi, M. Samragh, G. Fields, T. Javidi, F. Koushanfar, Cleann: accelerated trojan shield for embedded neural networks,in:Proceedings of the 2020 IEEE/ACM International Conference on Computer Aided Design (ICCAD), IEEE, 2020, pp. 1-9.

[53] B. Wang, Y. Yao, S. Shan, H. Li, B. Viswanath, H. Zheng, B.Y. Zhao, Neural cleanse: identifying and mitigating backdoor attacks in neural networks,in:Proceedings of the 2019 IEEE Symposium on Security and Privacy (SP), IEEE, 2019, pp. 707-723.

[54] H. Chen, C. Fu, J. Zhao, F. Koushanfar, Deepinspect: a black-box trojan detection and mitigation framework for deep neural networks,in: Proceedings of the International Joint Conference on Artificial Intelligence, 2019, pp. 4658-4664.

[55] X. Xu, Q. Wang, H. Li, N. Borisov, C. A. Gunter,B. Li, Detecting Ai Trojans Using Meta Neural Analysis, arXiv preprint arXiv:1910.03137.