Fast-M Adversarial Training Algorithm for Deep Neural Networks
Abstract
:1. Introduction
- This paper analyzes the differences between PGD adversarial training and free and fast adversarial training in training overhead and discusses the influence of the step size and the number of iterations on the training performance.
- This paper proposes an improved Fast-M adversarial training algorithm based on the fast adversarial training method to reduce the computational overload while maintaining the training performance.
- Extensive experiments are conducted with the MNIST, CIFAR10, and CIFAR100 datasets. Results show that the Fast-M algorithm achieves the same training effect as the commonly used adversarial training method, with a training time that is only one-third that of PGD adversarial training.
2. Related Work
3. Preliminary
3.1. Adversarial Attack
3.2. Fast Gradient Sign Method
3.3. Projected Gradient Descent
4. Discussion of Typical Adversarial Training Algorithms
4.1. PGD Adversarial Training
Algorithm 1. K-Step PGD Adversarial Training | |||
4.2. Free Adversarial Training
Algorithm 2. Free Adversarial Training | |||
4.3. Fast Adversarial Training
Algorithm 3. Fast Adversarial Training |
5. Fast-M Adversarial Training Algorithm
5.1. Design Rationale
5.2. Method
Algorithm 4. Fast-M Adversarial Training |
6. Experiments
6.1. Verification of Fast-M on MNIST
6.2. Verification of Fast-M on CIFAR10
6.2.1. Number of Iteration Stage Selection
6.2.2. Training Step Size Selection
6.2.3. Evaluation Comparison of Fast-M Adversarial Training
6.3. Verification of Fast-M on CIFAR100
7. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Akhtar, N.; Mian, A. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access 2018, 6, 14410–14430. [Google Scholar] [CrossRef]
- Szegedy, C.; Zaremba, W.; Sutskever, I.; Bruna, J.; Erhan, D.; Goodfellow, I.; Fergus, R. Intriguing properties of neural networks. arXiv 2013, arXiv:1312.6199. [Google Scholar]
- Goodfellow, I.J.; Shlens, J.; Szegedy, C. Explaining and harnessing adversarial examples. In Proceedings of the International Conference on Learning Representations, Banff, AB, Canada, 14–16 April 2014; Volume 101. [Google Scholar]
- Wong, E.; Rice, L.; Kolter, J.Z. Fast is better than free: Revisiting adversarial training. In Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA, 6–9 May 2019; Volume 103. [Google Scholar]
- Papernot, N.; McDaniel, P.; Wu, X.; Jha, S.; Swami, A. Distillation as a defense to adversarial perturbations against deep neural networks. In Proceedings of the 2016 IEEE Symposium on Security and Privacy (SP), San Jose, CA, USA, 22–26 May 2016; pp. 582–597. [Google Scholar]
- Metzen, J.H.; Genewein, T.; Fischer, V.; Bischoff, B. On detecting adversarial perturbations. In Proceedings of the International Conference on Learning Representations, Toulon, France, 24–26 April 2017. [Google Scholar]
- Feinman, R.; Curtin, R.R.; Shintre, S.; Gardner, A.B. Detecting adversarial samples from artifacts. arXiv 2017, arXiv:1703.00410. [Google Scholar]
- Pinot, R.; Ettedgui, R.; Rizk, G.; Chevaleyre, Y.; Atif, J. Randomization matters how to defend against strong adversarial attacks. In Proceedings of the 37th International Conference on Machine Learning, PMLR, Virtual Event, 13–18 July 2020; pp. 7717–7772. [Google Scholar]
- Madry, A.; Makelov, A.; Schmidt, L.; Tsipras, D.; Vladu, A. Towards deep learning models resistant to adversarial attacks. In Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada, 30 April–3 May 2018; Volume 97. [Google Scholar]
- Shafahi, A.; Najibi, M.; Ghiasi, M.A.; Xu, Z.; Dickerson, J.; Studer, C.; Davis, L.S.; Taylor, G.; Goldstein, T. Adversarial training for free! In Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019; Volume 32.
- Smith, L.N.; Topin, N. Super-convergence: Very fast training of neural networks using large learning rates. In Proceedings of the Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, SPIE, Baltimore, MD, USA, 14–18 April 2019; Volume 11006, pp. 369–386. [Google Scholar]
- Micikevicius, P.; Narang, S.; Alben, J.; Diamos, G.; Elsen, E.; Garcia, D.; Ginsburg, B.; Houston, M.; Kuchaiev, O.; Venkatesh, G.; et al. Mixed precision training. In Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada, 30 April–3 May 2018; Volume 95. [Google Scholar]
- Ye, N.; Li, Q.; Zhou, X.Y.; Zhu, Z. Amata: An annealing mechanism for adversarial training acceleration. In Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada, 2–9 February 2021; Volume 35, No. 12. pp. 10691–10699. [Google Scholar]
- Sriramanan, G.; Addepalli, S.; Baburaj, A. Guided adversarial attack for evaluating and enhancing adversarial defenses. In Proceedings of the Advances in Neural Information Processing Systems, Virtual, 6–12 December 2020; Volume 33, pp. 20297–20308. [Google Scholar]
- Kurakin, A.; Goodfellow, I.J.; Bengio, S. Adversarial examples in the physical world. In Artificial Intelligence Safety and Security; Chapman and Hall/CRC: Boca Raton, FL, USA, 2018; pp. 99–112. [Google Scholar]
- Moosavi-Dezfooli, S.M.; Fawzi, A.; Frossard, P. Deepfool: A simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 2574–2582. [Google Scholar]
- Carlini, N.; Wagner, D. Towards evaluating the robustness of neural networks. In Proceedings of the 2017 IEEE Symposium on Security and Privacy (SP), San Jose, CA, USA, 22–24 May 2017; pp. 39–57. [Google Scholar]
- Su, J.; Vargas, D.V.; Sakurai, K. One pixel attack for fooling deep neural networks. IEEE Trans. Evol. Comput. 2019, 23, 828–841. [Google Scholar] [CrossRef]
- Papernot, N.; McDaniel, P.; Goodfellow, I.; Jha, S.; Celik, Z.B.; Swami, A. Practical black-box attacks against deep learning systems using adversarial examples. arXiv arXiv:1602.02697, 2016.
- Chen, P.Y.; Zhang, H.; Sharma, Y.; Yi, J.; Hsieh, C.J. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, Dallas, TX, USA, 3 November 2017; pp. 15–26. [Google Scholar]
Epoch | Fast | Fast × 2 | |||||
---|---|---|---|---|---|---|---|
10 | 15 | 19 | 24 | 30 | 33 | 20 | 30 |
20 | 30 | 39 | 50 | 60 | 68 | 40 | 60 |
30 | 45 | 60 | 74 | 90 | 105 | 60 | 90 |
40 | 60 | 79 | 100 | 120 | 138 | 80 | 120 |
50 | 75 | 99 | 124 | 150 | 173 | 100 | 150 |
Algorithm | Standard | FGSM () | PGD-20 () | PGD-50 () | PGD-20 () | Training Time (min) |
---|---|---|---|---|---|---|
Fast | 84.13% | 54.8% | 46.73% | 46.28% | 36.63% | 25.670 |
Free (m = 4) | 86.04% | 53.92% | 44.89% | 44.57% | 33.41% | 43.820 |
Free (m = 8) | 82.12% | 53.93% | 47.94% | 47.76% | 38.48% | 85.819 |
PGD-7 | 81.26% | 55.8% | 50.99% | 50.75% | 42.14% | 83.241 |
PGD-10 | 80.67% | 55.45% | 50.85% | 50.76% | 42.67% | 112.321 |
Amata | 80.64% | 55.38% | 50.05% | 49.84% | 41.44% | 67.747 |
Fast-3 | 81.78% | 55.88% | 50.74% | 50.51% | 41.39% | 25.892 |
Fast-4 | 81.42% | 56.3% | 51.32% | 51.11% | 42.65% | 30.202 |
Fast-5 | 81.41% | 56.16% | 51.66% | 51.42% | 43.04% | 35.126 |
Fast-6 | 81.30% | 56.12% | 51.57% | 51.38% | 43.10% | 39.887 |
Algorithm | Standard | PGD-20 () | PGD-50 () | Training Time (min) |
---|---|---|---|---|
Fast | 58.24% | 0.54% | 0.3 | 53.11 |
Free (m = 8) | 49.13% | 12.14% | 11.93% | 107.16 |
Free (m = 8) | 56.06% | 20.55% | 20.26% | 213.55 |
PGD-7 | 55.51% | 20.32% | 20.06% | 211.32 |
PGD-10 | 54.89% | 21.38% | 21.15% | 289.33 |
Fast-3 (ours) | 59.02% | 20.31% | 20.01% | 54.39 |
Fast-4 (ours) | 57.81% | 21.53% | 21.28% | 67.05 |
Fast-5 (ours) | 57.78% | 22.18% | 21.88% | 80.11 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Ma, Y.; An, D.; Gu, Z.; Lin, J.; Liu, W. Fast-M Adversarial Training Algorithm for Deep Neural Networks. Appl. Sci. 2024, 14, 4607. https://doi.org/10.3390/app14114607
Ma Y, An D, Gu Z, Lin J, Liu W. Fast-M Adversarial Training Algorithm for Deep Neural Networks. Applied Sciences. 2024; 14(11):4607. https://doi.org/10.3390/app14114607
Chicago/Turabian StyleMa, Yu, Dou An, Zhixiang Gu, Jie Lin, and Weiyu Liu. 2024. "Fast-M Adversarial Training Algorithm for Deep Neural Networks" Applied Sciences 14, no. 11: 4607. https://doi.org/10.3390/app14114607
APA StyleMa, Y., An, D., Gu, Z., Lin, J., & Liu, W. (2024). Fast-M Adversarial Training Algorithm for Deep Neural Networks. Applied Sciences, 14(11), 4607. https://doi.org/10.3390/app14114607