Black-Box Watermarking and Blockchain for IP Protection of Voiceprint Recognition Model
Abstract
:1. Introduction
- (1)
- A black-box watermarking framework is proposed based on the Mel spectrogram to achieve copyright protection of speaker recognition models. It embeds the watermark information into the Mel spectrogram instead of directly embedding it into the original speech signal. This approach effectively improves the robustness and security of the watermark information;
- (2)
- A proactive defense framework based on blockchain access control is designed. This framework centers around the copyright owner, and visitors must obtain permission from the owner to share the model. By restricting the use of the model by unauthorized users, it achieves proactive protection of the speaker recognition model;
- (3)
- Experimental results demonstrate the effectiveness of the proposed watermarking framework.
2. Background
2.1. Blockchain Technology
2.2. Neural Network Backdoor
2.3. Spectrogram and Mel Spectrogram
2.4. Watermark Embedding Algorithm
3. Proposed Method
3.1. Blockchain Network
3.2. Watermark Generation
3.3. Watermark Embedding
Algorithm 1: the generation and embedding scheme of the watermark |
Input: training set of original audio , number of trigger audio samples , key information , |
Output: watermarked model |
; |
; /*extraction of Mel spectrogram from original audio*/ |
;/*selection of suitable matrix A in Mel spectrogram*/ |
; /*DCT transformation of matrix A row by row and embedding of key information,*/ |
; /*inverse DCT transformation of watermarked matrix A*/ |
; |
; |
; /*watermark generation*/ |
/*embedding of watermark*/ |
3.4. Watermark Verification
4. Experiment
4.1. Model and Datasets
4.2. Fidelity and Efficiency
4.3. Imperceptibility
4.4. Effectiveness
4.5. Robustness
4.5.1. Fine-Tuning
4.5.2. Pruning
4.6. Ambiguity Attacks
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Amodei, D.; Ananthanarayanan, S.; Anubhai, R.; Bai, J.; Battenberg, E.; Case, C.; Casper, J.; Catanzaro, B.; Cheng, Q.; Chen, G.; et al. Deep speech 2: End-to-end speech recognition in English and mandarin. In Proceedings of the International Conference on Machine Learning, New York, NY, USA, 19–24 June 2016; pp. 173–182. [Google Scholar]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 2012, 25, 1097–1105. [Google Scholar] [CrossRef]
- Sennrich, R.; Haddow, B.; Birch, A. Neural machine translation of rare words with subword units. arXiv 2015, arXiv:1508.07909. [Google Scholar] [CrossRef]
- You, H.; Li, C.; Xu, P.; Fu, Y.; Wang, Y.; Chen, X.; Baraniuk, R.G.; Wang, Z.; Lin, Y. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv 2019, arXiv:1909.11957. [Google Scholar] [CrossRef]
- Kapse, A.; Belokar, S.; Gorde, Y.; Rane, R.; Yewtkar, S. Digital image security using digital watermarking. Int. Res. J. Eng. Technol. 2018, 5, 163–166. [Google Scholar]
- Prajwalasimha, S.; Sowmyashree, A.; Suraksha, B.; Shashikumar, H.P. Logarithmic Transform based Digital Watermarking Scheme. In Proceedings of the International Conference on ISMAC in Computational Vision and Bio-Engineering, Palladam, India, 16–17 May 2018; Springer International Publishing: Cham, Switzerland, 2019; pp. 9–16. [Google Scholar]
- Kumaraswamy, E.; Kumar, G.M.; Mahender, K.; Bukkapatnam, K.; Prasad, C.R. Digital Watermarking: State of The Art and Research Challenges in Health Care & Multimedia Applications. In Proceedings of the IOP Conference Series: Materials Science and Engineering, Warangal, India, 9–10 October 2020; IOP: Bristol, UK, 2020; p. 032031. [Google Scholar]
- Fkirin, A.; Attiya, G.; El-Sayed, A.; Shouman, M.A. Copyright protection of deep neural network models using digital watermarking: A comparative study. Multimed. Tools Appl. 2022, 81, 15961–15975. [Google Scholar] [CrossRef]
- Vybornova, Y. Method for copyright protection of deep neural networks using digital watermarking. In Proceedings of the Fourteenth International Conference on Machine Vision (ICMV 2021), Rome, Italy, 8–12 November 2021; Volume 12084, pp. 297–304. [Google Scholar]
- Fan, X.; Gui, H.; Zhou, X. PCPT and ACPT: Copyright Protection and Traceability Scheme for DNN Model. arXiv 2022, arXiv:2206.02541. [Google Scholar]
- Zhong, H.; Chang, J.; Yang, Z.; Wu, T.; Arachchige, P.C.M.; Pathmabandu, C.; Xue, M. Copyright Protection and Accountability of Generative AI: Attack, Watermarking and Attribution. arXiv 2023, arXiv:2303.09272. [Google Scholar]
- Chen, X.; Wang, W.; Ding, Y.; Bender, C.; Jia, R.; Li, B.; Song, D. Leveraging unlabeled data for watermark removal of deep neural networks. In Proceedings of the ICML Workshop on Security and Privacy of Machine Learning, Long Beach, CA, USA, 16–20 June 2019. [Google Scholar]
- Fan, X.; Zhou, X.; Zhu, B.; Dong, J.; Niu, J.; Wang, H. Survey of copyright protection schemes based on DNN model. J. Comput. Res. Dev. 2022, 59, 953–977. [Google Scholar]
- Uchida, Y.; Nagai, Y.; Sakazawa, S.; Satoh, S. Embedding watermarks into deep neural networks. In Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, Bucharest, Romania, 6–9 June 2017; ACM: New York, NY, USA, 2017; pp. 269–277. [Google Scholar]
- Fan, L.; Ng, W.K.; Chan, C.S. Rethinking deep neural network ownership verification: Embedding passports to defeat ambiguity attacks. In Proceedings of the 33rd Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019; pp. 1–10. [Google Scholar]
- Li, Z.; Hu, C.; Zhang, Y.; Guo, S. How to prove your model belongs to you: A blind-watermark based framework to protect intellectual property of DNN. In Proceedings of the 35th Annual Computer Security Applications Conference, San Juan, Puerto Rico, 9–13 December 2019; pp. 126–137. [Google Scholar]
- Hua, G.; Teoh, A.B.J. Deep fidelity in DNN watermarking: A study of backdoor watermarking for classification models. Pattern Recognit. 2023, 144, 109844. [Google Scholar] [CrossRef]
- Li, F.Q.; Wang, S.L.; Zhu, Y. Fostering the robustness of white-box deep neural network watermarks by neuron alignment. In Proceedings of the ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore, 23–27 May 2022; pp. 3049–3053. [Google Scholar]
- Yan, Y.; Pan, X.; Zhang, M.; Yang, M. Rethinking White-Box Watermarks on Deep Learning Models under Neural Structural Obfuscation. In Proceedings of the 32th USENIX Security Symposium (USENIX Security 23), Anaheim Marriott Hotel in Anaheim, CA, USA, 9–11 August 2023. [Google Scholar]
- Kuribayashi, M.; Tanaka, T.; Suzuki, S.; Yasui, T.; Funabiki, N. White-box watermarking scheme for fully-connected layers in fine-tuning model. In Proceedings of the 2021 ACM Workshop on Information Hiding and Multimedia Security, Virtual Event, Belgium; 2021; pp. 165–170. [Google Scholar]
- Kuribayashi, M.; Yasui, T.; Malik, A. White Box Watermarking for Convolution Layers in Fine-Tuning Model Using the Constant Weight Code. J. Imaging 2023, 9, 117. [Google Scholar] [CrossRef]
- Lv, H.; Shen, S.; Lin, H.; Yuan, Y.; Duan, D. SVD Mark: A Novel Black-Box Watermarking for Protecting Intellectual Property of Deep Neural Network Model. In Proceedings of the 8th International Conference on Artificial Intelligence and Security, Qinghai, China, 15–20 July 2022; pp. 390–410. [Google Scholar]
- Liu, Y.; Wu, H.; Zhang, X. Robust and imperceptible black-box DNN watermarking based on Fourier perturbation analysis and frequency sensitivity clustering. arXiv 2022, arXiv:2208.03944. [Google Scholar]
- Meng, Y.; Chen, X.; Sun, X.; Liu, Y.; Wei, G. A Dual Model Watermarking Framework for Copyright Protection in Image Processing Networks. Cmc-Comput. Mater. Contin. 2023, 75, 831–844. [Google Scholar] [CrossRef]
- Chen, J.; Wang, J.; Peng, T.; Sun, Y.; Cheng, P.; Ji, S.; Ma, X.; Li, B.; Song, D. Copy, right? a testing framework for copyright protection of deep learning models. In Proceedings of the 2022 IEEE Symposium on Security and Privacy (SP), San Francisco, CA, USA, 22–26 May 2022; pp. 824–841. [Google Scholar]
- Vybornova, Y.; Ulyanov, D. Copyright protection for image classification models using pseudo-holographic watermarks. In Proceedings of the Fifteenth International Conference on Machine Vision (ICMV 2022), Rome, Italy, 18–20 November 2022; SPIE: Bellingham, WA, USA, 2023; Volume 12701, pp. 183–191. [Google Scholar]
- Shen, W.; Rong, J.; Liu, Y.; Zhao, Y. IrisMarkNet: Iris feature watermarking embedding and extraction network for image copyright protection. Appl. Intell. 2023, 53, 9992–10008. [Google Scholar] [CrossRef]
- Chen, H.; Rouhani, B.D.; Koushanfar, F. SpecMark: A Spectral Watermarking Framework for IP Protection of Speech Recognition Systems. Interspeech 2020, 2312–2316. [Google Scholar]
- Wang, Y.; Wu, H. Protecting the intellectual property of speaker recognition model by black-box watermarking in the frequency domain. Symmetry 2022, 14, 619. [Google Scholar] [CrossRef]
- Zhang, J.; Chen, D.; Liao, J.; Zhang, W.; Hua, G.; Yu, N. Passport-aware Normalization for Deep Model Protection. Adv. Neural Inf. Process. Syst. 2020, 33, 22619–22628. [Google Scholar]
- Yang, C.H.H.; Qi, J.; Chen, S.Y.C.; Chen, P.-Y.; Siniscalchi, S.M.; Ma, X.; Lee, C.H. Decentralizing feature extraction with quantum convolutional neural network for automatic speech recognition. In Proceedings of the ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 6–11 June 2021; pp. 6523–6527. [Google Scholar]
- Rajasekaran, A.S.; Azees, M.; Al-Turjman, F. A comprehensive survey on blockchain technology. Sustain. Energy Technol. Assess. 2022, 52, 102039. [Google Scholar] [CrossRef]
- Patil, P.; Sangeetha, M.; Bhaskar, V. Blockchain for IoT Access Control, Security and Privacy: A Review. Wirel. Pers. Commun. 2021, 117, 1815–1834. [Google Scholar] [CrossRef]
- Hewa, T.; Ylianttila, M.; Liyanage, M. Survey on blockchain based smart contracts: Applications, opportunities and challenges. J. Netw. Comput. Appl. 2021, 177, 102857. [Google Scholar] [CrossRef]
- Habib, G.; Sharma, S.; Ibrahim, S.; Ahmad, I.; Qureshi, S.; Ishfaq, M. Blockchain Technology: Benefits, Challenges, Applications, and Integration of Blockchain Technology with Cloud Computing. Future Internet 2022, 14, 341. [Google Scholar] [CrossRef]
- Kumar, P.; Sharma, S.K. An empirical evaluation of various digital signature scheme in wireless sensor network. IETE Tech. Rev. 2022, 39, 974–984. [Google Scholar] [CrossRef]
- Li, D.; Han, D.; Crespi, N.; Minerva, R.; Li, K. A blockchain-based secure storage and access control scheme for supply chain finance. J. Supercomput. 2023, 79, 109–138. [Google Scholar] [CrossRef]
- Yao, Y.; Li, H.; Zheng, H.; Zhao, Y.B. Latent backdoor attacks on deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, London, UK, 11–15 November 2019; pp. 2041–2055. [Google Scholar]
- Gao, Y.; Doan, B.G.; Zhang, Z.; Ma, S.; Zhang, J.; Fu, A.; Nepal, S.; Kim, H. Backdoor attacks and countermeasures on deep learning: A comprehensive review. arXiv 2020, arXiv:2007.10760. [Google Scholar]
- Zhang, J.; Gu, Z.; Jang, J.; Wu, H.; Stoecklin, M.; Huang, H.; Molloy, I. Protecting intellectual property of deep neural networks with watermarking. In Proceedings of the 2018 on Asia Conference on Computer and Communications Security, 2018; ACM: New York, NY, USA, 2018; pp. 159–172. [Google Scholar]
- Adi, Y.; Baum, C.; Cisse, M.; Pinkas, B.; Keshet, J. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In Proceedings of the 27th USENIX Security, Baltimore, MD, USA, 15–17 August 2018; USENIX Association: Berkeley, CA, USA, 2018; pp. 1615–1631. [Google Scholar]
- Namba, R.; Sakuma, J. Robust watermarking of neural network with exponential weighting. In Proceedings of the 2019 ACM Asia Conference on Computer and Communications Security; ACM: New York, NY, USA, 2019; pp. 228–240. [Google Scholar]
- Zhang, Y.; Sun, G. A watermark algorithm based on space-domain and transform-domain. In Proceedings of the 2019 IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC), Beijing, China, 12–14 July 2019; pp. 41–44. [Google Scholar]
- Zhong, Q.; Zhang, L.Y.; Zhang, J.; Gao, L.; Xiang, Y. Protecting IP of Deep Neural Networks with Watermarking: A New Label Helps. Adv. Knowl. Discov. DataMin. 2020, 12085, 462–474. [Google Scholar]
- Faheem, Z.B.; Ishaq, A.; Rustam, F.; de la Torre Díez, I.; Gavilanes, D.; Vergara, M.M.; Ashraf, I. Image Watermarking Using Least Significant Bit and Canny Edge Detection. Sensors 2023, 23, 1210. [Google Scholar] [CrossRef]
- Mohammed, A.A.; Salih, D.A.; Saeed, A.M.; Kheder, M.Q. An imperceptible semi-blind image watermarking scheme in DWT-SVD domain using a zigzag embedding technique. Multimed. Tools Appl. 2020, 79, 32095–32118. [Google Scholar] [CrossRef]
- Sharma, A.; Kumar, P.; Maddukuri, V.; Madamshetti, N.; Kishore, K.G.; Kavuru, S.S.S.; Raman, B. Fast Griffin Lim based waveform generation strategy for text-to-speech synthesis. Multimed. Tools Appl. 2020, 79, 30205–30233. [Google Scholar] [CrossRef]
- Ravanelli, M.; Bengio, Y. Speaker recognition from raw waveform with sincnet. In Proceedings of the 2018 IEEE Spoken Language Technology Workshop (SLT), Athens, Greece, 18–21 December 2018; pp. 1021–1028. [Google Scholar]
- Garofolo, J.S.; Lamel, L.F.; Fisher, W.M.; Fiscus, J.G.; Pallett, D.S.; Dahlgren, N.L.; Zue, V. TIMIT Acoustic-Phonetic Continuous Speech Corpus LDC93S1; Linguistic Data Consortium: Philadelphia, PA, USA, 1993. [Google Scholar]
- Panayotov, V.; Chen, G.; Povey, D.; Khudanpur, S. Librispeech: An asr corpus based on public domain audio books. In Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, QLD, Australia, 19–24 April 2015; pp. 5206–5210. [Google Scholar]
Datasets | TIMIT | LibriSpeech | ||||
---|---|---|---|---|---|---|
model | SincNet | Wang et al.’s | ours | SincNet | Wang et al.’s | ours |
FER(%) | 0.4873 | 0.5078 | 0.4880 | 0.4956 | 0.5139 | 0.4969 |
SER(%) | 0.0072 | 0.0192 | 0.0101 | 0.0158 | 0.0230 | 0.0173 |
running time | 4 h 57 min | 5 h 10 min | 5 h 26 min | 5 h 34 min | 5 h 40 min | 6 h 20 min |
Datasets | TIMIT | LibriSpeech | ||||
---|---|---|---|---|---|---|
model | CNN | Wang et al.’s | ours | CNN | Wang et al.’s | ours |
FER(%) | 0.4853 | 0.5063 | 0.4900 | 0.4892 | 0.5077 | 0.4898 |
SER(%) | 0.0093 | 0.0173 | 0.0123 | 0.0108 | 0.0166 | 0.0144 |
running time | 17 h 44 min | 18 h 39 min | 18 h 56 min | 18 h 40 min | 19 h 08 min | 19 h 58 min |
50 Samples | The SSIM Value | The Cosine Similarity |
---|---|---|
Sample1 | 0.9999 | 0.9803 |
Sample2 | 0.9999 | 0.9967 |
Sample3 | 0.9999 | 0.9875 |
Sample4 | 0.9999 | 0.9808 |
Sample5 | 0.9999 | 0.9974 |
Sample6 | 0.9999 | 0.9982 |
Sample47 | 0.9999 | 0.9866 |
Sample48 | 0.9999 | 0.9972 |
Sample49 | 0.9999 | 0.9977 |
Sample50 | 0.9999 | 0.9849 |
Datasets | Train/Test | SincNet | CNN |
---|---|---|---|
TIMIT | D (train) | 100% | 100% |
D′ (test) | 98.67% | 96.67% | |
LibriSpeech | D (train) | 100% | 100% |
D′ (test) | 97.33% | 95.33% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhang, J.; Dai, L.; Xu, L.; Ma, J.; Zhou, X. Black-Box Watermarking and Blockchain for IP Protection of Voiceprint Recognition Model. Electronics 2023, 12, 3697. https://doi.org/10.3390/electronics12173697
Zhang J, Dai L, Xu L, Ma J, Zhou X. Black-Box Watermarking and Blockchain for IP Protection of Voiceprint Recognition Model. Electronics. 2023; 12(17):3697. https://doi.org/10.3390/electronics12173697
Chicago/Turabian StyleZhang, Jing, Long Dai, Liaoran Xu, Jixin Ma, and Xiaoyi Zhou. 2023. "Black-Box Watermarking and Blockchain for IP Protection of Voiceprint Recognition Model" Electronics 12, no. 17: 3697. https://doi.org/10.3390/electronics12173697
APA StyleZhang, J., Dai, L., Xu, L., Ma, J., & Zhou, X. (2023). Black-Box Watermarking and Blockchain for IP Protection of Voiceprint Recognition Model. Electronics, 12(17), 3697. https://doi.org/10.3390/electronics12173697