# Coded DNN Watermark: Robustness against Pruning Models Using Constant Weight Code

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Preliminaries

#### 2.1. Notation

- N: Number of weight parameters in DNN model.
- L: Number of selected weight parameters.
- $w=({w}_{0},{w}_{1},\dots ,{w}_{L-1}),{w}_{i}\in \mathbb{R}$: Selected weight parameters.
- ${w}^{\prime}$: Selected weight parameters after pruning attack.
- R: Pruning rate.
- $p=\lfloor RN\rfloor $: Number of pruned weight parameters.
- k: Bit length of watermark.
- $\mathit{b}=({b}_{0},{b}_{1},\dots ,{b}_{k-1}),{b}_{i}\in \{0,1\}$: Binary watermark.
- $\mathit{c}=({c}_{0},{c}_{1},\dots ,{c}_{L-1}),{c}_{i}\in \{0,1\}$: Binary codeword of CWC.
- $\alpha =\sum {c}_{i}$: Hamming weight of codewords.
- ${T}_{1}$, ${T}_{0}$: Thresholds for binary classification, where $0<{T}_{0}<{T}_{1}$.
- $sort()$: Sort algorithm with ascending order.
- $sgn()$: Sign function.

#### 2.2. Pruning DNN Model

#### 2.3. Constant Weight Code

**b**into a codeword $\mathit{c}\in \mathcal{C}(\alpha ,L)$ is described in Algorithm 1. The k-bit information

**b**is recovered by decoding the codeword c using Algorithm 2, where “≫” denotes the right bit-shift operator.

Algorithm 1 Encode b into c |

Input: $\alpha $, L, $\mathit{b}=({b}_{0},{b}_{1},\dots ,{b}_{k-1})$, ${b}_{t}\in \{0,1\}$ |

Output: $\mathit{c}=({c}_{0},{c}_{1},\dots ,{c}_{L-1})$, ${c}_{t}\in \{0,1\}$ |

1: $B\leftarrow \sum _{t=0}^{k-1}{b}_{t}{2}^{t}$; |

2: $\mathcal{l}\leftarrow \alpha $; |

3: for
$t=0$ to
$L-1$
do |

4: if $B\ge \left(\genfrac{}{}{0pt}{}{L-t-1}{\mathcal{l}}\right)$ then |

5: ${c}_{L-t-1}=1$; |

6: $B\leftarrow B-\left(\genfrac{}{}{0pt}{}{L-t-1}{\mathcal{l}}\right)$; |

7: $\mathcal{l}\leftarrow \mathcal{l}-1$; |

8: else |

9: ${c}_{L-t-1}=0$; |

10: end if |

11: end for |

Algorithm 2 Decode c into b |

Input:
$\alpha $, L, $\mathit{c}=({c}_{0},{c}_{1},\dots ,{c}_{L-1})$, ${c}_{t}\in \{0,1\}$ |

Output:
$\mathit{b}=({b}_{0},{b}_{1},\dots ,{b}_{k-1})$, ${b}_{t}\in \{0,1\}$ |

1: $B\leftarrow 0$; |

2: $\mathcal{l}\leftarrow 0$; |

3: for
$t=0$ to
$L-1$
do |

4: if ${c}_{t}=1$ then |

5: $\mathcal{l}\leftarrow \mathcal{l}+1$; |

6: $B\leftarrow B+\left(\genfrac{}{}{0pt}{}{t}{\mathcal{l}}\right)$; |

7: end if |

8: end for |

9: for
$t=0$ to
$k-1$
do |

10: ${b}_{t}=B\phantom{\rule{4.44443pt}{0ex}}(mod\phantom{\rule{0.277778em}{0ex}}2)$ |

11: $B\leftarrow B\gg 1$; |

12: end for |

## 3. Proposed DNN Watermarking

**b**into the codeword c using CWC before the embedding operation. The weights corresponding to the elements ${c}_{i}=1$ become more than a higher threshold ${T}_{1}$, while the others corresponding to ${c}_{i}=0$ becomes less than a lower threshold ${T}_{0}$ by the embedding operation. In case the pruning attack is executed to round weight parameters ${w}_{i}$ with a small value to 0, those elements are judged as bit 0 in the codeword, and hence, there is no effect on the received codeword. As for bit 1, the corresponding weight parameters ${w}_{i}$ should be sufficiently large so that these are not cut off.

**w**are selected from N candidates according to a secret key. Then, an encoded watermark

**c**is embedded into

**w**under the following constraint:

- If ${c}_{i}=1$, then $|{w}_{i}|\ge {T}_{1}$; otherwise, $|{w}_{i}|\le {T}_{0}$, where ${T}_{0}$ and ${T}_{1}$ are thresholds satisfying $0<{T}_{0}<{T}_{1}$.

**w**selected for embedding

**c**are only controlled by the above restriction during the training process in the proposed method.

#### 3.1. Embedding

**b**into the codeword

**c**by using Algorithm 1. Here, the parameters $\alpha $ and L must satisfy the following condition:

**w**selected from the DNN model are modified into ${\mathit{w}}^{\mathbf{\u2020}}=({w}_{0}^{\u2020},{w}_{1}^{\u2020},\dots ,{w}_{L-1}^{\u2020})$ by using the two thresholds ${T}_{1}$ and ${T}_{0}$.

#### 3.2. Extraction

#### Detection

- Top $\alpha $-th weights: the uniform distribution in the range $[{T}_{1},\delta ]$, the mean is $(\delta +{T}_{1})/2$.
- Remainder: the uniform distribution in the range $[0,{T}_{0}]$, the mean is ${T}_{0}/2$.

#### 3.3. Recovery of Watermark

#### 3.4. Design of Two Thresholds

#### 3.5. Considerations

#### 3.6. Numerical Examples

## 4. Experimental Results

#### 4.1. Experimental Conditions

#### 4.1.1. Fine-Tuning Model

#### 4.1.2. Threshold

#### 4.2. Effect of Watermark on Original Task

#### 4.3. Detection Performance

#### 4.4. Robustness against Pruning Attacks

#### 4.5. Retrained DNN Model after Pruning Attack

#### 4.6. Comparison with Previous Studies

## 5. Conclusions and Future Works

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Chen, H.; Rouhani, B.D.; Fan, X.; Kilinc, O.C.; Koushanfar, F. Performance Comparison of Contemporary DNN Watermarking Techniques. arXiv
**2018**, arXiv:1811.03713. [Google Scholar] - Le Merrer, E.; Perez, P.; Trédan, G. Adversarial frontier stitching for remote neural network watermarking. Neural Comput. Appl.
**2020**, 32, 9233–9244. [Google Scholar] [CrossRef] [Green Version] - Wu, H.; Liu, G.; Yao, Y.; Zhang, X. Watermarking Neural Networks With Watermarked Images. IEEE Trans. Circuits Syst. Video Technol.
**2021**, 31, 2591–2601. [Google Scholar] [CrossRef] - Kong, Y.; Zhang, J. Adversarial audio: A new information hiding method and backdoor for dnn-based speech recognition models. arXiv
**2019**, arXiv:1904.03829. [Google Scholar] - Wang, Y.; Wu, H. Protecting the Intellectual Property of Speaker Recognition Model by Black-Box Watermarking in the Frequency Domain. Symmetry
**2022**, 14, 619. [Google Scholar] [CrossRef] - Uchida, Y.; Nagai, Y.; Sakazawa, S.; Satoh, S. Embedding watermarks into deep neural networks. In Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, Bucharest, Romania, 6–9 June 2017; pp. 269–277. [Google Scholar]
- Nagai, Y.; Uchida, Y.; Sakazawa, S.; Satoh, S. Digital watermarking for deep neural networks. Int. J. Multimed. Inf. Retr.
**2018**, 7, 3–16. [Google Scholar] [CrossRef] [Green Version] - Rouhani, B.D.; Chen, H.; Koushanfar, F. DeepSigns: An End-to-End Watermarking Framework for Ownership Protection of Deep Neural Networks. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, Providence, RI, USA, 13–17 April 2019; pp. 485–497. [Google Scholar]
- Chen, H.; Rouhani, B.D.; Fu, C.; Zhao, J.; Koushanfar, F. DeepMarks: A Secure Fingerprinting Framework for Digital Rights Management of Deep Learning Models. In Proceedings of the 2019 on International Conference on Multimedia Retrieval, Ottawa, ON, Canada, 10–13 June 2019; pp. 105–113. [Google Scholar]
- Choromanska, A.; Henaff, M.; Mathieu, M.; Arous, G.B.; LeCun, Y. The Loss Surfaces Of Multilayer Networks. In Proceedings of the Artificial Intelligence and Statistics, San Diego, CA, USA, 9–12 May 2015. [Google Scholar]
- Dauphin, Y.N.; Pascanu, R.; Gülçehre, Ç.; Cho, K.; Ganguli, S.; Bengio, Y. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; pp. 2933–2941. [Google Scholar]
- Kuribayashi, M.; Tanaka, T.; Suzuki, S.; Yasui, T.; Funabiki, N. White-box watermarking scheme for fully-connected layers in fine-tuning model. In Proceedings of the 2021 ACM Workshop on Information Hiding and Multimedia Security, Virtual Event. Belgium, 22–25 June 2021; pp. 165–170. [Google Scholar]
- Wang, J.; Hu, H.; Zhang, X.; Yao, Y. Watermarking in Deep Neural Networks via Error Back-propagation. In IS&T Electronic Imaging, Media Watermarking, Security and Forensics; SPIE: Bellingham, WA, USA, 2020. [Google Scholar]
- Liu, K.; Dolan-Gavitt, B.; Garg, S. Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks. In Research in Attacks, Intrusions, and Defenses; Bailey, M., Holz, T., Stamatogiannakis, M., Ioannidis, S., Eds.; Springer International Publishing: Cham, Switzerland, 2018; pp. 273–294. [Google Scholar]
- Li, Y.; Tondi, B.; Barni, M. Spread-Transform Dither Modulation Watermarking of Deep Neural Network. J. Inf. Secur. Appl.
**2021**, 63, 103004. [Google Scholar] [CrossRef] - Li, Y.; Wang, H.; Barni, M. A survey of deep neural network watermarking techniques. Neurocomputing
**2021**, 461, 171–193. [Google Scholar] [CrossRef] - Zhao, X.; Yao, Y.; Wu, H.; Zhang, X. Structural Watermarking to Deep Neural Networks via Network Channel Pruning. arXiv
**2021**, arXiv:2107.08688. [Google Scholar] - Schalkwijk, J.P.M. An algorithm for source coding. IEEE Trans. Inf. Theory
**1972**, 18, 395–399. [Google Scholar] [CrossRef] - Brouwer, A.E.; Shearer, J.B.; Sloane, N.J.A.; Smith, W. A new table of constant weight codes. IEEE Trans. Inf. Theory
**1990**, 36, 1334–1380. [Google Scholar] [CrossRef] - Kuribayashi, M.; Yasui, T.; Malik, A.; Funabiki, N. Immunization of Pruning Attack in DNN Watermarking Using Constant Weight Code. arXiv
**2021**, arXiv:2107.02961. [Google Scholar] - Denil, N.; Shakibi, B.; Dinh, L.; Ranzato, M.A.; Freitas, N.D. Predicting parameters in deep learning. In Proceedings of the Advances In Neural Information Processing Systems, Lake Tahoe, NV, USA, 5–10 December 2013; pp. 2148–2156. [Google Scholar]
- Guo, Y.; Yao, A.; Chen, Y. Dynamic network surgery for efficient DNNs. In Proceedings of the Advances In Neural Information Processing Systems, Barcelona, Spain, 5–10 December 2016; pp. 1379–1387. [Google Scholar]
- Han, S.; Mao, H.; Dally, W.J. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding, 2016. arXiv
**2015**, arXiv:1510.00149. [Google Scholar] - Molchanov, P.; Tyree, S.; Karras, T.; Aila, T.; Kautz, J. Pruning Convolutional Neural Networks for Resource Efficient Transfer Learning. arXiv
**2016**, arXiv:1611.06440. [Google Scholar] - Dong, X.; Chen, S.; Pan, S.J. Learning to Prune Deep Neural Networks via Layer-Wise Optimal Brain Surgeon. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 4860–4874. [Google Scholar]
- MacWilliams, F.J.; Sloane, N.J.A. The Theory of Error-Correcting Codes; Elsevier: Amsterdam, The Netherlands, 1977. [Google Scholar]
- Smith, D.H.; Hughes, L.A.; Perkins, S. A New Table of Constant Weight Codes of Length Greater than 28. Electron J. Comb.
**2006**, 13, A2. [Google Scholar] [CrossRef] [Green Version] - Nguyen, Q.A.; Gyorfi, L.; Massey, J.L. Constructions of binary constant-weight cyclic codes and cyclically permutable codes. IEEE Trans. Inf. Theory
**1992**, 38, 940–949. [Google Scholar] [CrossRef] [Green Version] - Bitan, S.; Etzion, T. Constructions for optimal constant weight cyclically permutable codes and difference families. IEEE Trans. Inf. Theory
**1995**, 41, 77–87. [Google Scholar] [CrossRef] - Etzion, T.; Vardy, A. A new construction for constant weight codes. In Proceedings of the 2014 International Symposium on Information Theory and Its Applications, Victoria, BC, Canada, 26–29 October 2014; pp. 338–342. [Google Scholar]
- Goodfellow, I.; Bengio, Y.; Courville, A. Deep Learning; MIT Press: Cambridge, MA, USA, 2016. [Google Scholar]
- Glorot, X.; Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Chia, Italy, 13–15 May 2010; Volume 9, pp. 249–256. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 7–13 December 2015; pp. 1026–1034. [Google Scholar]
- Kumar, S.K. On weight initialization in deep neural networks. arXiv
**2017**, arXiv:1704.08863. [Google Scholar] - Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv
**2014**, arXiv:1409.1556. [Google Scholar] - He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 30 June 2016. [Google Scholar]
- Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 248–255. [Google Scholar]

**Figure 4.**Example of probability distribution function of initial weight values (

**a**) Gaussian (

**b**) Uniform.

k | $\mathit{\alpha}$ | L | $\overline{\mathit{R}}$ |
---|---|---|---|

64 | 8 | 972 | 0.9918 |

9 | 583 | 0.9846 | |

10 | 393 | 0.9746 | |

11 | 288 | 0.9618 | |

128 | 16 | 1757 | 0.9909 |

18 | 1063 | 0.9831 | |

20 | 722 | 0.9723 | |

22 | 533 | 0.9587 | |

256 | 32 | 3307 | 0.9903 |

36 | 2011 | 0.9821 | |

40 | 1373 | 0.9709 | |

43 | 1090 | 0.9606 | |

512 | 63 | 6858 | 0.9908 |

73 | 3693 | 0.9802 | |

79 | 2780 | 0.9716 | |

85 | 2196 | 0.9613 | |

1024 | 127 | 12,955 | 0.9902 |

145 | 7443 | 0.9805 | |

159 | 5350 | 0.9703 | |

170 | 4323 | 0.9607 |

(a) VGG16 | ||||||

Metric | Phase | Original | $\mathcal{C}(16,1757)$ | $\mathcal{C}(18,1063)$ | $\mathcal{C}(20,722)$ | $\mathcal{C}(22,533)$ |

accuracy | training | 0.9639 | 0.9650 | 0.9648 | 0.9637 | 0.9634 |

validation | 0.9226 | 0.9137 | 0.9196 | 0.9238 | 0.9119 | |

test | 0.9071 | 0.9041 | 0.9029 | 0.9088 | 0.9068 | |

loss | training | 0.1207 | 0.1147 | 0.1189 | 0.1175 | 0.1204 |

validation | 0.2288 | 0.2393 | 0.2398 | 0.2184 | 0.2575 | |

test | 0.3703 | 0.3541 | 0.3614 | 0.3600 | 0.3662 | |

(b) ResNet50 | ||||||

Metric | Phase | Original | $\mathcal{C}(16,1757)$ | $\mathcal{C}(18,1063)$ | $\mathcal{C}(20,722)$ | $\mathcal{C}(22,533)$ |

accuracy | training | 0.9926 | 0.9924 | 0.9925 | 0.9923 | 0.9923 |

validation | 0.9310 | 0.9310 | 0.9375 | 0.9304 | 0.9405 | |

test | 0.9288 | 0.9326 | 0.9338 | 0.9300 | 0.9382 | |

loss | training | 0.0236 | 0.0251 | 0.0238 | 0.0246 | 0.0234 |

validation | 0.4501 | 0.4099 | 0.4581 | 0.4493 | 0.3696 | |

test | 0.5039 | 0.5558 | 0.5410 | 0.5478 | 0.5192 |

Base Model | Code | Pruning Rate $(\mathit{R})$ | ||||||||
---|---|---|---|---|---|---|---|---|---|---|

0.91 | 0.92 | 0.93 | 0.94 | 0.95 | 0.96 | 0.97 | 0.98 | 0.99 | ||

VGG-16 | $\mathcal{C}(16,1757)$ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 100 | 100 |

$\mathcal{C}(18,1063)$ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 100 | 100 | |

$\mathcal{C}(20,722)$ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 100 | 100 | |

$\mathcal{C}(22,533)$ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 100 | 100 | |

ResNet-50 | $\mathcal{C}(16,1757)$ | 0 | 0 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |

$\mathcal{C}(18,1063)$ | 0 | 0 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |

$\mathcal{C}(20,722)$ | 0 | 0 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |

$\mathcal{C}(22,533)$ | 0 | 0 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |

Base Model | Code | Pruning Rate $(\mathit{R})$ | ||||||||
---|---|---|---|---|---|---|---|---|---|---|

0.91 | 0.92 | 0.93 | 0.94 | 0.95 | 0.96 | 0.97 | 0.98 | 0.99 | ||

VGG-16 | $\mathcal{C}(16,1757)$ | 0 | 0 | 0 | 0 | 0 | 0 | 10 | 100 | 100 |

$\mathcal{C}(18,1063)$ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 100 | 100 | |

$\mathcal{C}(20,722)$ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 100 | 100 | |

$\mathcal{C}(22,533)$ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 100 | 100 | |

ResNet-50 | $\mathcal{C}(16,1757)$ | 90 | 90 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |

$\mathcal{C}(18,1063)$ | 90 | 90 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |

$\mathcal{C}(20,722)$ | 90 | 70 | 90 | 100 | 100 | 100 | 100 | 100 | 100 | |

$\mathcal{C}(22,533)$ | 70 | 60 | 60 | 100 | 100 | 100 | 100 | 100 | 100 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Yasui, T.; Tanaka, T.; Malik, A.; Kuribayashi, M.
Coded DNN Watermark: Robustness against Pruning Models Using Constant Weight Code. *J. Imaging* **2022**, *8*, 152.
https://doi.org/10.3390/jimaging8060152

**AMA Style**

Yasui T, Tanaka T, Malik A, Kuribayashi M.
Coded DNN Watermark: Robustness against Pruning Models Using Constant Weight Code. *Journal of Imaging*. 2022; 8(6):152.
https://doi.org/10.3390/jimaging8060152

**Chicago/Turabian Style**

Yasui, Tatsuya, Takuro Tanaka, Asad Malik, and Minoru Kuribayashi.
2022. "Coded DNN Watermark: Robustness against Pruning Models Using Constant Weight Code" *Journal of Imaging* 8, no. 6: 152.
https://doi.org/10.3390/jimaging8060152