IW-NeRF: Using Implicit Watermarks to Protect the Copyright of Neural Radiation Fields

Abstract

The neural radiance field (NeRF) has demonstrated significant advancements in computer vision. However, the training process for NeRF models necessitates extensive computational resources and ample training data. In the event of unauthorized usage or theft of the model, substantial losses can be incurred by the copyright holder. To address this concern, we present a novel algorithm that leverages the implicit neural representation (INR) watermarking technique to safeguard NeRF model copyrights. By encoding the watermark information implicitly, we integrate its parameters into the NeRF model’s network using a unique key. Through this key, the copyright owner can extract the embedded watermarks from the NeRF model for ownership verification. To the best of our knowledge, this is the pioneering implementation of INR watermarking for the protection of NeRF model copyrights. Our experimental results substantiate that our approach not only offers robustness and preserves high-quality 3D reconstructions but also ensures the flawless (100%) extraction of watermark content, thereby effectively securing the copyright of the NeRF model.

1. Introduction

Digital watermarking is a copyright protection technology that embeds copyright identifiers into digital media using algorithms [1,2,3]. In the event of copyright disputes, copyright owners can extract copyright information from digital media using the inverse operation of the embedding algorithm to confirm ownership. The concept of neural radiance fields (NeRFs), introduced by Mildenhall et al. [4], utilizes multilayer perceptrons (MLPs) to implicitly represent 3D scenes and render 2D images by synthesizing new perspectives. Due to the NeRF’s robust representational capabilities, high generalization, and ease of learning, it is poised to become a mainstream technique in digital media representation. However, training NeRF models is challenging, and the theft of such models can lead to significant losses for their owners. Therefore, safeguarding NeRF model copyrights is of paramount importance.

Traditional watermarking algorithms [5,6,7] typically rely on specific mathematical functions to embed watermarks into media by altering them. However, these methods often struggle to strike a balance between imperceptibility, robustness, and watermark capacity. The integration of deep learning in watermarking has revolutionized the field, eliminating the need for manually designing complex mathematical functions and showcasing superior performance. In deep learning watermarking, copyright holders encode watermark information into carrier images using an encoder and extract the watermark information from noisy images through a decoder [8], simulating a black-box scenario. While existing deep learning-based watermarking algorithms excel in robustness, imperceptibility, and embedding capacity, they are primarily tailored for multimedia data, like images [9], audio [10], and videos [11], with limited focus on watermarking implicit data such as NeRF.

Uchida et al. [12] pioneered embedding watermarks in deep neural network models, proposing a model watermarking scheme that embeds watermarks in model parameters using a regularization technique. Model watermarking has since evolved rapidly, proving effective in safeguarding network model copyrights. Leveraging the idea of model watermarking, researchers are exploring solutions to protect NeRF copyrights. For instance, Li et al. [13] introduced the StegaNeRF approach for information hiding, while Luo et al. [14] developed the CopyRNeRF scheme, aiming to safeguard NeRF model copyrights while preserving rendering quality and bit accuracy. However, these techniques may suffer from limited watermark capacity when using bit strings or images as watermark information, necessitating comprehensive representation of copyright holders’ logos in embedded watermark data.

To overcome the low watermark capacity and security vulnerabilities in existing NeRF watermarking techniques, we propose an implicit representation-based watermarking algorithm. This method encodes watermark information as continuous functions using implicit representation neurons, embedding these data into carrier networks via a key. By representing watermark information as neural network models instead of bit strings or images, we significantly enhance the watermark capacity. Our approach involves representing watermark information as continuous functions, embedding these parameters into an untrained network using a key, training the carrier network to encapsulate carrier information while sharing the trained watermark-containing carrier model across the network, and enabling copyright owners to extract watermark information using the key if model theft is suspected.

In summary, this work contributes the following:

• The first application of implicit representations in NeRF copyright protection to address existing NeRF watermarking challenges.
• A key-based carrier network construction method for lossless watermark information extraction.
• Validation of our method across various datasets, ensuring model quality and successful watermark extraction.
• Testing the robustness of our model to demonstrate that any attempts to remove the watermark would render the NeRF model unusable.

2. Related Work

2.1. Digital Watermarking for 2D Data

Traditionally, digital watermarking algorithms focused on embedding watermarks into two-dimensional data, categorized into spatial domain-based algorithms and transform domain-based algorithms depending on the embedding domains utilized. In spatial domain algorithms, classic examples include least significant bit (LSB) and patchwork algorithms. LSB algorithms [5,6,7] encode watermark information in the least significant bit of image pixels, while patchwork algorithms. utilize statistical characteristics of the carrier image for encoding. Transform domain-based watermarking algorithms [15,16,17] involve transforming the carrier image into a different domain and embedding the watermark by modifying the transformation domain coefficients based on the watermark value. While traditional watermarking algorithms excel in watermark extraction quality, they often lack robustness and struggle to withstand various attacks. The advancement of deep learning has revolutionized digital watermarking, with significant strides made in image watermarking methods leveraging deep learning techniques.

For instance, Kandi et al. [18] implemented watermark embedding using a convolutional neural network autoencoder, while Zhu et al. [19] introduced an end-to-end watermarking framework named HiDDeN. Building upon these innovations, Zhang et al. [20] delved into attack networks within the universal deep learning watermarking paradigm, enhancing the overall robustness of watermarking algorithms. Notably, deep learning-based watermarking algorithms have witnessed remarkable progress, showcasing their potential for robust and efficient watermark embedding and extraction. However, these existing methods do not cater to safeguarding the copyrights of 3D models effectively.

2.2. Digital Watermarking for 3D Data

Within the realm of 3D digital watermarking, traditional methods involve embedding watermarks in various representations of 3D models, such as point clouds [21], voxels [22], or triangular meshes [23], through transformations like translation, rotation, scaling [24], and parameter modifications [25]. Notably, Hou et al. [26] introduced a technique employing layered artifacts of 3D-printed objects for watermarking, while Hamidi et al. [27] enhanced robustness by leveraging wavelet transform to amplify grid saliency. Recognizing the complexities of traditional 3D digital watermarking approaches, researchers have turned towards integrating deep learning techniques in this domain. Wang et al. [28] pioneered a deep learning-based 3D mesh watermarking network, introducing a more versatile framework for 3D mesh watermarking. Building upon this, Yoo et al. [29] successfully embedded messages within 3D meshes and retrieved them from 2D renderings. However, existing methodologies predominantly cater to explicit 3D models, posing challenges when applying them to neural radiance field (NeRF) models lacking specific 3D structures, thereby limiting their efficacy in ensuring copyright protection for NeRF representations.

2.3. Model Watermarking Algorithm

With the burgeoning advancements in deep learning technology, a plethora of sophisticated deep learning models have surfaced, prompting the development of diverse watermarking strategies tailored to safeguard these models. These approaches can be broadly categorized into two groups: watermark embedding based on network weights and watermark insertion based on the classification labels of trigger sets. Uchida et al. [12] pioneered a method for embedding watermarks through network weights, utilizing a parameter regularizer to infuse watermarks into model parameters. Building on Uchida et al’s work, Wang et al. [30] enhanced this technique by introducing an additional neural network that maps network weights to the watermark space, although this is susceptible to ambiguity attacks. In response to these vulnerabilities, Fan et al. [31] devised a novel passport-based ownership verification scheme for deep neural networks (DNNs), showcasing resilience against network modifications and ambiguity attacks. Rouhani et al. [32] proposed DeepSigns, an end-to-end protective framework empowering developers to embed digital watermarks into pertinent deep learning models prior to dissemination. On the other hand, employing trigger set classification labels for watermark incorporation aims to introduce a backdoor into the network architecture, delivering exclusive activation rights to the copyright verifier. Adi et al. [33] introduced a framework integrating author signatures during DNN training to establish a backdoor-based watermarking mechanism, enabling verification of author identity through predefined signature patterns in watermarked DNNs. Additionally, Shafieinejad et al. [34] proposed a backdoor-inspired approach for watermarking DNNs. While these methodologies are tailored for specific network models and not directly applicable to neural radiance field (NeRF) copyright protection, insights from model watermarking can inform the design of watermark algorithms tailored for NeRF security.

2.4. Watermark Algorithm for Neural Radiation Field

The remarkable performance of neural radiance fields (NeRFs) in 3D reconstruction has garnered significant attention from researchers. Li et al. [13] introduced StegaNeRF, a method for embedding secret information in NeRF rendering. Their approach involved an optimization framework, initially training a standard NeRF model and subsequently conducting additional training to achieve the rendering of secret images while preserving the original visual quality. Similarly, Liu et al. [35] embedded secret messages within the implicit representation function of masked data, enabling direct extraction of these messages via a shared key between the sender and receiver. In contrast, Luo et al. [14] proposed an anti-distortion rendering scheme, replacing the original color representation in NeRF with a watermarked color representation to ensure stable extraction of watermark information in 2D rendered images. Additionally, Chen et al. [36] utilized the newly synthesized view of NeRF as a pivotal element for copyright verification, employing parameterized methods to train a watermark extractor for model validation. Notably, the majority of the aforementioned schemes rely on watermark extractors for copyright protection, which may entail certain security risks. Consequently, the development of an implicit representation watermarking algorithm specifically tailored for NeRF models has been pursued to address these concerns.

3. Preliminaries

The neural radiance field (NeRF) model utilizes an MLP network trained with the input of three-dimensional coordinate positions $x=(x,y,z)$ and spatial point directions $\mathrm{d}=(\theta ,\phi )$, where $\theta$ and $\phi$ represent horizontal and vertical azimuths, respectively. The network outputs the color of spatial points $\mathrm{c}=(r,g,b)$ and the density of corresponding positions (voxels) $\sigma$. In the specific implementation, the position information of x and d is encoded initially. Subsequently, x is fed into the MLP network to produce $\sigma$ and a 256-dimensional intermediate feature. This intermediate feature, along with d, is jointly input into a fully connected layer for color prediction, culminating in the generation of a two-dimensional image through volume rendering. The network structure of the NeRF is depicted in Figure 1.

Despite the high adaptability and strong learning capabilities of fully connected MLP networks, they exhibit notable parameter and structural redundancies. To validate the substantial redundancy within the NeRF model network, an 8-layer MLP network with a hidden layer size of 128 was trained. Following this, parameters of the trained NeRF model were pruned based on very small absolute values. The experimental outcomes, illustrated in Figure 2, demonstrate that even with a 50% parameter trim, the NeRF model retains high reconstruction quality. Hence, leveraging the inherent redundancy in MLP networks is considered in the design of our proposed solution.

4. Proposed Method

4.1. Framework

Our study centers on INR watermarking, utilizing the implicit representation of a NeRF as a case study to elucidate our approach, as depicted in Figure 3.

In our methodology, we denote the initial network conveying watermark information as $F_{\sigma }(·)$, the augmented model network with the watermark as $G_{\theta }(·)$, the watermark information dataset as $H_i,i\in [1,k]$, and the carrier model’s training dataset as $Q_i,i\in [1,k]$. The overall framework of our approach, delineated in Figure 4, is partitioned into three sequential stages. Initially, the content owner trains $H_i$ using $F_{\sigma }(·)$ to derive the watermark information network ${\widehat{F}}_{\sigma }(·)$ and integrates the model parameters of ${\widehat{F}}_{\sigma }(·)$ into $G_{\theta }(·)$ via the key K. Subsequently, the content owner trains $G_{\theta }(·)$ with the carrier dataset to construct a NeRF model ${\widehat{G}}_{\theta }(·)$ encompassing watermarks. In the final phase, during a copyright dispute, the content owner leverages the key K to extract the original parameters of ${\widehat{F}}_{\sigma }(·)$ from ${\widehat{G}}_{\theta }(·)$ for watermark information recovery. The embedding and extraction processes of the watermark information described above can be expressed through Equations (1) and (2):

$$\begin{array}{c}{G}_{\theta }(·)=Emb({\widehat{F}}_{\sigma }(·),K)\hfill \end{array}$$

(1)

$$\begin{array}{c}{\widehat{F}}_{\sigma }(·)=Ext({\widehat{G}}_{\theta }(·),K)\hfill \end{array}$$

(2)

4.2. Data Representation and Transformation

In the initial phase, we depict the watermark information data using implicit neural representations. The structure of the image noise-resilient (INR) model is a multilayer perceptron (MLP) network comprising n hidden layers of size D, employing the activation function $\sigma (·)$. The output of the INR model can be formulated as shown in Equation (3):

$$\begin{array}{c}\begin{array}{c}y=W^{\left(n\right)}(g_{n-1}o\cdots o{g}_{1}o{g}_0)\left(h_0\right)+b^{\left(n\right)},\hfill \\ \mathrm{where}{h}_{i+1}=g_i\left(h_i\right)=\sigma (W^{\left(i\right)}{h}_{\mathrm{i}}+b^{\left(i\right)})\hfill \end{array}\hfill \end{array}$$

(3)

Here, $h_i,i\in \{0,1,2,\dots ,n\}$ represents the input of the i-th layer, and y denotes the corresponding output value of $h_0$. Additionally, $g_i$ signifies the hidden layer, $W^{\left(i\right)}$ represents the weight matrix of the i-th layer, and $b^{\left(i\right)}$ denotes the bias parameter of the i-th layer. The INR network architecture utilizes a 10-layer MLP network structure with the ReLU activation function. In the context of a given input H, the output set M corresponding to Y is associated with a loss function as in Equation (4):

$$\begin{array}{c}{L}_w=min{\displaystyle \sum _{(h,y)\in M}}{∥F_{\sigma }\left(h\right)-y∥}_2^2\hfill \end{array}$$

(4)

Here, $F_{\sigma }(·)$ denotes the neural representation of the data. For instance, in the context of the NeRF model and a given set of images from diverse perspectives $H_i$, the set M comprises three-dimensional coordinates, camera pose $h=(x,y,z,\theta ,\phi )$, corresponding RGB values, and opacity $y=(r,g,b,\sigma )$.

4.3. Watermark Information Embedding Stage

This article is influenced by advancements in neural network watermarking, focusing on embedding watermark representations within carrier networks. The methodology leverages network unfolding techniques for watermark embedding, ensuring the preservation of watermark network parameters within a newly constructed neural network structure. By adopting implicit representations, especially through straightforward network architectures like MLP, embedding watermark parameters using a designated key is achieved without compromising performance. The new network design is centered on the watermark network, with inspiration drawn from models like the NeRF. Three key network deployment strategies are introduced, depicted in Figure 5, to optimize the embedding process efficiently while maintaining the integrity of the watermark information within the carrier network, as follows:

(1) Horizontal expansion. This strategy entails the insertion of new layers subsequent to the watermark network, preserving the original network structure, to construct a larger network, as illustrated in Figure 5a. (2) Vertical expansion. An alternative approach to expanding the network is to maintain the layer count unchanged. Given that both the watermark and extended networks embody the NeRF model, aside from the input and output layers, we augment solely the count of neurons within the hidden layers, as depicted in Figure 5b. (3) Mixed expansion. Mixed expansion encompasses both horizontal and vertical expansion, concurrently increasing the layer count and the neuron count within existing hidden layers. This constitutes a versatile operation, as demonstrated in Figure 5c, where the entirety of the watermark network’s content and parameters are encapsulated within a single network. Indeed, both horizontal and vertical expansion can be encapsulated under the umbrella of mixed expansion, and our objective is to construct the carrier network’s structure using expansion methodologies that are as simplistic as is feasible.

Employing watermark networks and carrier networks to encapsulate watermark information and NeRF models, we naturally leverage mixed dilation to construct carrier networks. Within implicitly represented 3D scenes, upon the training of the watermark network, we can assemble a carrier network through the watermark network. Due to the necessity of embedding the parameters of the trained watermark network within the carrier network, we designate a shared key K, comprising a randomized binary sequence of zeros and ones, as presented in Figure 2. Within our scheme, the key is defined as $K=\{k_0,k_1,k_2,\dots ,k_m\}$, wherein $k_0$ represents the count of layers of the watermark network neurons within the carrier network, with a binary value of 1 indicating that the layer includes neurons of the watermark network and 0 indicating otherwise. The sequence $\{k_1,k_2,\dots ,k_m\}$ denotes the location of the watermark network neurons within the carrier network, where each layer corresponds to a binary stream $k_m$ of length $d_{k_m}$, specifying the neuron count in that layer. Each bit within this sequence represents a neuron, with 1 indicating it belongs to the watermark information and 0 indicating it belongs to the carrier information. For example, in Figure 5c, the corresponding key K is $\{01110,00110,101101,01101\}$. This method significantly reduces the length of the key through $k_0$.

4.4. Training of Carrier Networks

We utilized Equation (4) to enable the implicit representation of the carrier NeRF model through the carrier network $G_{\theta }(·)$, where $\theta$ encompasses two distinct parameter categories: those of the watermark network and others representing carrier information. In ensuring the lossless extraction of ${\widehat{F}}_{\sigma }(·)$ from ${\widehat{G}}_{\theta }(·)$, it becomes imperative to immobilize the parameters associated with the watermark information. Subsequently, training is exclusively directed towards the parameters embodying carrier information, thereby optimizing ${\widehat{F}}_{\sigma }(·)$ with precision. In pursuit of this objective, we introduced a binary mask M to facilitate selective optimization of $G_{\theta }(·)$ in Equation (5):

$$\begin{array}{c}M\left[p\right]=\left\{\begin{array}{c}1,\theta \left[p\right]\in \phi \hfill \\ 0,else\hfill \end{array}\right.\hfill \end{array}$$

(5)

Among them, $\phi$ represents the parameter representing carrier information in $\theta$, and $\theta \left[p\right]$ represents the p-th parameter in $\theta$. Similar to Equation (6), we use the image set $Q_i$ to train the carrier network and define the loss of the carrier network as $L_c$. The formula for $L_c$ is as follows:

$$\begin{array}{c}{L}_c=min{\displaystyle \sum _{(q,y)\in M}}{∥G_{\theta }\left(q\right)-y∥}_2^2\hfill \end{array}$$

(6)

Let $\lambda$ be the learning rate and ⊙ be the product of elements, and update $G_{\theta }(·)$ using the gradient descent method, as shown in Equation (7).

$$\begin{array}{c}\theta =\theta -\lambda M\odot {\nabla }_{\theta }{L}_c\hfill \end{array}$$

(7)

Through training, we obtained the NeRF model ${\widehat{G}}_{\theta }(·)$ containing watermarks.

4.5. Watermark Information Extraction

After releasing model ${\widehat{G}}_{\theta }(·)$ online, regular users can engage with and explore 3D environments. Maintaining the watermark information parameters unaltered during the training phase of the carrier network $G_{\theta }(·)$ ensures that the party responsible for copyright protection can restore the watermark information network ${\widehat{F}}_{\sigma }(·)$ without any loss postacquisition of the watermark network parameter details utilizing the key K. Leveraging key K, we not only ascertain the layers within ${\widehat{F}}_{\sigma }(·)$ housing watermark information but also identify the neurons containing watermark information along with their pertinent parameter specifics. In essence, through the parameters $P_{{\widehat{F}}_{\sigma }(·)}$ and structure $S_{{\widehat{F}}_{\sigma }(·)}$ in key K, the watermark network can be regenerated. The overall process of our method is shown in Algorithm 1.

Algorithm 1 Training process of IW-NeRF.

1: Data: Watermark information dataset $H_i,i\in [1,k]$, Carrier model training dataset $Q_i,i\in [1,k]$, Random key K, learning rate $\eta$   2: Output: Watermark information network model ${\widehat{F}}_{\sigma }(·)$ Initial network containing watermark information $G_{\theta }(·)$ Watermarked NeRF model ${\widehat{G}}_{\theta }(·)$. Optimizing the parameter $\theta$ of model $G_{\theta }(·)$ through $Q_i$ Compute mask M for $\theta$ as in Equation (5)   3: for each training iteration t do     Compute Watermark information loss as Equation (4)     Compute Carrier network information loss as Equation (6)     Update $\theta$ as Equation (7)   4: end for

5. Experiments

5.1. Experimental Settings

Dataset. We assessed our algorithm using the NeRF Semantic and LLFF datasets sourced from the NeRF dataset. The LLFF dataset encompasses diverse scenes such as flowers, ferns, fortresses, and rooms, while the NeRF Semantic dataset features 360-degree scenes like LEGO structures, drums, and chairs. For evaluating the efficacy of our approach within the NeRF Semantic dataset, akin to a NeRF [1], our training regimen involves feeding in 100 views per scene. To gauge the visual fidelity of our methodology, we handpicked 20 images per scene from the test dataset. Furthermore, we conducted renders of 200 views for each scene to validate the watermark extraction precision across varying camera angles. Our comprehensive experimental methodology entails presenting all outcomes as averaged results for clarity and consistency.

Implicit neural representation setting. Our approach was implemented utilizing PyTorch, and both the watermark information and carrier network were structured employing an MLP network design. We employed 12 hidden layers of size 128 to encode the watermark information and 22 hidden layers of size 256 for the carrier network. The hyperparameters used include a learning rate of $5\times {10}^{-4}$, a batch size of 512, utilization of the Adam optimizer with default values of ${\beta }_1=0.9$, ${\beta }_2=0.999$, and regularization parameter $\lambda =1\times {10}^{-8}$. The experiments were conducted on an NVIDIA A100 GPU, with training of both the watermark network and carrier network executed using a stochastic gradient descent algorithm over 20,000 epochs.

Baselines. To the best of our knowledge, there exists limited research on watermarking specifically tailored for NeRF applications. Thus, we conducted a comparative analysis with established watermarking techniques adapted for NeRF to ensure a comprehensive evaluation: (1) LSB [6] + NeRF: here, we employed the LSB algorithm to embed watermark data into the dataset images before NeRF model training; (2) DeepStega [37] + NeRF: preceding NeRF model training, we leveraged the two-dimensional watermarking approach of DeepStega for image processing; (3) HiDDeN [19] + NeRF: prior to NeRF model training, the image underwent processing utilizing the HiDDeN scheme; (4) StegaNeRF [13]; and (5) CopyNeRF [14].

Distortion evaluation metric. In order to evaluate the distortion between the carrier data and the carrier data containing watermarks, as well as the distortion between the watermark data and the extracted watermark data, we visualized the implicitly represented data and used several evaluation metrics. For 2D images rendered from 3D models, we evaluate them using peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and perceived loss (LPIPS).

(1)

PSNR (peak signal-to-noise ratio).

PSNR is an indicator used to evaluate image quality based on the concept of root mean square error (MSE), which represents the peak signal-to-noise ratio of image signals. The higher the PSNR value, the better the quality of the evaluated image and the smaller the error. The calculation methods for MSE and PSNR are shown in Equations (8) and (9), respectively.

$$\begin{array}{c}MSE={\displaystyle \frac{1}{W\times H}}{\displaystyle \sum _{i=1}^W}{{\displaystyle \sum _{j=1}^H}(X_{i,j}-Y_{i,j})}^2\hfill \end{array}$$

(8)

$$\begin{array}{c}PSNR=20\times \lg \left(sc\right)-10\times \lg \left(\mathrm{MSE}\right)\hfill \end{array}$$

(9)

In the formulas, X and Y represent two images with a size of W × H, respectively, and sc represents the scaling factor, usually taken as 2.

(2)

SSIM (structural similarity).

SSIM is a metric that evaluates image quality by measuring the similarity between two images, with higher values indicating higher similarity. The calculation method of SSIM is shown in Equation (10).

$$\begin{array}{c}SSIM(x,y)={\displaystyle \frac{(2{\mu }_x{\mu }_y+c_1)(2{\sigma }_{xy}+c_2)}{({{\mu }_x}^2+{{\mu }_y}^2+c_1)({{\sigma }_x}^2+{{\sigma }_y}^2+c_2)}}\hfill \end{array}$$

(10)

In the above equation, ${\mu }_x$ and ${\sigma }_x$ are the mean and variance of image X, ${\mu }_y$ and ${\sigma }_y$ are the mean and variance of image Y, and ${\sigma }_{xy}$ is the covariance of X and Y. $c_1={({\mathrm{k}}_{1}L)}^2$, $c_2={({\mathrm{k}}_{2}L)}^2$ is a constant, ${\mathrm{k}}_1=0.01$, ${\mathrm{k}}_2=0.03$, L is the dynamic range of pixel values, and if the data are uint8-type, L takes a value of 255.

(3)

LPIPS (learned perceptual image patch similarity).

LPIPS is a method of measuring image similarity, which does not use mathematical formulas to implement but evaluates the perceptual differences between two images through deep learning models. The lower the value of LPIPS, the more similar the two images are; and the higher the value, the greater the difference. We use a pretrained LPIPS model Alex for evaluation.

5.2. Reconstruction and Watermark Extraction Quality

We conducted a qualitative comparison of the reconstruction quality against all baselines, with the results depicted in Figure 6. The visual analysis from Figure 6 indicates that all methods exhibit commendable reconstruction quality. While other schemes may outperform ours in reconstruction quality due to their limited capacity stemming from text or image information embedding, our approach stands out in terms of embedded capacity. Despite the reduction in reconstruction quality caused by the embedding of watermark network parameters, likely attributed to fixed parameters impacting the carrier network’s fitting effect, the overall NeRF model information remains retrievable visually. Notably, our designed key facilitates lossless recovery of watermark information from a carrier network containing watermarks, offering a significantly larger embedded capacity compared to alternative schemes. For a quantitative assessment, we further analyzed the rendering quality and watermark information extraction effects across various schemes, as detailed in

Table 1. Quantitative analysis of reconstruction quality and watermark extraction effect of NeRF models under different baselines (Synthetic-chair).

Method	NeRF Rendering			Watermark Extraction
	PSNR ↑	SSIM ↑	LPIPS ↓	Acc (%) ↑	SSIM ↑
Standard NeRF	33.23	0.9143	0.1113	N/A	N/A
LSB + NeRF	27.45	0.8446	0.1410	N/A	N/A
DeepStega + NeRF	26.41	0.8457	0.1429	N/A	N/A
HiDDeN + NeRF	27.88	0.8964	0.1418	N/A	N/A
StegaNeRF	30.31	0.9847	0.0276	100	0.9643
CopyRNeRF	30.54	0.9689	0.0327	100	N/A
IW-NeRF (ours)	21.32	0.6364	0.2852	100	1

and

Table 2. Quantitative analysis of reconstruction quality and watermark extraction effect of NeRF models under different baselines (LLFF-trex).

Method	NeRF Rendering			Watermark Extraction
	PSNR ↑	SSIM ↑	LPIPS ↓	Acc (%) ↑	SSIM ↑
Standard NeRF	27.76	0.8546	0.1453	N/A	N/A
LSB + NeRF	27.59	0.8435	0.1399	N/A	N/A
DeepStega + NeRF	26.98	0.8356	0.1269	N/A	N/A
HiDDeN + NeRF	27.64	0.8865	0.1512	N/A	N/A
StegaNeRF	28.21	0.8453	0.1423	100	0.9698
CopyRNeRF	30.67	0.9683	0.0457	100	N/A
IW-NeRF (ours)	22.54	0.6539	0.2798	100	1

.

By examining the outcomes outlined in Table 1 and Table 2, a notable observation is the ineffectiveness of the two-dimensional watermarking techniques applied to the NeRF model in retrieving watermark information. The alteration of information embedded via the two-dimensional watermarking method due to NeRF’s view synthesis led to the method’s failure. Conversely, the utilization of three-dimensional watermarking approaches facilitated accurate extraction of watermark information. Notably, while other algorithms incorporate bit strings into the model, our methodology visualizes the 3D reconstruction and watermark extraction effects. Our experimentation involved utilizing diverse 360-degree scenes such as ‘Lego’, ‘drums’, and ‘chairs’ from the NeRF Semantic dataset for training the watermark network and carrier network. The resulting experimental findings, as illustrated in Figure 7, showcase various images representing original watermark information, carrier information, samples from both the watermark and carrier networks, and extracted watermark network samples. Despite our method slightly affecting the NeRF model’s training performance, it demonstrates the ability to extract embedded watermark information from the carrier network without loss. Noteworthy is the incorporation of a neural network capable of implicitly representing a range of copyright materials (e.g., images, sounds, videos) in real-world scenarios.

5.3. Algorithm Capacity

Our approach involves embedding the parameters of a watermark network within a carrier model network, allowing for the assessment of algorithm capacity through scalability analysis. The expansion rate, as defined in Equation (11), serves as a metric to quantify this scalability.

$$\begin{array}{c}e={\displaystyle \frac{N_{carrier}}{N_{watermark}}}-1\hfill \end{array}$$

(11)

In this context, $N_{watermark}$ and $N_{\mathrm{carrier}}$ represent the parameter counts within the original watermark network and carrier network, respectively. Consequently, with $e⩾0$, the expansion rate is intricately linked to the sizes of the watermark network and carrier network. A lower expansion rate correlates with higher capacity. To assess the influence of watermark capacity on the outcomes of our proposed scheme, we executed the subsequent pair of experiments.

Watermark network layer modification. In our approach, the utilization of INR to represent watermark information enables a smaller watermark neural network to achieve superior carrier data reconstruction during the fitting process. By adjusting the size of the watermark network, we can influence the performance of the carrier network fitting. Specifically, we maintained a fixed size of 22 hidden layers for the carrier network and employed hidden layer networks with 16, 14, 12, 10, and 8 layers for training the watermark information. The hidden layer width for the carrier network was set at 256, while for the watermark network, it was set at 128. Our experiments were carried out using the LLFF dataset. Figure 8 illustrates the performance of carrier models trained with varying watermark networks, showcasing rendered images by both the carrier and watermark networks in the respective columns. Furthermore, Figure 9 displays the SSIM values between the rendered and real images of the carrier network across different watermark network levels.

Table 3. After modifying the number of layers in the watermark network, we download the PSNR value between the rendered image and the real image of the network with different expansion rates.

Expansion Rate	1.75	2.14	2.66	3.40	4.50
PSNR	19.88	21.67	22.99	23.41	25.86

depicts the impact of the expansion rate on the PSNR values between the rendered and real images of the carrier network. Notably, our experimental results highlight that with a consistent size for the carrier network, reducing the number of layers in the watermark network leads to an increase in the expansion rate and the complexity of the carrier network, consequently enhancing the quality of the carrier model.

Carrier network layer modification. To further optimize 3D reconstruction outcomes, adjustments to both the watermark network size and the carrier network dimensions were explored. The watermark network was kept at a fixed size of 10 layers, while carrier networks ranging from 18 to 26 layers were utilized in training the carrier model. The hidden layer width was set to 128 for the watermark network and 256 for the carrier network. PSNR values illustrating the comparison between images rendered by the carrier model trained on different layers of the carrier network and the original images is depicted in Figure 10. Additionally, Figure 11 showcases SSIM values representing the comparison between the rendered image and the actual image of the carrier network across varying layers. The impact of the expansion rate on the PSNR values between the rendered and real images of the carrier network is detailed in

Table 4. After modifying the number of layers in the carrier network, we download the PSNR value between the rendered image and the real image of the body network with different expansion rates.

Expansion Rate	2.60	3.00	3.40	3.80	4.20
PSNR	20.94	21.58	23.21	24.34	25.66

. The experimental findings reveal a positive correlation between the depth of the carrier network and the expansion rate, leading to an enhancement in the reconstruction quality of the carrier model. However, it is worth noting that an increase in the number of layers of the carrier network introduces higher computational complexity and training challenges, potentially reducing training efficiency. For future research directions, the adoption of multiresolution hash encoding [38] is under consideration to boost training speed.

5.4. The Robustness of the Model

In practical scenarios, the vulnerability of our model to malicious attacks necessitates an assessment of the robustness of our watermarking scheme against various model alterations. To evaluate the resilience of our algorithm, we employed two common modifications. Model pruning is a vital technique, especially for MLP networks known for their robust fitting capabilities but often burdened with numerous parameters due to deep network layers and high neuron counts. The abundance of parameters not only escalates computational demands in deep learning but also presents an opportunity for optimization through pruning. The primary objective of network pruning is to reduce redundant parameters while preserving the original network’s performance. Our experiments encompassed $L_1$ unstructured pruning, $L_n$ structured pruning, random structured pruning, and random unstructured pruning methods. During the pruning process, the parameter with the minimum absolute pruning rate p% (ranging from 10% to 90%) was set to 0 for the NeRF model embedded with watermarks. Subsequently, we compared the extracted watermark information postpruning with the prepruning data to assess its impact on our watermark framework. Ideally, a model thief aims to render the extracted watermark information inaccurate postpruning while retaining the model’s performance. Figure 12 and Figure 13 illustrate the influence of pruning rates on watermark extraction performance under various pruning methodologies. Notably, even with $L_1$ unstructured pruning removing 90% of parameters, our embedding method demonstrated high accuracy in watermark extraction (with an SSIM value of 0.8829 between the extracted and original watermark information). Conversely, $L_n$ structured pruning, random structured pruning, and random unstructured pruning methods exhibited a significant reduction in extracted watermark quality beyond a 10% pruning rate. However, while these pruning methods may compromise watermark quality, they also detrimentally impact the model’s performance, rendering the stolen model ineffective. Thus, our watermark algorithm exhibits robustness against pruning modifications, ensuring that any attempts to maintain the model’s quality post-theft come at the cost of significantly reduced model performance.

Model fine-tuning is a crucial aspect in the context of large-scale NeRF models. Training such models from scratch demands a substantial training dataset, and the lack of adequate data can impact model performance. Hence, in practical scenarios, fine-tuning existing models becomes a more feasible approach when data scarcity is a concern. Typically, fine-tuning is preferred when there is minimal variance in experimental performance between the original dataset and the dataset trained using a pretrained model. Consequently, for potential plagiarists, employing fine-tuning methods to train a new model based on the stolen model with limited training data becomes a viable strategy. This new model not only mirrors the performance of the original model but also diverges in experimental outcomes from the original model. In our experimental setup, each dataset underwent a partitioning of the test data into two segments: the first half facilitated the fine-tuning of the trained NeRF model, while the second half served to evaluate the performance of the newly derived model. Subsequently, we utilized the fine-tuned model to assess the resilience of our watermark scheme against modifications induced by the fine-tuning process.

Table 5. The watermark extraction effect on different datasets after model fine-tuning.

Dataset	Hot Dog	Lego	Fern	Flower
PSNR (dB)	46.34	45.28	44.57	46.13
SSIM	0.9843	0.9789	0.9648	0.9881

presents the PSNR and SSIM metrics comparing the original watermark information with the extracted watermark information post-model fine-tuning. The experimental findings indicate that fine-tuning has a minimal impact on watermark accuracy, regardless of whether synthetic or real datasets are utilized. This resilience to fine-tuning-induced modifications can be attributed to the presence of numerous redundant neurons within the MLP network, enhancing the robustness of our watermark algorithm against such adjustments.

5.5. Security Assessment

Contrary to discrete representations, implicit neural representations introduce uncertainty, leading to varied weight distributions for each dataset across rendering and watermark extraction processes. To assess the algorithm’s security, we conducted a comprehensive statistical analysis and visual examination of the weight distributions within the watermark network model and the carrier network post-watermark embedding. The watermark network comprised 10 layers, while the carrier network featured 22 layers. By comparing the corresponding layers’ weights in both networks, we observed distinct differences. Visualization techniques were employed to juxtapose the weights of networks with equivalent layer numbers in the watermark and carrier networks, depicted in Figure 14. The disparity in weight distributions between the watermark and carrier networks was evident, signifying the successful incorporation of watermark network parameters into the carrier network.

Within our watermarking framework, the pivotal role of the key in extracting watermark information underscores the need for a comprehensive evaluation of key sensitivity. Analogous to cryptographic protocols, the correct key serves as the sole means through which copyright stakeholders can access the embedded watermark data. In contrast to traditional cryptography, the performance of neural networks in extracting watermarks is subject to various influencing factors, including network topology, training datasets, parameter configurations, and optimization strategies, which may introduce performance variability. From an experimental standpoint, the fundamental criterion lies in the successful extraction of the watermark using the correct key, as even minor deviations in the key could result in extraction failures. Figure 15 showcases the average peak signal-to-noise ratio (PSNR) values between the original watermark and the extracted watermark across diverse datasets, highlighting the impact of varying error bits within the 2180-sized key. Additionally, Figure 16 visually demonstrates the consequences of erroneous key bits on the output quality of the extracted watermark network, showcasing a rapid decline in PSNR values as the number of incorrect key bits escalates, leading to perceptible image blurring. This observation underscores the sensitivity of the watermark extraction process to even minor alterations in the key. For individuals lacking direct access to the key, resorting to guesswork may introduce errors, with approximately 50% of incorrect key bits resulting in failed watermark extraction attempts. Future endeavors will focus on refining key generation methodologies to bolster the security of the watermarking scheme.

6. Limitations

The utilization of a network model, as opposed to a conventional bit string or image, in our watermarking scheme introduces a trade-off in the quality of the NeRF model. Our embedding and extraction processes operate akin to a white-box model, necessitating access to the NeRF model’s parameters for successful watermark extraction. This requirement may constrain the practical applicability of our algorithm, particularly in scenarios where parameter accessibility is restricted. Moving forward, our research endeavors will focus on enhancing the watermark embedding methodology to refine the overall efficiency and robustness of our watermarking scheme.

7. Conclusions

In light of the rapid advancements and diverse applications of NeRF technology, the preservation of copyright for NeRF models has emerged as a critical imperative. This study introduces a novel NeRF model watermarking scheme known as implicit neural representation watermarking for NeRF (IW-NeRF). Within this framework, the watermark information is intricately embedded into the NeRF model through implicit neural representations, ensuring a seamless and robust integration of copyright protection mechanisms within the model architecture. Subsequently, the extraction of the watermark information is facilitated through a designated key, effectively fortifying the copyright protection framework for NeRF models.

Advantages. The experimental findings demonstrate the distinct advantages of our proposed watermark scheme: (1) Security: Within our scheme, the watermark information is implicitly represented and then randomly embedded into the NeRF model using a key derived from network parameters. The extensive key space, stemming from the multitude of network parameters, ensures the security of our approach. (2) Capacity: By characterizing the watermark information implicitly through neural methods and incorporating it into a continuous function, the memory required for signal parameterization becomes independent of spatial resolution, solely influenced by the signal’s underlying complexity. This strategy enables the utilization of continuous functions for achieving high-capacity watermarks. (3) Robustness: Neural networks exhibit inherent redundancy in structure and neuron count. By embedding the watermark information within the NeRF model network, our scheme maintains resilience against minor alterations to the model; any significant attacks on the model would consequently impact its performance rather than compromise the embedded watermark. (4) Universality: Implicit neural representations possess the capability to depict diverse data forms. While our experimental focus entailed the utilization of the NeRF model as watermark information, the watermark network framework proposed in our study extends to various media types such as images and videos. Noteworthy is the pioneering introduction of implicit neural representation within the realm of digital watermarking in this research.