Robust Template-Based Watermarking for DIBR 3D Images

Abstract

Several depth image-based rendering (DIBR) watermarking methods have been proposed, but they have various drawbacks, such as non-blindness, low imperceptibility and vulnerability to signal or geometric distortion. This paper proposes a template-based DIBR watermarking method that overcomes the drawbacks of previous methods. The proposed method exploits two properties to resist DIBR attacks: the pixel is only moved horizontally by DIBR, and the smaller block is not distorted by DIBR. The one-dimensional (1D) discrete cosine transform (DCT) and curvelet domains are adopted to utilize these two properties. A template is inserted in the curvelet domain to restore the synchronization error caused by geometric distortion. A watermark is inserted in the 1D DCT domain to insert and detect a message from the DIBR image. Experimental results of the proposed method show high imperceptibility and robustness to various attacks, such as signal and geometric distortions. The proposed method is also robust to DIBR distortion and DIBR configuration adjustment, such as depth image preprocessing and baseline distance adjustment.

1. Introduction

Three-dimensional (3D) content has been steadily increasing in popularity because of its excellent lifelike appearance. With the recent development of 3D display technology, many new 3D applications have appeared to maximize realism, such as head-mounted displays, 360° virtual reality and ultra-high definition 3D content. Consequently, interest in 3D content and in the 3D market itself have both increased greatly.

Methods of representing 3D content are divided into stereo image recording (SIR) and depth-image-based rendering (DIBR). SIR stores the left and right views (as human eyes do) and provides a high-quality immersive view; however, this has many limitations, including a large data size, fixed depth, high cost and difficulty with multiple camera settings. Meanwhile, DIBR is a rendering method that creates various virtual viewpoint images using center and depth images [1,2,3,4]. The DIBR method has two main advantages: (1) the DIBR system is able to easily save and transmit 3D content, because it requires less data compared to SIR; (2) the DIBR system can provide various viewpoints, since it allows us to adjust the 3D configuration. These advantages have led to DIBR technology being employed in 2D-to-3D conversion [5,6,7,8,9,10] and auto-stereo [11,12] and multi-view stereo [13,14,15,16,17], which provide various viewpoints depending on the user’s position

Copyright protection techniques for DIBR content have received considerable attention due to DBIR’s important role and significant 3D content market growth. A typical copyright protection technique is watermarking, but many conventional two-dimensional (2D) watermarking techniques are difficult to apply to DIBR content. In the DIBR system, center image pixels are partially moved along the horizontal axis with distance depending on the depth image, using a process called non-linear geometric distortion. Watermarks inserted in the center image are strongly distorted and cannot be extracted.

Hence, several watermarking methods have been proposed as being robust to the DIBR process. Lin et al. proposed a method of embedding watermarks by predicting pixels’ moving distance [18]. Protecting the center image and both the left and right images required superimposing and embedding three watermarks. This method has a low bit error rate (BER) against the DIBR process and common distortions, such as JPEG or additive noise. However, this method is vulnerable when the depth information is modified, such as in depth image preprocessing or the change of baseline distance, since the moving distance for pixels during the DIBR process is predicted with unmodified depth information. Additionally, this method is vulnerable to geometric attacks due to the characteristics of the discrete cosine transform (DCT) domain.

Franco-Contreras et al. proposed a method of averaging the luminance of an image in the horizontal direction and then inserting a watermark in the averaged vector [19]. This method is robust against DIBR and signal processing attacks, but has the disadvantage of not considering geometric transformations.

Kim et al. suggested a watermarking method that employed quantization on dual tree complex wavelet transform (DT-CWT) domain coefficients [20]. The method used directional coefficients that were not significantly changed by the DIBR process. Row-wise quantization was performed to provide robustness to horizontal pixel shifts. The method showed robustness to DIBR; JPEG compression; image scaling; and DIBR configuration adjustments, such as depth image preprocessing and baseline distance changes. However, it was vulnerable to noise addition and geometric distortions.

Wang et al. [21], Miao et al. [22] and Cui et al. [23] proposed a DIBR watermarking method that used the scale-invariant feature transform (SIFT). The SIFT based-watermarking systems found similar parts between the center image and the synthesized image using a SIFT descriptor and then inserted the watermark into those parts. Because of the object matching with SIFT, these methods were robust to DIBR distortion. Additionally, the methods showed high robustness to both general signal distortions and geometric distortions. However, SIFT-based DIBR watermarking methods need SIFT descriptors during watermark extraction. These methods cannot extract watermarks blindly; therefore, SIFT-based watermarking methods are less practical than full-blind watermarking systems.

Rana et al. defined a dependent view region, which is a common area of the left and right eye images, and inserted a watermark using the DC coefficients of the DCT in the dependent view region [24]. This method achieved robustness against DIBR, JPEG compression and various noise attacks. Rana et al. also proposed a 3D video watermarking technique [25]. This method exploits the shift invariance and the directional property of the 2D DT-DWT to insert a watermark. This technique showed high invisibility and high robustness against 3D-high efficiency video coding compression, as well as DIBR attacks. However, neither of these methods consider geometric distortions.

Asikuzzaman et al. proposed a DT-CWT-based video watermarking method using color channels [26,27]. The method inserted a watermark into the U channel of the YUV color space and inserted the same watermark rotated 180° into the V channel. They showed that the method was robust to DIBR, due to the DT-CWT domain characteristics; and geometric attacks, such as scaling and rotation, since the U and V channels suffer the same geometric transformation. However, if the image center were changed due to attacks such as crop or translation, the watermark could not be detected, and under geometric distortion, it could only determine whether a watermark had been inserted or not, i.e., an on/off switch. Therefore, its application was somewhat limited.

Various templates have been proposed that are robust against geometric attacks [28,29,30,31]. However, these are only designed to be robust to linear distortions, such as affine transforms, and so are not robust to DIBR, which is a non-linear distortion, as discussed above.

The results of previous works show that blind watermarks have great difficulty surviving in geometrically-distorted DIBR images, because the watermark must be robust to both DIBR and geometric attacks, as shown in Figure 1. The combination of non-linear and linear deformation severely damages most watermarking domains. Therefore, it is desirable to solve this problem by combining two watermarking methods that have different characteristics, rather than using a single watermarking method.

This paper proposes a blind template-based watermarking system combining templates and message watermarks. The role of the template is to restore geometric distortion without being destroyed by the DIBR attack. The template is inserted into the curvelet domain in the form of peak points, and geometric distortion is estimated using the modified iterative closest point (ICP) method. The message watermark inserts and detects messages in the DIBR image. The proposed message watermark is inserted into the 1D-DCT domain, and the message is extracted from the DIBR image after geometric distortion recovery using the proposed template. The message watermark inserts the same information along the horizontal direction in the 1D-DCT domain, and the inserted message watermark is invariant to DIBR due to 1D-DCT linearity.

Experimental results showed that the proposed method has both high invisibility and robustness against various attacks. It achieved excellent scores in visual quality tests and low BER for common signal distortions, such as noise addition and JPEG compression. The watermarking system also exhibited good robustness against geometric distortions, a point of vulnerability in previous approaches, and excellent robustness against DIBR attacks and DIBR configuration adjustments.

This paper is organized as follows. Section 2 describes the DIBR system and curvelet transform, which are fundamental techniques for the proposed scheme. Section 3.1 demonstrates the main idea of the proposed watermarking method. The watermark embedding and extraction processes are presented in Section 3.2 and Section 3.3, respectively, and the experimental results and conclusion are given in Section 4 and Section 5, respectively.

2. Background

This section presents DIBR and the curvelet transform, fundamental techniques of the proposed watermarking system. First, the DIBR process is briefly introduced and DIBR analysis presented, and then, we provide an introduction to and analysis of the curvelet transform.

2.1. DIBR Process

The whole DIBR process is shown in Figure 2. DIBR consists of three steps: depth image preprocessing, pixel location warping and hole-filling.

Depth image preprocessing, the first step, improves the quality of the rendered image by reducing the number of holes [32,33,34]. When the viewpoint is moved by the DIBR, an area where no pixel information exists is generated. These areas, referred to as holes, are the main cause of 3D image quality degradation. Holes occur mainly when the depth difference between two adjacent pixels is large. Hence, the image quality can be improved by reducing the number of holes through depth image smoothing.

Pixel location warping, the second step, changes the position of pixels along the horizontal direction, allowing users to feel a 3D effect. The warping equation is as follows,

$$\begin{array}{c}\hfill x_L=x_C+\left(\frac{t_x}{2}\times \frac{f}{Z}\right),x_R=x_C-\left(\frac{t_x}{2}\times \frac{f}{Z}\right)\end{array}$$

(1)

where $x_C$ is the pixel position on the x-axis of the center image, $x_L$ and $x_R$ are the pixel positions on the x-axis of the left view and the right view, respectively, $t_x$ is the baseline distance, which means the distance from the center axis to the left and right, f is the focal length and Z is the value of the depth image. During warping, two or more pixels can overlap in one position. In this situation, the highest Z value of a pixel has to be selected to prevent an unnatural image.

The last step is hole-filling, which creates pixel values in holes caused by pixel location warping. There are several hole-filling techniques, such as interpolation and inpainting. This is a field that is constantly studied in pursuit of a more natural image [4,35,36].

2.2. Analysis of DIBR Attack

In the DIBR process, pixels are only translated horizontally, where the translation magnitude is determined by the depth. Similar to the cover model being considered as random, the depth image is also close to a random signal [37]; hence, the pixel’s moving distance can also be assumed to be random. Consequently, the pixels move irregularly, unlike in common translation. Thus, the 2D transformed domain coefficients are distorted by DIBR.

The watermark damage caused by DIBR can be confirmed by the average energy change of the middle frequency at which the watermark is inserted, since the average energy is the basis of the watermark embedding energy.

Table 1. Average energy change between coefficients of center and synthesized images.

		2D DCT	2D DFT (Mag)	2D DWT (Haar)
Average	Unmatched	209%	39%	215%
Energy change	Matched	207%	40%	4%

shows the average energy change between center and synthesized DIBR image coefficients. DIBR parameters used in this test were as recommended by [3], and average energy change was defined as:

$$\begin{array}{c}\hfill \frac{MSE(O,S)}{\mathrm{mean}\left(O^2\right)}\end{array}$$

(2)

where O and S are the transformed domain’s coefficients for the original and synthesized images, respectively; and MSE is mean squared error. The various transform domain coefficients’ energy is severely impaired by DIBR. Therefore, the watermark energy, inserted into the transform domain’s coefficients, is also damaged, and watermarks damaged by more than 40% are difficult to detect.

However, the average energy change of the Haar discrete wavelet transform (DWT) is reduced to 4% if the distorted coefficients are matched with the undistorted coefficients using the depth image. This is because the wavelet series represent the frequency information of the small spatial block, such as 2 × 2, 4 × 4, ⋯.

If the image is divided into small blocks, some of these will be undistorted, since the depth is similar between adjacent pixels within an object. If all depth values are the same in the block, pixel moving distances are all the same. DIBR is treated like a common translation for this case. A smaller block size implies a greater percentage of uncorrupted blocks, as shown in

Table 2. Average percentage of undistorted blocks after DIBR.

	Block Size
	5 × 5	10 × 10	20 × 20	40 × 40
Undistorted blocks (%)	94.32	85.68	70.67	48.35

for the example of 1800 synthesized DIBR images. Similar to the previous average energy change test, this test also used the recommended DIBR parameters from [3].

Since DCT and discrete Fourier transform (DFT) express a global frequency that does not include spatial information, the amount of average energy change after matching is still high. In other words, it can be seen that the magnitude of the coefficients is damaged in DCT and DFT, but the magnitude of the coefficients is maintained in DWT.

In summary, the following two properties of DIBR can be identified: (1) the pixel is moved only in the horizontal direction, and the moving distance is determined by the depth; (2) the percentage of undistorted blocks is high with a small block size. Due to the second property, the wavelet series are robust to DIBR.

2.3. Curvelet Transform

The curvelet transform is a multi-scale decomposition-like wavelet transform, and the curvelet represents the curve shape for the various directions in the spatial domain [38,39,40,41]. The curvelet transform is developed to improve the limitation of wavelet-based transforms and can represent edges more efficiently than conventional wavelet-based transforms. Moreover, curvelet bases cover all frequencies in contrast to other directional multi-scale transforms, such as the Gabor and ridgelet transforms [42]. The curvelet transform is expressed as follows,

$$\begin{array}{cc}\hfill C(g,l,k):=& \langle f,{\phi }_{g,l,k}\rangle ={\int }_{{\mathbf{R}}^2}f\left(x\right)\overline{{\phi }_{g,l,k}}dx\hfill \\ \hfill =& \frac{1}{{\left(2\pi \right)}^2}\int \widehat{f}\left(\omega \right)U_g\left(R_{{\theta }_l}\omega \right)e^{i\langle x_k^{(g,l)},\omega \rangle }d\omega ,\hfill \end{array}$$

(3)

$$U_g(r,\theta )=2^{-3g/4}W\left(2^{-g}r\right)V\left(\frac{2^{\lfloor g/2\rfloor }\theta }{2\pi }\right),$$

(4)

In (3), C is the curvelet coefficient, $g=0,1,2,\dots$ is the scale parameter, l is the rotation parameter and $k=(k_1,k_2)$ is the translation parameter. $U_g$ is a “wedge”-shaped frequency window represented in (4). $R_{\theta }$ is the rotation operator, and ${\theta }_l=2\pi ·2^{-\lfloor g/2\rfloor }·l$. In (4), W and V are the radial and angular windows, respectively.

The curvelet is illustrated in Figure 3. Figure 3a illustrates the tiling of the curvelet in the frequency domain, and the curvelet shape in several directions and scales in the spatial domain is shown in Figure 3b–d.

As shown in Equation (3) and Figure 3, the curvelet represents frequency information of a small spatial block similarly, so the curvelet is also not distorted by DIBR. In addition, energy conservation is better with the curvelet transform than with conventional Haar DWT when image rotation occurs [43]. For example, when the image rotates 10 degrees, the energy inserted into Haar DWT is reduced to 50%, but that inserted into the curvelet is maintained up to 85%. In the case of scaling attack, energy is well maintained in most DWT.

In summary, the curvelet transform is suitable for use as a template due to its robustness to DIBR and geometric transform.

3. Proposed Method

3.1. Main Idea of the Proposed Method

The overall flow of watermark embedding/extraction is shown in Figure 4. As shown, the proposed watermarking system consists of a template and message watermark. This section describes the characteristics and roles of the template and message.

The DIBR watermarking system must be robust to both DIBR and geometric attacks, as shown in Figure 1. The proposed method inserts a template into a curvelet domain robust against DIBR and geometric distortions. The inserted template enables restoring the image from geometric distortion without being destroyed by DIBR. Geometric distortion can be restored by inserting the template in peak point form and matching detected peaks with ground-truth positions, which can be obtained from the template key in the detection step.

However, although the peak matching method can recover global geometric distortion that occurs across the entire image, DIBR distortion cannot be recovered because it is treated as horizontal error. In addition, the peak point form template cannot insert messages. To address these problems, we designed an invariant message watermarking technique against DIBR, where DIBR deformations that cannot be restored by the template are handled by the message watermark. Message watermarks without templates are vulnerable to geometric attacks, but combining the advantages of the template and message watermark allows a robust DIBR watermarking system.

In summary, the template and message in the proposed watermarking system complement each other’s weaknesses. The curvelet with robustness to DIBR is utilized as a template to restore images from geometric attacks. The message watermark inserted in the 1D-DCT domain resolves the unrecovered damage caused by DIBR. The message watermark itself is not robust to geometric attacks, but images that have suffered geometric attacks can be restored using templates. The roles of templates and message watermarks are summarized in

Table 3. The roles of templates and message watermarks.

	Template	Message Watermark
Domain	Curvelet	1D-DCT
Geometric Distortion	Recoverable	Vulnerable
DIBR	Robust, but unrecoverable	Invariant
Payload	-	More than 1 bit per block

.

3.2. Proposed Watermark Embedding Method

This section gives a detailed description of the proposed watermarking procedure. The whole embedding process is presented in Figure 5.

3.2.1. Block Separation

If the template and the message watermark are inserted at the same position, the signals interfere with each other, and the robustness is degraded. To avoid overlapped insertion, the image is spatially divided into $M\times N$ blocks, as shown in Figure 6. The set of blocks is defined as B, and the template and message are inserted into the different blocks.

A random binary code is generated using the template key, and this code is then used to generate a 2D binary matrix, K, of size $M\times N$, which determines whether each block is a template or message watermark block in an image divided into blocks. To match the number of template blocks with the number of message watermarks, the average (K) should be 0.5. The template key is only used to determine the position of the template blocks, independent of the watermark key to be used later. Therefore, the template key has no effect on the message watermark.

The following rules distinguish the roles of the blocks.

$$B(x,y)=\left\{\begin{array}{c}\mathrm{Template}\mathrm{block},\mathrm{if}K(x,y)=1\hfill \\ \mathrm{Watermark}\mathrm{block},\mathrm{if}K(x,y)=0\hfill \end{array}\right.$$

(5)

where x and y are the horizontal and vertical coordinates of B and K, 0 ≤ x < M and 0 ≤ y < N.

3.2.2. Template Embedding

Since the curvelet coefficients contain spatial information, the template block position can be extracted from the curvelet coefficients. Therefore, the entire image is transformed into the curvelet domain without requiring a block-based curvelet transform. We then insert the peak point templates into the curvelet coefficients.

As shown in Figure 7, four peak points are inserted into one template block. If the number of peak points is too small, the robustness drops, and if there are too many points, the visual quality drops. Experimentally, four points were appropriate.

The template embedding process is divided into three steps:

(a)

Forward curvelet transform: The forward curvelet transform is applied to the whole image.

(b)

Peak points’ insertion: Select two directions of a curvelet, $l_1$ and $l_2$, with scale value g. $l_1$ and $l_2$ are selected such that they differ by 90°, to increase the detection rate by corner detection in the template extraction step, and g should be selected to be a middle frequency, as a compromise between invisibility and robustness. As discussed above, four peak points are inserted into one template block,

$$\begin{array}{c}\hfill \begin{array}{c}\hfill C_m(g,l_1,k)=C(g,l_1,k)·{\alpha }_t+{\beta }_t\\ \hfill C_m(g,l_2,k)=C(g,l_2,k)·{\alpha }_t+{\beta }_t\end{array}\end{array}$$

(6)

where $C_m$ is the modified curvelet coefficient, C is the original curvelet coefficient, ${\alpha }_t$ and ${\beta }_t$ represent the strength of the inserted template and k is the location of the template point.

(c)

Inverse curvelet transform: The inverse curvelet transform is applied to the modified coefficients.

3.2.3. Message Watermark Embedding

The message is inserted into the 1D-DCT domain using the spread spectrum method [44]. If a watermark having the same information in the horizontal direction is inserted into the 1D-DCT, invariance can be obtained against DIBR due to the first property in Section 2.2. The 1D-DCT watermark insertion equation is as follows,

$$\mathbf{DCT}\left(I_{i:}^{\prime }\right)=\mathbf{DCT}\left(I_{i:}\right)+\mathbf{w},$$

(7)

where $i=1,2,\dots ,m$, $\mathbf{w}={[w_1,w_2,\dots w_n]}^T$ is the watermark signal and $m,n$ are the horizontal and vertical sizes of the block, respectively. $I_{i:}$ denotes the i-th column of the original block; $I_{i:}^{\prime }$ denotes the i-th column of the watermarked block; and DCT denotes the 1D-DCT.

The DIBR attack is applied in the spatial domain, not in the transformed domain. Therefore, the inverse DCT (IDCT) is performed to check the change of the inserted watermark in the spatial domain. Using the IDCT, Equation (7) can be rewritten according to the linearity of the DCT as follows,

$$\begin{array}{cc}\hfill I_{i:}^{\prime }=& \mathbf{IDCT}\left(\mathbf{DCT}\left(I_{i:}^{\prime }\right)\right)\hfill \\ \hfill =& \mathbf{IDCT}(\mathbf{DCT}\left(I_{i:}\right)+\mathbf{w})=I_{i:}+\mathbf{IDCT}\left(\mathbf{w}\right)\hfill \\ \hfill =& I_{i:}+\mathbf{v},\hfill \end{array}$$

(8)

where $\mathbf{v}={[v_1,v_2,\dots ,v_n]}^T$ denotes the inverse transformed watermark.

The signal $\mathbf{v}$ is inserted as the same information in all columns of the spatial domain. This means that the embedded patterns from the j-th row have the same $v_j$ signal. Therefore, embedded watermark $\mathbf{v}$ has the following DIBR invariance,

$$\mathbf{D}\left([\mathbf{v},\mathbf{v},\dots ,\mathbf{v}]\right)=\left[\begin{array}{cccc}{v}_1& v_1& \cdots & v_1\\ v_2& v_2& \cdots & v_2\\ ⋮& ⋮& \ddots & ⋮\\ v_n& v_n& \cdots & v_n\end{array}\right]=[\mathbf{v},\mathbf{v},\dots ,\mathbf{v}]$$

(9)

where $\mathbf{D}(·)$ is the DIBR process described in Section 2.1.

Since DIBR only translates pixels in the horizontal direction, $\mathbf{v}$ is not deformed. Therefore, the watermark can be extracted in the frequency domain as follows,

$$\begin{array}{cc}\hfill \mathbf{DCT}\left(\mathbf{D}\left(I_{i:}^{\prime }\right)\right)& =\mathbf{DCT}\left(\mathbf{D}(I_{i:}+\mathbf{v})\right)\hfill \\ & =\mathbf{DCT}(\mathbf{D}\left(I_{i:}\right)+\mathbf{D}\left(\mathbf{v}\right))\hfill \\ & =\mathbf{DCT}(\mathbf{D}\left(I_{i:}\right)+\mathbf{v})\hfill \\ & =\mathbf{DCT}\left(\mathbf{D}\left(I_{i:}\right)\right)+\mathbf{DCT}\left(\mathbf{v}\right)\hfill \\ & =\mathbf{DCT}\left(\mathbf{D}\left(I_{i:}\right)\right)+\mathbf{w}.\hfill \end{array}$$

(10)

Hence, w can be extracted in the frequency domain without being damaged by DIBR in the extraction process.

To compensate for the capacity decrease caused by template insertion, a data coding technique can be utilized to insert more than 1 bit per block [37]. The message watermark embedding process is divided into four steps as follows.

(a)

Column-by-column 1D-DCT: Each column of the message block is transformed by 1D-DCT.

(b)

Data encoding: A pseudo random watermark pattern set (i.e., a set of watermark patterns) is generated. The length of the set is determined by the user, and the capacity of the block is determined according to the length. For example, to represent four bits of information per block, 16 (i.e., $2^4$) unique watermark patterns are generated [45].

(c)

Watermark embedding: The proposed method embeds the watermark based on the spread-spectrum [45]. The generated watermark pattern set ${\mathbf{w}}_b=\left\{w_{b1},w_{b2},\dots ,w_{bL}\right\}$ is embedded into the middle frequency of the DCT signal $\mathbf{s}=\left\{s_1,s_2,\dots ,s_L\right\}$ compromising both robustness and invisibility. The embedding equation is as follows,

$$\begin{array}{c}\hfill s_i^{\prime }=s_i+{\alpha }_m\left|s_i\right|w_{bi}\end{array}$$

(11)

where i is the block column, $1\le i\le L$, L is the length of the watermark signal, $\mathbf{s}=\left\{s_1,s_2,\dots ,s_L\right\}$ is the original signal, ${\mathbf{s}}^{\prime }=\left\{s_1^{\prime },s_2^{\prime },\dots ,s_L^{\prime }\right\}$ is the watermarked signal, ${\alpha }_m$ is the message watermark strength, $w_{bL}$ is the watermark pattern and b is the message inserted into the block. For example, if $b=7$ (111 in binary form), the message inserted in the block is 111 (in this case, the pseudo-random pattern set length = $2^3$). The embedding step (11) is repeated for all columns in the block.

(d)

1D-IDCT: The watermarked block is reconstructed by 1D-IDCT.

3.3. Proposed Watermark Extraction Method

This section describes the details of the proposed watermark extraction method. The overall process is illustrated in Figure 8. Before extracting the messages, the image must be synchronized using a template.

3.3.1. Template Decoding

(a)

Forward curvelet transform: The curvelet transform is applied to the test image.

(b)

Extract template points using corner detection: A peak was inserted into the $l_1$ and $l_2$ pair in the embedding step, with a 90° difference at each template point. Due to curvelet filter characteristics, the peak point spreads in a straight line in the corresponding direction. Therefore, the peak point is represented by an “X” shape when only coefficients of $l_1$ and $l_2$ are extracted, as shown in Figure 9, and can be found by corner detection. This paper employed Harris corner detection, but similar approaches would also suffice.

(c)

Estimate the degree of geometric deformation by exploiting modified ICP: The ICP method estimates the parameters of geometric distortion when there is no matching information of two point clouds. This method assumes that the closest points between the two point clouds match each other and repeats the process of minimizing the error [46].

This paper modifies the ICP to suit the problem in need of solving. Since the DIBR image has an error in the horizontal information, the weight of the horizontal distance error is set to 1/2.

Geometric distortion parameters are estimated using the modified ICP method, comparing detected corner points with the template point ground truth. The template ground truth can be generated using the key as in the embedding step.

(d)

Recover the test image using the estimated geometric distortion parameters.

If the degree of geometric transformation is large, the direction in which the template was inserted may have changed. For example, if the image was significantly rotated, the template inserted in the direction pairs $l_1$ and $l_2$ is moved to the direction pairs $l_1+1$ and $l_2+1$. In this case, the template decoding process is repeated for all direction pairs, and the estimated geometric parameters with the lowest ICP error are used.

3.3.2. Message Watermark Extraction

The message is extracted from the image recovered by the template. Message extraction consists of six steps:

(a)

Split the synchronized image into blocks: Divide the recovered image into blocks as in the embedding step.

(b)

Column-by-column 1D-DCT: The same process applied in the embedding step is applied.

(c)

Correlation: The correlation is conducted as follows,

$$\begin{array}{cc}\hfill Corr\left(b\right)=\frac{{\mathbf{s}}^{\prime }·{\mathbf{w}}_{\mathbf{b}}}{L}=\frac{1}{L}\sum _{i=1}^L{s}_i^{\prime }·& w_{bi}\hfill \end{array}$$

(12)

The notations are identical to those in the embedding step.

(d)

Bit decoding: Bits are decoded from the correlation result. For example, if the correlation value with $w_{3i}$ is the highest, then the bit decoded from the i-th column is 011. The length of the bit is determined by the length of the watermark pattern set, and in this case, the watermark pattern set length = $2^3$. Different bits may be decoded for each column, and the majority voting method is used to determine the bit of the block.

(e)

Restore messages by merging the bits: The messages are recovered by merging the bits extracted from each message block.

4. Experimental Results

This section evaluates the proposed method’s performance in terms of imperceptibility and robustness to various distortions. The proposed method was compared with other blind DIBR image watermarking systems, specifically Lin’s method [18] and Kim’s method [20]. The bit capacity of all methods was set to 64.

4.1. Experiment Setting

The test image sets were obtained from Heinrich Hertz Institute [2], Middlebury [47,48,49,50] and Microsoft Research 3D Video Datasets [51]. Figure 10 shows pairs of center and depth images of the test image sets. They have various resolutions from 900 × 720–1800 × 1500. The total number of images used in the experiment was about 1800.

The DIBR parameters are set to focal length $f=1$ and baseline distance $t_x=5\%$ of the image width, which are the recommended values for comfortable viewing conditions. Linear interpolation is used as hole-filling for simplicity and without loss of generality.

For Lin’s method, block size was set from 100 × 100–200 × 200 to match embedding capacity with other methods. Watermark strength was set as $\alpha$ = 1; watermark pattern length = 5120; and the beginning of the embedding position is the 250th coefficient of the zigzag scan order in the DCT domain.

For Kim’s method, block size was set to (w/8) × (h/8). The weighting factor for coefficient magnitude was set as $\mathbb{W}$ = 450, maximum quantization level $maxBit$ = 2 and minimum difference between paired coefficients $errMin$ = 8. These values are demonstrated in [20].

In the proposed method, the size of each block is set set to (w/8) × (h/8). In the template embedding process, the template strengths ${\alpha }_t$ and ${\beta }_t$ are set to five and 50, experimentally. In the message watermark embedding process, ${\alpha }_m$ is set to 0.5; the length of the watermark pattern is 40; and the start of the embedding position is the 45th coefficient of 1D-DCT.

4.2. Image Quality

As shown in Figure 11, the quality degradation of the watermarked image is not noticeable. For more accurate image quality measurements, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) [52] were measured. The average PSNR and SSIM are shown in

Table 4. Average PSNR and SSIM.

	PSNR	SSIM
Proposed method	50.03	0.9942
Lin’s method	45.12	0.9936
Kim’s method	44.02	0.9878

.

Lin’s method has a lower PSNR despite it exploiting a spread spectrum-based watermarking method that is similar to the proposed method. Since the 2D-DCT is not invariant to DIBR, Lin’s method must insert three watermarks in a superimposed manner in order to protect the left, right and center image. For this reason, the inserted watermark energy is very large.

In Kim’s method, the images are seriously blurred, because this method cuts the coefficient off excessively. As a result, both the PSNR and SSIM values were low.

The proposed method increases imperceptibility by taking advantage of the curvelet and 1D-DCT, which are robust against DIBR. Due to this robustness, the proposed method does not require excessive insertion of the watermark. In particular, the message watermark using the 1D-DCT does not require watermark insertion in a superimposed manner, so the insertion energy can be reduced to about one-third as compared with the 2D-DCT. In addition, since the template and the message can be inserted into different blocks after dividing the block, it is possible to prevent the invisibility degradation caused by the overlapping of templates and message watermarks. For these reasons, PSNR outperforms the other methods and shows similar performance to the best methods according to SSIM.

4.3. Robustness against DIBR

Table 5. Average BER of DIBR images.

	Center Image	Left Image	Right Image
Proposed method	0.002	0.002	0.002
Lin’s method	0.000	0.002	0.004
Kim’s method	0.006	0.008	0.009

shows the results of a DIBR attack. All three methods have low BER against the DIBR attack. Since Lin’s method inserts multiple watermarks, the BER is slightly higher on the right image. The previously inserted watermark (right watermark) was disturbed by the later inserted watermarks (left and center watermarks).

The major advantage of the DIBR process is that DIBR configurations can be adjusted to suit a user’s needs. As described in Section 2.1, the user can preprocess the depth image to increase the rendered image quality.

Table 6. Average BER of synthesized images using the preprocessed depth image.

	Left Image (Preprocessed Depth)	Right Image (Preprocessed Depth)
Proposed method	0.002	0.002
Lin’s method	0.003	0.007
Kim’s method	0.008	0.009

shows robustness results for synthesized images with a preprocessed depth image. Unlike other methods, Lin’s method raised the BER. This result shows that Lin’s method is susceptible to the shift distance of the pixel, because the left and right watermarks were inserted by predicting the shift distance of the pixels.

Baseline distance is another DIBR configuration aspect. Various viewpoints of an image can be synthesized depending on the baseline distance change. Figure 12 shows average BER where the baseline distance is adjusted from 1–10%. Lin’s method increases BER for the same reason as in depth image preprocessing, whereas Kim’s and the proposed method do not increase BER in the baseline distance adjustment.

4.4. Robustness against Signal Distortion

Table 7. Average BER for Gaussian noise attack.

Noise	Proposed			Lin			Kim
Var.	Left	Center	Right	Left	Center	Right	Left	Center	Right
10	0.00	0.00	0.00	0.00	0.00	0.00	0.04	0.03	0.04
40	0.00	0.00	0.00	0.00	0.00	0.00	0.12	0.09	0.111
300	0.01	0.01	0.01	0.00	0.00	0.00	0.38	0.36	0.38
1250	0.04	0.04	0.05	0.02	0.02	0.02	0.54	0.54	0.55

and

Table 8. Average BER for JPEG attack.

Quality	Proposed			Lin			Kim
Factor	Left	Center	Right	Left	Center	Right	Left	Center	Right
90	0.00	0.00	0.00	0.00	0.00	0.00	0.02	0.01	0.02
60	0.01	0.01	0.01	0.00	0.00	0.00	0.22	0.20	0.22
30	0.10	0.09	0.10	0.09	0.08	0.10	0.35	0.33	0.35

show the average BER for the signal distorted center, left and right images, and Figure 13 illustrates the average BER for the signal distorted right image.

For additive noise, Lin’s method has the best performance, and the proposed method has a slightly higher BER than Lin’s method. However, additive Gaussian noise with a variance of 2000 is a very severe attack, and such a large amount of noise barely occurs. Considering this, the proposed method is robust enough to additive noise.

For JPEG compression, the proposed method has a slightly higher BER than Lin’s method. However, since the proposed method exhibits error <0.1 for a JPEG quality factor of 30, which is very large compression, it can be considered sufficiently strong against a JPEG attack.

Meanwhile, Kim’s method is more vulnerable to signal distortion than the other methods, since the quantized DT-CWT coefficients are greatly affected by the signal distortion.

4.5. Robustness against Geometric Distortion

A robustness test against geometric distortion was also conducted with the center, left and right images. We did not consider the case where a geometric attack occurred before the DIBR attack. If a geometric attack takes place before a DIBR attack, the image becomes unnatural. For example, depth is measured horizontally, so if the DIBR is applied after the image is rotated, pixel warping will occur in an unintended direction.

Table 9. Average BER for scaling attack.

Scaling	Proposed			Lin			Kim
Factor	Left	Center	Right	Left	Center	Right	Left	Center	Right
0.6	0.01	0.01	0.02	0.48	0.49	0.49	0.08	0.01	0.10
0.8	0.02	0.01	0.02	0.47	0.50	0.52	0.01	0.02	0.03
1.4	0.09	0.06	0.08	0.52	0.51	0.49	0.01	0.01	0.01
2.0	0.04	0.03	0.04	0.49	0.50	0.50	0.02	0.02	0.02

,

Table 10. Average BER for rotation attack.

Rotation	Proposed			Lin			Kim
Degree	Left	Center	Right	Left	Center	Right	Left	Center	Right
10	0.09	0.05	0.10	0.45	0.54	0.51	0.43	0.41	0.42
20	0.10	0.07	0.09	0.52	0.50	0.49	0.54	0.55	0.51
30	0.09	0.06	0.09	0.50	0.53	0.53	0.52	0.54	0.54
40	0.11	0.08	0.12	0.55	0.56	0.50	0.50	0.55	0.54

,

Table 11. Average BER for translation attack.

Translation	Proposed			Lin			Kim
(%)	Left	Center	Right	Left	Center	Right	Left	Center	Right
10	0.08	0.06	0.09	0.54	0.51	0.51	0.24	0.22	0.22
20	0.07	0.05	0.08	0.53	0.50	0.53	0.55	0.55	0.55
30	0.11	0.07	0.12	0.49	0.47	0.50	0.53	0.53	0.53

and

Table 12. Average BER for shearing attack.

Shearing	Proposed			Lin			Kim
Factor	Left	Center	Right	Left	Center	Right	Left	Center	Right
0.1	0.03	0.02	0.02	0.54	0.51	0.52	0.24	0.22	0.22
0.2	0.03	0.03	0.03	0.48	0.53	0.49	0.53	0.50	0.51
0.3	0.05	0.05	0.05	0.50	0.51	0.47	0.52	0.53	0.56

show the average BER for the geometrically-distorted center, left and right images, and Figure 14 illustrates the average BER for the geometrically-distorted right image.

Lin’s method does not show good results against geometric distortion, because this method embeds the watermark in the DCT domain, which is vulnerable to geometric distortion.

Kim’s method shows good performance against scaling, but is weak to rotation, translation and shearing. This method does not lose synchronization information in scaling. However, the rotation, translation and shearing attacks break the block synchronization. This is because the block size is specified as the ratio of the image, such as $(width/M)\times (height/N)$.

The proposed method shows good performance on this test, because it utilizes a template robust to geometric attacks. Since the images were recovered from geometric distortion using this template, the message watermark has a low error.

Experiments were also conducted on the combination of geometric distortion in the proposed method. This experiments were conducted by combining two of rotation, translation and scaling, and the results are shown in

Table 13. Average BER of the proposed method for combination geometric distortion. The experiment was conducted on the synthesized right image.

Translation (%)	Rotation (°)			Scaling Factor	Rotation (°)			Scaling Factor	Translation (%)
	3	6	9		3	6	9		7	14	21
5	0.11	0.11	0.11	0.8	0.13	0.10	0.12	0.8	0.13	0.12	0.13
10	0.12	0.12	0.12	0.9	0.10	0.09	0.09	0.9	0.12	0.11	0.11
15	0.11	0.12	0.12	1.1	0.17	0.17	0.18	1.1	0.13	0.12	0.13
20	0.12	0.12	0.12	1.2	0.23	0.25	0.28	1.2	0.15	0.14	0.15

. Rotation with translation and scaling with translation show good results. However, when rotation and scaling occur at the same time, the watermark signal is greatly weakened, so the error is higher than with other geometric combination attacks.

5. Conclusions

Due to the advent of new 3D applications, DIBR has taken on an important role in 3D content. To protect the copyright of such content, this paper proposed a template-based DIBR watermarking system. Ensuring robustness against geometric attacks in rendered images requires that watermarks should be robust to both DIBR and geometric attacks. In order to have robustness to a combination of linear and non-linear attacks, a watermarking system was designed by combining two methods: template and message watermark. Inserting a template into the curvelet domain robust to DIBR allowed this method to restore the geometrically-distorted image. Then, the message was extracted using a 1D-DCT message watermarking method that was invariant to the DIBR. In the experimental results, the proposed method showed high image quality. In terms of robustness, it had low BER to DIBR configuration adjustment, as well as standard configured DIBR. Additionally, the results showed that the proposed method is very robust against noise addition and JPEG compression. For geometric distortion, such as scaling, rotation, translation and shearing, good performance was also demonstrated. However, this method still does not consider the robustness of video coding, which is often used in 3D video, such as high efficiency video coding (HEVC). Therefore, future work will focus on extending this method to 3D videos. In addition, since the proposed method does not use message encryption, we could further improve the security of the message by using encryption. In the future, we will investigate various other types of attack, such as copy-and-paste or transplantation.