Rational Spline Image Upscaling with Constraint Parameters

Abstract

Image interpolation is one of key contents in image processing. We present an interpolation algorithm based on a rational function model with constraint parameters. Firstly, based on the construction principle of the rational function, the detection threshold is selected through contour analysis. The smooth and non-smooth areas are interpolated by bicubic interpolation and general rational interpolation, respectively. In order to enhance the contrast in non-smooth areas and preserve the details, the parameter optimization technique is applied to get optimal shape parameters. Experimental results on benchmark test images demonstrate that the proposed method achieves competitive performance with the state-of-the-art interpolation algorithms, especially in image details and texture features.

1. Introduction

Interpolation acts as a bridge between the continuous world and the discrete one. As an important technique, it pervades many applications [1,2]. A digital image is not an exact snapshot of reality, it is only a discrete approximation. Image interpolation focuses on the issue which obtains a high-resolution image from a low-resolution one. Image interpolation plays an important role in image processing, and it is widely used in various fields, such as aerospace, medical, military, scientific research, communications, remote sensing satellite, television, film production, etc.

Generally speaking, the interpolation technique is used in nearly every geometric transformation, such as translation, scaling, rotation, etc. Such operations are utilized in many commercial digital image processing software [3]. The main issue of image interpolation is to maintain texture details and edge structure, while eliminating blocking artifacts, texture disorder, and other visual artifacts. Quantities of image interpolation methods have been proposed in several articles [4,5,6,7,8,9,10,11,12,13]. In these existing methods, linear filters (as the simplest technique) have been widely used in image interpolation, such as the bilinear, bicubic [4], and cubic spline algorithms [5]. Although these traditional methods are effective, they have disadvantages in ringing artifacts, blurred details, and so forth. Because the image can be affected by many factors, including the light, natural background, and the characteristics of its own texture, all in all, the relationship between pixels is not linear [14].

With ever-increasing capacity of computation power, many image interpolation methods have been proposed in the past few years. Jeong et al. [15] presented a multi-frame example-based super-resolution (SR) algorithm using locally directional self-similarity. Jha et al. [16] proposed a new image interpolation method using adaptive weights based on inverse gradients and distances from the pixels used in prediction. In [17], an edge-guided interpolation approach was proposed, which improved the accuracy of interpolation by detecting edges and fitting them with templates. In [18], predetermined edge patterns were used to obtain optimal parameters. As a consequence, according to the original images’ edge structures, high quality interpolated images could be obtained. Edge-Directed interpolation (EDI) methods have been presented [19]. Said and Pearlman [20] proposed a source model which focuses on the integrity of detected edges and modifies the interpolation to adapt to the original one. Then, a New Edge-Directed Interpolation (NEDI) adaptive method was presented, which obtains better subjective quality than EDI. Based on the geometric duality between the low-resolution covariance and the high-resolution covariance, the relationship of high-resolution covariance and the low-resolution one can be estimated [6]. A soft-decision interpolation technique (SAI) was proposed, which estimates unknown pixels in groups [13].

We dedicated our research to the integration of image interpolation and human visual perception. However, how to design the interpolation algorithm that combines with visual perception for the purpose of getting an “ideal” interpolated image is still a challenge for image interpolation methods. In this paper, a new image interpolation method based on rational function model is proposed. Considering the speed and quality of the interpolated image, the input image will be classified into smooth areas and non-smooth areas. For smooth areas, a bicubic interpolation is used to lower time complexity. For non-smooth areas, the proposed optimal rational interpolation function is used to improve the accuracy of interpolation. The bicubic interpolation function is a special form of the same interpolation function model on the condition that the parameters are chosen with some specific values. Namely, parameters α and β equal 1. The mean value of nine interpolation data points is the threshold, and is used for non-smooth area detection. Image interpolation, processing, region detection, different region interpolation, and visual contrast enhancement are all based on one rational interpolation model. Experimental results show good image quality results in both edges and details. The algorithm framework is shown in Figure 1.

This paper is organized as follows: Section 2 introduces the fundamental principles of the rational function model and its properties. Section 3 presents the proposed interpolation algorithm. Experimental results are shown for demonstration in Section 4. Section 5 concludes this paper.

2. A Bivariate Rational Interpolation

In recent years, the univariate rational spline interpolations with parameters have been developed [21,22,23]. These kinds of interpolation splines have a simple mathematical representation and can preserve property of interpolated curve and surface. Motivated by the univariate rational spline interpolation, the bivariate rational interpolation with parameters based on the function values has been studied in [24,25,26,27,28,29]. The interpolation function has a piecewise explicit rational mathematical representation with parameters, and it can be represented by its basis.

Let $\mathrm{\Omega }:[a,b;c,d]$ be the plane region, let $f(x,y)$ be a bivariate function defined in the region Ω, and let $a=x_1<x_2<\cdots <x_n<x_{n+1}=b$ and $c=y_1<y_2<\cdots <y_m<y_{m+1}=d$ be the knot sequences. Denote $f(x_i,y_j)$ by $f_{i,j}$; then, $\{(x_i,y_j,f_{i,j}),i=1,2,\dots n,n+1;j=1,2,\dots m,m+1\}$ are the given set of data points. For any point $(x,y)\in [x_i,x_{i+1};y_j,y_{j+1}]$ in the $xy$-plane, let $h_i=x_{i+1}-x_i$, $\theta =\frac{x-x_i}{h_i}$, and $l_j=y_{j+1}-y_j$, $\eta =\frac{y-y_j}{l_j}$. For each $y=y_j,j=1,2,\cdots m+1$, construct the x-direction interpolation curve; this is given by

$$\begin{array}{c}{P}_{i,j}^{*}\left(x\right)=\frac{p_{i,j}^{*}\left(x\right)}{q_{i,j}^{*}\left(x\right)},x\in [x_i,x_{i+1}],i=1,2,\cdots n-1,\end{array}$$

(1)

where

$$\begin{array}{ccc}\hfill p_{i,j}^{*}\left(x\right)& =& {(1-\theta )}^3{\alpha }_{i,j}{f}_{i,j}+\theta {(1-\theta )}^2{V}_{i,j}^{*}+{\theta }^2(1-\theta )W_{i,j}^{*}\hfill \\ & +& {\theta }^3{f}_{i+1,j},\hfill \\ \hfill q_{i,j}^{*}\left(x\right)& =& (1-\theta ){\alpha }_{i,j}+\theta ,\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill V_{i,j}^{*}& =& ({\alpha }_{i,j}+1)f_{i,j}+{\alpha }_{i,j}{f}_{i+1,j},\hfill \\ \hfill W_{i,j}^{*}& =& ({\alpha }_{i,j}+2)f_{i+1,j}-h_i{\mathrm{\Delta }}_{i+1,j}^{*},\hfill \end{array}$$

with ${\alpha }_{i,j}>0,$ and ${\mathrm{\Delta }}_{i,j}^{*}=(f_{i+1,j}-f_{i,j})/h_i.$ This interpolation is called the rational cubic interpolation based on function values which satisfies

$$p_{i,j}^{*}\left(x_i\right)=f_{i,j},p_{i,j}^{*}\left(x_{i+1}\right)=f_{i+1,j},$$

$$p_{i,j}^{{*}^{\prime }}\left(x_i\right)={\mathrm{\Delta }}_{i,j}^{*},p_{i,j}^{{*}^{\prime }}\left(x_{i+1}\right)={\mathrm{\Delta }}_{i+1,j}^{*}.$$

Obviously, the interpolation is a local one, it is defined in the interval $[x_i,x_{i+1}]$ and depends on the data at three points $\{(x_r,y_j,f_{r,j}),r=i,i+1,i+2\}$ and the parameter ${\alpha }_{i,j}$.

For each pair $(i,j),i=1,2,\cdots n-1$ and $j=1,2,\cdots m-1$, using the x-direction interpolation function $P_{i,j}^{*}\left(x\right)$, define the bivariate rational interpolating function $P_{i,j}(x,y)$ on $[x_i,x_{i+1};y_j,y_{j+1}]$ as follows

$$\begin{array}{c}{P}_{i,j}(x,y)=\frac{p_{i,j}(x,y)}{q_{i,j}\left(y\right)},\\ i=1,2,\cdots ,n-1;j=1,2,\cdots ,m-1\end{array}$$

(2)

where

$$\begin{array}{ccc}\hfill p_{i,j}(x,y)& =& {(1-\eta )}^3{\beta }_{i,j}{P}_{i,j}^{*}\left(x\right)+\eta {(1-\eta )}^2{V}_{i,j}\hfill \\ & +& {\eta }^2(1-\eta )W_{i,j}+{\eta }^3{P}_{i,j+1}^{*}\left(x\right),\hfill \\ \hfill q_{i,j}\left(y\right)& =& (1-\eta ){\beta }_{i,j}+\eta ,\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill V_{i,j}& =& ({\beta }_{i,j}+1)P_{i,j}^{*}\left(x\right)+{\beta }_{i,j}{P}_{i,j+1}^{*}\left(x\right),\hfill \\ \hfill W_{i,j}& =& ({\beta }_{i,j}+2)P_{i,j+1}^{*}\left(x\right)-l_j{\mathrm{\Delta }}_{i,j+1}\left(x\right),\hfill \end{array}$$

with ${\beta }_{i,j}>0,$ and ${\mathrm{\Delta }}_{i,j}\left(x\right)=(P_{i,j+1}^{*}(x)-P_{i,j}^{*}(x))/l_j.$

In the subregion $[x_i,x_{i+1};y_j,y_{j+1}]$, the function $P_{i,j}(x,y)$ is called a bivariate rational interpolation function based on function values. It depends on the data at nine points $\{(x_r,y_s,f_{r,s}),r=i,i+1,i+2,$ $s=j,j+1,j+2\}$ and which satisfies

$$P_{i,j}(x_r,y_s)=f(x_r,y_s),r=i,i+1,s=j,j+1.$$

Consider the equally spaced knots case; namely, for all $i=1,2,\cdots ,n$ and $j=1,2,\cdots ,m$, $h_i=h_j$ and $l_i=l_j$. The interpolating function $P_{i,j}(x,y)$ is $C^1$ in the whole interpolating region. The bivariate rational interpolating function $P_{i,j}(x,y)$ can be expressed as follows:

$$\begin{array}{c}{P}_{i,j}(x,y)=\sum _{r=0}^2\sum _{s=0}^2{\omega }_{rs}(\theta ,{\alpha }_i;\eta ,{\beta }_j)f_{i+r,j+s}\end{array}$$

(3)

where

$$\begin{array}{c}{\omega }_{rs}(\theta ,{\alpha }_i;\eta ,{\beta }_j)={\omega }_r(\theta ,{\alpha }_i){\omega }_s(\eta ,{\beta }_j)\end{array}$$

$$\begin{array}{c}{\omega }_0(\theta ,{\alpha }_i)=\frac{{(1-\theta )}^2({\alpha }_i+\theta )}{(1-\theta ){\alpha }_i+\theta }\end{array}$$

$$\begin{array}{c}{\omega }_1(\theta ,{\alpha }_i)=\frac{\theta (1-\theta ){\alpha }_i+3{\theta }^2-2{\theta }^3}{(1-\theta ){\alpha }_i+\theta }\end{array}$$

$$\begin{array}{c}{\omega }_2(\theta ,{\alpha }_i)=\frac{-{\theta }^2(1-\theta )}{(1-\theta ){\alpha }_i+\theta }.\end{array}$$

The terms ${\omega }_{rs}(\theta ,{\alpha }_i;\eta ,{\beta }_j),r=0,1,2;s=0,1,2$ are called the basis of the bivariate interpolation, which satisfy

$$\sum _{r=0}^2\sum _{s=0}^2{\omega }_{rs}(\theta ,{\alpha }_i;\eta ,{\beta }_j)=1.$$

From Equation (3), the interpolating function $P_{i,j}(x,y)$ is defined by a given set of data points $\{f_{i,j},f_{i+1,j},f_{i+2,j};f_{i,j+1},$ $f_{i+1,j+1},f_{i+2,j+1};f_{i,j+2},f_{i+1,j+2},f_{i+2,j+2}\}.$ Applying the interpolating function, we can construct a patch through points $f_{i,j}$, $f_{i+1,j},f_{i,j+1},$ $f_{i+1,j+1}$.

3. Basic Algorithms

Generally speaking, there is always a contradiction between the processing speed of interpolation and the resultant quality in image interpolation. The commonly used methods [6,7,13] usually adopt different interpolation functions for different regions Furthermore, the different region detection algorithm is not related to the interpolation. Namely, the used methods are not based on the interpolation function. Because the image has natural attributes, its inherent attributes are inevitably affected by using the commonly used methods. Thus, we present a novel interpolation model to solve this problem. Moreover, the processing speed of interpolation and the resultant quality are all preserved using the new algorithm.

3.1. Image Non-Smooth Areas Detection

In this paper, a variety of textures and edges of the image area are called non-smooth areas. The smooth areas contain abundant detailed information, and have critical effects on the image quality. First of all, the non-smooth areas are detected.

From the rational function, a patch can be constructed given nine data points. All of them have different effects on the patch through their weights ${\omega }_{rs}(\theta ,\alpha ;\eta ,\beta )$. When the weights present great differences from each other, the patch becomes more and more non-flat, considered as non-smooth area in the interpolated image. As mentioned above, the data points play a different role in the structure processing. The drawing and analyses of contours depends on the decision of the sign. On this basis, contour analysis is used to detect the non-smooth areas. The detection model is based on the interpolation model. As mentioned above, an interpolated patch is constructed using nine data points. The points play different roles in the structure processing. The vertices of the patch are the key points. The other data points are auxiliary. As the detection threshold, the mean value of the nine data points can be used for detecting the non-smooth areas in the interpolated image. Namely, for any given set of data ${\sum }_{r=0}^2{\sum }_{s=0}^2{f}_{r,s},r,s=0,1,2$, the detection threshold λ is given as follows:

$$\lambda =\frac{{\sum }_{r=0}^2{\sum }_{s=0}^2{f}_{r,s}}{9}.$$

Let $\delta =f_{r,s}-\lambda ,r,s=0,1$; from the contours drawing and analyses, if δ are all nonpositive and have nonnegative sign, the region is regarded as non-smooth. Otherwise it is smooth. Basically, the detection threshold is selected based on the rational function construction. The non-smooth detection results are shown in Figure 2.

3.2. Image Interpolation

An image is a set of values on a 2D plane. It is affected by the material of the object surface, the physical appearance of background objects , light strength and angle, and noise of the imaging progress. These data are purely random and can be constructed completely. The value of these data are gradually changed. In general, iit is nonlinear. As an nonlinear function, bivariate rational interpolation is a good choice to implement image interpolation. As is well known, interpolation processing speed is an important standard to measure an interpolation algorithm. Bivariate rational interpolation with parameters has nice properties. If the parameters equal to 1, it is a bicubic interpolation function. In this case, it may be written as

$$\begin{array}{c}{P}_{i,j}(x,y)=\sum _{r=0}^2\sum _{s=0}^2{\omega }_{rs}(\theta ,\eta )f_{i+r,j+s}\end{array}$$

(4)

where

$$\begin{array}{c}{\omega }_{rs}(\theta ,\eta )={\omega }_r\left(\theta \right){\omega }_s\left(\eta \right)\end{array}$$

and

$$\begin{array}{ccc}\hfill {\omega }_0\left(\theta \right)& =& {\theta }^3-{\theta }^2-\theta +1,\hfill \\ \hfill {\omega }_1\left(\theta \right)& =& -2{\theta }^3+{\theta }^2+\theta ,\hfill \\ \hfill {\omega }_2\left(\theta \right)& =& {\theta }^3-{\theta }^2.\hfill \end{array}$$

From Section 3.1, an image should be divided into multiscale edge regions and non-multiscale edge areas by using region detection. In order to reduce computational complexity, the multiscale edge regions and non-multiscale edge areas are interpolated by bicubic interpolation and rational interpolation, respectively. As follows, image interpolation based on the rational function model is introduced.

Given a $m\times n$ image $I_{m,n}$, let $f_{i,j}(0\le i\le m-1,0\le j\le n-1)$ be the gray value of the i line and the j row of $I_{m,n}$. The pixel coordinate is $(i,j)$. Denote a two-dimension discrete signal by $I_{m,n}$. At the integer points it is simple. Denote the data point by the gray value of each pixel. Then, the continuous interpolating surface can be constructed based on a bivariate rational interpolation method for a discrete image. For the random point $f_{i,j}$ in an image, our concern is how to construct the white points in a rectangular cell. First, the interpolation data must be ascertained. The data points are ${\sum }_{r=0}^2{\sum }_{s=0}^2{f}_{r,s},r,s=0,1,2$. Then, based on the ${\alpha }_i$ and ${\beta }_j$ values, we can obtain the interpolation function $P_1(x,y)$ by substitution of the interpolation data into Equation (3). Finally, the function values of the white points are calculated by proper x and y.

3.3. Parameters Optimization

For image interpolation based on the rational function, the following optimal equation is derived from Equation (3).

$$\begin{array}{c}\max grad\left(P_{i,j}(x,y)\right)=\max \sqrt{{\frac{\partial P}{\partial x}}^2+{\frac{\partial P}{\partial y}}^2}\end{array}$$

then

$$\begin{array}{c}\max grad\left(P_{i,j}(x,y)\right)\doteq \max (|\frac{\partial P}{\partial x}|+|\frac{\partial P}{\partial y}|).\end{array}$$

Denote

$$F(\theta ,{\alpha }_i;\eta ,{\beta }_j)=|\frac{\partial P}{\partial x}|+|\frac{\partial P}{\partial y}|$$

the following equation is derived.

$$\begin{array}{c}\max F(\theta ,{\alpha }_i;\eta ,{\beta }_j)s.t.{\alpha }_i>0,{\beta }_j>0\end{array}$$

(5)

the parameters ${\alpha }_i,{\beta }_j$ are satisfied with the above equation as the optimal solution. The optimal parameters ${\alpha }_i,{\beta }_j$ are denoted by ${\alpha }_i^{*},{\beta }_j^{*}$.

It must be pointed out that some interpolated image pixels are varied by selecting parameters ${\alpha }_i^{*},{\beta }_j^{*}$. $P_{i,j}(x,y)$ defined by 3 should satisfy the boundary property, and the interpolating patch can be modified when the parameters are different. Furthermore, for the interpolated image, it also satisfies $0\le P_{i,j}(x,y)\le 255$. For the case of the maximum value of $P_{i,j}(x,y)>255$, assume that $P_{i,j}(x,y)$ reaches its maximum value $P_{i,j}^{*}(x,y)$ at point $(x^{*},y^{*})$ in subregion $[x_i,x_{i+1};y_j,y_{j+1}]$; i.e., $P_{i,j}(x^{*},y^{*})>255$, $P_{i,j}(x,y)$ should be redefined by

$$255\times \frac{P_{i,j}(x,y)}{P_{i,j}(x^{*},y^{*})}.$$

4. Experiments

Experiments were conducted to evaluate the effectiveness of the interpolation algorithm. We compare three state-of-the-art interpolation algorithms, including NEDI [6], DFDF [7], SAI [13]. All experiments are performed with software provided by the authors of these algorithms. In our experiments, the four gray images shown in Figure 3 are applied to test visual effects.

Table 1. Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index (SSIM)  results of the reconstructed high-resolution (HR)  images by different methods.

	NEDI		DFDF		SAI		Our Method
	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
Light-tower	22.78	0.7937	23.25	0.7971	22.86	0.8005	23.37	0.8009
Dollar	19.10	0.8084	19.21	0.8066	19.24	0.8055	19.36	0.8118
Cliff	25.08	0.7115	25.05	0.7184	25.16	0.7233	25.22	0.7268
Barbara	22.35	0.8513	23.64	0.8766	23.54	0.8635	24.12	0.8801
Milkdrop	30.97	0.9156	34.36	0.9196	32.39	0.9176	34.48	0.9216
Couple	28.65	0.9391	29.06	0.9413	29.32	0.9443	29.14	0.9420
Goldhill	26.60	0.7645	26.69	0.7678	26.92	0.7772	26.92	0.7750
Door	33.12	0.9446	33.08	0.9447	31.16	0.9467	33.20	0.9478
Sky	28.41	0.9154	28.95	0.8608	29.05	0.9364	28.96	0.9378
Boat	25.82	0.8941	25.54	0.8378	25.43	0.9120	25.61	0.8973
Average	25.22	0.8207	25.74	0.8225	25.58	0.8326	25.98	0.8384

NEDI: new edge-directed interpolation; SAI: a soft-decision estimationtechnique for adaptive image interpolation; DFDF: directional filtering and data fusion.

gives the Peak Signal to Noise Ratio (PSNRs) and Structural Similarity Index (SSIMs) generated by all algorithms for the images. It can be seen that the proposed method has a higher average PSNR and SSIM among the compared algorithms.

We performed tests on natural images to show the improvements of the proposed method in the visual effects. Figure 4, Figure 5 and Figure 6 show the comparison of interpolation in test images. In Figure 5, all of the interpolation algorithms had aliasing in texture, but our method gave less. In Figure 6, we can see that NEDI suffers from some noisy interpolation artifacts. The SAI method also suffers from noisy artifacts. Our method had better performance in maintaining image features. As shown in Figure 7 and Figure 8, shape loss and noisy points. were introduced in images of a door and fence by the use of NEDI and SAI. The DFDF algorithm also introduced some errors of edge and texture. In general, experimental results on benchmark test images demonstrate that the proposed method achieved very competitive performance with the state-of-art interpolation algorithms, especially in image details and texture features.

5. Conclusions

Although image interpolation is not the most advanced research subject, research results in this field have come forth continuously in recent years. They are usually based on relatively simple theory and have broad applicability However, the interpolation expression forms are various. Therefore, according to the advantages of the different interpolation methods, we can take advantage of these methods and get good results. However, these combination methods have complicated forms, and do not meet the needs of timeliness and practicality. Basically, these methods often neglect the natural image attributes. The main contributions of our method are not combination, but adopting function constructed in Section 2. In this process, non-smooth area detection, image interpolation in different regions, and visual perception are merged into one interpolation function model. The image details and texture features are preserved by using this algorithm, and it can help to describe the image more objectively; hence, more features of the image can be preserved.