NMF is a recently developed matrix analysis algorithm [17,18], which can not only describe low-dimensional intrinsic structures in high-dimensional space, but achieves linear representation for original sample data by imposing non-negativity constraints on its bases and coefficients. It makes all the components non-negative (i.e., pure additive description) after being decomposed, as well as realizes the non-linear dimension reduction. NMF is defined as:

Conduct N times of investigation on a M-dimensional stochastic vector v, then record these data as v_j, j = 1,2,…, N, let V = [V_•1, V_•2, V_•N], where V_•j = v_j, j = 1,2,…, N. NMF is required to find a non-negative M × L base matrix W = [W_•1, W_•2,…, W_•N] and a L × N coefficient factor H = [H_•1, H_•2,…, H_•N], so that V ≈ WH [17]. The equation can also be wrote in a more intuitive form of that V . j ≈ ∑ i = 1 L W . i H . j, where L should be chose to satisfy (M + N) L < MN.

In the purpose of finding the appropriate factors W and H, the commonly used two objective functions are depicted as [18]: (1) E ( V ‖ W H ) = ‖ V − W H ‖ F 2 = ∑ i = 1 M ∑ j = 1 N ( V i j − ( W H ) i j ) 2 (2) D ( V ‖ W H ) = ∑ i = 1 M ∑ j = 1 N ( V i j log V i j ( W H ) i j − V i j + ( W H ) i j )

In respect to Equations (1) and (2), ∀i, a, j subject to W_ia > 0 and H_aj > 0, a is a integer. ‖•‖_F is the Frobenius norm, Equation (1) is called as the Euclid distance while Equation (2) is referred to as K-L divergence function. Note that, finding the approximate solution to V ≈ WH is considered equal to the optimization of the above mentioned two objective functions.

Roughly speaking, the NMF algorithm has high time complexity that results in limited advantages for the overall performance of algorithm, so that the introduction of improved iteration rules to optimize the NMF is extremely crucial to promote the efficiency. In the point of algorithm optimization, NMF is a majorization problem that contains a non-negative constraint. Until now, a wide range of decomposition algorithms have been investigated on the basis of non-negative constraints, such as the multiplicative iteration rules, interactive non-negative least squares, gradient method and projected gradient [29], among which the projected gradient approach is capable of reducing the time complexity of iteration to realize the NMF applications under mass data conditions. In addition, these works are distinguished by meaningful physical significance, effective sparse data, enhanced classification accuracy and striking time decreases. We propose a modified version of projected gradient NMF that will greatly reduce the complexity of iterations; the main idea of the algorithm is listed below.

As we know, the Lee-Seung algorithm continuously updates H and W, fixing the other, by taking a step in a certain weighted negative gradient direction, namely: (3) H i j ← H i j − η i j [ ∂ f ∂ H ] i j ≡ H i j + η i j ( W T A − W T W H ) i j (4) W i j ← W i j − ς i j [ ∂ f ∂ W ] i j ≡ W i j + ς i j ( A H T − W H H T ) i jwhere η_ij and ζ_ij are individual weights for the corresponding gradient elements, which are expressed like follows: (5) η i j = H i j ( W T W H ) i j , ς i j = W i j ( W H H T ) i jand then the updating formulas are: (6) H i j ← H i j ( W T A ) i j ( W T W H ) i j , W i j ← W i j ( A H T ) i j ( W H H T ) i j

We notice that the optimal H related to a fixed W can be obtained, column by column, by independently: (7) min 1 2 ‖ A e j − W H e j ‖ 2 2 s.t. H e j ≥ 0where e_j is the j^th column of the n × n identity matrix. Similarly, we can also acquire the optimal W relative to a fixed H by solving, row by row: (8) min 1 2 ‖ A T e i − H W T e i ‖ 2 2 s.t. W T e i ≥ 0where e_i is the i^th column of the m × m identity matrix. Actually, both Equations (7) and (8) can be changed into an ordinary form: (9) min 1 2 ‖ A x − b ‖ 2 2 s.t. x ≥ 0where A ≥ 0 and b ≥ 0. As the variables and given data are all nonnegative, the problem is therefore named the Totally Nonnegative Least Squares (TNNLS) issue.

We propose to revise the algorithm claimed in article [17] by using the same update rule with step-length α in [27] to the successive updates in improving the objective functions about the two TNNLS problems mentioned in Equations (7) and (8) As a result, this brings about a modified form of the Lee-Seung algorithm that successively updates the matrix H column by column and W row by row, with individual step-length α and β for each column of H and each row of W respectively. So we try to write the update rule as: (10) H i j ← H i j + α j η i j ( W T A − W T W H ) i j (11) W i j ← W i j + β i ς i j ( A H T − W H H T ) i jwhere η_ij and ζ_ij are set equal to some small positive number as described in [27], α_j (j = 1,2,…,n) and β_i (i = 1,2,…,m) are step-length parameters can be computed as follows. Let x > 0, q =A^T(b − Ax) and p = [x./(A^TAx)] ○ q, where the symbol “./” means component-wise division and “○” denotes multiplication. Then we introduce variable ô ∈ (0, 1): (12) α = min ( p T q p T A T A p , τ max { α ^ : x + α ^ p ≥ 0 } )

We can easily obtain the step-length formula of α_j or β_i if (A, b, x) is replaced by (W, Ae_j, He_j) or (H^T, A^Te_i, W^Te_i), respectively. It is necessary to point out that q is the negative gradient of the objective function, and the search direction p is a diagonally scaled negative gradient direction. The step-length α or β is either the minimum of the objective function in the search direction or a τ-fraction of the step to the boundary of the nonnegative quadrant.

Learning from article [27] that both quantities, p^Tq/p^TA^TAp and max{â : x + âp ≥ 0} are greater than 1 in the definition of the step α, thereby, we make α_j ≥ 1 and β_i ≥ 1 by treating τ sufficiently close to 1. In our experiment, we choose τ = 0.99 which practically guarantees that α and β are always greater than 1.

Obviously, when α←1 or β←1, update Equations (10) and (11) reduce to updates Equations (3) and (4) In our algorithm, the step-length parameters are allowed to be greater than 1. It is this indicates that for any given (W, H), we can get at least the same or greater decrease in the objective function than the algorithm in [27]. Hence, we call the proposed algorithm the Accelerated NMF (ANMF). Besides, the experiments in Section 5.5 will demonstrate that ANMF algorithm is indeed superior to that algorithm by generating better test results, especially when the amount of iterations is not too big.

As we know, approximation of an image belongs to the low-frequency part, while the high-frequency counterpart exhibits detailed features of edge and texture. In this paper, the NSCT method is utilized to separate the high and low components of the source image in the frequency domain, and then the two parts are dealt with different fusion rules according to their features. As a result, the fused image can be more complementary, reliable, clear and better understood.

By and large, the low-pass sub-band coefficients approximate the original image at low-resolution; it generally represents the image contour, but high-frequency details such as edges, region contours are not contained, so we take the ANMF algorithm to determine the low-pass sub-band coefficients which including holistic features of the two source images. The band-pass directional sub-band coefficients embody particular information, edges, lines, and boundaries of region, the main function of which is to obtain as many spatial details as possible. In our paper, a NHM based local self-adaptive fusion method is adopted in band-pass directional sub-band coefficients acquisition phase, by calculating the identical degree of the corresponding neighborhood to determine the selection for band-pass coefficients fusion rules (i.e., regional energy or global weighted).

Given that the two source images are A and B, with the same size, both have been registered, F is fused image. The fusion process is shown in Figure 2 and the steps are given as follows (Figure 2):

Adopt NSCT to implement the multi-scale and multi-direction decompositions for source images A and B, and the sub-band coefficients { C i 0 A ( m , n ) , C i , l A ( m , n ) }, { C i 0 B ( m , n ) , C i , l B ( m , n ) } can be obtained.

To verify the effectiveness of the proposed algorithm, three groups of images are used under the MATLAB 7.1 platform in this Section. All source images must be registered and with 256 gray levels. By comparison with the five typical algorithms below: NSCT-based method (M1), NMF-based method (classic NMF, M2), weighted NMF-based method (M3), PCA and wavelet, we can learn more about the one presented in our paper.

It may be possible to evaluate the image fusion subjectively, but subjective evaluation is likely affected by the biases of different observers, psychological status and even mental states. Consequently, it is absolutely necessary to establish a set of objective criteria for quantitative evaluation. In this paper, we select the Information Entropy (IE), Standard Deviation (SD), Average Gradient (AG), Peak Signal to Noise Ratio (PSNR), Q index [30], Mutual Information (MI), and Expanded Spectral Angle Mapper (ESAM) [31] as our evaluation metrics. IE is one of the most important evaluation indexes, whose value directly reflects the amount of information in the image. The larger the IE is the more information is contained in a fused image. SD indicates the deviation degree between the gray values of pixels and the average of the fused image. In a sense, the fusion effect is in direct proportion to the value of the SD. AG is capable of expressing the definition extent of the fused image, the definition extent will be better with an increasing AG value. PSNR is the ratio between the maximum possible power of a signal and the power of corrupting noise. The larger the PSNR is, the better is the image. MI is a quantity that measures the mutual dependence of the two random variables, so a better fusion effect makes for a bigger MI. Q index measures the amount of edge information “transferred” from source images to the fused one to give an estimation of the performance of the fusion algorithm. Here, larger Q value means better algorithm performance. ESAM is an especially informative metric in terms of measuring how close the pixel values of the two images are and we take the AE (average ESAM) as an overall quality index for measuring the difference between the two source images and the fused one. The higher the AE, the less the similarity of two images will be. The AE is computed using a sliding window approach, in this work, sliding windows with a size of 16 × 16, 32 × 32, and 64 × 64 are used.

A pair of “Balloon” images are chose to be source images, both are 200 by 160 in size. As can be seen from Figure 3(a), the left side of the image is in focus while the other side is out of focus. The opposite phenomenon emerges in Figure 3(b). Six variant approaches, M1–M3, PCA, wavelet (bi97), and our method, are applied to test the fusion performance. Figure 3c–h) show the simulated results.

From an intuitive point of view, the M1method produces a poor intensity that makes Figure 3(c) somewhat dim. On the contrary, the other five algorithms generate better performance in this aspect, but artifacts located in the middle right of Figure 3(e) can be found. Compared with the M2 and M3 methods, although the definition of the bottom left region in our method is slightly lower than that of the two algorithms, the holistic presentation is superior to the two. As for PCA and wavelet, the similar visual effects as Figure 3(h) are obtained, except the middle bottom balloon in Figure 3(f) is slightly blurred. Statistic results in Tables 2 and 3 verify the above visual effects further.

Table 2 illustrates that the proposed method has advantages over most of other algorithms since all the criteria, in both protection of image details and fusion of image information, are superior to that of M1–M3 and PCA. Of them, the indexes IE, SD, AG, PSNR of our method exceed those of M1, M2, M3 and PCA by 3.1%, 1.3%, 1.5% and 0.8% (for IE), 6.0%, 2.7%, 2.1% and 1.1% (for SD), 0.6%, 3.3%, 0.6% and 0.3 (for AG), 5.3%, 2.6%, 1.8% and 1.3% (for PSNR), respectively. These four basic indices indicate that our method provides a better visual effect. As for index Q, the 0.9844 value of our method means the best fusion algorithm performance when compared to values of the former four algorithms. In MI, our method is also the best, being superior to that of M1 by 19%; the latter is in effect the worst one. As for wavelet, four of six metrics are slightly inferior to ours while two of six metrics are inferior to ours. From Table 3, it can be found that our method has the lowest AE (AE_aF, AE_bF denote similarity between source image (a) and the fused one; (b) and the fused one, respectively) followed by wavelet, M3, PCA, M2, and method M1 has the highest AE. Therefore, in terms of transferring details, the performances of our method, wavelet, M3, PCA, M2, and M1 decrease.

Figure 4(a,b) are medical CT and MRI images whose sizes are 256 by 256. Six different methods, including our proposed one, are adopted to evaluate the fusion performance, and the simulated results are shown in Figure 4(c–h).

From Figure 4, images based on methods M2 and M3 are not fused well enough for the information in the MRI source image is not fully described yet. Although the external contour of M1 is clear, the overall effect is poor, which is confirmed by the low brightness of the image and the appearance of some undesirable artifacts observed on both sides of the cheek. Oppositely, PCA, wavelet and our methods not only produce distinct outlines and rationally control the brightness level, but also preserve and enhance image detailed information well. Related evaluations are recorded in Tables 4 and 5.

As revealed in Table 4, the proposed method is nearly the best based on the fact that the metrics of IE, SD, AG, PSNR of Figure 4(h) are all greater than that of the former four algorithms (percentages are not listed). The IE value of M1 is the lowest, which is precisely in accord with the image. Our method possesses an AG index of 29.209 which implies the image is clearer than images based on other approaches. In PSNR and SD, our method performs well, being second only to the wavelet approach, and the SD of M1 beats that of M2 and M3. As to Q index and MI, our method takes the second place in MI and the first place in Q, which indicates that the details and edges from source images are well inherited. These details and edges are extremely important for medical diagnosis. Like in experiment 1, our method achieves the lowest values both in AE_aF and AE_bF, and that of wavelet, M3, PCA, M2 and M1 arrange in ascending order.

A group of registered visible and infrared images with a size of 360 by 240 showing a person walking in front of a house are labeled as Figure 5(a,b).

Of these, Figure 5(a) has a clear background, but infrared thermal sources cannot be detected. Conversely, Figure 5(b) highlights the person and house but its ability to render other surroundings is weak. Effective fusions are achieved by the six methods. After concrete analysis on the six fused images, we draw the following conclusions: we can find that the image based on method M1 is the worst in overall effect, especially a dark area around the person, which is partly caused by the significant differences between two source images. Method M2 produces more smooth details than M1, as a case in point, the road on the right side of the image and the grass on the other side can easily be recognized for the enhancement of intensity. Approximate effects displayed in Figure 5(e–h) are achieved by using M3, PCA, wavelet and our method, from which we can easily distinguish most parts of the scene except the lighting beside the house in Figure 5(e) that can hardly be observed. It is difficult to judge the performances of the latter four methods through visual observation in case of the concrete data are not provided by Table 6.

In so far as IE, AG, and PSNR are concerned, the proposed technique is evidently better than the former four ones. Specially, the value of our method exceeds them by 1.6%, 4.9%, and 0.7% while the SD is slightly smaller when compared with M3. In index Q, the optimal value is obtained on the basis of the wavelet approach, while that of M1 holds the final place. As for MI, our method still ranks the first place in Table 6. Analogous effects are achieved in Table 7, statistics show that the similarities between visible light, infrared and fused images generated by our method are the best in that both AE_aF and AE_bF are the smallest.

In this section, we compare the performance of ANMF with that of algorithms presented in article [27] and [18] in order to prove its advantages. The algorithms are implemented in Matlab and applied to the Equinox face database [32]. The contrast experiments are conducted four times, where p is as described in Section 3.2 and n denotes for the number of images chosen from the face database. The Y axis of Figure 6 represents the number of iterations repeated by the three algorithms and the X axis is the time consumption scale. We choose one group of these experiments and demonstrate the results in Figure 6 with p = 100 and n = 1,000, in which algorithm in [18] is first performed for a given number of iterations and record the time elapsed and then run algorithm in [27] and our algorithm until the time consumed is equivalent to that of the former, respectively. We note that our algorithm offers improvements in all given time points, however, the relative improvement percentage of our method over other two algorithms goes down when the number of iterations increases. Actually, the performance of our method increases about 36.8%, 26.4%, 15.7%, 12.6%, 7.5% and 37.9%, 29.6%, 19.4%, 17.8%, 12.6%, respectively, when comparing with the algorithms in [27] and [18] at each of five time points. In other words, our method converges faster, especially at the early stages, but the percentage tends to decline, which implies that this attribute is merely useful for real-time applications without very large-scale data sets.

Image fusion with different models and numerical tests are conducted in our experiments, where the above four experiments indicate that the proposed method has a notable superiority in image fusion performance over the four other techniques examined (see Sections 5.2–5.4), and has better iteration efficiency (see Section 5.5). We observed that images based on wavelet and our proposed methods enjoy the best visual effect, and then the PCA, M3, M2, and M1 are the worst. In addition to visual inspection, quantitative analysis is also conducted to verify the validity of our algorithm from the angles of information amount, statistical features, gradient, signal to noise ratio, edge preservation, information theory and similarity of structure. The values in these metrics prove that the experiments achieve the desired objective.