Hyperspectral Super-Resolution Technique Using Histogram Matching and Endmember Optimization

Abstract

In most hyperspectral super-resolution (HSR) methods, which are techniques used to improve the resolution of hyperspectral images (HSIs), the HSI and the target RGB image are assumed to have identical fields of view. However, because implementing these identical fields of view is difficult in practical applications, in this paper, we propose a HSR method that is applicable when an HSI and a target RGB image have different spatial information. The proposed HSR method first creates a low-resolution RGB image from a given HSI. Next, a histogram matching is performed on a high-resolution RGB image and a low-resolution RGB image obtained from an HSI. Finally, the proposed method optimizes endmember abundance of the high-resolution HSI towards the histogram-matched high-resolution RGB image. The entire procedure is evaluated using an open HSI dataset, the Harvard dataset, by adding spatial mismatch to the dataset. The spatial mismatch is implemented by shear transformation and cutting off the upper and left sides of the target RGB image. The proposed method achieved a lower error rate across the entire dataset, confirming its capability for super-resolution using images that have different fields of view.

1. Introduction

Hyperspectral images (HSIs) are widely used in many scientific and engineering fields, because they are capable of collecting the light spectrum from a broad wavelength and saving the spectral data with high spectral resolution. Most HSI applications in scientific and engineering fields aim to identify materials in an image on the basis of the spectral characteristic of each chemical reflecting the light. HSIs have been used as a powerful tool for detecting certain materials in an object, or measuring quantitative chemical content in the fields of agriculture [1,2], environment [3,4], surveillance [5,6], biomedical imaging [7,8], and geosciences [9,10].

Low-resolution is, however, one of the shortcomings of HSIs in practice. The low-resolution results from the way in which hyperspectral cameras acquire the spectrum from light. Currently, two types of hyperspectral cameras are available—push-broom and snapshot. Push-broom cameras scan a line in an image and put the reflected light through optical spectroscopic sensors [11], while snapshot cameras use a focal-plane array to generate a two-dimensional image with a single shot [12]. Regardless of the hyperspectral camera type, all hyperspectral cameras slice the light input into small units of spectral bands to obtain fine spectral resolution, and this process inevitably leads to a low amount of energy collected by a pixel on the sensor. Therefore, pixel size of hyperspectral cameras should be larger than that of general cameras, to avoid high signal-to-noise ratios in the collected spectrum.

To overcome this innate limitation of hyperspectral cameras, much research has been conducted to increase the resolution of HSIs, by integrating other types of images. The techniques are variously referred to as hyperspectral image fusion [13,14,15], hyperspectral super-resolution (HSR) [16,17], or hyperspectral image upsampling [18]; HSR is the term used to refer to the technique in this paper. Most common HSR techniques are based on pan-sharpening, which can also be applied to RGB images. Pan-sharpening merges a low-resolution multispectral image with a high-resolution panchromatic image to create a single multispectral image with higher resolution features [19]. One of the most common types of pan-sharpening algorithms, component substitution [20], is comprised of six steps: up-sampling, alignment, forward transformation, intensity matching, component substitution, and reverse transformation. However, because HSR is an ill-posed problem, assumptions of the previous pan-sharpening techniques to resolve the ill-posedness are imperfect, which leads to the need for new assumptions, based on new perspectives [21]. HSR techniques based on deep learning have been developed in recent years, as deep learning techniques are rapidly emerging for image processing. Li et al. [22] proposed an HSR model combining a deep convolutional neural network with an extreme learning machine for HSI classification. Yuan et al. [23] trained the nonlinear relationship between low- and high-resolution images using a convolutional neural network, and proposed a collaborative non-negative matrix factorization, to enforce collaborations between the observed low-resolution HSI and the transferred high-resolution HSI.

Among various HSR techniques, those based on the linear mixing model (LMM) are widely studied by many researchers. The LMM represents an HSI as a matrix factorization of linear spectral bases, so-called endmembers, and it can solve difficult HSI fusion problems with multispectral images, including RGB images. Wycoff et al. [24] formulated an HSR problem in the form of a sparse non-negative matrix factorization, and applied alternating optimization and convex optimization solvers. Huang et al. [25] obtained a spectral dictionary using K-singular value decomposition, and computed sparse fractional abundances with a sparse coding technique of orthogonal matching pursuit. Simoes et al. [26] defined an HSR model as the standard linear inverse problem model, using a form of regularization based on vector total variation, by considering spatial and spectral characteristics of the given data. Kwon and Tai [18] proposed a two-step RGB-guided HSI upsampling scheme that consists of spatial upsampling and spectrum substitution stages. One aspect of note in this work is that the spectrum substitution stage learns a local spectral dictionary, from the superpixel neighborhood. Fang et al. [27] developed a sparse representation model based on superpixels, that learns a spectral dictionary via online dictionary learning. This method adopts joint sparse regularization to simultaneously decompose the superpixel on the transformed dictionary, to obtain the corresponding coefficient matrix.

However, in most HSR techniques, it is implicitly assumed that the mismatch between low-resolution and high-resolution images is negligible. When a hyperspectral camera is used together with an RGB camera, it is rarely possible, in practice, to match the fields of view (FOVs) of the two cameras exactly. In remote sensing, such as for satellite photography and aerial photography, the mismatch of FOVs resulting from different camera angles or lens distortion can be adjusted and made negligible because the imaged objects are far enough from the cameras. However, if imaged objects are close to the cameras, it is difficult to avoid these mismatches between images. Furthermore, the two cameras have different sensitivities, even in the same light bandwidths, owing to differences in imaging sensors and filming mechanisms.

HSR techniques based on the LMM potentially diverge in the process of finding the coefficient if there is a significant mismatch between images. This divergence problem occurs when an HSI and other RGB or multispectral images do not perfectly fit each other at the pixel level, because most of the linear-mixing-model-based HSR techniques simultaneously optimize the coefficient towards the pixel location of both the HSI and RGB or multispectral image. Lanaras et al. [28] formulated the super-resolution problem as a searching procedure for an image that has both high-spatial and high-spectral resolution. The high-resolution HSI is defined as a factorization of a matrix of endmembers and a matrix of per-pixel abundances. A low-resolution HSI is reconstructed by multiplying a spatial downsampling matrix, and a high-resolution RGB image is reconstructed by multiplying a spectral downsampling matrix. Then, the differences between reconstructed images and original images are minimized using a gradient descent optimization algorithm. However, if the input HSI and the RGB image are mismatched, the high-resolution HSI diverges, because it is impossible for the per-pixel abundance matrix to satisfy pixel locations of both images.

The divergence problem has to be overcome to develop an HSI–RGB image integrated system that can be used to take an image of a close object. We faced this problem while developing an HSI–RGB image integrated system detecting chemical leaching on the surface of a concrete structure. The features of chemical leaching on the concrete surface were too obscure to be detected, because of the low resolution of the HSI. One of the methods used to increase the resolution is to gain access closer to the concrete surface, but most concrete structures are too far away to be accessed. Therefore, it is necessary to develop an HSI–RGB image integrated system that can reconstruct a high-resolution HSI from an RGB image taken with different camera sensitivities and from different FOVs.

In this paper, we propose an HSR method that uses an HSI and an RGB image taken from a different FOV. This method mainly uses histogram matching and the endmember abundance optimization process proposed in [28]. First, a low-resolution RGB image is obtained, by applying spectral sensitivity of an RGB camera to a low-resolution HSI. The acquired low-resolution RGB image contains the color distribution of the low-resolution HSI. Next, histogram equalization is used to match the color distribution of the low-resolution RGB image and the high-resolution RGB image, taken from a slightly different viewing angle with the low-resolution HSI. Finally, the spatial endmember abundance of the high-resolution HSI is matched to the spatial information obtained from the high-resolution RGB image.

The proposed method can be described as a rearranging of the endmember of the HSI to fit into the pixel location of the high-resolution RGB image. Because there is no constraint on the pixel location of the low-resolution HSI, the proposed method can build a high-resolution HSI only if the endmember combination of both images is similar. The performance of the proposed method is evaluated by field experiments comparing the spectrum of the same material in the low- and high-resolution HSIs using a spectral angle mapper.

2. Problem Formulation

The goal of the HSR technique is to search for an image that has both high spectral and high spatial resolution. The image is formulated as $Z\in {\mathbb{R}}^{\mathrm{B}\times \mathrm{W}\times \mathrm{H}}$ with B, W, and H representing the number of spectral bands, width and height of the image, respectively. The HSR task of this research takes two inputs, an HSI $H\in {\mathbb{R}}^{\mathrm{B}\times \mathrm{w}\times \mathrm{h}}$ with high spectral resolution and low spatial resolution (w < W and h < H) and an RGB image $I\in {\mathbb{R}}^{\mathrm{b}\times \mathrm{W}\times \mathrm{H}}$ with low spectral resolution and high spatial resolution (b < B). According to the LMM of Lanaras et al. [28], the intensities $z\in {\mathbb{R}}^B$ at a given pixel of Z are described by an additive mixture:

$$z={\sum }_{j=1}^p{e}_j{a}_j,Z=EA$$

(1)

with a matrix $E\in {\mathbb{R}}^{\mathrm{B}\times \mathrm{p}}$ of endmembers and a matrix $A\in {\mathbb{R}}^{\mathrm{p}\times \mathrm{W}\times \mathrm{H}}$ of per-pixel abundances. By this definition, most p endmembers (materials) are present in the image. The endmembers E act as non-orthogonal bases to represent Z in a lower dimensional space ${\mathbb{R}}^{\mathrm{p}}$ with rank {Z} ≤ p.

The actual high-resolution RGB image I is a spectrally downsampled version of Z:

$$I=RZ=REA$$

(2)

where $R\in {\mathbb{R}}^{\mathrm{b}\times \mathrm{B}}$ is the spectral response function of the sensor. The spatial response function S of the hyperspectral camera and the spectral response function R of the conventional camera form part of the camera specifications and are assumed to be known. The spectral response function of the Nikon D700 offered in the work of Jun et al. [29] is used as R for this research.

The low-resolution HSI, H, is defined as:

$$\mathbf{H}={\mathbf{EA}}^{\prime }$$

(3)

where $A^{\prime }\in {\mathbb{R}}^{\mathrm{p}\times \mathrm{w}\times \mathrm{h}}$ is a low-resolution matrix of per-pixel abundances. The reason the per-pixel abundances of H are defined not as a factorization of EA and spatial downsampling operator, but as another form of matrix, is that the pixel location of the super-resolution result is not the same as that of H.

The low-resolution RGB image L is defined as:

$$\mathbf{L}={\mathbf{REA}}^{\prime }$$

(4a)

which represents the factorization of the spectral response function R and the low-resolution HSI, H. The constraint condition of this paper follows that of Lanaras et al. [28]. The main assumption of the constraint is that the endmembers are reflectance spectra of individual materials, and the abundances are proportions of those endmembers. As a consequence, the factorization is subject to the following constraints:

$$a_{ij}\ge 0 \forall i, j (\mathrm{non}-\mathrm{negative}\mathrm{abundance})$$

(4b)

$$1^{⊺}\mathrm{A}=1^{⊺}\left(\mathrm{abundances}\mathrm{sum}\mathrm{to}1\right)$$

(4c)

with $a_{ij}$ the elements of A. Here 1 denotes a vector of 1s compatible with the dimensions of A.

3. Proposed Solution

3.1. Overall Scheme

In this paper, we propose a solution to the problem formulated in Section 2, using histogram matching and the endmember abundance optimization process proposed in Lanaras et al. [28]. As briefly explained in Section 1, [28] defined a high-resolution HSI as a factorization of spectral endmembers and its per-pixel abundance. The authors of [28] find a high-resolution HSI by optimizing the spectral endmembers to a low-resolution HSI, and the per-pixel endmember abundances to a high-resolution RGB, image. The optimization step for endmember abundance is adopted in this method to find the per-pixel endmember abundance. The overall framework of the proposed solution is displayed in Figure 1. The HSR procedure proposed in this paper consists of three steps:

Step 1. Creating an RGB image from a low-resolution HSI, using a spectral response function of a commercial RGB camera.

Step 2. Matching the histogram of a high-resolution RGB image with that of a low-resolution RGB image, obtained in Step 1.

Step 3. Optimizing endmember abundance of a high-resolution HSI toward the histogram-matched high-resolution RGB image obtained in Step 2.

In Step 1, a low-resolution RGB image is reconstructed from a low-resolution HSI by factorizing a spectral response function of a commercial RGB camera. The reconstructed low-resolution RGB image is used as a color distribution standard point for both the low-resolution HSI and high-resolution RGB image. In Step 2, the histogram of the high-resolution RGB image is matched with the low-resolution RGB image obtained in Step 1. Since the high-resolution RGB image has the same color distribution as the low-resolution RGB image, it is less necessary to match the color distribution using an optimization process. In Step 3, the per-pixel endmember abundance optimization process of [28] is adopted, to match the spatial distribution of endmember abundance between the high-resolution HSI and the high-resolution RGB image. The per-pixel endmember abundance optimization is performed by reducing the subtraction between the target high-resolution RGB image and the high-resolution RGB image obtained from the high-resolution HSI. Because this optimization process disregards the objective function, representing the difference between low- and high-resolution HSIs, the endmember abundance freely converges to the pixel location of the high-resolution RGB image.

Each part of the proposed method is formulated as explained below. Step 1 factorizes the spectral response function of a RGB camera with a low-resolution HSI to reconstruct a low-resolution RGB image. An RGB image is reconstructed the equation formulated in [29]:

$${\mathit{I}}_{ch}={{\displaystyle \int }}_{400nm}^{720nm}{S}_{ch}\left(\lambda \right)L\left(\lambda \right)R_x\left(\lambda \right)d\lambda$$

(5)

where $S_{ch}\left(\lambda \right) (ch=\mathrm{R},\mathrm{G},\mathrm{B}$) is the spectral response function of an camera for each channel of RGB, $L\left(\lambda \right)$ is spectral power distribution of an illuminant and $R_x\left(\lambda \right)$ is a spectral reflectance of a point in an image. Equation (5) can also be written in a matrix form, for a discrete summation along the spectral range of a hyperspectral camera with a given bandwidth. Step 2 matches the histogram of an RGB image to that of a low-resolution RGB image L, obtained from an HSI. The histogram-matching process is a minimization of grayscale transformation T in the following equation:

$$\underset{T}{\min }\left|c_1\left(T\left(g\right)\right)-c_0\left(k\right)\right|$$

(6)

where $g$ is a specific index on a gray scale, $c_0$ is the cumulative distribution function of L’s histogram and $c_1$ is the cumulative distribution function of I’s histogram for all intensities k on a gray scale. Since the histogram equalization is defined on a gray scale, it has to be iteratively performed on each channel of a RGB image. T is a function that finds the index on $c_0$ that has the value most similar to the value of $c_1$ at a particular index $g$ on grayscale. After the function T is defined for g on all intensities on a gray scale, histogram equalization is performed by finding and mapping a value corresponding to L in the input image I, using T. In Step 3 of the proposed algorithm, an estimate of Z, or equivalently E and A, is needed. From the given super-resolution problem, the following constrained least-squares problem is formulated as:

$$\arg \underset{\mathrm{A}}{\min }{\parallel \mathbf{I}-\mathbf{REA}\parallel }_{\mathrm{F}}^2$$

(7a)

$$\mathrm{subject}\mathrm{to} a_{ij}\ge 0 \forall i, j$$

(7b)

with ${\parallel ·\parallel }_F$ denoting the Frobenius norm, and $a_{ij}$ is elements of A and constrained to non-negative values.

3.2. Overall Algorithm and Implementation

The method proposed in Section 3.1 proceeds with the following procedure, as described in

Table 1. Overall algorithm for the proposed solution.

Requires:  H (low-resolution hyperspectral image)  I (high-resolution RGB image)  C (RGB camera sensitivity)  S (upsampling rate)

Reconstruct L by applying C to H Match histogram of I to that of L Initialize $E^{\left(0\right)}$ with SISAL and $A^{\prime \left(0\right)}$ with SUnSAL from H Initialize $A^{\left(0\right)}$ by upsampling $A^{\prime \left(0\right)}$ with S k ← 0 while not converged do   k ← k + 1   $A^{\left(k\right)}$ ← $A^{\left(k-1\right)}S$; Estimate $E^{\left(k\right)}$ with (8a) and (8b) end while return $Z=E^{\left(k\right)}{A}^{\left(k\right)}$

. The algorithm requires H and I, which are low-resolution HSI and high-resolution RGB images, respectively. Because Z will be reconstructed using endmember abundance with the same resolution as I, the resolution of I has to be an integer multiple of the resolution of H with upsampling rate S. Additionally, RGB camera sensitivity C is required to reconstruct an RGB image from H.

The proposed method begins by constructing a low-resolution RGB image L from H, using C. The camera sensitivity for each RGB channel is multiplied by the spectral information of H of each pixel, and normalized into an 8-bit-precision RGB image L, for further histogram equalization with I. Then, the histogram of I is matched with that of L, so that I has the same color distribution as L. The histogram equalization is performed using a MATLAB built-in function [30]. The next step is to find the initial values, $E^{\left(0\right)}$ and $A^{\left(0\right)}$, to optimize Z, which consists of the endmember vector E and the per-pixel abundance vector A. Simplex identification via split augmented Lagrangian (SISAL) [31] initializes endmember E and sparse unmixing by variable splitting and augmented Lagrangian (SUnSAL) [32] initializes A’, respectively. SISAL is an algorithm for unsupervised hyperspectral linear unmixing and finds the minimum volume simplex containing the hyperspectral vectors, by augmented Lagrangian optimizations. SUnSAL is an eigen decomposition-based hyperspectral unmixing algorithm. The MATLAB code for SISAL and SUnSAL is available at the author’s webpage [33]. The low-resolution per-pixel abundance A’ is upsampled with S and will be used as the initial point of the low-resolution step. The optimization is performed with a projected gradient method for 7a. The equation for the projected gradient method is:

$${\mathbf{U}}^q = {\mathbf{E}}^{q-1}-\frac{1}{c}\left({\mathbf{E}}^{q-1}\mathbf{A}-\mathbf{H}\right){\mathbf{A}}^{⊺}$$

(8a)

$${\mathbf{E}}^q={\mathrm{prox}}_{\mathbf{E}}\left({\mathbf{U}}^q\right)$$

(8b)

where $\mathrm{c}={\mathsf{\gamma }}_2\parallel A{A}^{⊺}{\parallel }_{\mathrm{F}}$ is a nonzero constant and ${\mathrm{prox}}_E$ is a proximal operator that is constrained to 7b. The optimization procedure is repeated for $\mathrm{q}=1, 2, \dots$ until convergence, or until the pre-determined error rate, calculated by an error metric, is reached.

4. Experiment

4.1. Baseline Study for Spatial Information Mismatch

To confirm the limitation of current methods under color space difference and pixel mismatch, we applied the method of Lanaras et al. [28] to the cases where pixel mismatch exists, and color spectral functions are not identical for H and I. The method of Lanaras et al. implicitly assumes that there is no pixel mismatch or color difference between high resolution RGB image and low resolution HSI, and the effects of this assumption are investigated using images with pixel mismatch and color difference. A public hyperspectral database, called the Harvard dataset [34], was used for the evaluation of the proposed algorithm. Because the purpose of the Harvard dataset [34] is to establish the basic statistical structure of HSIs of real-world scenes, the dataset is in accord with the condition in which the proposed algorithm will be used. The Harvard dataset [34] has 50 indoor and outdoor images recorded under daylight illumination, and 27 images recorded under artificial or mixed illumination. The spatial resolution of the images is 1392 × 1040 pixels, with 31 spectral bands of width 10 nm, from 420 to 720 nm. The original HSIs are used as ground truth for the evaluation.

First, the upper and left sides of the original HSIs (Figure 2a) were cut off by 50 pixels in both horizontal and vertical directions to form the pixel mismatch dataset (Figure 2b). Comparing Figure 2a,b, it can be observed that the location of the window on the left side is slightly translated, potentially due to the lower per-pixel performance of the super-resolution process. Second, RGB images were reconstructed as displayed in Figure 2c, using a color spectral function, Nokia N900, which was different from the one used in optimization. The camera sensitivities of the Nokia N900 and Nikon D700 are displayed in Figure 3. Figure 2c has relatively brighter color space compared to Figure 2a, which causes low spectral accuracy of a super-resolution result. The maximum RGB value of the test was normalized to 1, which is a common image processing technique for enhancing the visibility of a reconstructed image. The experiments were conducted with the same implementation details as in [28] for both datasets, but the operation was forced to run 1500 iteration for the 11 selected images. For a fair comparison between the proposed algorithm and other methods, the same error rate method was used as the primary metric. The datasets were tested using the root–mean–square error (RMSE) of the estimated high-resolution HSI Z, with respect to the ground truth image $\widehat{\mathrm{Z}}$:

$$\mathrm{RMSE}=\sqrt{\frac{1}{B{N}_m}\sum \sum {\left({\widehat{z}}_{ij}-z_{ij}\right)}^2}=\sqrt{\frac{{\parallel \widehat{Z}-Z\parallel }_F^2}{B{N}_m}}$$

(9)

A spectral angle mapper (SAM) [35] was also adopted; it is defined as the angle in ${\mathbb{R}}^B$ between the estimated pixel ${\widehat{z}}_{ij}$ and the ground truth pixel $z_{ij}$, averaged over the entire image and given in degrees:

$$\mathrm{SAM}=\frac{1}{N_m}\sum \arccos \frac{{{\widehat{z}}_{ij}}^{⊺}{z}_{ij}}{{\parallel {\widehat{z}}_{ij}\parallel }_2{\parallel z_{ij}\parallel }_2}$$

(10)

where ${\parallel \cdot \parallel }_2$ is the $l_2$ vector norm.

The test results were tabulated in

Table 2. Results for the Harvard database.

Method	RMSE		SAM
	Average	Median	Average	Median
Original results for the entire dataset	1.7	1.5	2.9	2.7
Translation by cutting off	11.6	10.36	7.72	8.11
Histogram not matched	8.88	8.55	8.06	7.15

for comparison with the results of [28]. The algorithm of [28] showed accurate results compared to other methods, such as those in [24,26,36,37], and [38], but the experimental results indicate that a small pixel mismatch and color difference would lead to a higher error rate. This phenomenon is theoretically inevitable for algorithms that optimize the high-resolution HSI for both the RGB image and low-resolution HSI. For instance, the low-resolution step of [28], minimizing the objective function of endmembers in the image, is highly governed by the abundance of the high-resolution HSI. Therefore, inputting two images with pixel mismatch, as displayed in Figure 2a,b, means optimization toward two different points, and inevitably results in poor accuracy. This phenomenon appears in Figure 4a, which shows an error rate of > 10% where the spatial mismatch is significant. Color difference is another factor that should be considered in super-resolution problems. The high-resolution step requires an accurate spectral sensitivity function to recover an RGB image close to the input RGB image. However, the sensitivity of the hyperspectral camera usually differs from that of the RGB camera, as displayed in Figure 2a,c, complicating the initialization of the color space between the RGB image and HSI. As displayed in Figure 4b, the color distribution difference caused significant errors in parts of the HSI. The method is also inefficient in finding the spectral function of the RGB camera every time the input RGB image is changed. Therefore, poor results in both the pixel mismatch and color difference dataset indicate the need for the algorithm to account for these issues in the super-resolution problem.

4.2. Proposed Method Evaluation

To test the performance of the proposed algorithm under a condition in which a mismatch between a low-resolution image and a high-resolution image is significant, the original HSIs were edited in two ways. To construct the first dataset, the upper left corner of the original HSIs was cut off for the specific number of pixels, increasing in 20-unit increments from 20 to 100 pixels, as represented in Figure 5a. For the second dataset, an affine transformation was applied, as displayed in Figure 5b, to make horizontally sheared images from the pixels not cut off from the original images. The affine matrix for the shear transformation used in this research was:

$$\left[\begin{array}{ccc}1& C_x& 0\\ 0& 1& 0\\ 0& 0& 0\end{array}\right]$$

(11)

The numbers used for the constants $C_x$ in the shear transformation matrix are listed in

Table 3. Results for the Harvard database.

Method of Field of View Mismatch		Lanaras et al. [28]				Proposed Method
		RMSE		SAM		RMSE		SAM
		Average	Median	Average	Median	Average	Median	Average	Median
Without Transformation		2.68	2.11	5.58	5.47	2.68	2.57	6.69	5.32
Shear Transformation Using Affine Matrix	$C_x=$0.1	9.79	7.70	7.16	7.35	3.64	2.96	6.73	6.03
	$C_x=$0.2	11.76	9.33	7.88	8.00	4.4	3.56	6.9	6.26
	$C_x=$0.3	12.89	10.62	8.37	8.68	5.19	4.30	7.2	6.18
Translation by Cutting off Upper and Left Sides	$P_n= 20$	9.12	6.93	6.71	6.83	3.56	2.87	6.65	5.65
	$P_n= 40$	10.95	8.17	7.31	7.40	4.25	3.27	6.86	5.80
	$P_n= 60$	12.02	9.27	7.80	8.03	4.93	3.68	7.08	5.98
	$P_n= 80$	12.81	9.99	8.24	8.52	5.60	4.13	7.24	6.20

.

We ran our method with the maximum number of endmembers set to p = 10, which is sufficient for both datasets. The inner loops of optimization steps (8a) and (8b) were run for 5000 iterations. Operation times depended on the image size and the number of iterations. For a 1392 × 1040 pixel image with 31 channels, it took ~660 s on a single Intel i7-7700, operating at 3.60 GHz with GPU computation on a GTX 1080Ti (11 GB). The ratio between low-resolution image and high-resolution image is set to 1:8.

Table 3 compares the average and median RMSE and SAM values by the method of Lanaras et al. [28], with those by the proposed method for the eight cases, using images with and without geometric transformation. For all of the experimental cases, the color difference is applied. For the ideal images without any transformation, the method of Lanaras et al. [28] shows slightly lower RMSE and SAM values than the proposed method. If any transformation is applied to the images, however, the method of Lanaras et al. [28] results in significantly increased errors. With the small shear deformation ($C_x=$0.1), the average RMSE and SAM values increase from 2.68 to 9.79, and 5.58 to 7.16, respectively. With the small cut-off of images ($P_n=$20), the average RMSE and SAM values increase from 2.68 to 9.12, and 5.58 to 6.71, respectively. On the other hand, the proposed method results in slightly increased errors with the transformation. With the small shear deformation ($C_x=$0.1), the average RMSE and SAM values increase from 2.68 to 3.64, and 6.69 to 6.73, respectively. With the small cut-off of images ($P_n=$20), the average RMSE values increase from 2.68 to 3.56, and the average SAM values change negligibly, from 6.69 to 6.65. Even with larger transformations, the proposed method results in significantly smaller RMSE and SAM values than the method by Lanaras et al. [28]. The results quantitatively confirm that the proposed algorithm can solve the super-resolution problem of an HSI, where an HSI and a target RGB image have different fields of view.

To qualitatively evaluate the performance of the proposed method, we verified the performance of the per-pixel error rate image and the RGB image reconstructed from the high-resolution HSI, as shown in Figure 6. Figure 6a,c show the per-pixel RMSE and reconstructed RGB images of the high-resolution HSI of super-resolution in an environment where the shear transformation for $C_x$ was 0.3. Figure 6b,d show the per-pixel RMSE and reconstructed RGB images of high-resolution HSI of super-resolution, in an environment where $P_n$ was 80. Comparing Figure 6a,b with the per-pixel error image in Figure 4, we can observe that the error rate for the overall image has been significantly reduced. In particular, the phenomenon where two images are not merged into one image by the pixel mismatch in Figure 4a is significantly reduced in Figure 6a,b. Also, this phenomenon is hardly observed in the reconstructed RGB images of Figure 6c,d, which confirms that removing the high-resolution step in [28] gives a better performance in pixel-mismatch environments, by relaxing the spatial constraint on two images with different FOVs. Also, we showed that the color distribution difference that can be caused by removing the high-resolution step in [28] can be reduced by using histogram equalization. However, further research might be needed to study how to update the endmembers, even in an environment of pixel mismatch, using additional optimization techniques.

5. Conclusions

A new approach for hyperspectral super-resolution for an HSI and an RGB image taken with different camera sensitivities and fields of view is proposed. Matching different camera sensitivities and fields of view is a challenging task, that must be overcome to develop an HSI–RGB image integrated system for environments where this issue is inevitable. The proposed method employs a modification of the work of [28] to solve the super-resolution problem in an environment where an HSI and a target RGB image have slightly different fields of view. The basic principle of the proposed method is to obtain endmembers from the given his, and to optimize the per-pixel abundance with RGB images, for which the histogram is matched to that of the RGB image reconstructed from the HSI. The endmember optimization term between the low-resolution HSI and the high-resolution HSI is removed, to prevent divergence caused by different per-pixel abundances of the low-resolution HSI and the high-resolution RGB. A histogram-matching step between the given RGB image and the RGB image reconstructed from the HSI is added, to reduce the color difference caused by different camera sensitivities and the potential accuracy decrease caused by removal of the endmember optimization term.

We first conducted two experiments to confirm the effect of different camera sensitivities and fields of view. The datasets for the experiments were constructed by editing 11 selected images from the Harvard dataset, which is comprised of basic statistical structures of HSIs of real-world scenes. To confirm the effect of the color difference, the input high-resolution RGB image is reconstructed using camera spectral functions different from the one used for the optimization process. To confirm the effect of the different fields of view, the upper and left sides of the input high-resolution RGB image are cut off by 50 pixels. The experimental results demonstrate that these differences may lead to a high error rate in the super-resolution problem. The performance of the proposed method was also tested on the Harvard dataset. The experimental dataset was constructed by adding two types of geometric transformation, warping and transition, to the reconstructed RGB image, using camera spectral functions different from the one used for the optimization process. The low error rate of the experimental results in RMSE and SAM demonstrates that the proposed algorithm is applicable in environments with pixel mismatching and color differences.