Reproducibility of Pansharpening Methods and Quality Indexes versus Data Formats

Abstract

In this work, we investigate whether the performance of pansharpening methods depends on their input data format; in the case of spectral radiance, either in its original floating-point format or in an integer-packed fixed-point format. It is theoretically proven and experimentally demonstrated that methods based on multiresolution analysis are unaffected by the data format. Conversely, the format is crucial for methods based on component substitution, unless the intensity component is calculated by means of a multivariate linear regression between the upsampled bands and the lowpass-filtered Pan. Another concern related to data formats is whether quality measurements, carried out by means of normalized indexes depend on the format of the data on which they are calculated. We will focus on some of the most widely used with-reference indexes to provide a novel insight into their behaviors. Both theoretical analyses and computer simulations, carried out on GeoEye-1 and WorldView-2 datasets with the products of nine pansharpening methods, show that their performance does not depend on the data format for purely radiometric indexes, while it significantly depends on the data format, either floating-point or fixed-point, for a purely spectral index, like the spectral angle mapper. The dependence on the data format is weak for indexes that balance the spectral and radiometric similarity, like the family of indexes, Q2ⁿ, based on hypercomplex algebra.

1. Scenario and Motivations

Among remote sensing image fusion techniques, panchromatic (Pan) sharpening, or pansharpening, of multispectral (MS) images has recently received considerable attention [1,2]. Pansharpening techniques take advantage of the complementary characteristics of spatial and spectral resolutions of MS and Pan data, originated by constraints on the signal-to-noise ratio (SNR) of broad and narrow bands, to synthesize a unique product that exhibits as many spectral bands as the original MS image, each with same spatial resolution as the Pan image.

After the MS bands have been interpolated and co-registered to the Pan image [3], spatial details are extracted from Pan and added to the MS bands according to the injection model. The detail extraction step may follow the spectral approach, originally known as component substitution (CS), or the spatial approach, which may rely on multiresolution analysis (MRA), either separable [4] or not [5]. The dual classes of spectral and spatial methods exhibit complementary features in terms of tolerance to spatial and spectral impairments, respectively [6,7].

The Pan image is preliminarily histogram-matched, that is, radiometrically transformed by a constant gain and offset, in such a way that its lowpass version exhibits mean and variance equal to those of the component that shall be replaced [8]. The injection model rules the combination of the lowpass MS image with the spatial detail of Pan. Such a model is stated between each of the resampled MS bands and a lowpass version of the Pan image having the same spatial frequency content as the MS bands; a contextual adaptivity is generally beneficial [9]. The multiplicative injection model with haze correction [10,11,12,13] is the key to improving the fusion performance by exploiting the imaging mechanism through the atmosphere. The injection model is crucial for multimodal fusion, where the enhancing and enhanced datasets are produced by different physical imaging mechanisms, like thermal sharpening [14] and optical and SAR data fusion [15]. In the latter case, since measures of spectral reflectance and radar reflectivity of the surface cannot be directly merged, the optical views are enhanced by means of noise-robust texture features extracted from the SAR image [16].

Remote sensing image data are generally available in packed fixed-point formats, together with floating-point gains and offsets, a pair for each band of each scene, that allow floating-point calibrated values to be recovered [17]. While the maximum value of each band of the scene is mapped onto the largest digital number (DN) of the fixed-point representation, offsets are generally set equal to the minimum value, such that the active range of floating-point values of the scene is exactly mapped onto the dynamic range of the DN representation. If the offsets are taken all equal to zero, the DN and the floating-point representations differ only by a scaling factor, which, however, may be different from one band to another in the same scene, thereby originating an alteration in the spectral content of the data.

A problem seldom investigated in the literature is whether it makes difference if fusion is accomplished in the packed DN format or in the original floating-point format. In this work, we are concerned with how the performance of pansharpening methods depends on their input data format. It is theoretically proven and experimentally demonstrated that MRA methods are unaffected by the data format, which instead is crucial for CS methods, unless their intensity component is calculated by means of a multivariate linear regression between the upsampled bands and the lowpass-filtered Pan, as it is accomplished by the most advanced CS methods, e.g., [18].

Quality assessment of the pansharpened images is another debated problem. Notwithstanding achievements over the last years [19,20,21,22,23,24,25], the problem is still open, being inherently ill-posed. A further source of uncertainty, which has been explicitly addressed very seldomly [26], is that also the measured quality may depend on the data format. The quality check often entails the shortcoming of performing fusion with both MS and Pan datasets degraded at spatial resolutions lower than those of the originals, in order to use non-degraded MS originals as quality references [27]. In this study, several widespread with-reference (dis)similarity indexes are reviewed and discussed in terms of the reproducibility of their output values towards the data format. We wish to stress that in the present context the term quality represents the fidelity to a hypothetically available reference, and has no relationship with the intrinsic quality of the data produced by the instrument, which mainly depends on the modulation transfer function (MTF) of the multiband system and on the SNR, due to a mixed noise model, both photon and electronic [28,29].

In an experimental setup, GeoEye-1 and WorldView-2 data are either fused in their packed 11-bits DN format or converted to spectral radiance before fusion is accomplished, by applying gain and offset metadata. In the former case, fusion results are converted to spectral radiance before quality is measured. In the latter case, fusion results are preliminarily converted to DNs. Results exactly match the theoretical investigations. For the majority of CS fusion methods, which do not feature a regression-based intensity calculation, results are better whenever they are obtained from floating-point data. Furthermore, the assessment of nine pansharpened products from as many algorithms, carried out both on floating-point and on packed fixed-point data, reveals that the quality evaluations may be misleading whenever they are performed on fixed-point DN formats.

2. Data Formats and Products

Remote sensing optical data, specifically MS and Pan, are available in spectral radiance format, that is radiance normalized to the width of the spectral interval of the imaging instrument, and in reflectance formats. The reflectance, which is implicitly spectral, ranges in [0,1] and can be defined either as top-of-atmosphere (TOA) reflectance or as surface reflectance, also called bottom-of-atmosphere (BOA) reflectance, if it is measured at sea level. The former is the reflectance as viewed by the satellite and is given by the TOA spectral radiance rescaled by the TOA spectral irradiance of the Sun. The latter represents the spectral signature of the imaged surface; its determination requires the estimation, through parametric modeling and/or measurements, of the upward and downward transmittances of the atmosphere and of the upward scattered radiance at TOA, also known as path radiance [30].

Besides the TOA spectral radiance, which is a level-one (L1) product, the TOA reflectance, another L1 product, is generally available for systems featuring a nadiral acquisition and a global Earth coverage, like ASTER, Landsat 7 ETM+, Landsat 8 OLI, and Sentinel-2. On the contrary, extremely high resolution (EHR) systems (IKONOS, QuickBird, GeoEye-1, WorldView-2/3/4, Pléiades 1AB) perform sparse acquisitions with maneuverable acquisition angles and may not have the TOA reflectance format available. The surface reflectance is a level-two (L2) product and is distributed for global-coverage systems (OLI, Sentinel-2), only where an instrument network is available for atmospheric measurements [31], though it has been recently shown that robust estimates of surface reflectance can be achieved through the same data that shall be atmospherically corrected [32].

In order to store and distribute fixed-point data (typically 8 to 16 bits per pixel per band), more compact and practical than floating-point data, the spectral radiance/reflectance values are rescaled to fill the 256 to 65,536 digital number (DN) counts of the binary representation. A negative bias may be introduced to force the minimum radiance/reflectance value in the zero DN. If p denotes the wordlength of the fixed point representation and $L_{max}$ ($L_{min}$) the maximum (minimum) spectral radiance value, the conversion rule to DN is:

$$\mathrm{DN}=⌊\frac{L-L_{min}}{L_{max}-L_{min}}\times (2^p-1)+0.5⌋.$$

(1)

For each band, the reciprocal of the scaling factor, i.e. $(L_{max}-L_{min})/(2^p-1)$, and the bias changed of sign, i.e., $L_{min}$, which are generally different for each band, are placed in the file header as gain and offset metadata and are used to restore calibrated floating-point values from the DNs, which are identical for the three formats, spectral radiance, and reflectances; only gains and offsets change. In some cases, the offsets are set equal to zero, for all bands, including Pan, regardless of the actual minimum, $L_{min}$, which implies that the minimum DN may be greater than zero. This strategy is generally pursued when the wordlength of the packed DN is 11 bits or more. In the following of this study, we will show that such a choice is highly beneficial for the reproducibility of pansharpening methods and quality indexes.

We wish to remark that the packaging of floating-point data into DNs does not penalize the original precision of the calibrated data, which are obtained starting from the integer samples produced by the on-board analog-to-digital converter (ADC). Thus, the calibrated samples exhibit a finite number of floating-point values, which can be accommodated in a DN of suitable wordlength, comparable to that of the ADC; generally one bit less, because the ADC span is designed to encompass the dark signal, which is removed before calibration, and to leave an allowance to prevent saturation.

Eventually, for the kth spectral channel, the following relationships hold between the floating-point calibrated data, TOA spectral radiance, $L^{\mathrm{TOA}}$, TOA reflectance, ${\rho }^{\mathrm{TOA}}$, and surface reflectance, ${\rho }^{\mathrm{BOA}}$, and the packed fixed-point DN format that is distributed:

$$\begin{array}{cc}\hfill L_k^{\mathrm{TOA}}=& {\alpha }_k·{\mathrm{DN}}_k+{\beta }_k\hfill \end{array}$$

(2a)

$$\begin{array}{cc}\hfill {\rho }_k^{\mathrm{TOA}}=& {\alpha }_k^{\mathrm{TOA}}·{\mathrm{DN}}_k+{\beta }_k^{\mathrm{TOA}}\hfill \end{array}$$

(2b)

$$\begin{array}{cc}\hfill {\rho }_k^{\mathrm{BOA}}=& {\alpha }_k^{\mathrm{BOA}}·{\mathrm{DN}}_k+{\beta }_k^{\mathrm{BOA}}\hfill \end{array}$$

(2c)

in which the ${\mathrm{DN}}_k$s are the same for the three formats, while the gains, ${\alpha }_k$, and the offsets, ${\beta }_k$, are constant over the scene and variable from one band to another, including Pan. In the case of TOA reflectance, ${\alpha }_k^{\mathrm{TOA}}$ and ${\beta }_k^{\mathrm{TOA}}$ are equal to ${\alpha }_k$ and ${\beta }_k$ divided by the the solar irradiance, which cannot be assumed spatially constant if the scene is large. Therefore, ${\alpha }_k^{\mathrm{TOA}}$ and ${\beta }_k^{\mathrm{TOA}}$ are not constant over the whole scene, but on the sub-scenes of a partition, e.g. in square blocks. Analogous considerations hold for ${\alpha }_k^{\mathrm{BOA}}$ and ${\beta }_k^{\mathrm{BOA}}$. Without loss of generality, hereafter we will consider only the TOA spectral radiance in Equation (2a), which will be simply referred to as spectral radiance (SR). Figure 1a,b show the meaning of $\alpha$ and $\beta$: $\alpha$ is the slope and $\beta$ the intercept of the inverse trans-characteristic that maps DN counts onto SR values. For the direct conversion in Figure 1a the slope is $1/\alpha$ and the intercept $-\beta /\alpha$. So, if gains and offsets are different for the bands of an MS image, including Pan, Figure 1c shows that the same DN value is mapped onto different values of SR depending on the band. Thus, there is a spectral alteration, a sort of miscalibration occurring if the DN counts are not converted back to physical units before their use. As it clearly appears from Figure 1c, such a miscalibration vanishes if gains and offsets are identical for each band, including Pan.

3. Basics of CS and MRA Pansharpening

Classical pansharpening methods can be divided into CS, MRA, and hybrid methods. The unique difference between CS and MRA is the way to extract the Pan details, either by processing the stack of bands along the spectral direction or in the spatial domain. Hybrid methods, e.g., [33,34], are the cascade of CS and MRA, either CS followed by MRA or, more seldom, MRA followed by CS. In the former case, they are equivalent to MRA that inherits the injection model from the CS; in the latter case, they behave like CS, with the injection model borrowed from MRA [8]. Therefore, at least for what concerns the present analysis, hybrid methods are not a third class with specific properties. The notation used in this paper will be firstly shown. Afterward, a brief review of CS and MRA will follow.

3.1. Notation

The math notation used is detailed in the following. Vectors are indicated in bold lowercase (e.g., $\mathbf{x}$) with the i-th element indicated as $x_i$. 2-D and 3-D arrays are denoted in bold uppercase (e.g., $\mathbf{X}$). An MS image $\mathbf{M}={\left\{{\mathbf{M}}_k\right\}}_{k=1,\dots ,N}$ is a 3-D array composed by N spectral bands indexed by the subscript k. Accordingly, ${\mathbf{M}}_k$ indicates the kth spectral band of $\mathbf{M}$. The Pan image is a 2-D matrix indicated as $\mathbf{P}$. The MS interpolated and pansharpened MS bands are denoted as ${\tilde{\mathbf{M}}}_k$ and ${\hat{\mathbf{M}}}_k$, respectively. Unlike conventional matrix product and ratio, such operations are intended as product and ratio of terms of the same positions within the array.

3.2. CS

The class of CS, or spectral, methods is based on the projection of the MS image into another vector space, by assuming that the forward transformation splits the spatial structure and the spectral diversity into separate components. The problem may be stated as how to find the color space that is most suitable for fusion [35].

Under the hypothesis of substitution of a single component that is a linear combination of the input bands, the fusion process can be obtained without the explicit calculation of the forward and backward transformations, but through a proper injection scheme, thereby leading to the fast implementations of CS methods, whose general formulation is [1]:

$${\hat{\mathbf{M}}}_k={\tilde{\mathbf{M}}}_k+{\mathbf{G}}_k·\left({\overline{\mathbf{P}}}^{\left({\mathbf{I}}_L\right)}-{\mathbf{I}}_L\right),k=1,\dots ,N$$

(3)

in which k is the band index, $\mathbf{G}=[{\mathbf{G}}_1,\dots ,{\mathbf{G}}_k,\dots ,{\mathbf{G}}_N]$ the 3-D array of injection gains, which in principle may be different for each pixel each band, while the intensity, ${\mathbf{I}}_L$, is defined as

$${\mathbf{I}}_L=\stackrel{N}{\sum _{i=1}}{w}_i·{\tilde{\mathbf{M}}}_i$$

(4)

in which the weight vector $\mathbf{w}=[w_1,\dots ,w_i,\dots ,w_N]$ is the 1-D array of spectral weights, corresponding to the first row of the forward transformation matrix [1]. The term ${\overline{\mathbf{P}}}^{\left({\mathbf{I}}_L\right)}$ is $\mathbf{P}$ histogram-matched to ${\mathbf{I}}_L$

$${\overline{\mathbf{P}}}^{\left({\mathbf{I}}_L\right)}\triangleq (\mathbf{P}-{\mu }_{\mathbf{P}})·\frac{{\sigma }_{{\mathbf{I}}_L}}{{\sigma }_{{\mathbf{P}}_L}}+{\mu }_{{\mathbf{I}}_L}$$

(5)

in which $\mu$ and $\sigma$ denote mean and square root of variance, respectively, and ${\mathbf{P}}_L$ is a lowpass version of $\mathbf{P}$ having the same spatial frequency content as ${\mathbf{I}}_L$ [8]. Figure 2 shows the typical flowchart employed by CS methods.

The simplest CS fusion method is the Intensity-Hue-Saturation (IHS) [1], or better its generalization to an arbitrary number of bands, GIHS, which allows a fast implementation, given by Equation (3) with unitary injection gains, ${\mathbf{G}}_k=1$. The multiplicative or contrast-based injection model is a special case of Equation (3), in which space-varying injection gains, $\mathbf{G}$, are defined such that

$${\mathbf{G}}_k=\frac{{\tilde{\mathbf{M}}}_k}{{\mathbf{I}}_L},k=1,\dots ,N.$$

(6)

The resulting pansharpening method is described by

$${\hat{\mathbf{M}}}_k={\tilde{\mathbf{M}}}_k+\frac{{\tilde{\mathbf{M}}}_k}{{\mathbf{I}}_L}·({\overline{\mathbf{P}}}^{\left({\mathbf{I}}_L\right)}-{\mathbf{I}}_L)={\tilde{\mathbf{M}}}_k·\frac{{\overline{\mathbf{P}}}^{\left({\mathbf{I}}_L\right)}}{{\mathbf{I}}_L},k=1,\dots ,N$$

(7)

which, in the case of spectral weights all equal to $1/N$, is the widely known Brovey transform (BT) pansharpening method [1,36].

In Gram–Schmidt (GS) [37] spectral sharpening, the fusion process is described by Equation (3), where the injection gains are spatially uniform for each band, and thus denoted as ${\left\{g_k\right\}}_{k=1,\dots ,N}$. They are given by [8]:

$$g_k=\frac{\mathrm{cov}({\tilde{\mathbf{M}}}_k,{\mathbf{I}}_L)}{\mathrm{var}\left({\mathbf{I}}_L\right)}k=1,\dots ,N$$

(8)

in which $\mathrm{cov}\left(\mathbf{X},\mathbf{Y}\right)$ indicates the covariance between $\mathbf{X}$ and $\mathbf{Y}$, and $\mathrm{var}\left(\mathbf{Y}\right)$ is the variance of $\mathbf{Y}$. A multivariate linear regression has been exploited to model the relationship between the lowpass-filtered Pan, ${\mathbf{P}}_L$, and the interpolated MS bands [1,18],

$${\mathbf{P}}_L=\sum _{i=1}^N{\hat{w}}_i·{\tilde{\mathbf{M}}}_i+\hat{b}+ϵ\triangleq {\hat{\mathbf{I}}}_L+ϵ$$

(9)

in which ${\hat{\mathbf{I}}}_L$ is the optimal intensity component and $ϵ$ the least squares (LS) space-varying residue. The set of space-constant optimal weights, ${\left\{{\hat{w}}_k\right\}}_{k=1,\dots ,N}$, and $\hat{b}$, is calculated as the minimum MSE (MMSE) solution of Equation (9). A figure of merit of the matching achieved by the MMSE solution is given by the coefficient of determination (CD), namely $R^2$, defined as

$$R^2\triangleq 1-\frac{{\sigma }_{ϵ}^2}{{\sigma }_{{\mathbf{P}}_L}^2}$$

(10)

in which ${\sigma }_{ϵ}^2$ and ${\sigma }_{{\mathbf{P}}_L}^2$ denote the variance of the (zero-mean) LS residue, $ϵ$, and of the lowpass-filtered Pan image, respectively. Histogram-matching of Pan to the MMSE intensity component, ${\hat{\mathbf{I}}}_L$, should take into account that ${\mu }_{\mathbf{P}}={\mu }_{{\mathbf{P}}_L}={\mu }_{{\hat{\mathbf{I}}}_L}$, from Equation (9). Thus, from the definition of CD in Equation (10)

$${\overline{\mathbf{P}}}^{\left({\hat{\mathbf{I}}}_L\right)}=(\mathbf{P}-{\mu }_{\mathbf{P}})·R+{\mu }_{\mathbf{P}}.$$

(11)

This brief review does not include a popular CS fusion method employing principal component analysis (PCA). The reason is that PCA is a particular case of the more general GS transformation, in which ${\mathbf{I}}_L$ is equal to the maximum-variance first principal component, PC1, and the injection gains are those of GS, as in Equation (8), [1].

3.3. MRA

The spatial approach relies on the injection of highpass spatial details of Pan into the resampled MS bands. The most general MRA-based fusion may be stated as:

$${\hat{\mathbf{M}}}_k={\tilde{\mathbf{M}}}_k+{\mathbf{G}}_k·\left({\overline{\mathbf{P}}}^{\left({\tilde{\mathbf{M}}}_k\right)}-{\overline{\mathbf{P}}}_L^{\left({\tilde{\mathbf{M}}}_k\right)}\right),k=1,\dots ,N.$$

(12)

in which the Pan image is preliminarily histogram-matched to the interpolated kth MS band [8]

$${\overline{\mathbf{P}}}^{\left({\tilde{\mathbf{M}}}_k\right)}\triangleq (\mathbf{P}-{\mu }_{\mathbf{P}})·\frac{{\sigma }_{{\tilde{\mathbf{M}}}_k}}{{\sigma }_{{\mathbf{P}}_L}}+{\mu }_{{\tilde{\mathbf{M}}}_k}$$

(13)

and ${\overline{\mathbf{P}}}_L^{\left({\tilde{\mathbf{M}}}_k\right)}$ the lowpass-filtered version of ${\overline{\mathbf{P}}}^{\left({\tilde{\mathbf{M}}}_k\right)}$. It is noteworthy that according to either of Equations (5) and (13), histogram matching of $\mathbf{P}$ always implies the calculation of its lowpass version ${\mathbf{P}}_L$.

According to Equation (12) the different approaches and methods belonging to this class are uniquely characterized by the lowpass filter employed for obtaining the image ${\mathbf{P}}_L$, by the presence or absence of a decimator/interpolator pair [38] and by the set of space-varying injection gains, either spatially uniform, ${\left\{g_k\right\}}_{k=1,\dots ,N}$ or space-varying, ${\left\{{\mathbf{G}}_k\right\}}_{k=1,\dots ,N}$. The contrast-based version of MRA pansharpening is

$$\begin{array}{ccc}\hfill {\hat{\mathbf{M}}}_k& =& {\tilde{\mathbf{M}}}_k+\frac{{\tilde{\mathbf{M}}}_k}{{\overline{\mathbf{P}}}_L^{\left({\tilde{\mathbf{M}}}_k\right)}}·\left({\overline{\mathbf{P}}}^{\left({\tilde{\mathbf{M}}}_k\right)}-{\overline{\mathbf{P}}}_L^{\left({\tilde{\mathbf{M}}}_k\right)}\right)\hfill \\ & =& {\tilde{\mathbf{M}}}_k·\frac{{\overline{\mathbf{P}}}^{\left({\tilde{\mathbf{M}}}_k\right)}}{{\overline{\mathbf{P}}}_L^{\left({\tilde{\mathbf{M}}}_k\right)}},k=1,\dots ,N.\hfill \end{array}$$

(14)

It is noteworthy that, unlike what happens for Equation (7), Equation (14) does not preserve the spectral angle of ${\tilde{\mathbf{M}}}_k$, because the multiplicative sharpening term depends on k, through Equation (13). Eventually, the projective injection gains derived from GS (Equation (8)) can be extended to MRA fusion as

$$g_k=\frac{\mathrm{cov}({\tilde{\mathbf{M}}}_k,{\mathbf{P}}_L)}{\mathrm{var}\left({\mathbf{P}}_L\right)}k=1,\dots ,N$$

(15)

whose space varying-version ${\mathbf{G}}_k$, with statistics calculated locally on a sliding window, coupled with a pyramid MRA constitutes a popular pansharpening method known as GLP-CBD [1,39]. Figure 3 shows the general flowchart of spatial, a.k.a. MRA, pansharpening methods.

4. Reproducibility of Results of Pansharpening Methods with the Data Format

Here, we derive the conditions that allow CS and MRA pansharpening methods to attain reproducibility with the input data format. In particular, we will refer to the case of SR, even though the proof is also valid for reflectance formats:

$$\begin{array}{cc}\hfill {\mathbf{P}}^{SR}=& {\alpha }_P·{\mathbf{P}}^{DN}+{\beta }_P\hfill \\ \hfill {\tilde{\mathbf{M}}}_k^{SR}=& {\alpha }_k·{\tilde{\mathbf{M}}}_k^{DN}+{\beta }_k,k=1,\dots ,N\hfill \end{array}$$

(16)

in which the superscripts SR and DN indicate that the quantity is expressed in SR units (floating-point) or in DN (fixed-point), respectively.

Upon these premises, we consider a pansharpening method to be reproducible vs. data-format if:

$${\hat{\mathbf{M}}}_k^{SR}={\alpha }_k·{\hat{\mathbf{M}}}_k^{DN}+{\beta }_k,k=1,\dots ,N$$

(17)

or, equivalently, if the fusion process preserves the linear affine transformation of the data-format conversion.

4.1. CS

The analysis of data-format reproducibility of CS methods is more complicated than that of MRA methods because the spectral bands are combined together to produce the intensity component, unlike the band-to-band processing of MRA. Let us start from the general form of CS methods in Equation (3), considering the projective injection of coefficients of Equation (8) and the intensity component with spectral weights calculated as MMSE solution of Equation (9):

$${\hat{\mathbf{M}}}_k^{SR}={\tilde{\mathbf{M}}}_k^{SR}+g_k^{SR}·\left({\overline{\mathbf{P}}}^{\left({\hat{\mathbf{I}}}_L^{SR}\right)SR}-{\hat{\mathbf{I}}}_L^{SR}\right),k=1,\dots ,N,$$

(18)

in which

$${\hat{\mathbf{I}}}_L^{SR}=\sum _{i=1}^N{\hat{w}}_i^{SR}·{\tilde{\mathbf{M}}}_i^{SR}+{\hat{b}}^{SR}.$$

(19)

From the LS solution of Equation (9), it is easily proven that the following relationships hold between spectral weights and bias terms in DN and in SR formats:

$$\begin{array}{cc}\hfill {\hat{w}}_k^{SR}=& {\hat{w}}_k^{DN}·\frac{{\alpha }_P}{{\alpha }_k},k=1,\dots ,N\hfill \\ \hfill {\hat{b}}^{SR}=& {\alpha }_P·{\hat{b}}^{DN}+{\beta }_P-{\alpha }_P·\sum _{i=1}^N{\hat{w}}_i^{DN}·\frac{{\beta }_i}{{\alpha }_i}\hfill \\ \hfill R^{SR}=& R^{DN}=R\hfill \end{array}$$

(20)

It is noteworthy that the degree of matching achieved by the LS solution, given by Equation (10), is identical for the two formats. Substituting Equation (20) into Equation (19) yields the relationship between SR and DN intensity components:

$$\begin{array}{cc}\hfill {\hat{\mathbf{I}}}_L^{SR}& =\sum _{i=1}^N{\hat{w}}_i^{SR}·{\tilde{\mathbf{M}}}_i^{SR}+{\hat{b}}^{SR}\hfill \\ \hfill & =\sum _{i=1}^N{\hat{w}}_i^{DN}·\frac{{\alpha }_P}{{\alpha }_i}·({\alpha }_i·{\tilde{\mathbf{M}}}_i^{DN}+{\beta }_i)+{\alpha }_P·{\hat{b}}^{DN}+{\beta }_P-{\alpha }_P·\sum _{i=1}^N{\hat{w}}_i^{DN}·\frac{{\beta }_i}{{\alpha }_i}\hfill \\ \hfill & ={\alpha }_P·(\sum _{i=1}^N{\hat{w}}_i^{DN}·{\tilde{\mathbf{M}}}_i^{DN}+{\hat{b}}^{DN})+{\beta }_P\hfill \\ \hfill & ={\alpha }_P·{\hat{\mathbf{I}}}_L^{DN}+{\beta }_P\hfill \end{array}$$

(21)

Hence, the histogram-matching of Pan to ${\hat{\mathbf{I}}}_L^{SR}$ in Equation (11), becomes:

$$\begin{array}{cc}\hfill {\overline{\mathbf{P}}}^{\left({\hat{\mathbf{I}}}_L^{SR}\right)SR}& =({\mathbf{P}}^{SR}-{\mu }_{{\mathbf{P}}^{SR}})·R+{\mu }_{{\mathbf{P}}^{SR}}\hfill \\ \hfill & ={\alpha }_P·\left[({\mathbf{P}}^{DN}+{\beta }_P/{\alpha }_P-{\mu }_{{\mathbf{P}}^{DN}}-{\beta }_P/{\alpha }_P)·R+{\mu }_{{\mathbf{P}}^{DN}}\right]+{\beta }_P\hfill \\ \hfill & ={\alpha }_P·{\overline{\mathbf{P}}}^{\left({\hat{\mathbf{I}}}_L^{DN}\right)DN}+{\beta }_P.\hfill \end{array}$$

(22)

Furthermore, the SR and DN injections gains can be related as:

$$\begin{array}{cc}\hfill g_k^{SR}& =\frac{\mathrm{cov}({\tilde{\mathbf{M}}}_k^{SR},{\hat{\mathbf{I}}}_L^{SR})}{\mathrm{var}\left({\hat{\mathbf{I}}}_L^{SR}\right)}\hfill \\ \hfill & =\frac{{\alpha }_k}{{\alpha }_P}·\frac{\mathrm{cov}({\tilde{\mathbf{M}}}_k^{DN},{\hat{\mathbf{I}}}_L^{DN})}{\mathrm{var}\left({\hat{\mathbf{I}}}_L^{DN}\right)}\hfill \\ \hfill & =\frac{{\alpha }_k}{{\alpha }_P}·g_k^{DN}.\hfill \end{array}$$

(23)

Finally, substituting Equations (21)-(22)-(23) into Equation (18), we obtain:

$$\begin{array}{cc}\hfill {\hat{\mathbf{M}}}_k^{SR}& ={\tilde{\mathbf{M}}}_k^{SR}+g_k^{SR}·\left({\overline{\mathbf{P}}}^{\left({\hat{\mathbf{I}}}_L^{SR}\right)SR}-{\hat{\mathbf{I}}}_L^{SR}\right)\hfill \\ \hfill & ={\alpha }_k·{\tilde{\mathbf{M}}}_k^{DN}+{\beta }_k+\frac{{\alpha }_k}{{\alpha }_P}·g_k^{DN}·{\alpha }_P·\left({\overline{\mathbf{P}}}^{\left({\hat{\mathbf{I}}}_L^{DN}\right)DN}-{\hat{\mathbf{I}}}_L^{DN}\right)\hfill \\ \hfill & ={\alpha }_k·{\hat{\mathbf{M}}}_k^{DN}+{\beta }_k.\hfill \end{array}$$

(24)

It is important to remark that Equation (24) holds only if the relationships of Equation (20) are satisfied, which implicitly happens to regression-based CS methods. The unitary injection gain (GIHS) does not affect the reproducibility of fusion if Equation (20) holds. Conversely, it is easily proven that, when the offsets are nonzero, the multiplicative injection model of Equation (6) is never reproducible also if Equation (9), and hence Equation (20), hold. In summary, for CS methods, the key to achieve data-format reproducibility lies in the relationships of Equation (20) between the spectral weights calculated in the different data formats, which is implicitly satisfied if the intensity component is calculated as in Equation (9).

4.2. MRA

Starting from Equation (12), we firstly focus on the detail term and thus set the band-dependent injection gains all equal to unity:

$${\hat{\mathbf{M}}}_k^{SR}={\tilde{\mathbf{M}}}_k^{SR}+\left({\overline{\mathbf{P}}}^{\left({\tilde{\mathbf{M}}}_k^{SR}\right)SR}-{\overline{\mathbf{P}}}_L^{\left({\mathbf{M}}_k^{SR}\right)SR}\right),k=1,\dots ,N.$$

(25)

Starting from Equation (16), the relationship between histogram-matched Pan, in either DN or SR format, can be written as:

$$\begin{array}{cc}\hfill {\overline{\mathbf{P}}}^{\left({\tilde{\mathbf{M}}}_k^{SR}\right)SR}& =({\mathbf{P}}^{SR}-{\mu }_{{\mathbf{P}}^{SR}})·\frac{{\sigma }_{{\tilde{\mathbf{M}}}_k^{SR}}}{{\sigma }_{{\mathbf{P}}_L^{SR}}}+{\mu }_{{\tilde{\mathbf{M}}}_k^{SR}}\hfill \\ \hfill & ={\alpha }_P·({\mathbf{P}}^{DN}+{\beta }_P/{\alpha }_p-{\mu }_{{\mathbf{P}}^{DN}}-{\beta }_P/{\alpha }_P)·\frac{{\alpha }_k}{{\alpha }_P}·\frac{{\sigma }_{{\tilde{\mathbf{M}}}_k^{DN}}}{{\sigma }_{{\mathbf{P}}_L^{DN}}}+{\alpha }_k·{\mu }_{{\tilde{\mathbf{M}}}_k^{DN}}+{\beta }_k\hfill \\ \hfill & ={\alpha }_k·{\overline{P}}^{\left({\tilde{\mathbf{M}}}_k^{DN}\right)DN}+{\beta }_k\hfill \end{array}$$

(26)

where we have used the following relationships for the mean and standard deviation:

$$\left\{\begin{array}{cc}{\mu }_{{\mathbf{P}}^{SR}}={\alpha }_P·{\mu }_{{\mathbf{P}}^{DN}}+{\beta }_P,\hfill & {\sigma }_{{\mathbf{P}}_L^{SR}}={\alpha }_P·{\sigma }_{{\mathbf{P}}_L^{DN}}\hfill \\ {\mu }_{{\tilde{\mathbf{M}}}_k^{SR}}={\alpha }_k·{\mu }_{{\tilde{\mathbf{M}}}_k^{DN}}+{\beta }_k,\hfill & {\sigma }_{{\tilde{\mathbf{M}}}_k^{SR}}={\alpha }_k·{\sigma }_{{\tilde{\mathbf{M}}}_k^{SR}},k=1,\dots ,N.\hfill \end{array}\right.$$

(27)

Finally, Equation (25) can be rewritten as:

$$\begin{array}{cc}\hfill {\hat{\mathbf{M}}}_k^{SR}& ={\alpha }_k·{\tilde{\mathbf{M}}}_k^{DN}+{\beta }_k+{\alpha }_k·\left({\overline{\mathbf{P}}}^{\left({\tilde{\mathbf{M}}}_k^{DN}\right)DN}+{\beta }_k/{\alpha }_k-{\overline{\mathbf{P}}}_L^{\left({\tilde{\mathbf{M}}}_k^{DN}\right)DN}-{\beta }_k/{\alpha }_k\right)\hfill \\ \hfill & ={\alpha }_k·{\hat{\mathbf{M}}}_k^{DN}+{\beta }_k,k=1,\dots ,N.\hfill \end{array}$$

(28)

In the case of non-unitary injection gains, such as the projective and the multiplicative coefficients, it is easily proven that the former is independent of the format, analogously to Equation (23); the latter only if the offsets are zero. In summary, the key to data-format reproducibility of MRA methods lies in the band-by-band linear histogram matching operation of Equation (13), thanks to which the sharpening Pan inherits gain and offset from the MS band that shall be sharpened.

5. Reproducibility of Quality Indexes Varying with Data Formats

In this section, the most popular statistical similarity/dissimilarity indexes, suitable for multiband images, will be reviewed and their behavior for DN and calibrated data formats will be investigated.

5.1. SAM

Spectral angle mapper (SAM) was originally introduced for the discrimination of materials starting from their reflectance spectra [40]. Given two spectral vectors, $\mathbf{v}$ and $\hat{\mathbf{v}}$, both having N components, in which $\mathbf{v}=\{v_1,v_2,\dots ,v_N\}$ is the reference spectral pixel vector and $\hat{\mathbf{v}}=\{{\hat{v}}_1,{\hat{v}}_2,\dots ,{\hat{v}}_N\}$ is the test spectral pixel vector, SAM denotes the absolute value of the spectral angle between the two vectors:

$$\mathrm{SAM}(\mathbf{v},\hat{\mathbf{v}})\triangleq arccos\left(\frac{<\mathbf{v},\hat{\mathbf{v}}>}{{\left|\right|\mathbf{v}\left|\right|}_2·\left|\right|\hat{\mathbf{v}}{\left|\right|}_2}\right).$$

(29)

SAM is usually expressed in degrees and is equal to zero iff the test vector is spectrally identical to the reference vector, i.e., the two vectors are parallel and may differ only by their moduli. A global spectral dissimilarity, or distortion, index is obtained by averaging Equation (29) over the scene. From Equation (2) it is evident that whether Equation (29) is calculated with the two vectors in DN or SR, makes a difference. The angle measured in the two formats is identical if the offsets, ${\beta }_k$, are all zero and the gains, ${\alpha }_k$, are the same for each band. Whenever this does not occur, the angle measured by Equation (29) is different for the two formats, which is a reasonable conclusion for a spectral distortion measure.

5.2. ERGAS

ERGAS, the French acronym for relative dimensionless global error in synthesis [41], is a normalized dissimilarity index that offers a global indication of the distortion towards the reference of a test multiband image, as produced by a pansharpening method:

$$\mathrm{ERGAS}\triangleq 100\frac{d_h}{d_l}\sqrt{\frac{1}{N}{\sum }_{k=1}^N{\left(\frac{\mathrm{RMSE}\left(k\right)}{\mu \left(k\right)}\right)}^2}$$

(30)

where $d_h/d_l$ is the ratio between pixel sizes of Pan and MS, e.g., 1/4, $\mu \left(k\right)$ is the mean (average) of the kth band of the reference and N is the number of bands. Low values of ERGAS indicate high similarity between fused and reference MS data. Since ERGAS is nothing more than the cumulative normalized root mean square error (NRMSE), multiplied by the Pan-to-MS scale ratio and expressed in percentage, its value is insensitive to changes in the spectral gains, ${\alpha }_k$, of individual bands. Instead, the presence of nonzero band offsets, ${\beta }_k$, produces lower values of the mean, $\mu \left(k\right)$, and hence values of ERGAS higher than those with offsets that are all equal to zero.

5.3. Multivariate UIQI

Q$2^n$ is the multiband extension of the universal image quality index (UIQI) [42] and was introduced for quality assessment of pansharpened MS images [43]. Each pixel of an image with N spectral bands is accommodated into a hypercomplex (HC) number with one real part and $N-1$ imaginary parts.

Let $z=z(m,n)$ and $\hat{z}=\hat{z}(m,n)$ denote the HC representation of the reference and test spectral vectors at pixel $(m,n)$. Analogously to UIQI, namely, Q$2^0$ = Q, Q$2^n$ may be written as the product of three terms:

$$\mathrm{Q}{2}^n=\frac{|{\sigma }_{z\hat{z}}|}{{\sigma }_z{\sigma }_{\hat{z}}}·\frac{2{\sigma }_z{\sigma }_{\hat{z}}}{{\sigma }_z^2+{\sigma }_{\hat{z}}^2}·\frac{2\left|\overline{z}\right|\left|\overline{\hat{z}}\right|}{{\left|\overline{z}\right|}^2+{\left|\overline{\hat{z}}\right|}^2}$$

(31)

the first of which is the modulus of the HC correlation coefficient (HCCC) between z and $\hat{z}$. The second and third terms, respectively, measure contrast changes and mean bias on all bands simultaneously. Statistics are calculated on square blocks, typically, 32 × 32, and Q$2^n$ is averaged over the blocks of the whole image to yield the global score index. Q$2^n$ takes values in [0,1] and is equal to 1 iff$z=\hat{z}$ for all pixels. From the definitions of mean, variance, and covariance of HC random variables, it can be proven that the covariance, and hence the HCCC, is sensitive to gains, ${\alpha }_k$, but only if such gains are not equal for all bands. The contrast-change term is always insensitive to offsets, but not to band-varying gains. In the presence of nonzero band offsets, ${\beta }_k$, also the mean-bias term may change.

Analogously, it can be easily proven that the univariate index UIQI, often calculated on the individual bands and averaged to yield a unique quality index, is independent of any choice of gains, also different from one another, but is affected by the presence of offsets, also equal to one another.

In conclusion, while ERGAS and average UIQI are independent of gains, whichever they are, all the four indexes measure different values for the two formats, in the presence of offsets, also equal to one another.

6. Experimental Results and Discussion

6.1. Data Sets

Two test images, acquired by two different platforms, GeoEye-1 and WorldView-2, have been used in the simulations. In this section, besides describing the two datasets, we propose an index to measure the spectral alteration of the DN representation and present an analysis of the solution of the multivariate regression of Equation (9) for the two datasets.

6.1.1. Collazzone Dataset

The GeoEye-1 image has been acquired over the area of Collazzone, a rural locality in Central Italy, on 13 July 2010. The spatial sampling interval (SSI) is 2 m for MS and 0.5 m for Pan, resulting in a scale ratio equal to four. The MS image features four spectral bands: blue (B), green (G), red (R) and near infra-red (NIR). The image size is 512 × 512 pixels for MS and 2048 × 2048 for Pan. The radiometric resolution of the DN format is 11 bits; the conversion coefficients to the SR format, extracted from the metadata, are reported in

Table 1. Gains and offsets for conversion to SR of GeoEye-1—Collazzone.

GE-1	Pan	B	G	R	NIR	SIF%
${\alpha }_k$	0.0178	0.0250	0.0172	0.0277	0.0096	65.34%
${\beta }_k$	0	0	0	0	0	-

. It is noteworthy that the offsets are all equal to zero.

6.1.2. Sydney Dataset

The WorldView-2 image, acquired on 21 August 2012, portrays the city of Sydney, Australia. The SSI is 2 m for MS and 0.5 m for Pan; hence, the scale ratio is still four. The MS image comprises eight bands: coastal (C), B, G, yellow (Y), R, red edge (RE), NIR1, and NIR2. The covered area is 1,048,576 ${\mathrm{m}}^2$, corresponding to MS and Pan image sizes of 512 × 512 and 2048 × 2048 pixels, respectively. The fixed-point representation of the data employs 11 bits.

Table 2. Gains and offsets for conversion to SR of WorldView-2—Sydney.

WV-2	Pan	C	B	G	Y	R	RE	NIR1	NIR2	SIF%
${\alpha }_k$	0.1331	0.1965	0.2322	0.1542	0.1364	0.1923	0.1155	0.1238	0.0908	60.89%
${\beta }_k$	0	0	0	0	0	0	0	0	0	-

shows the corresponding radiometric calibration coefficients obtained from the metadata file. Also in this case the offsets are identically equal to zero.

6.2. Spectral Imbalance Factor

The conversion to SR of DN counts in Equation (2a) has the effect of restoring the original spectral inter-relationships of the calibrated data that have been lost after integer packaging. In other words, the integer-valued samples may be spectrally altered, if the gains and/or offsets are not equal to one another. A measure of such a spectral alteration can be defined as spectral imbalance factor (SIF):

$$\mathrm{SIF}\%\triangleq \frac{{\alpha }_{max}-{\alpha }_{min}}{{\alpha }_{max}}·exp\left(\frac{{\beta }_{max}-{\beta }_{min}}{{\beta }_{max}+c}\right)\times 100$$

(32)

where ${\alpha }_{min}$ (${\beta }_{min}$) and ${\alpha }_{max}$ (${\beta }_{max}$) are respectively the minimum and the maximum values of the gains (offsets) of the spectral bands, including Pan; c a machine tolerance to avoid zero-by-zero divisions. SIF% is zero iff the ${\alpha }_k$, including ${\alpha }_P$, are equal to one another and is independent of the offsets iff they are equal to one another, also if they are all zero. As reported in Table 1 and Table 2, SIF% is equal to 65.35% for the Collazzone dataset and 60.89% for the Sydney dataset. The presence of nonzero offsets would increase SIF%, as it appears from Equation (32). Note that, for both datasets, the minimum gain is that of the NIR channel (NIR2 for WorldView-2), the maximum gain is on the red channel for Collazzone and on the blue one for Sydney, which means that the red/blue bands have been the most compressed, in order to fit the 11-bit DN representation; the NIR/NIR2 bands, the least.

6.3. Analysis of the LS Intensity Component

The LS solution of the multivariate linear regression of Equation (9) is now discussed for the two test datasets.

Table 3. MMSE coefficients of the multivariate linear regression between interpolated MS bands and lowpass-filtered Pan, both in either DN or SR format, for the GeoEye-1 Collazzone image. The bias, $\hat{b}$, is either set equal to zero or left as a further variable to optimize.

GE-1	B	G	R	NIR	$\hat{\mathit{b}}$	$\sum {\hat{\mathit{w}}}_{\mathit{k}}$	CD
${\hat{w}}_k^{DN}$	0.0637	0.4398	0.4609	0.1557	−17.5223	1.1200	0.9858
${\hat{w}}_k^{DN}$	−0.1010	0.5408	0.4474	0.1474	-	1.0346	0.9856
${\hat{w}}_k^{SR}$	0.0453	0.4554	0.2956	0.2888	−0.3117	1.0850	0.9858
${\hat{w}}_k^{SR}$	−0.0718	0.5600	0.2869	0.2734	-	1.0486	0.9856

and

Table 4. MMSE coefficients of the multivariate linear regression between interpolated MS bands and lowpass-filtered Pan, both in either DN or SR format, for the WorldView-2 Sydney image. The bias, $\hat{b}$, is either set equal to zero or left as a further variable to optimize.

WV-2	C	B	G	Y	R	RE	NIR1	NIR2	$\hat{\mathit{b}}$	$\sum {\hat{\mathit{w}}}_{\mathit{k}}$	CD
${\hat{w}}_k^{DN}$	0.3643	−0.0269	0.1127	0.2368	0.2636	0.1353	0.0225	0.1477	−37.5550	1.2560	0.9857
${\hat{w}}_k^{DN}$	0.1170	0.1245	0.0987	0.2934	0.2245	0.1251	0.0233	0.1506	-	1.1571	0.9855
${\hat{w}}_k^{SR}$	0.2468	−0.0154	0.0973	0.2311	0.1825	0.1560	0.0242	0.2166	−4.9996	1.1390	0.9857
${\hat{w}}_k^{SR}$	0.0792	0.0714	0.0852	0.2864	0.1555	0.1442	0.0251	0.2208	-	1.0678	0.9855

report the MMSE spectral weights of each band and the bias coefficient, when it is included or not, for both the DN and the SR formats. All the experimental values match the theoretical investigations of Equation (20); in particular, the CD, also called $R^2$, is identical for the two formats. The presence or absence of the bias term produces ${\hat{w}}_k$ somewhat different in the two cases, but this has a very limited impact on the degree of matching, measured by CD, which changes on the fourth decimal digit. This happens because in both the test images offsets are not present, i.e., they are identically zero. In the presence of nonzero offsets, the regression must include the bias term, which compensates for the offsets of the DN format, according to Equation (20). Note that the sum of the spectral weights slightly exceeds one, in all cases, less for GE-1 than for WV-2. This result is in agreement with the layout of spectral bands of the two instruments, since weights summing to one would indicate that the spectral width of Pan is equal to the total span of the accompanying MS bands.

6.4. Simulations

Nine pansharpening algorithms, including the plain interpolated MS image, without injection of details, denoted by EXP, have been selected from the Pansharpening Toolbox [39]. Figure 4 shows the flowchart of the experimental setup. The goal is twofold: (i) to determine the possible dependence of quality assessment metrics on the specific data format, in which they are evaluated; (ii) to check which pansharpening methods perform identically, regardless of the data format. After the original MS + Pan dataset has been spatially degraded, all fusion methods are run twice: once on the DN data and once on the DN data preliminarily converted to SR. In this way, two sets of fused images are made available, one in DN and one in SR. Each fused dataset is then converted to the other format: the output of the fusion process in DN is converted to SR and the output of fusion in SR is converted back to DN. This procedure allows reproducibility of methods and reproducibility of indexes to be evaluated in a unique simulation session. Thus, data-format reproducible pansharpening methods and data-format reproducible quality indexes are jointly identified.

Table 5. Scores of GeoEye-1 Collazzone towards ground truth (REF) for nine fusion methods described in [39]. The upper subtable refers to assessments carried out on data in DN; the lower one in SR. Each entry of the table contains the values of a quality/distortion index relativ to fusion performed in either DN or SR. Reproducible indexes are highlighted in green; non-reproducible in red. Reproducible methods are marked by green squares; non-reproducible by red squares.

	DN	Qavg ●↓	Q4 ●↓	SAM ●↓	ERGAS ●↓
DN|SR
REF		1	1	0	0
EXP ■→		0.7814 | 0.7814	0.7779 | 0.7779	2.3534 | 2.3534	3.1821 | 3.1821
GS ■→		0.8456 | 0.8544	0.8182 | 0.8291	2.5814 | 2.8921	2.8376 | 2.8127
GSA ■→		0.8595 | 0.8595	0.8673 | 0.8673	2.9923 | 2.9923	2.8786 | 2.8786
GIHS ■→		0.8378 | 0.8694	0.8175 | 0.8654	2.8407 | 2.3580	2.9159 | 2.4197
PCA ■→		0.8490 | 0.8140	0.8332 | 0.8241	2.6032 | 3.6255	3.0049 | 3.3031
BT ■→		0.8670 | 0.8677	0.8599 | 0.8625	2.3534 | 2.3534	2.4315 | 2.4556
MTF-GLP ■→		0.8625 | 0.8625	0.8625 | 0.8625	2.5338 | 2.5338	2.9208 | 2.9208
BDSD ■→		0.8994 | 0.8994	0.9027 | 0.9027	2.5043 | 2.5043	2.1717 | 2.1717
ATWT ■→		0.8723 | 0.8723	0.8688 | 0.8688	2.4686 | 2.4686	2.6631 | 2.6631
	SR	Qavg ●↓	Q4 ●↓	SAM ●↓	ERGAS ●↓
DN|SR
EXP ■→		0.7814 | 0.7814	0.7048 | 0.7048	2.9896 | 2.9896	3.1821 | 3.1821
GS ■→		0.8456 | 0.8544	0.7377 | 0.7472	3.2935 | 3.6044	2.8376 | 2.8127
GSA ■→		0.8595 | 0.8595	0.8021 | 0.8021	3.6930 | 3.6930	2.8786 | 2.8786
GIHS ■→		0.8378 | 0.8694	0.7534 | 0.7998	3.6549 | 2.9516	2.9159 | 2.4197
PCA ■→		0.8490 | 0.8140	0.7652 | 0.7635	3.4756 | 4.5339	3.0049 | 3.3031
BT ■→		0.8670 | 0.8677	0.7862 | 0.7979	2.9896 | 2.9896	2.4315 | 2.4556
MTF-GLP ■→		0.8625 | 0.8625	0.8001 | 0.8001	3.2838 | 3.2838	2.9208 | 2.9208
BDSD ■→		0.8994 | 0.8994	0.8352 | 0.8352	3.0302 | 3.0302	2.1717 | 2.1717
ATWT ■→		0.8723 | 0.8723	0.8033 | 0.8033	3.1489 | 3.1489	2.6631 | 2.6631
REF		1	1	0	0

●(●): data-format (non-)reproducible quality index. ■(■): data-format (non-)reproducible pansharpening method.

and

Table 6. Scores of WV-2 Sydney towards ground truth (REF) for nine fusion methods described in [39]. The upper subtable refers to assessments carried out on data in DN; the lower one in SR. Each entry of the table contains the values of a quality/distortion index relatively to fusion performed in either DN or SR. Reproducible indexes are highlighted in green; non-reproducible in red. Reproducible methods are marked by green squares; non-reproducible by red squares.

	DN	Qavg ●↓	Q8 ●↓	SAM ●↓	ERGAS ●↓
DN|SR
REF		1	1	0	0
EXP ■→		0.7077 | 0.7077	0.7000 | 0.7000	5.0636 | 5.0636	6.4271 | 6.4271
GS ■→		0.8158 | 0.8288	0.7487 | 0.7861	7.2114 | 6.3689	4.8309 | 4.1664
GSA ■→		0.8460 | 0.8460	0.8495 | 0.8495	4.5789 | 4.5789	3.3967 | 3.3967
GIHS ■→		0.7750 | 0.8247	0.6760 | 0.7888	6.0590 | 4.9979	4.7732 | 3.5853
PCA ■→		0.7912 | 0.8271	0.7138 | 0.7460	7.4372 | 6.1357	7.9097 | 5.9057
BT ■→		0.8030 | 0.8167	0.6940 | 0.7554	5.0636 | 5.0636	4.2299 | 3.4069
MTF-GLP ■→		0.8302 | 0.8302	0.8247 | 0.8247	5.0954 | 5.0954	4.0518 | 4.0518
BDSD ■→		0.8439 | 0.8439	0.8430 | 0.8430	5.0104 | 5.0104	3.8416 | 3.8416
ATWT ■→		0.8366 | 0.8366	0.8310 | 0.8310	4.9530 | 4.9530	3.8678 | 3.8678
	SR	Qavg ●↓	Q8 ●↓	SAM ●↓	ERGAS ●↓
DN|SR
EXP ■→		0.7077 | 0.7077	0.6906 | 0.6906	4.8254 | 4.8254	6.4271 | 6.4271
GS ■→		0.8158 | 0.8288	0.7465 | 0.7835	6.0258 | 5.3841	4.8309 | 4.1664
GSA ■→		0.8460 | 0.8460	0.8377 | 0.8377	4.0870 | 4.0870	3.3967 | 3.3967
GIHS ■→		0.7750 | 0.8247	0.6735 | 0.7807	5.9163 | 4.6071	4.7732 | 3.5853
PCA ■→		0.7912 | 0.8271	0.7100 | 0.7427	6.8330 | 5.4468	7.9097 | 5.9057
BT ■→		0.8030 | 0.8167	0.6924 | 0.7503	4.8254 | 4.8254	4.2299 | 3.4069
MTF-GLP ■→		0.8302 | 0.8302	0.8133 | 0.8133	4.3498 | 4.3498	4.0518 | 4.0518
BDSD ■→		0.8439 | 0.8439	0.8371 | 0.8371	4.5094 | 4.5094	3.8416 | 3.8416
ATWT ■→		0.8366 | 0.8366	0.8199 | 0.8199	4.2917 | 4.2917	3.8678 | 3.8678
REF		1	1	0	0

●(●): data-format (non-)reproducible quality index. ■(■): data-format (non-)reproducible pansharpening method.

report the numerical results of the simulations performed on the two datasets. The organization of the tables requires some explications. Each table is composed of two subtables: the upper one reports the values of quality indexes calculated on data in DN format; the lower subtable reports the same quality metrics calculated in SR format. The cell in each table contains two values: the left term refers to fusion performed in DN; the right one to fusion in SR. In order to check whether a quality index is invariant towards the data format or not, it is sufficient to check whether the columns in the two subtables relative to the same index are identical or not.

The data-format reproducible quality/distortion indexes have been marked with a green bullet beside a down arrow, while the non-reproducible ones with a red bullet beside a down arrow. The upper and lower matching columns are highlighted in light green; the columns of non-reproducible indexes are highlighted in light red. Therefore, average UIQI and ERGAS are format-reproducible; multivariate UIQI and SAM are not. These results match the theoretical investigations, because the offsets are all zero. Otherwise, none of the indexes would be reproducible. Analogously, in order to check the data-format reproducibility of a pansharpening method, we need to look at rows instead of columns. For a given method, if the two values inside every cell of the row are identical to each other, the pansharpening method is data-format reproducible; otherwise, it is not. Data-format reproducible methods are marked with a green square beside a right arrow; non-reproducible methods with red square and arrow. We can verify that all MRA methods (ATWT and MTF-GLP) and some CS methods (GSA and BDSD) provide results invariant with the data format. Conversely, all first-generation CS pansharpening methods, such as GS, GIHS, BT, and PCA, provide different results depending on the input data format. The experimental results match the theoretical investigations presented in Section 4, within the limits that both the available test datasets have zero offsets. In the presence of nonzero offsets. Incidentally, since ${\hat{\mathbf{M}}}_k={\tilde{\mathbf{M}}}_k$, EXP is, by its definition, format-reproducible. Note that, for methods that are not reproducible, results are always better whenever the algorithms are run on data in SR format.

The results of the fusion process at reduced resolution for each of the methods are shown in Figure 5 and Figure 6, for the datasets of Collazzone and Sydney, respectively. For merely visual convenience, fusion is carried out in DNs and the results are displayed in the same format. This fact is not surprising, because the earlier SPOT data (G, R, NIR) were originally distributed in a packed 8-bit format, in order to be directly mapped onto displaying devices, which usually have 8 bits per color channel.

7. Concluding Remarks

In this work, we have investigated how the performance of a pansharpening method depends on the input data format, either packed DNs of Equation (1) or SR, as well as any other floating-point calibrated format. It is theoretically proven and experimentally demonstrated that MRA methods are unaffected by the data format, which instead is crucial for CS methods, unless their intensity component is calculated by means of a multivariate linear regression between the interpolated bands and the lowpass-filtered Pan, as it is accomplished by the most advanced CS methods. For CS fusion methods that do not feature a regression-based intensity calculation, results are better whenever they are obtained from floating-point data. For this reason, whenever the data that are merged come from different platforms [44] and/or are related to different intervals of the electromagnetic spectrum [45], the use of floating-point formats is highly recommended. This study has also demonstrated the necessary non-reproducibility versus the data format of normalized spectral similarity indexes for multiband images. This is necessary because if an index yields the same values when it is calculated from data in packed fixed-point or floating-point formats, it will not necessarily be a spectral index. Conversely, normalized spatial/radiometric indexes are those and only those that do not depend on the format of the data that are compared, at least if the band offsets are all zero. Thus, spectral similarity should be measured on data that are represented as physical units, e.g., SR. Use of packed DNs may lead to misestimation of quality, because DN data are spectrally altered, and hence miscalibrated, if the gains are not equal to one another and the offsets are nonzero.

A viable escape to retain the advantages of a fixed-point processing, mandatory for most of dedicated hardware implementations, and avoid the drawbacks of the spectral distortion, originated by the packaging of floating-point calibrated data of Equation (1), could be:

• convert the available packed DNs to floating-point SR, or any other physical format, by using Equation (2a);
• convert the SR data obtained at the previous step back to DN by using Equation (1), in which $L_{max}$ is not the maximum of the individual band of the scene, but of the whole scene, and $L_{min}=0$.

Thus, it is easily verified that the gains ${\alpha }_k$ are identical to one another and the offsets ${\beta }_k$ are all zero. Hence, it turns out that the distortion factor SIF%, Equation (32), is identically zero. As a final remark, notwithstanding that remote sensing data fusion by means of extremely sophisticated machine learning (ML) tools, e.g., [46], has nowadays reached the state of the art [2], the behaviors of such methods can be hardly modeled and hence predicted because they are based on the outcome of a complex learning process and not on simple constitutive relations. Consequently, it is not easy to prove or foresee if an ML fusion method trained on packed DN data gives the same results as the same method trained on floating-point spectral radiance data. Only an experimental analysis, as carried out in Section 6, would be feasible for ML-based pansharpening methods, but in this case, the analysis and its outcome would not be reproducible, thereby invalidating its motivations. Therefore it is recommended that all the training data are used in floating-point format and fusion is accomplished and assessed in the same format.