# Introduction and Assessment of Measures for Quantitative Model-Data Comparison Using Satellite Images

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Nomenclature

#### 2.1. Definition of Symbols

- ${A}_{\mathit{NaN}}\cap {B}_{\mathit{NaN}}$,
- ${A}_{\mathit{real}}\cap {B}_{\mathit{real}}={X}_{\mathit{real}}$,
- ${A}_{\mathit{real}}\cap {B}_{\mathit{NaN}}$ and
- ${A}_{\mathit{NaN}}\cap {B}_{\mathit{real}}$,

**Figure 1.**Schematic of a satellite taking an image (left panel) and corresponding subregions on the model grid (right panel). Subregion 1 in the right panel corresponds to ${X}_{\mathit{real}}={A}_{\mathit{real}}\cap {B}_{\mathit{real}}$, where the view of the ocean is clear. The satellite image does not cover the entire model domain and clouds as well as other interferences cause missing values in the data, creating subregion 2 (${A}_{\mathit{real}}\cap {B}_{\mathit{NaN}}$). The model domain includes land, resulting in subregion 3 (${A}_{\mathit{NaN}}\cap {B}_{\mathit{NaN}}$).

## 3. Image Comparison Methods

#### 3.1. Parametrisations

#### 3.2. Pixel-by-Pixel Measures

#### Root Mean Square Error (RMS)

**Table 1.**The comparison measures and the names of their parametrisations used in the tests in Section 3.

image comparison measure | p-b-p ^{a} | reference | scaling parameter | name | parametrization | complexity |

root mean square error | √ | RMS | O$\left(\right)open="("\; close=")">nm$ | |||

normalized cross-correlation | √ | NXC | O$\left(\right)open="("\; close=")">nm$ | |||

entropic distance ${D}_{2}$ | √ | [17] | D2 | O$\left(\right)open="("\; close=")">nm$ | ||

adapted Hausdorff distance | [18] | ${g}_{\mathrm{max}}$ ^{c} | AHD^{d} | $p\to \infty $, ${k}_{\mathrm{max}}=1$ | O$\left(\right)open="("\; close=")">{\left(mn\right)}^{2}$ | |

AHD2 ^{d} | $p\to \infty $, ${k}_{\mathrm{max}}=20$ | |||||

averaged distance | [16] | w, ${g}_{\mathrm{max}}$ ^{c} | AVG ^{d} | $w=10$ | O$\left(\right)open="("\; close=")">(m-w)(n-w){\left(2w\right)}^{2}$ | |

Delta-g (${\Delta}_{g}$) | [19] | c ^{b} | DG | ${n}_{\mathrm{lev}}=32$, $c=20$ | O$\left(\right)open="("\; close=")">{n}_{\mathrm{lev}}{\left(mn\right)}^{2}$ | |

image euclidean distance | [20] | σ | IE | $\sigma =1$ | O$\left(\right)open="("\; close=")">{\left(mn\right)}^{2}$ | |

adapted gray block distance | [21] | w | AGB | $w=1$ | O$\left(\right)open="("\; close=")">{log}_{2}(max(m,n))max{(m,n)}^{2}$ ^{e} |

^{a}pixel-by-pixel

^{b}see [19]

^{c}relative to ${m}_{\mathrm{max}}$, ${n}_{\mathrm{max}}$

^{d}AHD, AHD2, AVG use ${g}_{\mathrm{max}}=g$, ${m}_{\mathrm{max}}={n}_{\mathrm{max}}=max(m,n)$

^{e}see Section 3.3. for more details

#### Normalized Cross-Correlation (NXC)

#### Entropic Distance ${D}_{2}$ (D2)

#### 3.3. Neighborhood-Based Measures

#### Adapted Hausdorff Distance (AHD, AHD2)

#### Averaged Distance (AVG)

#### Delta-g (DG)

#### Image Euclidean Distance (IE)

#### Adapted Gray Block Distance (AGB)

**Figure 2.**Original (blue, solid line) and alternate (red, dotted line) division of a ${2}^{{n}_{\mathrm{ext}}}\times {2}^{{n}_{\mathrm{ext}}}$ image into blocks for resolution levels $r=2$ (left image) and $r=3$ (right image).

## 4. Image Comparison Tests & Results

#### 4.1. Test 1: Translated and Masked Features

**Figure 3.**Examples of the manually created images used for the translated and the masked feature tests. In each test case, the first image represents the satellite image D which is compared to the images ${M}_{1}$ (center image) and ${M}_{2}$ (right image). D and ${M}_{1}$ share a common feature that does not appear in ${M}_{2}$. The location of this feature in ${M}_{1}$ is highlighted by a white dotted circle in every image. In the masked feature tests (c) and (d), this location is masked by $\mathit{NaN}$-values in D.

**Figure 4.**Results of tests for the image series shown in Figure 3, expressed as ratios $\frac{d(D,{M}_{1})}{d(D,{M}_{2})}$. Ratios smaller than 1 are desirable

#### 4.2. Test 2: Translation & Rotation of Images

**neighbor test:**- A comparison measure d passes the neighbor test if for any given image in the series, the distance to one of the neighboring images in the series is smaller than the minimum distance to any non-neighboring image, i.e., if$$\underset{j\in \{i-1,i+1\}}{min}d({A}_{i},{A}_{j})<\underset{j\in \{1,\cdots ,i-2\}\cup \{i+2,\cdots ,q\}}{min}d({A}_{i},{A}_{j})\phantom{\rule{4.pt}{0ex}}\mathrm{for}\phantom{\rule{4.pt}{0ex}}i=1,2,3,\cdots ,q$$
**monotonic test:**- d passes the monotonic test if, for the first image in the series, the distance to the other images in the series is monotonically increasing, i.e., if$$d({A}_{1},{A}_{2})<d({A}_{1},{A}_{3})<d({A}_{1},{A}_{4})<\cdots <d({A}_{1},{A}_{q}).$$

**Figure 5.**A series of 4 translated images (top row) and a series of rotated images (bottom row) used in Test 2. Markers are inserted into the images to illustrate the direction of the translation and the angles of the rotation, respectively.

**Figure 6.**Fractions of passed neighbor tests and monotonic tests for 100 series of translated and rotated images.

#### 4.3. Test 3: Noise Sensitivity

#### 4.4. Test 4: NaN-Sensitivity

**Figure 8.**3 Realizations (Rows) of the Same Image Covered Randomly in Missing Values. From Left to Right the Number of Missing Values is Increased in Each Column from 10% to 90% in Increments of 20%.

**Figure 9.**Mean and Standard Deviation of Image Distances for Different Levels of Missing Values. For Each Distance Measure, the Mean Values have been Normalized, so that Their Mean is 1.

#### 4.5. Test 5: NaN-Translation

#### 4.6. Test 6: Time-Series

**Figure 11.**An extraction of 4 consecutive images in a time series of images generated by a physical-biological ocean model.

**Figure 12.**Results of the time-series test. Bar height indicates average fraction of passed tests for the 6 time series of images.

**Table 2.**Qualitative performance ratings of the image comparison measures for the 6 Tests in Section 4. The symbols indicate relative performance in the tests: + is good performance, ○ is average performance and – is bad performance.

Test | RMS | NXC | D2 | AHD | AHD2 | AVG | DG | IE | AGB |

Test 1: translated & masked features | – | – | – | + | + | ○ | + | – | + |

Test 2: translation & rotation | + | – | + | – | + | ○ | ○ | + | + |

Test 3: noise sensitivity | + | – | + | – | ○ | – | – | + | + |

Test 4: NaN-sensitivity | – | – | + | – | + | +^{a} | ○ | –^{a} | + |

Test 5: NaN-translation | + | – | + | – | + | – | ○ | ○ | + |

Test 6: time-series | ○ | – | + | – | ○ | ○ | – | ○ | + |

^{a}The significant bias introduced through missing values is not included in the rating.

## 5. Discussion

## 6. Conclusions

## Acknowledgments

## References

- Allen, J.; Holt, J.; Blackford, J.; Proctor, R. Error quantification of a high-resolution coupled hydrodynamic-ecosystem coastal-ocean model: Part 2. Chlorophyll-a, nutrients and SPM. J. Marine Syst.
**2007**, 68, 381–404. [Google Scholar] [CrossRef] - Stow, C.; Jolliff, J.; McGillicuddy, D.; Doney, S.; Allen, J.; Friedrichs, M.; Rose, K.; Wallhead, P. Skill assessment for coupled biological/physical models of marine systems. J. Marine Syst.
**2009**, 76, 4–15. [Google Scholar] [CrossRef] - Lehmann, M. K.; Fennel, K.; He, R. Statistical validation of a 3-D bio-physical model of the western North Atlantic. Biogeosciences
**2009**, 6, 1961–1974. [Google Scholar] [CrossRef] - Bennett, A. Inverse Modeling of the Ocean and Atmosphere; Cambridge University Press: Cambridge, UK, 2002. [Google Scholar]
- Dowd, M. Bayesian statistical data assimilation for ecosystem models using Markov Chain Monte Carlo. J. Marine Syst.
**2007**, 68, 439–456. [Google Scholar] [CrossRef] - Avcıbaş, İ.; Sankur, B.; Sayood, K. Statistical evaluation of image quality measures. J. Electron. Imag.
**2002**, 11, 206–245. [Google Scholar] - Mannino, A.; Russ, M.; Hooker, S. Algorithm development and validation for satellite-derived distributions of DOC and CDOM in the US Middle Atlantic Bight. J. Geophys. Res.
**2008**, 113, C07051. [Google Scholar] - Lehmann, T.; Sovakar, A.; Schmiti, W.; Repges, R. A comparison of similarity measures for digital subtraction radiography. Comput. Biol. Med.
**1997**, 27, 151–167. [Google Scholar] [CrossRef] - Le Moigne, J.; Tilton, J. Refining image segmentation by integration of edge and region data. IEEE T. Geosci. Remote Sens.
**1995**, 33, 605–615. [Google Scholar] [CrossRef] - Alberga, V. Similarity measures of remotely sensed multi-sensor images for change detection applications. Remote Sens.
**2009**, 1, 122–143. [Google Scholar] [CrossRef] - Holyer, R.; Peckinpaugh, S. Edge detection applied to satellite imagery of the oceans. IEEE T. Geosci. Remote Sens.
**1989**, 27, 46–56. [Google Scholar] [CrossRef] - Cayula, J.; Cornillon, P. Edge detection algorithm for SST images. J. Atmos. Ocean. Tech.
**1992**, 9, 67–80. [Google Scholar] [CrossRef] - Belkin, I.; O’Reilly, J. An algorithm for oceanic front detection in chlorophyll and SST satellite imagery. J. Marine Syst.
**2009**, 78, 319–326. [Google Scholar] [CrossRef] - Nichol, D. Autonomous extraction of an eddy-like structure from infrared images of the ocean. IEEE T. Geosci. Remote Sens.
**1987**, 25, 28–34. [Google Scholar] [CrossRef] - Santini, S.; Jain, R. Similarity measures. IEEE T. Pattern Anal.
**1999**, 21, 871–883. [Google Scholar] [CrossRef] - Di Gesú, V.; Starovoitov, V. Distance-based functions for image comparison. Pattern Recog. Lett.
**1999**, 20, 207–214. [Google Scholar] [CrossRef] - Di Gesú, V.; Roy, S. Pictorial indexes and soft image distances. Springer: Berlin/Heidelberg, Germany, 2002; pp. 63–79. [Google Scholar] [CrossRef]
- Huttenlocher, D.; Klanderman, G.; Rucklidge, W. Comparing images using the Hausdorff distance. IEEE T. Pattern Anal.
**1993**, 15, 850–863. [Google Scholar] [CrossRef] - Wilson, D.; Baddeley, A.; Owens, R. A New Metric for Grey-Scale Image Comparison. Int. J. Comput. Vision
**1997**, 24, 5–17. [Google Scholar] [CrossRef] - Wang, L.; Zhang, Y.; Feng, J. On the Euclidean distance of images. IEEE T. Pattern Anal.
**2005**, 27, 1334–1339. [Google Scholar] [CrossRef] [PubMed] - Juffs, P.; Beggs, E.; Deravi, F. A multiresolution distance measure for images. IEEE Sig. Proc. Lett.
**1998**, 5, 138–140. [Google Scholar] [CrossRef] - Dubuisson, M.; Jain, A. A modified Hausdorff distance for object matching. In Proceedings of the 12th IAPR International Conference on Computer Vision & Image Processing, Jerusalem, Israel, 1994; pp. 566–568.
- Sim, D.; Kwon, O.; Park, R. Object matching algorithms using robust Hausdorff distance measures. IEEE T. Image Proc.
**1999**, 8, 425–429. [Google Scholar] - Fennel, K.; Wilkin, J.; Previdi, M.; Najjar, R. Denitrification effects on air-sea CO
_{2}flux in the coastal ocean: simulations for the Northwest North Atlantic. Geophys. Res. Lett.**2008**, 35, L24608. [Google Scholar] [CrossRef]

© 2010 by the authors; licensee Molecular Diversity Preservation International, Basel, Switzerland. This article is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license http://creativecommons.org/licenses/by/3.0/.

## Share and Cite

**MDPI and ACS Style**

Mattern, J.P.; Fennel, K.; Dowd, M.
Introduction and Assessment of Measures for Quantitative Model-Data Comparison Using Satellite Images. *Remote Sens.* **2010**, *2*, 794-818.
https://doi.org/10.3390/rs2030794

**AMA Style**

Mattern JP, Fennel K, Dowd M.
Introduction and Assessment of Measures for Quantitative Model-Data Comparison Using Satellite Images. *Remote Sensing*. 2010; 2(3):794-818.
https://doi.org/10.3390/rs2030794

**Chicago/Turabian Style**

Mattern, Jann Paul, Katja Fennel, and Michael Dowd.
2010. "Introduction and Assessment of Measures for Quantitative Model-Data Comparison Using Satellite Images" *Remote Sensing* 2, no. 3: 794-818.
https://doi.org/10.3390/rs2030794