# An Uncertainty-Aware Visual System for Image Pre-Processing

^{1}

^{2}

^{3}

^{4}

^{*}

## Abstract

**:**

- A summary of uncertainty propagation rules
- Propagation of image uncertainty throughout arbitrary image pre-processing operations
- An intuitive visual system to create arbitrary image pre-processing pipelines and review the impact of the pipeline design to the image uncertainty

## 1. Related Work

#### 1.1. Uncertainty Propagation in Image Pre-Processing Operations

#### 1.2. Uncertainty-Aware Computer Vision

#### 1.3. Uncertainty-Aware Visualization of Images

## 2. Requirements for Uncertainty-Aware Image Pre-Processing

#### 2.1. Uncertainty-Awareness

**U1**Quantify uncertainty in each point**U2**Visualize uncertainty information**U3**Enable interactive uncertainty exploration**U4**Propagate and aggregate uncertainty

#### 2.2. Meaningful Visualization

**M1**Usability**M2**Effectiveness**M3**Correctness**M4**Intuitiveness**M5**Flexibility

## 3. Definitions

#### 3.1. Image Definition

- ${I}_{({t}_{1},\dots ,{t}_{d}),{s}^{c}}$ defines a new image I with $\mathbb{V}=\{1,\dots ,{t}_{1}\}\times \dots \times \{1,\dots ,{t}_{d}\}$ and $\mathbb{X}:={\{1,\dots ,s\}}^{c}$
- $I(x,y)$ is the selection of all Image compartments where x and y are valid
- $t(I,i)$, $d\left(I\right)$, $s\left(I\right)$, and $c\left(I\right)$ outputting t, d, s and c as defined for images
- $n(I,v,k)$ outputting all pixels an Image that have the distance k to the pixel v
- $\Sigma \left(I\right)$ and $\sigma \left(I\right)$ outputting, the average pixel value and the standard deviation of all pixel values
- $h\left(I\right(x\left)\right)$ defines the histogram of the image. This functions outputs the occurrences of a specific $x\phantom{\rule{4.pt}{0ex}}\mathrm{in}\phantom{\rule{4.pt}{0ex}}\mathbb{X}$ in the given input image

#### 3.2. Uncertainty Propagation Rules

#### 3.2.1. Multiplication by a Constant

#### 3.2.2. Addition/Subtraction of Two or More Values

#### 3.2.3. Multiplication/Division of Two Values

#### 3.2.4. Raising to a Power

#### 3.2.5. Arbitrary Formulas with One Variable

#### 3.2.6. Formulas with Multiple Variables

#### 3.2.7. Important Composed Functions

## 4. Methods

#### 4.1. Uncertainty Quantification of Input Images

#### 4.2. Visualization of Images and Their Uncertainty

- Juxtaposition of all uncertainty dimensions
- Superposition of all uncertainty dimensions
- Embedded iso-surface visualization of all uncertainty dimensions.

#### 4.3. Pre-Processing Steps Under Uncertainty

#### 4.3.1. Thresholding

#### 4.3.2. Histogram Equalization

#### 4.3.3. Kernel Operations

#### 4.4. Multiple Uncertainty-Aware Operators

## 5. Results

#### 5.1. Sobel Operator

#### 5.2. Medical Image Data

#### 5.3. Satellite Images

## 6. Discussion

#### 6.1. Discussion of Uncertainty Principle

**C1: Quantify uncertainty in each component.**The presented methodology is able to quantify the uncertainty of each image pixel. Here, we utilize different error measures to create an arbitrary high-dimensional uncertainty space capturing multiple aspects of uncertainty. Users are enabled to select the utilized error measures to achieve an uncertainty quantification that meets their needs and fits to the underlying image data best.

**C2: Visualize uncertainty information.**In the presented methodology, visualization is a key concept. Each computational step and its uncertainty is visually presented. Here, users can select from three different visualization modes depending on the visualization purpose.

**C3: Enable interactive uncertainty exploration.**As shown in the presented results, our methodology allows an interactive composition of multiple uncertainty-aware image pre-processing operations. All parameters and settings can be selected by the user. Users can choose steps and regions they are particularly interested in.

**C4: Propagate and aggregate uncertainty.**The propagation and aggregation of uncertainty is ensured by refining a variety of image preprocessing algorithms such that they are able to transform the image properly. This means that the image pre-processing operations are including uncertainty information throughout their computation. In addition, the presented operations are able to adjust the underlying image’s uncertainty as well. The presented system is designed such that these transformations are propagated along arbitrary image pre-processing pipelines.

#### 6.2. Discussion of Requirements for a Real World Use

**M1: Usability.**The presented methodology is easy to use for domain scientists as is allows to generate arbitrary pre-processing pipelines. The visualization and arrangement of individual computational steps help users to understand the composition of the single steps and how they change the underlying uncertainty. Visualization types can be chosen according to individual needs.

**M2: Effectiveness.**The effectiveness of the presented methodology is ensured although the additional information needs to be stored and computed. When considering n as the number of pixels, the required storage for the presented methodology is $q\xb7r\xb7(n+1)$, where q is the number of uncertainty measures and r the number of computations selected by the users. The computational effort is highly depending on the underlying image operation. When assuming O as an upper bound of all utilized image operations, the resulting runtime complexity can be expressed by $q\xb7r\xb7O$. For both, storage efficiency and runtime efficiency, the presented methodology solely adds a linear factor. In many related approaches, the runtime exceeds linearity. In addition, the approach can be parallelized as different cores do not require the input image in its entirety.

**M3: Correctness.**As the results show, the presented uncertainty-aware image pre-processing operations output correct results, such as the classic versions of each operator does. In addition it provides a refinement of these operators that allows to express the uncertainty of the computational result.

**M4: Intuitiveness.**The presented methodology is intuitive as it allows an easy to understand selection and connection process of the provided uncertainty-aware pre-processing operations. The computational results are sorted by their order to allow an intuitive reviewing by users.

**M5: Flexibility.**The flexibility of our system is high as we allow a variety of uncertainty measures and their combination. The presented system additionally allows the inclusion of further uncertainty measures. Furthermore, it is able to handle datasets that already have an uncertainty quantification. For the visualization of the uncertainty quantification in each step, we allow three different visualization types: juxtaposition, superposition, and embedded iso-surface visualizations. Furthermore, the methodology is not restricted in terms of image data that can be utilized as input. 2D or 3D images, color-images or grey-scale, and different image types can be considered. Still, an extension of the visualization methodologies would be required when considering 3D images [43]. In such cases, visual clutter can occur and a proper visualization strategy for uncertain pixels need to be found. A further limitation of the presented approach can be reached, when images became very large. At some point, not all intermediate steps can be displayed at the same time without a huge zoom factor. Here, focus and context approaches, such as hierarchical clustering, are required [44]. The flexibility of the presented methodology can be increased further when adding new uncertainty-aware image pre-processing operations. The code is designed such that users are able to implement new operations when required regarding the given uncertainty-propagation principles.

## 7. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Sonka, M.; Hlavac, V.; Boyle, R. Image pre-processing. In Image Processing, Analysis and Machine Vision; Springer: Boston, MA, USA, 1993; pp. 56–111. [Google Scholar]
- Ekman, U.; Agostinho, D.; Thylstrup, N.B.; Veel, K. The uncertainty of the uncertain image. Digit. Creat.
**2017**, 28, 255–264. [Google Scholar] [CrossRef] - Gillmann, C.; Wischgoll, T.; Hagen, H. Uncertainty-Awareness in Open Source Visualization Solutions. In Proceedings of the IEEE Visualization Conference (VIS)—VIP Workshop, Baltimore, MD, USA, 23–28 October 2016. [Google Scholar]
- Wilson, R.; Granlund, G.H. The Uncertainty Principle in Image Processing. IEEE Trans. Pattern Anal. Mach. Intell.
**1984**, PAMI-6, 758–767. [Google Scholar] [CrossRef] - PAL, S.K. Fuzzy image processing and recognition: uncertainty handling and applications. Int. J. Image Gr.
**2001**, 1, 169–195. [Google Scholar] [CrossRef] - Mencattini, A.; Rabottino, G.; Salicone, S.; Salmeri, M. Uncertainty handling and propagation in X-ray images analysis systems by means of Random-Fuzzy Variables. In Proceedings of the 2008 IEEE International Workshop on Advanced Methods for Uncertainty Estimation in Measurement, Trento, Italy, 21–22 July 2008; pp. 50–55. [Google Scholar] [CrossRef]
- Ji, Q.; Haralick, R.M. Error propagation for the Hough transform. Pattern Recognit. Lett.
**2001**, 22, 813–823. [Google Scholar] [CrossRef] - Davis, T.J.; Keller, C.P. Modelling and Visualizing Multiple Spatial Uncertainties. Comput. Geosci.
**1997**, 23, 397–408. [Google Scholar] [CrossRef] - Yi, S.; Haralick, R.M.; Shapiro, L.G. Error propagation in machine vision. Mach. Vis. Appl.
**1994**, 7, 93–114. [Google Scholar] [CrossRef] - Krishnapuram, R.; Keller, J.M. Fuzzy set theoretic approach to computer vision: An overview. In Proceedings of the IEEE International Conference on Fuzzy Systems, San Diego, CA, USA, 8–12 March 1992; pp. 135–142. [Google Scholar] [CrossRef]
- Rangarajan, C.; Jayaramamurthy, S.; Huntsberger, T. Representation of Uncertainty in Computer Vision Using Fuzzy Sets. IEEE Trans. Comput.
**1986**, 35, 145–156. [Google Scholar] [CrossRef] - Haralick, R.M. Propagating Covariance in Computer Vision. In Performance Characterization in Computer Vision; Klette, R., Stiehl, H.S., Viergever, M.A., Vincken, K.L., Eds.; Springer: Dordrecht, The Netherlands, 2000; pp. 95–114. [Google Scholar]
- Hua, M.; Pei, J. Probabilistic Path Queries in Road Networks: Traffic Uncertainty Aware Path Selection. In Proceedings of the 13th International Conference on Extending Database Technology, Lausanne, Switzerland, 22–26 March 2010; pp. 347–358. [Google Scholar] [CrossRef]
- Stachowiak, A.; Żywica, P.; Dyczkowski, K.; Wójtowicz, A. An Interval-Valued Fuzzy Classifier Based on an Uncertainty-Aware Similarity Measure. In Intelligent Systems’2014; Angelov, P., Atanassov, K., Doukovska, L., Hadjiski, M., Jotsov, V., Kacprzyk, J., Kasabov, N., Sotirov, S., Szmidt, E., Zadrożny, S., Eds.; Springer International Publishing: Cham, Switzerland, 2015; pp. 741–751. [Google Scholar]
- Bonneau, G.P.; Hege, H.C.; Johnson, C.R.; Oliveira, M.M.; Potter, K.; Rheingans, P.; Schultz, T. Overview and State-of-the-Art of Uncertainty Visualization. In Scientific Visualization: Uncertainty, Multifield, Biomedical, and Scalable Visualization; Hansen, C.D., Chen, M., Johnson, C.R., Kaufman, A.E., Hagen, H., Eds.; Springer: London, UK, 2014; pp. 3–27. [Google Scholar]
- Lundström, C.; Ljung, P.; Persson, A.; Ynnerman, A. Uncertainty Visualization in Medical Volume Rendering Using Probabilistic Animation. IEEE Trans. Vis. Comput. Gr.
**2007**, 13, 1648–1655. [Google Scholar] [CrossRef] [PubMed] - Fleßner, M.; Müller, A.; Götz, D.; Helmecke, E.; Hausotte, T. Assessment of the single point uncertainty of dimensional CT measurements. In Proceedings of the 6th Conference on Industrial Computed Tomography, Wels, Austria, 9–12 February 2016. [Google Scholar]
- Wong, P.C.; Bergeron, R.D. 30 Years of Multidimensional Multivariate Visualization. In Scientific Visualization, Overviews, Methodologies, and Techniques; IEEE Computer Society: Washington, DC, USA, 1997; pp. 3–33. [Google Scholar]
- Ding, Z.; Ding, Z.; Chen, W.; Chen, H.; Tao, Y.; Li, X.; Chen, W. Visual inspection of multivariate volume data based on multi-class noise sampling. Vis. Comput.
**2016**, 32, 465–478. [Google Scholar] [CrossRef] - Akiba, H.; Ma, K.L.; Chen, J.H.; Hawkes, E.R. Visualizing Multivariate Volume Data from Turbulent Combustion Simulations. Comput. Sci. Eng.
**2007**, 9, 76–83. [Google Scholar] [CrossRef][Green Version] - Ma, K.; Lum, E.B.; Muraki, S. Recent advances in hardware-accelerated volume rendering. Comput. Gr.
**2003**, 27, 725–734. [Google Scholar] [CrossRef] - Gillmann, C.; Arbeláez, P.; Hernández, J.T.; Hagen, H.; Wischgoll, T. Intuitive Error Space Exploration of Medical Image Data in Clinical Daily Routine. In EuroVis 2017—Short Papers; Kozlikova, B., Schreck, T., Wischgoll, T., Eds.; The Eurographics Association: Lyon, France, 2017. [Google Scholar] [CrossRef]
- Sacha, D.; Senaratne, H.; Kwon, B.C.; Ellis, G.; Keim, D.A. The Role of Uncertainty, Awareness, and Trust in Visual Analytics. IEEE Trans. Vis. Comput. Gr.
**2016**, 22, 240–249. [Google Scholar] [CrossRef] [PubMed] - Maack, R.G.C.; Hagen, H.; Gillmann, C. Evaluation of the uncertainty-awareness principleusing a tumor detection scenario. In Proceedings of the VisGuides: 2nd Workshop on the Creation, Curation, Critique and Conditioning of Principles and Guidelines in Visualization (IEEE Vis 2018), Berlin, Germany, 21–22 October 2018. [Google Scholar]
- Gillmann, C.; Leitte, H.; Wischgoll, T.; Hagen, H. From Theory to Usage: Requirements for successful Visualizations in Applications. In Proceedings of the IEEE Visualization Conference (VIS)—C4PGV Workshop, Baltimore, MA, USA, 23 October 2016. [Google Scholar]
- Taylor, J. Introduction To Error Analysis: The Study of Uncertainties in Physical Measurements; A Series of Books in Physics; University Science Books: South Orange, NJ, USA, 1997. [Google Scholar]
- Rumelhart, D.; Hinton, G.; Williams, R.; University of California, S.D.I.f.C.S. Learning Internal Representations by Error Propagation; ICS Report; Institute for Cognitive Science; University of California: San Diego, CA, USA, 1985. [Google Scholar]
- Gertsbakh, I. Measurement Uncertainty: Error Propagation Formula. In Measurement Theory for Engineers; Springer: Berlin/Heidelberg, Germany, 2003; pp. 87–94. [Google Scholar]
- Bevington, P.R.; Robinson, D.K. Data Reduction and Error Analysis for the Physical Sciences, 3rd ed.; McGraw-Hill: New York, NY, USA, 2003. [Google Scholar]
- Nguyen, H.T. A Survey of Uncertainty Measures: Structures and Applications. In Proceeding of the 1984 American Control Conference, San Diego, CA, USA, 6–8 June 1984; pp. 549–553. [Google Scholar] [CrossRef]
- Artmann, U. Image quality assessment using the dead leaves target: experience with the latest approach and further investigations. Proc. SPIE
**2015**, 9404, 94040J. [Google Scholar] - Pizer, S.M.; Amburn, E.P.; Austin, J.D.; Cromartie, R.; Geselowitz, A.; Greer, T.; Romeny, B.T.H.; Zimmerman, J.B. Adaptive Histogram Equalization and Its Variations. Comput. Vis. Gr. Image Process.
**1987**, 39, 355–368. [Google Scholar] [CrossRef] - Fienup, J.R. Invariant error metrics for image reconstruction. Appl. Opt.
**1997**, 36, 8352–8357. [Google Scholar] [CrossRef] [PubMed] - Boyat, A.K.; Joshi, B.K. A Review Paper: Noise Models in Digital Image Processing. arXiv, 2015; arXiv:1505.03489. [Google Scholar] [CrossRef]
- L, J.V.; Gopikakumari, R. Article: IEM: A New Image Enhancement Metric for Contrast and Sharpness Measurements. Int. J. Comput. Appl.
**2013**, 79, 1–9. [Google Scholar] - Tian, Q.; Xue, Q.; Sebe, N.; Huang, T. Error metric analysis and its applications. Proc. SPIE Int. Soc. Opt. Eng.
**2004**, 5601, 46–57. [Google Scholar] - Irum, I.; Sharif, M.; Raza, M.; Mohsin, S. A Nonlinear Hybrid Filter for Salt & Pepper Noise Removal from Color Images. J. Appl. Res. Technol.
**2015**, 13, 79–85. [Google Scholar] - Sahoo, P.; Soltani, S.; Wong, A. A survey of thresholding techniques. Comput. Vis. Gr. Image Proc.
**1988**, 41, 233–260. [Google Scholar] [CrossRef] - Zuiderveld, K. Graphics Gems IV; Chapter Contrast Limited Adaptive Histogram Equalization; Academic Press Professional, Inc.: San Diego, CA, USA, 1994; pp. 474–485. [Google Scholar]
- Shawe-Taylor, J.; Cristianini, N. Kernel Methods for Pattern Analysis; Cambridge University Press: New York, NY, USA, 2004. [Google Scholar]
- Huo, Y.; Choy, J.S.; Wischgoll, T.; Luo, T.; Teague, S.D.; Bhatt, D.L.; Gebab, G.S. Computed tomography-based diagnosis of diffuse compensatory enlargment of coronary arteries using scaling power laws Interface. J. R. Soc. Interface
**2002**, 10, H514–H523. [Google Scholar] - Planet Team. Planet Application Program Interface: In Space for Life on Earth; Planet Team: San Francisco, CA, USA, 2017; Available online: https://api.planet.com (accessed on 28 August 2018 ).
- Djurcilov, S.; Kim, K.; Lermusiaux, P.; Pang, A. Visualizing Scalar Volumetric Data with Uncertainty. Comput. Gr.
**2002**, 26, 239–248. [Google Scholar] [CrossRef] - Chen, Z.; Healey, C. Large Image Collection Visualization Using Perception-Based Similarity with Color Features. In International Symposium on Visual Computing; Springer: Cham, Switzerland, 2016; pp. 379–390. [Google Scholar]

**Figure 1.**Uncertainty propagation rules for different operations. Top: input signals and their uncertainty. Bottom Output signals and their transformed uncertainty. (

**a**,

**b**) Multiplication with a constant c; (

**c**,

**d**) Addition of two signals; (

**e**,

**f**) Multiplication of two signals; (

**g**,

**h**) Raising a signal to the power 1.5.

**Figure 2.**Overview of the presented uncertainty-aware image pre-processing. Input images can be quantified by an arbitrary amount of uncertainty measures. Users can depict adjusted image pre-processing operations transforming the input image and the input image’s uncertainty. This can be repeated arbitrarily often.

**Figure 3.**Visualization of uncertainty for a single step in the pre-processing pipeline (

**a**); (

**b**–

**h**) Juxtaposition of the utilized uncertainty measures Uncertainty quantification measures for images (Acutance, Contrast stretch, Gaussian error, Brightness, Local contrast, Local range, Salt and pepper); (

**i**) Superposition of all uncertainty measures utilizing the length of vector as a metric. The color scales reach from no uncertainty (light blue) over medium uncertainty (white uncertainty) to maximum uncertainty (orange); (

**j**) Embedded iso-surface visualization utilizing the method presented by Gillmann et al. [22].

**Figure 4.**Comparison of image pixels under uncertainty. (

**a**) First input image and the corresponding uncertainty utilizing the Gaussian error; (

**b**) Second input image utilizing the Gaussian error; (

**c**) Resulting image and modified uncertainty, when utilizing the uncertainty-aware pixel-wise image comparison.

**Figure 5.**Histogram equalization with and without considering the input images uncertainty. (

**a**) Input image; (

**b**) Histogram equalization without considering uncertainty information; (

**c**) Histogram equalization when considering uncertainty information.

**Figure 6.**Uncertainty-aware kernel operations. (

**a**) The input image and it’s uncertainty quantification; (

**b**) Image utilizing a Gaussian kernel (5 × 5) and its resulting uncertainty; (

**c**) Image utilizing the Laplace edge detection and the resulting uncertainty.

**Figure 7.**Connection of multiple image pre-processing operations to form the Sobel operator. Starting from the input image and its uncertainty (top), the Sobel operator in x and y direction can be computed (center). Resulting from this, the length of vector function considering both Sobel operators as input can be computed to achieve the overall result of the Sobel edge detection.

**Figure 8.**Edge detection for vascular analysis. (

**a**) Input image and uncertainty quantification; Sobel operator in x-direction (

**b**) and y-direction (

**c**) combined with the length of vector operation (

**e**); The resulting uncertainty quantification is shown respectively; Gaussian smoothing (

**d**) followed by a Laplace edge detection (

**f**); Both edge detection can be combined by utilizing the length of vector function as shown in (

**g**).

**Figure 9.**Uncertainty-aware image pre-processing applied to a satellite image of Tokyo. (

**a**) Input image; (

**b**) Contrast equalization; (

**c**) Gaussian Smoothing; (

**d**) Sobel operator in x-direction; (

**e**) Sobel operator in y-direction; (

**f**) Length of vector.

**Figure 10.**Closeup of the final result and its uncertainty of the uncertainty-aware image pre-processing pipeline shown in Figure 9f.

Name | Formula | Uncertainty Propagation |
---|---|---|

Exponential Function | $y={e}^{x}$ | $\Delta y={e}^{\left({\Delta}_{z}\right)}$ |

Logarithm | $y=log\left(x\right)$ | $\Delta y=\frac{1}{\Delta x}$ |

Sine | $y=sin\left(x\right)$ | $\Delta y=cos(\Delta x)$ |

Cosine | $y=cos\left(x\right)$ | $\Delta y=-sin(\Delta x)$ |

Chain Rule | $y=f\left(g\right(x\left)\right)$ | ${f}^{\prime}\left(g\left(x\right)\right)\xb7{g}^{\prime}\left(x\right)$ |

Product Rule | $y=f\left(x\right)\xb7g\left(x\right)$ | ${f}^{\prime}\left(x\right)\xb7g\left(x\right)+f\left(x\right)\xb7{g}^{\prime}\left(x\right)$ |

Quotient Rule | $y=f\left(x\right)/g\left(x\right)$ | $\frac{{g}^{\prime}\left(x\right)h\left(x\right)-g\left(x\right){h}^{\prime}\left(x\right)}{h{\left(x\right)}^{2}}$ |

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Gillmann, C.; Arbelaez, P.; Hernandez, J.T.; Hagen, H.; Wischgoll, T. An Uncertainty-Aware Visual System for Image Pre-Processing. *J. Imaging* **2018**, *4*, 109.
https://doi.org/10.3390/jimaging4090109

**AMA Style**

Gillmann C, Arbelaez P, Hernandez JT, Hagen H, Wischgoll T. An Uncertainty-Aware Visual System for Image Pre-Processing. *Journal of Imaging*. 2018; 4(9):109.
https://doi.org/10.3390/jimaging4090109

**Chicago/Turabian Style**

Gillmann, Christina, Pablo Arbelaez, Jose Tiberio Hernandez, Hans Hagen, and Thomas Wischgoll. 2018. "An Uncertainty-Aware Visual System for Image Pre-Processing" *Journal of Imaging* 4, no. 9: 109.
https://doi.org/10.3390/jimaging4090109