# Confidence Estimation for Machine Learning-Based Quantitative Photoacoustics

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Materials and Methods

**Method for Confidence Estimation.**Our approach to confidence estimation can be applied to any qPAI method designed to convert an input image I (${p}_{0}$ or raw time-series data) into an image of optical absorption ${I}_{{\mu}_{a}}$. In order to not restrict the qPAI method to a certain class (e.g., a deep learning-based method), we made the design decision to base the confidence quantification method on an external observing method. For this, we use a neural network, which is presented tuples of input image I and absorption quantification error ${e}_{{\mu}_{a}}$ in the training phase. When applying the method to a previously unseen image I, the following steps are performed (cf. Figure 1).

- (1)
- Quantification of aleatoric uncertainty: I is converted into an image ${I}^{\mathrm{aleatoric}}$ reflecting the aleatoric uncertainty. For this purpose, we use the contrast-to-noise-ratio (CNR), as defined by Welvaert and Rosseel [45], as $\mathrm{CNR}=\left(S-avg\left(b\right)\right)/std\left(b\right)$, with S being the pixel signal intensity, and $avg\left(b\right)$ and $std\left(b\right)$ being the mean and standard deviation of all background pixels in the dataset. Using this metric, we predefine our ROI to comprise all pixels with CNR > 5.
- (2)
- Quantification of epistemic confidence: I is converted into an image ${I}^{\mathrm{epistemic}}$ reflecting the epistemic confidence. For this purpose, we use the external model to estimate the quantification error ${e}_{{\mu}_{a}}$ of the qPAI algorithm.
- (3)
- Output generation: A threshold over ${I}^{\mathrm{aleatoric}}$ yields a binary image with an ROI representing confident pixels in ${I}_{{\mu}_{a}}$ according to the input signal intensity. We then proceed to narrow down the ROI by applying a confidence threshold ($CT$) which removes the n% least confident pixels according to ${I}^{\mathrm{epistemic}}$.

**Deep Learning Model.**As our external observing network, we used an adapted version of a standard U-Net [46] implemented in PyTorch [47]. The model uses $2\times 2$ max pooling for downscaling, $2\times 2$ transpose convolutions for upscaling, and all convolution layers have a kernel size of $3\times 3$ and a padding of $1\times 1$. We modified the skip connections to incorporate a total of three convolution layers and thus generate an asymmetric U-Net capable of dealing with different input and output sizes (cf. Figure 2). This is necessary to enable the network to directly output reconstructed initial pressure or optical absorption distributions when receiving raw time-series data as input, as the data have different sizes on the y-axis. Specifically, the second of these convolutions was modified to have a kernel size of $3\times 20$, a stride of $1\times 20$, and a padding of $1\times 9$, effectively scaling down the input along the y-axis by a factor of 20. To be more robust to overfitting, we added dropout layers with a dropout rate of $25\%$ to each convolutional layer of the network. Note that a recent study [48] suggests that the U-net architecture is particularly well-suited for medical imaging applications because of its ability to generate data representations on many abstraction levels.

**Quantitative PAI Methods.**We applied our approach to confidence estimation to three different qPAI methods; a naïve quantification method, as well as two different deep learning-based approaches (cf. Figure 3). The three methods are detailed in the following paragraphs.

**Validation Data.**We simulated an in silico dataset containing 3600 training samples, 400 validation samples, as well as 150 calibration and test samples which were used in all experiments. Each data sample consists of the ground truth optical tissue parameters, the corresponding light fluence, and the initial pressure distribution simulated with the mcxyz framework [51], which is a Monte Carlo simulation of photon propagation where we assume a symmetric illumination geometry with two laser outputs. The data sample also comprises raw time-series data simulated using the k-Wave [52] toolkit with the first-order 2D k-space method, assuming a 128-element linear array ultrasound transducer with a central frequency of 7.5 MHz, a bandwidth of 60%, and a sample rate of 1.5 $\times \phantom{\rule{3.33333pt}{0ex}}{10}^{-8}$ s. The illumination and ultrasound geometry are depicted in Figure 4. Each tissue volume sample comprises 1–10 tubular vessel structures, whose absorption coefficients ${\mu}_{a}$ range from 2 to 10 cm${}^{-1}$ in vessel structures and are assumed constant with 0.1 cm${}^{-1}$ in the background. We chose a constant reduced scattering coefficient of 15 cm${}^{-1}$ in both background and vessel structures. Additional details on the simulation parameters can also be found in our previous work [53]. The raw time-series data was noised after k-space simulation with an additive Gaussian noise model of recorded noise of our system [54]. For the experiments where we directly use ${p}_{0}$, we noise the initial pressure distribution with a Gaussian additive noise model, as also described in [7]. In our case, the noise model comprised an additive component with $(5\pm 5)\%$ of the mean signal and a multiplicative component with a standard deviation of $20\%$ to simulate imperfect reconstructions of ${p}_{0}$. The data used in this study is available in a Zenodo repository [55].

**Experimental Design.**We performed five in silico experiments to validate our approach to confidence estimation in qPAI—one experiment with naïve fluence correction applied to ${p}_{0}$ data, as well as our different configurations for quantification with deep learning. Both methods shown in Figure 3 are applied to initial pressure ${p}_{0}$, as well as raw time-series data. We used the Trixi [56] framework to perform the experiments. All qPAI models were trained on the training set and the progress was supervised with the validation set. We also used the validation set for hyperparameter optimization of the number of training epochs and batch sizes. We trained for 50 epochs, showing the network ${10}^{4}$ randomly drawn and augmented samples from the training set; each epoch had a learning rate of ${10}^{-4}$, used an ${L}_{1}$ loss function, and augmented every sample with a white Gaussian multiplicative noise model and horizontal mirroring to prevent the model from overfitting. Afterward, we estimated the optical absorption ${\widehat{\mu}}_{a}$ of the validation set, calculated the relative errors ${e}_{{\mu}_{a}}=|{\widehat{\mu}}_{a}-{\mu}_{a}|/{\mu}_{a}$, and trained the external observing neural networks on the errors of the validation set with the same hyperparameters, supervising the progression on the calibration set. For better convergence, we used the weights of the ${p}_{0}$ estimation model as a starting point for the ${e}_{{\mu}_{a}}$ estimation deep learning models. All results presented in this paper were computed on the test set.

## 3. Results

## 4. Discussion

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## Abbreviations

CNR | Contrast-To-Noise-Ratio |

PA | Photoacoustic |

PAI | Photoacoustic Imaging |

qPAI | quantitative PAI |

ROI | Region of Interest |

CT | Confidence Threshold |

## Appendix A. Results for the Naïve Fluence Compensation Method

**Figure A1.**Sample images showing the best, the worst, and the median performance of our method when considering only the 50% most confident quantification estimations. All images show the (

**a**) ground truth absorption coefficients (

**b**) reconstructed absorption, (

**c**) error estimate from external model, (

**d**) the actual quantification error.

## Appendix B. Results for Fluence Correction on ${\mathit{p}}_{\mathbf{0}}$ Data

**Figure A2.**Sample images showing the best, the worst, and the median performance of our method when considering only the 50% most confident quantification estimations. All images show the (

**a**) ground truth absorption coefficients (

**b**) reconstructed absorption, (

**c**) error estimate from external model, (

**d**) the actual quantification error.

## Appendix C. Results for Fluence Correction on Raw PA Time Series Data

**Figure A3.**Sample images showing the best, the worst, and the median performance of our method when considering only the 50% most confident quantification estimations. All images show the (

**a**) ground truth absorption coefficients (

**b**) reconstructed absorption, (

**c**) error estimate from external model, (

**d**) the actual quantification error.

## Appendix D. Results for Direct ${\mathit{\mu}}_{\mathit{a}}$ Estimation on Raw PA Time Series Data

**Figure A4.**Sample images showing the best, the worst, and the median performance of our method when considering only the 50% most confident quantification estimations. All images show the (

**a**) ground truth absorption coefficients (

**b**) reconstructed absorption, (

**c**) error estimate from external model, (

**d**) the actual quantification error.

## References

- Valluru, K.S.; Willmann, J.K. Clinical photoacoustic imaging of cancer. Ultrason
**2016**, 35, 267–280. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Knieling, F.; Neufert, C.; Hartmann, A.; Claussen, J.; Urich, A.; Egger, C.; Vetter, M.; Fischer, S.; Pfeifer, L.; Hagel, A.; et al. Multispectral Optoacoustic Tomography for Assessment of Crohn’s Disease Activity. N. Engl. J. Med.
**2017**, 376, 1292–1294. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Laufer, J. Photoacoustic Imaging: Principles and Applications. In Quantification of Biophysical Parameters in Medical Imaging; Springer: Cham, The Netherlands, 2018; pp. 303–324. [Google Scholar] [CrossRef]
- Mitcham, T.; Taghavi, H.; Long, J.; Wood, C.; Fuentes, D.; Stefan, W.; Ward, J.; Bouchard, R. Photoacoustic-based SO
_{2}estimation through excised bovine prostate tissue with interstitial light delivery. Photoacoust**2017**, 7, 47–56. [Google Scholar] [CrossRef] [PubMed] - Jacques, S.L. Optical properties of biological tissues: A review. Phys. Med. Boil.
**2013**, 58, R37. [Google Scholar] [CrossRef] [PubMed] - Tzoumas, S.; Nunes, A.; Olefir, I.; Stangl, S.; Symvoulidis, P.; Glasl, S.; Bayer, C.; Multhoff, G.; Ntziachristos, V. Eigenspectra optoacoustic tomography achieves quantitative blood oxygenation imaging deep in tissues. Nat. Commun.
**2016**, 7, 12121. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Cox, B.T.; Arridge, S.R.; Köstli, K.P.; Beard, P.C. Two-dimensional quantitative photoacoustic image reconstruction of absorption distributions in scattering media by use of a simple iterative method. Appl. Opt.
**2006**, 45, 1866–1875. [Google Scholar] [CrossRef] [PubMed] - Cox, B.; Laufer, J.; Beard, P. The challenges for quantitative photoacoustic imaging. Photons Plus Ultrasound: Imaging and Sensing. Int. Soc. Opt. Photonics
**2009**, 7177, 717713. [Google Scholar] - Cox, B.; Laufer, J.G.; Arridge, S.R.; Beard, P.C. Quantitative spectroscopic photoacoustic imaging: A review. J. Biomed. Opt.
**2012**, 17, 061202. [Google Scholar] [CrossRef] - Yuan, Z.; Jiang, H. Quantitative photoacoustic tomography: Recovery of optical absorption coefficient maps of heterogeneous media. Appl. Phys. Lett.
**2006**, 88, 231101. [Google Scholar] [CrossRef] - Yuan, Z.; Jiang, H. Quantitative photoacoustic tomography. Philos. Trans. R. Soc. Lond. A Math. Phys. Eng. Sci.
**2009**, 367, 3043–3054. [Google Scholar] [CrossRef] [Green Version] - Wang, Y.; He, J.; Li, J.; Lu, T.; Li, Y.; Ma, W.; Zhang, L.; Zhou, Z.; Zhao, H.; Gao, F. Toward whole-body quantitative photoacoustic tomography of small-animals with multi-angle light-sheet illuminations. Biomed. Opt. Express
**2017**, 8, 3778–3795. [Google Scholar] [CrossRef] [PubMed] - Saratoon, T.; Tarvainen, T.; Cox, B.; Arridge, S. A gradient-based method for quantitative photoacoustic tomography using the radiative transfer equation. Inverse Probl.
**2013**, 29, 075006. [Google Scholar] [CrossRef] - Tarvainen, T.; Pulkkinen, A.; Cox, B.T.; Arridge, S.R. Utilising the radiative transfer equation in quantitative photoacoustic tomography. Photons Plus Ultrasound Imaging Sens.
**2017**, 10064. [Google Scholar] [CrossRef] - Haltmeier, M.; Neumann, L.; Rabanser, S. Single-stage reconstruction algorithm for quantitative photoacoustic tomography. Inverse Probl.
**2015**, 31, 065005. [Google Scholar] [CrossRef] [Green Version] - Kaplan, B.A.; Buchmann, J.; Prohaska, S.; Laufer, J. Monte-Carlo-based inversion scheme for 3D quantitative photoacoustic tomography. Photons Plus Ultrasound: Imaging and Sensing. Int. Soc. Opt. Photonics
**2017**, 10064, 100645J. [Google Scholar] - Tzoumas, S.; Ntziachristos, V. Spectral unmixing techniques for optoacoustic imaging of tissue pathophysiology. Philos. Trans. R. Soc. A
**2017**, 375, 20170262. [Google Scholar] [CrossRef] [PubMed] - Perekatova, V.; Subochev, P.; Kleshnin, M.; Turchin, I. Optimal wavelengths for optoacoustic measurements of blood oxygen saturation in biological tissues. Biomed. Opt. Express
**2016**, 7, 3979–3995. [Google Scholar] [CrossRef] - Glatz, J.; Deliolanis, N.C.; Buehler, A.; Razansky, D.; Ntziachristos, V. Blind source unmixing in multi-spectral optoacoustic tomography. Opt. Express
**2011**, 19, 3175–3184. [Google Scholar] [CrossRef] - Cai, C.; Deng, K.; Ma, C.; Luo, J. End-to-end deep neural network for optical inversion in quantitative photoacoustic imaging. Opt. Lett.
**2018**, 43, 2752–2755. [Google Scholar] [CrossRef] - Kirchner, T.; Gröhl, J.; Maier-Hein, L. Context encoding enables machine learning-based quantitative photoacoustics. J. Biomed. Opt.
**2018**, 23, 056008. [Google Scholar] [CrossRef] [Green Version] - Fonseca, M.; Saratoon, T.; Zeqiri, B.; Beard, P.; Cox, B. Sensitivity of quantitative photoacoustic tomography inversion schemes to experimental uncertainty. SPIE BiOS. Int. Soc. Opt. Photonics
**2016**, 9708, 97084X. [Google Scholar] - Maier-Hein, L.; Franz, A.M.; Santos, T.R.D.; Schmidt, M.; Fangerau, M.; Meinzer, H.; Fitzpatrick, J.M. Convergent Iterative Closest-Point Algorithm to Accomodate Anisotropic and Inhomogenous Localization Error. IEEE Trans. Pattern Anal. Mach. Intell.
**2012**, 34, 1520–1532. [Google Scholar] [CrossRef] [PubMed] - Alterovitz, R.; Branicky, M.; Goldberg, K. Motion Planning Under Uncertainty for Image-guided Medical Needle Steering. Int. J. Robot. Res.
**2008**, 27, 1361–1374. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Sykes, J.R.; Brettle, D.S.; Magee, D.R.; Thwaites, D.I. Investigation of uncertainties in image registration of cone beam CT to CT on an image-guided radiotherapy system. Phys. Med. Boil.
**2009**, 54, 7263–7283. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Risholm, P.; Janoos, F.; Norton, I.; Golby, A.J.; Wells, W.M. Bayesian characterization of uncertainty in intra-subject non-rigid registration. Med. Image Anal.
**2013**, 17, 538–555. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Nair, T.; Precup, D.; Arnold, D.L.; Arbel, T. Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Granada, Spain, 16–20 September 2018; Springer: Berlin, Germany, 2018; pp. 655–663. [Google Scholar]
- Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res.
**2014**, 15, 1929–1958. [Google Scholar] - Gal, Y.; Ghahramani, Z. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the International Conference on Machine Learning, New York, NY, USA, 19–24 June 2016; pp. 1050–1059. [Google Scholar]
- Li, Y.; Gal, Y. Dropout Inference in Bayesian Neural Networks with Alpha-divergences. arXiv, 2017; arXiv:1703.02914. [Google Scholar]
- Leibig, C.; Allken, V.; Ayhan, M.S.; Berens, P.; Wahl, S. Leveraging uncertainty information from deep neural networks for disease detection. Sci. Rep.
**2017**, 7, 17816. [Google Scholar] [CrossRef] [Green Version] - Feindt, M. A Neural Bayesian Estimator for Conditional Probability Densities. arXiv, 2004; arXiv:Phys./0402093. [Google Scholar]
- Zhu, Y.; Zabaras, N. Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification. J. Comput. Phys.
**2018**, 366, 415–447. [Google Scholar] [CrossRef] - Kohl, S.A.; Romera-Paredes, B.; Meyer, C.; De Fauw, J.; Ledsam, J.R.; Maier-Hein, K.H.; Eslami, S.; Rezende, D.J.; Ronneberger, O. A Probabilistic U-Net for Segmentation of Ambiguous Images. arXiv, 2018; arXiv:1806.05034. [Google Scholar]
- Kingma, D.P.; Welling, M. Auto-encoding variational bayes. arXiv, 2013; arXiv:1312.6114. [Google Scholar]
- Mescheder, L.; Nowozin, S.; Geiger, A. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. arXiv, 2017; arXiv:1701.04722. [Google Scholar]
- Ardizzone, L.; Kruse, J.; Wirkert, S.; Rahner, D.; Pellegrini, E.W.; Klessen, R.S.; Maier-Hein, L.; Rother, C.; Köthe, U. Analyzing Inverse Problems with Invertible Neural Networks. arXiv, 2018; arXiv:1808.04730. [Google Scholar]
- Lakshminarayanan, B.; Pritzel, A.; Blundell, C. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In Advances in Neural Information Processing Systems 30; Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R., Eds.; Curran Associates, Inc.: Red Hook, NY, USA, 2017; pp. 6402–6413. [Google Scholar]
- Smith, L.; Gal, Y. Understanding Measures of Uncertainty for Adversarial Example Detection. arXiv, 2018; arXiv:1803.08533. [Google Scholar]
- Pulkkinen, A.; Cox, B.T.; Arridge, S.R.; Goh, H.; Kaipio, J.P.; Tarvainen, T. Direct estimation of optical parameters from photoacoustic time series in quantitative photoacoustic tomography. IEEE Trans. Med. Imaging
**2016**, 35, 2497–2508. [Google Scholar] [CrossRef] [PubMed] - Pulkkinen, A.; Cox, B.T.; Arridge, S.R.; Kaipio, J.P.; Tarvainen, T. Estimation and uncertainty quantification of optical properties directly from the photoacoustic time series. Photons Plus Ultrasound: Imaging and Sensing 2017. Int. Soc. Opt. Photonics
**2017**, 10064, 100643N. [Google Scholar] - Tick, J.; Pulkkinen, A.; Tarvainen, T. Image reconstruction with uncertainty quantification in photoacoustic tomography. J. Acoust. Soc. Am.
**2016**, 139, 1951–1961. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Tick, J.; Pulkkinen, A.; Tarvainen, T. Photoacoustic image reconstruction with uncertainty quantification. In EMBEC & NBC 2017; Eskola, H., Väisänen, O., Viik, J., Hyttinen, J., Eds.; Springer: Singapore, 2018; pp. 113–116. [Google Scholar]
- Gröhl, J.; Kirchner, T.; Maier-Hein, L. Confidence estimation for quantitative photoacoustic imaging. Photons Plus Ultrasound: Imaging and Sensing 2018. Int. Soc. Opt. Photonics
**2018**, 10494, 104941C. [Google Scholar] [CrossRef] - Welvaert, M.; Rosseel, Y. On the Definition of Signal-To-Noise Ratio and Contrast-To-Noise Ratio for fMRI Data. PLoS ONE
**2013**, 8, e77089. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI); Springer: Berlin, Germany, 2015; Volume 9351, pp. 234–241. [Google Scholar]
- Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; Lerer, A. Automatic Differentiation in PyTorch. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017. [Google Scholar]
- Isensee, F.; Petersen, J.; Klein, A.; Zimmerer, D.; Jaeger, P.F.; Kohl, S.; Wasserthal, J.; Koehler, G.; Norajitra, T.; Wirkert, S.; et al. nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation. arXiv, 2018; arXiv:1809.10486. [Google Scholar]
- Bauer, A.Q.; Nothdurft, R.E.; Erpelding, T.N.; Wang, L.V.; Culver, J.P. Quantitative photoacoustic imaging: Correcting for heterogeneous light fluence distributions using diffuse optical tomography. J. Biomed. Opt.
**2011**, 16, 096016–096016. [Google Scholar] [CrossRef] - Daoudi, K.; Hussain, A.; Hondebrink, E.; Steenbergen, W. Correcting photoacoustic signals for fluence variations using acousto-optic modulation. Opt. Express
**2012**, 20, 14117–14129. [Google Scholar] [CrossRef] [PubMed] - Jacques, S.L. Coupling 3D Monte Carlo light transport in optically heterogeneous tissues to photoacoustic signal generation. Photoacoust
**2014**, 2, 137–142. [Google Scholar] [CrossRef] [PubMed] - Treeby, B.E.; Cox, B.T. k-Wave: MATLAB toolbox for the simulation and reconstruction of photoacoustic wave fields. J. Biomed. Opt.
**2010**, 15, 021314. [Google Scholar] [CrossRef] [PubMed] - Waibel, D.; Gröhl, J.; Isensee, F.; Kirchner, T.; Maier-Hein, K.; Maier-Hein, L. Reconstruction of initial pressure from limited view photoacoustic images using deep learning. Photons Plus Ultrasound: Imaging and Sensing 2018. Int. Soc. Opt. Photonics
**2018**, 10494, 104942S. [Google Scholar] [CrossRef] - Kirchner, T.; Wild, E.; Maier-Hein, K.H.; Maier-Hein, L. Freehand photoacoustic tomography for 3D angiography using local gradient information. Photons Plus Ultrasound Imaging Sens.
**2016**, 9708, 97083G. [Google Scholar] [CrossRef] - Gröhl, J.; Kirchner, T.; Adler, T.; Maier-Hein, L. Silico 2D Photoacoustic Imaging Data; Zenodo: Meyrin, Switzerland, 2018. [Google Scholar] [CrossRef]
- Zimmerer, D.; Petersen, J.; Koehler, G.; Wasserthal, J.; Adler, T.; Wirkert, A. MIC-DKFZ/Trixi: Pre-Release; Zenodo: Meyrin, Switzerland, 2018. [Google Scholar] [CrossRef]
- Hauptmann, A.; Lucka, F.; Betcke, M.; Huynh, N.; Adler, J.; Cox, B.; Beard, P.; Ourselin, S.; Arridge, S. Model-Based Learning for Accelerated, Limited-View 3-D Photoacoustic Tomography. IEEE Trans. Med. Imaging
**2018**, 37, 1382–1393. [Google Scholar] [CrossRef] - Antholzer, S.; Haltmeier, M.; Schwab, J. Deep learning for photoacoustic tomography from sparse data. Inverse Probl. Sci. Eng.
**2018**. [Google Scholar] [CrossRef]

**Figure 1.**Visualization of the proposed method for confidence estimation using an observing neural network as an error model. The estimator generates an output for a given input and the error model is used to obtain an estimate of the quantification error from the same input data. The region of interest (ROI), which is based on the aleatoric uncertainty ${I}^{\mathrm{aleatoric}}$ extracted from the input data, can then be refined using the error estimates ${I}^{\mathrm{epistemic}}$ of the error model as a confidence threshold ($CT$).

**Figure 2.**Visualization of the deep learning model used in the experiments: a standard U-Net structure with slight modifications to the skip connections. The (x, y, c) numbers shown represent the x and y dimensions of the layers, as well as the number of channels c. Specifically, in this figure, they show the values for a 128 × 128 input and 128 × 128 output. The center consists of an additional convolution and a skip convolution layer to enable different input and output sizes.

**Figure 3.**Visualization of the two methods for absorption quantification with subsequent confidence estimation. (

**a**) A quantification approach based on fluence estimation. In our implementation, the denoised initial pressure $\widehat{{p}_{0}}$ and the fluence $\widehat{\varphi}$ distributions are estimated using deep learning models. These are used to calculate the underlying absorption ${\widehat{\mu}}_{a}$. An error model then estimates the quantification error and, in combination with an aleatoric uncertainty metric, a region of interest is defined. (

**b**) An approach in which one model is used to directly estimate ${\widehat{\mu}}_{a}$ from the input data. ${I}^{\mathrm{aleatoric}}$ is calculated on the basis of the input data and ${I}^{\mathrm{epistemic}}$ is estimated with a second model.

**Figure 4.**Depiction of the illumination geometry and the transducer design which is based on our Fraunhofer DiPhAS photoacoustic imaging (PAI) system [54]. (

**a**) The ultrasound transducer design with the position of the laser output and the transducer elements, as well as the imaging plane; (

**b**) one-half of the symmetric transducer design, where the laser outputs are in parallel left and right to the transducer elements over a length of 2.45 cm.

**Figure 5.**Quantification error as a function of the confidence threshold ($CT$) for five different quantification methods. The line shows the median relative absorption estimation error when only evaluating with the most confident estimates regarding $CT$, and the transparent background shows the corresponding interquartile range. Naïve: Fluence correction with a homogeneous fluence estimate; Fluence raw/${p}_{0}$: Deep learning-based quantification of the fluence applied to ${p}_{0}$ and raw time-series input data and subsequent estimation of ${\mu}_{a}$; Direct raw/${p}_{0}$: End-to-end deep learning-based quantification of ${\mu}_{a}$ applied to ${p}_{0}$ and raw time-series input data.

**Figure 6.**Visualization of the changes in the distribution of the absorption quantification error ${e}_{{\mu}_{a}}$ when applying different confidence thresholds $CT=\{100\%,\phantom{\rule{4pt}{0ex}}50\%,\phantom{\rule{4pt}{0ex}}10\%\}$. The plot shows results for all five conducted experiments. The white line denotes the median, and the black box corresponds to the interquartile range. Outliers of ${e}_{{\mu}_{a}}$ with a value greater than 100% have been omitted from this plot.

**Figure 7.**Sample images of the end-to-end direct ${\mu}_{a}$ quantification method applied to ${p}_{0}$ data, showing the best, the worst, and the median performance of our method when considering only the 50% most confident quantification estimations. All images show the (

**a**) ground truth absorption coefficients (

**b**) reconstructed absorption, (

**c**) error estimate from external model, (

**d**) the actual quantification error.

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Gröhl, J.; Kirchner, T.; Adler, T.; Maier-Hein, L.
Confidence Estimation for Machine Learning-Based Quantitative Photoacoustics. *J. Imaging* **2018**, *4*, 147.
https://doi.org/10.3390/jimaging4120147

**AMA Style**

Gröhl J, Kirchner T, Adler T, Maier-Hein L.
Confidence Estimation for Machine Learning-Based Quantitative Photoacoustics. *Journal of Imaging*. 2018; 4(12):147.
https://doi.org/10.3390/jimaging4120147

**Chicago/Turabian Style**

Gröhl, Janek, Thomas Kirchner, Tim Adler, and Lena Maier-Hein.
2018. "Confidence Estimation for Machine Learning-Based Quantitative Photoacoustics" *Journal of Imaging* 4, no. 12: 147.
https://doi.org/10.3390/jimaging4120147