# Explicit Physics-Informed Deep Learning for Computer-Aided Diagnostic Tasks in Medical Imaging

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Physics-Based Loss Terms

## 3. Unrolled Networks

## 4. Generative Models

## 5. Plug-and-Play Methods

## 6. Proposed Approach: Explicit Physics-Informed Learning

#### 6.1. Explicit Physics Priors and EPI-Learning

#### 6.2. EPI-Learning for Nonlinear Regression Tasks—General Problem Formulation

#### 6.3. EPI-Learning for a Model of the Form of a Sum of Exponentials—Case Example

#### 6.4. Proposed Simulated Experiments for the Sum-of-Exponentials Model

## 7. Summary and Future Challenges

## Funding

## Institutional Review Board Statement

## Conflicts of Interest

## References

- Egger, J.; Gsaxner, C.; Pepe, A.; Pomykala, K.L.; Jonske, F.; Kurz, M.; Li, J.; Kleesiek, J. Medical deep learning—A systematic meta-review. Comput. Methods Programs Biomed.
**2022**, 221, 106874. [Google Scholar] [CrossRef] - Chernyakova, T.; Eldar, Y.C. Exploiting FRI signal structure for sub-Nyquist sampling and processing in medical ultrasound. In Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, Australia, 19–24 April 2015; IEEE: New York, NY, USA, 2015; pp. 5947–5951. [Google Scholar]
- Mamistvalov, A.; Amar, A.; Kessler, N.; Eldar, Y.C. Deep-learning based adaptive ultrasound imaging from sub-Nyquist channel data. IEEE Trans. Ultrason. Ferroelectr. Freq. Control
**2022**, 69, 1638–1648. [Google Scholar] [CrossRef] - Cooper, M.A.; Nguyen, T.D.; Xu, B.; Prince, M.R.; Elad, M.; Wang, Y.; Spincemaille, P. Patch based reconstruction of undersampled data (PROUD) for high signal-to-noise ratio and high frame rate contrast enhanced liver imaging. Magn. Reson. Med.
**2015**, 74, 1587–1597. [Google Scholar] [CrossRef] - Chen, Z.; Basarab, A.; Kouamé, D. Joint compressive sampling and deconvolution in ultrasound medical imaging. In Proceedings of the 2015 IEEE International Ultrasonics Symposium (IUS), Taipei, Taiwan, 21–24 October 2015; IEEE: New York, NY, USA; pp. 1–4. [Google Scholar]
- Korngut, N.; Rotman, E.; Afacan, O.; Kurugol, S.; Zaffrani-Reznikov, Y.; Nemirovsky-Rotman, S.; Warfield, S.; Freiman, M. SUPER-IVIM-DC: Intra-voxel incoherent motion based Fetal lung maturity assessment from limited DWI data using supervised learning coupled with data-consistency. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Singapore, 18–22 September 2022; Springer Nature: Cham, Switzerland, 2022; pp. 743–752. [Google Scholar]
- Zhou, S.; Nie, D.; Adeli, E.; Gao, Y.; Wang, L.; Yin, J.; Shen, D. Fine-grained segmentation using hierarchical dilated neural networks. In Proceedings of the Medical Image Computing and Computer Assisted Intervention—MICCAI 2018: 21st International Conference, Granada, Spain, 16–20 September 2018; Springer International Publishing: Cham, Switzerland, 2018; pp. 488–496. [Google Scholar]
- Zhang, L.; Wang, M.; Liu, M.; Zhang, D. A survey on deep learning for neuroimaging-based brain disorder analysis. Front. Neurosci.
**2020**, 14, 779. [Google Scholar] [CrossRef] [PubMed] - Wang, M.; Lian, C.; Yao, D.; Zhang, D.; Liu, M.; Shen, D. Spatial-temporal dependency modeling and network hub detection for functional MRI analysis via convolutional-recurrent network. IEEE Trans. Biomed. Eng.
**2019**, 67, 2241–2252. [Google Scholar] [CrossRef] [PubMed] - Jie, B.; Liu, M.; Lian, C.; Shi, F.; Shen, D. Designing weighted correlation kernels in convolutional neural networks for functional connectivity based brain disease diagnosis. Med. Image Anal.
**2020**, 63, 101709. [Google Scholar] [CrossRef] - Aggarwal, R.; Sounderajah, V.; Martin, G.; Ting, D.S.; Karthikesalingam, A.; King, D.; Ashrafian, H.; Darzi, A. Diagnostic accuracy of deep learning in medical imaging: A systematic review and meta-analysis. NPJ Digit. Med.
**2021**, 4, 65. [Google Scholar] [CrossRef] [PubMed] - Finlayson, S.G.; Chung, H.W.; Kohane, I.S.; Beam, A.L. Adversarial attacks against medical deep learning systems. arXiv
**2018**, arXiv:1804.05296. [Google Scholar] - Topol, E.J. High-performance medicine: The convergence of human and artificial intelligence. Nat. Med.
**2019**, 25, 44–56. [Google Scholar] [CrossRef] - Benjamens, S.; Dhunnoo, P.; Meskó, B. The state of artificial intelligence-based FDA-approved medical devices and algorithms: An online database. npj Digit. Med.
**2020**, 3, 118. [Google Scholar] [CrossRef] - Xu, M.; Zhang, T.; Li, Z.; Liu, M.; Zhang, D. Towards evaluating the robustness of deep diagnostic models by adversarial attack. Med. Image Anal.
**2021**, 69, 101977. [Google Scholar] [CrossRef] [PubMed] - Springenberg, M.; Frommholz, A.; Wenzel, M.; Weicken, E.; Ma, J.; Strodthoff, N. From modern CNNs to vision transformers: Assessing the performance, robustness, and classification strategies of deep learning models in histopathology. Med. Image Anal.
**2023**, 87, 102809. [Google Scholar] [CrossRef] [PubMed] - Billot, B.; Greve, D.N.; Puonti, O.; Thielscher, A.; Van Leemput, K.; Fischl, B.; Dalca, A.V.; Iglesias, J.E. SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining. Med. Image Anal.
**2023**, 86, 102789. [Google Scholar] [CrossRef] - Sudarshan, V.P.; Upadhyay, U.; Egan, G.F.; Chen, Z.; Awate, S.P. Towards lower-dose pet using physics-based uncertainty-aware multimodal learning with robustness to out-of-distribution data. Med. Image Anal.
**2021**, 73, 102187. [Google Scholar] [CrossRef] [PubMed] - Monga, V.; Li, Y.; Eldar, Y.C. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Process. Mag.
**2021**, 38, 18–44. [Google Scholar] [CrossRef] - Barbieri, S.; Gurney-Champion, O.J.; Klaassen, R.; Thoeny, H.C. Deep learning how to fit an intravoxel incoherent motion model to diffusion-weighted MRI. Magn. Reson. Med.
**2020**, 83, 312–321. [Google Scholar] [CrossRef] [PubMed] - Bortsova, G.; González-Gonzalo, C.; Wetstein, S.C.; Dubost, F.; Katramados, I.; Hogeweg, L.; Liefers, B.; van Ginneken, B.; Pluim, J.P.; Veta, M.; et al. Adversarial attack vulnerability of medical image analysis systems: Unexplored factors. Med. Image Anal.
**2021**, 73, 102141. [Google Scholar] [CrossRef] [PubMed] - Szegedy, C.; Zaremba, W.; Sutskever, I.; Bruna, J.; Erhan, D.; Goodfellow, I.; Fergus, R. Intriguing properties of neural networks. arXiv
**2013**, arXiv:1312.6199. [Google Scholar] - Ortiz-Jiménez, G.; Modas, A.; Moosavi-Dezfooli, S.M.; Frossard, P. Optimism in the face of adversity: Understanding and improving deep learning through adversarial robustness. Proc. IEEE
**2021**, 109, 635–659. [Google Scholar] [CrossRef] - Carlini, N.; Wagner, D. Towards evaluating the robustness of neural networks. In Proceedings of the 2017 IEEE Symposium on Security and Privacy (sp), San Jose, CA, USA, 22–26 May 2017; pp. 39–57. [Google Scholar]
- Xie, X.; Niu, J.; Liu, X.; Chen, Z.; Tang, S.; Yu, S. A survey on incorporating domain knowledge into deep learning for medical image analysis. Med. Image Anal.
**2021**, 69, 101985. [Google Scholar] [CrossRef] - Burwinkel, H.; Matz, H.; Saur, S.; Hauger, C.; Trost, M.; Hirnschall, N.; Findl, O.; Navab, N.; Ahmadi, S.A. Physics-aware learning and domain-specific loss design in ophthalmology. Med. Image Anal.
**2022**, 76, 102314. [Google Scholar] [CrossRef] - Lucas, A.; Iliadis, M.; Molina, R.; Katsaggelos, A.K. Using deep neural networks for inverse problems in imaging: Beyond analytical methods. IEEE Signal Process. Mag.
**2018**, 35, 20–36. [Google Scholar] [CrossRef] - Ba, Y.; Zhao, G.; Kadambi, A. Blending diverse physical priors with neural networks. arXiv
**2019**, arXiv:1910.00201. [Google Scholar] - Hammernik, K.; Küstner, T.; Yaman, B.; Huang, Z.; Rueckert, D.; Knoll, F.; Akçakaya, M. Physics-Driven Deep Learning for Computational Magnetic Resonance Imaging: Combining physics and machine learning for improved medical imaging. IEEE Signal Process. Mag.
**2023**, 40, 98–114. [Google Scholar] [CrossRef] [PubMed] - Kaandorp, M.P.; Barbieri, S.; Klaassen, R.; van Laarhoven, H.W.; Crezee, H.; While, P.T.; Nederveen, A.J.; Gurney-Champion, O.J. Improved unsupervised physics-informed deep learning for intravoxel incoherent motion modeling and evaluation in pancreatic cancer patients. Magn. Reson. Med.
**2021**, 86, 2250–2265. [Google Scholar] [CrossRef] [PubMed] - Buoso, S.; Joyce, T.; Kozerke, S. Personalising left-ventricular biophysical models of the heart using parametric physics-informed neural networks. Med. Image Anal.
**2021**, 71, 102066. [Google Scholar] [CrossRef] [PubMed] - Gao, Z.; Wu, S.; Liu, Z.; Luo, J.; Zhang, H.; Gong, M.; Li, S. Learning the implicit strain reconstruction in ultrasound elastography using privileged information. Med. Image Anal.
**2019**, 58, 101534. [Google Scholar] [CrossRef] [PubMed] - Schlemper, J.; Caballero, J.; Hajnal, J.V.; Price, A.N.; Rueckert, D. A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Trans. Med. Imaging
**2017**, 37, 491–503. [Google Scholar] [CrossRef] [PubMed] - Yang, G.; Yu, S.; Dong, H.; Slabaugh, G.; Dragotti, P.L.; Ye, X.; Liu, F.; Arridge, S.; Keegan, J.; Guo, Y.; et al. DAGAN: Deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction. IEEE Trans. Med. Imaging
**2017**, 37, 1310–1321. [Google Scholar] [CrossRef] [PubMed] - Hyun, C.M.; Kim, H.P.; Lee, S.M.; Lee, S.; Seo, J.K. Deep learning for under-sampled MRI reconstruction. Phys. Med. Biol.
**2018**, 63, 135007. [Google Scholar] [CrossRef] - Gregor, K.; LeCun, Y. Learning fast approximations of sparse coding. In Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel, 21–24 June 2010; pp. 399–406. [Google Scholar]
- Ye, C. Tissue microstructure estimation using a deep network inspired by a dictionary-based framework. Med. Image Anal.
**2017**, 42, 288–299. [Google Scholar] [CrossRef] - Kamiya, K.; Hori, M.; Aoki, S. NODDI in clinical research. J. Neurosci. Methods
**2020**, 346, 108908. [Google Scholar] [CrossRef] [PubMed] - Golkov, V.; Dosovitskiy, A.; Sperl, J.I.; Menzel, M.I.; Czisch, M.; Sämann, P.; Brox, T.; Cremers, D. Q-space deep learning: Twelve-fold shorter and model-free diffusion MRI scans. IEEE Trans. Med. Imaging
**2016**, 35, 1344–1351. [Google Scholar] [CrossRef] [PubMed] - Solomon, O.; Cohen, R.; Zhang, Y.; Yang, Y.; He, Q.; Luo, J.; van Sloun, R.J.; Eldar, Y.C. Deep unfolded robust PCA with application to clutter suppression in ultrasound. IEEE Trans. Med. Imaging
**2019**, 39, 1051–1063. [Google Scholar] [CrossRef] - Adler, J.; Öktem, O. Learned primal-dual reconstruction. IEEE Trans. Med. Imaging
**2018**, 37, 1322–1332. [Google Scholar] [CrossRef] - Yang, J.; Zhang, Y.; Yin, W. A fast alternating direction method for TVL1-L2 signal reconstruction from partial Fourier data. IEEE J. Sel. Top. Signal Process.
**2010**, 4, 288–297. [Google Scholar] [CrossRef] - Sun, J.; Li, H.; Xu, Z. Deep ADMM-Net for compressive sensing MRI. In Proceedings of the 30th Conference on Neural Information Processing Systems, Barcelona, Spain, 5–10 December 2016. [Google Scholar]
- Yang, Y.; Sun, J.; Li, H.; Xu, Z. ADMM-CSNet: A deep learning approach for image compressive sensing. IEEE Trans. Pattern Anal. Mach. Intell.
**2018**, 42, 521–538. [Google Scholar] [CrossRef] - Hammernik, K.; Klatzer, T.; Kobler, E.; Recht, M.P.; Sodickson, D.K.; Pock, T.; Knoll, F. Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson. Med.
**2018**, 79, 3055–3071. [Google Scholar] [CrossRef] - Qin, C.; Schlemper, J.; Caballero, J.; Price, A.N.; Hajnal, J.V.; Rueckert, D. Convolutional recurrent neural networks for dynamic MR image reconstruction. IEEE Trans. Med. Imaging
**2018**, 38, 280–290. [Google Scholar] [CrossRef] [PubMed] - Aggarwal, H.K.; Mani, M.P.; Jacob, M. MoDL: Model-based deep learning architecture for inverse problems. IEEE Trans. Med. Imaging
**2018**, 38, 394–405. [Google Scholar] [CrossRef] - Ulyanov, D.; Vedaldi, A.; Lempitsky, V. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 9446–9454. [Google Scholar]
- Golts, A.; Freedman, D.; Elad, M. Deep energy: Using Energy Functions for Unsupervised Training of DNNs. 2018. Available online: https://github.com/AlonaGolts/Deep_Energy (accessed on 3 December 2023).
- Yazdanpanah, A.P.; Afacan, O.; Warfield, S.K. Non-learning based deep parallel MRI reconstruction (NLDpMRI). In Proceedings of the Medical Imaging 2019: Image Processing, San Diego, CA, USA, 19–21 February 2019; SPIE: Paris, France, 2019; Volume 10949, pp. 21–27. [Google Scholar]
- Mardani, M.; Gong, E.; Cheng, J.Y.; Vasanawala, S.S.; Zaharchuk, G.; Xing, L.; Pauly, J.M. Deep generative adversarial neural networks for compressive sensing MRI. IEEE Trans. Med. Imaging
**2018**, 38, 167–179. [Google Scholar] [CrossRef] - Ahmad, R.; Bouman, C.A.; Buzzard, G.T.; Chan, S.; Liu, S.; Reehorst, E.T.; Schniter, P. Plug-and-play methods for magnetic resonance imaging: Using denoisers for image recovery. IEEE Signal Process. Mag.
**2020**, 37, 105–116. [Google Scholar] [CrossRef] [PubMed] - Mani, M.; Magnotta, V.A.; Jacob, M. qModeL: A plug-and-play model-based reconstruction for highly accelerated multi-shot diffusion MRI using learned priors. Magn. Reson. Med.
**2021**, 86, 835–851. [Google Scholar] [CrossRef] [PubMed] - Liu, J.; Sun, Y.; Eldeniz, C.; Gan, W.; An, H.; Kamilov, U.S. RARE: Image reconstruction using deep priors learned without groundtruth. IEEE J. Sel. Top. Signal Process.
**2020**, 14, 1088–1099. [Google Scholar] [CrossRef] - Nemirovsky-Rotman, S.; Rotman, E.; Afacan, O.; Kurugol, S.; Warfield, S.; Freiman, M. Physically Motivated Deep-Neural Networks of the Intravoxel Incoherent Motion Signal Decay Model for Quantitative Diffusion-Weighted MRI. In Proceedings of the Annual Conference of the ISMRM (International Society of Magnetic Resonance in Medicine, Virtual, 15–20 May 2021. [Google Scholar]
- Sorantin, E.; Grasser, M.G.; Hemmelmayr, A.; Tschauner, S.; Hrzic, F.; Weiss, V.; Lacekova, J.; Holzinger, A. The augmented radiologist: Artificial intelligence in the practice of radiology. Pediatr. Radiol.
**2021**, 52, 1–13. [Google Scholar] [CrossRef]

**Figure 1.**Categories for physics-based deep neural networks: (

**a**) prior-based loss term; (

**b**) unrolled networks; (

**c**) generative models; and (

**d**) plug-and-play. For each category, the part of the architecture that is induced with the prior is marked in blue.

**Figure 2.**Extending a basic network architecture with the explicit prior information coordinates (EPI-Learning): (

**a**) the original network and (

**b**) the proposed EPI-Learning architecture.

**Figure 3.**Extending a basic network architecture with the explicit prior information coordinates (EPI-Learning).

**Figure 4.**Generating training data sets for the basic architecture (

**a**) and its “EPI-Learning” counterpart (

**b**). Green and red arrows correspond to the basic and “EPI-Learning” architectures, respectively.

**Figure 5.**Categories for physics-based deep neural networks (marked with hexagons) and their potential contributions to the network performance (marked with circles).

**Table 1.**Summary of surveyed works categorized into their corresponding physics-based methods, imaging modality, and application domain.

Physics-Based Method | Task/Imaging Modality | Application Domain | Reference | Year |
---|---|---|---|---|

- Physics-based loss term
| Generating simulated images | Cardiac modeling | Buoso et al. [31] | 2018 |

Parmeter estimation for diffusion MRI (dMRI) | Pancreatic imaging | Kaandorp et al. [30] | 2021 | |

Parmeter estimation for optical coherence tomography | Ophthalmology | Burwinkel et al. [26] | 2022 | |

Tissue elasticity map prediction for ultrasound elastography imaging | Liver and breast imaging | Gao et al. [32] | 2019 | |

Magnetic resonance (MR) image reconstruction for undersampled data (compressed sensing) | Brain imaging | Yang et al. [34] | 2017 | |

MRI reconstruction—compressed sensing | Brain imaging | Hyun et al. [35] | 2018 | |

Positron emission tomography (PET) | Brain imaging | Sudarshan et al. [18] | 2021 | |

- 2.
- Unrolled networks
| Parameter estimation for dMRI | Brain imaging | Ye [37] | 2017 |

MRI reconstruction—compressed sensing | Brain imaging Chest imaging | Sun et al. [43] | 2016 | |

MRI reconstruction—compressed sensing | Brain imaging | Yang et al. [44] | 2018 | |

MRI reconstruction—compressed sensing | Musculoskeletal imaging | Hammernik et al. [45] | 2018 | |

MRI reconstruction—compressed sensing | Cardiac imaging | Qin et al. [46] | 2018 | |

MRI reconstruction—compressed sensing | Brain imaging | Aggarwal et al. [47] | 2018 | |

Clutter suppression in ultrasound imaging | Vascular imaging | Solomon et al. [40] | 2019 | |

Computer tomography (CT) reconstruction | Demonstrated for human phantoms | Adler and Öktem [41] | 2018 | |

- 3.
- Generative models (GANs)
| MRI reconstruction—compressed sensing | Brain imaging Knee imaging | Yazdanpanah et al. [50] | 2019 |

MRI reconstruction—compressed sensing | Pediatric imaging (abdomen and knee scans) | Mardani et al. [51] | 2018 | |

- 4.
- Plug-and-play (PnP) methods
| MRI reconstruction—compressed sensing | Cardiac imaging Knee imaging | Ahmad et al. [52] | 2020 |

Image reconstruction for undersampled dMRI | Brain imaging | Mani et al. [53] | 2021 | |

MRI reconstruction—compressed sensing | Liver imaging | Liu et al. [54] | 2020 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Nemirovsky-Rotman, S.; Bercovich, E.
Explicit Physics-Informed Deep Learning for Computer-Aided Diagnostic Tasks in Medical Imaging. *Mach. Learn. Knowl. Extr.* **2024**, *6*, 385-401.
https://doi.org/10.3390/make6010019

**AMA Style**

Nemirovsky-Rotman S, Bercovich E.
Explicit Physics-Informed Deep Learning for Computer-Aided Diagnostic Tasks in Medical Imaging. *Machine Learning and Knowledge Extraction*. 2024; 6(1):385-401.
https://doi.org/10.3390/make6010019

**Chicago/Turabian Style**

Nemirovsky-Rotman, Shira, and Eyal Bercovich.
2024. "Explicit Physics-Informed Deep Learning for Computer-Aided Diagnostic Tasks in Medical Imaging" *Machine Learning and Knowledge Extraction* 6, no. 1: 385-401.
https://doi.org/10.3390/make6010019