# Benchmarking MRI Reconstruction Neural Networks on Large Public Datasets

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

- They are usually iterative involving the computation of transforms on large data, and therefore, take a lot of time (2 min for a $512\times 512$ − 500 µm in plane resolution slice [7], on a machine with 8 cores).
- The regularization is usually not perfectly suited to MRI data (it is indeed very difficult to come up with a prior that perfectly reflects MR images).

- Because they are implemented efficiently on GPU and do not use an iterative algorithm, the deep learning algorithms run extremely fast.
- If they have enough capacity, they can learn a better prior of the MR images from the training set.

- Benchmark different neural networks for MRI reconstruction on two datasets: the fastMRI dataset, containing raw complex-valued knee data, and the OASIS dataset [13] containing DICOM real-valued brain data.
- Provide reproducible code and the networks’ weights (https://github.com/zaccharieramzi/fastmri-reproducible-benchmark), using Keras [14] with a TensorFlow backend [15].

## 2. Related Works

- The IXI database (http://brain-development.org/ixi-dataset/) (brains),
- The Data Science Bowl challenge (https://www.kaggle.com/c/second-annual-data-science-bowl/data) (chests).

- The brain real-valued data set provided by the Alzheimer’s Disease Neuroimaging Initiative (ADNI) [20],
- Two proprietary datasets with raw complex-valued data of brain data.

## 3. Models

#### 3.1. Idealized Inverse Problem

**y**is the acquired Fourier coefficients, also called the k-space data, $\mathsf{\Omega}$ is the sub-sampling pattern or mask, ${\mathit{F}}_{\mathsf{\Omega}}$ is the non-uniform Fourier transform (or masked Fourier transform in the case of Cartesian under-sampling), and

**x**is the real anatomical image. Here, we will only deal with Cartesian under-sampling, and there we have ${\mathit{F}}_{\mathsf{\Omega}}={M}_{\mathsf{\Omega}}\mathit{F}$, where ${M}_{\mathsf{\Omega}}$ is a mask, and

**F**is the classical Fourier transform. This model is also valid for 3D (volumewise) imaging, but in the following, we only consider 2D (slicewise) imaging.

#### 3.2. Classic Models

- The dictionary learning step, where both the dictionary D and the sparse codes ${\alpha}_{ij}$ are updated alternatively.
- The reconstruction step, where x is updated. Since this subproblem is quadratic, it admits an analytical solution, which amounts to averaging patches and then performing a data consistency in which the sampled frequencies are replaced in the patch-average result.

#### 3.3. Neural Networks

#### 3.3.1. Single-Domain Networks

#### 3.3.2. Cross-Domains Networks

**Cascade-net**[11] is based on the dictionary learning optimization Problem (4). The idea is to replace the dictionary learning step by convolutional neural networks and still keep the data consistency step in the k-space. The optimization algorithm is then unrolled to allow us to perform back-propagation. The authors of [11] show that we can perform back-propagation through the data consistency step (which is linear) and derive the corresponding Jacobian. The parameters used here for the implementation are the same as those in the original paper, except the number of filters, which was decreased from 64 to 48 to fit on a single GPU. This network is illustrated in Figure 3.

**KIKI-net**[10] is an extension of the Cascade-net where they additionally perform convolutions after the data consistency step in the k-space. The parameters used here for the implementation are the same as those in the original paper. This network is illustrated in Figure 4.

**Primal-Dual-net (PD-net)**was introduced by [9] and applied to MRI by [32], is based on the wavelet-based denoising (3), and in particular, the resolution of the corresponding optimization problem with the PDHG [23] algorithm. Here, the algorithm is unrolled, and the proximity operators (present in the general case of PDHG) are replaced by convolutional neural networks. For our implementation, for a fairer comparison with Cascade-net and the U-net, we used a ReLU non-linearity instead of a PReLU [33]. This network is illustrated in Figure 5.

#### 3.4. Training

## 4. Data

#### 4.1. Under-Sampling

#### 4.2. FastMRI

#### 4.3. OASIS

## 5. Results

#### 5.1. Metrics

- The Peak Signal-to-Noise Ratio (PSNR);
- The Structural SIMilarity index (SSIM) [45];
- The number of trainable parameters in the network;
- The runtime in seconds of the neural network on a single volume.

#### 5.2. Quantitative Results

#### 5.3. Qualitative Results

## 6. Discussion

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## Abbreviations

MRI | Magnetic Resonance Imaging |

CS-MRI | Compressed Sensing MRI |

GPU | Graphical Processing Unit |

ReLU | Rectified Linear Unit |

PReLU | Parametrized ReLU |

PSNR | Peak Signal-to-Noise Ratio |

SSIM | Structural SIMilarity index |

## References

- Ramzi, Z.; Ciuciu, P.; Starck, J.L. Benchmarking Deep Nets MRI Reconstruction Models on the FastMRI Publicly Available Dataset. In Proceedings of the ISBI 2020—International Symposium on Biomedical Imaging, Iowa City, IA, USA, 3–7 April 2020. [Google Scholar]
- Smith-Bindman, R.; Miglioretti, D.L.; Larson, E.B. Rising use of diagnostic medical imaging in a large integrated health system. Health Aff.
**2008**, 27, 1491–1502. [Google Scholar] [CrossRef] [PubMed][Green Version] - Roemer, P.B.; Edelstein, W.A.; Hayes, C.E.; Souza, S.P.; Mueller, O.M. The NMR phased array. Magn. Reson. Med.
**1990**, 16, 192–225. [Google Scholar] [CrossRef] [PubMed] - Lustig, M.; Donoho, D.; Pauly, J.M. Sparse MRI: The Application of Compressed Sensing for Rapid MR Imaging Michael. Magn. Reson. Med.
**2007**. [Google Scholar] [CrossRef] [PubMed] - NHS. NHS Eebsite. Available online: https://www.nhs.uk/conditions/mri-scan/what-happens/ (accessed on 4 March 2020).
- AIM Specialty Health. Clinical Appropriateness Guidelines: Advanced Imaging. 2017. Available online: https://www.aimspecialtyhealth.com/PDF/Guidelines/2017/Sept05/AIM_Guidelines.pdf (accessed on 4 March 2020).
- Ramzi, Z.; Ciuciu, P.; Starck, J.L. Benchmarking proximal methods acceleration enhancements for CS-acquired MR image analysis reconstruction. In Proceedings of the SPARS 2019—Signal Processing with Adaptive Sparse Structured Representations Workshop, Toulouse, France, 1–4 July 2019. [Google Scholar]
- Zhu, B.; Liu, J.Z.; Cauley, S.F.; Rosen, B.R.; Rosen, M.S. Image reconstruction by domain-transform manifold learning. Nature
**2018**, 555, 487–492. [Google Scholar] [CrossRef] [PubMed][Green Version] - Adler, J.; Öktem, O. Learned Primal-Dual Reconstruction. IEEE Trans. Med. Imaging
**2018**, 37, 1322–1332. [Google Scholar] [CrossRef] [PubMed][Green Version] - Eo, T.; Jun, Y.; Kim, T.; Jang, J.; Lee, H.J.; Hwang, D. KIKI-net: Cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images. Magn. Reson. Med.
**2018**, 80, 2188–2201. [Google Scholar] [CrossRef] - Schlemper, J.; Caballero, J.; Hajnal, J.V.; Price, A.; Rueckert, D. A Deep Cascade of Convolutional Neural Networks for MR Image Reconstruction. IEEE Trans. Med. Imaging
**2018**, 37, 491–503. [Google Scholar] [CrossRef][Green Version] - Zbontar, J.; Knoll, F.; Sriram, A.; Muckley, M.J.; Bruno, M.; Defazio, A.; Parente, M.; Geras, K.J.; Katsnelson, J.; Chandarana, H.; et al. fastMRI: An Open Dataset and Benchmarks for Accelerated MRI. arXiv
**2018**, arXiv:1811.08839. [Google Scholar] - LaMontagne, P.J.; Keefe, S.; Lauren, W.; Xiong, C.; Grant, E.A.; Moulder, K.L.; Morris, J.C.; Benzinger, T.L.; Marcus, D.S. OASIS-3: Longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and Alzheimer’s disease. Alzheimer’s Dementia J. Alzheimer’s Assoc.
**2018**, 14, P1097. [Google Scholar] [CrossRef] - Chollet, F. Keras. 2015. Available online: https://keras.io (accessed on 4 March 2020).
- Abadi, M.; Agarwal, A.; Barham, P.; Brevdo, E.; Chen, Z.; Citro, C.; Corrado, G.S.; Davis, A.; Dean, J.; Devin, M.; et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. 2015. Available online: tensorflow.org (accessed on 4 March 2020).
- Virtue, P.; Stella, X.Y.; Lustig, M. Better than real: Complex-valued neural nets for MRI fingerprinting. In Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China, 17–20 September 2017; pp. 3953–3957. [Google Scholar]
- Aggarwal, H.K.; Mani, M.P.; Jacob, M. Multi-Shot Sensitivity-Encoded Diffusion MRI Using Model-Based Deep Learning (Modl-Mussels). In Proceedings of the 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), Venice, Italy, 8–11 April 2019; pp. 1541–1544. [Google Scholar]
- Minh Quan, T.; Nguyen-Duc, T.; Jeong, W.K. Compressed Sensing MRI Reconstruction using a Generative Adversarial Network with a Cyclic Loss. IEEE Trans. Med. Imaging
**2018**, 37, 1488–1497. [Google Scholar] [CrossRef][Green Version] - Yang, Y.; Sun, J.; Li, H.; Xu, Z.; Sun, J.; Xu, Z. Deep ADMM-net for compressive sensing MRI. In Proceedings of the Neural Information Processing Systems, Barcelona, Spain, 5–10 December 2016; pp. 10–18. [Google Scholar] [CrossRef]
- Petersen, R.C.; Aisen, P.; Beckett, L.A.; Donohue, M.; Gamst, A.; Harvey, D.J.; Jack, C.; Jagust, W.; Shaw, L.; Toga, A.; et al. Alzheimer’s disease neuroimaging initiative (ADNI): Clinical characterization. Neurology
**2010**, 74, 201–209. [Google Scholar] [CrossRef][Green Version] - Haldar, J.P. Low-Rank Modeling of Local k-Space Neighborhoods (LORAKS) for Constrained MRI. IEEE Trans. Med. Imaging
**2014**, 33, 668–681. [Google Scholar] [CrossRef] [PubMed][Green Version] - Condat, L. A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms. J. Optim. Theory Appl.
**2013**, 158, 460–479. [Google Scholar] [CrossRef][Green Version] - Chambolle, A.; Pock, T. A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging. J. Math. Imaging Vis.
**2011**, 120–145. [Google Scholar] [CrossRef][Green Version] - Beck, A.; Teboulle, M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci.
**2009**, 2, 183–202. [Google Scholar] [CrossRef][Green Version] - Kim, D.; Fessler, J.A. Adaptive restart of the optimized gradient method for convex optimization. J. Optim. Theory Appl.
**2018**, 178, 240–263. [Google Scholar] [CrossRef][Green Version] - Ravishankar, S.; Bresler, Y. Magnetic Resonance Image Reconstruction from Highly Undersampled K-Space Data Using Dictionary Learning. IEEE Trans. Med. Imaging
**2011**, 30, 1028–1041. [Google Scholar] [CrossRef] - Caballero, J.; Price, A.N.; Rueckert, D.; Hajnal, J.V. Dictionary learning and time sparsity for dynamic MR data reconstruction. IEEE Trans. Med. Imaging
**2014**, 33, 979–994. [Google Scholar] [CrossRef][Green Version] - Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv
**2015**, arXiv:1505.04597v1. [Google Scholar] - Han, Y.; Sunwoo, L.; Chul Ye, J. k-Space Deep Learning for Accelerated MRI. arXiv
**2019**, arXiv:1805.03779v2. [Google Scholar] [CrossRef][Green Version] - Min Hyun, C.; Pyung Kim, H.; Min Lee, S.; Lee, S.; Keun Seo, J. Deep learning for undersampled MRI reconstruction. arXiv
**2019**, arXiv:1709.02576v3. [Google Scholar] - Gregor, K.; Lecun, Y. Learning Fast Approximations of Sparse Coding. In Proceedings of the 27thInternational Confer-ence on Machine Learning, Haifa, Israel, 13–15 June 2010. [Google Scholar]
- Cheng, J.; Wang, H.; Ying, L.; Liang, D. Model Learning: Primal Dual Networks for Fast MR imaging. arXiv
**2019**, arXiv:1908.02426. [Google Scholar] - He, K.; Zhang, X.; Ren, S.; Sun, J. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. arXiv
**2015**, arXiv:1502.01852. [Google Scholar] - Kingma, D.P.; Ba, J. Adam: A Method for Stochastic Optimization. arXiv
**2014**, arXiv:1412.6980. [Google Scholar] - Pascanu, R.; Mikolov, T.; Bengio, Y. On the difficulty of training Recurrent Neural Networks. arXiv
**2012**, arXiv:1211.5063. [Google Scholar] - Chauffert, N.; Ciuciu, P.; Kahn, J.; Weiss, P. Variable density sampling with continuous trajectories. SIAM J. Imaging Sci.
**2014**, 7, 1962–1992. [Google Scholar] [CrossRef][Green Version] - Irarrazabal, P.; Nishimura, D.G. Fast Three Dimensional Magnetic Resonance Imaging. Magn. Reson. Med.
**1995**, 33, 656–662. [Google Scholar] [CrossRef] - Meyer, C.H.; Hu, B.S.; Nishimura, D.G.; Macovski, A. Fast Spiral Coronary Artery Imaging. Magn. Reson. Med.
**1992**, 28, 202–213. [Google Scholar] [CrossRef] - Lazarus, C.; Weiss, P.; Chauffert, N.; Mauconduit, F.; El Gueddari, L.; Destrieux, C.; Zemmoura, I.; Vignaud, A.; Ciuciu, P. SPARKLING: Variable-density k-space filling curves for accelerated T 2 * -weighted MRI. Magn. Reson. Med.
**2018**, 1–19. [Google Scholar] [CrossRef][Green Version] - Sanchez, T.; Gözcü, B.; van Heeswijk, R.B.; Eftekhari, A.; Ilıcak, E.; Çukur, T.; Cevher, V. Scalable learning-based sampling optimization for compressive dynamic MRI. arXiv
**2019**, arXiv:1902.00386. [Google Scholar] - Sherry, F.; Benning, M.; Reyes, J.C.D.l.; Graves, M.J.; Maierhofer, G.; Williams, G.; Schönlieb, C.B.; Ehrhardt, M.J. Learning the sampling pattern for MRI. arXiv
**2019**, arXiv:1906.08754. [Google Scholar] - Aggarwal, H.K.; Jacob, M. J-MoDL: Joint Model-Based Deep Learning for Optimized Sampling and Reconstruction. arXiv
**2019**, arXiv:1911.02945. [Google Scholar] - Wu, Y.; Rosca, M.; Lillicrap, T. Deep compressed sensing. arXiv
**2019**, arXiv:1905.06723. [Google Scholar] - Weiss, T.; Senouf, O.; Vedula, S.; Michailovich, O.; Zibulevsky, M.; Bronstein, A. PILOT: Physics-Informed Learned Optimal Trajectories for Accelerated MRI. arXiv
**2019**, arXiv:1909.05773. [Google Scholar] - Wang, Z.; Bovik, A.C.; Rahim Sheikh, H.; Simoncelli, E.P. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Trans. Image Process.
**2004**, 13. [Google Scholar] [CrossRef][Green Version] - Horvath, K.A.; Li, M.; Mazilu, D.; Guttman, M.A.; McVeigh, E.R. Real-time magnetic resonance imaging guidance for cardiovascular procedures. In Seminars in Thoracic and Cardiovascular Surgery; Elsevier: New York, NY, USA, 2007; Volume 19, pp. 330–335. [Google Scholar]
- Schlemper, J.; Sadegh, S.; Salehi, M.; Kundu, P.; Lazarus, C.; Dyvorne, H.; Rueckert, D.; Sofka, M. Nonuniform Variational Network: Deep Learning for Accelerated Nonuniform MR Image Reconstruction. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Shenzhen, China, 13–17 October 2019. [Google Scholar]
- El Gueddari, L.; Ciuciu, P.; Chouzenoux, E.; Vignaud, A.; Pesquet, J.C. Calibrationless OSCAR-based image reconstruction in compressed sensing parallel MRI. In Proceedings of the 16th IEEE International Symposium on Biomedical Imaging, Venice, Italy, 8–11 April 2019; pp. 1532–1536. [Google Scholar]
- Dragotti, P.L.; Dong, H.; Yang, G.; Guo, Y.; Firmin, D.; Slabaugh, G.; Yu, S.; Keegan, J.; Ye, X.; Liu, F.; et al. DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction. IEEE Trans. Med. Imaging
**2017**, 37, 1310–1321. [Google Scholar] [CrossRef][Green Version]

**Figure 1.**Illustration of the U-net from [28]. In our case, the output is not a segmentation map but a reconstructed image of the same size (we perform zero-padding to prevent decreasing sizes in convolutions).

**Figure 2.**The common backbone between the Cascade net, the KIKI-net, and the PD-net. US mask stands for under-sampling mask. DC stands for data consistency. (I)FFT stands for (Inverse) Fast Fourier Transform. ${N}_{k,d}$ is the number of convolution layers applied in the k-space. ${N}_{i,d}$ is the number of convolution layers applied in the image space. ${N}_{C}$ is the total number of alternations between the k-space and the image-space. It is to be noted that in the case of PD-net, the data consistency step is not performed, the Fourier operators are performed with the original under-sampling mask, and a buffer is concatenated along with the current iteration to allow for some memory between iterations and to learn the acceleration (in the k-space net—dual net—it is also concatenated with the original k-space input). In the case of the Cascade net, ${N}_{k,d}=0$, only the data consistency is performed in the k-space. In the case of the KIKI-net, there is no residual connection in the k-space. However, the k-space nets and image space nets could potentially be any kind of image-to-image neural network.

**Figure 3.**Illustration of the Cascade-net from [11]. Here, each ${C}_{i}$ is a convolutional block of 64 filters (48 in our implementation) followed by a ReLU non-linearity, ${n}_{d}$ is the number of such convolutional blocks forming a convolutional subnetwork between each data consistency layer $DC$, and ${n}_{c}$ is the number of convolutional subnetworks.

**Figure 4.**Illustration of the KIKI-net from [10]. The KCNN and ICNN are convolutional neural networks composed of a number of convolutional blocks varying between 5 and 25 (we implemented 25 blocks for both KCNN and ICNN), each followed by a ReLU non-linearity and featuring between 8 and 64 filters (we implemented 32 filters). For both the varying numbers, the supplementary material of [10] shows that the higher the better. The ICNN also features a residual connection.

**Figure 5.**Illustration of the PD-net from [9]. Here, $\mathcal{T}$ denotes the measurement operator, which in our case is the under-sampled Fourier transform, ${\mathcal{T}}^{*}$ its adjoint, g is the measurements, which in our case are the undersampled k-space measurements, and ${f}_{0}$ and ${h}_{0}$ are the initial guesses for the direct and measurement spaces (the image and k-space in our case). The initial guesses are zero tensors. Because we transform complex-valued data into 2-channel real-valued data, the number of channels at the input and the output of the convolutional subnetworks are multiplied by 2 in our implementation.

**Figure 6.**Reconstruction results for a specific slice (16th slice of

`file1000196`, part of the validation set). The first row represents the reconstruction using the different methods, while the second represents the absolute error when compared to the reference.

**Figure 7.**Reconstruction results for a specific slice (15th slice of

`sub-OAS30367_ses-d3396_T1w.nii.gz`, part of the validation set). The top row represents the reconstruction using the different methods, while the bottom row represents the absolute error when compared to the reference.

**Table 1.**Quantitative results for the

**fastMRI**dataset. Peak Signal-to-Noise Ratio (PSNR) and Structural SIMilarity index (SSIM) mean and standard deviations are computed over the 200 validation volumes. Runtimes are given for the reconstruction of a volume with 35 slices.

Network | PSNR-mean (std) (dB) | SSIM-mean (std) | #params | Runtime (s) |
---|---|---|---|---|

Zero-filled | 29.61 ( 5.28) | 0.657 ( 0.23) | 0 | 0.68 |

KIKI-net | 31.38 (3.02) | 0.712 (0.13) | 1.25M | 8.22 |

U-net | 31.78 ( 6.53) | 0.720 ( 0.25) | 482k | 0.61 |

Cascade net | 31.97 ( 6.95) | 0.719 ( 0.27) | 425k | 3.58 |

PD-net | 32.15 ( 6.90) | 0.729 ( 0.26) | 318k | 5.55 |

**Table 2.**Quantitative results for the

**fastMRI**dataset with the

**Proton density fat suppression (PDFS)**contrast. PSNR and SSIM mean and standard deviations are computed over the 99 validation volumes. Runtimes are given for the reconstruction of a volume with 35 slices.

Network | PSNR-mean (std) (dB) | SSIM-mean (std) | # params | Runtime (s) |
---|---|---|---|---|

Zero-filled | 28.44 (2.62) | 0.578 (0.095) | 0 | 0.41 |

KIKI-net | 29.57 (2.64) | 0.6271 (0.10) | 1.25M | 8.88 |

Cascade-net | 29.88 (2.82) | 0.6251 (0.11) | 425K | 3.57 |

U-net | 29.89 (2.74) | 0.6334 (0.10) | 482K | 1.34 |

PD-net | 30.06 (2.82) | 0.6394 (0.10) | 318K | 5.38 |

**Table 3.**Quantitative results for the

**fastMRI**dataset with the

**Proton density (PD)**contrast. PSNR and SSIM mean and standard deviations are computed over the 100 validation volumes. Runtimes are given for the reconstruction of a volume with 40 slices.

Network | PSNR-mean (std) (dB) | SSIM-mean (std) | # params | Runtime (s) |
---|---|---|---|---|

Zero-filled | 30.63 (2.1) | 0.727 (0.087) | 0 | 0.52 |

KIKI-net | 32.86 (2.4) | 0.797 (0.082) | 1.25M | 11.83 |

U-net | 33.64 (2.6) | 0.807 (0.084) | 482K | 1.07 |

Cascade-net | 33.98 (2.7) | 0.811 (0.086) | 425K | 4.22 |

PD-net | 34.2 (2.7) | 0.818 (0.084) | 318280 | 6.08 |

**Table 4.**Quantitative results for the

**OASIS**dataset. PSNR and SSIM mean and standard deviations are computed over the 200 validation volumes. Runtimes are given for the reconstruction of a volume with 32 slices.

Network | PSNR-mean (std) (dB) | SSIM-mean (std) | # params | Runtime (s) |
---|---|---|---|---|

Zero-filled | 26.11 (1.45) | 0.672 (0.0307) | 0 | 0.165 |

U-net | 29.8 (1.39) | 0.847 (0.0398) | 482k | 1.202 |

KIKI-net | 30.08 (1.43) | 0.853 (0.0336) | 1.25M | 3.567 |

Cascade-net | 32.0 (1.731) | 0.887 (0.0327) | 425k | 2.234 |

PD-net | 33.22 (1.912) | 0.910 (0.0358) | 318k | 2.758 |

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Ramzi, Z.; Ciuciu, P.; Starck, J.-L.
Benchmarking MRI Reconstruction Neural Networks on Large Public Datasets. *Appl. Sci.* **2020**, *10*, 1816.
https://doi.org/10.3390/app10051816

**AMA Style**

Ramzi Z, Ciuciu P, Starck J-L.
Benchmarking MRI Reconstruction Neural Networks on Large Public Datasets. *Applied Sciences*. 2020; 10(5):1816.
https://doi.org/10.3390/app10051816

**Chicago/Turabian Style**

Ramzi, Zaccharie, Philippe Ciuciu, and Jean-Luc Starck.
2020. "Benchmarking MRI Reconstruction Neural Networks on Large Public Datasets" *Applied Sciences* 10, no. 5: 1816.
https://doi.org/10.3390/app10051816