Next Article in Journal
Medical Radiology: Current Progress
Next Article in Special Issue
Effect of Low Tube Voltage (100 kV) Combined with ASIR-V on the Visualization and Image Quality of the Adamkiewicz Artery: A Comparison with 120 kV Protocol
Previous Article in Journal
Pediatric COVID-19 and Diabetes: An Investigation into the Intersection of Two Pandemics
Previous Article in Special Issue
Head-to-Head Comparison between FDG and 11C-Methionine in Multiple Myeloma: A Systematic Review
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Deep Learning Convolutional Neural Network Reconstruction and Radial k-Space Acquisition MR Technique for Enhanced Detection of Retropatellar Cartilage Lesions of the Knee Joint

by
Malwina Kaniewska
1,2,*,
Eva Deininger-Czermak
1,2,3,
Maelene Lohezic
4,
Falko Ensle
1,2 and
Roman Guggenberger
1,2
1
Institute of Diagnostic and Interventional Radiology, University Hospital Zurich (USZ), Raemistrasse 100, 8091 Zurich, Switzerland
2
Institute of Diagnostic and Interventional Radiology, University of Zurich (UZH), Raemistrasse 100, 8091 Zurich, Switzerland
3
Department of Forensic Medicine and Imaging, Institute of Forensic Medicine, University of Zurich, 8152 Zurich, Switzerland
4
Advanced Technology, Science and Technology Organization, GE HealthCare, 8152 Zurich, Switzerland
*
Author to whom correspondence should be addressed.
Diagnostics 2023, 13(14), 2438; https://doi.org/10.3390/diagnostics13142438
Submission received: 16 May 2023 / Revised: 13 July 2023 / Accepted: 15 July 2023 / Published: 21 July 2023
(This article belongs to the Special Issue What's New in Diagnostic Radiological Imaging?)

Abstract

:
Objectives: To assess diagnostic performance of standard radial k-space (PROPELLER) MRI sequences and compare with accelerated acquisitions combined with a deep learning-based convolutional neural network (DL-CNN) reconstruction for evaluation of the knee joint. Methods: Thirty-five patients undergoing MR imaging of the knee at 1.5 T were prospectively included. Two readers evaluated image quality and diagnostic confidence of standard and DL-CNN accelerated PROPELLER MR sequences using a four-point Likert scale. Pathological findings of bone, cartilage, cruciate and collateral ligaments, menisci, and joint space were analyzed. Inter-reader agreement (IRA) for image quality and diagnostic confidence was assessed using intraclass coefficients (ICC). Cohen’s Kappa method was used for evaluation of IRA and consensus between sequences in assessing different structures. In addition, image quality was quantitatively evaluated by signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) measurements. Results: Mean acquisition time of standard vs. DL-CNN sequences was 10 min 3 s vs. 4 min 45 s. DL-CNN sequences showed significantly superior image quality and diagnostic confidence compared to standard MR sequences. There was moderate and good IRA for assessment of image quality in standard and DL-CNN sequences with ICC of 0.524 and 0.830, respectively. Pathological findings of the knee joint could be equally well detected in both sequences (κ-value of 0.8). Retropatellar cartilage could be significantly better assessed on DL-CNN sequences. SNR and CNR was significantly higher for DL-CNN sequences (both p < 0.05). Conclusions: In MR imaging of the knee, DL-CNN sequences showed significantly higher image quality and diagnostic confidence compared to standard PROPELLER sequences, while reducing acquisition time substantially. Both sequences perform comparably in the detection of knee-joint pathologies, while DL-CNN sequences are superior for evaluation of retropatellar cartilage lesions.

1. Introduction

Magnetic resonance imaging (MRI) is the preferred imaging modality for assessment of soft tissues in and around joints and is thus widely used in musculoskeletal imaging [1,2,3,4]. In MR imaging of the knee joint, fast spin echo (FSE) sequences are usually acquired in 2D or 3D as they allow precise visualization of internal and external structures of the knee joint, such as menisci, cruciate and collateral ligaments, and cartilage, but also soft tissues, like muscle and fat around the joint [5,6]. However, motion artifacts, caused by blood flow of the adjacent vessels or patient motion, can hamper image quality and may be an important limiting factor when reporting MRI scans of the knee.
To increase time efficiency of MR scanners while simultaneously reducing motion artifacts, recent efforts have been focusing on decreasing scan times. A significant reduction in scan time in musculoskeletal imaging may be achieved using parallel imaging, simultaneous multislice acquisition, compressed sensing-based sampling, and synthetic MRI techniques [7,8]. While these techniques provide a significant reduction in acquisition time and artifacts, they also usually result in a lower signal-to-noise ratio (SNR) and loss of image quality when compared to conventional acquisition techniques [7]. Alternatively, the PROPELLER method which relies on radial k-space image acquisition can be used for reduction of motion artifacts. [9,10,11]. In the PROPELLER acquisition technique, data are collected using parallel lines in circle around the k-space, which helps minimizing motion artifacts while increasing scan times [9,12]. This technique effectively reduces motion artifacts while increasing image quality, and therefore has been successfully used in imaging of the joints [9].
Application of deep learning-based convolutional neural networks (DL-CNN) reconstruction has been recently described as an effective method to accelerate MRI scan times, while reducing image noise and maintaining optimal image contrast [13,14,15]. The combination of conventional FSE sequences with DL-CNN has already been successfully implemented in imaging of the shoulder and knee joint [14,16].
The combination of a radial k-space acquisition technique with DL-CNN image reconstruction has the potential to simultaneously reduce motion artifacts and image noise in addition to substantial scan time reduction. However, this has not been yet investigated, especially for the imaging of the joints.
Therefore, the aim of this study was to assess the diagnostic performance of standard radial k-space (PROPELLER) MRI sequences and compare them with accelerated acquisitions using the PROPELLER technique combined with a deep learning-based convolutional neural network (DL-CNN) reconstruction for evaluation of the knee.

2. Materials and Methods

This prospective cohort study received ethical board approval, and written informed consent was obtained from every patient included.

2.1. Participants

Between October and December 2021, patients referred for an MRI of the knee joint were prospectively included in the study. Two image sets of standard and accelerated MR PROPELLER sequences were acquired in each patient in coronal, sagittal, and axial plane. In total, 35 patients were included in the study (Flow chart in Figure 1).

2.2. MRI Examination

The MRI examinations were performed using a 1.5 Tesla MRI scanner (SIGNA Artist, Waukesha, WI, USA) with a dedicated 16-channel knee coil.
MR protocols included coronal T1-weighted, axial proton density (PD) fat-saturated (FS) and sagittal PD sequences. Standard and accelerated sequences were acquired using the PROPELLER technique. Accelerated sequences were further reconstructed using DL-CNN reconstruction AIRTM Recon DL® (Waukesha, WI, USA) with a medium level of SNR improvement (chosen between low, medium, and high), thereafter referred to as DL-CNN sequences. Table 1 shows detailed MRI parameters of the standard and accelerated sequences.
The AIRTM Recon DL pipeline incorporates a deep convolutional neural network that operates on raw, complex-valued imaging data to generate a clean output image. Specifically, the CNN is designed to allow users to adjust the level of image noise reduction, minimize truncation artifacts and enhance edge sharpness. Integration into the scanner’s native reconstruction pipeline is crucial to access the complete, high bit-depth data. The CNN consists of approximately 10,000 kernels, totaling 4.4 million trainable parameters. Being a convolutional network, it can accommodate MR images of various sizes. The CNN was trained using a supervised learning approach, using pairs of images representing nearly perfect and conventional MRI images. The near-perfect training data consisted of high-resolution images with minimal ringing and very low noise levels. The conventional training data were synthesized from the near-perfect images, employing established methods to create lower-resolution versions with increased truncation artifacts and higher noise levels [17].
Subsequently, all image datasets were forwarded to the Picture Archiving and Communication System (PACS) of our department (IMPAX 6; Agfa-Gevaert N.V., Mortsel, Belgium) for subsequent analysis.

2.3. Assessment of Image Quality and Diagnostic Confidence

As a part of the initial reading training, a set of diverse knee joint examinations, including both standard sequences and DL-CNN sequences unrelated to the study, were reviewed by two readers. The readers consisted of a board-certified radiologist with 6 years of experience and a radiology resident with 3 years of experience. Readers were blinded to any clinical information and to the sequence identifiers. Discrepancies were thoroughly discussed until agreement was achieved.
Then, the same two readers assessed MRI images of the knee joint of 35 patients included in the study. All image sets underwent a process of removing sequence identifiers (standard sequence vs. DL-CNN sequences) and were subsequently combined in a mixed order. The readers conducted the review of all images in a randomized sequence. Following the completion of the readings, the sequence type information was disclosed for the purpose of statistical analysis.
Image quality of standard and DL-CNN PROPELLER sequences for tibial, femoral and retropatellar bone and cartilage, tibiofibular joint, muscle (vastus medialis and lateralis muscle), anterior and posterior cruciate ligament, medial and lateral collateral ligament, medial and lateral meniscus, and subcutaneous tissue at the level of the medial femoral condyle was assessed using a four-point Likert scale. The scale ranged from 0 (poor) to 4 (perfect), with intermediate ratings of 1 (moderate) and 2 (good).
Additionally, the diagnostic confidence for the aforementioned structures was evaluated, along with the assessment of contour sharpness and homogeneity of fat saturation in the central and peripheral field-of-view (FOV). A four-point Likert scale (0-poor, 1-moderate, 2-good, 3-perfect) was utilized for this evaluation. The central FOV was defined at the level of the knee joint, while the peripheral FOV was identified as the most medial part of the vastus medialis muscle. Motion artifacts were evaluated independently for each image set.

2.4. Assessment of Pathological Findings

Evaluation of bone was assessed as (0) normal bone, or (1) minimal, (2) moderate and (3) advanced productive changes as usually associated with osteoarthritis. The assessment of cartilage involved categorizing it into different levels: (0) normal and homogeneous, (1) superficial inhomogeneities with normal cartilage contour, (2) partial thickness cartilage loss of less than 50% without subchondral edema, and (3) full-thickness cartilage loss with subchondral edema. Soft tissues were assessed as (0) normal, (1) muscle edema, (2) muscle fatty degeneration.
Anterior and posterior cruciate ligament was categorized as (0) normal, (1) mucoid degeneration with slightly hyperintense signal in PD fs sequences, (2) strain or partial-thickness tear with high signal intensity, swelling or thinning of the ligament or wavy course, and (3) full-thickness tear with complete lack of continuity as described by Ng et al. [18]. Medial and lateral collateral ligament were characterized as (0) normal, (1) tendinopathy when the ligament was thickened with slightly hyperintense signal in PD fs sequences, (2) partial thickness tear, and (3) full-thickness tear with complete lack of ligamentous continuity. Medial and lateral meniscus was evaluated as follows: (0) normal, (1) mucoid degeneration, and (2) torn. The knee joint was assessed as (0) normal, or (1) minimal, (2) moderate, or (3) marked joint effusion.
Each structure was assessed in all planes of the acquired MR images and sequences.

2.5. Signal-to-Noise and Contrast-to-Noise Ratio

For quantitative assessment of image quality, the signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) were measured for both sequences. Regions of interest (ROIs) with an area of 5 mm2 were placed individually on each set to determine the signal intensity (SI) in the bone (specifically the distal femur), muscle (vastus medialis muscle), and subcutaneous fat at the level of the medial femoral condyle. The noise was quantified as the standard deviation (SD) of the SI in an ROI measured in extracorporeal air.
The SNR and CNR were calculated using the following formulas:
S N R = S I S D ( a i r )
C N R ( b o n e ) = S I b o n e S I ( m u s c l e ) S D a i r
C N R ( f a t ) = S I f a t S I ( m u s c l e ) S D a i r

2.6. Statistical Analysis

A Shapiro–Wilk test was applied to assess the normal distribution of findings [19,20,21]. A Wilcoxon signed-rank test was used to compare the findings of image quality and diagnostic confidence between standard and DL-CNN sequences [22]. If a significant difference between sequences was noticed, a Bonferroni–Holm post hoc test for multiple comparison was additionally performed [23].
All identified pathologies in the evaluated structures were documented individually for each reader and dichotomized as either not pathological (scores 0 and 1) or pathological (scores 2 and 3).
The agreement between standard and DL-CNN sequences, as well as the inter-reader agreement (IRA) for assessing image quality and diagnostic confidence, were determined using the intraclass coefficient (ICC) [24]. ICC values below 0.5 were considered poor, values between 0.5 and 0.75 were categorized as moderate, values between 0.75 and 0.9 were considered good, and values above 0.9 were regarded as excellent indicators of reliability [25].
Cohen’s Kappa statistic was applied for assessment of the IRA and agreement between standard and DL-CNN sequences in evaluation of the pathological findings of the knee joint. Kappa values between 0.41–0.60 were considered moderate, between 0.61–0.80 substantial, and above 0.81 almost perfect agreement [26,27]. p-value < 0.05 was considered significant. To calculate an appropriate minimum sample size, an a priori power analysis was conducted with following parameters: Cohen’s effect size was determined as 0.8, α = 0.05 and Power (1 − β) = 0.95 [28]. A power analysis was performed using a G* Power software, v. 3.1.9.1 (Heinrich-Heine-Universitaet Duesseldorf, Dusesseldorf, Germany) [29]. Other statistical analyses were conducted using SPSS, v. 26.0 (IBM, Armonk, NY, USA).

3. Results

A minimum sample size of 24 was calculated in the power analysis for comparison of two matched samples. Finally, 35 patients between 18 and 73 years of age (mean: 47 years old SD: 16 years old; male n = 17, female n = 18) were included in the study. The mean acquisition time of standard sequences was 10 min 3 s, and of accelerated sequences 4 min 45 s.

3.1. The Quality of the Images and the Level of Diagnostic Confidence

The average image quality scores for bone assessment were 2.0 for standard sequences and 2.9 for DL-CNN sequences, of cartilage 1.9 and 2.9, of anterior cruciate ligament 1.8 and 2.6, of posterior cruciate ligament 2.2 and 2.9, of medial meniscus 1.9 and 2.9, and of lateral meniscus 1.9 and 2.8, respectively.
The mean overall diagnostic confidence in standard and DL-CNN sequences was 2.0 and 2.9, respectively.
Significant better results were observed in the average image quality and diagnostic confidence of all analyzed structures when utilizing DL-CNN sequences in comparison to the standard sequences (p < 0.05). No motion artifacts were detected in any of the image sets.
Overall, there was a moderate and good IRA for image quality in standard and DL-CNN sequences with ICC of 0.52 and 0.83. The inter-reader agreement for the assessment of diagnostic confidence in both standard and DL-CNN sequences showed moderate agreement, with ICC values of 0.54 and 0.55, respectively. There was a good agreement between standard and DL-CNN sequences for assessment of image quality with ICC of 0.77 and a moderate agreement for diagnostic confidence with ICC of 0.59.
For detailed information, please refer to Table 2 and Table 3. Figure 2 shows examples of both sequences.

3.2. Pathological Findings

Tibial and femoral bones were evaluated as normal (grades 0 and 1) in 16 and 18 patients, and in 19 and 17 patients as pathologic (grades 2 and 3) in standard and DL-CNN sequences, respectively. Tibial and femoral cartilage was evaluated as normal in 24 and 23 patients, and in 11 and 12 patients as pathologic (grades 2 and 3) in standard and DL-CNN sequences. ACL was assessed as normal in 30 patients in both standard and DL-CNN sequences and as pathologic in 5 patients. PCL was considered pathologic in one and two patients in standard and DL-CNN sequences, respectively. Medial meniscus was evaluated as pathologic in 10 and 9 patients in standard and DL-CNN sequences, while lateral meniscus in 5 and 4 patients.
In general, there was a moderate inter-reader agreement when assessing pathological findings in the knee joint using both standard and DL-CNN sequences, as indicated by Kappa values of 0.54 and 0.55, respectively. There was a substantial level of agreement between standard and DL-CNN sequences in the assessment of pathological findings of the knee joint with a κ-value of 0.8.
The summary of results of pathological findings of all analyzed structures in standard and DL-CNN sequences and IRAs is depicted in Table 4. Figure 3, Figure 4 and Figure 5 showcase various examples of pathological findings observed in both standard and DL-CNN sequences.

3.3. Signal-to-Noise and Contrast-to-Noise Ratios

The mean signal-to-noise ratio for bone, fat, and muscle was significantly higher in DL-CNN sequences compared to standard sequences (p < 0.05). Mean CNR was significantly higher in DL-CNN than in standard sequences (p < 0.05). For details refer to Figure 6.

4. Discussion

This study represents the first attempt to integrate PROPELLER MR acquisition technique with DL-CNN image reconstruction for knee joint imaging (Table S1 Supplementary Materials).
Deep-learning based reconstruction algorithms in musculoskeletal imaging can be applied both to minimize image noise and to reduce the scan time. Recht et al. described that DL can be used to reconstruct fourfold accelerated images resulting in higher image quality than standard sequences. Moreover, the DL-accelerated sequences performed interchangeably with standard sequences for the detection of pathology of the knee [14]. Additionally, a super-resolution technique using deep learning and convolutional neural networks have been used for post-processing of the lower-resolution MR images of the knee joint and abdomen, resulting in superior image quality and diagnostic performance compared to the standard sequences [30,31,32,33]. As already shown in different studies, deep learning reconstructions can be also used for computed tomography (i.e., for planning of different orthopedic implants), as they serve as a robust method for dose reduction, while maintaining the image quality [34,35,36,37].
The PROPELLER acquisition technique is used when correction of motion and pulsatile flow artifacts is needed and has been described for imaging of the joints, head, and abdomen [9,10,11,12,38,39]. In 2011, Dietrich et al. reported that the PROPELLER technique in the MRI of the shoulder can be successfully used to reduce motion artifacts, while increasing image quality [9]. Similarly, application of radial k-space acquisition technique for imaging of the knee joint in the study of Lavdas et al. resulted in significant elimination of motion and pulsatile flow artifacts and thus higher image quality when compared to the conventional fast spin-echo (FSE) sequences [11] at the expense of increased scan time [9,12].
We sought to combine advantages of both approaches, i.e., artifact reduction by the motion insensitive PROPELLER technique with image noise and scan time reduction by using DL-CNN image reconstruction. The results confirmed a significant improvement in both image quality and diagnostic confidence when utilizing DL-CNN sequences in comparison to standard sequences, while reducing acquisition time from 10 min 3 s to 4 min 45 s. As expected, there were no motion artifacts in any of the image sets that would have hampered image analysis.
In our study, there was a significantly superior image quality of the DL-CNN sequences compared to the standard PROPELLER for imaging the knee joint, subsequently resulting in higher diagnostic confidence. Moreover, the IRA for image quality was better in DL-CNN than in standard sequences with ICC of 0.830 and 0.524, correspondingly.
A moderate level of IRA was observed for the evaluation of diagnostic confidence in both standard and DL-CNN sequences. We assume that a new imaging technique might have caused a different image impression to the radiologists who were experienced in reading conventional MRI sequences, resulting in different subjective perception of analyzed images.
The substantial agreement between standard and DL-CNN sequences for assessment of pathological findings of the knee joint is consistent with the fact that pathologies could be properly detected in both standard and DL-CNN sequences. The DL-CNN sequences however were significantly superior for the assessment of retropatellar cartilage, especially for evaluation of superficial cartilage defects and chondral delamination. Different methods for imaging and segmentation of knee cartilage have been defined including ultra-short echo time and T1ρ sequences and deep convolutional neural networks; however, the utilization of the PROPELLER technique combined with DL-CNN reconstruction for knee cartilage imaging has not been described thus far [32,40,41,42]. Our study shows that a PROPELLER acquisition technique can be successfully applied for imaging of the knee joint in combination with DL-CNN sequences and enables superior depiction of superficial cartilage defects and cartilage delamination.
Our study has limitations. First, this is a small pilot study including only 35 patients that looks at different structures of the knee joint. Large sample sizes and a more focused study hypothesis may be warranted for future studies comparing standard to DL-CNN MR imaging. Second, all sequences were acquired using the PROPELLER technique and no conventional FSE sequences as part of most standard MR protocols were included. However, this study was specifically designed to address the lack of literature regarding the application of the PROPELLER technique in generating DL-CNN sequences [16,38,43]. Finally, there was no arthroscopic reference standard for the different joint pathologies detected by MRI. However, as described in previous studies, MRI is a reliable non-invasive method for evaluation of the knee joint with high correlation with arthroscopic findings [5,44].
In conclusion, MR imaging of the knee using accelerated PROPELLER sequences combined with DL-CNN reconstructions showed significantly higher image quality and diagnostic confidence compared to standard PROPELLER sequences, while reducing acquisition time substantially from 10 min 3 s to 4 min 45 s. Both sequences perform comparably in the detection of knee-joint pathologies, while DL-CNN images are superior for evaluation of retropatellar cartilage lesions.
These findings suggest that a combination of PROPELLER acquisition and the DL-CNN reconstruction technique may become the new reference standard for motion-artifact free, fast MR imaging of the knee with a superior depiction of superficial cartilage defects and cartilage delamination compared to a standard PROPELLER technique.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/diagnostics13142438/s1, Table S1: Database search table.

Author Contributions

M.K.: Conceptualization, Validation, Investigation, Data Curation, Writing—Original Draft, Writing—Review & Editing, Project administration; E.D.-C.: Methodology, Validation, Formal analysis, Writing—Review & Editing; M.L.: Software, Data Curation, Writing—Review & Editing; F.E.: Investigation, Data Curation, Writing—Review & Editing; R.G.: Conceptualization, Writing—Review & Editing, Supervision. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Approval of the cantonal ethics committee in Zurich (BASEC 2018-01752).

Informed Consent Statement

Not applicable.

Data Availability Statement

Research data are to be found on the local computer of the University Hospital of Zurich and it is its property.

Acknowledgments

We acknowledge support from GE HealthCare for providing the prototype AIR TM Recon DL® reconstruction for PROPELLER and from Slavisa Milanovic (GE HealthCare) for assisting with the optimization of the accelerated PROPELLER protocol.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

ACLanterior cruciate ligament
CNRcontrast-to-noise ratio
DL-CNNdeep learning-based convolutional neural networks
FOVfield of view
FSEfast spin-echo sequences
ICCintraclass correlation coefficient
IRAinter-reader agreement
LCLlateral collateral ligament
MCLmedial collateral ligament
PCLposterior cruciate ligament
PROPELLERperiodically rotated overlapping parallel lines with enhanced reconstruction
ROIregion of interest
SDstandard deviation
SIsignal intensity
SNRsignal-to-noise ratio

References

  1. Ben-Galim, P.; Steinberg, E.L.; Amir, H.; Ash, N.; Dekel, S.; Arbel, R. Accuracy of Magnetic Resonance Imaging of the Knee and Unjustified Surgery. Clin. Orthop. Relat. Res. 2006, 447, 100–104. [Google Scholar] [CrossRef] [PubMed]
  2. Gold, G.E.; Han, E.; Stainsby, J.; Wright, G.; Brittain, J.; Beaulieu, C. Musculoskeletal MRI at 3.0 T: Relaxation Times and Image Contrast. Am. J. Roentgenol. 2004, 183, 343–351. [Google Scholar] [CrossRef]
  3. Nacey, N.C.; Geeslin, M.G.; Miller, G.W.; Pierce, J.L. Magnetic resonance imaging of the knee: An overview and update of conventional and state of the art imaging. J. Magn. Reson. Imaging 2017, 45, 1257–1275. [Google Scholar] [CrossRef] [PubMed]
  4. Shapiro, L.; Harish, M.; Hargreaves, B.; Staroswiecki, E.; Gold, G. Advances in musculoskeletal MRI: Technical considerations. J. Magn. Reson. Imaging 2012, 36, 775–787. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  5. Friemert, B.; Schwarz, W.; Danz, B.; Oberländer, Y.; Häberle, H.J.; Bähren, W.; Gerngroß, H. Diagnosis of chondral lesions of the knee joint: Can MRI replace arthroscopy? Knee Surg. Sports Traumatol. Arthrosc. 2004, 12, 58–64. [Google Scholar] [CrossRef] [PubMed]
  6. Jackson, D.; Jennings, L.; Maywood, R.; Berger, P. Magnetic resonance imaging of the knee. Am. J. Sports Med. 1988, 16, 29–38. [Google Scholar] [CrossRef]
  7. Fritz, J.; Guggenberger, R.; del Grande, F. Rapid Musculoskeletal MRI in 2021: Clinical Application of Advanced Accelerated Techniques. Am. J. Roentgenol. 2021, 216, 718–733. [Google Scholar] [CrossRef]
  8. Del Grande, F.; Rashidi, A.; Luna, R.; Delcogliano, M.; Stern, S.E.; Dalili, D.; Fritz, J. Five-Minute Five-Sequence Knee MRI Using Combined Simultaneous Multislice and Parallel Imaging Acceleration: Comparison with 10-Minute Parallel Imaging Knee MRI. Radiology 2021, 299, 635–646. [Google Scholar] [CrossRef]
  9. Dietrich, T.J.; Ulbrich, E.J.; Zanetti, M.; Fucentese, S.F.; Pfirrmann, C.W.A. PROPELLER Technique to Improve Image Quality of MRI of the Shoulder. Am. J. Roentgenol. 2011, 197, W1093–W1100. [Google Scholar] [CrossRef]
  10. Jia-gao, F.; Yang, F.; Lian-jin, N. Clinical Application of 3.0 T MRI BLADE Technique in Correcting Joint Motion Artifacts. Prog. Mod. Biomed. 2010, 23, 4547–4549. [Google Scholar]
  11. Lavdas, E.; Vlychou, M.; Zaloni, E.; Vassiou, K.; Tsagkalis, A.; Dailiana, Z.H.; Fezoulidis, I.V. Elimination of motion and pulsation artifacts using BLADE sequences in shoulder MR imaging. Skelet. Radiol. 2015, 44, 1619–1626. [Google Scholar] [CrossRef]
  12. Pipe, J.G. Motion correction with PROPELLER MRI: Application to head motion and free-breathing cardiac imaging. Magn. Reson. Med. Off. J. Int. Soc. Magn. Reson. Med. 1999, 42, 963–969. [Google Scholar] [CrossRef]
  13. Lebel, R.M. Performance characterization of a novel deep learning-based MR image reconstruction pipeline. arXiv 2020, arXiv:200806559. [Google Scholar]
  14. Recht, M.P.; Zbontar, J.; Sodickson, D.K.; Knoll, F.; Yakubova, N.; Sriram, A.; Murrell, T.; Defazio, A.; Rabbat, M.; Rybak, L.; et al. Using Deep Learning to Accelerate Knee MRI at 3 T: Results of an Interchangeability Study. Am. J. Roentgenol. 2020, 215, 1421–1429. [Google Scholar] [CrossRef] [PubMed]
  15. Yaman, B.; Hosseini, S.A.H.; Moeller, S.; Ellermann, J.; Uğurbil, K.; Akçakaya, M. Self-Supervised Physics-Based Deep Learning MRI Reconstruction Without Fully-Sampled Data. In Proceedings of the 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), Iowa City, IA, USA, 3–7 April 2020. [Google Scholar]
  16. Hahn, S.; Yi, J.; Lee, H.-J.; Lee, Y.; Lim, Y.-J.; Bang, J.-Y.; Kim, H.; Lee, J. Image Quality and Diagnostic Performance of Accelerated Shoulder MRI with Deep Learning–Based Reconstruction. Am. J. Roentgenol. 2021, 218, 506–516. [Google Scholar] [CrossRef] [PubMed]
  17. Peters, R.D.; Heide, H.R. The Clinical Benefits of AIR™ Recon DL for MR Image Reconstruction; GE Healthcare: Singapore, 2020. [Google Scholar]
  18. Ng, A.W.H.; Griffith, J.F.; Hung, E.H.Y.; Law, K.Y.; Yung, P.S.H. MRI diagnosis of ACL bundle tears: Value of oblique axial imaging. Skelet. Radiol. 2013, 42, 209–217. [Google Scholar] [CrossRef]
  19. Ghasemi, A.; Zahediasl, S. Normality Tests for Statistical Analysis: A Guide for Non-Statisticians. Int. J. Endocrinol. Metab. 2012, 10, 486–489. [Google Scholar] [CrossRef] [Green Version]
  20. Razali, N.M.; Wah, Y.B. Power comparisons of shapiro-wilk, kolmogorov-smirnov, lilliefors and anderson-darling tests. J. Stat. Model. Anal. 2011, 2, 21–33. [Google Scholar]
  21. Zar, J.H. Spearman rank correlation. In Encyclopedia of Biostatistics; Wiley: Hoboken, NJ, USA, 2005; Volume 7. [Google Scholar]
  22. Zimmerman, D.W.; Zumbo, B.D. Relative power of the Wilcoxon test, the Friedman test, and repeated-measures ANOVA on ranks. J. Exp. Educ. 1993, 62, 75–86. [Google Scholar] [CrossRef]
  23. Abdi, H. Holm’s sequential Bonferroni procedure. Encycl. Res. Des. 2010, 1, 1–8. [Google Scholar]
  24. McGraw, K.O.; Wong, S.P. Forming inferences about some intraclass correlation coefficients. Psychol. Methods 1996, 1, 30. [Google Scholar] [CrossRef]
  25. Koo, T.K.; Li, M.Y. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J. Chiropr. Med. 2016, 15, 155–163. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  26. Fleiss, J.L.; Levin, B.; Paik, M.C. The measurement of interrater agreement. Stat. Methods Rates Proportions 1981, 2, 22–23. [Google Scholar]
  27. McHugh, M.L. Interrater reliability: The kappa statistic. Biochem. Medica 2012, 22, 276–282. [Google Scholar] [CrossRef]
  28. Cohen, J. Statistical power analysis. Curr. Dir. Psychol. Sci. 1992, 1, 98–101. [Google Scholar] [CrossRef]
  29. Faul, F.; Erdfelder, E.; Lang, A.-G.; Buchner, A. G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods 2007, 39, 175–191. [Google Scholar] [CrossRef]
  30. Afat, S.; Wessling, D.; Afat, C.; Nickel, D.; Arberet, S.; Herrmann, J.; Othman, A.E.; Gassenmaier, S. Analysis of a Deep Learning-Based Superresolution Algorithm Tailored to Partial Fourier Gradient Echo Sequences of the Abdomen at 1.5 T: Reduction of Breath-Hold Time and Improvement of Image Quality. Investig. Radiol. 2022, 57, 157–162. [Google Scholar] [CrossRef]
  31. Chaudhari, A.S.; Fang, Z.; Kogan, F.; Wood, J.; Stevens, K.J.; Gibbons, E.K.; Lee, J.H.; Gold, G.E.; Hargreaves, B.A. Super-resolution musculoskeletalMRIusing deep learning. Magn. Reson. Med. 2018, 80, 2139–2154. [Google Scholar] [CrossRef]
  32. Kelkar, V.; Zhang, X.; Granstedt, J.; Li, H.; Anastasio, M. Task-based evaluation of deep image super-resolution in medical imaging. In Medical Imaging 2021: Image Perception, Observer Performance, and Technology Assessment; SPIE: Bellingham, WA, USA, 2021. [Google Scholar]
  33. Wang, Z.; Chen, J.; Hoi, S.C.H. Deep Learning for Image Super-Resolution: A Survey. IEEE Trans. Pattern Anal. Mach. Intell. 2021, 43, 3365–3387. [Google Scholar] [CrossRef] [Green Version]
  34. McLeavy, C.; Chunara, M.; Gravell, R.; Rauf, A.; Cushnie, A.; Talbot, C.S.; Hawkins, R. The future of CT: Deep learning reconstruction. Clin. Radiol. 2021, 76, 407–415. [Google Scholar] [CrossRef]
  35. Salaha, Z.F.M.; Ammarullah, M.I.; Abdullah, N.N.A.A.; Aziz, A.U.A.; Gan, H.-S.; Abdullah, A.H.; Kadir, M.R.A.; Ramlee, M.H. Biomechanical Effects of the Porous Structure of Gyroid and Voronoi Hip Implants: A Finite Element Analysis Using an Experimentally Validated Model. Materials 2023, 16, 3298. [Google Scholar] [CrossRef] [PubMed]
  36. Jamari, J.; Ammarullah, M.I.; Santoso, G.; Sugiharto, S.; Supriyono, T.; Permana, M.S.; Winarni, T.I.; van der Heide, E. Adopted walking condition for computational simulation approach on bearing of hip joint prosthesis: Review over the past 30 years. Heliyon 2022, 8, e12050. [Google Scholar] [CrossRef]
  37. Ammarullah, M.I.; Hartono, R.; Supriyono, T.; Santoso, G.; Sugiharto, S.; Permana, M.S. Polycrystalline Diamond as a Potential Material for the Hard-on-Hard Bearing of Total Hip Prosthesis: Von Mises Stress Analysis. Biomedicines 2023, 11, 951. [Google Scholar] [CrossRef] [PubMed]
  38. Shanbhogue, K.; Tong, A.; Smereka, P.; Nickel, D.; Arberet, S.; Anthopolos, R.; Chandarana, H. Accelerated single-shot T2-weighted fat-suppressed (FS) MRI of the liver with deep learning–based image reconstruction: Qualitative and quantitative comparison of image quality with conventional T2-weighted FS sequence. Eur. Radiol. 2021, 31, 8447–8457. [Google Scholar] [CrossRef] [PubMed]
  39. Hirokawa, Y.; Isoda, H.; Maetani, Y.S.; Arizono, S.; Shimada, K.; Togashi, K. MRI Artifact Reduction and Quality Improvement in the Upper Abdomen with PROPELLER and Prospective Acquisition Correction (PACE) Technique. Am. J. Roentgenol. 2008, 191, 1154–1158. [Google Scholar] [CrossRef]
  40. Ebrahimkhani, S.; Dharmaratne, A.; Jaward, M.H.; Wang, Y.; Cicuttini, F.M. Automated segmentation of knee articular cartilage: Joint deep and hand-crafted learning-based framework using diffeomorphic mapping. Neurocomputing 2022, 467, 36–55. [Google Scholar] [CrossRef]
  41. Shen, S.; Tian, H.; Lu, H.; Chen, H.-L.; Nie, C.; Zhang, J.; Zhang, C.-C.; Wang, H. T1ρ magnetic resonance imaging for quantitative assessment of patella cartilage in healthy subjects and athletes. Measurement 2022, 191, 110785. [Google Scholar] [CrossRef]
  42. Wu, M.; Ma, Y.-J.; Liu, M.; Xue, Y.; Gong, L.; Wei, Z.; Jerban, S.; Jang, H.; Chang, D.G.; Chang, E.Y.; et al. Quantitative assessment of articular cartilage degeneration using 3D ultrashort echo time cones adiabatic T1ρ (3D UTE-Cones-AdiabT1ρ) imaging. Eur. Radiol. 2022, 32, 6178–6186. [Google Scholar] [CrossRef]
  43. Kidoh, M.; Shinoda, K.; Kitajima, M.; Isogawa, K.; Nambu, M.; Uetani, H.; Morita, K.; Nakaura, T.; Tateishi, M.; Yamashita, Y.; et al. Deep Learning Based Noise Reduction for Brain MR Imaging: Tests on Phantoms and Healthy Volunteers. Magn. Reson. Med. Sci. 2020, 19, 195–206. [Google Scholar] [CrossRef] [Green Version]
  44. Kladny, B.; Hofmann, G.; Glückert, K.; Blank-Schäl, A. MRI of the knee joint with a 3-D gradient echo sequence. Arch. Orthop. Trauma Surg. 1992, 112, 5–14. [Google Scholar] [CrossRef]
Figure 1. Flow diagram of patient selection. Data search in 3 different data bases (PubMed, Scopus and Web of Science).
Figure 1. Flow diagram of patient selection. Data search in 3 different data bases (PubMed, Scopus and Web of Science).
Diagnostics 13 02438 g001
Figure 2. MR images of the right knee joint of a 38-year-old male without any symptoms. (a) Coronal T1-weighted (T1w) standard PROPELLER MR image, (b) coronal T1w DL-CNN MR image, (c) sagittal proton-density (PD) standard PROPELLER MR image, (d) sagittal PD DL-CNN MR image, (e) axial PD FS PROPELLER MR image, and (f) axial PD FS DL-CNN MR image.
Figure 2. MR images of the right knee joint of a 38-year-old male without any symptoms. (a) Coronal T1-weighted (T1w) standard PROPELLER MR image, (b) coronal T1w DL-CNN MR image, (c) sagittal proton-density (PD) standard PROPELLER MR image, (d) sagittal PD DL-CNN MR image, (e) axial PD FS PROPELLER MR image, and (f) axial PD FS DL-CNN MR image.
Diagnostics 13 02438 g002
Figure 3. MR images of the right knee joint from a 44-year-old male experiencing knee pain following a direct trauma to the medial aspect of the knee. (a) Axial proton-density (PD) fat-saturated (FS) standard PROPELLER MR image, and (b) axial PD FS DL-CNN MR images show a superficial defect (arrow) of patellar cartilage with a horizontal course along lateral facet of the patella suggestive of a non-displaced delamination. Note a bone marrow edema of medial femoral condyle due to a direct trauma.
Figure 3. MR images of the right knee joint from a 44-year-old male experiencing knee pain following a direct trauma to the medial aspect of the knee. (a) Axial proton-density (PD) fat-saturated (FS) standard PROPELLER MR image, and (b) axial PD FS DL-CNN MR images show a superficial defect (arrow) of patellar cartilage with a horizontal course along lateral facet of the patella suggestive of a non-displaced delamination. Note a bone marrow edema of medial femoral condyle due to a direct trauma.
Diagnostics 13 02438 g003
Figure 4. MR images of the right knee joint from a 63-year-old male presenting with pain and a clicking sound in the knee joint after a twisting injury. (a) Axial proton-density (PD) fat-saturated (FS) standard PROPELLER and (b) axial PD FS DL-CNN MR images show subtle superficial defects of patellar cartilage (arrows). (c) Sagittal PD standard PROPELLER and (d) sagittal PD DL-CNN MR images reveal a horizontal tear of posterior horn of the medial meniscus extending to the tibial surface (arrow).
Figure 4. MR images of the right knee joint from a 63-year-old male presenting with pain and a clicking sound in the knee joint after a twisting injury. (a) Axial proton-density (PD) fat-saturated (FS) standard PROPELLER and (b) axial PD FS DL-CNN MR images show subtle superficial defects of patellar cartilage (arrows). (c) Sagittal PD standard PROPELLER and (d) sagittal PD DL-CNN MR images reveal a horizontal tear of posterior horn of the medial meniscus extending to the tibial surface (arrow).
Diagnostics 13 02438 g004
Figure 5. MR images of the right knee joint from a 51-year-old male suffering from chronic knee pain. (a) Sagittal proton density (PD) standard PROPELLER, (b) sagittal PD DL-CNN MR images and (c) axial PD FS PROPELLER, and (d) axial PD FS DL-CNN MR images show a radial tear of anterior horn of the lateral meniscus (arrow). Pathologies can be depicted in both sequences; however, DL-CNN sequences are less noisy, so pathological findings can be easier recognized.
Figure 5. MR images of the right knee joint from a 51-year-old male suffering from chronic knee pain. (a) Sagittal proton density (PD) standard PROPELLER, (b) sagittal PD DL-CNN MR images and (c) axial PD FS PROPELLER, and (d) axial PD FS DL-CNN MR images show a radial tear of anterior horn of the lateral meniscus (arrow). Pathologies can be depicted in both sequences; however, DL-CNN sequences are less noisy, so pathological findings can be easier recognized.
Diagnostics 13 02438 g005
Figure 6. Signal-to-noise ratio (SNR) for bone, fat, and muscle (top) and contrast-to-noise ratios (CNR) (bottom) for standard and DL-CNN sequences. The mean signal-to-noise ratio for bone, fat, and muscle was significantly higher in DL-CNN sequences compared to standard sequences (p < 0.05). Mean CNR was significantly higher in DL-CNN than in standard sequences (p < 0.05).
Figure 6. Signal-to-noise ratio (SNR) for bone, fat, and muscle (top) and contrast-to-noise ratios (CNR) (bottom) for standard and DL-CNN sequences. The mean signal-to-noise ratio for bone, fat, and muscle was significantly higher in DL-CNN sequences compared to standard sequences (p < 0.05). Mean CNR was significantly higher in DL-CNN than in standard sequences (p < 0.05).
Diagnostics 13 02438 g006
Table 1. Standard sequences and DL-CNN PROPELLER sequences. ax—axial, cor—coronal, FS—fat saturated, PD—proton density, sag—sagittal oblique, SL—slice thickness, T1w—T1-weighted, T2w—T2-weighted, TE—echo time, TR—repetition time.
Table 1. Standard sequences and DL-CNN PROPELLER sequences. ax—axial, cor—coronal, FS—fat saturated, PD—proton density, sag—sagittal oblique, SL—slice thickness, T1w—T1-weighted, T2w—T2-weighted, TE—echo time, TR—repetition time.
Standard PROPELLERDL-CNN Sequences
ax PD FSsag PDcor T1wax PD FSsag PDcor T1w FS
TE (ms)49.944.212.846.742.942.2
TR (ms)4649251751144212500541
SL (mm)333333
Spacing between slices444444
Echo train length1514715147
Echo numbers111111
Matrix300 × 300320 × 320340 × 340320 × 320360 × 360360 × 360
Flip angle (°)160160160160160160
Receiver bandwidth (kHz)162.7195.3195.3195.3244.1244.1
Number of averages3.02.82.61.61.51.6
Imaging frequency63.863.863.863.863.863.8
Acquisition time (min:s)04:0902:4603:0802:0001:0901:36
Overall acquisition time (min:s)10:0304:45
Table 2. The image quality of all evaluated structures in the knee joint was assessed using both standard and DL-CNN sequences. The results of the Wilcoxon-signed rank test for comparing the MR sequences are provided. The evaluation of bone quality included femoral, tibial, and retropatellar bones. Patellar cartilage was evaluated separately. Image quality was measured using four-point Likert scale (0-poor, 1-moderate, 2-good, 3-perfect).
Table 2. The image quality of all evaluated structures in the knee joint was assessed using both standard and DL-CNN sequences. The results of the Wilcoxon-signed rank test for comparing the MR sequences are provided. The evaluation of bone quality included femoral, tibial, and retropatellar bones. Patellar cartilage was evaluated separately. Image quality was measured using four-point Likert scale (0-poor, 1-moderate, 2-good, 3-perfect).
Standard
PROPELLER
(Mean ± SD)
DL-CNN (Mean ± SD)Wilcoxon-Signed Rank Test (p-Value)
Bone2.05 ± 0.572.90 ± 0.08
Cartilage1.95 ± 0.292.85 ± 0.22
Retropatellar cartilage1.94 ± 0.052.95 ± 0.50
Muscle2.22 ± 0.512.42 ± 0.16
Anterior cruciate ligament1.80 ± 0.052.60 ± 0.51
Posterior cruciate ligament2.24 ± 0.312.97 ± 0.20
Medial collateral ligament1.98 ± 0.022.91 ± 0.11
Lateral collateral ligament1.95 ± 0.022.88 ± 0.05
Medial meniscus1.91 ± 0.112.82 ± 0.30
Lateral meniscus2.04 ± 0.202.92 ± 0.86
Contour sharpness in central FOV1.95 ± 0.422.92 ± 0.02
Contour sharpness in peripheral FOV2.05 ± 0.402.85 ± 0.32
Homogeneity of fat saturation in central FOV2.32 ± 1.052.40 ± 0.65
Homogeneity of fat saturation in peripheral FOV1.84 ± 1.342.30 ± 0.71all < 0.05
Table 3. Diagnostic confidence of all analyzed structures of the knee joint in standard and DL-CNN sequences together with results of Wilcoxon-signed rank test for MR sequence comparison. The knee structures were assessed as groups as follows: Bone and cartilage were assessed in the femoral, tibial, and patellar bone and retropatellar cartilage; muscles were evaluated in the medial and lateral vastus muscle; ligaments were assessed as the anterior and posterior cruciate ligament, and medial and lateral collateral ligaments. The assessment of diagnostic confidence was conducted utilizing a four-point Likert scale, with ratings ranging from 0 (poor) to 3 (perfect).
Table 3. Diagnostic confidence of all analyzed structures of the knee joint in standard and DL-CNN sequences together with results of Wilcoxon-signed rank test for MR sequence comparison. The knee structures were assessed as groups as follows: Bone and cartilage were assessed in the femoral, tibial, and patellar bone and retropatellar cartilage; muscles were evaluated in the medial and lateral vastus muscle; ligaments were assessed as the anterior and posterior cruciate ligament, and medial and lateral collateral ligaments. The assessment of diagnostic confidence was conducted utilizing a four-point Likert scale, with ratings ranging from 0 (poor) to 3 (perfect).
Standard
PROPELLER
(Mean ± SD)
DL-CNN
(Mean ± SD)
Wilcoxon Signed-Rank Test (p-Value)
Bone2.07 ± 0.472.94 ± 0.23
Cartilage1.91 ± 0.402.85 ± 0.35
Muscle2.41 ± 0.552.92 ± 0.25
Ligaments2.14 ± 0.482.87 ± 0.33
Medial and lateral meniscus2.12 ± 0.522.89 ± 0.31
Subcutaneous fat tissue2.49 ± 0.552.90 ± 0.33
Overall2.04 ± 0.312.95 ± 0.20all < 0.05
Table 4. The summary of pathologies observed in all analyzed structures using both standard and DL-CNN sequences, along with the corresponding inter-reader agreements, is presented. The assessment of inter-reader agreement was performed using Cohen’s Kappa statistic. Kappa values between 0.41–0.60 were considered moderate, between 0.61–0.80 substantial, and above 0.81 almost perfect agreement.
Table 4. The summary of pathologies observed in all analyzed structures using both standard and DL-CNN sequences, along with the corresponding inter-reader agreements, is presented. The assessment of inter-reader agreement was performed using Cohen’s Kappa statistic. Kappa values between 0.41–0.60 were considered moderate, between 0.61–0.80 substantial, and above 0.81 almost perfect agreement.
Assessment of PathologiesInter-Reader Agreement
Standard
PROPELLER
(Mean ± SD)
DL-CNN (Mean ± SD)Standard PROPELLER
(κ-Value)
DL-CNN
(κ-Value)
Bone1.87 ± 0.891.80 ± 0.890.4790.508
Cartilage1.31 ± 0.461.34 ± 0.470.7390.832
Patellar cartilage1.37 ± 0.481.45 ± 0.500.7560.770
Anterior cruciate ligament1.12 ± 0.331.14 ± 0.350.6240.717
Posterior cruciate ligament1.02 ± 0.161.04 ± 0.200.6530.729
Medial collateral ligament1.12 ± 0.331.12 ± 0.330.8730.873
Lateral collateral ligament1.01 ± 0.111.01 ± 0.110.8100.814
Medial meniscus1.27 ± 0.441.24 ± 0.430.7840.767
Lateral meniscus1.14 ± 0.351.11 ± 0.320.7100.720
Joint effusion1.32 ± 0.471.31 ± 0.460.8070.736
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Kaniewska, M.; Deininger-Czermak, E.; Lohezic, M.; Ensle, F.; Guggenberger, R. Deep Learning Convolutional Neural Network Reconstruction and Radial k-Space Acquisition MR Technique for Enhanced Detection of Retropatellar Cartilage Lesions of the Knee Joint. Diagnostics 2023, 13, 2438. https://doi.org/10.3390/diagnostics13142438

AMA Style

Kaniewska M, Deininger-Czermak E, Lohezic M, Ensle F, Guggenberger R. Deep Learning Convolutional Neural Network Reconstruction and Radial k-Space Acquisition MR Technique for Enhanced Detection of Retropatellar Cartilage Lesions of the Knee Joint. Diagnostics. 2023; 13(14):2438. https://doi.org/10.3390/diagnostics13142438

Chicago/Turabian Style

Kaniewska, Malwina, Eva Deininger-Czermak, Maelene Lohezic, Falko Ensle, and Roman Guggenberger. 2023. "Deep Learning Convolutional Neural Network Reconstruction and Radial k-Space Acquisition MR Technique for Enhanced Detection of Retropatellar Cartilage Lesions of the Knee Joint" Diagnostics 13, no. 14: 2438. https://doi.org/10.3390/diagnostics13142438

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop