You are currently viewing a new version of our website. To view the old version click .
MDPIMDPI
Search all articles
Diagnostics
Download PDF
  • Article
  • Open Access

9 September 2022

Phantom Study on the Robustness of MR Radiomics Features: Comparing the Applicability of 3D Printed and Biological Phantoms

,
,
,
,
,
,
and
1
Division of Radiology and Imaging Science, Department of Medical Imaging, Faculty of Medicine, University of Debrecen, 4032 Debrecen, Hungary
2
Doctoral School of Neuroscience, Faculty of Medicine, University of Debrecen, 4032 Debrecen, Hungary
3
Doctoral School of Molecular Medicine, Faculty of Medicine, University of Debrecen, 4032 Debrecen, Hungary
4
Division of Nuclear Medicine and Translational Imaging, Department of Medical Imaging, Faculty of Medicine, University of Debrecen, 4032 Debrecen, Hungary
Diagnostics2022, 12(9), 2196;https://doi.org/10.3390/diagnostics12092196
This article belongs to the Section Medical Imaging and Theranostics
Version Notes
Order Reprints
Review Reports

Abstract

The objectives of our study were to (a) evaluate the feasibility of using 3D printed phantoms in magnetic resonance imaging (MR) in assessing the robustness and repeatability of radiomic parameters and (b) to compare the results obtained from the 3D printed phantoms to metrics obtained in biological phantoms. To this end, three different 3D phantoms were printed: a Hilbert cube (5 × 5 × 5 cm3) and two cubic quick response (QR) code phantoms (a large phantom (large QR) (5 × 5 × 4 cm3) and a small phantom (small QR) (4 × 4 × 3 cm3)). All 3D printed and biological phantoms (kiwis, tomatoes, and onions) were scanned thrice on clinical 1.5 T and 3 T MR with 1 mm and 2 mm isotropic resolution. Subsequent analyses included analyses of several radiomics indices (RI), their repeatability and reliability were calculated using the coefficient of variation (CV), the relative percentage difference (RPD), and the interclass coefficient (ICC) parameters. Additionally, the readability of QR codes obtained from the MR images was examined with several mobile phones and algorithms. The best repeatability (CV ≤ 10%) is reported for the acquisition protocols with the highest spatial resolution. In general, the repeatability and reliability of RI were better in data obtained at 1.5 T (CV = 1.9) than at 3 T (CV = 2.11). Furthermore, we report good agreements between results obtained for the 3D phantoms and biological phantoms. Finally, analyses of the read-out rate of the QR code revealed better texture analyses for images with a spatial resolution of 1 mm than 2 mm. In conclusion, 3D printing techniques offer a unique solution to create textures for analyzing the reliability of radiomic data from MR scans.

1. Introduction

Conventional analyses of cancerous lesions rely on invasive histological samples, which have several limitations, including discomfort for the patient and potential problems extracting the whole lesion for total-lesion heterogeneity analysis. Therefore, alternatives to the invasive methods are desired. One possible solution is the non-invasive, in-vivo radiomic assessment of images acquired in radiological and nuclear medicine settings [1,2,3,4,5,6]. The major strength in using radiomics as an alternative to histological sampling is the multitude of imaging series, modalities, and follow-up acquisitions employed in the clinical assessment of patients. Radiomic assessments of the various imaging modalities may give more personalized treatment regimes without requiring additional testing of the patients.
However, several challenges have been identified in the clinical translation of radiomics which may affect their applicability in the clinical routine. First, the spatial resolution of the diagnostic imaging modalities (spatial resolutions of 0.5–5 mm) are orders of magnitude worse than the histological samples (spatial resolution of 10−4–10−3 mm). Second, the choice of radiomic features may be affected by the diagnostic image modality and their modality-specific reconstruction settings [7,8,9,10,11]. Third, image discretization and normalization procedures may affect the radiomic analyses. While these problems have been identified, previous studies have reported great potential in the radiomic assessments for both CT and nuclear medicine imaging settings, suggesting that the discrepancy in spatial resolution nor the choice of imaging modality does not hinder radiomic assessments.
In terms of discretization of the images, two methods exist: the fixed bin size (FBS) and fixed bin number (FBN) [12]. Previous studies have identified that FBS may be the method of choice for quantitative imaging modalities (PET and CT); however, its feasibility on non-quantitative images (MRI) has not yet been evaluated. Therefore, it is of interest to assess how different discretization processes and radiomic features may be employed in MR imaging. Recently, the Image Biomarker Standardization Initiative (IBSI) has sought to standardize the radiomic feature extraction process, resulting in nearly 1000 different radiomics parameters [12]. One way to assess the influence of the discretization processes may include using physical phantoms, such as fruits and vegetables. However, identifying the most suitable phantom in MR imaging is not trivial, as the choice may affect both repeatability and reliability of the radiomics indexes (RIs) during analyses [9,10,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27]. Previous studies have mainly relied on biological phantoms, which decay over time, thus limiting their applicability in longitudinal and multi-center studies [14,15,16]. Recent developments in 3D printing have facilitated the possibility of printing image modality-specific phantoms at a low price and fast production time, which may serve as an alternative to biological phantoms [27,28,29,30,31]. The 3D-printing technique permits manufacturing complicated phantoms, which may arise from mathematical objects or real patient data. These phantoms are strict in shape and consistency and give the possibility to ignore or emphasize even a small number of details. However, dedicated 3D designed and printed MR radiomics phantoms have not yet appeared in the MR imaging literature.
This study aimed to compare radiomics obtained for biological and 3D printed phantoms. To this end, we developed fillable Quick Response (QR) code- and Hilbert curve-based 3D printed models to introduce new MR radiomic phantoms. QR codes and Hilbert curves are well structured and precisely defined mathematical objects, ensuring decent reproducibility in manufacturing [32,33]. QR codes may provide an additional analytical option through the potential of the read-out success rate of the information stored on a deeper level in MR images [34]. The 3D-printed phantoms were compared to biological phantoms that have previously been found useful in the literature (kiwis, onions, and tomatoes) [35].QR codes and Hilbert curves are well structured and precisely defined mathematical objects, ensuring decent reproducibility in manufacturing [32]. QR codes may provide an additional analytical option through the potential of the read-out success rate of the information stored on a deeper level in MR images [34].

2. Methods

2.1. Biological Phantoms

Given the high water content, vegetables and fruits reflect different signal intensities, shapes, and “tissue” textures in MR images, making them ideal as biological phantoms [13,15,25,36]. For this study, the biological phantoms were selected using the following criteria: the phantom must be water-rich, have an appropriate size (below 5 × 5 × 5 cm3), have certain degrees of hardness and heterogeneity, and have stable structural characteristics. Following the selection criteria, three fruits and vegetable types were considered: kiwis (n = 4), tomatoes (n = 3), and onions (n = 3). Representative MR images of one of the selected kiwis are shown in Figure 1. For every acquisition, all the fruits and vegetables were placed in two separate holders to fix and standardize the place of the phantoms during the acquisition steps. Supplementary Photo S1 is shown the placement of biological phantoms. Further, the fixation ensured a harmonic orientation of the phantoms across different acquisitions. The first holder contained four kiwis, while the second contained the tomatoes and onions. A pre-selected kiwi was rotated perpendicularly to its primary axis’s (labeled by kiwirot) between its three repetitions to examine the possible influence of the phantom orientation on the computed RI [36,37,38,39].
Figure 1. Representative orthogonal 3T MR images of a kiwi: 3D T1 weighted sagittal, coronal, and axial high resolution (1 × 1 × 1 mm3).

2.2. 3D Printed Phantoms

Two types of distinct, hollow plastic objects were planned and constructed. First, a Quick Response (QR) code was modeled within a fillable container, including the “UNIDEB MRI Texture Analysis Phantom” text information. We used a web application for QR code production (https://www.qrcode-monkey.com/#text (accessed on 6 May 2021)). The QR code was printed into two 3D models with different sizes (large QR (5 × 5 × 4 cm3) and small QR (4 × 4 × 3 cm3), with respective heights of the QR code of 3 and 2 cm) using the Trimble SketchUp Pro 2020 (Trimble Inc., Sunnyvale, CA, USA). Both cubic QR code containers were constructed and saved in Standard Tessellation Language (STL) file format (Figure 2, first row) [33,34,40,41].
Figure 2. Representative photo and model images of the 3D printed phantoms: the top row shows a photo and the design image of the QR code, while the bottom row shows the design images of the modified Hilbert cube after and before modeling from left to right.
As the second type of printed phantom, we chose a 3D Hilbert cube with a volume of 5 × 5 × 5 cm3 [42]. The recursive process fills the entire space with one continuous line. We used an STL file of a pre-created Hilbert cube from the Thingiverse website to embed it to the mentioned volume (https://www.thingiverse.com/thing:1762713 (accessed on 10 June 2021)). From the downloaded STL files, the second recursion level was chosen due to its sufficient complexity and ease of redesign. We redesigned the original model containing several small vertical and horizontal support lines to achieve a more robust form to ensure successful printing and make an extra chessboard-like pattern in three dimensions. This model has an outside pattern with a plastic-air alternating rectangle and includes the so-called Hilbert square pipe (Figure 2, second row).
All three phantoms were 3D printed from Polylactic Acid (3DJake ecoPLA, White) filament on a Creality Ender 3, Fused Deposition Modeling (FDM) type 3D printer with a 0.4 mm nozzle diameter [27,30]. The print plan was achieved using Repetier-Host (Hot-World GmbH and Co. KG, Willich, Germany) software with the following settings: 100% infill; 0.2 mm layer height; no support; brim adhesion; 40 mm/s print speed; 210 °C hot end, and 60 °C print bed temperatures. The print time was around 4 h for each object [30,43]. Each 3D printed phantom was filled with 0.01 mM NiCL2 solution. Representative MR images of the QR and the Hilbert cube phantoms are presented in Figure 3.
Figure 3. Representative coronal MR images of 3D printed phantoms. The left is the large QR cube, and the right is the Hilbert cube. The interested readers can read the presented QR code MR image using their smartphone’s QR code reader.

2.3. MR Scanning

All phantoms were scanned at the MR imaging facility of the Clinical Center, University of Debrecen. The phantoms were scanned in two MR systems, a 3 T Philips Achieva and a 1.5 T Siemens Magnetom Essenza MRI scanner. The scans were performed with a 6-channel head coil in the 1.5 T MR and an 8-channel head or a 32-channel neurovascular coil in the 3 T MRI system. In both devices, isotropic 3D T2-weighted and 3D T1-weighted sequences were obtained using protocols employed in the clinical routine, with minor adjustments to the number of repetitions to ensure sufficient image quality in the smaller volumes. In the Philips MR system, the T1- and T2-weighted measurements were obtained using a 3D BrainVIEW protocol which provides high-resolution isotropic data. For the Siemens system, the SPACE (Sampling Perfection with Application optimized Contrasts using different flip angle Evolution) sequence was applied for T2 acquisitions, which provides isotropic 3D images. For T1-weighted Gradient-Echo (GRE) measurements, the sequence MPRAGE (Magnetization-Prepared Rapid GRE) was utilized [44,45,46].
The acquisition parameters for the acquisition protocols are listed in Table 1. Each imaging protocol was performed using two different isotropic voxel resolutions (1 × 1 × 1 mm3 and 2 × 2 × 2 mm3). The phantoms were scanned thrice for each setting in both MR systems, denoted as repeated scans or repetitions below. Specific to the Philips MR system, a new table position had to be chosen before each repetition. After completing imaging protocol, the acquisition software automatically moved the table from the gantry. However, the phantoms were always placed in the isocenter, the most homogeneous magnetic field region.
Table 1. Parameters of the sequences used in this work. TR = repetition time, TE = echo time, NSA = number of signals averaged.

2.4. Image Visualization and Segmentation

A radiologist and a radiologist MR specialist with 30 and 10 years of MRI experience, respectively, analyzed the image data qualitatively in 3D Slicer software [47,48] to determine how close the imaged texture of each object was to the actual pattern and heterogeneity.
Image alignment was performed semiautomatically using the 3D Slicer open-source software (version 4.10.2 r28257) [49]. A cubic volume of interest (VOI) was placed separately on each 3D printed phantom using the CreateModels tool from the SlicerIGT and the Segmentations modules. VOI sizes were 50 × 50 × 50, 55 × 55 × 55, and 45 × 45 × 45 mm3 for the Hilbert cube, the large QR, and the small QR phantoms, respectively. The “Grow from seeds” algorithm was applied for biological phantoms to define VOIs that blend into the surface of fruits and vegetables as much as possible. The semiautomatically generated VOIs were manually corrected by excluding the border zone between fruit/vegetable and surrounding air and the most apical and basal slice of each fruit/vegetable using a brush-erase tool. Corresponding label maps from segmentations of all images were exported and saved to Neuroimaging Informatics Technology Initiative (NIfTI) file format for further processing and calculation [50,51].

2.5. Normalization and Discretization

In this study, we used a so-called µ ± 3σ normalization technique with µ being the mean and σ the standard deviation of the image [52,53,54,55,56]. Two different discretization techniques were considered; the FBS and the FBN techniques. The FBS method is defined as
I F B S i = I i B − 1
where I(i) and IFBS(i) are the original and the transformed intensity level of the ith voxel [53,54,57,58]. The [] brackets stand for the ceil operation. We choose B = 0.15 for normalized images and B = 50 for non-normalized ones to have a similar number of bins in both cases [7,12,59].
The FBN method is calculated by
I F B N i = 1 I i = I m i n D · I i − I m i n I m a x − I m i n o t h e r w i s e
where IFBN(i) is the new intensity value of the i-th voxel intensity after the FBN discretization, Imax is the maximum, Imin is the minimum original voxel intensity of the particular lesion, and D is the number of bin parameters. We set D to 64.

2.6. Texture Calculation

RIs were extracted using the GLCM (Gray Level Co-Occurrence Matrix), GLSZM (Gray Level Size Zone Matrix), and GLRLM (Gray Level Run Length Matrix)-based algorithms implemented in MATLAB (version 2020) [7]. A total of 40 radiomic features were extracted from each VOI, divided into 18 GLCM, 11 GLSZM, and 11 GLRLM metrics. All 40 RIs (Supplementary Table S1) were calculated according to the IBSI guideline [53,59]. In addition, five basic histogram-based statistical parameters were also determined as the minimum and maximum value, mean, median, and VOI volume in voxel numbers. All 45 features were determined for the segmented volumes for each acquisition and discretization setup (Table 2).
Table 2. Abbreviations for the different imaging setups. The second column shows the setup parameters, listing the filed strength, the weighting (T1 or T2), the applied coil type, the acquisition voxel size, and the discretization method. A total of 24 settings were considered for each RI calculation.

3. Statistical Analysis

3.1. Coefficient of Variation

For each RI and each acquisition, we report the mean, standard deviation, and coefficient of variation (CV), defined as
C V = s t d R I m e a n R I · 100
where std(RI) and mean(RI) represent the standard deviation and mean for the three computed radiomics indices. We utilized the CV parameter as a measure of the repeatability of a group or object [9,18,60,61].

3.2. Relative Parameter Differences

The relative parameter difference (RPD) value was employed to measure the relative RI difference of a given phantom between two different measurement setups. The relative parameter difference was defined as:
R P D =   m e a n ( R I s e t u p 1 ) − m e a n ( R I s e t u p 2 ) m e a n ( R I s e t u p 1 ) · 100
With m e a n ( R I s e t u p 1 ) and m e a n ( R I s e t u p 2 ) being the mean of the three RI at two different acquisition setups (for the definitions of acquisition setups, see Table 2).
Table 3 shows all the acquisition setup pairs and the related abbreviation used in the comparative analysis for each phantom type for T1 contrast. For the T2 contrast, the number of comparisons was the same, giving a total of 14 comparisons for both contrasts.
Table 3. List of abbreviations for calculating the RPD at T1 and T2 contrast. The first column contains the abbreviations, and the second and third columns show the serial number of the acquisition setup defined in Table 2.

3.3. Interclass Correlation Coefficient

For the repeatability test, the interclass correlation coefficient (ICC) for two-way mixed effects [62] was calculated for each RI based on absolute agreement, single rater/measurement model. The ICC was determined by matching results (averages of the repeated measures) from two different measurement setups and using every phantom with the following formula:
I C C = M S R − M S E M S R + k − 1 M S E + k n M S C − M S E
where M S R stands for mean square for rows, M S E is the mean square of error, M S C the mean square of columns, n, and k are the numbers of subjects and the number of raters/measurements. Based on the ICC-values the RI were considered to be either excellent (ICC > 0.9), good (0.75 < ICC ≤ 0.9), moderate (0.5 < ICC ≤ 0.75) and poor (ICC ≤ 0.5) repeatability [21,63,64,65,66]. Every calculation was performed using MATLAB (2020, The MathWorks Inc., Natick, MA, USA) and Microsoft Office Excel software.

3.4. QR Code Readability Test

The QR phantoms were evaluated using two analyses, the QR codes readability in the MR scans and the radiomics analysis of the texture. The readability of the QR code was tested for each of the coronal 2D images. An unsuccessful read-out of the QR code for any coronal slice was defined as a significant image distortion (>25% data loss) and thereby detrimental texture loss. The reading of the QR codes relied on two types of decoding methods; first, we wrote a Python program, while the second decoding was obtained using cell phone readings of the MR images. For the Python implementation, we utilized the Pyzbar module (https://pypi.org/project/pyzbar (accessed on 29 June 2021)) to read the coded textural information. The Pyzbar module ensures that multiple QR codes can be decoded simultaneously from an image with multiple QR codes. The original DICOM images must be converted to Portable Network Graphics (.png) format for this process. The resulting .png images will be mentioned as ‘original images’ in this manuscript. The original images were processed further and resized to 1024 × 1024-pixel resolution using the Lanczos interpolation method implemented in the Python Pillow library to ensure the read-out yield stability. The interpolated, 2D images will be referred to as ‘interpolated images’ below. The evaluation program reads every coronal image from a scan of QR code phantoms and tries to read out each coded information. The number of successful readings was expressed as a percentage of the total number of coronal images, and this parameter was defined as the reading ratio. For the second read-out method, we used commercial mobile phones to detect the QR codes from the same monitor screen (ViewSonic VP2030B monitor with 100% luminance and 1600 × 1200-pixel resolution) displaying the previously constructed PNG images of the QR phantoms [67,68,69,70]. The displayed size of each PNG image has been set to display the QR phantoms in their original, true size. Five phones were involved from three manufacturers: iPhone 12, iPhone SE 2020, Samsung Galaxy A51, Xiaomi Redmi 6A, and Xiaomi MI 9 lite, referred to respectively as Phone 1 through Phone 5 below. From a given scan, only the middle coronal image was processed by each phone.

4. Results

4.1. Visual Comparison

Figure 4A shows representative images of the large QR and the Hilbert phantoms at T1 weighted contrast and different acquisition resolutions (1 mm on the left and 2 mm on the right). We report a robust read-out of the Hilbert cube disregarding the image resolutions. In contrast, significant changes in the high-frequency patterns were reported for the QR cubes when the image resolution was changed (Figure 4C). After visual inspection of the high-resolution and low-resolution images, a radiologist and a radiographer MR specialist reported that the textures observed in the biological phantoms were comparable to the real heterogeneity. However, the finer structures (high-frequency features) were blurred (Figure 4B,D).
Figure 4. Representative 3D T1- and T2-weighted 3T MR coronal images of the 3D printed and biological phantoms. The pairwise left and right images had isotropic resolutions of 1 × 1 × 1 mm3 and 2 × 2 × 2 mm3, respectively. (A): 3D T1-weighted 3T MR coronal images of the Hilbert cube and the large QR phantoms. (B): 3D T1-weighted images of 3 tomatoes and three onions. (C): 3D T2-weighted 3T MR coronal images of small and large QR code phantoms. (D): 3D T2-weighted images of the same three tomatoes and three onions.

4.2. Coefficient of Variation

For repeatability purposes, CVs were determined using normalized and non-normalized data from each object with each MR acquisition setup (Supplementary Figures S1 and S2). Improved repeatability (lower CV) measures were observed for all phantoms following the normalization of the data (Table 4 and Table 5).
Table 4. Repeatability is expressed via CVaverage, in the case of normalized and non-normalized images. At normalized images, lower CV values can be observed in most cases.
Table 5. CVaverage computed for each radiomics parameter group for all objects.
Below, we present only results from the normalized data. Given the poor repeatability (CV > 10%), a subset of RIs were excluded from the subsequent analyses (Jmax, Energy, ClusterShade, HGRE, SRHGE, LRHGE, LZE, LZLGE, LZHGE).
The CVs for the remaining 31 RI were obtained using the different weighting (T1 and T2), and their repeatability coefficients are shown in Figure 5, Figure 6 and Figure 7 and Supplementary Figure S3. Table 6 comprises the corresponding CVaverage values for different property groups (Acquisition setup, Texture parameter, and Discretization method). In general, it may be observed that most texture features’ repeatability was better at 1.5 T field strengths than at 3 T (CVaverage is 1.94 at 1.5 T and 2.11 at 3 T) (Figure 5, Figure 6 and Figure 7 and Table 4). In addition, no major differences were observed between the FBN and FBS discretization (CVaverage(FBN) = 2.04 and CVaverage(FBS) = 2.06).
Figure 5. CV of radiomics parameters for Hilbert, large QR, and small QR code cubes. Four radiomics parameter groups (GLCM, GLRLM, GLRLM, and histogram-based) are shown on the horizontal axis. The CV is expressed in %.
Figure 6. CV of radiomics data obtained for the second tomato, onion, and kiwi. Corresponding figures for all phantoms can be found in the Supplementary Figure S3. The CV is expressed in %.
Figure 7. CV data for kiwirot and kiwi2 objects. The CV is expressed in %.
Table 6. Average CV values for each object and different settings or properties. The last row shows the CVaverage for the case when all objects’ data are included.
Each object’s CV result from different points of view is presented in Table 6. The choice of discretization did not affect the repeatability measures (Table 6). In the texture parameters group, increased histogram-based CV values can be observed at the kiwis compared to the other objects’ same calculations. This phenomenon was not identified for the kiwirot (kiwi rotated through its primary axis between measurements). Columns of the acquisition setup group reveal that CV values are usually smaller at 1.5 T field strength and 1 mm resolution. Further, when excluding the kiwis, no difference in the repeatability was observed between the biological and 3D phantoms. Detailed CV data for all objects and acquisition setups are presented in Figure 5, Figure 6 and Figure 7 heatmaps.
Figure 7 shows the sensitivity of the radiomics when the scanned object rotates in the MR system. The CV calculated from the three orthogonal orientations (kiwirot) is larger than the CVs obtained from three repetitions of a given kiwi in the conventional acquisition orientation.

4.3. Relative Difference

The relative parameter difference (RPD) values are shown in the Supplementary Figure S6. The biggest relative differences are observed when comparing results obtained at different MR field strengths.

4.4. Interclass Correlation Coefficient

Figure 8 represents all ICC data of RI for the same comparison types as in the RPD analysis (in the Supplementary Figure S5). The high ICC correlates well with the larger RPD values for each object and RI.
Figure 8. Heatmap of the ICCs calculated for the 14 different comparisons and computed RI. Black squares correspond to ICCs with a value below zero, indicating high error rates between the measurements. ICC = interclass correlation coefficient.

4.5. QR Code Readability Test

QR code readability outcomes show apparent dependency on the acquisition resolution with a standalone Python code and mobile phones. None of the applied read-out methods could detect the information from the 2 × 2 × 2 mm3 resolution scans, and only some of the 1 × 1 × 1 mm3 resolution images could be successfully read. Read-out ratios originating from the Python code-based process are presented in Figure 9. Ratios are calculated from the three repeated measurements as the average read-out ratio for each type of measurement.
Figure 9. Averaged read-out ratios of the three repeated measurements in the case of the Python program processing. All original and interpolated coronal images of small QR and large QR phantoms were read by the developed Phyton code. Data from the 2 × 2 × 2 mm3 resolution are not presented due to the lack of successful decoding.
Mobile phone read-out numbers are shown in Table 7. The related score distribution is similar to the result shown in Figure 9. Many of the T1 images of the QR cubes were readable by the phones.
Table 7. Read-out results by different smartphones. The middle coronal images of the three consecutive measurements were used to show the ratio of the successful and total decrypting of the QR code information. Data from the 2 × 2 × 2 mm3 resolution are not presented due to the lack of successful decoding.
Representative QR code images are shown in Figure 10 at T1 and T2 weighting and 3T field strength.
Figure 10. Representative coronal images from the small and the large QR phantom scans at 3 Tesla field strength and 1 × 1 × 1 mm3 resolution. T1 and T2 contrast images are in the left and right columns, respectively. The encoded information can only be read out with a smartphone from the T1 image of the large QR phantom.

5. Discussion

This study aimed to test the feasibility of using printed 3D phantoms with structural information to analyze the robustness and repeatability of radiomics in MR imaging. To this end, this was performed using three different types of printed 3D phantom models and three types of biological phantoms (kiwis, tomatoes, and onions). The main finding of this study was that 3D printed phantoms provide similar robustness and repeatability metrics as biological phantoms and thus, may be favorable in identifying the optimal radiomics in MR imaging protocols. Of note, for high-resolution MR images (isotropic volume of 1 × 1 × 1 mm3), it is possible to retrieve the structural QR information, thus providing acceptable image quality for radiomic analyses. To the best of our knowledge, this finding has not been previously demonstrated in studies evaluating radiomics.
Radiological assessments in tumor staging are primarily focused on the number and the size of lesions in radiological settings when using MR; nevertheless, there is also a growing interest in measuring and analyzing radiomic characteristics, which may provide more insight into the lesion composition [71,72,73,74,75,76,77,78,79,80]. Furthermore, it is not fully understood in all its details how the different MR systems and acquisition protocols affect the robustness and reliability of radiomics parameters [64,81,82,83,84]. In general terms, radiomic assessments are challenged by the non-quantitative nature of MR imaging. The intensities in the MR images may be affected by shimming protocols and positioning of the scanned objects, among other factors, which may pose challenges in the test-retest assessment of the radiomic information [77,85]. In addition to the standard imaging parameters such as field of view, spatial resolution, and reconstruction algorithm, other factors such as magnetic field strength, repetition number, echo time, number of excitations (NEX or NSA), or the signal-to-noise ratio itself have a high impact on calculations [86]. Nevertheless, there is also a growing interest in measuring and analyzing radiomic characteristics [71,72,73,74,75,76,77,78,79,80]. One way to introduce radiomics into MR imaging protocols and to identify the relevant metrics for MR images may be through phantom studies. Previous studies have focused on biological phantoms, showing that fruits and vegetables are suitable as phantoms because they have sufficient water and organically fine texture [15,16,36,38,87]. In addition, Werz et al. showed T1 and T2 relaxation times for fruits and vegetables are similar to human tissues [36]. They found that the measured image quality of biological objects has good reproducibility, making them useful for test measurements, sequence developments, and optimization. Despite the positive findings as potential phantoms, the fruit and vegetables have limited applicability in the multi-center studies of radiomics because of their relatively rapid biological decay. Therefore, this study sought to identify if 3D printed phantoms may act as replacements. Figure 5 and Figure 6 and Table 4 and Table 6 showed that the 3D-printed phantoms provided repeatability metrics similar to those obtained from the biological phantoms. Further, it was found that the discretization process did not affect the repeatability metrics.
In radiomics, it is important to identify the most relevant metrics for tumor characterization. The literature and IBSI guidelines have identified several hundred radiomic metrics that may be extracted from segmented VOIs in medical images [1,3,53,54,57,88,89,90,91]. While all metrics may contribute to the lesion assessment, many may be relevant only for a subset of organs and tumor phenotypes, while others may have imaging modality-specific performances. Therefore, a pre-selection of the metrics used for the initial radiomic assessments may improve both the reliability of the identified metrics and the overall processing speed of the applied machine-learning models [21,80,92,93]. Our proposed 3D printed phantoms may suit such purposes.
Several studies of biological phantoms have proven that the segmentation procedure may affect texture analysis; hence we chose two robust volume delineation methods: the fixed cube-shaped VOI definition and the “grow from seeds” 3D Slicer tool [7,47]. Figure 4A demonstrates that the visible robust texture of the Hilbert cube is not affected by the spatial resolution when using 1 mm and 2 mm isotropic volumes. However, the image structure of the finer patterned QR cubes deteriorates at 2 mm spatial resolution for both the large QR and small QR cubes (Figure 4A,C). Similar findings were reported for the biological phantoms (Figure 4B,D), where high-frequency objects such as the seeds were smeared. These results emphasize that the performance level of RI is highly dependent on the spatial resolution of the input images.
The radiomic analyses’ repeatability is important in translating their use into the clinical routine. In this study, we investigated the repeatability measures at different steps in the analyses, the normalization process, and the RI, in addition to how they were affected by object size and imaging parameters. We report improved repeatability for both biological and printed phantoms when normalization is applied to the data (Supplementary Figures S1 and S2, Table 4), findings that are in concordance with previous studies [7,52,56,94,95,96,97]. Similarly, when normalization was applied to the data, improvements were observed for the two discretization methods (FBS and FBN). This finding is concordant with previous findings for the FBS normalization [56,71,94,98]. Similarly, the radiomics parameter groups (Table 5) had improved repeatability when normalization was applied to the data, stressing the importance of using normalization of MR images before radiomic analyses. Despite normalization, we observed that a subset of RIs was performing poorly (CV > 10%) for all MR imaging protocols and objects. Because of their poor performance, Jmax, Energy, ClusterShade, HGRE, SRHGE, LRHGE, LZE, LZLGE, and LZHGE were omitted from further analyses in this study, and only normalized data were used in all following evaluations. The resulting RIs were observed to perform better at lower MR field strengths, in agreement with previous reports (Figure 5, Figure 6 and Figure 7, Table 6, and Supplementary Figure S3), findings likely caused by worsened field homogeneity at 3 T [10].
In terms of parameter performance, we observed that the relative difference (RPD) might change by more than 20% for the same objects, depending on the MR systems’ field strengths (Table 3, Supplementary Figure S5, and Figure 6). Further, a bias was observed for the RPD when altering the imaging resolution, regardless of the image weighting (T1 or T2) and phantom type. Based on RPD comparisons, it can be concluded that, from the applicability point of view, 3D printed phantoms are as good as biological ones. It should also be noted that artificial phantoms eliminate some problems related to biological properties, such as putrefaction.
In terms of repeatability, high test-retest variance (high CV, poor repeatability) was observed for the mean and median values obtained in the objects (Table 6). The poor repeatability measures are likely introduced by the shimming the B0 field before each scan (introducing small alterations in signal intensities) and the non-quantitative intensities of the MR images [65,93,99,100]. The ICC was calculated by averaging the RI from the three repetitions of a given setup and matching these results between two different measurement setups (Table 3) involving all phantoms. Figure 8 shows the calculated ICCs of RI for the same comparison types as in the RPD analysis (Supplementary Figure S4). The expected negative correlation of the ICC and RPD values is clearly visible within each parameter group. In general, a larger ICC corresponds to a smaller RPD value. Results of the manufactured QR cubes show how to degrade the stored deep-level information by MR imaging. Applying different coils, field strength, or even acquisition resolution influences the level of information loss [10,15,71,72,101]. The metric of the degradation of the embedded information could be used as a new, useful parameter besides the existing radiomics characteristics.
In this study, the quality of the texture information analysis was also evaluated by the read-out yield of QR codes. It was hypothesized that a successful read-out of the QR code meant that the MR image series preserved most of the physical properties of the original texture. Using Python read-out of the QR codes from the converted .png files, we report read-out successes < 5% for the small QR when using the T2 sequence at 3 T, while no read-outs were possible for the original (non-interpolated images) at 1.5 T (Figure 9). However, at 3 T, the original and interpolated large QR phantom may be decrypted when using the T1-weighted images. These findings were reproduced by reading the QR code with smartphones (Table 7). In general, the read-out success rates were of varying success, with read-out percentages ranging from 19% to 39%. The smartphone read-out success rates highlight that the successful decoding of stored information depends on the acquisition settings and the reader algorithm. This phenomenon is analog to the clinical situation where lesion detection may depend on the physician’s practice.
The 3D printed phantoms developed in this study have many advantages and could be helpful in the quality assurance of radiomic studies involving MR datasets. Printed phantoms using the same materials and print settings at the local sites are an easy and affordable method to compare the radiomics performance of different MR scanners. For the analysis of radiomic characteristics, the flexibility of 3D printing could be a favorable method. The newly proposed 3D printed phantoms and the results with the biological phantoms of this study may benefit the radiomics community, which seeks to standardize both imaging protocols and radiomic analysis strategies [21,43].
Our study has some limitations. First, the results from phantom studies cannot always be transferred directly into clinical studies. In clinical studies, the patients are known to move during the acquisition protocols, these patterns cannot be reproduced by simple 3D-printed phantoms examined in this study [92,102]. Therefore, the results obtained here simulate a best-case scenario where no patient motion is observed. Unlike analyses solely based on 3D printed and biological phantoms, results based on actual human tissue can better indicate the usefulness of radiomics. This is because whatever phantom is proposed, they are always a simplification of a real human tissue environment [35,38,103,104]. Second, we examined only the 3D radiomic characteristics of certain texture classes. However, we wanted to include only the most frequently used ones in this work [13,35]. In addition, the number of MR devices and sequences included in the study was also limited.

6. Conclusions

We report good agreements between observations obtained from biological and 3D-printed phantoms. Three-dimensional-printed QR codes provide a unique opportunity to analyze the reliability and challenges of radiomics in MR imaging protocols. This study found that the large QR phantom provides better insight into identifying radiomic features of interest than the usual Hilbert phantom. Further, the QR codes permit analyses of texture distortion through external validation of the readability of the QR codes using smartphone read-out success rates.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/diagnostics12092196/s1, Table S1: Nomenclature of the calculated Texture Indices; Figure S1: Coefficient of Variation for all non-normalized Texture Indices and for every phantom; Figure S2: Coefficient of Variation for all normalized Texture Indices and for every phantom; Figure S3: Coefficient of Variation for the selected normalized Texture Indices; Figure S4: Relative Parameter Difference for the selected normalized Texture Indices; Figure S5: Interclass Correlation Coefficient for all normalized Texture Indices; Figure S6: Relative Parameter Difference with different acquisition setups for every fantom; Photo S1: Placement of biological phantoms.

Author Contributions

Conceptualization, L.B., J.K. and G.V.; methodology, L.B., J.K. and E.B.; software, J.K., N.F.V., P.K.-B. and N.B.M.; investigation, L.B., J.K. and G.V.; resources, G.V.; writing—original draft preparation, G.V. and J.K.; writing—review and editing, L.B. and M.L.L.; visualization, G.V. and J.K.; supervision, M.L.L.; funding acquisition, L.B. All authors have read and agreed to the published version of the manuscript.

Funding

The research was supported by the Thematic Excellence Program (TKP2020-NKA-04) of the Ministry for Innovation and Technology in Hungary.

Institutional Review Board Statement

Not applicable.

Data Availability Statement

The data presented in this study are available in the article and the corresponding supplementary materials.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Gillies, R.J.; Kinahan, P.E.; Hricak, H. Radiomics: Images Are More than Pictures, They Are Data. Radiology 2016, 278, 563–577. [Google Scholar] [CrossRef] [PubMed]
  2. Aerts, H.J.W.L.; Velazquez, E.R.; Leijenaar, R.T.H.; Parmar, C.; Grossmann, P.; Cavalho, S.; Bussink, J.; Monshouwer, R.; Haibe-Kains, B.; Rietveld, D.; et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat. Commun. 2014, 5, 4006. [Google Scholar] [CrossRef]
  3. Lambin, P.; Rios-Velazquez, E.; Leijenaar, R.; Carvalho, S.; Van Stiphout, R.G.P.M.; Granton, P.; Zegers, C.M.L.; Gillies, R.; Boellard, R.; Dekker, A.; et al. Radiomics: Extracting more information from medical images using advanced feature analysis. Eur. J. Cancer 2012, 48, 441–446. [Google Scholar] [CrossRef]
  4. Schick, U.; Lucia, F.; Bourbonne, V.; Dissaux, G.; Pradier, O.; Jaouen, V.; Tixier, F.; Visvikis, D.; Hatt, M. Use of radiomics in the radiation oncology setting: Where do we stand and what do we need? Cancer/Radiotherapie 2020, 24, 755–761. [Google Scholar] [CrossRef] [PubMed]
  5. Bibault, J.E.; Xing, L.; Giraud, P.; El Ayachy, R.; Giraud, N.; Decazes, P.; Burgun, A. Radiomics: A primer for the radiation oncologist. Cancer/Radiotherapie 2020, 24, 403–410. [Google Scholar] [CrossRef]
  6. Lambin, P.; Leijenaar, R.T.H.; Deist, T.M.; Peerlings, J.; De Jong, E.E.C.; Van Timmeren, J.; Sanduleanu, S.; Larue, R.T.H.M.; Even, A.J.G.; Jochems, A.; et al. Radiomics: The bridge between medical imaging and personalized medicine. Nat. Rev. Clin. Oncol. 2017, 14, 749–762. [Google Scholar] [CrossRef]
  7. Veres, G.; Vas, N.F.; Lassen, M.L.; Béresová, M.; Krizsan, A.K.; Forgács, A.; Berényi, E.; Balkay, L. Effect of grey-level discretization on texture feature on different weighted MRI images of diverse disease groups. PLoS ONE 2021, 16, e0253419. [Google Scholar] [CrossRef]
  8. Mayerhoefer, M.E.; Szomolanyi, P.; Jirak, D.; Materka, A.; Trattnig, S. Effects of MRI acquisition parameter variations and protocol heterogeneity on the results of texture analysis and pattern discrimination: An application-oriented study. Med. Phys. 2009, 36, 1236–1243. [Google Scholar] [CrossRef]
  9. Mi, H.; Yuan, M.; Suo, S.; Cheng, J.; Li, S.; Duan, S.; Lu, Q. Impact of different scanners and acquisition parameters on robustness of MR radiomics features based on women’s cervix. Sci. Rep. 2020, 10, 20407. [Google Scholar] [CrossRef] [PubMed]
  10. Ammari, S.; Pitre-Champagnat, S.; Dercle, L.; Chouzenoux, E.; Moalla, S.; Reuze, S.; Talbot, H.; Mokoyoko, T.; Hadchiti, J.; Diffetocq, S.; et al. Influence of Magnetic Field Strength on Magnetic Resonance Imaging Radiomics Features in Brain Imaging, an In Vitro and In Vivo Study. Front. Oncol. 2021, 10, 541663. [Google Scholar] [CrossRef] [PubMed]
  11. Forgács, A.; Béresová, M.; Garai, I.; Lassen, M.L.; Beyer, T.; Difranco, M.D.; Berényi, E.; Balkay, L. Impact of intensity discretization on textural indices of [18F]FDG-PET tumour heterogeneity in lung cancer patients. Phys. Med. Biol. 2019, 64, 125016. [Google Scholar] [CrossRef] [PubMed]
  12. Zwanenburg, A.; Abdalah, M.A.; Apte, A.; Ashrafinia, S.; Beukinga, J.; Bogowicz, M.; Dinh, C.V.; Götz, M.; Hatt, M.; Leijenaar, R.T.H.; et al. PO-0981: Results from the Image Biomarker Standardisation Initiative. Radiother. Oncol. 2018, 127, 258–264. [Google Scholar] [CrossRef]
  13. Bernatz, S.; Zhdanovich, Y.; Ackermann, J.; Koch, I.; Wild, P.J.; dos Santos, D.P.; Vogl, T.J.; Kaltenbach, B.; Rosbach, N. Impact of rescanning and repositioning on radiomic features employing a multi-object phantom in magnetic resonance imaging. Sci. Rep. 2021, 11, 14248. [Google Scholar] [CrossRef] [PubMed]
  14. Dreher, C.; Kuder, T.A.; König, F.; Mlynarska-Bujny, A.; Tenconi, C.; Paech, D.; Schlemmer, H.P.; Ladd, M.E.; Bickelhaupt, S. Radiomics in diffusion data: A test–retest, inter- and intra-reader DWI phantom study. Clin. Radiol. 2020, 75, 798.e13–798.e22. [Google Scholar] [CrossRef] [PubMed]
  15. Baek, S.; Lim, J.; Lee, J.G.; McCarthy, M.J.; Kim, S.M. Investigation of the maturity changes of cherry tomato using magnetic resonance imaging. Appl. Sci. 2020, 10, 5188. [Google Scholar] [CrossRef]
  16. Kalne, A.; Kotwaliwale, N.; Singh, K.; Singh, V.K. Non-Destructive Inspection of Mango Fruit Using Digital Radiography, Computed Tomography and Magnetic Resonance Imaging. J. Agric. Eng. 2012, 49, 33–41. [Google Scholar]
  17. Mayerhoefer, M.E.; Szomolanyi, P.; Jirak, D.; Berg, A.; Materka, A.; Dirisamer, A.; Trattnig, S. Effects of magnetic resonance image interpolation on the results of texture-based pattern classification a phantom study. Investig. Radiol. 2009, 44, 405–411. [Google Scholar] [CrossRef]
  18. Wong, O.L.; Yuan, J.; Zhou, Y.; Yu, S.K.; Cheung, K.Y. Longitudinal acquisition repeatability of MRI radiomics features: An ACR MRI phantom study on two MRI scanners using a 3D T1W TSE sequence. Med. Phys. 2021, 48, 1239–1249. [Google Scholar] [CrossRef]
  19. Jirák, D.; Dezortová, M.; Hájek, M. Phantoms for texture analysis of MR images. Long-term and multi-center study. Med. Phys. 2004, 31, 616–622. [Google Scholar] [CrossRef] [PubMed]
  20. Valladares, A.; Beyer, T.; Rausch, I. Physical imaging phantoms for simulation of tumor heterogeneity in PET, CT, and MRI: An overview of existing designs. Med. Phys. 2020, 47, 2023–2037. [Google Scholar] [CrossRef]
  21. Rai, R.; Holloway, L.C.; Brink, C.; Field, M.; Christiansen, R.L.; Sun, Y.; Barton, M.B.; Liney, G.P. Multicenter evaluation of MRI-based radiomic features: A phantom study. Med. Phys. 2020, 47, 3054–3063. [Google Scholar] [CrossRef] [PubMed]
  22. Bianchini, L.; Botta, F.; Origgi, D.; Rizzo, S.; Mariani, M.; Summers, P.; García-Polo, P.; Cremonesi, M.; Lascialfari, A. PETER PHAN: An MRI phantom for the optimisation of radiomic studies of the female pelvis. Phys. Medica 2020, 71, 71–81. [Google Scholar] [CrossRef] [PubMed]
  23. Bianchini, L.; Santinha, J.; Loução, N.; Figueiredo, M.; Botta, F.; Origgi, D.; Cremonesi, M.; Cassano, E.; Papanikolaou, N.; Lascialfari, A. A multicenter study on radiomic features from T2-weighted images of a customized MR pelvic phantom setting the basis for robust radiomic models in clinics. Magn. Reson. Med. 2020, 85, 1713–1726. [Google Scholar] [CrossRef] [PubMed]
  24. Orlhac, F.; Lecler, A.; Savatovski, J.; Goya-Outi, J.; Nioche, C.; Charbonneau, F.; Ayache, N.; Frouin, F.; Duron, L.; Buvat, I. How can we combat multicenter variability in MR radiomics? Validation of a correction procedure. Eur. Radiol. 2021, 31, 2272–2280. [Google Scholar] [CrossRef]
  25. Baeßler, B.; Weiss, K.; Santos, D.P. Dos Robustness and Reproducibility of Radiomics in Magnetic Resonance Imaging: A Phantom Study. Investig. Radiol. 2019, 54, 221–228. [Google Scholar] [CrossRef]
  26. Cattell, R.; Chen, S.; Huang, C. Robustness of radiomic features in magnetic resonance imaging: Review and a phantom study. Vis. Comput. Ind. Biomed. Art 2019, 2, 19. [Google Scholar] [CrossRef]
  27. Rai, R.; Wang, Y.F.; Manton, D.; Dong, B.; Deshpande, S.; Liney, G.P. Development of multi-purpose 3D printed phantoms for MRI. Phys. Med. Biol. 2019, 64, 075010. [Google Scholar] [CrossRef]
  28. Filippou, V.; Tsoumpas, C. Recent advances on the development of phantoms using 3D printing for imaging with CT, MRI, PET, SPECT, and ultrasound. Med. Phys. 2018, 45, e740–e760. [Google Scholar] [CrossRef]
  29. Jin, Z.; Li, Y.; Yu, K.; Liu, L.; Fu, J.; Yao, X.; Zhang, A.; He, Y. 3D Printing of Physical Organ Models: Recent Developments and Challenges. Adv. Sci. 2021, 8, 2101394. [Google Scholar] [CrossRef]
  30. Aimar, A.; Palermo, A.; Innocenti, B. The Role of 3D Printing in Medical Applications: A State of the Art. J. Healthc. Eng. 2019, 2019, 5340616. [Google Scholar] [CrossRef]
  31. Bieniosek, M.F.; Lee, B.J.; Levin, C.S. Technical Note: Characterization of custom 3D printed multimodality imaging phantoms. Med. Phys. 2015, 42, 5913–5918. [Google Scholar] [CrossRef] [PubMed]
  32. Sagan, H. A three-dimensional hilbert curve. Int. J. Math. Educ. Sci. Technol. 1993, 24, 541–545. [Google Scholar] [CrossRef]
  33. Denso Wave. QRcode.com. 2014. Available online: https://www.QRCode.com. (accessed on 1 July 2022).
  34. Tiwari, S. An introduction to QR code technology. In Proceedings of the 2016 15th International Conference on Information Technology, Bhubaneswar, India, 22–24 December 2016. [Google Scholar]
  35. Teramoto, D.; Ushioda, Y.; Sasaki, A.; Sakurai, Y.; Nagahama, H.; Nakamura, M.; Sugimori, H.; Sakata, M. [Can fruits and vegetables be used as substitute phantoms for normal human brain tissues in magnetic resonance imaging?]. Nihon Hoshasen Gijutsu Gakkai Zasshi 2013, 69, 1146–1152. [Google Scholar] [CrossRef]
  36. Werz, K.; Braun, H.; Vitha, D.; Bruno, G.; Martirosian, P.; Steidle, G.; Schick, F. Relaxation times T1, T2, and T2 * of apples, pears, citrus fruits, and potatoes with a comparison to human tissues. Z. Med. Phys. 2011, 21, 206–215. [Google Scholar] [CrossRef] [PubMed]
  37. Berenguer, R.; Pastor-Juan, M.D.R.; Canales-Vázquez, J.; Castro-García, M.; Villas, M.V.; Legorburo, F.M.; Sabater, S. Radiomics of CT features may be nonreproducible and redundant: Influence of CT acquisition parameters. Radiology 2018, 288, 407–415. [Google Scholar] [CrossRef]
  38. Mueller-Lisse, U.G.; Murer, S.; Mueller-Lisse, U.L.; Kuhn, M.; Scheidler, J.; Scherr, M. Everyman’s prostate phantom: Kiwi-fruit substitute for human prostates at magnetic resonance imaging, diffusion-weighted imaging and magnetic resonance spectroscopy. Eur. Radiol. 2017, 27, 3362–3371. [Google Scholar] [CrossRef] [PubMed]
  39. Stupic, K.F.; Ainslie, M.; Boss, M.A.; Charles, C.; Dienstfrey, A.M.; Evelhoch, J.L.; Finn, P.; Gimbutas, Z.; Gunter, J.L.; Hill, D.L.G.; et al. A standard system phantom for magnetic resonance imaging. Magn. Reson. Med. 2021, 86, 1194–1211. [Google Scholar] [CrossRef] [PubMed]
  40. Yang, J.; Peng, H.; Liu, L.; Lu, L. 3D printed perforated QR codes. Comput. Graph. 2019, 81, 117–124. [Google Scholar] [CrossRef]
  41. Papp, G.; Hoffmann, M.; Papp, I. Improved Embedding of QR Codes onto Surfaces to be 3D Printed. CAD Comput. Aided Des. 2021, 131, 102961. [Google Scholar] [CrossRef]
  42. Alber, J.; Niedermeier, R. On multidimensional curves with hilbert property. Theory Comput. Syst. 2000, 33, 295–312. [Google Scholar] [CrossRef]
  43. Garcia, J.; Yang, Z.L.; Mongrain, R.; Leask, R.L.; Lachapelle, K. 3D printing materials and their use in medical education: A review of current technology and trends for the future. BMJ Simul. Technol. Enhanc. Learn. 2017, 4, 27–40. [Google Scholar] [CrossRef] [PubMed]
  44. Mugler, J.P. Optimized three-dimensional fast-spin-echo MRI. J. Magn. Reson. Imaging 2014, 39, 745–767. [Google Scholar] [CrossRef] [PubMed]
  45. Bapst, B.; Amegnizin, J.L.; Vignaud, A.; Kauv, P.; Maraval, A.; Kalsoum, E.; Tuilier, T.; Benaissa, A.; Brugières, P.; Leclerc, X.; et al. Post-contrast 3D T1-weighted TSE MR sequences (SPACE, CUBE, VISTA/BRAINVIEW, isoFSE, 3D MVOX): Technical aspects and clinical applications. J. Neuroradiol. 2020, 47, 358–368. [Google Scholar] [CrossRef] [PubMed]
  46. Nair, G.; Absinta, M.; Reich, D.S. Optimized T1-MPRAGE sequence for better visualization of spinal cord multiple sclerosis lesions at 3T. Am. J. Neuroradiol. 2013, 34, 2215–2222. [Google Scholar] [CrossRef] [PubMed]
  47. Pinter, C.; Lasso, A.; Fichtinger, G. Polymorph segmentation representation for medical image computing. Comput. Methods Programs Biomed. 2019, 171, 19–26. [Google Scholar] [CrossRef]
  48. Fedorov, A.; Beichel, R.; Kalpathy-Cramer, J.; Finet, J.; Fillion-Robin, J.-C.; Pujol, S.; Bauer, C.; Jennings, D.; Fennessy, F.; Sonka, M.; et al. 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magn. Reson. Imaging 2012, 30, 1323–1341. [Google Scholar] [CrossRef]
  49. Parmar, C.; Velazquez, E.R.; Leijenaar, R.; Jermoumi, M.; Carvalho, S.; Mak, R.H.; Mitra, S.; Shankar, B.U.; Kikinis, R.; Haibe-Kains, B.; et al. Robust radiomics feature quantification using semiautomatic volumetric segmentation. PLoS ONE 2014, 9, e102107. [Google Scholar] [CrossRef]
  50. NIfTI: Neuroimaging Informatics Technology Initiative. 2012. Available online: https://nifti.nimh.nih.gov. (accessed on 1 July 2022).
  51. Samuel, S.; Moore, M.; Sheridan, H.; Sorensen, C.; Patterson, B. Touring a Data Curation Network Primer: A Focus on Neuroimaging Data. J. eScience Libr. 2021, 10, 5. [Google Scholar] [CrossRef]
  52. Collewet, G.; Strzelecki, M.; Mariette, F. Influence of MRI acquisition protocols and image intensity normalization methods on texture classification. Magn. Reson. Imaging 2004, 22, 81–91. [Google Scholar] [CrossRef] [PubMed]
  53. Hatt, M.; Vallieres, M.; Visvikis, D.; Zwanenburg, A. IBSI: An international community radiomics standardization initiative. J. Nucl. Med. 2018, 59, 287. [Google Scholar]
  54. Depeursinge, A.; Andrearczyk, V.; Whybra, P.; van Griethuysen, J.; Müller, H.; Schaer, R.; Vallières, M.; Zwanenburg, A. Standardised convolutional filtering for radiomics: Image biomarker standardisation initiative (IBSI). arXiv 2020, arXiv:2006.05470v5. [Google Scholar]
  55. Lacroix, M.; Frouin, F.; Dirand, A.S.; Nioche, C.; Orlhac, F.; Bernaudin, J.F.; Brillet, P.Y.; Buvat, I. Correction for Magnetic Field Inhomogeneities and Normalization of Voxel Values Are Needed to Better Reveal the Potential of MR Radiomic Features in Lung Cancer. Front. Oncol. 2020, 10, 43. [Google Scholar] [CrossRef] [PubMed]
  56. Hu, Z.; Zhuang, Q.; Xiao, Y.; Wu, G.; Shi, Z.; Chen, L.; Wang, Y.; Yu, J. MIL normalization —— prerequisites for accurate MRI radiomics analysis. Comput. Biol. Med. 2021, 133, 104403. [Google Scholar] [CrossRef]
  57. Zwanenburg, A.; Leger, S.; Vallières, M.; Löck, S. Radiomics Digital Phantom. CancerData 2016, 41, 366–373. [Google Scholar]
  58. Orlhac, F.; Soussan, M.; Maisonobe, J.-A.; Garcia, C.A.; Vanderlinden, B.; Buvat, I. Tumor Texture Analysis in 18F-FDG PET: Relationships Between Texture Parameters, Histogram Indices, Standardized Uptake Values, Metabolic Volumes, and Total Lesion Glycolysis. J. Nucl. Med. 2014, 55, 414–422. [Google Scholar] [CrossRef]
  59. Zwanenburg, A.; Vallières, M.; Abdalah, M.A.; Aerts, H.J.W.L.; Andrearczyk, V.; Apte, A.; Ashrafinia, S.; Bakas, S.; Beukinga, R.J.; Boellaard, R.; et al. The image biomarker standardization initiative: Standardized quantitative radiomics for high-throughput image-based phenotyping. Radiology 2020, 295, 328–338. [Google Scholar] [CrossRef]
  60. Schwier, M.; van Griethuysen, J.; Vangel, M.G.; Pieper, S.; Peled, S.; Tempany, C.; Aerts, H.J.W.L.; Kikinis, R.; Fennessy, F.M.; Fedorov, A. Repeatability of Multiparametric Prostate MRI Radiomics Features. Sci. Rep. 2019, 9, 9441. [Google Scholar] [CrossRef]
  61. Carbonell, G.; Kennedy, P.; Bane, O.; Kirmani, A.; El Homsi, M.; Stocker, D.; Said, D.; Mukherjee, P.; Gevaert, O.; Lewis, S.; et al. Precision of MRI radiomics features in the liver and hepatocellular carcinoma. Eur. Radiol. 2021, 32, 2030–2040. [Google Scholar] [CrossRef]
  62. Koo, T.K.; Li, M.Y. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J. Chiropr. Med. 2016, 15, 155–163. [Google Scholar] [CrossRef]
  63. Scalco, E.; Belfatto, A.; Mastropietro, A.; Rancati, T.; Avuzzi, B.; Messina, A.; Valdagni, R.; Rizzo, G. T2w-MRI signal normalization affects radiomics features reproducibility. Med. Phys. 2020, 47, 1680–1691. [Google Scholar] [CrossRef]
  64. Ta, D.; Khan, M.; Ishaque, A.; Seres, P.; Eurich, D.; Yang, Y.H.; Kalra, S. Reliability of 3D texture analysis: A multicenter MRI study of the brain. J. Magn. Reson. Imaging 2020, 51, 1200–1209. [Google Scholar] [CrossRef] [PubMed]
  65. Moradmand, H.; Aghamiri, S.M.R.; Ghaderi, R. Impact of image preprocessing methods on reproducibility of radiomic features in multimodal magnetic resonance imaging in glioblastoma. J. Appl. Clin. Med. Phys. 2020, 21, 179–190. [Google Scholar] [CrossRef] [PubMed]
  66. Fiset, S.; Welch, M.L.; Weiss, J.; Pintilie, M.; Conway, J.L.; Milosevic, M.; Fyles, A.; Traverso, A.; Jaffray, D.; Metser, U.; et al. Repeatability and reproducibility of MRI-based radiomic features in cervical cancer. Radiother. Oncol. 2019, 135, 107–114. [Google Scholar] [CrossRef] [PubMed]
  67. Qian, J.; Xing, B.; Zhang, B.; Yang, H. Optimizing QR code readability for curved agro-food packages using response surface methodology to improve mobile phone-based traceability. Food Packag. Shelf Life 2021, 28, 100638. [Google Scholar] [CrossRef]
  68. Tarjan, L.; Šenk, I.; Tegeltija, S.; Stankovski, S.; Ostojic, G. A readability analysis for QR code application in a traceability system. Comput. Electron. Agric. 2014, 109, 1–11. [Google Scholar] [CrossRef]
  69. Chen, R.; Zheng, Z.; Zhao, H.; Ren, J.; Tan, H. Fast Blind Restoration of QR Code Images Based on Blurred Imaging Mechanism. Guangzi Xuebao/Acta Photonica Sin. 2021, 50, 91. [Google Scholar] [CrossRef]
  70. Kim, Y.; Cho, K.-O.; Han, J.; Cho, M. Three-dimensional QR Code Using Integral Imaging. J. Korea Inst. Inf. Commun. Eng. 2016, 20, 2363–2369. [Google Scholar] [CrossRef][Green Version]
  71. Calabrese, A.; Santucci, D.; Landi, R.; Beomonte Zobel, B.; Faiella, E.; de Felice, C. Radiomics MRI for lymph node status prediction in breast cancer patients: The state of art. J. Cancer Res. Clin. Oncol. 2021, 147, 1587–1597. [Google Scholar] [CrossRef]
  72. Hassani, C.; Saremi, F.; Varghese, B.A.; Duddalwar, V. Myocardial radiomics in cardiac MRI. Am. J. Roentgenol. 2020, 214, 536–545. [Google Scholar] [CrossRef]
  73. Lohmann, P.; Kocher, M.; Ruge, M.I.; Visser-Vandewalle, V.; Shah, N.J.; Fink, G.R.; Langen, K.J.; Galldiks, N. PET/MRI Radiomics in Patients With Brain Metastases. Front. Neurol. 2020, 11, 1. [Google Scholar] [CrossRef]
  74. Sun, Y.; Reynolds, H.M.; Parameswaran, B.; Wraith, D.; Finnegan, M.E.; Williams, S.; Haworth, A. Multiparametric MRI and radiomics in prostate cancer: A review. Australas. Phys. Eng. Sci. Med. 2019, 42, 3–25. [Google Scholar] [CrossRef] [PubMed]
  75. Cutaia, G.; la Tona, G.; Comelli, A.; Vernuccio, F.; Agnello, F.; Gagliardo, C.; Salvaggio, L.; Quartuccio, N.; Sturiale, L.; Stefano, A.; et al. Radiomics and prostate MRI: Current role and future applications. J. Imaging 2021, 7, 34. [Google Scholar] [CrossRef] [PubMed]
  76. Ye, D.M.; Wang, H.T.; Yu, T. The application of radiomics in breast MRI: A review. Technol. Cancer Res. Treat. 2020, 19, 1–16. [Google Scholar] [CrossRef] [PubMed]
  77. Schick, U.; Lucia, F.; Dissaux, G.; Visvikis, D.; Badic, B.; Masson, I.; Pradier, O.; Bourbonne, V.; Hatt, M. MRI-derived radiomics: Methodology and clinical applications in the field of pelvic oncology. Br. J. Radiol. 2019, 92, 20190105. [Google Scholar] [CrossRef] [PubMed]
  78. Caruso, D.; Polici, M.; Zerunian, M.; Pucciarelli, F.; Guido, G.; Polidori, T.; Landolfi, F.; Nicolai, M.; Lucertini, E.; Tarallo, M.; et al. Radiomics in oncology, part 1: Technical principles and gastrointestinal application in ct and mri. Cancers 2021, 13, 2522. [Google Scholar] [CrossRef] [PubMed]
  79. Bartoli, M.; Barat, M.; Dohan, A.; Gaujoux, S.; Coriat, R.; Hoeffel, C.; Cassinotto, C.; Chassagnon, G.; Soyer, P. CT and MRI of pancreatic tumors: An update in the era of radiomics. Jpn. J. Radiol. 2020, 38, 1111–1124. [Google Scholar] [CrossRef]
  80. Feng, Q.; Ding, Z. MRI Radiomics Classification and Prediction in Alzheimer’s Disease and Mild Cognitive Impairment: A Review. Curr. Alzheimer Res. 2020, 17, 297–309. [Google Scholar] [CrossRef]
  81. Cusumano, D.; Meijer, G.; Lenkowicz, J.; Chiloiro, G.; Boldrini, L.; Masciocchi, C.; Dinapoli, N.; Gatta, R.; Casà, C.; Damiani, A.; et al. A field strength independent MR radiomics model to predict pathological complete response in locally advanced rectal cancer. Radiol. Medica 2021, 126, 421–429. [Google Scholar] [CrossRef]
  82. Doran, S.J.; Kumar, S.; Orton, M.; d’Arcy, J.; Kwaks, F.; O’Flynn, E.; Ahmed, Z.; Downey, K.; Dowsett, M.; Turner, N.; et al. “Real-world” radiomics from multi-vendor MRI: An original retrospective study on the prediction of nodal status and disease survival in breast cancer, as an exemplar to promote discussion of the wider issues. Cancer Imaging 2021, 21. [Google Scholar] [CrossRef] [PubMed]
  83. Xu, Y.; He, X.; Li, Y.; Pang, P.; Shu, Z.; Gong, X. The Nomogram of MRI-based Radiomics with Complementary Visual Features by Machine Learning Improves Stratification of Glioblastoma Patients: A Multicenter Study. J. Magn. Reson. Imaging 2021, 54, 37. [Google Scholar] [CrossRef]
  84. Alvarez-Jimenez, C.; Antunes, J.T.; Talasila, N.; Bera, K.; Brady, J.T.; Gollamudi, J.; Marderstein, E.; Kalady, M.F.; Purysko, A.; Willis, J.E.; et al. Radiomic texture and shape descriptors of the rectal environment on post-chemoradiation T2-weighted MRI are associated with pathologic tumor stage regression in rectal cancers: A retrospective, multi-institution study. Cancers 2020, 12, 2027. [Google Scholar] [CrossRef]
  85. Masokano, I.B.; Liu, W.; Xie, S.; Marcellin, D.F.H.; Pei, Y.; Li, W. The application of texture quantification in hepatocellular carcinoma using CT and MRI: A review of perspectives and challenges. Cancer Imaging 2020, 20, 67. [Google Scholar] [CrossRef] [PubMed]
  86. Leandrou, S.; Lamnisos, D.; Kyriacou, P.A.; Constanti, S.; Pattichis, C.S. Comparison of 1.5 T and 3 T MRI hippocampus texture features in the assessment of Alzheimer’s disease. Biomed. Signal Process. Control 2020, 62, 102098. [Google Scholar] [CrossRef]
  87. Srivastava, R.K.; Talluri, S.; Beebi, S.K.; Rajesh Kumar, B. Magnetic Resonance Imaging for Quality Evaluation of Fruits: A Review. Food Anal. Methods 2018, 11, 2943–2960. [Google Scholar] [CrossRef]
  88. Lee, J.; Steinmann, A.; Ding, Y.; Lee, H.; Owens, C.; Wang, J.; Yang, J.; Followill, D.; Ger, R.; MacKin, D.; et al. Radiomics feature robustness as measured using an MRI phantom. Sci. Rep. 2021, 11, 3973. [Google Scholar] [CrossRef]
  89. Mayerhoefer, M.E.; Materka, A.; Langs, G.; Häggström, I.; Szczypiński, P.; Gibbs, P.; Cook, G. Introduction to Radiomics. J. Nucl. Med. 2020, 61, 488–495. [Google Scholar] [CrossRef] [PubMed]
  90. Rizzo, S.; Botta, F.; Raimondi, S.; Origgi, D.; Fanciullo, C.; Morganti, A.G.; Bellomi, M. Radiomics: The facts and the challenges of image analysis. Eur. Radiol. Exp. 2018, 2, 36. [Google Scholar] [CrossRef]
  91. Haralick, R.M.; Dinstein, I.; Shanmugam, K. Textural Features for Image Classification. IEEE Trans. Syst. Man Cybern. 1973, SMC-3, 610–621. [Google Scholar] [CrossRef]
  92. Tian, Z.; Chen, C.; Fan, Y.; Ou, X.; Wang, J.; Ma, X.; Xu, J. Glioblastoma and Anaplastic Astrocytoma: Differentiation Using MRI Texture Analysis. Front. Oncol. 2019, 9, 1–7. [Google Scholar] [CrossRef]
  93. Zhang, B.; Tian, J.; Dong, D.; Gu, D.; Dong, Y.; Zhang, L.; Lian, Z.; Liu, J.; Luo, X.; Pei, S.; et al. Radiomics features of multiparametric MRI as novel prognostic factors in advanced nasopharyngeal carcinoma. Clin. Cancer Res. 2017, 23, 4259–4269. [Google Scholar] [CrossRef] [PubMed]
  94. Loizou, C.P.; Pantziaris, M.; Seimenis, I.; Pattichis, C.S. Brain MR image normalization in texture analysis of multiple sclerosis. In Proceedings of the the Final Program and Abstract Book—9th International Conference on Information Technology and Applications in Biomedicine, Larnaka, Cyprus, 4–7 November 2009. [Google Scholar]
  95. Sun, X.; Shi, L.; Luo, Y.; Yang, W.; Li, H.; Liang, P.; Li, K.; Mok, V.C.T.; Chu, W.C.W.; Wang, D. Histogram-based normalization technique on human brain magnetic resonance images from different acquisitions. Biomed. Eng. Online 2015, 14, 73. [Google Scholar] [CrossRef] [PubMed]
  96. Shinohara, R.T.; Sweeney, E.M.; Goldsmith, J.; Shiee, N.; Mateen, F.J.; Calabresi, P.A.; Jarso, S.; Pham, D.L.; Reich, D.S.; Crainiceanu, C.M. Statistical normalization techniques for magnetic resonance imaging. NeuroImage Clin. 2014, 6, 9–19. [Google Scholar] [CrossRef]
  97. Cernadas, E.; Fernández-Delgado, M.; González-Rufino, E.; Carrión, P. Influence of normalization and color space to color texture classification. Pattern Recognit. 2017, 61, 120–138. [Google Scholar] [CrossRef]
  98. Sollini, M.; Cozzi, L.; Antunovic, L.; Chiti, A.; Kirienko, M. PET Radiomics in NSCLC: State of the art and a proposal for harmonization of methodology. Sci. Rep. 2017, 7, 358. [Google Scholar] [CrossRef] [PubMed]
  99. Yan, Q.; Yi, Y.; Shen, J.; Shan, F.; Zhang, Z.; Yang, G.; Shi, Y. Preliminary study of 3 T-MRI native T1-mapping radiomics in differential diagnosis of non-calcified solid pulmonary nodules/masses. Cancer Cell Int. 2021, 20, 346. [Google Scholar] [CrossRef] [PubMed]
  100. Ahn, S.J.; Kwon, H.; Yang, J.J.; Park, M.; Cha, Y.J.; Suh, S.H.; Lee, J.M. Contrast-enhanced T1-weighted image radiomics of brain metastases may predict EGFR mutation status in primary lung cancer. Sci. Rep. 2020, 10, 8905. [Google Scholar] [CrossRef]
  101. Carré, A.; Klausner, G.; Edjlali, M.; Lerousseau, M.; Briend-Diop, J.; Sun, R.; Ammari, S.; Reuzé, S.; Andres, E.A.; Estienne, T.; et al. Standardization of brain MR images across machines and protocols: Bridging the gap for MRI-based radiomics. Sci. Rep. 2020, 10, 1–15. [Google Scholar] [CrossRef]
  102. Xiao, D.D.; Yan, P.F.; Wang, Y.X.; Osman, M.S.; Zhao, H.Y. Glioblastoma and primary central nervous system lymphoma: Preoperative differentiation by using MRI-based 3D texture analysis. Clin. Neurol. Neurosurg. 2018, 173, 1–26. [Google Scholar] [CrossRef]
  103. Forgacs, A.; Kallos-Balogh, P.; Nagy, F.; Krizsan, A.K.; Garai, I.; Tron, L.; Dahlbom, M.; Balkay, L. Activity painting: PET images of freely defined activity distributions applying a novel phantom technique. PLoS ONE 2019, 14, e0207658. [Google Scholar] [CrossRef]
  104. Jha, A.K.; Mithun, S.; Jaiswar, V.; Sherkhane, U.B.; Purandare, N.C.; Prabhash, K.; Rangarajan, V.; Dekker, A.; Wee, L.; Traverso, A. Repeatability and reproducibility study of radiomic features on a phantom and human cohort. Sci. Rep. 2021, 11, 2055. [Google Scholar] [CrossRef]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Article Metrics

Citations

Article Access Statistics

Journal Statistics
Multiple requests from the same IP address are counted as one view.
Download PDF
PRAME Immunoexpression in 275 Cutaneous Melanocytic Lesions: A Double Institutional Experience
Previous
Elucidation of Ischemic Mechanisms of Early Pressure Injury during Post-Decompression and Detecting Methods
Next