Comparative Analysis of Computer-Aided Diagnosis and Computer-Assisted Subjective Assessment in Thyroid Ultrasound
Abstract
:1. Introduction
2. Materials and Methods
2.1. Study Type and Data Sources
2.2. Image Selection Criteria
2.3. Analyses of the Thyroid Nodule Images
2.3.1. Computer-Assisted Subjective Risk Assessment
2.3.2. CAD Assessment
2.4. Data Analysis and Statistical Analysis
3. Results
3.1. Demographics and Thyroid Nodule Characteristics
3.2. Nodule Sonographic Feature Classifications by Human Subjective Assessment and CAD
3.3. Classification Correlation Comparisons between Subjective Ratings and CAD
3.4. Rater Agreement Based on TIRADS
3.5. Diagnostic Performance Assessment of CAD and Computer-Assisted Raters for Matched TIRADS
4. Discussion
4.1. Interpretation of the Study Findings for Sonographic Feature Ratings between the CAD and Computer-Assisted Approaches
4.2. Interpretation of the Study’s Diagnostic Performance Outcomes
4.3. Meaning of the Study and Implications
4.4. Limitations and Directions for Future Research
Supplementary Materials
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Vaccarella, S.; Franceschi, S.; Bray, F.; Wild, C.P.; Plummer, M.; Maso, L.D. Worldwide Thyroid-Cancer Epidemic? The Increasing Impact of Overdiagnosis. N. Engl. J. Med. 2016, 375, 614–617. [Google Scholar] [CrossRef]
- Schnadig, V.J. Overdiagnosis of Thyroid Cancer: Is This Not an Ethical Issue for Pathologists as Well as Radiologists and Clinicians? Arch. Pathol. Lab. Med. 2018, 142, 1018–1020. [Google Scholar] [CrossRef]
- Gharib, H.; Papini, E.; Garber, J.R.; Duick, D.S.; Harrell, R.M.; Hegedus, L.; Paschke, R.; Valcavi, R.; Vitti, P. American Association of Clinical Endocrinologists, American College of Endocrinology, and Associazione Medici Endocrinologi Medical Guidelines for Clinical Practice for the Diagnosis and Management of Thyroid Nodules - 2016 Update Appendix. Endocr. Pr. 2016, 22, 1–60. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Haugen, B.R.; Alexander, E.K.; Bible, K.C.; Doherty, G.M.; Mandel, S.J.; Nikiforov, Y.E.; Pacini, F.; Randolph, G.W.; Sawka, A.M.; Schlumberger, M.; et al. 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid 2016, 26, 1–133. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Shin, J.H.; Baek, J.H.; Chung, J.; Ha, E.J.; Kim, J.-H.; Lee, Y.H.; Lim, H.K.; Moon, W.-J.; Na, D.G.; Park, J.S.; et al. Ultrasonography Diagnosis and Imaging-Based Management of Thyroid Nodules: Revised Korean Society of Thyroid Radiology Consensus Statement and Recommendations. Korean J. Radiol. 2016, 17, 370–395. [Google Scholar] [CrossRef] [Green Version]
- Russ, G.; Bonnema, S.J.; Erdogan, M.F.; Durante, C.; Ngu, R.; Leenhardt, L. European Thyroid Association Guidelines for Ultrasound Malignancy Risk Stratification of Thyroid Nodules in Adults: The EU-TIRADS. Eur. Thyroid. J. 2017, 6, 225–237. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Yoon, J.H.; Lee, H.S.; Kim, E.-K.; Moon, H.J.; Park, V.Y.; Kwak, J.Y. Pattern-based vs. score-based guidelines using ultrasound features have different strengths in risk stratification of thyroid nodules. Eur. Radiol. 2020, 30, 3793–3802. [Google Scholar] [CrossRef]
- Choi, Y.J.; Baek, J.H.; Baek, S.H.; Shim, W.H.; Lee, K.D.; Lee, H.S.; Shong, Y.K.; Ha, E.J.; Lee, J.H. Web-Based Malignancy Risk Estimation for Thyroid Nodules Using Ultrasonography Characteristics: Development and Validation of a Predictive Model. Thyroid 2015, 25, 1306–1312. [Google Scholar] [CrossRef]
- Ha, S.M.; Ahn, H.S.; Baek, J.H.; Ahn, H.Y.; Chung, Y.J.; Cho, B.Y.; Bin Park, S. Validation of Three Scoring Risk-Stratification Models for Thyroid Nodules. Thyroid 2017, 27, 1550–1557. [Google Scholar] [CrossRef]
- Li, T.; Jiang, Z.; Lu, M.; Zou, S.; Wu, M.; Wei, T.; Wang, L.; Li, J.; Hu, Z.; Cheng, X.; et al. Computer-aided diagnosis system of thyroid nodules ultrasonography: Diagnostic performance difference between computer-aided diagnosis and 111 radiologists. Medicine 2020, 99, e20634. [Google Scholar] [CrossRef] [PubMed]
- Ye, F.-Y.; Lyu, G.-R.; Li, S.-Q.; You, J.-H.; Wang, K.-J.; Cai, M.-L.; Su, Q.-C. Diagnostic Performance of Ultrasound Computer-Aided Diagnosis Software Compared with That of Radiologists with Different Levels of Expertise for Thyroid Malignancy: A Multicenter Prospective Study. Ultrasound Med. Biol. 2021, 47, 114–124. [Google Scholar] [CrossRef]
- Ha, E.J.; Baek, J.H. Applications of machine learning and deep learning to thyroid imaging: Where do we stand? Ultrasonography 2021, 40, 23–29. [Google Scholar] [CrossRef]
- Moon, H.J.; Kwak, J.Y.; Kim, E.-K.; Kim, M.J. A Taller-Than-Wide Shape in Thyroid Nodules in Transverse and Longitudinal Ultrasonographic Planes and the Prediction of Malignancy. Thyroid. 2011, 21, 1249–1253. [Google Scholar] [CrossRef]
- Grani, G.; Lamartina, L.; Ramundo, V.; Falcone, R.; Lomonaco, C.; Ciotti, L.; Barone, M.; Maranghi, M.; Cantisani, V.; Filetti, S.; et al. Taller-Than-Wide Shape: A New Definition Improves the Specificity of TIRADS Systems. Eur. Thyroid. J. 2020, 9, 85–91. [Google Scholar] [CrossRef]
- Hinkle, D.E.; Wiersma, W.; Jurs, S.G. Correlation: A measure of relationship. In Applied Statistics for the Behavioral Sciences, 5th ed.; Houghton Mifflin: Boston, MA, USA, 2003. [Google Scholar]
- Landis, J.R.; Koch, G.G. The Measurement of Observer Agreement for Categorical Data. Biometrics 1977, 33, 159–174. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Thomas, J.; Ledger, G.A.; Mamillapalli, C.K. Use of artificial intelligence and machine learning for estimating malignancy risk of thyroid nodules. Curr. Opin. Endocrinol. Diabetes Obes. 2020, 27, 345–350. [Google Scholar] [CrossRef]
- Park, S.H. Artificial intelligence for ultrasonography: Unique opportunities and challenges. Ultrasonography 2021, 40, 3–6. [Google Scholar] [CrossRef]
- Xu, L.; Gao, J.; Wang, Q.; Yin, J.; Yu, P.; Bai, B.; Pei, R.; Chen, D.; Yang, G.; Wang, S.; et al. Computer-Aided Diagnosis Systems in Diagnosing Malignant Thyroid Nodules on Ultrasonography: A Systematic Review and Meta-Analysis. Eur. Thyroid. J. 2020, 9, 186–193. [Google Scholar] [CrossRef]
- Han, M.; Ha, E.; Park, J. Computer-Aided Diagnostic System for Thyroid Nodules on Ultrasonography: Diagnostic Performance Based on the Thyroid Imaging Reporting and Data System Classification and Dichotomous Outcomes. Am. J. Neuroradiol. 2021, 42, 559–565. [Google Scholar] [CrossRef]
- Xia, S.; Yao, J.; Zhou, W.; Dong, Y.; Xu, S.; Zhou, J.; Zhan, W. A computer-aided diagnosing system in the evaluation of thyroid nodules—experience in a specialized thyroid center. World J. Surg. Oncol. 2019, 17, 1–8. [Google Scholar] [CrossRef] [PubMed]
- Kim, H.L.; Ha, E.J.; Han, M. Real-World Performance of Computer-Aided Diagnosis System for Thyroid Nodules Using Ultrasonography. Ultrasound Med. Biol. 2019, 45, 2672–2678. [Google Scholar] [CrossRef] [PubMed]
- Jeong, E.Y.; Kim, H.L.; Ha, E.J.; Park, S.Y.; Cho, Y.J.; Han, M. Computer-aided diagnosis system for thyroid nodules on ultrasonography: Diagnostic performance and reproducibility based on the experience level of operators. Eur. Radiol. 2019, 29, 1978–1985. [Google Scholar] [CrossRef]
- Reverter, J.L.; Puig-Jove, C.; Vazquez, F.; Puig-Domingo, M. Diagnostic performance evaluation of a computer-assisted imaging analysis system for ultrasound risk stratification of thyroid nodules. Endocr. Abstr. 2019, 10, 1–6. [Google Scholar] [CrossRef] [PubMed]
- Zhao, W.-J.; Fu, L.-R.; Huang, Z.-M.; Zhu, J.-Q.; Ma, B.-Y. Effectiveness evaluation of computer-aided diagnosis system for the diagnosis of thyroid nodules on ultrasound. Medicine 2019, 98, e16379. [Google Scholar] [CrossRef] [PubMed]
- Fresilli, D.; Grani, G.; De Pascali, M.L.; Alagna, G.; Tassone, E.; Ramundo, V.; Ascoli, V.; Bosco, D.; Biffoni, M.; Bononi, M.; et al. Computer-aided diagnostic system for thyroid nodule sonographic evaluation outperforms the specificity of less experienced examiners. J. Ultrasound 2020, 23, 169–174. [Google Scholar] [CrossRef]
- Glas, A.S.; Lijmer, J.G.; Prins, M.H.; Bonsel, G.J.; Bossuyt, P.M.M. The diagnostic odds ratio: A single indicator of test performance. J. Clin. Epidemiol. 2003, 56, 1129–1135. [Google Scholar] [CrossRef]
- Eusebi, P. Diagnostic Accuracy Measures. Cerebrovasc. Dis. 2013, 36, 267–272. [Google Scholar] [CrossRef]
Sonographic | R1 | R2 | CAD | p-Values | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Feature | M = 62 | B = 100 | T = 162 | M = 62 | B = 100 | T= 162 | M = 62 | B = 100 | T = 162 | R1 | R2 | CAD |
Echogenicity | ||||||||||||
Isoechoic | 9 (14.5%) | 35 (35%) | 44 (27.2%) | 12 (19.4%) | 45 (45%) | 57 (35.2%) | 11 (17.7%) | 27 (27%) | 38 (23.5%) | 0.006 | <0.001 | 0.13 |
Hyperechoic | 5 (8.1%) | 13 (13%) | 18 (11.1%) | 0 (0%) | 5 (5%) | 5 (3.1%) | 13 (21%) | 31 (31%) | 44 (27.2%) | |||
Hypoechoic | 41 (66.1%) | 48 (48%) | 89 (54.9%) | 45 (72.6%) | 49 (49%) | 94 (58%) | 30 (48.4%) | 34 (34%) | 64 (39.5%) | |||
M-hypoechoic | 7 (11.3%) | 4 (4%) | 11 (6.8%) | 5 (8.1%) | 1 (1%) | 6 (3.7%) | 8 (12.9%) | 8 (8%) | 16 (9.9%) | |||
Calcifications | ||||||||||||
None | 18 (29%) | 61 (61%) | 79 (48.8%) | 28 (45.2%) | 78 (78%) | 106 (65.4%) | 27 (43.5%) | 71 (71%) | 98 (60.5%) | <0.001 | <0.001 | 0.001 |
Macro-calc | 2 (3.2%) | 10 (10%) | 12 (7.4%) | 2 (3.2%) | 2 (2%) | 4 (2.5%) | 0 (0%) | 0 (0%) | 0 (0%) | |||
Micro-calc | 29 (46.8%) | 25 (25%) | 54 (33.3%) | 28 (45.2%) | 19 (19%) | 47 (29%) | 32 (51.6%) | 23 (23%) | 55 (34%) | |||
Rim calc | 1 (1.6%) | 1 (1%) | 2 (1.2%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | |||
Mixed calc | 12 (19.4%) | 3 (3%) | 15 (9.3%) | 4 (6.5%) | 1 (1%) | 5 (3.1%) | 3 (4.8%) | 6 (6%) | 9 (5.6%) | |||
Margins | ||||||||||||
Well-defined | 26 (41.9%) | 71 (71%) | 97 (59.9%) | 31 (50%) | 70 (70%) | 101 (62.3%) | 42 (67.7%) | 80 (80%) | 122 (75.3%) | 0.001 | 0.001 | 0.093 |
Irregular | 32 (51.6%) | 26 (26%) | 58 (35.8%) | 18 (29%) | 26 (26%) | 44 (27.2%) | 20 (32.3%) | 20 (20%) | 40 (24.7%) | |||
Microlobulated | 0 (0%) | 0 (0%) | 0 (0%) | 1 (1.6%) | 2 (2%) | 3 (1.9%) | 0 (0%) | 0 (0%) | 0 (0%) | |||
Spiculated | 4 (6.5%) | 3 (3%) | 7 (4.3%) | 12 (19.4%) | 2 (2%) | 14 (8.6%) | 0 (0%) | 0 (0%) | 0 (0%) | |||
Composition | ||||||||||||
Spongiform | 3 (4.8%) | 15 (15%) | 18 (11.1%) | 1 (1.6%) | 13 (13%) | 14 (8.6%) | 0 (0%) | 1 (1%) | 1 (0.6%) | 0.005 | 0.014 | 0.9 |
Pred. cystic | 0 (0%) | 7 (7%) | 7 (4.3%) | 1 (1.6%) | 7 (7%) | 8 (4.9%) | 0 (0%) | 1 (1%) | 1 (0.6%) | |||
Pred. solid | 10 (16.1%) | 23 (23%) | 33 (20.4%) | 34 (54.8%) | 51 (51%) | 85 (52.5%) | 24 (38.7%) | 35 (35%) | 59 (36.4%) | |||
Solid | 49 (79%) | 55 (55%) | 104 (64.2%) | 26 (41.9%) | 29 (29%) | 55 (34%) | 38 (61.3%) | 63 (63%) | 101 (62.3%) | |||
Shape | ||||||||||||
Round/Ovoid | 25 (40.3%) | 74 (74%) | 99 (61.1%) | 28 (45.2%) | 75 (75%) | 103 (63.6%) | 49 (79%) | 90 (90%) | 139 (85.8%) | <0.001 | <0.001 | 0.057 |
Taller than wide | 28 (45.2%) | 20 (20%) | 48 (29.6%) | 27 (43.5%) | 21 (21%) | 48 (29.6%) | 13 (21%) | 9 (9%) | 22 (13.6%) | |||
Irregular | 9 (14.5%) | 6 (6%) | 15 (9.3%) | 7 (11.3%) | 4 (4%) | 11 (6.8%) | 0 (0%) | 1 (1%) | 1 (0.6%) |
Sonographic Feature | R1 vs. CAD | R2 vs. CAD | R1 vs. R2 | ||||||
---|---|---|---|---|---|---|---|---|---|
M | B | T | M | B | T | M | B | T | |
Echogenicity | 0.74 | 0.55 | 0.64 | 0.73 | 0.63 | 0.67 | 0.91 | 0.64 | 0.78 |
Calcifications | 0.30 | 0.45 | 0.49 | 0.33 | 0.26 | 0.41 | 0.84 | 0.78 | 0.85 |
Margins | 0.39 | 0.14 | 0.33 | 0.37 | 0.22 | 0.35 | 0.03 | 0.63 | 0.45 |
Composition | −0.25 | 0.46 | 0.28 | 0.15 | 0.60 | 0.46 | 0.81 | 0.79 | 0.83 |
Shape | 0.65 | 0.81 | 0.76 | 0.40 | 0.86 | 0.71 | 0.41 | 0.85 | 0.72 |
Raters | Nodules | TIRADS | |||
---|---|---|---|---|---|
AACE | ATA | EU | KSThR | ||
R1 vs. CAD | M | 0.12 | 0.69 | 0.23 | 0.43 |
B | 0.32 | 0.68 | 0.40 | 0.46 | |
ALL | 0.35 | 0.75 | 0.46 | 0.53 | |
R2 vs. CAD | M | 0.18 | 0.57 | 0.45 | 0.45 |
B | 0.12 | 0.40 | 0.24 | 0.23 | |
ALL | 0.21 | 0.56 | 0.38 | 0.37 | |
R1 vs. R2 | M | 0.52 | 0.71 | 0.40 | 0.50 |
B | 0.60 | 0.71 | 0.57 | 0.43 | |
ALL | 0.65 | 0.77 | 0.59 | 0.54 |
Rater by TIRADS | N | SEN % (CI) | SPE (%) (CI) | PLR (CI) | NLR (CI) | DOR (CI) | AUROC (CI) |
---|---|---|---|---|---|---|---|
EU-CAD | 162 | 79.0 (66.8; 88.3) | 55.0 (44.7; 65.0) | 1.76 (1.37; 2.26) | 0.38 (0.23; 0.64) | 4.61 (2.23; 9.53) | 79.0 (66.8; 88.3) |
EU-R1 | 162 | 85.5 (74.2; 93.1) | 62.0 (51.8; 71.5) | 2.25 (1.72; 2.95) | 0.23 (0.12; 0.42) | 9.61 (4.26; 21.68) | 85.5 (74.2; 93.1) |
EU-R2 | 162 | 71.0 (58.1; 81.8) | 64.0 (53.8; 73.4) | 1.97 (1.45; 2.68) | 0.45 (0.30; 0.69) | 4.35 (2.16; 8.37) | 71.0 (58.1; 81.8) |
KSThR-CAD | 162 | 83.9 (72.3; 92.0) | 46.0 (36.0; 56.3) | 1.55 (1.26; 1.92) | 0.35 (0.19; 0.64) | 4.43 (2.03; 9.69) | 83.9 (72.3; 92.0) |
KSThR-R1 | 162 | 90.3 (80.1; 96.4) | 51.0 (40.8; 61.4) | 1.84 (1.49; 2.29) | 0.19 (0.09; 0.42) | 9.71 (3.84; 24.59) | 90.3 (80.1; 96.4) |
KSThR-R2 | 162 | 75.8 (63.3; 85.8) | 61.0 (50.7; 70.6) | 1.94 (1.47; 2.58) | 0.40 (0.25; 0.63) | 4.90 (2.42; 9.93) | 75.8 (63.3; 85.8) |
AACE-CAD | 114 | 92.5 (81.8; 97.9) | 26.2 (15.8; 39.1) | 1.25 (1.06; 1.48) | 0.29 (0.10; 0.81) | 4.36 (1.35; 14.01) | 92.5 (81.8; 97.9) |
AACE-R1 | 114 | 88.7 (77.0; 95.7) | 54.1 (40.9; 66.9) | 1.93 (1.45; 2.58) | 0.21 (0.10; 0.46) | 9.23 (3.44; 24.79) | 88.7 (77.0; 95.7) |
AACE-R2 | 114 | 79.3 (65.9; 89.2) | 62.3 (49.0; 74.4) | 2.10 (1.48; 2.98) | 0.33 (0.19; 0.58) | 6.31 (2.72; 14.64) | 79.3 (65.9; 89.2) |
ATA-CAD | 96 | 79.5 (63.5; 90.7) | 66.7 (52.9; 78.6) | 2.38 (1.60; 3.56) | 0.31 (0.16; 0.59) | 7.75 (2.99; 20.09) | 79.5 (63.5; 90.7) |
ATA-R1 | 96 | 79.5 (63.5; 90.7) | 70.2 (56.6; 81.6) | 2.67 (1.70; 4.19) | 0.29 (0.15; 0.55) | 9.12 (3.48; 23.87) | 79.5 (63.5; 90.7) |
ATA-R2 | 96 | 74.4 (57.9; 87.0) | 68.4 (54.8; 80.1) | 2.35 (1.54; 3.60) | 0.37 (0.21; 0.66) | 6.28 (2.53; 15.61) | 74.4 (57.9; 87.0) |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chambara, N.; Liu, S.Y.W.; Lo, X.; Ying, M. Comparative Analysis of Computer-Aided Diagnosis and Computer-Assisted Subjective Assessment in Thyroid Ultrasound. Life 2021, 11, 1148. https://doi.org/10.3390/life11111148
Chambara N, Liu SYW, Lo X, Ying M. Comparative Analysis of Computer-Aided Diagnosis and Computer-Assisted Subjective Assessment in Thyroid Ultrasound. Life. 2021; 11(11):1148. https://doi.org/10.3390/life11111148
Chicago/Turabian StyleChambara, Nonhlanhla, Shirley Yuk Wah Liu, Xina Lo, and Michael Ying. 2021. "Comparative Analysis of Computer-Aided Diagnosis and Computer-Assisted Subjective Assessment in Thyroid Ultrasound" Life 11, no. 11: 1148. https://doi.org/10.3390/life11111148