Deep Learning Enhances Radiologists’ Detection of Potential Spinal Malignancies in CT Scans
Abstract
:Featured Application
Abstract
1. Introduction
2. Materials and Methods
2.1. Data Acquisition
2.2. Establishment of the Ground Truth
2.3. AI Algorithm Development
2.4. Experimental Setup
2.5. Statistical Analysis
3. Results
3.1. Demographics of the Dataset and Spinal Lesion Assessment in the Reference Standard
3.2. Algorithm Performance
3.3. Intra-Observer Agreement without and with the Aid of the DLA
4. Discussion
5. Conclusions
Supplementary Materials
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Waite, S.; Scott, J.; Gale, B.; Fuchs, T.; Kolla, S.; Reede, D. Interpretive Error in Radiology. AJR Am. J. Roentgenol. 2017, 208, 739–749. [Google Scholar] [CrossRef]
- Fitzgerald, R. Error in radiology. Clin. Radiol. 2001, 56, 938–946. [Google Scholar] [CrossRef]
- McCreadie, G.; Oliver, T.B. Eight CT lessons that we learned the hard way: An analysis of current patterns of radiological error and discrepancy with particular emphasis on CT. Clin. Radiol. 2009, 64, 491–499; discussion 500-1. [Google Scholar] [CrossRef]
- Terreblanche, O.D.; Andronikou, S.; Hlabangana, L.T.; Brown, T.; Boshoff, P.E. Should registrars be reporting after-hours CT scans? A calculation of error rate and the influencing factors in South Africa. Acta Radiol. 2012, 53, 61–68. [Google Scholar] [CrossRef] [PubMed]
- Ziu, E.; Viswanathan, V.K.; Mesfin, F.B. Spinal Metastasis; StatPearls Publishing: Treasure Island, FL, USA, 2022. [Google Scholar]
- Rodallec, M.H.; Feydy, A.; Larousserie, F.; Anract, P.; Campagna, R.; Babinet, A.; Zins, M.; Drapé, J.-L. Diagnostic Imaging of Solitary Tumors of the Spine: What to Do and Say. Radiographics 2008, 28, 1019–1041. [Google Scholar] [CrossRef] [Green Version]
- Kim, Y.S.; Han, I.H.; Lee, I.S.; Lee, J.S.; Choi, B.K. Imaging findings of solitary spinal bony lesions and the differential diagnosis of benign and malignant lesions. J. Korean Neurosurg. Soc. 2012, 52, 126–132. [Google Scholar] [CrossRef]
- Nguyen, T.T.; Thelen, J.C.; Bhatt, A.A. Bone up on spinal osseous lesions: A case review series. Insights Imaging 2020, 11, 80. [Google Scholar] [CrossRef] [PubMed]
- Coleman, R.E. Metastatic bone disease: Clinical features, pathophysiology and treatment strategies. Cancer Treat. Rev. 2001, 27, 165–176. [Google Scholar] [CrossRef] [PubMed]
- Martin, M.; Bell, R.; Bourgeois, H.; Brufsky, A.; Diel, I.; Eniu, A.; Fallowfield, L.; Fujiwara, Y.; Jassem, J.; Paterson, A.H.; et al. Bone-related complications and quality of life in advanced breast cancer: Results from a randomized phase III trial of denosumab versus zoledronic acid. Clin. Cancer Res. 2012, 18, 4841–4849. [Google Scholar] [CrossRef] [Green Version]
- Jarvik, J.G.; Deyo, R.A. Diagnostic evaluation of low back pain with emphasis on imaging. Ann. Intern. Med. 2002, 137, 586–597. [Google Scholar] [CrossRef]
- Wang, J.; Fang, Z.; Lang, N.; Yuan, H.; Su, M.Y.; Baldi, P. A multi-resolution approach for spinal metastasis detection using deep Siamese neural networks. Comput. Biol. Med. 2017, 84, 137–146. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Noguchi, S.; Nishio, M.; Sakamoto, R.; Yakami, M.; Fujimoto, K.; Emoto, Y.; Kubo, T.; Iizuka, Y.; Nakagomi, K.; Miyasa, K.; et al. Deep learning–based algorithm improved radiologists’ performance in bone metastases detection on CT. Eur. Radiol. 2022. [CrossRef] [PubMed]
- Li, Z.; Wu, F.; Hong, F.; Gai, X.; Cao, W.; Zhang, Z.; Yang, T.; Wang, J.; Gao, S.; Peng, C. Computer-Aided Diagnosis of Spinal Tuberculosis From CT Images Based on Deep Learning with Multimodal Feature Fusion. Front. Microbiol. 2022, 13, 823324. [Google Scholar] [CrossRef] [PubMed]
- Liang, M.; Tang, W.; Xu, D.M.; Jirapatnakul, A.C.; Reeves, A.P.; Henschke, C.I.; Yankelevitz, D. Low-Dose CT Screening for Lung Cancer: Computer-aided Detection of Missed Lung Cancers. Radiology. 2016, 281, 279–288. [Google Scholar] [CrossRef] [Green Version]
- McKinney, S.M.; Sieniek, M.; Godbole, V.; Godwin, J.; Antropova, N.; Ashrafian, H.; Back, T.; Chesus, M.; Corrado, G.S.; Darzi, A.; et al. International evaluation of an AI system for breast cancer screening. Nature 2020, 577, 89–94. [Google Scholar] [CrossRef]
- Loom. Available online: https://www.loom.com/ (accessed on 21 June 2023).
- Visian. Available online: https://visian.org/ (accessed on 21 June 2023).
- Salvo, N.; Christakis, M.; Rubenstein, J.; de Sa, E.; Napolskikh, J.; Sinclair, E.; Ford, M.; Goh, P.; Chow, E. The Role of Plain Radiographs in Management of Bone Metastases. J. Palliat Med. 2009, 12, 195–198. [Google Scholar] [CrossRef]
- Williams, A.L.; Al-Busaidi, A.; Sparrow, P.J.; Adams, J.E.; Whitehouse, R.W. Under-reporting of osteoporotic vertebral fractures on computed tomography. Eur. J. Radiol. 2009, 69, 179–183. [Google Scholar] [CrossRef]
- Bartalena, T.; Rinaldi, M.F.; Modolon, C.; Braccaioli, L.; Sverzellati, N.; Rossi, G.; Rimondi, E.; Busacca, M.; Albisinni, U.; Resnick, D. Incidental vertebral compression fractures in imaging studies: Lessons not learned by radiologists. World J. Radiol. 2010, 2, 399–404. [Google Scholar] [CrossRef]
- Donald, J.J.; Barnard, S.A. Common patterns in 558 diagnostic radiology errors. J. Med. Imaging Radiat. Oncol. 2012, 56, 173–178. [Google Scholar] [CrossRef]
- Smith, M.J. Error and Variation in Diagnostic Radiology; C.C. Thomas: Springfield, IL, USA, 1967. [Google Scholar]
- Markus, J.B.; Somers, S.; O’Malley, B.P.; Stevenson, G.W. Double-contrast barium enema studies: Effect of multiple reading on perception error. Radiology 1990, 175, 155–156. [Google Scholar] [CrossRef] [PubMed]
- Wakeley, C.J.; Jones, A.M.; Kabala, J.E.; Prince, D.; Goddard, P.R. Audit of the value of double reading magnetic resonance imaging films. Br. J. Radiol. 1995, 68, 358–360. [Google Scholar] [CrossRef] [PubMed]
- Gillies, R.J.; Kinahan, P.E.; Hricak, H. Radiomics: Images Are More than Pictures, They Are Data. Radiology 2016, 278, 563–577. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Masoudi, S.; Mehralivand, S.; Harmon, S.; Walker, S.; Pinto, P.A.; Wood, B.J.; Citrin, D.E.; Karzai, F.; Gulley, J.L.; Madan, R.A.; et al. Artificial intelligence assisted bone lesion detection and classification in computed tomography scans of prostate cancer patients. J. Clin. Orthod. 2020, 38, e17567. [Google Scholar] [CrossRef]
- Li, Y.; Zhang, Y.; Zhang, E.; Chen, Y.; Wang, Q.; Liu, K.; Yu, H.J.; Yuan, H.; Lang, N.; Su, M.-Y. Differential diagnosis of benign and malignant vertebral fracture on CT using deep learning. Eur. Radiol. 2021, 31, 9612–9619. [Google Scholar] [CrossRef]
- Han, S.; Oh, J.S.; Lee, J.J. Diagnostic performance of deep learning models for detecting bone metastasis on whole-body bone scan in prostate cancer. Eur. J. Nucl. Med. Mol. Imaging 2022, 49, 585–595. [Google Scholar] [CrossRef]
- Ohno, Y.; Aoyagi, K.; Takenaka, D.; Yoshikawa, T.; Ikezaki, A.; Fujisawa, Y.; Murayama, K.; Hattori, H.; Toyama, H. Machine learning for lung CT texture analysis: Improvement of inter-observer agreement for radiological finding classification in patients with pulmonary diseases. Eur. J. Radiol. 2021, 134, 109410. [Google Scholar] [CrossRef]
- Kato, S.; Amemiya, S.; Takao, H.; Yamashita, H.; Sakamoto, N.; Miki, S.; Watanabe, Y.; Suzuki, F.; Fujimoto, K.; Mizuki, M.; et al. Computer-aided detection improves brain metastasis identification on non-enhanced CT in less experienced radiologists. Acta Radiol. 2023, 64, 1958–1965. [Google Scholar] [CrossRef]
- Yoda, T.; Maki, S.; Furuya, T.; Yokota, H.; Matsumoto, K.; Takaoka, H.; Miyamoto, T.; Okimatsu, S.; Shiga, Y.; Inage, K.; et al. Automated Differentiation between Osteoporotic Vertebral Fracture and Malignant Vertebral Fracture on MRI Using a Deep Convolutional Neural Network. Spine 2022, 47, E347–E352. [Google Scholar] [CrossRef]
- Yeh, L.-R.; Zhang, Y.; Chen, J.-H.; Liu, Y.-L.; Wang, A.-C.; Yang, J.-Y.; Yeh, W.-C.; Cheng, C.-S.; Chen, L.-K.; Su, M.-Y. A deep learning-based method for the diagnosis of vertebral fractures on spine MRI: Retrospective training and validation of ResNet. Eur. Spine J. 2022, 31, 2022–2030. [Google Scholar] [CrossRef]
- Ouyang, H.; Meng, F.; Liu, J.; Song, X.; Li, Y.; Yuan, Y.; Wang, C.; Lang, N.; Tian, S.; Yao, M.; et al. Evaluation of Deep Learning-Based Automated Detection of Primary Spine Tumors on MRI Using the Turing Test. Front. Oncol. 2022, 12, 814667. [Google Scholar] [CrossRef]
- Liu, H.; Jiao, M.; Yuan, Y.; Ouyang, H.; Liu, J.; Li, Y.; Wang, C.; Lang, N.; Qian, Y.; Jiang, L.; et al. Benign and malignant diagnosis of spinal tumors based on deep learning and weighted fusion framework on MRI. Insights Imaging 2022, 13, 87. [Google Scholar] [CrossRef] [PubMed]
Positive Cases | Total Number of Detected Objects | Number of Detected Spinal Lesions (TP) | Number of Undetected Spinal Lesions (FN) | Number of Falsely Detected Spinal Lesions (FP) | |
---|---|---|---|---|---|
Gold Standard | 16 | 27 | 27 | N/A | N/A |
DLA | 11 | 40 | 12 | 15 | 13 |
Sensitivity (TP Rate) | Specificity (TN Rate) | Accuracy | |||
---|---|---|---|---|---|
Case Level | Object Level | Case Level | Object Level | Case Level | Object Level |
75.00% | 44.44% | 56.25% | N/A | 65.63% | N/A |
Participants | |||||||
1 | 2 | 3 | 4 | 5 | 6 | ||
Experience (years) | 6 | 7 | 5 | 1 | 11 | 1 | |
Phase 1—no support from DLA | Mean | ||||||
Suspicious lesions reported (n) | 2.00 | 4.00 | 3.00 | 0.00 | 2.00 | 0.00 | 1.83 |
False positives (n) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
Sensitivity (case level) | 12.50% | 25.00% | 18.75% | 0.00% | 12.50% | 0.00% | 11.46% |
Sensitivity (object level) | 7.41% | 14.81% | 11.11% | 0.00% | 7.41% | 0.00% | 6.79% |
Specificity (case level) | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
Accuracy (case level) | 56.25% | 62.50% | 59.38% | 50.00% | 56.25% | 50.00% | 55.73% |
Accuracy (object level) | 53.70% | 57.41% | 55.56% | 50.00% | 53.70% | 50.00% | 53.40% |
False positive rate | 0.00% | 0.000% | 0.000% | 0.000% | 0.000% | 0.000% | 0.000% |
Phase 2—with support from DLA | Mean | ||||||
Suspicious lesions reported (n) | 5.00 | 5.00 | 8.00 | 0.00 | 14.00 | 3.00 | 5.83 |
False positives (n) | 1.00 | 0.00 | 1.00 | 0.00 | 2.00 | 1.00 | 0.83 |
Sensitivity (case level) | 31.25% | 31.25% | 43.75% | 0.00% | 75.00% | 12.50% | 32.29% |
Sensitivity (object level) | 14.81% | 18.52% | 25.93% | 0.00% | 44.44% | 7.41% | 18.52% |
Specificity (case level) | 93.75% | 100.00% | 93.75% | 100.00% | 87.50% | 93.75% | 94.79% |
Accuracy (case level) | 62.25% | 65.62% | 68.75% | 50.00% | 81.25% | 53.12% | 63.54% |
Accuracy (object level) | 55.56% | 59.26% | 68.12% | 50.00% | 68.52% | 51.85% | 58.89% |
False-positive rate | 3.70% | 0.00% | 3.57% | 0.00% | 7.41% | 3.70% | 3.06% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Gilberg, L.; Teodorescu, B.; Maerkisch, L.; Baumgart, A.; Ramaesh, R.; Gomes Ataide, E.J.; Koç, A.M. Deep Learning Enhances Radiologists’ Detection of Potential Spinal Malignancies in CT Scans. Appl. Sci. 2023, 13, 8140. https://doi.org/10.3390/app13148140
Gilberg L, Teodorescu B, Maerkisch L, Baumgart A, Ramaesh R, Gomes Ataide EJ, Koç AM. Deep Learning Enhances Radiologists’ Detection of Potential Spinal Malignancies in CT Scans. Applied Sciences. 2023; 13(14):8140. https://doi.org/10.3390/app13148140
Chicago/Turabian StyleGilberg, Leonard, Bianca Teodorescu, Leander Maerkisch, Andre Baumgart, Rishi Ramaesh, Elmer Jeto Gomes Ataide, and Ali Murat Koç. 2023. "Deep Learning Enhances Radiologists’ Detection of Potential Spinal Malignancies in CT Scans" Applied Sciences 13, no. 14: 8140. https://doi.org/10.3390/app13148140