Abstract
The aim was to assess the precision and accuracy of cephalometric analyses performed by artificial intelligence (AI) with and without human augmentation. Four dental professionals with varying experience levels identified 31 landmarks on 30 cephalometric radiographs twice. These landmarks were re-identified by all examiners with the aid of AI. Precision and accuracy were assessed by using intraclass correlation coefficients (ICCs) and mean absolute errors (MAEs). AI revealed the highest precision, with a mean ICC of 0.97, while the dental student had the lowest (mean ICC: 0.77). The AI/human augmentation method significantly improved the precision of the orthodontist, resident, dentist, and dental student by 3.26%, 2.17%, 19.75%, and 23.38%, respectively. The orthodontist demonstrated the highest accuracy with an MAE of 1.57 mm/°. The AI/human augmentation method improved the accuracy of the orthodontist, resident, dentist, and dental student by 12.74%, 19.10%, 35.69%, and 33.96%, respectively. AI demonstrated excellent precision and good accuracy in automated cephalometric analysis. The precision and accuracy of the examiners with the aid of AI improved by 10.47% and 27.27%, respectively. The AI/human augmentation method significantly improved the precision and accuracy of less experienced dental professionals to the level of an experienced orthodontist.
1. Introduction
Since the development of cephalometric radiography by Broadbent and Hofrath in 1931 [], a multitude of lateral cephalometric analyses have been developed for the purpose of orthodontic diagnosis and treatment planning. These analyses allow orthodontists to quantify and analyze relationships in the craniofacial region between the skeleton, dentition, and soft tissue. However, an accurate orthodontic diagnosis derived from a cephalometric radiograph requires accurate identification of key landmarks. Current methods of landmark identification include hand tracing with acetate paper or the use of computer software to manually mark digitized cephalometric radiographs, both of which have been shown to be equally reliable [].
Regardless of the method of cephalometric tracing, manual identification of anatomical structures presents inherent challenges. Factors such as differences in training and radiograph quality can cause variations in intra-examiner and inter-examiner reliability []. Additionally, reliability varies depending on the landmark being identified. The landmarks condylion, gnathion, orbitale, and anterior nasal spine tended to have the lowest inter-examiner reliability among experienced dental radiologists and orthodontists []. Computing technology demonstrates the potential to automate this process through artificial intelligence (AI).
AI refers to the development of computer systems that can perform tasks that would typically require human intelligence. These tasks include but are not limited to: learning, reasoning, problem-solving, perception, and language understanding. AI systems often employ algorithms and models to process and interpret data, enabling them to make informed decisions or predictions. In dentistry, AI has been used in a variety of applications, such as to assist in the diagnosis of vertical root fractures, diagnosis of proximal caries and localization of root canals []. One area in which AI has been used in orthodontics is automatic lateral cephalometric landmark identification. Pioneered by Cohen in 1984, preliminary attempts to locate landmarks using AI were limited to easily identifiable landmarks such as sella and menton []. Since Cohen’s attempt to automate landmark identification, several AI methods have been developed to increase accuracy and precision. Success rates of these methods, defined as a deviation of less than 2 mm between the distance measured from the AI-marked landmark and the expert-marked landmark, have varied from 35 to 84.7% depending on the methodology []. Convolutional neural networks (CNNs), a type of deep-learning algorithm, have demonstrated some of the highest accuracy in automatic lateral cephalometric landmark identification []. The CNN You-Only-Look-Once version 3 (YOLOv3) is a frontrunner in this aspect [,].
More recent studies have attempted to improve upon the precision and accuracy of the YOLOv3 algorithm, albeit with limited success. Yao used a customized CNN using PyTorch software (Facebook Artificial Intelligence Research; Facebook, Menlo Park, CA, USA) and found a mean radial error of about 1 mm, averaged among 37 landmarks []. However, the mean radial errors varied significantly, ranging from 0.5 mm for pronasale to 2 mm for pogonion []. Jeon utilized a CNN using CephX software and found significant differences between AI and manual measurements for saddle angle, U1-NA (mm), and L1-NB (mm) measurements []. Manual adjustment of landmarks identified by AI was recommended for further improvement in accuracy []. A systematic review conducted to evaluate the accuracy of automatic cephalometric landmark identification concluded that most CNNs demonstrated accuracy that matched the accuracy of an experienced orthodontist for many measurements [].
In addition to two-dimensional landmark identification, studies have been conducted to evaluate the accuracy of AI in landmark identification in three dimensions. Bao evaluated the accuracy of cephalometric landmark identification with AI on lateral cephalometric radiographs generated from 3D cone-beam computed tomography (CBCT) images. It was discovered that 15 of 23 cephalometric measurements were within the clinically acceptable range, or within 2 mm or degrees from the gold standard []. A meta-analysis by Serafin evaluated the accuracy of 3D landmark identification on CBCT images using only recent studies (from 2020 to 2022). As technology advanced, the mean error improved by over 2 mm over the 2-year period []. Despite the improvements, AI has fared better in 2D landmark identification compared to 3D, likely due to smaller sample sizes and increased computational complexity in CBCT studies [].
Although cephalometric landmark identification with AI has improved in the last few decades, a few limitations still exist. Most studies have used far below the recommended 2300 images to train the AI in order to achieve the same level of accuracy as human examiners []. Additionally, few studies have used AI trained from multiple types of x-ray machines, which limits real-world clinical application of the software []. A study by Silva et al. addressed these limitations by training a CNN named CEFBOT (RadioMemory Ltd., Belo Horizonte, Brazil) with over 250,000 images from various x-ray sources to automatically perform Arnett’s analysis measurements [].
AI-based fully automated decision-making processes put machines in the driver’s seat and aim to replace human decisions. However, can machines and humans work together in making better decisions? This new concept is defined as AI/human augmentation, where the goal is to improve human decision-making with machine insights and recommendations. Although previous studies show a promising future for AI-based automatic cephalometric landmark identification and analysis, machines have not yet shown the ability to outperform humans. On average, 80% of landmarks are accurately identified by AI alone within 2 mm of the ideal location []. The errors in the remaining 20% of landmarks can drastically affect cephalometric angular and linear measurements to the point that the results are not of diagnostic value. The ability to manually alter the location of the inaccurate landmarks could result in significantly improved accuracy levels. However, there is a lack of studies in the literature that specifically evaluate the extent to which clinicians with different levels of experience can benefit from this collaborative approach. To the best of our knowledge, this study will be the first to assess the impact of AI on the precision and accuracy of cephalometric analyses conducted by examiners with varying levels of experience. The objective of this study was to evaluate the precision and accuracy of cephalometric analyses performed using AI alone and in combination with human augmentation, and to determine whether this collaboration leads to varying levels of improvement among examiners with different levels of experience. Our hypothesis is that the least experienced clinicians would derive the greatest benefit from the AI/human augmentation in cephalometric analyses.
2. Materials and Methods
This study was reviewed and marked as IRB exempt by Indiana University Institutional Review Board (Protocol:14334) on 14 March 2022.
2.1. Sample Size Justification
Prior to the start of the study, a power analysis was conducted to determine the sample size. With a sample size of 30 images, the width of the 95% confidence interval for the intraclass correlation coefficients (ICCs) ranged from 0.08 and 0.28 from the estimated ICC, assuming the ICC is between 0.8 and 0.95. In addition, the width of the 95% confidence interval for the percentage of measurements within 1.5 mm of the ideal ranged between 20% and 30% from the estimated percentage, assuming the percentage is between 75% and 95%.
2.2. Data Collection
Thirty lateral cephalograms were randomly selected from a pool of adult (age 18 or older) orthodontic patients treated in the Indiana University School of Dentistry (IUSD). Inclusion criteria included patients with a class I skeletal relationship (defined as ANB between 0.5 to 5.0°) and an angle class I dental relationship. Patients were excluded if they had craniofacial anomalies, missing or unerupted permanent incisors, missing or unerupted permanent molars, or poor quality initial cephalometric radiographs. The cephalometric radiographs were saved as JPEG files at a resolution of 200 dpi.
A calibration session led by an orthodontist with 25 years of experience was conducted to clarify the cephalometric landmark identification process and to define each of the landmarks. Four examiners, including an orthodontist with two years of experience, a second-year orthodontics resident, a fourth-year dental student, and a general dentist with one year of experience, each separately and manually identified 31 cephalometric landmarks on the 30 radiographs (Figure 1). The same examiners retraced all radiographs after a period of one week in a randomized order. After completion of the manual tracings, the examiners traced the same cephalometric radiographs with the aid of a deep-learning AI (AI/human augmentation method). For these tracings, AI first located all of the landmarks, and the examiners later adjusted points as needed. The AI/human augmentation method tracings were completed again after one week in a randomized order. Additionally, AI alone identified each of the landmarks without any adjustments and repeated the process one week later.
Figure 1.
Cephalometric landmarks used in this study. 1: Nasion, 2: soft tissue nasion, 3: orbitale, 4: sella, 5: porion, 6: basion, 7: condylion, 8: articulare, 9: gonion, 10: menton, 11: pogonion, 12: gnathion, 13: B point, 14: A point, 15: anterior nasal spine, 16: posterior nasal spine, 17: distal of upper first molar, 18: mesial of upper first molar, 19: mesial of lower first molar, 20: lower incisor root apex, 21: lower incisor incisal edge, 22: upper incisor incisal edge, 23: upper incisor root apex, 24: soft tissue pogonion, 25: soft tissue B point, 26: lower lip, 27: upper lip, 28: soft tissue A point, 29: subnasale, 30: pronasale, 31: soft tissue menton.
The gold standard for landmark positions was identified with the collaboration of an orthodontist and dental radiologist, each with 25 years of experience within their respective fields. Any disagreements resulted in the point being located at the midpoint both vertically and horizontally between the two points.
2.3. Software
RadioCef software v.3 (RadioMemory Ltd., Belo Horizonte, Brazil) with CEFBOT module was used for automated cephalometric landmark identification. This software uses four subsystems to locate cephalometric landmarks. Subsystem 1 is a convolutional neural network (CNN) that processes radiographs into probability maps of cephalometric landmark positions. Subsystem 2 is another CNN that partitions radiographs by vectorizing skeletal and soft tissue borders and estimates the positions of cranial landmarks. Subsystem 3 combines the two CNN subsystems to estimate the position of landmarks, whereas subsystem 4 acts as a quality check to validate the previous results []. The CEFBOT was trained with substantial cephalometric data comprising over 250,000 cephalometric images from multiple X-ray sources and providers [].
2.4. Data Analyses
Using the traced landmarks, 26 cephalometric measurements were measured for each of the traced radiographs (Supplementary Table S1). Precision was evaluated by comparing the initial tracing to the final tracing completed after a period of one week for each of the examiners, with and without the aid of AI. The accuracy of each of these measurements was evaluated by comparing each examiner’s values, with and without the aid of AI, to the gold standard values through mean absolute error (MAE). MAEs were expressed as millimeters or degrees depending on whether it was a linear or angular measurement.
2.5. Statistical Analyses
For both precision and accuracy analyses, MAEs with 95% confidence intervals were provided for differences between the gold standard and each measurement. For precision analysis, ICCs and Bland–Altman plots were used to measure the agreement between initial and final measurements. One-way analysis of variance (ANOVA) with a random effect was used to test for the differences between the 9 examiner groups. All the analyses were done using SAS 9.4 (Cary, NC, USA) software.
3. Results
3.1. Precision Analysis
Table 1 shows the ICCs of the cephalometric measurements for each group. Mean ICC values were also calculated for all groups of examiners by averaging the individual cephalometric measurement ICC values. According to the results, AI showed excellent precision (ICCs > 0.90). The precision of the examiners without the aid of AI generally followed their level of experience and exposure to the cephalometric analyses. The orthodontist, resident, dentist, and dental student had a mean ICC of 0.92, 0.92, 0.81, and 0.77, respectively. With the aid of AI, the precision of the orthodontist, resident, dentist, and dental student improved by 3.26%, 2.17%, 19.75%, and 23.38%, respectively (Figure 2). Overall, the precision of the examiners improved by 10.47% with the aid of AI. As we expected, the least experienced clinicians benefitted the most from the help of AI.
Table 1.
Precision of the examiners with and without the aid of AI given by the intraclass correlation coefficients (ICCs).
Figure 2.
Mean intraclass correlation coefficient for each examiner group and percentages of improvement in precision.
Without the aid of AI, the cephalometric measurements that yielded moderate precision were maxillary length, U1-NA (mm), SNA, ANB, and convexity angles. With the AI/human augmentation method, 23 out of 26 measurements showed excellent precision, while U1-NA (mm), ANB, and convexity angles improved to good precision (Table 1).
Table 2 shows the results of the one-way ANOVA analysis, with a random effect used to test for the differences in precision between the methods. A statistically significant difference was found in 13 out of 26 measurements (p < 0.05). In general, the dentist and student, both without the aid of AI, had more significant differences in precision compared to the rest of the examiner groups.
Table 2.
The results of the one-way ANOVA analysis with a random effect used to test for the differences in precision between the methods.
3.2. Accuracy Analysis
To compare each examiner’s overall accuracy with and without AI’s aid, an average of the MAEs was calculated for each group (Table 3). The average MAE of CEFBOT was 1.83 (in mm/°), which was within the clinically acceptable range (MAEs < 2 mm/°). The examiners’ accuracy without AI generally followed their level of experience and exposure to the cephalometric analyses. The orthodontist and resident both had MAEs within the clinically acceptable range (1.57 and 1.99 mm/°, respectively) while the accuracy of the dentist and dental student were out of the range, with MAEs of 2.55 and 2.68 mm/°, respectively. With the aid of AI, the accuracy of the orthodontist, resident, dentist, and dental student improved by 12.74%, 19.10%, 35.69%, and 33.96%, respectively (Figure 3). The AI/human augmentation method revealed clinically acceptable accuracies for all of the examiner groups. Although the examiners’ accuracy improved by 27.27% with the aid of AI, the least experienced clinicians benefitted the most from the AI/human augmentation method of cephalometric landmark identification.
Table 3.
Accuracy of the examiners with and without the aid of AI given by the mean absolute errors (MAEs) from the gold standard.
Figure 3.
Average mean absolute errors for each examiner group and the percentages of improvement in accuracy.
Without the aid of AI, 13 out of 26 cephalometric measurements (maxillary and mandibular lengths, posterior face height, SNA, FMA, cranial base, U1-SN, L1-MP, L1-NB, interincisal, convexity, and facial angles) yielded MAEs out of the clinically acceptable range (Table 3). With the AI/human augmentation method, five of these measurements (SNA, maxillary and mandibular lengths, convexity, and facial angles) improved to be in the clinically acceptable range, while others remained out of the range although they showed some improvement in their accuracies (Table 3).
Table 4 shows the results of the one-way ANOVA analysis with a random effect used to test for the differences in accuracies between the methods. A statistically significant difference was found in 23 out of 26 measurements (p < 0.0001). No significant differences between groups were found in the ANB, L1-NB (°), and NA-Apo measurements (p > 0.05).
Table 4.
The results of the one-way ANOVA analysis with a random effect used to test for the differences in accuracies between the methods.
4. Discussion
Accurate identification of cephalometric landmarks is crucial in orthodontic diagnosis and treatment planning. Approximately 20% of landmarks identified by AI deviate by more than 2 mm from the ideal location []. These landmarks are used for angular and linear cephalometric measurements. A slight deviation in the position of a single landmark can significantly alter the resulting measurement and reduce the diagnostic quality of the traced cephalometric radiograph. Manual adjustment of the misidentified landmarks has been suggested to improve accuracy and allow orthodontists to use the radiograph in a diagnostic manner []. Currently, there is no literature that compares the precision and accuracy of AI/human modified landmarks to AI alone. Additionally, no studies have evaluated how less experienced examiners, such as dentists and dental students, fare in cephalometric landmark identification, with or without the aid of AI. As the proportion of orthodontic cases treated by general dentists compared to orthodontists increases, it is important to evaluate the accuracy of these less experienced clinicians.
The precision of AI alone obtained in this study was similar to the levels of precision reported in other studies [,]. Silva and his colleagues found that the ICC of AI ranged from 0.768 to 0.997 for the 10 Arnett’s measurements. Excluding the lowest ICC measurement of Frankfort plane to true horizontal line (ICC = 0.768), the remaining ICC values were greater than 0.94, indicating excellent precision []. This is similar to our study, in that the ICC values of AI traced cephalometric measurements ranging from 0.890 to 1.00 across the 26 cephalometric measurements and demonstrated the greatest precision of all examiners. Furthermore, Hwang found that the YOLOv3 deep-learning software had excellent precision and marked each landmark at the exact same position compared to the human inter-examiner reliability of 0.97 mm []. The accuracy of AI-derived cephalometric measurements differed from Silva’s study, which found that there were no statistically significant differences in accuracy of nine of the 10 Arnett’s measurements used, and AI was unable to measure the 10th measurement []. In this study, 10 out of 26 AI-derived measurements revealed accuracies out of the clinically acceptable range. An interesting note was that of the four of these measurements involved the gonion landmark (posterior face height, FMA, SN-MP, and L1-MP). Posterior face height and L1-MP were significantly underestimated, whereas SN-MP and FMA were overestimated. These errors were consistent with AI software identifying gonion too superiorly relative to the gold standard, which is a likely source contributing to the mean error. This error was not limited to our study, as Hwang found that the mean error of YOLOv3 was 2.9 mm in the identification of the gonion landmark []. Additionally, a meta-analysis conducted by Schwendicke showed that porion, subspinale, gonion, articulare, and anterior nasal spine were the least accurate landmarks identified by AI, although it varied depending on the specific AI software being tested [].
Of the five cephalometric measurements that demonstrated moderate precision without the aid of AI, four of them (U1-NA distance, SNA, ANB, and convexity angles) involved “Point A” landmark. This suggests “Point A” may pose significant challenges in terms of reproducibility. Possibilities for this include the proximity of this point to the maxillary incisor root, superimposition of nearby structures, blurriness of the curvature in the anterior maxilla, and proximity of the canine fossa. Subspinale, also known as Point A, has shown good reliability in the x-axis but significant variation in the y-axis []. Jeon also reported a significant reduction in accuracy in U1-NA (mm) compared to other measurements and theorized it may be due to superimposition of surrounding anatomy [].
In terms of accuracy averaged among all the examiners, 13 of the 26 measurements demonstrated MAEs greater than 2 mm or degrees without the use of AI. The increased MAEs of FMA and posterior face height may be due to the difficulty of identification of the gonion landmark. Gonion is a constructed point that relies on accurate identification of other landmarks, such as articulare and menton, to create tangent lines to the inferior body of the mandible and posterior border of the ramus. Slight inaccuracies in the tangent lines can dramatically affect the location of gonion. Among the 16 landmarks evaluated by Baumrind and Frantz, gonion was found to be the least reliable landmark in both the x and y-axes in lateral cephalometric films []. Additionally, in the case of FMA, it is possible that the orbitale landmark was not accurately identified by all examiners. Orbitale has been shown to have significant variability in lateral cephalometric identification, especially in the y-axis []. Inaccuracy of the maxillary length measurement may be due to difficulty of identification of the posterior nasal spine landmark, as it is often superimposed with the third molar germ. Condylion and basion landmarks are also challenging to identify due to the superimposition of other craniofacial structures and thus may explain the higher MAEs of mandibular length and cranial base angle. Basion has shown significant variation in identification in both the x and y-coordinates []. The incisor angulation measurements (U1-SN, U1-NA, L1-MP, L1-NB, and interincisal angle) also demonstrated increased MAEs. This could be due to difficulty in the identification of upper and lower incisor apices. Stabrun and Danielson found that upper and lower incisor apices were not reproducibly identified in 75% of cases []. Misidentification of the incisor root apices results in alterations of the long axis of the upper and lower incisors, which affects the accuracy of U1-SN, U1-NA, L1-MP, L1-NB, and interincisal angles.
The AI/human augmentation method was a novel aspect of this study. This method demonstrated an increase in the precision and accuracy of all examiner groups compared to the manual landmark identification technique, especially with the less experienced student and dentist examiners. This is likely due to the inherent ability of the AI to predictably locate each landmark in a nearly identical position []. The less experienced examiners may not have changed the position of landmarks unless there was a significant discrepancy due to a lack of skill in tracing lateral cephalometric radiographs. Most measurements that showed clinical significance overlapped between AI alone and the AI/human augmentation method (posterior face height, FMA, L1-MP, L1-NB, and interincisal angles). This reinforces the idea that the location of difficult to locate landmarks, such as gonion and incisor apices, may not have been modified by examiners.
Overall, the results from this study show that the human/AI augmentation method demonstrates greater accuracy and precision for all examiner groups compared to manual landmark identification. Additionally, the mean accuracy of every examiner with the aid of AI was greater than the accuracy of AI alone. Therefore, this method improves the diagnostic quality of lateral cephalometric radiographs compared to manual or AI identification alone.
One limitation of this study is that a single examiner represented an entire dental professional group. Ideally, each experience level would comprise multiple examiners. Additionally, we were unable to compare the accuracy or precision of individual landmarks. Future studies with multiple examiners and evaluation of the precision and accuracy of individual landmarks are recommended. Despite significant improvements in computing technology and AI in automatic lateral cephalometric landmark identification, AI alone cannot fully replace experienced clinicians. However, AI can serve as an educational tool and aid in landmark identification through the human/AI augmentation method, especially for less experienced clinicians.
5. Conclusions
Deep learning AI demonstrated excellent precision and good accuracy in automated cephalometric analysis. Overall, the precision and accuracy of the examiners with the aid of AI improved by 10.47% and 27.27%, respectively. The AI/human augmentation method significantly improved the precision and accuracy of less experienced dental professionals and students in comparison to that of an experienced orthodontist. These groups benefited the most from the approach.
Supplementary Materials
The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/app13126921/s1, Table S1: Cephalometric measurements and their definitions.
Author Contributions
Conceptualization, S.P. and H.T.; data curation, S.P., A.Z., E.H., A.W., V.D. and H.T.; formal analysis, S.P., S.S.B., G.E., V.D. and H.T.; investigation, S.P., V.D. and H.T.; methodology, S.P., A.Z., E.H., A.W. and V.D.; project administration, H.T.; software, H.T.; supervision, H.T.; validation, S.P.; writing—original draft, S.P. and H.T.; writing—review & editing, S.P., A.W., V.D. and H.T. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
This study was reviewed and marked as IRB exempt by Indiana University Institutional Review Board (Protocol:14334) on 14 March 2022.
Informed Consent Statement
Not applicable.
Data Availability Statement
The data underlying this article are available in the article.
Acknowledgments
We would like to thank Craig Eberhardt, EHR systems, for help in filtering patient data.
Conflicts of Interest
The authors declare no conflict of interest.
References
- Leonardi, R.; Giordano, D.; Maiorana, F.; Spampinato, C. Automatic cephalometric analysis. Angle Orthod. 2008, 78, 145–151. [Google Scholar] [CrossRef] [PubMed]
- Albarakati, S.F.; Kula, K.S.; Ghoneima, A.A. The reliability and reproducibility of cephalometric measurements: A comparison of conventional and digital methods. Dentomaxillofac. Radiol. 2012, 41, 11–17. [Google Scholar] [CrossRef]
- Liu, J.K.; Chen, Y.T.; Cheng, K.S. Accuracy of computerized automatic identification of cephalometric landmarks. Am. J. Orthod. Dentofac. Orthop. 2000, 118, 535–540. [Google Scholar] [CrossRef]
- Durão, A.P.; Morosolli, A.; Pittayapat, P.; Bolstad, N.; Ferreira, A.P.; Jacobs, R. Cephalometric landmark variability among orthodontists and dentomaxillofacial radiologists: A comparative study. Imaging Sci. Dent. 2015, 45, 213–220. [Google Scholar] [CrossRef] [PubMed]
- Hung, K.; Montalvao, C.; Tanaka, R.; Kawai, T.; Bornstein, M.M. The use and performance of artificial intelligence applications in dental and maxillofacial radiology: A systematic review. Dentomaxillofac. Radiol. 2020, 49, 20190107. [Google Scholar] [CrossRef] [PubMed]
- Cohen, A.M.; Ip, H.H.; Linney, A.D. A preliminary study of computer recognition and identification of skeletal landmarks as a new method of cephalometric analysis. Br. J. Orthod. 1984, 11, 143–154. [Google Scholar] [CrossRef] [PubMed]
- Ren, R.; Luo, H.; Su, C.; Yao, Y.; Liao, W. Machine learning in dental, oral and craniofacial imaging: A review of recent progress. PeerJ 2021, 9, e11451. [Google Scholar] [CrossRef]
- Park, J.H.; Hwang, H.W.; Moon, J.H.; Yu, Y.; Kim, H.; Her, S.B.; Srinivasan, G.; Aljanabi, M.N.A.; Donatelli, R.E.; Lee, S.J. Automated identification of cephalometric landmarks: Part 1-Comparisons between the latest deep-learning methods YOLOV3 and SSD. Angle Orthod. 2019, 89, 903–909. [Google Scholar] [CrossRef]
- Hwang, H.-W.; Park, J.-H.; Moon, J.-H.; Yu, Y.; Kim, H.; Her, S.-B.; Srinivasan, G.; Aljanabi, M.N.A.; Donatelli, R.E.; Lee, S.-J. Automated Identification of Cephalometric Landmarks: Part 2-Might It Be Better Than human? Angle Orthod. 2019, 90, 69–76. [Google Scholar] [CrossRef]
- Yao, J.; Zeng, W.; He, T.; Zhou, S.; Zhang, Y.; Guo, J.; Tang, W. Automatic localization of cephalometric landmarks based on convolutional neural network. Am. J. Orthod. Dentofac. Orthop. 2022, 161, e250–e259. [Google Scholar] [CrossRef]
- Jeon, S.; Lee, K.C. Comparison of cephalometric measurements between conventional and automatic cephalometric analysis using convolutional neural network. Prog. Orthod. 2021, 22, 14. [Google Scholar] [CrossRef]
- Junaid, N.; Khan, N.; Ahmed, N.; Abbasi, M.S.; Das, G.; Maqsood, A.; Ahmed, A.R.; Marya, A.; Alam, M.K.; Heboyan, A. Development, Application, and Performance of Artificial Intelligence in Cephalometric Landmark Identification and Diagnosis: A Systematic Review. Healthcare 2022, 10, 2454. [Google Scholar] [CrossRef]
- Bao, H.; Zhang, K.; Yu, C.; Li, H.; Cao, D.; Shu, H.; Liu, L.; Yan, B. Evaluating the accuracy of automated cephalometric analysis based on artificial intelligence. BMC Oral Health 2023, 23, 191. [Google Scholar] [CrossRef] [PubMed]
- Serafin, M.; Baldini, B.; Cabitza, F.; Carrafiello, G.; Baselli, G.; Del Fabbro, M.; Sforza, C.; Caprioglio, A.; Tartaglia, G.M. Accuracy of automated 3D cephalometric landmarks by deep learning algorithms: Systematic review and meta-analysis. Radiol. Med. 2023, 128, 544–555. [Google Scholar] [CrossRef] [PubMed]
- Moon, J.H.; Hwang, H.W.; Yu, Y.; Kim, M.G.; Donatelli, R.E.; Lee, S.J. How much deep learning is enough for automatic identification to be reliable? Angle Orthod. 2020, 90, 823–830. [Google Scholar] [CrossRef]
- Kim, J.; Kim, I.; Kim, Y.-J.; Kim, M.; Cho, J.-H.; Hong, M.; Kang, K.-H.; Lim, S.-H.; Kim, S.-J.; Kim, Y.H.; et al. Accuracy of automated identification of lateral cephalometric landmarks using cascade convolutional neural networks on lateral cephalograms from nationwide multi-centres. Orthod. Craniofac. Res. 2021, 24, 59–67. [Google Scholar] [CrossRef] [PubMed]
- Silva, T.P.; Hughes, M.M.; Menezes, L.D.S.; de Melo, M.F.B.; Takeshita, W.M.; Freitas, P.H.L. Artificial intelligence-based cephalometric landmark annotation and measurements according to Arnett’s analysis: Can we trust a bot to do that? Dentomaxillofac. Radiol. 2022, 51, 20200548. [Google Scholar] [CrossRef]
- Schwendicke, F.; Chaurasia, A.; Arsiwala, L.; Lee, J.H.; Elhennawy, K.; Jost-Brinkmann, P.G.; Demarco, F.; Krois, J. Deep learning for cephalometric landmark detection: Systematic review and meta-analysis. Clin. Oral Investig. 2021, 25, 4299–4309. [Google Scholar] [CrossRef]
- Broch, J.; Slagsvold, O.; Røsler, M. Error in landmark identification in lateral radiographic headplates. Eur. J. Orthod. 1981, 3, 9–13. [Google Scholar] [CrossRef] [PubMed]
- Baumrind, S.; Frantz, R.C. The reliability of head film measurements. 1. Landmark identification. Am. J. Orthod. 1971, 60, 111–127. [Google Scholar] [CrossRef] [PubMed]
- Trpkova, B.; Major, P.; Prasad, N.; Nebbe, B. Cephalometric landmarks identification and reproducibility: A meta analysis. Am. J. Orthod. Dentofac. Orthop. 1997, 112, 165–170. [Google Scholar] [CrossRef] [PubMed]
- Stabrun, A.E.; Danielsen, K. Precision in cephalometric landmark identification. Eur. J. Orthod. 1982, 4, 185–196. [Google Scholar] [CrossRef] [PubMed]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).