Machine Learning and Feature Selection in Pediatric Appendicitis

Abstract

Background/Objectives: Accurate prediction of pediatric appendicitis diagnosis, management, and severity is critical for clinical decision-making. We aimed to evaluate the predictive performance of a wide range of machine learning models, combined with various feature selection techniques, on a pediatric appendicitis dataset. A particular focus was placed on the role of ultrasound (US) image-descriptive features in model performance and explainability. Methods: We conducted a retrospective cohort study on a dataset of 781 pediatric patients aged 0–18 presenting to Children’s Hospital St. Hedwig in Regensburg, Germany, between January 2016 and February 2023. We developed and validated predictive models; machine learning algorithms included the random forest, logistic regression, stochastic gradient descent, and the light gradient boosting machine (LGBM). These were paired exhaustively with feature selection methods spanning filter-based (association and prediction), embedded (LGBM and linear), and a novel redundancy-aware step-up wrapper approach. We employed a machine learning benchmarking study design where AI models were trained to predict diagnosis, management, and severity outcomes, both with and without US image-descriptive features, and evaluated on held-out testing samples. Model performance was assessed using overall accuracy and area under the receiver operating characteristic curve (AUROC). A deep learner optimized for tabular data, GANDALF, was also evaluated in these applications. Results: US features significantly improved diagnostic accuracy, supporting their use in reducing model bias. However, they were not essential for maximizing accuracy in predicting management or severity. In summary, our best-performing models were, for diagnosis, the random forest with embedded LGBM feature selection (98.1% accuracy, AUROC: 0.993), for management, the random forest without feature selection (93.9% accuracy, AUROC: 0.980), and for severity, the LGBM with filter-based association feature selection (90.1% accuracy, AUROC: 0.931). Conclusions: Our results demonstrate that high-performing, interpretable machine learning models can predict key clinical outcomes in pediatric appendicitis. US image features improve diagnostic accuracy but are not critical for predicting management or severity.

1. Introduction

Pediatric appendicitis is characterized by inflammation of the appendix found in patients aged eighteen years and younger. When inflamed, the appendix causes pain and can lead to serious complications for the patient, including peritonitis and infection [1]. Symptoms can include nausea, loss of appetite, constipation, bloating, and abdominal pain [1]. Symptoms are not always easily identified or caught in time in younger patients, as they may not communicate as well and often experience fewer symptoms [2]. Appendicitis is typically caused by a blockage in the lumen, leading to an infection that then causes the appendix to expand and potentially burst [1]. While appendicitis can occur in both males and females, males have been found to be at a slightly higher risk, and most cases occur between the ages of ten and thirty [1]. A highly effective way to diagnose appendicitis is to evaluate the current state of the appendix using medical imaging. This is performed through computed tomography (CT), ultrasound (US), or magnetic resonance imaging (MRI), with CT being the most accurate of the three [3]. A shortcoming of these imaging techniques is that they are expensive and potentially time-consuming. MRI may not always be readily accessible due to high costs, limited availability, and the need for specialized interpretation, all of which can delay diagnosis and treatment. Additionally, CT relies upon ionizing radiation, which for most adults is safe, but may be risky for younger patients due to the radiation’s potential negative effects on their growing bodies [3].

Supervised machine learning is a common technology applied to predictive applications, such as diagnosing a given medical condition. The algorithms are provided with ground-truth training data, which are represented by sets of samples/instances, each containing a set of feature measurements that can inform predictions, and a target variable to be predicted. During training, algorithms establish complex correlational relationships between predictor variables and the target variable, supporting the creation of technologies that can be relied upon to make predictions on samples that were not trained upon. As such, as long as correlations exist between predictor variables and the target variable, AI has the potential to create highly accurate predictive models.

Using artificial intelligence technologies to diagnose appendicitis has been the subject of previous analyses. One study from Saudi Arabia used K-nearest neighbours (KNN), decision trees (DT), bagging, and stacking to identify acute appendicitis and found their stacking model to be the most successful with training accuracy, testing accuracy, testing precision, and testing F1 scores of 97.51%, 92.63%, 95.29%, and 92.04%, respectively [4]. From their study, they found their most important features to be neutrophils, white blood cell count, length of stay, and symptom days for their stacking model [4]. Another study [5] was conducted using results from previous studies [6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34] to determine whether using AI models is an effective way for diagnosing acute appendicitis in adults. This review analyzed twenty-nine studies on diagnosis and prognosis of acute appendicitis, and found that the model most commonly used was the artificial neural network (ANN) [5]. These ANNs produced accuracy scores typically above 80% with some reporting the area under the receiver operating characteristic curve (AUC) nearing 0.99 [5]. However, it should be noted that this analysis was based on an adult population, and so the findings therein may not hold in a pediatric population.

Several recent studies have applied machine learning approaches to pediatric appendicitis using subsets of the dataset analyzed in the present work. A foundational study [35] on a subset of the dataset addressed in this research [36] was previously conducted and included 430 patients. The machine learning models used were logistic regression, random forest, and generalized boosted regression model, all in the R programming language. Their results are summarized: “A random forest classifier achieved areas under the precision-recall curve of 0.94, 0.92, and 0.70, respectively, for the diagnosis, management, and severity of appendicitis”, based on held-out test samples as part of 10-fold validation [35]. A subsequent analysis, as part of a larger team, was performed using a larger subset (579 patients) of the dataset addressed in this study to diagnose pediatric appendicitis using deep learning methods with concept bottleneck models (CBMs) with a primary focus on the ultrasound images [37]. While the dataset contains images and corresponding descriptions of the images, some patients included do not contain a complete set of all of these features. The images are taken from multiple views of the same target area to help ensure imaging has captured key features of the appendix being analyzed. To handle this, the study used a semi-supervised extension in addition to the CBM [37]. They first used a shared encoder neural network to map the images to features, which are then aggregated across imaging views to produce representations and concepts understandable by humans, contributing to the prediction of the target class [37]. Results of 0.80 AUROC were reported for predicting the diagnosis of appendicitis. Two additional studies have been conducted on the updated dataset used in this analysis, focused exclusively on diagnosis [38,39]. This includes an approach achieving 94.5% accuracy with the random forest [38], and an approach based on the Hybrid Bat algorithm achieving 94% accuracy [39]. An additional analysis focused on diagnosis and severity [40] but did not report accuracy statistics.

Hypotheses and Contributions

Our objective in this study is to address the following hypotheses. We hypothesize that:

▪

The use of open-source machine learning software applied to the Regensburg Pediatric Appendicitis Dataset may produce useful technology for predicting aspects of pediatric appendicitis patient care.

▪

By creating technologies that can predict diagnosis, severity, and management of pediatric appendicitis, both by using and withholding US image-derived features, we can assess the apparent value of US imaging in the context of AI predictive technology.

▪

Our models will be able to more accurately predict their respective target variables (diagnosis, management, and severity), as compared to previous works on this topic, by thoroughly examining a large set of combinations of machine learning and feature selection algorithms.

▪

Feature selection subsets will be informative to clinicians and researchers as to factors that are predictive of diagnosis, management, and severity of pediatric appendicitis, respectively.

Contributions provided by this study include the consideration of a large selection of feature selection (FS) algorithms, including a novel redundancy-aware FS algorithm developed in our lab, consideration of novel subsets of features identified by FS, consideration of a variety of high-performing machine learning algorithms, including the computationally efficient light gradient boosting machine and a deep learner optimized for tabular data, known as Gandalf, evaluation of our study findings on an updated pediatric appendicitis dataset with more patients/samples than those included in the early work on this topic, confirmation of the value of ultrasound imaging features as assisting in mitigating bias in prediction for diagnosis of appendicitis, and finally, demonstrating strong predictive performance from the models developed across three AI applications in pediatric appendicitis.

We introduced an overview of pediatric appendicitis, related AI technological development, and closely related work on the same dataset in Section 1, as well as provided a Hypotheses and Contributions subsection. The rest of the paper will proceed as follows: we will provide a study design overview in Section 2.1, an outline of the study participants in Section 2.2, a detailed dataset description of the variables/measurements in Section 2.3, an outline of the preprocessing performed on the dataset in Section 2.4, the machine learning methods used are presented in Section 2.5, and the statistics relied upon for machine learning evaluation are presented in Section 2.6. The results for predicting diagnosis are provided in Section 3.1, the results for predicting management are provided in Section 3.2, the results for predicting severity are provided in Section 3.3, and the Gandalf deep learner results are provided in Section 3.4. A discussion of interactions between machine learning and feature selection technologies employed is provided in Section 4.1, a discussion of Gandalf results is provided in Section 4.2, a discussion of the value of Ultrasound features is provided in Section 4.3, a literature comparison is provided in Section 4.4, future work is presented in Section 4.5, followed by our conclusions in Section 5.

2. Materials and Methods

2.1. Study Design Overview

We conducted a retrospective cohort study on a dataset of 781 pediatric patients aged 0–18 presenting to Children’s Hospital St. Hedwig in Regensburg, Germany, between January 2016 and February 2023. This study employed a comparative AI benchmarking approach using publicly available benchmarking software applied to an open-access pediatric appendicitis dataset. The analysis covered three clinical tasks: diagnosis (the AI is tasked with performing a diagnosis of appendicitis or not), management (the AI is tasked with predicting the treatment option for the patient), and severity (the AI is tasked with predicting the state of the patient’s appendicitis). The potential value from the inclusion of ultrasound image features was considered for all applications. This study was performed retrospectively on a public domain dataset; as such, no ethics committee approval was required for this analysis.

2.2. Participants

The dataset examined was initially assembled by Marcinkevičs et al., and their analysis was previously published [35]. The dataset was revisited [37] with an extended observation timeline, more patients, and additionally collected ultrasound images for many of the patients. The dataset previously studied [37] included records for 579 patients, whereas we examined an updated version of this dataset with 781 observations. The data was obtained from patients admitted to the tertiary Children’s Hospital St. Hedwig in Regensburg, Germany, with suspected appendicitis between 2016 and 2021. All aspects of the methods of this study were completed by the study authors except for the patient recruitment and data acquisition/curation previously completed [35,37].

2.3. Variables/Measurements

Patient data included demographic information, clinical examinations, laboratory tests, scoring results, and (potentially multiple per patient) ultrasound (US) images and expert-interpreted findings from the images. Descriptions of the feature measurements and target variables are detailed in Table 1, Table 2 and Table 3, and their numeric feature distributions in Table 4. The categorical feature statistics tables have also been provided in Appendix A (Table A1, Table A2 and Table A3). Detailed feature descriptions are also provided in Appendix A, broken down for different feature types, see Table A4, Table A5, Table A6, Table A7, Table A8 and Table A9. Note that there was a single patient/sample with a missing diagnosis field in this dataset; as such, it needed to be excluded from the diagnosis application, resulting in a count of 780 samples for the diagnosis application, whereas we were able to maintain the full sample count of 781 for the remaining two target variable applications. Predictive models were created to target the same three variables previously targeted [35] for binary classification:

Table 1. Diagnosis target variable.

	Appendicitis	No Appendicitis
Frequency	463	317
Proportion	463/780	317/780

Table 2. Management target variable.

	Conservative	Primary Surgical	Secondary Surgical	Simultaneous Appendectomy
Frequency	483	270	27	1
Proportion	483/781	270/781	27/781	1/781
Relative Frequency	61.84%	34.57%	3.46%	0.13%

Table 3. Severity target variable.

	Uncomplicated	Complicated
Frequency	662	119
Proportion	662/781	119/781
Relative Frequency	84.76%	15.24%

Table 4. Numeric feature statistics for patients with and without appendicitis.

Variable	Appendicitis: Mean, SD	No Appendicitis: Mean, SD
Age	11.08, 3.56	11.72, 3.46
BMI	18.45, 4.16	19.56, 4.62
Height	146.93, 20.43	149.51, 18.64
Weight	41.72, 17.47	45.25, 17.11
Length_of_Stay	5.11, 2.98	3.09, 0.98
Alvarado_Score	6.67, 1.93	4.83, 2.0
Paedriatic_Appendicitis_Score	5.82, 1.85	4.42, 1.81
Appendix_Diameter	8.7, 2.18	5.04, 1.17
Body_Temperature	37.52, 0.81	37.24, 1.0
WBC_Count	14.28, 5.34	10.33, 4.48
Neutrophil_Percentage	76.03, 12.63	65.6, 14.76
Segmented_Neutrophils	71.6, 12.51	55.23, 13.29
RBC_Count	4.79, 0.37	4.82, 0.64
Hemoglobin	13.38, 1.61	13.38, 1.02
RDW	13.4, 5.86	12.87, 0.87
Thrombocyte_Count	285.79, 70.83	284.48, 74.92
Ketones_in_Urine	1.15, 1.28	0.69, 1.11
RBC_in_Urine	0.4, 0.82	0.32, 0.71
WBC_in_Urine	0.24, 0.63	0.19, 0.55
CRP	44.9, 68.51	11.72, 24.92

• Diagnosis: Appendicitis (n = 463, 59.36%) or no appendicitis (n = 317, 40.64%).
• Management: Surgical (n = 298, 38.16%) or conservative (n = 483, 61.84%).
• Severity: Complicated (n = 119, 15.24%) or uncomplicated (n = 662, 84.76%).

2.4. Data Preprocessing

Df-analyze, the software relied upon for our machine learning and feature selection analysis, performs its own data cleaning [41], so null value handling was left to its imputation feature with median selection. A variety of preprocessing steps were applied prior to the use of df-analyze. The US number was dropped as it acted as a unique ID. All urine sample features were converted from categorical features to an ordinal scale from 0 to 3, so the relationship between values was encapsulated in the feature encodings. The management target variable was reduced to a binary class by combining primary surgical, secondary surgical, and simultaneous appendectomy in a single surgical class, as df-analyze requires substantial class representation for all target values for its validation to function. The data summary suggests secondary surgical management indicates surgery after their initial stay, when the patient data was recorded. As part of the previous analysis [35], patients were contacted at least 6 months after discharge and classified their management as (secondary) surgical if they had since had an appendectomy. As was previously investigated [35], we predict whether a patient required surgery, as it could potentially prevent a second visit to the hospital. Length of stay was also dropped from the dataset, as we have created technologies with potential real-world utility, in which we would want to be able to predict important target variables, such as diagnosis, severity, and management as early on in their hospital admission as possible, and we cannot establish the correct length of stay value for each patient until the end of their hospitalization.

The presumptive diagnosis feature may not always match the final diagnosis and may provide additional information reinforced by the managing doctors’ education and expertise, which could be particularly useful in smaller datasets. However, the feature may bias a machine learning model, or in real-world applications, may not be available for input. As such, this feature was excluded from our dataset.

Lymph Nodes Location, Abscess Location, and Gynecological findings were excluded from our dataset, as they were all described as free-form text, mostly in German. When divided into classes by unique values, Abscess Location and Gynecological findings’ largest class had fewer than 20 instances, which is too few for informing reliable predictions in df-analyze [41]. Lymph Nodes Location had some unique values with at least 20 instances, but many of its classes were combinations of others, and the feature is null for more than 80% of records; as such, it was also excluded. To facilitate reproducibility, custom pre-processing code for this dataset is provided in clean-tabular-dataset.py [42].

2.5. Machine Learning

The machine learning software used in this study is df-analyze [41]. The models considered in this study include the light gradient boosting machine (LGBM), random forest (RF), linear regression (LR), stochastic gradient descent (SGD), k-nearest neighbours (KNN), and a dummy model that predicts the class with the largest number of samples as a baseline. Df-analyze also supports assessment of a variety of feature selection (FS) technologies [41], each of which is exhaustively combined with all supported aforementioned machine learning methods. This includes two types of filter-based FS: association (assoc) and prediction (pred) [41], two types of embedded FS: linear (embed_linear) and LGBM (embed_lgbm) [41], and an emerging redundancy-aware step-up feature selection method (wrap) unique to df-analyze [43], as well as no (none) FS. The target features in this study were predicted from exhaustive combinations of supported machine learning and FS algorithms trained and tested individually as part of a fair comparison validation. For each target variable, models are constructed with each FS method. Optuna hyperparameter tuning is supported in df-analyze [41] and was used in this analysis for all machine learning techniques.

The code for running all configurations of our dataset with command line interfaces (CLIs) is provided in run-df-analyze.sh [42]. Each target variable was run with and without US image features. Thus, our analysis involves six runs of df-analyze as follows:

• Targeting Diagnosis with US Image Features Included;
• Targeting Diagnosis without US Image Features Included;
• Targeting Management with US Image Features Included;
• Targeting Management without US Image Features Included;
• Targeting Severity with US Image Features Included;
• Targeting Severity without US Image Features Included.

2.6. Statistical Analysis

Df-analyze conducts statistical analyses of each classification model paired with each FS method, using eight different metrics. These metrics are: overall accuracy (acc—the proportion of correct predictions out of all predictions), balanced accuracy (bal-acc—the expected accuracy if the dataset classes were balanced), F1-score (f1—the harmonic mean of recall and precision), negative predicted value (npv—the proportion of negative predictions that are correct), positive predicted value (ppv—the proportion of positive predictions that are correct), sensitivity (sens—the proportion of the group of interest predicted correctly), specificity (spec—the proportion of the group not-of-interest predicted correctly), and the area under ROC curve (AUROC or AUC—the area under the curve outlining the tradeoff between sensitivity and specificity across operating points). The primary metrics used to evaluate each model are overall accuracy and AUROC. Two validation methods were employed, including holdout set validation and K-Fold validation on the hold-out set. The hold-out set was established with a large 40% of the samples randomly selected in order to assist with reliability and reproducibility. Validation was performed on the holdout set, as well as with K-Fold validation on the holdout set with K = 5. Optuna hyperparameter tuning was completed with 50 runs. After completion of the above methods, a new version of df-analyze was released with support for an emerging deep learning method designed for tabular data, known as Gated Adaptive Network for Deep Automated Learning of Features (GANDALF) [44]. Df-analyze was re-accessed to assess this method as well (df-analyze access date: November 2024), and the experiments were re-run with GANDALF enabled. Due to the additional computational demands of GANDALF relative to the other machine learning methods assessed, df-analyze was run without redundancy-aware step-up feature selection enabled, as this was the slowest of our considered feature selection methods.

3. Results

3.1. Predicting Diagnosis

For predicting diagnosis when including US image features, the best-performing model was the random forest (RF) with embedded LGBM-based feature selection, achieving an accuracy of 98.1% and an AUROC of 99.3% across both validation methods, see Appendix B. The features that this model relied upon are outlined in Appendix C, which provides a ranking of their respective apparent importance to inform prediction.

When excluding the US image-based features, the best-performing model was LGBM with no (none) feature selection, achieving an accuracy of 80.1% and an AUROC of 87.3–88.0% across both validation methods, see Appendix D.

The Optuna hyperparameter-tuned model parameters for the leading techniques are provided in Appendix E. A comparative visualization of leading findings is provided in Figure 1.

Figure 1. Comparative bar plot of leading models for predicting diagnosis with and without US features.

3.2. Predicting Management

For predicting management, the best-performing models that included US-based image features were the random forest with association filter-based feature selection (assoc), achieving accuracies of 92.0–93.6% accuracy and an AUROC of 97.3–98.4% across both validation methods, see Appendix F. The association feature selection method selected for a large number of the available features in this dataset and is provided in detail in Appendix G. Note that a sorting of the importance of the features is provided both for numerical and categorical features, respectively. The leading features informing prediction, according to the association filter-based method’s reliance on mutual information, were C-reactive protein, Alvarado score, the appendix diameter, white blood cell count, and neutrophil percentage for the numerical variables, and ipsilateral rebound tenderness, diagnosis, peritonitis, severity, and surrounding tissue reaction for the categorical variables.

When predicting management without the US-based image features, the best-performing model was the random forest (RF) with no (none) feature selection, achieving accuracies of 92.0–93.9% and an AUROC of 97.0–98.0% across both validation methods, see Appendix H. Noteworthy is that our emerging redundancy-aware step-up feature selection method (wrap), which is biased in favour of unusually small feature sets, achieved near equal accuracies of 92.0–92.7% and an AUROC of 94.2–96.0%, based on just 11 features, as outlined in Appendix I. The leading features relied upon were peritonitis, white blood cell count, body temperature, weight, severity, and C-reactive protein.

The Optuna hyperparameter-tuned model parameters for leading techniques are provided in Appendix E. A visualization of leading findings is provided in Figure 2.

Figure 2. Comparative bar plot of leading models for predicting management with and without US features.

3.3. Predicting Severity

For predicting severity, with US image features included, the best-performing model was logistic regression (LR) with wrapper-based redundancy-aware step-up feature selection (wrap), which achieved accuracy of 89.1–89.5% and an AUROC of 82.0–83.4% across both validation methods, see Appendix J. The feature selection results are provided in Appendix K. Leading features were meteorism (excess gas in the digestive tract), dysuria, weight, lower right abdominal pain, and free fluids.

When predicting severity with US image features excluded, the best-performing model was LGBM with filter-based association (assoc) feature selection, achieving an accuracy of 89.2–90.1% and an AUROC of 89.6–93.1% across both validation methods, see Appendix L. As is common, the association-based feature selection method selects a large number of the available features in this dataset. Also of interest, redundancy-aware step-up feature selection (wrap) produced similar results, achieving an accuracy of 88.8% and an AUROC of 80.5–81.1% when combined with logistic regression based on just five features, as outlined in Appendix M. The five features included were peritonitis, coughing pain, body temperature, thrombocyte count, and C-reactive protein.

The Optuna hyperparameter-tuned model parameters for leading techniques are provided in Appendix E. A visualization of leading findings is provided in Figure 3.

Figure 3. Comparative bar plot of leading models for predicting severity with and without US features.

3.4. GANDALF Results

GANDALF [44] was run with an updated version of df-analyze, and so the results presented can only be roughly compared with the findings presented above due to it being run as an additional round of validation with unique randomization. When predicting diagnosis, the leading accuracy/AUROC for GANDALF was 80.5/90.6% with US features (filter-based prediction feature selection), and 66.7/75.7% without US features (filter-based prediction feature selection). When predicting management, the leading accuracy/AUROC for GANDALF was 91.5/96.9% with US features (no feature selection), and 90.5/97.5% without US features (embedded linear feature selection). When predicting severity, the leading accuracy/AUROC for GANDALF was 81.1/77.7% with US features (filter-based prediction feature selection), and 85.4/81.1% without US features (embedded linear feature selection).

4. Discussion

We performed a detailed study comparing several machine learning algorithms combined exhaustively with a variety of feature selection approaches applied to pediatric appendicitis diagnostics, management (treatment prediction), and severity. Results demonstrate that we are able to create high-performing models for each of the three main predictive tasks addressed. Our extensive use of feature selection has provided a variety of feature sets predictive of our three addressed target variables, information that can potentially assist in the clinical management of appendicitis and may inform the development of future AI technologies in this domain.

4.1. Interactions Between Machine Learning and Feature Selection Technologies

Our df-analyze benchmarking software has been previously used to assess machine learning and feature selection combinations that produce high-quality AI models to assist in schizophrenia diagnostics [45], chronic kidney disease diagnosis [46], mitigating bias in traffic stop outcomes [47], and studying proteins potentially linked with learning in the cerebral cortex [48]. In this study, we investigated the tool’s potential for use in three applications of pediatric appendicitis.

Logistic regression (LR) and stochastic gradient descent (SGD) were only among our top performers when using a feature selection method, suggesting that those methods are sensitive to being negatively biased from the inclusion of noisy, useless, and/or redundant features. In contrast, the light gradient boosting machine (LGBM) and the random forest (RF) models often performed well in predicting appendicitis diagnosis, management, and severity with and without feature selection methods. These results imply that the LGBM and RF are strong at ignoring noisy, useless, and/or redundant features in this application. These observations are expected as the LGBM and RF are both based on collections of decision tree classifiers, which are inherently capable of ignoring weak features, as they strongly tend not to be selected for in the splitting process that creates decisions at each split in each base learner decision tree. Our results also demonstrate potential from our novel redundancy-aware feature selection (FS) method, contributing to high-performing models in both management and severity prediction, based on relatively small feature sets. Such solutions have the potential to improve the explainability of our AI technologies through a greatly reduced feature set size. For management, our redundancy-aware FS method identified 11 features (see Appendix I), with the leading features relied upon being peritonitis, white blood cell count, body temperature, weight, severity, and C-reactive protein. For severity, our redundancy-aware FS method identified five features (see Appendix M): peritonitis, coughing pain, body temperature, thrombocyte count, and C-reactive protein. These feature sets are highly predictive of management and severity, respectively, and so may represent useful information for clinicians responsible for patient management.

4.2. Discussion of GANDALF Results

GANDALF [44], an emerging deep learning architecture designed for tabular data, upon which deep learners have traditionally been underperformers, was assessed as an addendum to this study. Results demonstrate overall good performance from GANDALF; however, it was not the leading AI technology in our trials in terms of predictive accuracy. That said, GANDALF was very competitive in predicting management and severity, especially in terms of AUROC scores, implying the method is capable of creating internal embeddings of feature representations that assist in delineating between our target classes of interest as assessed by AUROC. It is well known that deep learners in particular benefit from large sample sizes to train upon, and so it is expected that in this application, with relatively few samples compared with many other machine learning studies, GANDALF is disadvantaged.

4.3. Predictive Significance of US Image Features

For predicting diagnosis, the performance tables in Appendix H and Appendix J consistently show a decrease in predictive accuracy of our top-performing models of 10–20% in both performance on holdout set and 5-fold cross-validation on the holdout set when withholding US image features. The significant drop in performance suggests information in the US image features is important for diagnosing appendicitis and contributes to a mitigation in how biassed the resultant models are from predicting ground-truth diagnoses. When predicting management, there is no drop in performance across our top-performing models when US image features are removed (see Appendix C and Appendix L). Similar findings were observed in comparative performance when US image features are included/excluded when predicting Severity (see Appendix G and Appendix I). These results suggest US image-derived features are either not useful in predicting the management and severity of pediatric appendicitis or are redundant to non-US-based features available in this dataset.

4.4. Literature Comparison

The appendicitis dataset relied upon has been updated since the earliest publications focused on this work [35,37], supporting a more statistically powered analysis with 781 patients in our study, as opposed to 430 patients [35] and 579 patients [37]. Thus, any comparisons between our findings and the foundational papers on this dataset in the literature [35,37] are not exact comparisons due to the dataset size, as well as inevitably employing different validation strategies. Having more samples in the total dataset is expected to help improve predictive accuracies, as more samples are available for training, which is well known to improve the performance of machine learning models generally. Also noteworthy, our validation approach involved reserving 40% of the samples in the dataset for our hold-out testing to help ensure reliability. This has the potential to reduce our reported predictive accuracy, as only 60% of the total samples were available for training in a relatively small dataset. Previous work on this dataset employed validation with 10% of samples included in the testing pools [35]. Results indicate that our leading models produced AUROC scores of 0.993 for predicting diagnosis, 0.973–0.984 for predicting management, and 0.896–0.931 for predicting severity across our two validation methods. This compares favourably with literature work on a subset of this dataset [35], which reported AUROC scores of 0.96 (+/−0.01) for predicting diagnosis, and 0.94 (+/−0.02) for predicting management; however, our findings were approximately the same for predicting severity, with the literature reporting AUROC scores of 0.91 (+/−0.07) [35]. Our results are roughly in line with those from the literature [35], with some noteworthy improvements in AUROC scores in predicting diagnosis and management. The improved performance of our models may be attributable to the increased sample size available in our dataset, and features of df-analyze, such as Optuna hyperparameter tuning, extensive feature selection techniques evaluated, using state-of-the-art scikit learn implementations of learning machines in Python (as opposed to relying on R), and consideration of lightweight high-performing algorithms such as the light gradient boosting machine (LGBM), and LGBM-based embedded feature selection. It should also be noted that two additional studies have been conducted on the updated dataset used in this analysis, focused exclusively on diagnosis [38,39]. This includes an approach achieving 94.5% accuracy with the random forest [38], and an approach based on the Hybrid Bat algorithm achieving 94% accuracy [39]. An additional study was based on recursive feature elimination and the random forest, which did not report overall accuracies [39], but reported AUROC scores for diagnosis of 0.96 +/−0.02 [40]. In contrast, our approach, enhanced by Optuna hyperparameter tuning and feature selection, compares favourably with 98.1% accuracy and AUROC scores of 0.993 for diagnosis.

4.5. Future Work and Limitations

An interesting consideration that has resulted from this study relates to interactions between the target variables. There is potential value, for instance, in predicting diagnosis with and without knowledge of management, or predicting management with or without knowledge of the diagnosis. For instance, diagnosis is often not established until after surgical management, so the method selected for surgical management can potentially be a useful informative feature assisting in the predictive capacity of diagnosis. Conversely, management may benefit from knowledge of the final diagnosis if it is available. However, in situations where it is not (the patient’s final diagnosis is unknown), but the patient is proceeding to management/surgery, then a management prediction algorithm should not be informed as to the patient’s diagnosis when creating a technology to be relied upon clinically. Confounding issues, such as these, are important when creating a series of technologies to be relied upon for aiding clinical management of patients. Models can be created with and without knowledge of the other target variables of interest; thus, appropriate AI models can theoretically be relied upon clinically based on the availability (or not) of given target variables that may be helpful in informing prediction. Furthermore, AI technologies can be created that input a prediction of a target variable assessed by a different AI model. While this study is a research endeavour, and the models developed have not been clinically deployed, it is important for AI developers in medical applications to appreciate the various trade-offs and varying clinical utility of nearly identical models trained on almost the same set of potential predictor variables. Preliminary experiments indicate that high-performing models can be built with df-analyze for these applications with and without the inclusion of alternate target variables as features informing prediction. Limitations include that this study was performed on a single dataset, as this is the only dataset of its type publicly available; thus, independent dataset validation was not possible. Future work should involve validation on additional independent datasets in different healthcare environments to assess their generalisability across diverse pediatric populations. Future work should also involve consideration of emerging learning algorithms, such as updates to deep learners focused on tabular data.

5. Conclusions

We investigated the use of several machine learning technologies exhaustively combined with a variety of feature selection algorithms for predicting the diagnosis, management, and severity of pediatric appendicitis, with and without the inclusion of ultrasound image-derived features. Ultrasound image features were found to be important for maximizing accuracy when performing diagnostics, providing support for the value of imaging features in mitigating bias in the AI model relative to ground-truth diagnoses. However, findings imply that image-derived features are not as useful when predicting the management and severity of the condition. A variety of leading learning machines were presented based on variable subsets of the features identified by our redundancy-aware FS, providing detailed information that can potentially aid in the explainability of our AI models. The methods outlined in this study produced AI technologies with robust predictive potential in three applications focused on pediatric appendicitis as assessed by the area under the receiver operating characteristic curve. The technologies developed in this study could potentially help identify and manage young patients with suspected appendicitis. Advantages of the approach taken in this study include the consideration of a novel redundancy-aware step-up feature selection algorithm, consideration of an emerging deep learner optimized for tabular data (Gandalf), assessment of the value of US-derived features, and the creation of highly accurate AI models for three applications. Disadvantages include that this study did not consider convolutional neural networks that process the US images available in this dataset, as well as being reliant on a single dataset for all analyses. Future work will investigate the role of image analysis deep learners, including on additional datasets.