A Web-Based Dynamic Nomogram to Predict the Risk of Methicillin-Resistant Staphylococcal Infection in Patients with Pneumonia

The aim of this study was to create a dynamic web-based tool to predict the risks of methicillin-resistant Staphylococcus spp. (MRS) infection in patients with pneumonia. We conducted an observational study of patients with pneumonia at Cho Ray Hospital from March 2021 to March 2023. The Bayesian model averaging method and stepwise selection were applied to identify different sets of independent predictors. The final model was internally validated using the bootstrap method. We used receiver operator characteristic (ROC) curve, calibration, and decision curve analyses to assess the nomogram model’s predictive performance. Based on the American Thoracic Society, British Thoracic Society recommendations, and our data, we developed a model with significant risk factors, including tracheostomies or endotracheal tubes, skin infections, pleural effusions, and pneumatoceles, and used 0.3 as the optimal cut-off point. ROC curve analysis indicated an area under the curve of 0.7 (0.63–0.77) in the dataset and 0.71 (0.64–0.78) in 1000 bootstrap samples, with sensitivities of 92.39% and 91.11%, respectively. Calibration analysis demonstrated good agreement between the observed and predicted probability curves. When the threshold is above 0.3, we recommend empiric antibiotic therapy for MRS. The web-based dynamic interface also makes our model easier to use.

The risk of MRS infection in pneumonia needs to be addressed when designing empirical therapy regimens.According to previous studies, risk factors for MRS infection included tobacco use, chronic obstructive pulmonary disease, recent antibiotic exposure, illicit drug use, and chest tubes [2,7].MRSA pneumonia was also associated with male gender, age over 74, diabetes, a recent nursing home or hospital stay, recent exposure to fluoroquinolones or antibiotics for Gram-positive organisms, and severe pneumonia [8].Therefore, clinical settings need to be based on these risk factors to cover MRS early to improve prognosis in these patients.
Coverage treatment based on guidelines helps provide early MRS treatment; however, related risk factors need to be considered more.The American Thoracic Society (ATS) and the British Thoracic Society (BTS) both made suggestions about risk factors for treating MRSA.These included having used an intravenous antibiotic in the past 90 days, having severe pneumonia, having been infected with MRSA before, being treated in units with more than 10-20% MRSA isolates [9], having a shadow on both lungs, and having frequent lung cavitation [10].The goals of the guidelines help improve targeted treatment for MRSA; however, the frequency of MRSA and MRCoNS infection and the outcome have been the topic of much discussion.Several studies have been developed from these recommendations to identify additional risk factors beyond those already included in existing guidelines to better identify patients at higher risk, which could help tailor treatment decisions more precisely, reducing the risk of overtreatment.
Nomograms are graph-based tools used to calculate the predicted probability of an event of interest using the input parameters.Compared to traditional forms, webbased dynamic nomograms are gaining widespread use in prognosis and diagnosis of pulmonology and other disciplines thanks to their user-friendliness and accessibility [11,12].The application simplifies the process by integrating the underlying regression formula of the prediction model into its calculation, thereby enabling rapid results for predicted risk.Its convenience may allow for tailored clinical decisions [11].
This study aimed to develop a predictive model to estimate the individual risk of MRS infection in patients with pneumonia and build a dynamic nomogram to facilitate clinical use.

Study Population and Design
This observational study enrolled pneumonia patients who were admitted to Cho Ray Hospital from March 2021 to March 2023.The inclusion criteria included (1) being over 18 years of age, (2) having a confirmed diagnosis of pneumonia, and (3) having MRS risks according to ATS recommendations.These included prior intravenous antibiotic use within 90 days, severe pneumonia, especially if intubated, and shock sepsis.Patients with COVID-19 infections or poor-quality specimens were excluded.

Data Collection
Baseline demographics included age, sex, comorbidities, previous infection with MRS, severity of pneumonia, clinical symptoms, and characteristics of chest X-rays or computed tomography scans.All were recorded when the patients had pneumonia.

Definition of Variables
Pneumonia was identified by the presence of a new or worsened lung infiltration on a chest X-ray and meeting at least two of the following criteria: temperature above 38 • C, leukocytosis (≥12,000 WBC/mm 3 ) or leukopenia (≤4000/mm 3 ), and purulent sputum [2,13].
Comorbid illnesses were identified by the Charlson comorbidity index and using ICD-10 cm codes [15].The method to calculate the CCI score has already been mentioned elsewhere [14,15].The diagnosis of MRS pneumonia was made based on clinical characteristics, laboratory testing, imaging findings, microbiological tests, and treatment outcomes.MRS (MRSA and MRCoNS) that was detected from blood or a respiratory source (such as sputum, endotracheal aspirate, bronchoalveolar lavage, or pleura) through culture and/or real-time PCR [2,13] are deemed clinically significant infection.

Statistical Analysis
Continuous variables were summarized in means and standard deviations if normally distributed, or median and percentile range if otherwise.Measurement data were analyzed by t tests (normal distribution) and Mann-Whitney U tests (non-normal distribution).Categorical variables are shown as numbers and percentages for comparison using the χ 2 test.

Predictive Model Development and Validation
The initial dataset was used for model development.
Stepwise selection and Bayesian model averaging (BMA) were applied to build a prediction model.Covariates were chosen based on p values and Bayesian information criterion [16,17].
For stepwise selection, univariable logistic regression was used in the dataset to identify the independent risk factors for MRS infection.Three approaches were considered for the final model.Models 1 and 2 included all significant predictors from univariable logistic regression with significant p-value levels below 0.1 and 0.05, respectively.In contrast, model 3 retained only those that were significant (p-value < 0.1) in the multivariable logistic regression of model 1 [16].
In BMA approach, 5 models with the lowest Bayesian information criterion were considered [16,17].
The model's discrimination for MRS infection was evaluated using the receiver operating characteristic (ROC) curve and the area under the curve (AUC).With the threshold, we chose a high sensitivity value, and the net benefit of using the MRS model criteria was significantly different for all treatments.
The calibration curve determined the model's goodness of fit.Decision curve analysis and a clinical impact curve were employed to assess the clinical impact of the model.The model with the best AUC value in the dataset was chosen as an optimal model, which was then translated to a static and a web-based nomogram.
A total of 1000 bootstrap samples (with replacement) were employed to evaluate the model.The potential impact of utilizing the MRS model for screening MRS testing was measured.Initially, the infected case rate was used as a testing threshold for the MRS model to determine the expected infection risk.Within the bootstrap samples, the AUC and sensitivity of the model were calculated.The net benefit of employing the MRS model criteria was compared to that of treating all patients [18].
All statistical analyses were performed using the R Studio program version 4.2.3.R software (rms, tidyverse, caret, and BMA package) to build the nomogram models.Then, DynNom and Shiny packages were used to build a web application.

Predictive Model Development and Validation
All baseline characteristics in the dataset were included in the univariable logistic regression analysis.At the a priori significance level p-value of 0.1, prior intravenous antibiotic use within 90 days, severity of pneumonia, respiratory failure, tracheostomy or endotracheal tube, urinary catheterization, central venous catheterization, nasogastric intubation, skin infection, and pneumatoceles were associated with the risk of MRS infection (Table 1).Furthermore, skin infection and pneumatoceles were significantly associated with the risk of MRS infection in the multivariable logistic regression analysis (Table 2).As shown in Figure 2, model 7 with four predictors (tracheostomies and endotracheal tube, skin infection, pleural effusions, and pneumatoceles) outperformed other models with regard to their ROC and sensitivity values in the dataset.ROC analyses showed an area under the curve of 0.70 (0.63-0.77) and 0.71 (0.64-0.78) (Figure 3A,B) in the two datasets.As shown in Figure 3C, the net benefit of the decision curve for the nomogram was higher, suggesting that our nomogram could improve clinical prediction.Additionally, the calibration curve revealed a close alignment between the predicted probability and the actual estimates (Mean Absolute Error = 0.02) (Figure 3D), affirming the model's goodness of fit.Furthermore, at the threshold of 0.3, the model showed a sensitivity of 92.39% and 91.11% in the dataset and 1000 bootstrap samples, respectively (Table 3).With this threshold, we reduced24 patients who did not need to receive anti-MRSA therapy (Table S1).At the threshold of 0.8, the model showed a specificity of 100% and 95.73% in the order of samples given (Table 3).The nomogram was thereby developed based on model 7 and is available at "https://bvv2023.shinyapps.io/MRSapp/(accessed on 11 March 2024)" (Figure 4).

Discussion
In this study, prior intravenous antibiotic use within 90 days, severity of pneumonia, respiratory failure, tracheostomy or endotracheal tube, urinary catheterization, central venous catheterization, nasogastric intubation, skin infection, and pneumatoceles were associated with MRS pneumonia at a p-value of below 0.1.These risk factors for MRS infection are in line with the current literature [7,8].According to the American Thoracic Society (ATS), hospitalization history, especially frequent hospital admissions and prolonged hospital stays, is independently associated with MRSA infection [19].Additionally, colonization and infection of the skin by Staphylococcus spp.also increase the risk of pneumonia and bloodstream infections [20].Patients with tracheostomies and endotracheal tubes were vulnerable to biofilm-producing and highly resistant bacteria quickly colonizing the surfaces of the tracheostomy tubes.A retrospective study of intensive care units over 10 years revealed that Staphylococcus spp., including CoNS and S. aureus, were 1 of 14 common microorganisms isolated from tracheostomy tubes [21].Furthermore, pleural effusions, pneumatoceles, and pneumothoraces are also common findings in radiological investigations and a computed tomography scan in pneumonia caused by S. aureus [22].
Scoring systems and nomograms are necessary and convenient for predicting MRS infection.A cohort study derived a risk score for MRSA infection from 5975 patients, with 14% having MRSA infections.The score consisted of eight variables and a possible total score of 10.The MRSA infection probability was lower by 10% at a score of 0 to 1 and could be above 30% when the score was 6 or greater [7].Another retrospective study showed that a history of MRSA infection (OR 5.6, 95% CI 1.56-20.63)and osteomyelitis (OR 2.5, 95% CI 1.00-6.79)was linked to a higher chance of MRSA isolation (C statistic of 0.7).Previous work also introduced a predictive nomogram with moderate to good discrimination for MRSA infection in patients with infected foot ulcers [23].In this study, we developed a model and used the bootstrap method to validate the nomogram with an AUC value of 0.70 and a sensitivity of 92.39% at a threshold of 0.3 in the dataset (Figures 2 and 3A,B and Table 3).Our model demonstrated good calibration, as substantial agreement between the observed and predicted probability curves was found in the calibration analysis (Figure 3C,D).
Early diagnosis and timely treatment play a crucial role in MRS pneumonia management.MRSA bacteremia substantially increases hospital costs, as patients with this infection required an additional 12,818 (95% CI 7246-19,966) hospital bed days, which consequently cost the hospitals an extra USD 24,366,741 (95% CI USD 13,774,548-37,954,686) per year [24].Therefore, ATS has recommended covering MRSA early in patients who have had prior intravenous antibiotic use within 90 days, severe pneumonia, or a previous infection with MRSA [9,13].Regarding the pathogen's prevalence, however, among 3562 cases, only 5.2% were positive for MRSA.The recommended empirical anti-MRSA coverage over a low infection prevalence might render up to 94.8% of patients overtreated [25].Among 88,605 patients admitted to the hospital, empirical anti-MRSA therapy was given to 33,632 individuals (38%).When compared to standard treatment alone, the addition of empirical anti-MRSA therapy alongside standard therapy showed a significant association with increased adjusted risks of death (95% CI 1.3-1.5),kidney injury (95% CI 1.3-1.5),secondary Clostridium difficile infections (95% CI 1.3-1.9),vancomycin-resistant Enterococcus spp.infections (95% CI 1.0-2.3),and secondary Gram-negative rod infections (95% CI 1.2-1.8)[26].Furthermore, CoNS are one of the major opportunistic pathogens, and there are fewer reports of these pathogens in pneumonia; however, it was indicated that the antibiotic resistance, failed rate of treatment, and 30-day mortality for CoNS are higher than those for other organisms [27].Obviously, a pathogen-specific treatment should be favored over cover therapies in view of treatment costs and adverse effects such as kidney injury, secondary infection, and antibiotic resistance.Our model had a good net benefit, as shown in the decision curve, implying greater benefit when applied to guide treatment (Figure 3C).In patients with pneumonia who had prior intravenous antibiotic use within 90 days, severe pneumonia, or previous MRS infection plus a positive prediction of our MRS app, we reduced 24 cases of pneumonia with coverage treatment for MRS of the ATS recommendation (Table S1).
Our research should bring some benefits to clinical practice.ATS and BTS recommendations helped to determine the best empiric therapy for MRS infection, but imbalances between cost effectiveness, complications, and positive outcome have been the subjects of many debates, so our model could serve as a screening tool plus recommendations.In addition, with a high sensitivity, a good AUC, a high net benefit in decision curve analysis, and good calibration, our model represents a useful and effective prediction tool.The model was made available as a web-based interactive tool to ease clinical use and enhance its accessibility.The number of MRS cases in our research was sufficient to build a model that responded to at least 200 observations [28] and complied with the rules of thumb (1:10) [29] or Claudia Beleites (5-25 independent samples per class) [30] and other predictive research [31].Furthermore, the quality of the model is measured based on two factors, including accuracy and precision of estimation.In our research, we used many methods to select variables for building and comparing models and used bootstrapping to estimate the precision of the unknown population characteristics.
This study may be subject to limitations that warrant consideration.Firstly, the research was conducted at a single center, which inherently restricts the generalizability of the model.Secondly, the sample size was relatively small in relation to the number of covariates included, potentially leading to an overfitted model.It is crucial to conduct additional validation studies in diverse settings to corroborate the findings and enhance the robustness of the model.

Conclusions
This study reaffirmed the significance of prior intravenous antibiotic use within 90 days or severe pneumonia as important risk factors for MRS pneumonia.The nomogram model demonstrated commendable performance in predicting MRS infection.Exploring the potential benefits of implementing this model for early administration of MRS antibiotics merits further investigation.

Figure 4 .
Figure 4.A dynamic nomogram to predict the risk of methicillin-resistant Staphylococcal infection in patients with pneumonia.MRS-Methicillin-resistant Staphylococcus spp.

Figure 4 .
Figure 4.A dynamic nomogram to predict the risk of methicillin-resistant Staphylococcal infection in patients with pneumonia.MRS-Methicillin-resistant Staphylococcus spp.

Table 1 .
Demographics and clinical characteristics of the study.

Table 1 .
Demographics and clinical characteristics of the study.
Data are presented as the mean and SD or number and proportion of participants in each diagnosis category (%) where data were available.MRS-methicillin-resistant Staphylococcus spp.; AIDS-acquired immune deficiency syndrome; MRSA-methicillin-resistant Staphylococcus aureus; BMI-body mass index; HAP-hospital-acquired pneumonia; VAP-ventilator-associated pneumonia; CAP, community-acquired pneumonia; SD-standard deviation.

Table 2 .
Multivariable logistic regression analysis of the dataset.

Table 3 .
Sensitivity and specificity of the dataset and 1000 bootstrap samples to different threshold values.