Head-to-Head Comparison: P-POSSUM and ACS-NSQIP® in Predicting Perioperative Risk in Robotic Surgery for Gynaecological Cancers

Simple Summary Estimating the risk of postoperative complications is important for shared decision-making with the patient. This also helps multidisciplinary teams to plan preoperative management and postoperative care and predict duration of recovery. In this retrospective pilot study, we aim to evaluate the accuracy and reliability of the P-POSSUM and ACS-NSQIP surgical risk calculators in predicting postoperative complications in gynaecological–oncological (GO) robotic surgery (RS). We were able to demonstrate that the ACS-NSQIP risk tool showed 90% accuracy in prediction of five major complications and mortality. Abstract Purpose: In this retrospective pilot study, we aim to evaluate the accuracy and reliability of the P-POSSUM and ACS-NSQIP surgical risk calculators in predicting postoperative complications in gynaecological–oncological (GO) robotic surgery (RS). Methods: Retrospective data collection undertaken through a dedicated GO database and patient notes at a tertiary referral cancer centre. Following data lock with the actual post-op event/complication, the risk calculators were used to measure predictive scores for each patient. Baseline analysis of 153 patients, based on statistician advice, was undertaken to evaluate P-POSSUM and ACS-NSQIP validity and relevance in GO patients undergoing RS performed. Results: P-POSSUM reports on mortality and morbidity only; ACS-NSQIP reports some individual complications as well. ACS-NSQIP risk prediction was most accurate for venous thromboembolism (VTE) (area under the curve (AUC)-0.793) and pneumonia (AUC-0.657) and it showed 90% accuracy in prediction of five major complications (Brier score 0.01). Morbidity was much better predicted by ACS-NSQIP than by P-POSSUM (AUC-0.608 vs. AUC-0.551) with the same result in mortality prediction (Brier score 0.0000). Moreover, a statistically significant overestimation of morbidity has been shown by the P-POSSUM calculator (p = 0.018). Conclusions: Despite the limitations of this pilot study, the ACS-NSQIP risk calculator appears to be a better predictor of major complications and mortality, making it suitable for use by GO surgeons as an informed consent tool. Larger data collection and analyses are ongoing to validate this further.


Introduction
Evidence suggests that despite undergoing a preoperative consultation and participating in the informed consent process, patients frequently exhibit limited understanding regarding potential postoperative complications [1][2][3].There is cumulative evidence in the literature to suggest that decision aids often increase patients' understanding of proposed treatment and interventions and assist in the informed consent process [4].Hence, estimating the risk of postoperative complications is crucial in the shared decision-making process.It helps clinicians and multidisciplinary teams to plan preoperative management and postoperative care, including inpatient stay, and to predict likely duration of recovery.
Over the last decade, the burden of cancer incidence as well as mortality has grown rapidly worldwide and is reflected in the rates of gynaecological cancer increase, with cervical and uterine cancer incidence and mortality being the seventh most common cancers in women [5].With the increasing rates of GO operations, especially technically challenging minimally invasive surgery (MIS) [6,7] in obese patients and patients with multiple comorbidities, it is even more important to be able to accurately predict the likelihood of perioperative complications and involve patients in joint decision-making about their surgery.Iyer et al. were able to demonstrate, in their prospective multi-centre study, that the intraoperative rate of complications amongst patients undergoing surgery for cancer was 5.4%, whereas the postoperative complications rate was 27.1% [8].
According to a recent review, cardiopulmonary exercise testing (CPET) is providing an objective evaluation of cardiorespiratory fitness and functional capacity resulting in individualised risk profiles, which could guide shared decision-making [9].However, although it is increasingly being used in the UK to evaluate preoperatively the postoperative morbidity, it is a very expensive method with limited capacity [10].
Several online risk scoring tools are in clinical use, all of which attempt to predict the risk of postoperative morbidity and mortality during major surgical procedures.One such calculator, originally developed by Copeland et al. in 1991, is the Physiologic and Operative Severity Score for the Enumeration of Mortality and Morbidity (POSSUM) risk scoring [11], subsequently modified to Portsmouth POSSUM (P-POSSUM) to provide a more accurate prediction [12], which is widely accepted in the UK for postoperative mortality and morbidity risk prediction.P-POSSUM uses the preoperative physiological scores and intraoperative surgical scores of patients for generation of postoperative risk scores [11].However, published data across different specialties, including GO, suggest that it may overestimate risk, especially the mortality rates [13][14][15][16][17], which may cause undue patient anxiety, influence the surgical decision-making and potentially prolong the length of stay (LOS).Moreover, P-POSSUM requires intraoperative parameters for morbidity and mortality calculation, which in turn makes the prediction less accurate.
The American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) surgical risk calculator was developed in 2013 based on the NSQIP database, which was designed to measure and improve the quality of surgical care [18].The database was derived from standardised prospective surgical data collection, across more than 700 hospitals and nine specialties in the USA, and further evaluation of the postoperative complications.The calculator derived from this database is a validated web-based tool requiring 19 preoperative risk factors for the prediction of LOS and 11 postoperative outcomes within 30 days of surgery (plus 2 outcomes in surgery involving gastrointestinal tract-ileus and anastomotic leak), which allows surgeons to receive a customised, patientspecific risk for their surgery [19].
A number of studies have explored the applicability and efficacy of the ACS-NSQIP surgical risk calculator in the context of GO [20][21][22][23].However, none of these studies have compared ACS-NSQIP to any other tools and there have not previously been attempts to analyse the tool in a cohort of patients undergoing RS.
The objective of our study was to explore the validity and suitability, as well as to compare the performance, of the P-POSSUM and ACS-NSQIP surgical risk calculators in the setting of GO minimally invasive robotic-assisted operations and hereby we present our pilot study that shows the performance of ACS-NSQIP compared to P-POSSUM.

Ethics and Approvals
This project was approved by the Audit and Quality Improvement department as part of the Research and Development office of the Royal Surrey NHS Foundation Trust (RSNFT) as part of the studentship with University of Brighton for a retrospective analysis of 1500 patients undergoing RS and 1500 patients undergoing open surgery.

Cohort Selection
Yearly operation lists were obtained from the designated electronic GO database of RSNFT.All patients were divided by the year of the operation (2009-2020) and by the type of surgery they had undergone (open/robotic/laparoscopic).Women who underwent other diagnostic procedures have been excluded from this study, as well as cases where there was not enough information for the ACS-NSQIP score calculation or the P-POSSUM score.While data collection is ongoing, the first 153 robotic cases were felt to be adequate for statistical analysis for this pilot and hence the first 153 robotic cases with complete data have been analysed.
In this report, we present the preliminary analysis of 153 patients who had undergone Da Vinci-assisted RS for suspected or confirmed gynaecological malignancies at RSNFT, based on the intention to treat analysis.

Data Collection
For all patients undergoing surgery at our cancer centre, data are recorded prospectively on a dedicated database.For each patient, the electronic database record was retrieved to collect P-POSSUM scores, along with the 19 pre-operative parameters required to calculate the ACS-NSQIP risk (Table 1).The missing data were collected from paper patient records.Following their surgery and discharge, patients have the option of contacting their local hospital, general practitioner (GP) or RSNFT for any concerns, and all patients at RSNFT are invited for a follow-up appointment within 30 days postoperatively when the course of recovery +/− further management/surveillance is discussed with the treating specialist.All readmissions and presentations to different trusts, visits to GP and treatments commenced within that period are discussed with the patient.Clinical nurse specialists are also contacted by other trusts when patients present/are admitted to other hospitals.These events are documented within the electronic database and/or patients' records.The electronic database and paper records were also analysed to identify any actual postoperative 30-day complications and the LOS, which were recorded along with the rest of the data.Morbidity was defined by either the Clavien-Dindo score and/or the Postoperative Morbidity Survey (POMS) score within the 30 days after the surgery.All complications above Clavien-Dindo IIIa were classified as serious complications.
In addition to the variables listed above, a number of tumour characteristics were collected for each of the patients for general purposes, such as tumour site, neoplasm type and subtype, FIGO stage and grade.
The required parameters were entered into the online ACS-NSQIP surgical risk calculator tool [24] for each patient, and the Common Procedural Terminology (CPT) code was selected as accurately as possible.In accordance with previous studies, if there was more than one procedure within the surgery, then more than one CPT code was selected and the scores for each were calculated; the highest percentage of each of the complications was selected for analysis [20,23].Finally, the estimated LOS and risk of 11 postoperative outcomes were calculated (Table 2)

Anastomotic leak *
This includes air, fluid, GI contents or contrast material.With or without treatment.The presence of an infection/abscess thought to be related to an anastomosis, even if the leak cannot be definitively identified as visualised during an operation, or by contrast extravasation, would still be considered an anastomotic leak if this is indicated by the surgeon.
* For operations involving GI tract.

Statistical Analysis
Firstly, baseline demographic variables were analysed to describe the selected cohort.Patients were categorised by age, ASA class, degree of obesity and types of comorbidities.Tumours were categorised as being malignant or benign, as well as by the site of neoplasm.
Statistical analysis was undertaken using Statistical Package for the Social Sciences (SPSS) version 24.0.The difference between each of the predicted postoperative outcomes, hence the definitiveness of the ACS-NSQIP surgical risk calculator, and the actual complications were measured using the Brier score.The closer the reliability term was to zero, the more reliable was the prediction of morbidity, mortality by P-POSSUM and ACS tools or specific complication by the ACS tool (https://riskcalculator.facs.org/RiskCalculator/PatientInfo.jsp,accessed 2 March 2023).
Within the logistic regression model, we also determined the c-statistic standard measure of the predictive accuracy, which is equivalent to the area under the Receiver Operating Characteristic (ROC) curve when the outcome is binary [25].The ROC curve values range from 0 to 1, where 0 indicates an ineffective model and 1 means that the model is perfect for the prediction of complications [26].

Results
In this report, we present results for the initial 153 patients.These patients had undergone a Da Vinci RS for gynaecological malignancy at RSNFT.Preoperative demographic data are presented in Table 3.The vast majority of the patients were under 65 years of age (56.2%) and were classed as ASA 2 (66%).All patients had undergone planned procedures with no emergencies.Similar numbers of patients were of normal weight and overweight (30% and 27%, accordingly) and the majority of the patients were classed as obese (class 1-3) (43%).In terms of comorbidities, a substantial proportion of the patients were found to have hypertension (35.3%), followed by 15% of smokers and 9.8% of non-insulin-dependent diabetes mellitus patients.Only a small number of patients were suffering from insulindependent diabetes mellitus (2.6%) and COPD (0.6%).
The majority of patients underwent surgery for malignancy (85%); 4.6% had borderline condition and 10.4% were found to have a benign tumour.A total of 64.7% of the tumours were located in endometrium, 29.4% in the cervix, 5.2% in the ovaries and one patient had a primary in another site (Table 4).
The main cohort of patients had undergone hysterectomy with or without salpingoophorectomy and a staging procedure, lymphadenectomy and/or omentectomy with or without hysterectomy and/or salpingoophorectomy.The breakdown of the patients by the type of surgery is shown in Table 4.
Of the 153 patients, 23 (15%) were found to have one or more complications and 13 (8.5%) had an event constituting a serious complication, as described previously.The breakdown of the actual postoperative complications is demonstrated in Table 5.As shown, the most frequent complications in the sequence of their frequency are UTI (n = 10), readmission (n = 7), pneumonia (n = 3) and SSI (n = 3).Rarer complications included return to theatre (n = 2), VTE (n = 2), renal failure (n = 1) and discharge to post-acute care facility (n = 2).There were no deaths or cardiac complications observed in this cohort of patients.When comparing the morbidity prediction ability of P-POSSUM and ACS-NSQIP surgical risk calculators, ACS-NSQIP showed a better predictive value than P-POSSUM with AUC 0.608 against 0.551 (Figure 1).ACS-NSQIP was also found to statistically significantly predict postoperative complications better than P-POSSUM in patients with ASA class I and II.When comparing the morbidity prediction ability of P-POSSUM and ACS-NSQIP surgical risk calculators, ACS-NSQIP showed a better predictive value than P-POSSUM with AUC 0.608 against 0.551 (Figure 1).ACS-NSQIP was also found to statistically significantly predict postoperative complications better than P-POSSUM in patients with ASA class I and II.The ACS-NSQIP calculator was found to perform best in predicting the risk of postoperative VTE (AUC = 0.793) and pneumonia (AUC = 0.657), followed by a worse prediction of readmission (AUC = 0.587) and UTI (AUC = 0.515).(Figure 2).
When comparing the mortality predictions by ACS-NSQIP and P-POSSUM, the best results were shown by ACS-NSQIP (Brier (P-POSSUM mortality) = 0.0014; Brier (ACS mortality) = 0.000; p = 0.084) (Figure 3a) (Table 6).Given the lack of deaths in the cohort, it is evident that the P-POSSUM risk calculator is overestimating the mortality prediction compared to the ACS risk calculator, which has accurately predicted the mortality.
P-POSSUM was also found to statistically significantly overestimate morbidity, similarly to mortality risks (p = 0.018), as shown on Figure 3b and Table 6.
Lastly, the performance of ACS-NSQIP in predicting individual complications was measured using the Brier score.The risk prediction tool showed the best accuracy in the prediction of major complications as follows: death, cardiac event, renal failure, VTE, return to theatre and SSI (Figure 4) (Table 7).
Given the lack of deaths in the cohort, it is evident that the P-POSSUM risk calculator is overestimating the mortality prediction compared to the ACS risk calculator, which has accurately predicted the mortality.

Discussion
To our knowledge, this study has been the first attempt to validate and compare the predictive value of surgical risk calculators, P-POSSUM and ACS-NSQIP, in the context of RS in GO.Previous studies have identified the lack of accuracy in the risk prediction of the P-POSSUM tool.It has been shown that P-POSSUM tends to overestimate mortality risk prediction in patients undergoing different types of surgery, specifically in GO [13,27], and our study has also confirmed that.This can affect the potential patients' understanding of possible outcomes of surgery and influence the recovery, as the psychological aspect plays a vital role in cancer patient recovery [28].Moreover, we have been able to show that P-POSSUM overestimates morbidity along with mortality scores amongst our cohort of patients, which makes it an unreliable and potentially harmful tool to be used in this group of patients.
On the other hand, the results of our study show that ACS-NSQIP would have higher predictive value and would be a better tool to be used in GO.A study by Murray et al. with 142 patients had shown that ACS-NSQIP has a potential applicability in GO, especially since it could be used as a visual aid when counselling patients before the operation [22].ACS-NSQIP has been shown to be an accurate predictor, especially for death and VTE, which is in agreement with the study by Szender et  Thigpen et al. in their retrospective study, which included a cohort of patients who had undergone total abdominal or laparoscopic hysterectomy for benign gynaecological disease, found that the ACS-NSQIP calculator was effective in prediction of VTE and SSI but did not perform well in other categories of complications [30].These previous studies seem to be in

Discussion
To our knowledge, this study has been the first attempt to validate and compare the predictive value of surgical risk calculators, P-POSSUM and ACS-NSQIP, in the context of RS in GO.Previous studies have identified the lack of accuracy in the risk prediction of the P-POSSUM tool.It has been shown that P-POSSUM tends to overestimate mortality risk prediction in patients undergoing different types of surgery, specifically in GO [13,27], and our study has also confirmed that.This can affect the potential patients' understanding of possible outcomes of surgery and influence the recovery, as the psychological aspect plays a vital role in cancer patient recovery [28].Moreover, we have been able to show that P-POSSUM overestimates morbidity along with mortality scores amongst our cohort of patients, which makes it an unreliable and potentially harmful tool to be used in this group of patients.
On the other hand, the results of our study show that ACS-NSQIP would have higher predictive value and would be a better tool to be used in GO.A study by Murray et al. with 142 patients had shown that ACS-NSQIP has a potential applicability in GO, especially since it could be used as a visual aid when counselling patients before the operation [22].ACS-NSQIP has been shown to be an accurate predictor, especially for death and VTE, which is in agreement with the study by Szender et [20].
Thigpen et al. in their retrospective study, which included a cohort of patients who had undergone total abdominal or laparoscopic hysterectomy for benign gynaecological disease, found that the ACS-NSQIP calculator was effective in prediction of VTE and SSI but did not perform well in other categories of complications [30].These previous studies seem to be in concordance with our current research with regards to the efficacy of preoperative prediction of VTE during the postoperative period.In contrast, Teoh et al. report that the ACS-NSQIP risk calculator performs poorly in predicting 30-day postoperative complications in GO patients, especially compared to the general and colorectal surgical population [21].In their study, a mixed population of patients who had undergone MIS (laparoscopic or robotic) was reviewed in contrast to our pilot study, where we have reviewed exclusively robotic cases.This could influence the result, as patients selected for RS cases often have multiple comorbidities and are considered more complex cases.Manning-Geist et al. in their study (n = 261) had also shown that ACS-NSQIP performed poorly in patients with ovarian cancer undergoing interval debulking surgery [23].
Past studies have also shown that the presence of the following parameters increases the risk of postoperative complications in GO operations: diabetes, any comorbidity, increasing ASA class, age and BMI.The risk also depended on whether the approach was laparoscopic or open [8].These listed parameters are all included in the ACS-NSQIP surgical risk calculator.Nonetheless, Iyer et al. also showed that intra-operative factors also play a significant role in the prediction of postoperative complications.Those include estimated blood loss, duration of surgery and previous abdominal surgery.[8] This could potentially be the reason the ACS-NSQIP tool has not shown such accurate predictive abilities in GO patients compared to the colorectal surgical cohort [31].Other reasons, as described by Rivard et al., could include the variety of complexity of procedures because of intraperitoneal disease burden, the need for a greater amount of CPT codes to be taken into account considering the complexity of the procedures and the frequent poor nutritional status of the GO patients weeks prior to the operations [20].

Limitations and Insights
Being a pilot, retrospective data collection, the authors acknowledge that this poses a limitation to our study, potentially resulting in underreporting of postoperative complications due to inadequate follow-up or insufficient documentation in medical records.
All attempts were made to obtain complete records to obtain the results from both the ACS and P-POSSUM calculators for comparison.Nonetheless, data collection is currently underway with a much larger cohort of patients undergoing both open and minimally invasive surgery as well as validation in a prospective setting.It has been difficult to account for rare postoperative events, considering the number of participants, but plans exist to overcome this limitation in the future.The demographic profile of Surrey reveals a population characterised by relatively favourable ageing and affluence in comparison to broader demographics observed across the United Kingdom, though obesity as a risk factor prevails.In the ongoing data analysis, we strategically integrate multi-centre datasets to mitigate potential biases from specific demographic profiles.Lack of case selection has resulted in cases across all stages being incorporated, which the authors acknowledge is a limitation as there is evidence that the complexity of surgery is related to the disease burden as well as patient factors.Once again, being a pilot study, it does not allow for a stratified data analysis, which will be possible with larger datasets in the future.This is the first study to analyse a separate cohort of patients who underwent Da Vinci robot-assisted operations, which has not been undertaken previously.Furthermore, a head-to-head comparison of the performance of two surgical risk calculators, P-POSSUM and ACS-NSQIP, in this cohort of patients, which is the main strength of this study, and has been undertaken at a tertiary referral cancer centre where patients are referred following a suspected or confirmed case of malignancy.
This preliminary analysis suggests a notable disparity in the prognostic accuracy between the ACS-NSQIP scoring system and the P-POSSUM model in the context of mortality prediction, discerning a distinct advantage in favour of the ACS-NSQIP scoring system, which demonstrated a null mortality rate, as opposed to the P-POSSUM model, which failed to anticipate such outcomes.This observation bears significant implications for our ongoing research endeavours, substantiating the heightened efficacy of the ACS-NSQIP scoring system in mortality prediction and thereby providing a more robust foundation for future investigations.

Conclusions
This pilot study, along with others, suggests that the P-POSSUM surgical risk tool overestimates morbidity and mortality scoring in GO patients undergoing RS and as such should not be the preferred risk calculator for this cohort of patients.The ACS-NSQIP surgical risk calculator informative value has been higher, especially in predicting major complications, and it may be used as an informed consent tool.However, future larger studies are necessary to further evaluate these prediction tools.

Figure 1 .
Figure1.The ROC area under the curve (AUC), which ranges from 0.0 to 1.0, is a diagnostic measure for evaluating the accuracy of a predictor.The diagonal green reference line has an AUC of 0.5 and means random guessing.This ROC curve represents the comparative analysis of ACS-NSQIP and P-POSSUM general morbidity prediction.

Figure 1 .
Figure1.The ROC area under the curve (AUC), which ranges from 0.0 to 1.0, is a diagnostic measure for evaluating the accuracy of a predictor.The diagonal green reference line has an AUC of 0.5 and means random guessing.This ROC curve represents the comparative analysis of ACS-NSQIP and P-POSSUM general morbidity prediction.

Figure 3 .
Figure 3. Brier score measures the accuracy of probabilistic predictions: the lower the Brier score is for a set of predictions, the better the predictions are calibrated.(a) Mortality prediction: ACS (Brier Score 0.000), P-POSSUM (Brier score 0.0014).(b) Morbidity prediction: ACS (Brier score 0.1827), P-POSSUM (Brier score 0.1359).

Figure 3 .
Figure 3. Brier score measures the accuracy of probabilistic predictions: the lower the Brier score is for a set of predictions, the better the predictions are calibrated.(a) Mortality prediction: ACS (Brier Score 0.000), P-POSSUM (Brier score 0.0014).(b) Morbidity prediction: ACS (Brier score 0.1827), P-POSSUM (Brier score 0.1359).
al., which demonstrated that the Brier score approached the threshold for death, pneumonia, UTI and VTE in cancer patients, although the patient cohort of n = 577 appear to have all had open surgery [29].Rivard et al. in their cohort of 1094 open cases were able to demonstrate a c-statistic value of >0.8 for death and 0.7 for cardiac events and renal failure [20].

Table 4 .
Breakdown of the patients by tumour site, staging and the type of operation undergone.
* Patient underwent two-stage procedure.

Table 5 .
Rates of complications which occurred within 30 days of the operation.

Table 6 .
Paired T-test sample statistics for morbidity and mortality prediction of P-POSSUM and ACS risk calculators.

Table 6 .
Paired T-test sample statistics for morbidity and mortality prediction of P-POSSUM and ACS risk calculators.

Table 7 .
Statistics for prediction of individual postoperative complications using the ACS risk calculator.
al., which demonstrated that the Brier score approached the threshold for death, pneumonia, UTI and VTE in cancer patients, although the patient cohort of n = 577 appear to have all had open surgery [29].Rivard et al. in their cohort of 1094 open cases were able to demonstrate a c-statistic value of >0.8 for death and 0.7 for cardiac events and renal failure