Proposal of a Standardized Questionnaire to Structure Clinical Peer Reviews of Mortality and Failure of Rescue in Pancreatic Surgery

Background: Quality management tools such as clinical peer reviews facilitate root cause analysis and may, ultimately, help to reduce surgery-related morbidity and mortality. This study aimed to evaluate the reliability of a standardized questionnaire for clinical peer reviews in pancreatic surgery. Methods: All cases of in-hospital-mortality following pancreatic surgery at two high-volume centers (n = 86) were reviewed by two pancreatic surgeons. A standardized mortality review questionnaire was developed and applied to all cases. In a second step, 20 cases were randomly assigned to an online re-review that was completed by seven pancreatic surgeons. The overall consistency of the results between the peer review and online re-review was determined by Cohen’s kappa (κ). The inter-rater reliability of the online re-review was assessed by Fleiss’ kappa (κ). Results: The clinical peer review showed that 80% of the patient mortality was related to surgery. Post-operative pancreatic fistula (POPF) (36%) followed by post-pancreatectomy hemorrhage (PPH) (22%) were the most common surgical underlying (index) complications leading to in-hospital mortality. Most of the index complications yielded in abdominal sepsis (62%); 60% of the cases exhibited potential of improvement, especially through timely diagnosis and therapy (42%). There was a moderate to substantial strength of agreement between the peer review and the online re-review in regard to the category of death (surgical vs. non-surgical; κ = 0.886), type of surgical index complication (κ = 0.714) as well as surgical and non-surgical index complications (κ = 0.492 and κ = 0.793). Fleiss’ kappa showed a moderate to substantial inter-rater agreement of the online re-review in terms of category of death (κ = 0.724), category of common surgical index complications (κ = 0.455) and surgical index complication (κ = 0.424). Conclusion: The proposed questionnaire to structure clinical peer reviews is a reliable tool for root cause analyses of in-hospital mortality and may help to identify specific options to improve outcomes in pancreatic surgery. However, the reliability of the peer feedback decreases with an increasing specificity of the review questions.


Introduction
Several studies have found relevant mortality in pancreatic surgery, which entails a need to investigate the underlying causes [1][2][3]. The hospital volume was identified as one of the most important risk factors of perioperative mortality. This effect is mainly based on a significantly lower rate of failure to rescue indicating a more sufficient complication management in high-volume centers [4][5][6][7][8]. Moreover, there is evidence that the experience of the surgeon has a significant influence on perioperative outcomes of pancreatic surgery [9]. As the quality of patient care varies widely among institutions, relevant morbidity and mortality may arise from a variety of other causes that are not detectable by measures of systematic clinical research. In such cases, however, quality management tools provide the possibility to identify the cause of death of individual patient cases. One known tool for the analysis and improvement of medical processes is the peer review process [10][11][12]. The peer review process focuses on the direct exchange of expert knowledge and the mutual assessment at eye level, which helps to generate opportunities for quality and safety improvement in future cases. During the peer review, cases with poor outcome (e.g., inhospital mortality) are selected and evaluated by other clinicians in order to determine root causes [12]. In the context of complex surgery, such as pancreatic surgery, it is unknown how reliable peer reviews reveal such root causes. In addition, a standardized approach for peer reviews of patient mortality following pancreatic surgery does not yet exist.
The aim of the present study was (1) to identify common categories of underlying (index) complications and causes of mortality in pancreatic surgery, (2) to develop a standardized review questionnaire and (3) to examine, if a peer review with a standardized questionnaire is suitable for root cause analyses of in-hospital mortality.

Study Cohort
All cases of in-hospital mortality following major pancreatic surgery (n = 86) at two high-volume centers were included ( Figure 1). Major pancreatic surgery was defined as one of the following types of resection: pancreatic head resection, segmental pancreatectomy, distal pancreatectomy and total pancreatectomy with or without additional vascular or adjacent organ resections. The study cohort included patients treated at the Department of Surgery of University Hospital Erlangen between January 2002 and December 2017 (annual volume in 2017: 90 major pancreatic resections; 4 surgeons) and at the Department of Surgery of University Hospital Dresden between January 1994 and December 2014 (annual volume in 2014: 127 major pancreatic resections; 5 surgeons). In accordance with the guidelines for human subject research, approval was obtained from the ethics committee at the Carl Gustav Carus University Hospital (decision number EK 404102018).

Clinical Peer Review
Two pancreatic surgeons that were not involved in patient treatment performed a comprehensive chart review of all included cases ( Figure 1). Anonymized case summaries containing necessary clinical data for a mortality review were compiled. For each case with suspected surgical cause of death, a causal chain including at least the underlying (index) complication and the ultimate cause of death was derived. Based on this root cause analysis, definitions for the categories of common index complications and causes of death were established. Subsequently, a questionnaire to standardize mortality reviews in pancreatic surgery was developed. This standardized review questionnaire was applied to all included patient cases. To analyze the delay in recognizing and treating complications, the time between the surgery and the first deviation (any item) from textbook outcome as well as the time between the surgery and the first clinical sign specific to the index complication was determined [13].

Online Re-Review
Using the research randomizer (www.randomizer.org, accessed date: 9 March 2020), 20 cases of the patient cohort were randomly selected to undergo online re-reviews by seven experienced pancreatic surgeons ( Figure 1). Theses re-reviews were based on the anonymized case summaries and the standardized review questionnaire.

Statistics
Data analysis was performed with SPSS software (SPSS, Inc., Chicago, IL, USA). Descriptive data are presented as number (%) or mean [range]. The relationships between surgical index complications and the categories of common surgical index complications and causes of death were graphically illustrated using a Sankey diagram. The distribution of the questionnaire responses according to patient characteristics was assessed by the Chi-square test. Comparisons of metric data were calculated with the Student t-test.
To assess the test-retest reliability between the peer review and the online re-review, the majority vote of each questionnaire item of the online re-review had to be determined. Subsequently, the test-retest reliability was examined by Cohen's kappa (κ). At last, the inter-rater reliability analysis using Fleiss' kappa (κ) statistics was performed to determine consistency among raters of the online re-review. A p value ≤ 0.05 was considered statistically significant.

Clinical Peer Review
Mean postoperative time to death was 34 days [range 1-146]. Cause of death was most often related to surgery (80%). In 20% of all cases, patients died from a non-surgical index complication (e.g., myocardial infarction and pulmonary embolism). The relationships and most important contributions between surgical index complications and causes of deaths are visualized in Figure 2. The most common surgical index complications leading to inhospital mortality were post-operative pancreatic fistula (POPF, 36%), post-pancreatectomy hemorrhage (22%), abdominal thrombosis (12%), delayed gastric emptying causing aspiration (10%) and anastomotic leakage other than pancreatic anastomosis (9%). These surgical index complications were attributed to four categories (62% anastomotic complications, 26% vascular complications, 9% other complications and 2% intraoperative complications). The causes of deaths that resulted from surgical index complications were categorized in abdominal sepsis with organ failure (62%), abdominal ischemia (17%), pulmonary sepsis with organ failure (12%) and hemorrhagic shock (9%), respectively. Non-surgical index complications are listed in Table 2. The first clinical sign specific to the index complication occurred significantly later compared to the first deviation from textbook outcome (p = 0.002) ( Table 3). This difference was also significant in cases of a surgical cause of death and POPF (p = 0.004 respectively p = 0.036). In 42% of the cases a delay in recognizing or treating the complication was observed ( Table 3).
The standardized review questionnaire derived from these data is given in Figure 3. Table 2. Non-surgical causes of death in patients with in-hospital mortality after pancreatic surgery.

Distribution of Questionnaire Responses According to Patient Characteristics and Surgical Parameters
The distribution of questionnaire responses according to patient characteristics and surgical parameters is given in Tables 4 and 5. Stratification of cases according to preoperative (oncological vs. non oncological, pre-existing diseases <2 vs. ≥2) and surgical parameters (pancreatic head resection vs. pancreatic left resection, vascular vs. no vascular resection, multivisceral vs. no multivisceral resection) showed that patients with more than one pre-existing diseases died earlier (40 vs. 30 days, p = 0.050) and more often due to pulmonary septic organ failure and less often due to hemorrhagic shock than those with no or one pre-existing disease (p = 0.010) ( Table 4). Moreover, the number of patients with a surgical cause of death was significant higher following pancreatic head resections as compared to those with left pancreatectomy (85% vs. 57%, p = 0.026). Patients with vascular resection had higher rates of vascular complications (50% vs. 16%, p = 0.012) and abdominal ischemia (40% vs. 8%, p = 0.012) compared to patients with no vascular resection (Table 5).

Online Re-Review
Out of 20 re-reviewed cases, seven were classified non-surgical and 13 were classified surgical by majority voting. The majority votes for the classification of the cause of death were attained with an average agreement of 76% (non-surgical) and 87% (surgical), respectively. In case of non-surgical classified deaths, majority decisions on the category of common index complications were reached with an average agreement of 73%. The average agreement on the type of index complication and the category of cause of death in surgical classified deaths was 84% and 61%, respectively. Cohen's κ was run to determine if there was agreement between the peer review and online re-review on all questionnaire items (Table 6). There was a near perfect agreement between the peer review and the online re-review on the category of death (surgical vs. non-surgical; κ = 0.886 (p < 0.001); 95% confidence interval (CI 0.670; 1.101)), a substantial strength of agreement on the potential of improvement (κ = 0.765 (p = 0.001); 95% CI (0.465; 1.065), the category of common surgical index complications (κ = 0.714 (p < 0.001); 95% CI (0.377; 1.051)), the category of cause of death (κ = 0.694 (p < 0.001); 95% CI (0.327; 1.061)) as well as non-surgical index complications (κ = 0.793 (p < 0.001); 95% CI (0.458; 1.128) and a moderate strength of agreement on surgical index complications (κ = 0.492 (p < 0.001); 95% CI (0.178; 0.806)) ( Table 6). The inter-rater reliability between all participants of the online re-review assessed by Fleiss' kappa is given in Table 7. Fleiss' kappa showed a substantial inter-rater agreement on the category of death (κ = 0.724 (p < 0.001); 95% CI (0.581; 0.866)) and a moderate inter-rater agreement on the category of common surgical index complications (κ = 0.455 (p < 0001) 95%CI (0326; 0.584)), the category of cause of death (κ = 0.533 (p < 0.001); 95% CI (0.311; 0.745)) and surgical index complications (κ = 0.424 (p < 0.001); 95% CI (0.329; 0.518)), respectively.        Table 7. Inter-rater reliability between participants on standardized items of the online re-review.

Discussion
This comprehensive clinical peer review of 86 patients with in-hospital mortality following pancreatic surgery identified four categories of common surgical index complications and causes of death. These categories as well as identified common surgical and non-surgical index complications were incorporated in a standardized questionnaire. Reliability analysis of the questionnaire items found a moderate to substantial test-retest reliability as well as inter-rater reliability.
Pancreatic surgery is associated with relevant morbidity and mortality, which entails the need to investigate the underlying causes [1][2][3]. Several studies have focused on the investigation of mortality rates and the development of prediction models in order to determine the risk of morbidity and mortality in pancreatic surgery. Patient demographics, hospital volume and the expertise of the surgeons have been identified as strong risk factors that affect the perioperative morbidity and mortality rates [1][2][3][4][5][6][7][8][9]. While patient demographics are hard to control, hospital and surgeon volume have been addressed through centralization. Several studies have already reported the benefits of centralization in pancreatic surgery [14][15][16]. As the centralization effect is probably exhausted in numerous countries, additional approaches are necessary to gain further improvements of outcomes. Clinical peer reviews may be an option, but its effect on outcomes is unclear. In addition, such targeted root cause analyses demand a comprehensive analysis of each individual patient case. This study found that surgical index complications accounted for 80% of all deaths. Most of these index complications yielded in sepsis with abdominal organ failure (62%). Data from other studies suggest that at least half of all surgical complications are avoidable [17,18]. In line with an Australian mortality review report, the present data show that a delay in recognizing and treating complications may occur in more than a quarter of deaths [19]. Subsequently, prevention of delayed recognition of complications may improve outcomes.
Pancreatic resections are complex procedures that may result in even more complex post-operative courses as soon as complications arise. In addition, complications originate from and are boosted by preoperative, intraoperative or post-operative errors in patient care. The resulting complexity hampers retrospective root cause analysis including the determination of the final cause of death and the surgical index complications. Therefore, clinical peer reviews will most likely have a low reliability that, in turn, leads to inconsistent proposals of measures to change subsequent practice. This study shows that structuring of the peer review process by means of a standardized questionnaire provides sufficient reliability in regard to index complications, causes of death and potential preventive measures. Of note, the reliability of the peer feedback decreases with an increasing specificity of the review questions (e.g., category of common surgical index complication: 'anastomotic complication' vs. surgical index complication: 'POPF'). This effect probably increases in a multicausal setting that comprises more than one complication and error cause, and subsequently a larger scope of preventive measures. In addition, the inter-rater reliability analysis of the peer re-review showed a lower degree of agreement especially for the last item of the questionnaire (potential of improvement). In view of these facts, it should be noted that the proposed questionnaire has limitations and is not intended to be exhaustive. Especially, specific measures derived from root cause analysis are usually not generalizable due to multifactorial influences in complex clinical settings of individual patient cases. Nevertheless, it should be considered as a tool to structure clinical peer reviews that may prompt in-depth perceptions.
In this study, surgical in-hospital mortality was mainly caused by anastomotic complications. This is in line with the current literature on post-pancreatectomy complications, such as POPF, PPH and DGE. In this regard, the current analysis underlines the relevance of PPH that causes a relevant part of the overall mortality (22%), although it is significantly less common than POPF or DGE. Vascular complications led to in-hospital mortality in a significant part of cases, which corresponds to results of an Australian mortality review report [19]. This category of complications is important to consider and may be due to pre-existing stenoses or inadvertent vascular injury [19]. The latter may be avoided by preoperative anticipation of aberrant vascular anatomy using a novel 3-D visualization technology for post-processing of computed tomography (CT) images, called cinematic rendering [20,21].
This study has limitations that need to be considered. First, the limited sample size and the retrospective design of our study may have incurred some biases. Second, peer reviewing is a subjective assessment and thus prone to bias. Consequently, the standardized questionnaire derived from the clinical peer review in this study should be considered imperfect and improvable. These limitations may be overcome by future investigations with a larger patient cohort in an international, multicenter setting and a prospective study design. Such studies should also include a large panel of international reviewers who are experts in the field pancreatic surgery.

Conclusions
Patients with in-hospital mortality following pancreatic surgery frequently have long and difficult post-operative courses. The resulting complexity hampers the determination of the final causes of death and the surgical index complications. The use of standardized questionnaires helps to structure the peer review process. The fact that structured clinical peer reviews provide consistent results between different reviewers underlines the usefulness of this quality management tool for root cause analyses of in-hospital mortality following pancreatic surgery. Of note, the overall consistency of the peer feedback decreases with an increasing specificity of the review questions. Further studies are needed to improve the suggested standardized questionnaire and to investigate whether quality enhancement measures derived from clinical peer reviews lead to a sustained improvement of post-operative outcomes. Informed Consent Statement: Patient's consent was waived due to the retrospective design of the study without identifying data.
Data Availability Statement: All data presented in this study are available in the article.