Next Article in Journal
Gene-Transcript Expression in Urine Supernatant and Urine Cell-Sediment Are Different but Equally Useful for Detecting Prostate Cancer
Next Article in Special Issue
Cytoreductive Surgery and Hyperthermic Intraperitoneal Chemotherapy with Intra-Operative Radiotherapy for Patients with Locally Advanced or Locally Recurrent Rectal Cancer and Peritoneal Metastases
Previous Article in Journal
Is There a Role for Molecular Testing for Low-Risk Differentiated Thyroid Cancer? A Cost-Effectiveness Analysis
Previous Article in Special Issue
Combined Hyperthermia and Re-Irradiation in Non-Breast Cancer Patients: A Systematic Review
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

The Impact of Cytoreductive Surgery and Hyperthermic Intraperitoneal Chemotherapy (CRS-HIPEC) versus Conventional Surgery on Patient-Reported Outcomes: A Comparative Cohort Study between the CAIRO6 Trial and the PROCORE Study

by
Checca Bakkers
1,
Vincent C. J. van de Vlasakker
1,
Koen P. B. Rovers
1,
Robin J. Lurvink
1,
Simon W. Nienhuijs
1,
Jacobus W. A. Burger
1,
Geert-Jan M. Creemers
2,
Cynthia S. Bonhof
3,
Floortje Mols
3,4 and
Ignace H. J. T. de Hingh
1,4,5,*
1
Department of Surgery, Catharina Cancer Institute, P.O. Box 1350, 5602 ZA Eindhoven, The Netherlands
2
Department of Medical Oncology, Catharina Cancer Institute, P.O. Box 1350, 5602 ZA Eindhoven, The Netherlands
3
Center of Research on Psychological and Somatic Disorders, Department of Medical and Clinical Psychology, Tilburg University, P.O. Box 90153, 5000 LE Tilburg, The Netherlands
4
Department of Research, Netherlands Comprehensive Cancer Organization, P.O. Box 19079, 3501 DB Utrecht, The Netherlands
5
GROW—School for Oncology and Reproduction, Maastricht University, P.O. Box 616, 6200 MD Maastricht, The Netherlands
*
Author to whom correspondence should be addressed.
Cancers 2023, 15(3), 788; https://doi.org/10.3390/cancers15030788
Submission received: 5 January 2023 / Revised: 24 January 2023 / Accepted: 25 January 2023 / Published: 27 January 2023
(This article belongs to the Collection Hyperthermia in Cancer Therapy)

Abstract

:

Simple Summary

Patients treated with cytoreductive surgery and hyperthermic intraperitoneal chemotherapy (CRS-HIPEC) are at risk of significant treatment burden. Multiple studies have reported on patient-reported outcome (PRO) measurements of these patients. However, outcomes are difficult to interpret as no comparison has been made between CRS-HIPEC and conventional surgery. The present study compares several PROs at three different timepoints between patients with colorectal peritoneal metastases treated with CRS-HIPEC and colorectal cancer (CRC) patients treated with conventional surgery. PROs were obtained from two Dutch prospective trials. Despite a more extensive procedure with greater risk of morbidity, CRS-HIPEC in patients with colorectal peritoneal metastases did not have a greater negative impact on PROs than conventional surgery in patients with CRC. Furthermore, systemic therapy did not affect these PROs. These findings may facilitate future patient counseling and shared decision making in clinical practice.

Abstract

Purpose—To compare patient-reported outcomes (PROs) of patients undergoing cytoreductive surgery and hyperthermic intraperitoneal chemotherapy (CRS-HIPEC) for colorectal peritoneal metastases to PROs of colorectal cancer (CRC) patients undergoing conventional surgery. Methods—Data were extracted from the CAIRO6 trial (CRS-HIPEC group) and the PROCORE study (conventional surgery group). Nine predefined PROs (derived from the EORTC QLQ-C30 questionnaire) were compared at baseline, in the early postoperative period and one year postoperatively, with correction for treatment with systemic therapy using linear mixed modeling. Results—In total, 331 patients were included: 71 in the CRS-HIPEC group and 260 in the conventional surgery group. All predefined PROs (fatigue, diarrhea, C30 summary score, Global Health Status, physical, role, emotional, cognitive, and social functioning) did not differ significantly between the groups at all three timepoints, and differential effects over time for all PROs did not differ significantly between the groups. Significant worsening of fatigue, C30 summary score, physical and role functioning (both groups), and cognitive and social functioning (conventional surgery group only) was present in the early postoperative period. All scores returned to baseline at one year postoperatively, except for physical and cognitive functioning in the conventional surgery group. Emotional functioning improved postoperatively in both groups compared to baseline. Conclusion—Despite a more extensive procedure with greater risk of morbidity, CRS-HIPEC in patients with colorectal peritoneal metastases did not have a greater negative impact on PROs than conventional surgery in patients with CRC. Further, systemic therapy did not affect these PROs. These findings may facilitate future patient counseling and shared decision making in clinical practice.

1. Introduction

Over the last few decades, treatments for both non-metastatic and metastatic colorectal cancer (CRC) have greatly improved by evolving into multimodality treatments, including surgery, systemic therapy, and/or radiotherapy. This has resulted in prolonged survival and, consequently, with this growing number of CRC survivors, more emphasis on patient-reported outcomes (PROs) is warranted in clinical research [1].
A randomized clinical trial published in 2003 showed that cytoreductive surgery followed by hyperthermic intraperitoneal chemotherapy (CRS-HIPEC) in selected patients resulted in improved survival, as compared to palliative chemotherapy alone [2,3]. Ever since, CRS-HIPEC has been performed with an increasing frequency in patients with colorectal peritoneal metastases and it is recommended in various (inter)national guidelines [4,5].
Although the increasingly frequent application of CRS-HIPEC in this patient group has improved prognosis significantly, it is an invasive treatment with considerable risk of morbidity and mortality [6]. While aiming for macroscopic radical resection of the tumor, CRS-HIPEC usually requires multiple visceral resections, leading to higher risks of morbidity, as compared to conventional surgery for primary CRC [7,8,9,10]. This could result in a significant treatment burden, leading to a decrease in quality of life.
Patients with non-metastatic primary CRC comprise over a million new patients yearly worldwide [11], and PROs are broadly investigated in these patients [12,13]. Likewise, several studies on patients undergoing CRS-HIPEC focusing on PROs have been published [14,15,16,17,18,19,20]. However, no comparative studies are available. Therefore, it remains unknown whether CRS-HIPEC affects PROs more extensively than conventional surgery for primary CRC. In addition, for both patients with primary CRC undergoing conventional surgery and for patients with colorectal peritoneal metastases undergoing CRS-HIPEC, systemic therapy is often part of treatment. This may additionally cause serious side effects, which might consequently affect PROs [21,22,23].
The aims of this study were to compare PROs in patients undergoing CRS-HIPEC for colorectal peritoneal metastases to PROs in patients undergoing conventional surgery for CRC, and to investigate the effect of systemic therapy for these surgical treatments on PROs. Herewith, more insight into the burden of treatment in these patients undergoing extensive treatment for metastatic CRC could be provided.

2. Materials and Methods

2.1. Study Design and Setting

In this cohort study, PROs of patients undergoing CRS-HIPEC for CPM were compared with PROs of patients undergoing conventional surgery for primary CRC. PROs of patients undergoing CRS-HIPEC for CPM were prospectively collected in the phase 2 part of the CAIRO6 trial [24]. CAIRO6 is an open-label parallel-group trial in all Dutch tertiary centers for the surgical treatment of CPM, with randomization of patients with resectable CPM to perioperative systemic therapy and CRS-HIPEC or upfront CRS-HIPEC alone. PROs of patients undergoing conventional surgery for primary CRC were prospectively collected in the PROCORE study [25]. PROCORE is a population-based, prospective cohort study in four Dutch centers that aims to assess PROs of CRC treatment. The CAIRO6 trial and the PROCORE study were both approved by a central ethics committee (MEC-U, Nieuwegein, Netherlands, numbers NL57644.100.16 (CAIRO6) and NL51119.060.14 (PROCORE)) and the institutional review boards of all participating centers.

2.2. Participants

Patients enrolled in CAIRO6 were adults with a WHO performance score of 0–1 and isolated resectable synchronous or metachronous CPM who did not receive systemic therapy ≤6 months prior to enrolment. CAIRO6 patients were included in the present study if they underwent complete CRS-HIPEC. These patients constituted the ‘CRS-HIPEC group’. Patients enrolled in the PROCORE study were adults with newly diagnosed CRC before the start of CRC treatment. To balance patient groups, PROCORE patients were included in the present study if they underwent curative-intent resection for a T3-4N0-2 primary tumor. These patients constituted the ‘conventional surgery group’. Participants in the PROCORE study who underwent CRS-HIPEC for colorectal peritoneal metastases were included as crossovers in the CRS-HIPEC group.

2.3. Treatments

All patients in the CRS-HIPEC group underwent cytoreductive surgery (CRS), followed by the perfusion of heated chemotherapy through the peritoneal cavity (HIPEC) to eradicate residual cancer cells. Depending on randomization, patients either underwent CRS-HIPEC alone (control arm of CAIRO6) or CRS-HIPEC with ±3 months of neoadjuvant systemic combination chemotherapy (experimental arm of CAIRO6) (preferentially consisting of capecitabine–oxaliplatin (CAPOX) with bevacizumab, or alternatively consisting of 5-fluorouracil–leucovorin–oxaliplatin (FOLFOX) with bevacizumab or 5-fluorouracil–leucovorin–irinotecan (FOLFIRI) with bevacizumab) and ±3 months of adjuvant systemic combination chemotherapy (consisting of CAPOX, FOLFOX, or 5-fluorouracil).
All patients in the conventional surgery group underwent laparoscopic or open tumor resection (right hemicolectomy, left hemicolectomy, sigmoidectomy, low-anterior resection (LAR), or abdomino-perineal resection (APR)), and (neo)adjuvant systemic therapy (consisting of 5-fluorouracil- or capecitabine-based regimens) if indicated according to the Dutch CRC guideline [26].

2.4. PROs

Patients from both groups were asked to complete EORTC QLQ-C30 questionnaire at three time points: at baseline (i.e., before the start of treatment), in the early postoperative period (i.e., ±12 weeks postoperatively in the CRS-HIPEC group, ±5 weeks postoperatively in the conventional surgery group), and one year postoperatively. Scores of all PROs were calculated according to the manuals of EORTC [27,28]. PROs can be divided into function scales (with higher score representing better functioning, e.g., physical functioning) and symptom scales (with higher scores indicating worse symptoms, e.g., fatigue). For the present study, the following PROs were predefined as the most appropriate to assess overall health and treatment tolerability in this setting: C30 summary score, Global Health Scale (GHS), physical functioning, role functioning, emotional functioning, cognitive functioning, social functioning, fatigue, and diarrhea.

2.5. Data Collection

Collected patient, tumor, and treatment characteristics included sex, age, American Society of Anesthesiologists (ASA) classification, primary tumor location, and treatment with (neo)adjuvant systemic therapy. For the CRS-HIPEC group, these data were collected using the CAIRO6 trial database. For the conventional surgery group, these data were extracted from the Netherlands Cancer Registry, which registers data from all Dutch patients diagnosed with cancer.

Statistical Analyses

All categorical patient and treatment characteristics were compared between the two groups using Chi-square tests. For all patients, mean scores of all PROs at baseline, in the early postoperative period, and one year postoperatively were calculated. Patients who completed questionnaires at least at two time points were included in further analysis. For the comparison of nine predefined PROs between the groups, differential effects over time and scores at each timepoint were compared using linear mixed modeling, with the use of maximum likelihood estimation and an unstructured covariance matrix, with a two-level structure (i.e., repeated timepoints (lower level), patients (higher level)). To adjust for the possible effects of systemic therapy on PROs, the same analyses were performed with correction for treatment with systemic therapy. Within the two groups, longitudinal comparisons between different timepoints were conducted, also using linear mixed modeling. For all PROs being significantly different between the groups, Cohen’s D (CD) effect sizes were calculated to express clinical relevance. CD values ≥ 0.5 were considered clinically relevant. All tests were performed two-sided and pragmatically conducted at p < 0.01 to account for multiple testing. All statistical analyses were performed using IBM SPSS Statistics version 25.0 for Windows (IBM Corp, Armonk, NY, USA).

3. Results

3.1. Patients and Treatments

In total, 331 patients were included: 71 in the CRS-HIPEC group and 260 in the conventional surgery group (Figure 1). Patient, tumor, and treatment characteristics are presented in Table 1. Patients in the CRS-HIPEC group were significantly younger than patients in the conventional surgery group (18% vs. 4% ≤50 years, respectively, p < 0.001), less frequently had tumors located in the rectum (4% vs. 26% in the conventional surgery group, p < 0.001), and more frequently received neoadjuvant systemic therapy (48% vs. 10% in the conventional surgery group, p < 0.001). The types of surgery in the conventional surgery group were right hemicolectomy (n = 93), left hemicolectomy (n = 23), sigmoid resection (n = 70), LAR (n = 63), APR (n = 7), and (sub)total hemicolectomy (n = 4).

3.2. Questionnaire Completion Rates

In the CRS-HIPEC group, 99% of patients (70/71) completed baseline questionnaires, 89% of patients (63/71) completed questionnaires in the early postoperative period, and 69% of patients (49/71) completed questionnaires one year postoperatively. In the conventional surgery group, 98% of patients (256/260) completed baseline questionnaires, 85% of patients (221/260) completed questionnaires in the early postoperative period, and 72% of patients (187/260) completed questionnaires one year postoperatively.

3.3. Comparative Analyses of PROs between Groups and Longitudinal Comparisons within the Groups

Comparisons and course of all predefined PROs between both groups are presented in Figure 2, with corresponding linear mixed models in Table 2. Longitudinal comparisons within groups are presented in Table 3.

3.4. Functional Scales

3.4.1. C30 Summary Score

Differential effects over time (p = 0.015) and scores at each timepoint did not differ significantly between the groups, neither before nor after adjustment for systemic therapy (Figure 2A, Table 2). In both groups, the C30 summary score worsened in the early postoperative period (mean difference (MD) −7 (95% CI −11–−4), p < 0.001, clinically relevant (CD 0.66), in the CRS-HIPEC group; MD −5 (95% CI −7–−3), p < 0.001, non-clinically relevant (CD 0.38) in the conventional surgery group) but returned to baseline values at one year postoperatively (Table 3).

3.4.2. Global Health Status

Differential effects over time (p = 0.811) and scores at each timepoint did not differ significantly between the groups, neither before nor after adjustment for systemic therapy (Figure 2B, Table 2). No significant worsening of Global Health Status was present in either of the groups in the early postoperative period; however, a significant improvement at one year postoperatively was present when compared to baseline in the conventional surgery group (MD +7 (95% CI 4–10), p < 0.001, non-clinically relevant (CD 0.38), Table 3).

3.4.3. Physical Functioning

Differential effects over time (p = 0.033) and scores at each timepoint did not differ significantly between the groups, neither before nor after adjustment for systemic therapy (Figure 2C, Table 2). In both groups, physical functioning worsened in the early postoperative period (MD −9 (95% CI −14–−4), p < 0.001, clinically relevant (CD 0.58) in the CRS-HIPEC group; MD −8 (95% CI −11–−7), p < 0.001, clinically relevant (CD 0.53) in the conventional surgery group). In the CRS-HIPEC group, physical functioning returned to baseline scores at one year postoperatively. In the conventional surgery group, physical functioning remained worsened at one year postoperatively in comparison to baseline scores (MD −3 (95% CI −6–−1), p = 0.003, non-clinically relevant (CD 0.18), Table 3).

3.4.4. Role Functioning

Differential effects over time (p = 0.029) and scores at each timepoint did not differ significantly between the groups, neither before nor after adjustment for systemic therapy (Figure 2D, Table 2). In both groups, role functioning worsened in the early postoperative period (MD −12 (95% CI −25–−7), p < 0.001, clinically relevant (CD 0.50) in the CRS-HIPEC group; MD −20 (95% CI −24–−16), p < 0.001, clinically relevant (CD 0.68) in the conventional surgery group), but returned to baseline values at one year postoperatively (Table 3).

3.4.5. Emotional Functioning

Differential effects over time (p = 0.059) and scores at each timepoint did not differ significantly between the groups, neither before nor after adjustment for systemic therapy (Figure 2E, Table 2). In the CRS-HIPEC group, no significant differences in emotional functioning were observed in the early postoperative period, but a significant improvement in emotional functioning at one year postoperatively was observed (MD +13 (95% CI 9–15), p = 0.007, clinically relevant (CD 0.50)) as compared to baseline. In the conventional surgery group, emotional functioning improved in the early postoperative period (MD +7 (95% CI 4–9), p < 0.001, non-clinically relevant (CD 0.37)) and one year postoperatively (MD +8 (95% CI 2–13), p < 0.001, clinically relevant (CD 0.65)) as compared to baseline (Table 3).

3.4.6. Cognitive Functioning

Differential effects over time (p = 0.701) and scores at each timepoint did not differ significantly between the groups, neither before nor after adjustment for systemic therapy (Figure 2F, Table 2). In the CRS-HIPEC group, scores on cognitive functioning remained stable at all timepoints. In the conventional surgery group, significant worsening of cognitive functioning was present both in the early postoperative period (MD −4 (95% CI −6–−1), p = 0.003, non-clinically relevant (CD 0.18)) and one year postoperatively (MD −3 (95% CI −5–0), p < 0.001, non-clinically relevant (CD 0.11)) as compared to baseline (Table 3).

3.4.7. Social Functioning

Differential effects over time (p = 0.006) and scores at each timepoint did not differ significantly between the groups, neither before nor after adjustment for systemic therapy (Figure 2G, Table 2). In the CRS-HIPEC group, scores on social functioning remained stable at all timepoints. In the conventional surgery group, significant worsening of social functioning was present in the early postoperative period (MD −8 (95% CI −12–−6), p < 0.001, non-clinically relevant (CD 0.37)); however, this returned to baseline at one year postoperatively (Table 3).

3.5. Symptom Scales

3.5.1. Fatigue

Differential effects over time (p = 0.047) and scores at each timepoint did not differ significantly between the groups, neither before nor after correction for treatment with systemic therapy (Figure 2H, Table 2). In both groups, fatigue worsened in the early postoperative period (MD +14 (95% CI 8–20), p < 0.001, clinically relevant (CD 0.72) in the CRS-HIPEC group; MD +12 (95% CI 9–16), p < 0.001, non-clinically relevant (CD 0.49) in the conventional surgery group), but returned to baseline scores at one year postoperatively (Table 3).

3.5.2. Diarrhea

Differential effects over time (p = 0.976) and scores at each timepoint did not differ significantly between the groups, neither before nor after correction for treatment with systemic therapy (Figure 2I, Table 2). No significant worsening of diarrhea was found in either group, neither in the early postoperative period nor one year postoperatively.

4. Discussion

This cohort study compared PROs of patients who underwent CRS-HIPEC for colorectal peritoneal metastases to PROs of patients who underwent conventional surgery for CRC. At all timepoints, PROs did not differ significantly between the groups, neither before nor after correction for possible effects of systemic therapy. Therefore, the results of the present study suggest that CRS-HIPEC in patients with colorectal peritoneal metastases does not affect PROs more extensively than conventional surgery in patients with CRC. Further, systemic therapy did not affect these PROs.
With CRS-HIPEC, the aim is to radically resect all of the visible tumor, including the primary tumor if present. This is typically an extensive surgical procedure, including multiple visceral resections, which is associated with a considerably higher risk of severe postoperative morbidity than conventional surgery alone [8,9,10]. Therefore, this could lead to higher treatment and symptom burden. Previous studies investigating PROs in patients undergoing CRS-HIPEC, indeed, showed a worsening of PROs early after surgery with recovery to baseline levels approximately 6 to 12 months postoperatively, which is comparable with the findings in the present study [14,15,16,17,18,19,20]. Although both studies reporting PROs after CRS-HIPEC and studies reporting PROs after curative colon resection for primary CRC are conducted extensively [12,13], no comparative studies are available. Therefore, the present study provides new insight into the burden of treatment in patients undergoing CRS-HIPEC for colorectal peritoneal metastases and may inform both clinicians and patients about the burden of CRS-HIPEC, thereby facilitating future patient counseling. Despite the more extensive treatment in the CRS-HIPEC group, no significant worsening of PROs was observed in comparison to the conventional surgery group. This indicates that CRS-HIPEC does not have a more substantial negative impact on PROs than conventional surgery only.
As the addition of systemic therapy to the surgical treatment of any type of CRC inevitably prolongs and intensifies treatment and possibly leads to (severe) toxicity, systemic therapy may have an impact on PROs during surgical treatment [29]. Therefore, correction for treatment with systemic therapy was performed through comparative linear mixed modeling. However, no significant effect of systemic therapy on PROs was observed in either group.
At one year postoperatively, all PROs in the CRS-HIPEC group had returned to baseline values, but in the conventional surgery group, a worse physical and cognitive functioning remained, as compared to baseline. Even though other researchers have previously described residual cognitive impairment at one and two years after treatment in CRC patients [30], these effects were not clinically relevant in the present study.
Despite being the first comparative study on PROs in patients undergoing CRS-HIPEC or conventional surgery, this study has some limitations. Firstly, patients in the CRS-HIPEC group were significantly younger than patients in the conventional surgery group, which may have affected PROs. Secondly, the selection procedure of patients for the two study groups was different. In the PROCORE study, all patients undergoing conventional surgery for CRC in four hospitals were asked to participate, while the CAIRO6 trial population comprised patients having to meet much stricter criteria, due to its design and experimental interventions. This additionally resulted in different group sizes, being the third limitation in the present study. However, in order to balance both groups as much as possible, only patients diagnosed with T stage 3–4 were selected from the PROCORE study for the conventional surgery group, as the vast majority of patients with colorectal peritoneal metastases also present with T stage 3–4 CRC. Furthermore, appropriate statistical analyses by means of linear mixed modeling were performed to account for small group sizes. Another limitation is that data on postoperative morbidity and on the presence of any type of ostomy, which might have affected PROs (particularly in the early postoperative period), were not available. Another limitation regarding data is the missing data on specific (neo)adjuvant treatment regimens. Although, in general, no significant effect of systemic therapy on PROs was observed in either group, it is possible that PROs were affected to a greater or lesser extent by the specific systemic therapy regimen (e.g., the toxicity profile of oxaliplatin is more strongly associated with neuropathy than that of irinotecan and, as such, oxaliplatin might have affected the PRO physical functioning more strongly). However, as the PROs are affected by multiple variables and since the percentage of patients in each group that received systemic therapy was comparable, it is expected that the possible effect of the specific systemic therapy regimen was minor.
Lastly, patients in the conventional surgery group completed the questionnaire in the early postoperative period about five weeks postoperatively, as compared to about 12 weeks postoperatively in the CRS-HIPEC group. This could have led to a worsened score from PROs in the conventional surgery group at the early postoperative timepoint as compared to the CRS-HIPEC group due to patients being in an earlier postoperative recovery phase.

5. Conclusions

Despite a more extensive procedure with greater risk of morbidity, CRS-HIPEC in patients with colorectal peritoneal metastases did not have a greater negative impact on PROs than conventional surgery in patients with CRC. Systemic therapy in addition to surgical treatment did not significantly affect PROs in either of the groups. These results are valuable for patient counselling and support shared decision making for the treatment of patients with colorectal peritoneal metastases.

Author Contributions

Conceptualization, C.S.B., F.M. and I.H.J.T.d.H.; methodology, K.P.B.R. and C.S.B., S.W.N., J.W.A.B. and G.-J.M.C.; software, C.B. and K.P.B.R.; validation, C.B., V.C.J.v.d.V. and R.J.L.; formal analysis, C.S.B.; investigation, C.S.B.; resources, C.B., V.C.J.v.d.V., K.P.B.R., C.S.B. and F.M.; data curation, C.S.B. and C.S.B.; writing—original draft preparation, C.S.B., R.J.L. and K.P.B.R.; writing—review and editing, J.W.A.B., C.S.B. and V.C.J.v.d.V.; visualization, C.S.B. and R.J.L.; supervision, I.H.J.T.d.H.; project administration, F.M.; funding acquisition, I.H.J.T.d.H. All authors have read and agreed to the published version of the manuscript.

Funding

The CAIRO6 trial is funded by the Dutch Cancer Society (grant number 10795) and F. Hoffman-La Roche (grant number ML39718). The PROCORE study was funded by the Netherlands Comprehensive Cancer Organisation, Utrecht, the Netherlands; the Center of Research on Psychological and Somatic Disorders (CoRPS), Tilburg University, the Netherlands; and an Investment Grant Large of the Dutch Research Council (2016/04981/ZONMW-91101002). The funders had no role in study design, in the collection, analysis, and interpretation of data, in the writing of the report, and in the decision to submit the article for publication.

Institutional Review Board Statement

The CAIRO6 trial and the PROCORE study were both approved by a central ethics committee (MEC-U, Nieuwegein, Netherlands, numbers NL57644.100.16 (CAIRO6) and NL51119.060.14 (PROCORE)) and the institutional review boards of all participating centers.

Informed Consent Statement

Informed consent was obtained from all subjects involved in either study.

Data Availability Statement

Data will be made readily available upon written request sent to the corresponding author.

Conflicts of Interest

De Hingh: An unrestricted research grant from RanD/QP&S and Roche, paid to the institute, unrelated to the submitted work. All other authors did not have any disclosures.

References

  1. Wibe, A. Multidisciplinary Treatment: Influence on Outcomes. Multidisciplinary Treatment of Colorectal Cancer; Springer: Berlin/Heidelberg, Germany, 2021; pp. 11–21. [Google Scholar]
  2. Verwaal, V.J.; Bruin, S.; Boot, H.; Van Slooten, G.; Van Tinteren, H. 8-year follow-up of randomized trial: Cytoreduction and hyperthermic intraperitoneal chemotherapy versus systemic chemotherapy in patients with peritoneal carcinomatosis of colorectal cancer. Ann. Surg. Oncol. 2008, 15, 2426–2432. [Google Scholar] [CrossRef] [PubMed]
  3. Verwaal, V.J.; van Ruth, S.; de Bree, E.; van Slooten, G.W.; van Tinteren, H.; Boot, H.; Zoetmulder, F.A. Randomized trial of cytoreduction and hyperthermic intraperitoneal chemotherapy versus systemic chemotherapy and palliative surgery in patients with peritoneal carcinomatosis of colorectal cancer. J. Clin. Oncol. 2003, 21, 3737–3743. [Google Scholar] [CrossRef] [PubMed]
  4. Bushati, M.; Rovers, K.; Sommariva, A.; Sugarbaker, P.; Morris, D.; Yonemura, Y.; Quadros, C.; Somashekhar, S.; Ceelen, W.; Dubé, P.; et al. The current practice of cytoreductive surgery and HIPEC for colorectal peritoneal metastases: Results of a worldwide web-based survey of the Peritoneal Surface Oncology Group International (PSOGI). Eur. J. Surg. Oncol. 2018, 44, 1942–1948. [Google Scholar] [CrossRef] [PubMed]
  5. Klaver, C.E.L.; Groenen, H.; Morton, D.G.; Laurberg, S.; Bemelman, W.A.; Tanis, P.J. The research committee of the European Society of Coloproctology Recommendations and consensus on the treatment of peritoneal metastases of colorectal origin: A systematic review of national and international guidelines. Color. Dis. 2017, 19, 224–236. [Google Scholar] [CrossRef] [PubMed]
  6. Baratti, D.; Kusamura, S.; Pietrantonio, F.; Guaglio, M.; Niger, M.; Deraco, M. Progress in treatments for colorectal cancer peritoneal metastases during the years 2010–2015. A systematic review. Crit. Rev. Oncol. Hematol. 2016, 100, 209–222. [Google Scholar] [CrossRef]
  7. Hall, B.; Padussis, J.; Foster, J.M. Cytoreduction and Hyperthermic Intraperitoneal Chemotherapy in the Management of Colorectal Peritoneal Metastasis. Surg. Clin. N. Am. 2017, 97, 671–682. [Google Scholar] [CrossRef]
  8. Nakafusa, Y.; Tanaka, T.; Tanaka, M.; Kitajima, Y.; Sato, S.; Miyazaki, K. Comparison of Multivisceral Resection and Standard Operation for Locally Advanced Colorectal Cancer: Analysis of Prognostic Factors for Short-Term and Long-Term Outcome. Dis. Colon Rectum 2004, 47, 2055–2063. [Google Scholar] [CrossRef]
  9. Simkens, G.A.; Van Oudheusden, T.R.; Braam, H.J.; Luyer, M.D.; Wiezer, M.J.; Van Ramshorst, B.; Nienhuijs, S.W.; De Hingh, I.H. Treatment-Related Mortality After Cytoreductive Surgery and HIPEC in Patients with Colorectal Peritoneal Carcinomatosis is Underestimated by Conventional Parameters. Ann. Surg. Oncol. 2016, 23, 99–105. [Google Scholar] [CrossRef]
  10. Simkens, G.A.; Van Oudheusden, T.R.; Luyer, M.D.; Nienhuijs, S.W.; Nieuwenhuijzen, G.A.; Rutten, H.J.; De Hingh, I.H. Predictors of Severe Morbidity After Cytoreductive Surgery and Hyperthermic Intraperitoneal Chemotherapy for Patients With Colorectal Peritoneal Carcinomatosis. Ann. Surg. Oncol. 2016, 23, 833–841. [Google Scholar] [CrossRef]
  11. Global Cancer Observatory 2022. Available online: globocan.iarc.fr (accessed on 9 June 2020).
  12. Reale, M.L.; De Luca, E.; Lombardi, P.; Marandino, L.; Zichi, C.; Pignataro, D.; Ghisoni, E.; Di Stefano, R.F.; Mariniello, A.; Trevisi, E.; et al. Quality of life analysis in lung cancer: A systematic review of phase III trials published between 2012 and 2018. Lung Cancer 2020, 139, 47–54. [Google Scholar] [CrossRef] [Green Version]
  13. Wiltink, L.M.; White, K.; King, M.T.; Rutherford, C. Systematic review of clinical practice guidelines for colorectal and anal cancer: The extent of recommendations for managing long-term symptoms and functional impairments. Support. Care Cancer 2020, 28, 2523–2532. [Google Scholar] [CrossRef] [Green Version]
  14. da Silva, R.G.; Sugarbaker, P.H. Analysis of prognostic factors in seventy patients having a complete cytoreduction plus perioperative intraperitoneal chemotherapy for carcinomatosis from colorectal cancer. J. Am. Coll. Surg. 2006, 203, 878–886. [Google Scholar] [CrossRef]
  15. Hill, A.R.; Mcquellon, R.P.; Russell, G.B.; Shen, P.; Stewart, J.H.; Levine, E.A. Survival and Quality of Life Following Cytoreductive Surgery Plus Hyperthermic Intraperitoneal Chemotherapy for Peritoneal Carcinomatosis of Colonic Origin. Ann. Surg. Oncol. 2011, 18, 3673–3679. [Google Scholar] [CrossRef] [Green Version]
  16. Ihemelandu, C.U.; Mcquellon, R.; Shen, P.; Stewart, J.H.; Votanopoulos, K.; Levine, E.A. Predicting Postoperative Morbidity Following Cytoreductive Surgery with Hyperthermic Intraperitoneal Chemotherapy (CS+HIPEC) with Preoperative FACT-C (Functional Assessment of Cancer Therapy) and Patient-Rated Performance Status. Ann. Surg. Oncol. 2013, 20, 3519–3526. [Google Scholar] [CrossRef]
  17. Mcquellon, R.P.; Danhauer, S.C.; Russell, G.B.; Shen, P.; Fenstermaker, J.; Stewart, J.H.; Levine, E.A. Monitoring Health Outcomes Following Cytoreductive Surgery Plus Intraperitoneal Hyperthermic Chemotherapy for Peritoneal Carcinomatosis. Ann. Surg. Oncol. 2007, 14, 1105–1113. [Google Scholar] [CrossRef]
  18. Passot, G.; Bakrin, N.; Roux, A.; Vaudoyer, D.; Gilly, F.-N.; Glehen, O.; Cotte, E. Quality of life after cytoreductive surgery plus hyperthermic intraperitoneal chemotherapy: A prospective study of 216 patients. Eur. J. Surg. Oncol. 2014, 40, 529–535. [Google Scholar] [CrossRef]
  19. Steffens, D.; Koh, C.; Ansari, N.; Solomon, M.J.; Brown, K.; McBride, K.; Young, J.; Young, C.J.; Moran, B. Quality of Life After Cytoreductive Surgery and Hyperthermic Intraperitoneal Chemotherapy: Early Results from a Prospective Cohort Study of 115 Patients. Ann. Surg. Oncol. 2020, 27, 3986–3994. [Google Scholar] [CrossRef]
  20. Tsilimparis, N.; Bockelmann, C.; Raue, W.; Menenakos, C.; Perez, S.; Rau, B.; Hartmann, J. Quality of life in patients after cytoreductive surgery and hyperthermic intraperitoneal chemotherapy: Is it worth the risk? Ann. Surg. Oncol. 2013, 20, 226–232. [Google Scholar] [CrossRef]
  21. Cleeland, C.S. Symptom Burden: Multiple Symptoms and Their Impact as Patient-Reported Outcomes. JNCI Monogr. 2007, 2007, 16–21. [Google Scholar] [CrossRef]
  22. Curt, G.A.; Breitbart, W.; Cella, D.; Groopman, J.E.; Horning, S.J.; Itri, L.M.; Johnson, D.H.; Miaskowski, C.; Scherr, S.L.; Portenoy, R.K.; et al. Impact of Cancer-Related Fatigue on the Lives of Patients: New Findings From the Fatigue Coalition. Oncologist 2000, 5, 353–360. [Google Scholar] [CrossRef] [Green Version]
  23. Mols, F.; Beijers, T.; Lemmens, V.; Hurk, C.J.V.D.; Vreugdenhil, G.; van de Poll-Franse, L.V. Chemotherapy-Induced Neuropathy and Its Association With Quality of Life Among 2- to 11-Year Colorectal Cancer Survivors: Results From the Population-Based PROFILES Registry. J. Clin. Oncol. 2013, 31, 2699–2707. [Google Scholar] [CrossRef] [PubMed]
  24. Rovers, K.P.; Bakkers, C.; Simkens, G.A.A.M.; Burger, J.W.A.; Nienhuijs, S.W.; Creemers, G.-J.M.; Thijs, A.M.J.; Brandt-Kerkhof, A.R.M.; Madsen, E.V.E.; Ayez, N.; et al. Perioperative systemic therapy and cytoreductive surgery with HIPEC versus upfront cytoreductive surgery with HIPEC alone for isolated resectable colorectal peritoneal metastases: Protocol of a multicentre, open-label, parallel-group, phase II-III, randomised, superiority study (CAIRO6). BMC Cancer 2019, 19, 390. [Google Scholar]
  25. Bonhof, C.S.; van de Poll-Franse, L.V.; Wasowicz, D.K.; Beerepoot, L.V.; Vreugdenhil, G.; Mols, F. The course of peripheral neuropathy and its association with health-related quality of life among colorectal cancer patients. J. Cancer Surviv. 2021, 15, 190–200. [Google Scholar] [CrossRef] [PubMed]
  26. Vereniging Klinische Genetica Nederland; Landelijke Werkgroep Gastrointestinale Tumoren. Oncoline Landelijke Richtlijn Erfelijke Darmkanker Versie.2.0; Integraal Kanker Centrum Nederland: Utrecht, The Netherlands, 2015. [Google Scholar]
  27. Aaronson, N.K.; Ahmedzai, S.; Bergman, B.; Bullinger, M.; Cull, A.; Duez, N.J.; Filiberti, A.; Flechtner, H.; Fleishman, S.B.; De Haes, J.C.J.M.; et al. The European Organization for Research and Treatment of Cancer QLQ-C30: A Quality-of-Life Instrument for Use in International Clinical Trials in Oncology. J. Natl. Cancer Inst. 1993, 85, 365–376. [Google Scholar] [CrossRef]
  28. Giesinger, J.M.; Kieffer, J.M.; Fayers, P.M.; Groenvold, M.; Petersen, M.A.; Scott, N.W.; Sprangers, M.A.; Velikova, G.; Aaronson, N.K. Replication and validation of higher order models demonstrated that a summary score for the EORTC QLQ-C30 is robust. J. Clin. Epidemiol. 2016, 69, 79–88. [Google Scholar] [CrossRef] [Green Version]
  29. Braun, M.S.; Seymour, M.T. Balancing the efficacy and toxicity of chemotherapy in colorectal cancer. Ther. Adv. Med. Oncol. 2011, 3, 43–52. [Google Scholar] [CrossRef] [Green Version]
  30. Vardy, J.L.; Dhillon, H.M.; Pond, G.R.; Rourke, S.B.; Bekele, T.; Renton, C.; Dodd, A.; Zhang, H.; Beale, P.; Clarke, S.; et al. Cognitive function in patients with colorectal cancer who do and do not receive chemotherapy: A prospective, longitudinal, controlled study. J. Clin. Oncol. 2015, 33, 4085. [Google Scholar] [CrossRef] [Green Version]
Figure 1. Patient flowchart. CRS-HIPEC cytoreductive surgery and hyperthermic intraperitoneal chemotherapy; a ± 12 weeks postoperatively; b ± 5 weeks postoperatively.
Figure 1. Patient flowchart. CRS-HIPEC cytoreductive surgery and hyperthermic intraperitoneal chemotherapy; a ± 12 weeks postoperatively; b ± 5 weeks postoperatively.
Cancers 15 00788 g001
Figure 2. Comparison of predefined patient-reported outcomes between the CRS-HIPEC group and the conventional surgery group. Continuous lines represent mean PRO scores; dotted lines represent standard deviations of PRO scores. (A) C30 Summary score; (B) Global Health Status; (C) physical functioning; (D) role functioning; (E) emotional functioning; (F) cognitive functioning; (G) social functioning; (H) fatigue; (I) diarrhea.
Figure 2. Comparison of predefined patient-reported outcomes between the CRS-HIPEC group and the conventional surgery group. Continuous lines represent mean PRO scores; dotted lines represent standard deviations of PRO scores. (A) C30 Summary score; (B) Global Health Status; (C) physical functioning; (D) role functioning; (E) emotional functioning; (F) cognitive functioning; (G) social functioning; (H) fatigue; (I) diarrhea.
Cancers 15 00788 g002
Table 1. Comparison of patient, tumor, and treatment characteristics between both groups.
Table 1. Comparison of patient, tumor, and treatment characteristics between both groups.
CRS-HIPEC Group
N = 71
Conventional Surgery Group
N = 260
p-Value
Sex, n (%) 0.176
Male36 (51)156 (60)
Female35 (49)104 (40)
Age, n (%) <0.001
≤5013 (18)11 (4)
51–7040 (56)135 (52)
>7018 (25)114 (44)
ASA, n (%) 0.112
1–265 (92)211 (82)
3–46 (8)40 (15)
Unknown-9 (3)
Primary tumor location, n (%) <0.001
Right colon24 (34)96 (37)
Left colon42 (59)94 (36)
Rectum3 (4)68 (26)
Unknown2 (3)2 (1)
Any (neo)adjuvant systemic therapy, n (%) 0.133
No38 (54)165 (64)
Yes33 (47)95 (36)
Neoadjuvant systemic therapy, n (%) <0.001
No37 (52)235 (90)
Yes34 (48)25 (10)
Adjuvant systemic therapy, n (%) 0.553
No49 (69)189 (73)
Yes22 (31)71 (27)
CRS-HIPEC: cytoreductive surgery and hyperthermic intraperitoneal chemotherapy, ASA: American Society of Anesthesiologists classification.
Table 2. Linear mixed modeling on comparisons between both groups, with and without correction for systemic therapy.
Table 2. Linear mixed modeling on comparisons between both groups, with and without correction for systemic therapy.
UnadjustedAdjusted for Systemic Therapy
PROMean Difference d95% CIp-Value95% CIp-Value
C30 summary score a
Comparison of differential effects over time between both groupsNANA0.015NA0.012
Comparisons between the groups at each measurement c
Baseline−1−6–20.337−6–20.312
Early postoperative period d−3−8–−10.024−8–−10.021
One year postoperatively−2−9–−10.020−9–−10.017
Global Health Status a
Comparison of differential effects over time between both groupsNANA0.811NA0.838
Comparisons between the groups at each measurement c
Baseline+6−2–90.199−2–90.206
Early postoperative periodd+1−5–60.736−5–60.753
One year postoperatively−1−10–20.232−10–20.222
Physical functioning a
Comparison of differential effects over time between both groupsNANA0.033NA0.026
Comparisons between the groups at each measurement c
Baseline−2−9–10.116−9–10.102
Early postoperative period d−3−9–10.105−9–10.090
One year postoperatively−2−11–00.069−11–00.057
Role functioning a
Comparison of differential effects over time between both groupsNANA0.029NA0.029
Comparisons between the groups at each measurement c
Baseline−9−16–00.057−16–00.059
Early postoperative period d−1−11–50.496−11–50.501
One year postoperatively−6−16–20.115−16–20.118
Emotional functioning a
Comparison of differential effects over time between both groupsNA 0.059NA0.048
Comparisons between the groups at each measurement c
Baseline−1−8–30.359−8–30.329
Early postoperative periodd−2−9–20.229−9–20.205
One year postoperatively−4−13–−10.017−14–−20.014
Cognitive functioning a
Comparison of differential effects over time between both groupsNANA0.701NA0.690
Comparisons between the groups at each measurement c
Baseline−1−5–50.940−5–50.932
Early postoperative periodd0−6–40.610−6–40.603
One year postoperatively0−7–50.721−7–50.711
Social functioning a
Comparison of differential effects over time between both groupsNANA0.006NA0.006
Comparisons between the groups at each measurement c
Baseline−3−12–−00.042−12 –−00.044
Early postoperative periodd−4−11–10.090−11–10.094
One year postoperatively−5−15–−20.015−15–−20.016
Fatigue b
Comparison of differential effects over time between both groupsNANA0.047NA0.048
Comparisons between the groups at each measurement c
Baseline+23–100.248−3–100.251
Early postoperative periodd+4−1–120.105−1–120.107
One year postoperatively+5−1–140.096−1–140.099
Diarrhea b
Comparison of differential effects over time between both groupsNANA0.976NA0.958
Comparisons between the groups at each measurement c
Baseline−7−12–10.083−12–10.086
Early postoperative periodd0−4–90.495−4–90.486
One year postoperatively+3−2–120.189−2–120.183
PRO patient-reported outcome; CI confidence interval; NA not applicable; a higher scores represent better functioning; b higher scores represent worse symptoms; c calculated as CRS-HIPEC as compared to conventional surgery (conventional surgery as reference); d +/− 12 weeks postoperatively in CRS-HIPEC group, +/− 5 weeks postoperatively in conventional surgery group.
Table 3. Linear mixed modeling on comparisons of scores over time within the groups.
Table 3. Linear mixed modeling on comparisons of scores over time within the groups.
CRS-HIPEC
N = 66
Conventional Surgery
N = 237
PROMean Difference c95% CIp-ValueCohen’s D eMean Difference c95% CIp-ValueCohen’s D e
C30 summary score a
Comparisons between time points within groups
Baseline vs. early postoperative period d−7−11–−4<0.0010.66−5−7–−3<0.0010.38
Early postoperative period d vs. one year postoperatively+83–100.0020.61+75–9<0.0010.54
Baseline vs. one year postoperatively+1−5–−30.701NA+20–40.018NA
Global Health Status a
Comparisons between time points within groups
Baseline vs. early postoperative period d−7−10–00.066NA−2−5–10.119NA
Early postoperative period d vs. one year postoperatively+7−2–100.163NA+96–12<0.0010.47
Baseline vs. one year postoperatively0−6–50.815NA+74–10<0.0010.38
Physical functioning a
Comparisons between time points within groups
Baseline vs. early postoperative period d−9−14–−4<0.0010.58−8−11–−7<0.0010.53
Early postoperative period d vs. one year postoperatively+6−1–100.146NA+53–8<0.0010.34
Baseline vs. one year postoperatively−3−10–00.074NA−3−6–−10.0030.18
Role functioning a
Comparisons between time points within groups
Baseline vs. early postoperative period d−12−25–−7<0.0010.50−20−24–−16<0.0010.68
Early postoperative period d vs. one year postoperatively+186–260.0020.67+2115–25<0.0010.75
Baseline vs. one year postoperatively+6−10–90.913NA+1−5–50.968NA
Emotional functioning a
Comparisons between time points within groups
Baseline vs. early postoperative period d+61–110.0120.37+74 –9<0.0010.37
Early postoperative period d vs. one year postoperatively+7−4–70.635NA+13–8<0.0010.29
Baseline vs. one year postoperatively+139–150.0070.50+82–13<0.0010.65
Cognitive functioning a
Comparisons between time points within groups
Baseline vs. early postoperative period d−3−9–00.031NA−4−6–−10.0030.18
Early postoperative period d vs. one year postoperatively+1−3–60.525NA+1−1–34<0.0010.05
Baseline vs. one year postoperatively−2−8–20.193NA−3−5–0<0.0010.11
Social functioning a
Comparisons between time points within groups
Baseline vs. early postoperative period d−9−15–−10.018NA−8−12–−6<0.0010.37
Early postoperative period d vs. one year postoperatively+9−9–60.749NA+107–14<0.0010.47
Baseline vs. one year postoperatively0−1–150.073NA+2−2–50.323NA
Fatigue b
Comparisons between time points within groups
Baseline vs. early postoperative period d+148–20<0.0010.72+129–16<0.0010.49
Early postoperative period d vs. one year postoperatively−14−20–7<0.0010.62−15−18–−11<0.0010.61
Baseline vs. one year postoperatively0−6–70.908NA−1−6–20.279NA
Diarrhea b
Comparisons between time points within groups
Baseline vs. early postoperative period d+72–140.012NA0−4–30.781NA
Early postoperative period d vs. one year postoperatively−1−9–60.669NA−4−8–00.049NA
Baseline vs. one year postoperatively+60–140.064NA−4−8–−10.022NA
PRO patient-reported outcome; CI confidence interval; NA not applicable; a higher scores represent better functioning; b higher scores represent worse symptoms; c calculated as second mentioned (mean) timepoint minus first mentioned (mean) timepoint for longitudinal comparisons; d +/− 12 weeks postoperatively in CRS-HIPEC group, +/− 5 weeks postoperatively in conventional surgery group; e Cohen’s D effect sizes were calculated in case of statistically significant differences between two timepoints.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Bakkers, C.; van de Vlasakker, V.C.J.; Rovers, K.P.B.; Lurvink, R.J.; Nienhuijs, S.W.; Burger, J.W.A.; Creemers, G.-J.M.; Bonhof, C.S.; Mols, F.; de Hingh, I.H.J.T. The Impact of Cytoreductive Surgery and Hyperthermic Intraperitoneal Chemotherapy (CRS-HIPEC) versus Conventional Surgery on Patient-Reported Outcomes: A Comparative Cohort Study between the CAIRO6 Trial and the PROCORE Study. Cancers 2023, 15, 788. https://doi.org/10.3390/cancers15030788

AMA Style

Bakkers C, van de Vlasakker VCJ, Rovers KPB, Lurvink RJ, Nienhuijs SW, Burger JWA, Creemers G-JM, Bonhof CS, Mols F, de Hingh IHJT. The Impact of Cytoreductive Surgery and Hyperthermic Intraperitoneal Chemotherapy (CRS-HIPEC) versus Conventional Surgery on Patient-Reported Outcomes: A Comparative Cohort Study between the CAIRO6 Trial and the PROCORE Study. Cancers. 2023; 15(3):788. https://doi.org/10.3390/cancers15030788

Chicago/Turabian Style

Bakkers, Checca, Vincent C. J. van de Vlasakker, Koen P. B. Rovers, Robin J. Lurvink, Simon W. Nienhuijs, Jacobus W. A. Burger, Geert-Jan M. Creemers, Cynthia S. Bonhof, Floortje Mols, and Ignace H. J. T. de Hingh. 2023. "The Impact of Cytoreductive Surgery and Hyperthermic Intraperitoneal Chemotherapy (CRS-HIPEC) versus Conventional Surgery on Patient-Reported Outcomes: A Comparative Cohort Study between the CAIRO6 Trial and the PROCORE Study" Cancers 15, no. 3: 788. https://doi.org/10.3390/cancers15030788

APA Style

Bakkers, C., van de Vlasakker, V. C. J., Rovers, K. P. B., Lurvink, R. J., Nienhuijs, S. W., Burger, J. W. A., Creemers, G. -J. M., Bonhof, C. S., Mols, F., & de Hingh, I. H. J. T. (2023). The Impact of Cytoreductive Surgery and Hyperthermic Intraperitoneal Chemotherapy (CRS-HIPEC) versus Conventional Surgery on Patient-Reported Outcomes: A Comparative Cohort Study between the CAIRO6 Trial and the PROCORE Study. Cancers, 15(3), 788. https://doi.org/10.3390/cancers15030788

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop