Development of Population-Based Cancer Indicators and a Measurement of Cancer Care Continuum Using a Modified Delphi Method

Simple Summary This study intended to identify the current state of national cancer management and prevention policies and provide a useful framework for policy establishment. The development of population-based cancer indicators project was conducted as part of a Nationwide Population-based Cancer Registration Project and aimed to applying quality of health care system approaches to the cancer care in population-based monitoring system. The literature review and grey literature review were conducted prior to Delphi method. Our findings confirm 26 cancer indicators and classify them into “Primary prevention,” “Secondary prevention,” “Treatment,” “Quality of care,” “Survivor management,” and “End-of-life care.” The Donabedian model used for health services and the Institute of Medicine quality of healthcare domains of six dimensions were applied to the measurement system. The developed cancer indicators and measurements will be able to provide useful information for prioritizing the operational tasks that can be used by health authorities and policymakers from the other countries. Abstract To identify population-based cancer indicators and construct monitoring systems for the entire lifecycle of cancer patients using a modified Delphi method. A modified Delphi method was used to identify the cancer indicators and measurement by scoping review and gray literature. The final list of cancer indicators was developed by consensus of 11 multidisciplinary experts over multiple rounds and rating scored the importance of each indicator on a 10-point scale. Frequency analysis was performed to rate with median scores ≥7 and finalized the list of indicators according to the priority. Initially, 254 indicators were identified, of which 94 were considered important and feasible. After two rounds of rating by the experts and panel discussions, 26 indicators were finalized in six domains: primary prevention (n = 7), secondary prevention (n = 11), treatment (n = 2), quality of life (n = 4), survivor management (n = 1), and end-of-life care (n = 1). The Donabedian model used for examining health services and the Institute of Medicine quality of healthcare domains were applied to the measurement system. Panel experts identified cancer indicators based on priorities with a high level of consensus, providing a scrupulous foundation for community-based monitoring of cancer patients.


Introduction
The global cancer burden has risen to 18.1 million cases, and 9.6 million cancer deaths occurred in 2018 [1]. Cancer incidence and mortality are rapidly increasing worldwide. During 2012-2016, people diagnosed with cancer were 69% as likely to survive for at least 5 years after being diagnosed as was the overall population [2].
In Korea, one in four deaths per year is attributable to cancer, and more than 200,000 new cancer cases were diagnosed in 2014 [3]. The mid-year population and cancer mortality data from 1983 to 2016 were obtained from Statistics Korea [4]. The number of cancer incidences and deaths are expected to increase with the increase in aging population and adoption of a westernized lifestyle [5].
In this context, cancer statistics are the most important indicators of the national cancer burden and can be used to develop cancer prevention and control strategies [6]. To date, most of these efforts have focused on measuring regional performance [7]. Healthcare cancer indicators, as well as performance indicators or quality measures, are used worldwide to quantify, grade, monitor, and improve the quality of healthcare [8]. The identification and management of care for patients with cancer are important but complex [9]. In addition, cancer has a very significant effect on the patient's physical, psychological, and social well-being, and various medical professionals are involved in the prevention, diagnosis, treatment, and follow-up care of these patients. Therefore, complexity can lead to an imbalance in care, resulting in discontinuity of cancer care [9,10].
To date, development of indicators related to cancer management, have focused one of the following, national cancer screenings or early detection [11,12], diagnosis [13,14], treatment [14], quality of care [15][16][17][18], survivor management [19,20], and end-of-life care [21]. Although there has been much work involved in the development of indicators related to the quality or performance of specific types of cancer [16][17][18][19][20][21], minimal attention has been paid to the lifecycle of cancer or cancer care continuum.
Our previous preliminary study described the systematic development of cancer indicators based on the entire lifecycle of cancer care as the first step in population-based monitoring. In addition, indicators were grouped into prevention, treatment, quality of care, survivor management, and end-of-life care domains. The participants of measurement system were patients, healthcare providers, and administrators or managers with results based on the prioritized cancer indicators and supporting evidence. We chose a modified Delphi approach to develop cancer indicators and measurements for this project [22]. The Delphi methodology is commonly used when sufficient evidence is available to derive indicators, which require review and deliberation by experts [23]. This study aimed to develop the indicators that were selected by expert consensus. In addition, panel discussion and the Delphi method were used to finalize the prioritized cancer indicators and population-based measurement system as a community-based measurement system.

Materials and Methods
This study was performed using a well-structured step-by-step approach. The approaches were (1) scoping review, (2) gray literature based on national and international guidelines of cancer care, and (3) expert opinions and panel discussions. Expert opinions were obtained through email, and the panel discussion was conducted with face-to face meeting and online discussion facility was arranged for the experts who could not physically attend discussion due to the coronavirus disease (COVID-19) pandemic.

Literature Search
A scoping review was conducted to develop the first draft of a list of cancer indicators and followed the six stages outlined in the Joanna Briggs Institute manual [JBI, 2015-18] Cancers 2021, 13, 4826 3 of 13 within the compliance of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement for scoping reviews [24]. PubMed, CINAHL, Cochrane library, Ovid-EMBASE, RISS, KISS, and KoreaMed databases were systematically searched from May 2020 to September 2020 using a predefined search strategy. Article selection and data extraction were independently performed by two researchers. Articles were included if they defined, described, or recommended appropriate cancer care.

Gray Literature
At the time of the review of literature report submission, our expert committee was instructed to identify additional published and unpublished (government reports, policy statements and papers) references based on cancer care guideline. Therefore, this study diverted to conduct gray literature additionally. Gray literature is composed of knowledge artifact review process, it has been more specifically conceptualized in narrow and broad ways [25]. In extension, we have searched an existing national and international guideline for the entire cycle of cancer care from the official site of the American Society of Clinical Oncology and European Union guidelines, Guidelines of the World Health Organization, International Agency for Research on Cancer, Organization for Economic Cooperation and Development, National Health Service. Guidelines from Canada, Japan, and South Korea were followed, and certain non-English guidelines were translated by the research team. Furthermore, problems resulting from unclear translation or unclear formulation were resolved by discussion. The sources of grey literature and characteristics are given in Supplementary Table S1.
The next process was initiated by combining both the literature review and information acquired from the guidelines. Each concept was converted into an indicator by formulating a definition, numerator, and denominator. All converted topics were checked for loss of information due to translation by a research team.

Delphi Rounds
This study adapted the RAND-modified Delphi panel process, validated by the University of California Los Angeles, which involves a 2-round process of multidisciplinary experts, combining scientific evidence and their experiences to rate the proposed measures [26].
A multidisciplinary panel included forty-two experts to select the cancer measures according to the priority. We divided the experts into three groups: researchers, policymakers, and clinicians. Then, we randomly selected five researchers, three policymakers, and three clinicians. Of the five researchers, two were from internal research group, and the other three from the national cancer center and other colleges. For the three policymakers, one was a director of a public or community health service center, another from the Ministry of Health and Welfare, and the other from the National Cancer Center. For clinicians, we selected one oncologist, one internal physician, and a radio-oncologist. All panel members were contacted via email; the purpose of the project was explained, and consent was obtained.
The literature search yielded 254 indicators, which were reassessed by three rounds of the Delphi method to finalize the appropriate cancer indicators as follows: (1) First round conducted a modified Delphi method, which involved rating or scoring by experts (using frequency and mean scores); (2) second round of qualitative suggestions given by experts during the panel discussions (using comments from both the Delphi rounds) were reviewed; (3) sorting and correcting the appropriate indicators based on the priority and possibility of measuring based on the population were performed.

Round 1
A list of potential indicators was prepared using an Excel spreadsheet based on the literature search and sent to all the experts via email along with a list of references from which the potential indicators were extracted. The experts were asked to rate each indicator based on a 10-point Likert scale by the following criteria: "highly important, direct implementation, long-term monitoring care". The experts were asked to express their opinions and comment on each indicator.
All the opinions of the experts were collected and entered in a Microsoft Excel database, the mean and frequencies were calculated, and a summary report was prepared. The indicators were organized based on whether a strong consensus for acceptance was achieved with a mean score of ≥8 obtained from the panel members. Information on the selected indicators was distributed to the panel members as the basis for discussion at the in-person meeting.
The first-round analysis revealed a set of 94 indicators that were recommended to be ensured by the experts. All potential indicators were reviewed and confirmed based on the discussion among the research team members who either accepted or rejected these indicators. The research team considered the expert comments and the indicators proposed in the internal meeting. This discussion allowed for the outright rejection of some indicators and the merging of several indicators. The panel experts were asked to rate the indicators in a manner like that of Round 1. Responses were analyzed again; a mixed online and offline mode of discussion was conducted to confirm the acceptance or rejection.

Round 2
The cancer indicators remaining after Rounds 1 were subjected to another panel discussion as Round 2. All panel experts could not be present in person due to the COVID-19 pandemic; therefore, a mixed mode using offline and online Zoom discussions was performed, to facilitate and encourage interaction to share their opinions. The criterion for strong consensus for acceptance or rejection was based on whether a majority of responses was obtained in favor of or against an indicator, respectively.

Quality of the Healthcare Measurement Framework
In this study, the framework for the quality of healthcare measurement focused on three models: (1)

Literature Search and Gray Literature
The scoping review search retrieved a total of 6202 articles from electronic databases, and 3234 were selected after eliminating duplicates. There were 310 articles screened for eligibility after carefully reading the abstract and title. Total of 275 articles that did not meet the selection criteria were excluded after careful revision; and the remaining 35 articles were included in the final analysis.
The search of gray literature revealed more than 600 indicators based on the different types of cancer, diagnosis, treatment, quality of care, survivor management, and end-oflife care. All research teams are involved in the analysis and synthesis of the national and international guidelines. We have finalized 254 indicators, and extraction was performed based on the importance and nationwide demand for the indicators (Supplementary Table  S1). The framework for monitoring the measurement of cancer indicators is shown in Figure 1, which presents the concept of the decision-making process involving the patient and their family, healthcare providers, and administrators from organizations. The Donabedian model [27] evaluates three concepts of quality of care that underpin improvement of the care provided. The three components are structure, process, and outcomes. The six domains of healthcare quality [28] are interrelated with both the decision-making authority and healthcare quality measurements.

Literature Search and Gray Literature
The scoping review search retrieved a total of 6202 articles from electronic databases, and 3234 were selected after eliminating duplicates. There were 310 articles screened for eligibility after carefully reading the abstract and title. Total of 275 articles that did not meet the selection criteria were excluded after careful revision; and the remaining 35 articles were included in the final analysis.
The search of gray literature revealed more than 600 indicators based on the different types of cancer, diagnosis, treatment, quality of care, survivor management, and end-of-life care. All research teams are involved in the analysis and synthesis of the national and international guidelines. We have finalized 254 indicators, and extraction was performed based on the importance and nationwide demand for the indicators (Supplementary Table S1).

Selection Process of Cancer Indicators in the Delphi Rounds
The selection process of cancer indicators is shown in Figure 2. After several meetings and discussions by the entire research team, 254 unique indicators were selected for the panel discussions.

Distribution of Cancer-Related Monitoring Indicators according to Donabedian's Quality of Healthcare
The selected 26 indicators are analyzed by using Donabedian's quality of healthcare measures for quality of care throughout the lifecycle of cancer. The results shown in Table  2 indicate that most of the indicators were related to the outcome (20, 76.9%), process (4, 15.4%), and structure (2, 7.7%) domains, with maximum involvement of the administrator  Table 1.

Distribution of Cancer-Related Monitoring Indicators According to Donabedian's Quality of Healthcare
The selected 26 indicators are analyzed by using Donabedian's quality of healthcare measures for quality of care throughout the lifecycle of cancer. The results shown in Table 2 indicate that most of the indicators were related to the outcome (20, 76.9%), process (4, 15.4%), and structure (2, 7.7%) domains, with maximum involvement of the administrator (14, 53.8%), followed by a combination of the patient and administrator (7, 26.9%), whereas the involvement of provider was lesser (2, 7.7%) than that of others.

Discussion
Quality of care initiatives have become common among cancer care providers, and policymakers strive to systematically measure and improve the quality of cancer care. In the present Delphi method, we defined criteria for including and excluding ratings in the in three rounds [12]. For the first Delphi round, we used the median scores given by the experts for each indicator for inclusion or exclusion as the median is less affected by outliers, which are important in group discussions. We had a strict exclusion criterion (median score ≤ 5 on a scale of 1-10) and inclusion criterion (median score ≥ 7 on a scale of 1-10) in the first round of Delphi; therefore, only indicators with a strong agreement among the expert's opinions are included.

Discussion
Quality of care initiatives have become common among cancer care providers, and policymakers strive to systematically measure and improve the quality of cancer care. In the present Delphi method, we defined criteria for including and excluding ratings in the in three rounds [12]. For the first Delphi round, we used the median scores given by the experts for each indicator for inclusion or exclusion as the median is less affected by outliers, which are important in group discussions. We had a strict exclusion criterion (median score ≤ 5 on a scale of 1-10) and inclusion criterion (median score ≥ 7 on a scale of 1-10) in the first round of Delphi; therefore, only indicators with a strong agreement among the expert's opinions are included.
Although the cancer indicators were chosen from or among existing measures in the literature, the perspective and framework used for selection are unique as most quality measurement efforts to date both within and outside cancer have been disease-focused or procedure-focused [12][13][14]16]. Incorporating this perspective into the development of measurement systems based on cancer care indicators for the population is very important to minimize the risk that quality improvement initiatives pose in widening the gaps in the quality of cancer care [8].
The most unique aspect of our study is the perspective method by which the indicators were selected. Cancer is a leading cause of morbidity and mortality in Korea. Furthermore, the diagnosis and treatment of cancer, even if the outcome is not fatal, has a significant impact on the quality of life, morbidity, and healthcare utilization. More importantly, the available literature points to significant differences in cancer screening and treatment with respect to factors such as socioeconomic status [9], health behavior [9,10,13], screening, and treatment process [15][16][17][18], palliative care, and end-of-life care [14,17,[19][20][21], highlighting the need to incorporate these perspectives into cancer measurement for the total population.
Another unique aspect of our study is the comprehensive selection of cancer indicators and measurement strategies. Our indicators are valid for the entire lifecycle of cancer from screening through end-of-life care and across the different types of cancers, all of which are important causes of morbidity and mortality in the total population. Specifically, the selected indicators will allow us to look at differences in cancer care in different aspects of the healthcare delivery, including primary care (cancer screening), secondary care (early diagnosis and interventions), tertiary care (treatment and special procedures), quality of life care (follow-up care), survivor management (education and regular health visits), and end-of-life care (palliative and hospice care). This will allow us to determine how different sociodemographic factors influence the quality of cancer care in different healthcare sectors.
Furthermore, we have identified a number of indicators that can be used in a population based on structure, process, and outcome of cancer indicators according to the Donabedian's model [27]. Studies have concluded that a comparison of outcomes is challenging at the national and international levels because of the non-parametric nature of the indicators [11,12,20,[22][23][24]. Our assessment of indicators showed that the structure of cancer care is not a high priority in the healthcare system of Korea, which is already well established. Regarding the process, we could identify six indicators in secondary prevention and survivor management. Most of our indicators were outcome-focused, which is very important for the quality of healthcare systems.
Post this analysis, we have used these indicators to measure cancer care in the total population and identify several gaps in care that are currently being provided. We hope that our cancer indicators and population-based measurements will be of use directly or indirectly to other countries or regions interested according to their organizational pattern to improve their quality of cancer care.

Strengths and Limitations
This study had several strengths. First, a step-by-step process of indicator selection, including scoping review, gray literature surveying of existing national and international guidelines, and three rounds of the panel discussions were considered as a very rigorous cancer indicator identification procedure. Second, the panel meeting involved experts from all provinces of Korea, which contributed to an efficient and systematic discussion of all prioritized cancer indicators.
We have certain limitations in this study. First, there were only 11 experts involved in the Delphi rounds due to the COVID-19; optimal size and composition of the panel need to be obtained in the further study. Second, the third consensus round was conducted online and offline due to the social distancing. Therefore, most of the panel members did not have a face-to-face interaction, all of them were contacted by zoom meeting at the time of panel discussion. Third, the selected cancer indicators were not submitted for external review. However, the limited involvement of external reviews can be adequately addressed in future research. Four, further work, which may include a review of the potential baseline indicators such as diet, family history, patient satisfaction, information systems, and patient experience measures to identify relevant data items and develop new data items where required, is necessary to support the implementation of these indicators. There is a need to examine the extent to which the measurement of an indicator may contribute to changes in service provision and patient outcomes and identify the mechanisms by which service delivery processes affect patient outcomes.

Conclusions
The Delphi method provided guidance in selection of indicators that are widely accepted by those who are willing to improve population-based cancer care. These indicators are highlighting the combination of clinical and community-oriented cancer care practice, health care professionals, communication, and quality of health care system. Currently, there is substantial interest in the development of cancer indicators. Thus, not only the process but also the resulting indicators may prove of interest to policymakers, clinicians, and researchers across South Korea.