Next Article in Journal
Aberrant Methylation of the Imprinted C19MC and MIR371-3 Clusters in Patients with Non-Small Cell Lung Cancer
Next Article in Special Issue
The Role of Type 2 Diabetes in Patient Symptom Attribution, Help-Seeking, and Attitudes to Investigations for Colorectal Cancer Symptoms: An Online Vignette Study
Previous Article in Journal
Population-Based Analysis of National Comprehensive Cancer Network (NCCN) Guideline Adherence for Patients with Anal Squamous Cell Carcinoma in California
Previous Article in Special Issue
Urine CA125 and HE4 for the Detection of Ovarian Cancer in Symptomatic Women
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Cancers
Volume 15
Issue 5
10.3390/cancers15051462
cancers-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Impact of the COVID-19 Outbreak—Delayed Referral of Colorectal and Lung Cancer in Primary Care: A National Retrospective Cohort Study

by
Charles W. Helsper
1,*,
Carla H. Van Gils
1,
Nicole F. Van Erp
1,
Marinde F. R. Siepman van den Berg
1,
Omar Rogouti
1,
Kristel M. Van Asselt
2,
Otto R. Maarsingh
2,
Jean Muris
3,
Daan Brandenbarg
4,
Sabine Siesling
5,6,
Niek J. De Wit
1,
Matthew P. Grant
1 and
on behalf of the COVID and Cancer Consortium
†
1
Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Heidelberglaan 100, 3584 CS Utrecht, The Netherlands
2
Department of General Practice, Amsterdam UMC, Amsterdam Public Health Research Institiute, Meibergdreef 9, 1105 AZ Amsterdam, The Netherlands
3
Department of General Practice, Care and Public Health Research Institute, Maastricht University, P. Debeyeplein 1, 6229 HA Maastricht, The Netherlands
4
Department of General Practice and Elderly Care Medicine, University Medical Centre Groningen, University of Groningen, 9700 AD Groningen, The Netherlands
5
IKNL—Department of Research and Development, Netherlands Comprehensive Cancer Organisation (IKNL), Godebaldkwartier 419, 3511 DT Utrecht, The Netherlands
6
Department of Health Technology and Services Research, Utrecht & Technical Medical Centre, Twente University, Technohal, Hallenweg 5, 7522 NH Enschede, The Netherlands
*
Author to whom correspondence should be addressed.
†
Collaborators of the COVID and Cancer Consortium are indicated in the Acknowledgments.
Cancers 2023, 15(5), 1462; https://doi.org/10.3390/cancers15051462
Submission received: 30 January 2023 / Revised: 14 February 2023 / Accepted: 20 February 2023 / Published: 25 February 2023
(This article belongs to the Special Issue Cancer Detection in Primary Care)
Browse Figures
Review Reports Versions Notes

Abstract

:

Simple Summary

The COVID-19 pandemic impacted health care. We studied the impact of the COVID-19 outbreak on the referral and diagnosis of cancer patients, thereby aiming to help prevent the delay of cancer diagnosis in future pandemics. We found that during the COVID-19 outbreak, the time between presentation and referral in primary care for patients with colorectal and lung cancer was substantially increased. This was also seen for patients who presented with alarm symptoms. This shows that, in future pandemics, targeted GP support is needed to maintain effective cancer diagnosis.

Abstract

The Coronavirus disease 2019 (COVID-19) outbreak impacted health care. We investigated its impact on the time to referral and diagnosis for symptomatic cancer patients in The Netherlands. We performed a national retrospective cohort study utilizing primary care records linked to The Netherlands Cancer Registry. For patients with symptomatic colorectal, lung, breast, or melanoma cancer, we manually explored free and coded texts to determine the durations of the primary care (IPC) and secondary care (ISC) diagnostic intervals during the first COVID-19 wave and pre-COVID-19. We found that the median IPC duration increased for colorectal cancer from 5 days (Interquartile Range (IQR) 1–29 days) pre-COVID-19 to 44 days (IQR 6–230, p < 0.01) during the first COVID-19 wave, and for lung cancer, the duration increased from 15 days (IQR) 3–47) to 41 days (IQR 7–102, p < 0.01). For breast cancer and melanoma, the change in IPC duration was negligible. The median ISC duration only increased for breast cancer, from 3 (IQR 2–7) to 6 days (IQR 3–9, p < 0.01). For colorectal cancer, lung cancer, and melanoma, the median ISC durations were 17.5 (IQR (9–52), 18 (IQR 7–40), and 9 (IQR 3–44) days, respectively, similar to pre-COVID-19 results. In conclusion, for colorectal and lung cancer, the time to primary care referral was substantially prolonged during the first COVID-19 wave. In such crises, targeted primary care support is needed to maintain effective cancer diagnosis.

1. Introduction

The Coronavirus disease 2019 (COVID-19) pandemic has had a substantial impact on health care worldwide, altering the manner in which patients accessed care, health professionals provided care, and health care systems functioned [1,2,3]. As the first wave of COVID-19 struck Europe in March 2020, health care provision and government policy focused on care for patients with COVID-19 and the prevention of its transmission. Lockdowns were implemented, and usual care processes were interrupted or stopped, producing major impacts on care for patients with non-COVID-19 conditions, such as cancer [3,4,5]. This is evident from cancer diagnosis data from March to June 2020, with numbers of cancer diagnoses reportedly decreasing between 25–61% internationally [6,7,8].
Delay in recognition, referral, and diagnosis of cancer patients can have a substantial impact on the disease burden and prognosis [9,10]. Such delay is associated with later stages of cancer at diagnosis, more invasive treatments, greater impacts on patient lives and quality of life, and worse morbidity and mortality [9,10]. In countries with primary care-based healthcare systems, the diagnosis of cancer largely occurs through the general practitioner (GP), with previous studies demonstrating that over 80% of cancer patients in Western Europe are diagnosed after contact with their GP [11]. The first steps in the diagnostic process are integral for timely diagnosis and treatment; patients first recognize their symptoms, then present to their GP, who details these symptoms, investigates them if needed, and, if indicated, refers the patients to secondary care for diagnosis and treatment [12].
During the first COVID-19 wave, the opportunities to perform face to face consultations and physical examination were restricted and in GP consultations there was a dominant focus on COVID-19, both by patients and GPs [5,13,14,15,16]. Referral pathways were also affected, because of interruptions in routine investigations and overburdening of secondary care services [16]. Therefore, we hypothesize that the first COVID-19 wave had a substantial impact on the time to recognition, referral, and diagnosis of cancer patients presenting to primary care.
To enable the targeted prevention of delay in cancer diagnosis in future pandemics, detailed knowledge about the extent of such delay and its occurrence in specific populations is needed. To achieve this goal, we aim to map the impact of the first wave of the COVID-19 pandemic on the duration of the diagnostic pathway before and after primary care referral for symptomatic cancer patients in The Netherlands.

2. Materials and Methods

2.1. Study Design

We performed a retrospective cohort study. To build our cohort, we linked routine primary care data to cancer registry data collected in primary care practices and hospitals in The Netherlands. This study was reported to be in line with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement [17]. This research was reviewed by the institutional review board of the UMC Utrecht (21-144/C) and judged not to be subject to the Medical Research Involving Human Subjects Act of The Netherlands.

2.2. Data

Primary care data was obtained from The Intercity Data Network, which combines data from the dynamic primary care cohorts of five academic primary care networks: the Academic Network of General Practice at Amsterdam University Medical Centre; location VU Medical Center (ANH VUmc) and location Amsterdam Medical Center (AHA AMC), the Research Network Family Medicine (RNFM) Maastricht, the Academic General Practitioner Development Network Groningen (AHON), and the Julius General Practitioners’ Network Utrecht (JGPN). Together, they contain data from over 1.2 million adult primary care patients, which are representative of the Dutch population [18].
This data contains deidentified coded and free text data, including free text descriptions of patient symptoms, physical examinations, working diagnoses, and case management. The symptoms and diagnoses are coded according to the International Classification of Primary Care (ICPC-1) coding system.
Data from patients with diagnostic codes for cancer (colorectal, lung, and breast cancer, and melanoma) in the primary care dataset were linked to the Dutch National Cancer Registry (NCR) database by a third trusted party. The NCR is a national database which includes over 99% of Dutch cancer patients [19]. NCR data include patient demographics, tumour histology, stage and localisation of cancer, date of diagnosis, and subsequent treatment. The Intercity Data Network, the NCR, and the linkage process have been described previously [20,21,22].

2.3. Population

To select cancer patients for whom the primary care diagnostic interval (partially) overlapped with the first COVID-19 wave, we included patients from each database in the intercity network with a newly attributed ICPC-1 code for colorectal (code: D75), lung (R84), or breast (X76) cancer, or melanoma (S77.03) used after 1 March 2020. The diagnosis of cancer for each patient was confirmed by manual exploration of the free text data, thereby employing an established validation process described previously [20,21,22]. The first wave of the COVID-19 pandemic was defined as “from the first of March to the 30 June 2020,” informed by the National Institute for Public Health and the Environment (RIVM) [23]. We selected patients who presented to the GP with cancer related symptoms, who were then referred to secondary care, and whose primary care diagnostic pathway overlapped with all or part of the first COVID-19 wave. We ceased screening routine care data for such patients for each cancer type if there were no new inclusions for two consecutive months after the first COVID-19 wave. Patients diagnosed through screening programs, specialists, or emergency departments, or who were asymptomatic, were excluded. For melanoma, patients who had a melanoma resected by the GP (and thus, were not referred to secondary care) were not included.
Data for patients whose diagnostic interval served as the control period were collected in the previous Dickens project, as elaborated in the thesis of Nicole van Erp [22]. These data describe the duration of the diagnostic process of symptomatic patients diagnosed with colorectal, lung, and breast cancer, and melanoma, prior to the COVID-19 pandemic, from 1 January 2012 to 31 December 2015.

2.4. Data Collection

Information used to detail the time intervals of the diagnostic process in primary care was manually collected from the free text routine care data. All data collection was performed according to a data collection manual, which was developed in the previous Dickens project. We used identical methods for the pre-COVID-19 and COVID-19 cohorts [22]. Data collection was performed by the research team comprised of medical doctors and medical students. If there were uncertainties, these were discussed and resolved within the study group. All data required for analysis was entered into Castor, an electronic data capture system, using a standardized capture procedure.
The diagnostic time intervals employed in this study were developed from the Aarhus statement, described in Figure 1 [12]. The primary care interval (IPC) was defined as the period between the first contact with primary care for suspected cancer-related signs or symptoms (which could be in-person or through video/telephone consultation) and the date of referral to specialist care. The date of referral is generally explicitly mentioned in primary care data. If not, the moment when responsibility for patient care was transferred to specialist care was identified in the free text and selected. In the case of multiple referrals, the first referral relating to cancer-related symptoms was chosen.
The secondary care interval (ISC) was defined as the period between referral by the GP to secondary care and the histological cancer diagnosis, as retrieved from the NCR.
Patient demographics, clinical characteristics, signs and symptoms, and dates of first consultation and referral were collected from primary care databases. Comorbidities were manually collected for each patient in accordance with the methods of O’Halloran et al. [24]. Cancer alarm symptoms were assessed per cancer type according to predefined definitions, similar to those previously employed in the Dickens project to determine the pre-COVID-19 durations of IPC and ISC [22].

2.5. Analyses

Patient demographics and clinical characteristics were detailed using descriptive statistics. The durations of the diagnostic intervals, for each cancer population and its subgroups (patient and clinical characteristics) were calculated in days for each cohort. Durations were expressed in medians and interquartile ranges because of the expected non-parametric distribution. Consistent with previous studies, to define same day referrals as ‘one day,’ one day was added to all intervals. The number of GP consultations with cancer-related complaints (for the specific cancer type) in IPC was counted with a maximum of one consultation per day and included the initial and referral consultation. Mann–Whitney U tests were employed for comparisons of differences regarding duration and number of consultations (COVID-19 versus pre-COVID-19), and a p-value of less than 0.05 was considered statistically significant. IBM SPSS software version 27 (Microsoft, Chicago, IL, USA) was employed for data analysis.

3. Results

To assess the impact of the first wave of the COVID-19 pandemic on the duration of the diagnostic pathway before and after primary care referral for symptomatic cancer patients in The Netherlands, we performed a retrospective cohort study using routine primary care data linked to The Netherlands Cancer Registry.
We screened 3182 cancer patients for eligibility (IPC overlap the first COVID-19 wave). Overlap of the first COVID-19 wave with IPC was found for 415 cancer patients, and with ISC for 273 patients. The inclusion process is described in a flow diagram in the Supplementary Materials (‘Supplementary flow diagram’). In the prior Dickens project, which describes the pre-COVID-19 period, IPC and ISC durations were determined for 979 cancer patients [22].

3.1. Population

Table 1 describes the patient demographics and clinical characteristics of the patients whose IPC or ISC overlapped with the first COVID-19 wave, and of patients included in the pre-COVID-19 study. The mean age at the first GP consultation was between 55 and 70 years, with the majority of patients (64–92%) having registered comorbid conditions.

3.2. Primary Care Interval (IPC)

As shown in Figure 2, the primary care interval was prolonged during the first COVID-19 wave for patients with colorectal and lung cancer, but not for patients with breast cancer and melanoma. For colorectal cancer patients, the median IPC duration increased by 39 days, from 5 days (IQR 1–29) pre-COVID-19, to 44 days (IQR 6–230, p < 0.01) during the first wave. For lung cancer patients, the median duration from first consultation to referral increased by 26 days, from 15 days (IQR 3–47) to 41 days (IQR 7–101.75, p < 0.01). For breast cancer and melanoma patients the median IPC duration remained 1 day, and the P75 value, indicating the value above which the 25% longest durations occur, shifted from 1 to 2 days (not statistically significant). This lack of relevant change for breast cancer and melanoma was observed in all subgroup analyses, with median durations remaining at 1 day in all subgroups (not shown in Table).
Subgroup analyses suggest differences in impact of the first COVID-19 wave on IPC duration for colorectal and lung cancer, as detailed in Table 2. For colorectal cancer, the observed increase in duration was the largest for females, young patients (<65), those with less than two comorbidities, and those with psychiatric comorbidity. For those with colorectal cancer and alarm symptoms (e.g., rectal bleeding), the median IPC duration increased from one day (IQR 1–18) pre-COVID-19 to one month during COVID-19 (IQR 13–169, p < 0.01). In contrast, for lung cancer, the largest absolute increase was observed for males, elderly patients (≥65), and those with more than two comorbidities or psychiatric comorbidity.
The number of cancer symptom related GP consultations in IPC—for all cancer types—were significantly increased during the first COVID-19 wave. For colorectal and lung cancer, the median number of consultations in IPC was 3 (IQR; 2–6) and 4 (IQR; 3–6), as compared to 2 (IQR; 1–4) (p value for change: <0.01) and 3 (IQR; 2–5) (p value for change: <0.01) pre-COVID-19. For breast cancer and melanoma, the median number of consultations remained at 1, but a larger proportion of patients required more than one consultation (p value for change: both <0.05).

3.3. Secondary Care Interval (ISC)

The ISC duration was only prolonged for breast cancer, increasing from a median of 3 days (IQR 2–7) before COVID-19, to 6 days (IQR 3–9, p < 0.01) during the first COVID-19 wave. ISC durations during the first COVID-19 wave for colorectal and lung cancer and melanoma were 17.5 (IQR 9–52), 18 (IQR 7–40), and 9 (IQR 3–44) days, respectively, which is similar to the pre-COVID-19 period. Only for lung cancer, the subgroup analyses showed differences; a significant increase in ISC duration for those with psychiatric co-morbidity, from a median of 16 days (IQR 8–30.5) pre-COVID-19 to 40 days during COVID-19 (IQR 14–49). Details on ISC duration before and during the first COVID-19 wave are available in the Supplementary Materials, Scheme S1.

4. Discussion

4.1. Main Findings

These results describe substantial delays in primary care for colorectal and lung cancer patients during the first COVID-19 wave, particularly among the subgroups. For colorectal and lung cancer, the median primary care diagnostic interval during the first COVID-19 wave was 39 and 26 days longer than the pre-COVID-19 interval, respectively. Subgroup differences were suggested, affecting different groups for colorectal and lung cancer. The impact of COVID-19 on the secondary care interval was minimal, with only a small increase of three days observed for breast cancer.
The delays observed for colorectal and lung cancer in primary care are worrying, given the extent of these delays and the tendency of these tumor types for rapid progression. Sud et al. studied the effect of a 2-month delay for major cancer types in a modeling study that showed an 11% and 7% worsening in 10-year survival rates for lung and colorectal cancer (estimated for a 65-year old patient) [9]. Especially when considering the observed increase in duration of two to six months in P75 values (longest: 25%) for the diagnostic interval time in our study, the potential impact of the COVID-19-related delay on cancer burden and prognosis is worrisome.
There were concerns that the impact of the COVID-19 pandemic would disproportionally affect vulnerable patients [13]. Our results suggest that this may be true for colorectal and lung cancer patients with psychiatric comorbidities, for whom the impact of COVID-19 on the duration of the primary care pathway was observed to be twice as severe. Another worrying finding is that patients with colorectal and lung cancer presenting with alarm symptoms were also at increased risk of delay during the first COVID-19 wave. For both these patient populations, targeted delay prevention appears warranted in the case of future pandemics or similar disruptions to health systems.

4.2. Implications for Practice

The finding that delays during the first COVID-19 wave occurred in colorectal and lung cancer patients, and hardly at all in melanoma and breast cancer, reflects the known challenges in detecting ‘hard-to-recognise’ cancers. While the symptoms of melanoma and breast cancer are generally clear, those of lung and colorectal cancer tend to be less specific. During the first COVID-19 wave, recent studies suggest that recognizing and acting upon less specific symptoms was increasingly challenging [13,16,25], particularly for lung cancer, since its symptoms (e.g., coughing) were considered a reason to shift to the use of video or telephone consultation instead of physical consultation during the first COVID-19 wave. This additional challenge in recognizing and referring cancer is reflected not only in a longer duration of the primary care pathway, but also in the additional consultations needed prior to referral. This is likely to be partially related to the increased use of telehealth during this period, resulting in many initial consultations through this medium, requiring a secondary consultation to facilitate a physical examination and a more detailed assessment [25]. However, this alone is unlikely to be responsible for the observed delay. On a patient level, recent studies described patient-related barriers to being referred to specialist services during this period, including a reluctance to ‘make a fuss,’ and concerns of overburdening the health system or being infected with COVID-19 [16,25]. On a system level, health professionals described challenges in accessing investigations and communicating with and referring to secondary care [25]. Such barriers for referral are likely to be accentuated for cancer types that generally present with alarm symptoms that are less distinct and for which referral pathways are more complex.
The secondary care diagnostic pathway did not appear to have experienced the same extensive delays during the first COVID-19 wave as those seen in the primary care phase. This is likely to be the effect of the well-developed and tailored nature of secondary care cancer pathways and their prioritization of cancer care in times of increased pressure on health care.
Future directions: our findings show that the impact of the first wave of the COVID-19 pandemic on the diagnostic pathway of symptomatic cancer was the largest in primary care. The mechanisms underlying primary care delay seem multifactorial and inconsistent between cancer types and subgroups. To reduce the impact of future outbreaks on effective cancer diagnosis, improving understanding of the mechanisms leading to delay would facilitate the development of targeted support for the detection and referral of cancer patients by GPs.

4.3. Strength and Limitations

This study employed routine care data from patients throughout The Netherlands for both the prior Dickens project and the current study. The main strength of such data is the large and diverse population from which the data is obtained, as well as the detailed clinical nature of the data. The availability of free text data, which can be used as a prospectively collected transcribed verbatim summary of daily practice consultations, allows very rich and detailed information to be used to determine milestones which mark the beginning and end of the primary care phase of the diagnostic pathway. It also enables the detailed registration of a broad range of symptoms and characteristics, which are not always registered in coded data. As symptoms are registered at the time of their occurrence, recall bias is eliminated. This rich primary care dataset includes over 1.2 million adult primary care patients, from different geographic and socioeconomic regions, which was then linked with the Netherlands Cancer Registry (NCR) database. Linkage to the NCR allows for the use of detailed and reliable information, such as the validation of cancer diagnosis and its date. Data from the pre-COVID-19 period (2012–2015) were used as a comparator. They originated from the same datasets, and we employed the same methods and measures for data collection, providing methodological consistency in order to compare results.
Several limitations of this study should be taken into account. The numbers of cancer cases occurring in this study are substantially fewer than those in the prior Dickens study, which was used as the comparator. This is largely due to the inclusion criteria, which demand that presentation to the GP occurs during the first COVID-19 wave, which is a much shorter timeframe than that used in the previous Dickens study. Despite the dataset including over 1.2 million patients, only a small proportion of patients were diagnosed with these cancer types and had exhibited their diagnostic periods within this timeframe. We expect to have included all eligible patients within this inclusion period because of continued screening for patients up to one year after the end of the first COVID-19 wave. However, it is possible that, in the event that cancer was diagnosed more than one year after the first COVID-19 wave, we missed some patients with extensive delays.
Routine care data contains complexities, errors, and intricacies. This data is entered by many different health practitioners while—and for the purpose of—providing clinical care for patients. Thus, data collection was very time consuming and can be subject to interpretation. For optimal yield and the prevention of error and subjectivity, data collection was done according to a set framework, and was performed by different researchers with clinical experience, such as medical students or doctors. Similar to the prior Dickens project, if there were uncertainties or missing information in the data that made the diagnostic pathway of the patient unclear, these patients were excluded. Despite this extensive effort, our data may be incomplete and subject to misinterpretation. Routine care data can be incomplete or sub-optimally coded for research purposes. Consequently, missing diagnostic codes could have occurred; i.e., for some patients, the right ICPC code for the cancer diagnosis might not (yet) have been attributed by the GP [20]. This would lead to missing cancer patients in our dataset. Additionally, if there were uncertainties about the cancer diagnostic processes (i.e., the presence of cancer related symptoms, or when a referral occurred), these patients were excluded. A total of 555 patients were excluded because of an ‘unclear diagnostic pathway’. Fortunately, the number of patients which could be included were consistent with our original estimates, which were based on the previous Dickens project. We believe the total numbers included are representative and sufficient to robustly demonstrate clinically relevant differences in the diagnostic periods in comparison with the pre-COVID-19 data. Moreover, by adding NCR data to the primary care data required linkage using pre-approved third trusted-party employment and privacy procedures, ensuring the non-reducibility of the data. Such procedures, similar to those in preceding projects, lead to loss of patients, which is generally thought to be a non-selective loss [22].
It should be noted that our analyses of the impact of the COVID-19 outbreak do not include the delay in the diagnostic process which may have occurred before presentation to primary care, in other words, the patient interval from the time of first symptom recognition to consulting primary care. Recent findings suggest that there was widespread avoidance of primary care during the first COVID-19 wave, with reductions of up to 34% in patient presentations with cancer-related symptoms during this period [14,26,27,28]. This delay due to care avoidance is likely to add to the delay observed in our study.
Moreover, our findings only describe the impact of the first COVID-19 wave and not the impact during the full COVID-19 pandemic. The impact of the first wave on both health care performance and patient behavior is likely to be larger during the first wave than during consecutive waves [3,27,28,29].
Finally, it should be taken into account that—for colorectal and breast cancer—a national screening program exists in The Netherlands, which was halted because of COVID-19 halfway through March 2020. Additionally, we did not include patients visiting the emergency room. Therefore, our findings are not representative of all new cancer patients, and changes in these other pathways may have influenced the selection of patients visiting primary care with symptoms [30]. Given our focus only on symptomatic patients presenting to primary care during the relatively short first wave of COVID-19, we do not expect this to have impacted our findings.

5. Conclusions

This study demonstrates that time to referral in primary care was substantially prolonged during the first COVID-19 wave for patients with colorectal or lung cancer. This impact seems to vary for subgroups, and it is inconsistent between cancer types. For these ‘hard-to-diagnose-cancer-types’ that exhibit less distinct referral criteria and pathways, delay also occurred in patients presenting with alarm symptoms. For patient populations at risk of pandemic-related delays in primary care, targeted support for detection and referral by GPs seems warranted to ensure effective cancer diagnosis in future pandemics or similar crises.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/cancers15051462/s1 Supplementary flow diagram (Scheme S1)—study inclusion of patients diagnosed with cancer during first COVID-19 wave; Supplementary Table S1—Duration of the secondary care interval (ISC), and time between GP referral and histological diagnosis, before and during the first COVID-19 wave for colorectal, lung, breast cancer, and melanoma.

Author Contributions

The below-mentioned authors contributed to the following aspects of the research and manuscript: conceptualization, C.W.H., C.H.V.G., N.F.V.E., J.M. and N.J.D.W.; data curation, C.W.H., N.F.V.E., K.M.V.A., J.M., D.B., S.S. and M.P.G.; formal analysis, C.W.H., C.H.V.G., M.F.R.S.v.d.B., O.R. and M.P.G.; funding acquisition, C.W.H., C.H.V.G., S.S. and N.J.D.W.; investigation, C.W.H. and N.F.V.E.; methodology, C.W.H., C.H.V.G., N.F.V.E., M.F.R.S.v.d.B., O.R. and M.P.G.; project administration, C.W.H.; resources, C.W.H.; supervision, C.W.H., C.H.V.G. and M.P.G.; visualization, C.W.H., N.F.V.E. and M.P.G.; writing—original draft, C.W.H. and M.P.G.; Writing—review and editing, C.W.H., C.H.V.G., N.F.V.E., M.F.R.S.v.d.B., O.R., K.M.V.A., O.R.M., J.M., D.B., N.J.D.W., M.P.G. and The COVID-19 and Cancer Consortium. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by ZonMw—The Netherlands Organization for Health Research and Development (grant number 10430022010014). The funding sources had no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report, and in the decision to submit the article for publication.

Institutional Review Board Statement

This study was not subject to the Dutch Medical Research Involving Human Subjects Act, and therefore, the Research Ethics Committee of the University Medical Center Utrecht judged the study exempt from assessment (21-440/C).

Informed Consent Statement

This study utilizes aggregated data from a dynamic longitudinal cohort, with informed opt-out consent. Patient consent was waived for this particular study because non-identifiable aggregated data were used.

Data Availability Statement

The de-identified participant data used for the analysis can be disclosed by the corresponding author upon reasonable request.

Acknowledgments

We would like to thank over 40 medical students from UU medical school who supported the data collection. We also thank the data managers and coordinators affiliated with the retrospective regional databases for sharing and extracting the data for the purposes of this scientific research: Pauline Slottje, Research Coordinator ANH-VUmc Academic Network of General Practice database, Amsterdam UMC location Vrije Universiteit; Hanna Joosten, the Academic Network of General Practice Amsterdam UMC (ANHA); Feikje Groenhof and Ronald Wilmink, University of Groningen, University Medical Center Groningen (AHON database); and Huibert Tange and Donovan de Jonge, Maastricht University, Care and Public Health Research Institute, Department of General Practice (RNFM database). In particular, thanks go to Marloes van Beurden and Nicole Boekema, Julius Center for Health Sciences and Primary Care, for developing the queries employed in the databases that made this research possible and for sharing and extracting data of the JGPN—Julius Center, University Medical Center Utrecht. The authors thank the registration team of The Netherlands Comprehensive Cancer Organization (IKNL) for the collection of data for The Netherlands Cancer Registry. The authors also acknowledge the other members of the COVID-19 and Cancer-NL Consortium: J.C. van Hoeve, department of research and development, Netherlands Comprehensive Cancer Organization (IKNL), Utrecht; M.A.W. Merkx, Department of Research and Development, Netherlands Comprehensive Cancer Organization (IKNL), Utrecht; Department IQ Healthcare, Radboud University Nijmegen Medical Center, Nijmegen; I. Dingemans, Dutch Federation of Cancer Patient Organizations (NFK), Utrecht; I.D. Nagtegaal, department of Pathology, Radboud University Nijmegen Medical Center, Nijmegen, on behalf of the Automated Pathology Archive (PALGA); M. van der Schaaf, Department of Insight and Innovation, Dutch Hospital Data (DHD), Utrecht; H.C.P.M. van Weert, Department of General Practice, Amsterdam Public Health, Amsterdam UMC location AMC, Amsterdam; M. Verheij, Department of Radiation Oncology, Radboud University Medical Center, Nijmegen, on behalf of SONCOS (Dutch Multidisciplinary Oncology Foundation); all The Netherlands. This research has been conducted using the ANH VUmc, AHA AMC, AHON, RNFM, and JGPN databases. These are ongoing longitudinal databases containing pseudonymized routine general practice care data extracted from the electronic medical records of participating general practices in different regions of The Netherlands. These databases can be used for scientific research relevant for (general practice) care. Patients are informed about the databases by their general practices. The databases contain data from all enlisted patients, except for those who object to this (opt-out). Detailed information about the cancer diagnoses was obtained from The Netherlands Cancer Registration (NCR). The NCR is a population-based registry with detailed diagnostic and therapeutic data of over 99% of Dutch cancer patients since 1989.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Joy, M.; McGagh, D.; Jones, N.; Liyanage, H.; Sherlock, J.; Parimalanathan, V.; Akinyemi, O.; Van Vlymen, J.; Howsam, G.; Marshall, M.; et al. Reorganisation of primary care for older adults during COVID-19: A cross-sectional database study in the UK. Br. J. Gen. Pract. 2020, 70, e540–e547. [Google Scholar] [CrossRef] [PubMed]
  2. Naughton, F.; Ward, E.; Khondoker, M.; Belderson, P.; Marie Minihane, A.; Dainty, J.; Hanson, S.; Holland, R.; Brown, T.; Notley, C. Health behaviour change during the UK COVID-19 lockdown: Findings from the first wave of the C-19 health behaviour and well-being daily tracker study. Br. J. Health Psychol. 2021, 26, 624–643. [Google Scholar] [CrossRef]
  3. Moynihan, R.; Sanders, S.; Michaleff, Z.A.; Scott, A.M.; Clark, J.; To, E.J.; Jones, M.; Kitchener, E.; Fox, M.; Johansson, M.; et al. Impact of COVID-19 pandemic on utilisation of healthcare services: A systematic review. BMJ Open 2021, 11, e045343. [Google Scholar] [CrossRef]
  4. Spicer, J.; Chamberlain, C.; Papa, S. Provision of cancer care during the COVID-19 pandemic. Nat. Rev. Clin. Oncol. 2020, 17, 329–331. [Google Scholar] [CrossRef] [Green Version]
  5. Helsper, C.W.; Campbell, C.; Emery, J.; Neal, R.D.; Li, L.; Rubin, G.; van Weert, H.; Vedsted, P.; Walter, F.M.; Weller, D.; et al. Cancer has not gone away: A primary care perspective to support a balanced approach for timely cancer diagnosis during COVID-19. Eur. J. Cancer Care 2020, 29, e13290. [Google Scholar] [CrossRef] [PubMed]
  6. De Vincentiis, L.; Carr, R.A.; Mariani, M.P.; Ferrara, G. Cancer diagnostic rates during the 2020 ‘lockdown’, due to COVID-19 pandemic, compared with the 2018–2019: An audit study from cellular pathology. J. Clin. Pathol. 2021, 74, 187–189. [Google Scholar] [CrossRef] [PubMed]
  7. Dinmohamed, A.G.; Visser, O.; Verhoeven, R.H.A.; Louwman, M.W.J.; van Nederveen, F.H.; Willems, S.M.; Merkx, M.A.W.; Lemmens, V.E.P.P.; Nagtegaal, I.D.; Siesling, S. Fewer cancer diagnoses during the COVID-19 epidemic in the Netherlands. Lancet Oncol. 2020, 21, 750–751. [Google Scholar] [CrossRef]
  8. Marques, N.P.; Silveira, D.M.M.; Marques, N.C.T.; Martelli, D.R.B.; Oliveira, E.A.; Martelli-Júnior, H. Cancer diagnosis in Brazil in the COVID-19 era. Semin. Oncol. 2021, 48, 156–159. [Google Scholar] [CrossRef]
  9. Sud, A.; Torr, B.; Jones, M.E.; Broggio, J.; Scott, S.; Loveday, C.; Garrett, A.; Gronthoud, F.; Nicol, D.L.; Jhanji, S.; et al. Effect of delays in the 2-week-wait cancer referral pathway during the COVID-19 pandemic on cancer survival in the UK: A modelling study. Lancet Oncol. 2020, 21, 1035–1044. [Google Scholar] [CrossRef]
  10. Neal, R.D.; Tharmanathan, P.; France, B.; Din, N.U.; Cotton, S.; Fallon-Ferguson, J.; Hamilton, W.; Hendry, A.; Hendry, M.; Lewis, R.; et al. Is increased time to diagnosis and treatment in symptomatic cancer associated with poorer outcomes? Systematic review. Br. J. Cancer 2015, 112, S92–S107. [Google Scholar] [CrossRef] [Green Version]
  11. Hansen, R.P.; Vedsted, P.; Sokolowski, I.; Søndergaard, J.; Olesen, F. Time intervals from first symptom to treatment of cancer: A cohort study of 2,212 newly diagnosed cancer patients. BMC Health Serv. Res. 2011, 11, 284. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  12. Weller, D.; Vedsted, P.; Rubin, G.; Walter, F.M.; Emery, J.; Scott, S.; Campbell, C.; Andersen, R.S.; Hamilton, W.; Olesen, F.; et al. The Aarhus statement: Improving design and reporting of studies on early cancer diagnosis. Br. J. Cancer 2012, 106, 1262–1267. [Google Scholar] [CrossRef]
  13. Van Weert, H. After the first wave: What effects did the COVID-19 measures have on regular care and how can general practitioners respond to this? Eur. J. Gen. Pract. 2020, 26, 126–128. [Google Scholar] [CrossRef] [PubMed]
  14. Nicholson, B.D.; Ordóñez-Mena, J.M.; Lay-Flurrie, S.; Sheppard, J.P.; Liyanage, H.; McGagh, D.; Sherlock, J.; Williams, J.; Smith, M.; Drakesmith, C.W.; et al. Consultations for clinical features of possible cancer and associated urgent referrals before and during the COVID-19 pandemic: An observational cohort study from English primary care. Br. J. Cancer 2021, 126, 948–956. [Google Scholar] [CrossRef] [PubMed]
  15. Jones, D.; Neal, R.D.; Duffy, S.R.G.; Scott, S.E.; Whitaker, K.L.; Brain, K. Impact of the COVID-19 pandemic on the symptomatic diagnosis of cancer: The view from primary care. Lancet Oncol. 2020, 21, 748–750. [Google Scholar] [CrossRef] [PubMed]
  16. Siepman Van den Berg, M.F.R.; Grant, M.; van Erp, N.; van Gils, C.H.; Muris, J.; Brandenbarg, D.; van Asselt, K.M.; de Wit, N.J.; Helsper, C.W. Cancer detection during the COVID-19 pandemic—Experiences in primary care and recommendations for the future. J. Gen. Fam. Med. 2022. [Google Scholar] [CrossRef]
  17. Von Elm, E.; Altman, D.G.; Egger, M.; Pocock, S.J.; Gøtzsche, P.C.; Vandenbroucke, J.P. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: Guidelines for reporting observational studies. Bull. World Health Organ. 2007, 85, 867–872. [Google Scholar] [CrossRef]
  18. Smeets, H.M.; Kortekaas, M.F.; Rutten, F.H.; Bots, M.L.; Van Der Kraan, W.; Daggelders, G.; Smits-Pelser, H.; Helsper, C.W.; Hoes, A.W.; De Wit, N.J. Routine primary care data for scientific research, quality of care programs and educational purposes: The Julius General Practitioners’ Network (JGPN). BMC Health Serv. Res. 2018, 18, 735. [Google Scholar] [CrossRef]
  19. Van der Sanden, G.A.C.; Coebergh, J.-W.; Schouten, L.J.; Visser, O.; van Leeuwen, F.E. Cancer incidence in the Netherlands in 1989 and 1990: First results of the nationwide Netherlands cancer registry. Eur. J. Cancer 1995, 31, 1822–1829. [Google Scholar] [CrossRef]
  20. Sollie, A.; Roskam, J.; Sijmons, R.H.; Numans, M.E.; Helsper, C.W. Do GPs know their patients with cancer? Assessing the quality of cancer registration in Dutch primary care: A cross-sectional validation study. BMJ Open 2016, 6, e012669. [Google Scholar] [CrossRef] [Green Version]
  21. Helsper, C.W.; van Erp, N.F.; Peeters, P.H.M.; de Wit, N.J. Time to diagnosis and treatment for cancer patients in the Netherlands: Room for improvement? Eur. J. Cancer 2017, 87, 113–121. [Google Scholar] [CrossRef] [Green Version]
  22. Van Erp, N.F. Exploring the Diagnostic Pathway of Symptomatic Cancer Patients in The Netherlands. Ph.D. Thesis, Utrecht University, Utrecht, The Netherlands, 2021; pp. 1–208. [Google Scholar] [CrossRef]
  23. Coyer, L.; Boyd, A.; Schinkel, J.; Agyemang, C.; Galenkamp, H.; Koopman, A.D.; Leenstra, T.; van Duijnhoven, Y.T.; van Charante, E.P.M.; Born, B.-J.H.V.D.; et al. Differences in SARS-CoV-2 infections during the first and second wave of SARS-CoV-2 between six ethnic groups in Amsterdam, the Netherlands: A population-based longitudinal serological study. Lancet Reg. Health Eur. 2021, 13, 100284. [Google Scholar] [CrossRef]
  24. O’Halloran, J.; Miller, G.C.; Britt, H. Defining chronic conditions for primary care with ICPC-2. Fam. Pract. 2004, 21, 381–386. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  25. Archer, S.; Calanzani, N.; Honey, S.; Johnson, M.; Neal, R.; Scott, S.; Walter, F. Impact of the COVID-19 pandemic on cancer assessment in primary care: A qualitative study of GP views. BJGP Open 2021, 5, BJGPO.2021.0056. [Google Scholar] [CrossRef] [PubMed]
  26. Splinter, M.J.; Velek, P.; Ikram, M.K.; Kieboom, B.C.T.; Peeters, R.P.; Bindels, P.J.E.; Wolters, F.J.; Leening, M.J.G.; de Schepper, E.I.T.; Licher, S. Prevalence and determinants of healthcare avoidance during the COVID-19 pandemic: A population-based cross-sectional study. PLoS Med. 2021, 18, e1003854. [Google Scholar] [CrossRef] [PubMed]
  27. Grant, M.P.; Helsper, C.W.; Stellato, R.; van Erp, N.; van Asselt, K.M.; Slottje, P.; Muris, J.; Brandenbarg, D.; de Wit, N.J.; van Gils, C.H. The Impact of the COVID Pandemic on the Incidence of Presentations with Cancer-Related Symptoms in Primary Care. Cancers 2022, 14, 5353. [Google Scholar] [CrossRef]
  28. Schuster, N.A.; de Breij, S.; Schaap, L.A.; van Schoor, N.M.; Peters, M.J.L.; de Jongh, R.T.; Huisman, M.; Hoogendijk, E.O. Older adults report cancellation or avoidance of medical care during the COVID-19 pandemic: Results from the Longitudinal Aging Study Amsterdam. Eur. Geriatr. Med. 2021, 12, 1075–1083. [Google Scholar] [CrossRef]
  29. Smith, C.F.; Nicholson, B.D.; Hirst, Y.; Fleming, S.; Bankhead, C.R. Primary care practice and cancer suspicion during the first three COVID-19 lockdowns in the UK: A qualitative study. Br. J. Gen. Pract. 2022, 72, e713–e721. [Google Scholar] [CrossRef]
  30. Ling, Y.; Cheung, M.C.; Chan, K.K.; Lofters, A.; Fox, C.; Patrikar, A.; Liu, N.; Singh, S. Primary care utilization for patients with newly diagnosed cancer during the COVID-19 pandemic: A population-based study. BMC Cancer 2022, 22, 1–9. [Google Scholar] [CrossRef]
Cancers 15 01462 g001 550
Figure 1. Primary care cancer diagnostic intervals included in the pathway from time of first symptom presentation to diagnosis, including the primary care interval (IPC) and the secondary care interval (ISC), as detailed in the Aarhus statement [12].
Figure 1. Primary care cancer diagnostic intervals included in the pathway from time of first symptom presentation to diagnosis, including the primary care interval (IPC) and the secondary care interval (ISC), as detailed in the Aarhus statement [12].
Cancers 15 01462 g001
Cancers 15 01462 g002 550
Figure 2. Distribution of primary care interval (IPC) durations for each cancer type, comparing IPC durations pre-COVID-19 with those during COVID-19. P50 marks the median, P25 the 25th percentile, and P75 the 75th percentile.
Figure 2. Distribution of primary care interval (IPC) durations for each cancer type, comparing IPC durations pre-COVID-19 with those during COVID-19. P50 marks the median, P25 the 25th percentile, and P75 the 75th percentile.
Cancers 15 01462 g002
Table
Table 1. Patient demographic and clinical characteristics per cancer type for pre-COVID-19 (2012–2015) and the first COVID-19 wave (March–June 2020). Registered comorbidity was defined in accordance with O’Halloran et al. [24].
Table 1. Patient demographic and clinical characteristics per cancer type for pre-COVID-19 (2012–2015) and the first COVID-19 wave (March–June 2020). Registered comorbidity was defined in accordance with O’Halloran et al. [24].
Cancer TypeColorectalLungBreastMelanoma
Time PeriodPre-COVID-19COVID-19Pre-COVID-19COVID-19Pre-COVID-19COVID-19Pre-COVID-19COVID-19
Primary Care Interval
N31311023611830614012447
Female—n (%)154 (49)66 (60)104 (44)62 (53)306 (100)140 (100)61 (49)28 (60)
Age at first GP consultation—mean (SD)69.5 (12.5)68.4 (14.4)68.6 (11.1)66.4 (10.9)57.5 (18.2)58.9 (17.4)55.0 (17.0)60.7 (18.1)
Registered comorbidity—n (%)287 (92)90 (82)218 (92)96 (81)241 (79)90 (64)85 (69)31 (66)
Psychiatric comorbidity—n (%)79 (25)20 (18)48 (20)27 (23)52 (17)42 (30)28 (20)8 (17)
Secondary Care Interval
N259621977525611010626
Female—n (%)130 (50)33 (53)87 (44)42 (56)256 (100)110 (100)52 (49)14 (54)
Age—mean (SD)69.1 (12.4)67.8 (14.7)68.6 (10.6)67.0 (9.6)57.8 (17.4)57.3 (17.3)58.3 (15.2)65.0 (16.8)
Registered comorbidity—n (%)236 (91)52 (84)182 (92)65 (87)202 (79)74 (67)77 (73)23 (89)
Psychiatric comorbidity—n (%)66 (26)13 (21)40 (20)14 (19)44 (17)32 (29)19 (18)2 (8)
Table
Table 2. Impact of first COVID-19 wave on duration of the primary care interval—total and subgroups.
Table 2. Impact of first COVID-19 wave on duration of the primary care interval—total and subgroups.
Cancer Type and COVID-19 ImpactColorectal CancerLung Cancer
Pre COVID-191st Wave COVID-19p for Diff.Absolute Increase Median Duration (Days)Pre COVID-191st Wave COVID-19p for Diff. Absolute Increase Median Duration (Days)
Median IPC Duration in Days (IQR)Median IPC Duration Days (IQR)
Total
Number of patients (N)
N = CPC: 313 CC: 110 LPC: 166 LC: 118		5
(1–29)		44
(6–230)		<0.013915
(3–47)		41
(7–102)		<0.0126
Gender
Male—N: CPC: 159 CC: 44 LPC: 132 LC: 56		3
(1–25)		30
(2–177)		<0.012713
(2–40)		46
(7–98)		<0.0133
Female—N: CPC: 154 CC:66 LPC: 104 LC: 627
(1–30)		66
(8–252)		<0.015918
(5–50)		40
(6–107)		0.0422
Age
<65—N: CPC: 100 CC: 37 LPC: 83 LC: 50		4
(1–24)		67
(12–240)		<0.016315
(5–51)		31
(7–92)		0.0516
≥65—N: CPC: 213 CC: 73 LPC: 153 LC: 686
(1–32)		37
(4–231)		<0.013115
(3–44)		47
(7–105)		<0.0132
Comorbidities present
<2—N: CPC: 60 CC: 44 LPC: 53 LC: 43		2
(1–15)		57
(2–231)		<0.015510
(1–24)		29
(8–99)		<0.0119
≥2—N: CPC: 247 CC: 66 LPC: 177 LC: 756
(1–31)		42
(8–231)		<0.013620
(5–51)		46
(6–108)		<0.0127
Psychiatric comorbidity
Yes—N: CPC: 79 CC: 20 LPC: 48 LC: 27		5
(1–19)		70
(22–259)		<0.016519
(5–50)		55
(11–123)		<0.0136
No—N: CPC: 230 CC: 90 LPC: 187 LC: 915
(1–33)		27.5
(4–216)		<0.0122.515
(2–44)		32
(5–92)		<0.0117
Alarm symptom at first consult
Yes—N: CPC: 169 CC: 45 LPC: 188 LC: 45		1
(1–18)		37
(13–169)		<0.013618
(1–18)		37
(2–169)		0.619
No—N: CPC: 144 CC: 65 LPC: 188 LC: 10111.5
(1–35)		52
(13–252)		<0.014018
(6–50)		48
(11–108)		<0.0130
GP consult frequency—prior year
<5—N: CPC: 130 CC: 56 LPC: 120 LC: 71		4
(1–26)		25
(4–189)		<0.012111
(2–37)		39
(9–88)		<0.0128
≥5—N: CPC: 103 CC: 54 LPC: 66 LC: 476
(1–29)		50
(10–232)		<0.014418.5
(6–50)		45
(5–108)		0.0426.5
(N): number of patients; CPC: colorectal cancer pre-COVID-19; CC: colorectal cancer 1st wave COVID-19; LPC: lung cancer pre-COVID-19; LC: lung cancer 1st wave COVID-19. If numbers do not add up to expected total, this is due to missing (left out) information. p. for diff.: p-value for difference in IPC durations pre-COVID-19 and during first COVID-19 wave. No significance testing for differences in COVID-19 impact between subgroups within cancer populations was performed because of low N. IQR: interquartile range. Periods: pre-COVID-19: 2012–2015; first COVID-19 wave: March–June 2020. Comorbidity was defined in accordance with O’Halloran et al. [24].
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Helsper, C.W.; Van Gils, C.H.; Van Erp, N.F.; Siepman van den Berg, M.F.R.; Rogouti, O.; Van Asselt, K.M.; Maarsingh, O.R.; Muris, J.; Brandenbarg, D.; Siesling, S.; et al. Impact of the COVID-19 Outbreak—Delayed Referral of Colorectal and Lung Cancer in Primary Care: A National Retrospective Cohort Study. Cancers 2023, 15, 1462. https://doi.org/10.3390/cancers15051462

AMA Style

Helsper CW, Van Gils CH, Van Erp NF, Siepman van den Berg MFR, Rogouti O, Van Asselt KM, Maarsingh OR, Muris J, Brandenbarg D, Siesling S, et al. Impact of the COVID-19 Outbreak—Delayed Referral of Colorectal and Lung Cancer in Primary Care: A National Retrospective Cohort Study. Cancers. 2023; 15(5):1462. https://doi.org/10.3390/cancers15051462

Chicago/Turabian Style

Helsper, Charles W., Carla H. Van Gils, Nicole F. Van Erp, Marinde F. R. Siepman van den Berg, Omar Rogouti, Kristel M. Van Asselt, Otto R. Maarsingh, Jean Muris, Daan Brandenbarg, Sabine Siesling, and et al. 2023. "Impact of the COVID-19 Outbreak—Delayed Referral of Colorectal and Lung Cancer in Primary Care: A National Retrospective Cohort Study" Cancers 15, no. 5: 1462. https://doi.org/10.3390/cancers15051462

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop