Next Article in Journal
Association between the blaCTX-M-14-harboring Escherichia coli Isolated from Weasels and Domestic Animals Reared on a University Campus
Next Article in Special Issue
Exploring the One Health Perspective in Sweden’s Policies for Containing Antibiotic Resistance
Previous Article in Journal
Natural and Enantiopure Alkylglycerols as Antibiofilms Against Clinical Bacterial Isolates and Quorum Sensing Inhibitors of Chromobacterium violaceum ATCC 12472
Previous Article in Special Issue
The Governance and Implementation of the National Action Plan on Antimicrobial Resistance in Tanzania: A Qualitative Study
Open AccessSystematic Review

Quality and Utility of Information Captured by Surveillance Systems Relevant to Antimicrobial Resistance (AMR): A Systematic Review

1
Department of Health Services Research and Policy, London School of Hygiene and Tropical Medicine, London WC1H 9SH, UK
2
Department of Public Health, Environments and Society, London School of Hygiene and Tropical Medicine, London WC1E 7HT, UK
*
Author to whom correspondence should be addressed.
Academic Editor: Helena Legido-Quigley
Antibiotics 2021, 10(4), 431; https://doi.org/10.3390/antibiotics10040431
Received: 10 March 2021 / Revised: 9 April 2021 / Accepted: 11 April 2021 / Published: 13 April 2021

Abstract

Health surveillance systems are considered vital for combatting antimicrobial resistance (AMR); however, the evidence-base on the effectiveness of these systems in providing information that can be used by healthcare professionals, or the acceptability of these systems by users, has not been reviewed. A systematic review was conducted of a number of databases to synthesise the evidence. The review identified 43 studies that met the inclusion criteria, conducted in 18 countries and used 11 attributes in their assessment of surveillance systems. The majority of systems evaluated were for monitoring the incidence of tuberculosis. The studies found that most surveillance systems were underperforming in key attributes that relate to both effectiveness and acceptability. We identified that two features of systems (ease of use and users’ awareness of systems) were associated with greater acceptability and completeness of systems. We recommend prioritising these for the improvement of existing systems, as well as ensuring consistency in the definition of attributes studied, to allow a more consistent approach in evaluations of surveillance systems, and to facilitate the identification of the attributes that have the greatest impact on the utility of data produced.
Keywords: antimicrobial resistance; surveillance systems; attributes antimicrobial resistance; surveillance systems; attributes

1. Introduction

The process of antimicrobial resistance (AMR), whereby microbes evolve over time to become less susceptible to medicines, has reduced the ability of healthcare providers to treat common infections. It is a naturally occurring process that takes place through genetic mutations. The main drivers for AMR include the misuse and overuse of antimicrobials; lack of access to clean water, sanitation and hygiene; poor infection and disease prevention and control; poor access to quality, affordable medicines, vaccines and diagnostics; and lack of enforcement of legislation [1]. In the context of combating AMR, the World Health Organisation (WHO) views surveillance systems as being essential for providing information on the magnitude and trends in AMR and for monitoring the effects of interventions [2].
The most widely recognised guidelines for establishing the utility of the outputs of a surveillance system are those developed by the US Centers for Disease Control and Prevention (CDC) [3]. These describe the usefulness of a system in terms of its contribution to the prevention and control of adverse health events, including an understanding of the implications of those events, based on consideration of nine attributes: Simplicity, Flexibility, Data Quality, Acceptability, Sensitivity, Positive Predictive Value (PPV), Representativeness, Timeliness, and Stability (See Appendix A for definitions). The overall usefulness of a surveillance system is considered to be influenced (to varying degrees depending on the aim of the system) by these attributes [3]. For example, a public health surveillance system that is simple, flexible, acceptable, and stable is more likely to be useful for public health action. However, the CDC acknowledges that there is no perfect system and that focusing resources to improve one attribute might have an adverse effect on another; for example, as sensitivity increases, the PPV might decrease, and efforts to increase sensitivity and PPV could result in a more complex surveillance system with reduced acceptability and timeliness [3]. There is uncertainty about what constitute the most effective characteristics of a system.
A rapid review of the literature identified no systematic reviews of evaluations of the ability of surveillance systems to provide information that can be used by health care professionals to combat AMR. Such a synthesis could contribute to the evidence base when deciding whether to invest or continue investing in these systems in the future. Hence, the aim of this systematic review was to synthesise the evidence from evaluations of the quality and utility of information produced by surveillance systems that monitor organisms and conditions important in the AMR context, with a focus on human rather than animal surveillance systems. The research questions were:
  • What is the effectiveness of AMR-relevant surveillance systems in providing information that can be used to inform healthcare professionals?
  • What is the acceptability of these systems to users?

2. Materials and Methods

The study protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO), registration number: CRD42018085346.
In order to answer the first research question (the effectiveness of systems), the following study designs were eligible for inclusion:
  • Prospective observational studies (controlled and uncontrolled before and after studies).
  • Retrospective observational evaluations, including case-control studies, retrospective cohort studies, and audits. Data sources included primary data collected for research and secondary data (for example, health insurance claim data).
  • Interventions using an experimental design.
Evidence from qualitative research (such as interviews and focus groups) was considered eligible for answering the second research question on the acceptability of systems.
Evaluations of surveillance systems that monitored the following organisms were eligible for inclusion:
  • Bacteria whose antibiotic susceptibility status was recorded by the surveillance system.
  • Bacteria relevant to AMR. A list was collated from the key AMR threats that have been identified by the WHO [4], the CDC [5], European Centre for Disease Control (ECDC) [6], European Food Safety Authority (EFSA) [7], and the key drug-bug combinations identified by Public Health England in the UK AMR Strategy [8].
  • The following types of evaluation study were excluded from the review:
  • Evaluations of public surveillance systems that monitor non-bacterial microorganisms (for example, viruses or fungi).
  • Evaluations of surveillance systems that monitor bacterial microorganisms that are not on any of the priority lists described in the inclusion criteria above.
  • Screening systems that are limited to a single or group of hospitals, and where the information is not shared outside the hospital system.
  • Studies published prior to 1988, when the first CDC guidelines for evaluating Public Health Surveillance systems were published.
  • Articles published in languages other than English.
The following databases were searched for relevant articles from 1 January 1988 until 1 June 2018: OVID Medline; EMBASE; Cochrane Central Register of Controlled Trials (CENTRAL); Global Health (OVID); Web of Science; Open Grey and Scopus.
The search terms used can be found in Appendix B. They were adapted for databases in accordance with the repository’s interface and search options. All search strings were run in English, and all the records were exported to Endnote reference management software v 18.0.2 and Excel 2016. In addition to searching databases, we also performed reference searches of the identified.

2.1. Outcomes

The primary outcome for inclusion in the review, to answer the research question on effectiveness, were attributes of surveillance systems, as described by the authors. Aspects of acceptability, such as ease of use, reported in qualitative data, were the outcomes of interest for answering the second research question.

2.2. Quality Assessment of Studies

The Critical Appraisal Skills Programme (CASP) checklist [9] was used for assessing the quality of reporting of the qualitative studies included in the review, and the Effective Public Health Practice Project (EPHPP) Quality Assessment Tool was used for those that included a quantitative component [10].

2.3. Analysis

Given the heterogeneity among study designs and contexts, a narrative review was conducted following guidelines for narrative synthesis [11].

3. Results

3.1. Outcome of Study Identification Process

A total of 49,000 records were identified from databases and manual searches, of which 43 studies were included in the review. Figure 1 provides the details of the records excluded at each stage of the screening process.

3.2. Characteristics of Included Studies

3.2.1. Study Design

The majority of studies were exclusively quantitative (38/43), with the other five including a qualitative component. None of the studies was purely qualitative. The majority were retrospective analyses of routinely collected data (41/43) (Table 1). Of these, five combined retrospective quantitative analysis with semi-structured interviews and another five incorporated capture/recapture methods to measure the completeness of data.
In over half of the studies (29/43), the system under investigation was compared to other sources of data (laboratory reports, prescription data, medical records, or health insurance claims data) or alternative notification technologies. These comparators had been treated by researchers in two different ways:
  • Use of another surveillance system (such as the CDC Emerging Infections Programme) as a high-quality reference standard against which to compare (for example, Nguyen et al. [12])
  • Comparison between different methods of data collection and reporting, including comparing electronic reporting against other forms of reporting, for example, Saeed et al. [13].
The sample size of the included studies, which in most instances was the number of patients or samples, ranged from 35 in an evaluation of a system for reporting active Tuberculosis (TB) among US military service members [14] to a maximum of 251,693 records in an evaluation of TB surveillance in Brazilian micro-regions [15].

3.2.2. Quality of Studies

The CASP checklist [9] of qualitative research identified a number of problems in the five studies that included such methods [13,15,16,17,18]. These included the suitability of using a qualitative approach for analysing predominately closed questions [13,15] and the absence in all studies of details of how data were collected or analysed. However, all studies included an introductory section that stated clearly the aims of the study.
The quality assessment of the studies that included a quantitative component, using the Effective Public Health Practice Project (EPHPP) Quality Assessment Tool for Quantitative Studies [19], resulted in all of the studies receiving an overall rating of weak. This was primarily the result of the non-randomised sampling cases from systems and the lack of clarity over how representative the samples (case records and/or time periods covered) were of the wider system under investigation.
An analysis of the potential for sampling bias in the 30 studies [12,18,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47] that used retrospective analysis of routinely collected data revealed that the rationale for the choice of the time period and the geographic area was not provided in all studies. Furthermore, the studies did not all include a description of who had collected and analysed the data and whether they had an affiliation with the programme (i.e., whether they were independent or not). These are potential sources of data collection and analysis bias. However, studies that included the use of the capture-recapture statistical method to measure the completeness of systems [14,42,48,49,50] all acknowledged the potential sources of bias that could result in the violation of any of the principles that underlie the capture-recapture method [51].

3.2.3. Setting

The evaluations included took place in 18 countries (Table 2), with the largest number in the US (ten studies), followed by Australia and the UK (four evaluations each). The included articles were published between 1991 and 2016.

3.2.4. Surveillance Systems’ Attributes Evaluated

The 43 studies included used 18 attributes in their assessment of health surveillance systems. However, on inspecting the description of these attributes and how they were applied in the studies, they could be grouped into 11, as some authors were using different terms to describe the same attributes. For example, the term “completeness” was sometimes used to describe the proportion of cases of a condition that were detected by the surveillance system under examination. The same definition was used to describe “sensitivity” in other studies. Furthermore, the authors used the same term to describe different characteristics of systems. For example, “completeness” (defined above) was also used to describe the extent to which forms containing the details of each case identified were complete. A description of attributes as they were used in the studies included in this review can be found in Table 3.
The number of attributes examined in each evaluation ranged from one to ten. The attribute most commonly used (20/43) was “completeness” (the proportion of cases identified by the system). Appendix C provides the list of attributes used in each evaluation.
Two of the attributes identified in the systematic review were not identified in the CDC guidelines [3]; these were concordance and specificity. However, specificity was described by the CDC as influencing the positive predictive value (an attribute included in the CDC list).

3.2.5. Health Conditions and Microorganisms Monitored by Systems

The majority of surveillance systems were monitoring the incidence of TB (22/43). Table 4 provides details of the number of evaluations by health condition. Appendix D lists the attributes examined by health condition or microorganism.

3.3. Performance of Systems in Relation to Attributes Assessed

The findings of the narrative synthesis are presented for each of the 11 attributes of surveillance systems (as defined in Table 3) examined in relevant studies, starting with the attributes that relate to effectiveness.

3.3.1. Specificity

The only study to consider this attribute [39] was an evaluation of the National Mycobacterial Surveillance System that screened for TB among new refugees in New South Wales, Australia. The study used retrospectively collected data and reported that in nearly one-third of the cases notified as having active TB (n = 60), the individuals did not actually have active disease, which suggests low specificity and over-estimation of disease levels.

3.3.2. Usefulness

Saeed et al. [13] was the only included study to look at this attribute. It investigated the possible use of the data generated by two surveillance systems in Afghanistan: the Health Management Information System (HMIS) and the National Tuberculosis Control Programme (NTP), using interviews with key informants. They reported that the HMIS data were useful for planning and monitoring but less so for detecting outbreaks (as a result of poor coordination with the national programme). In contrast, proper analysis of the data from the NTP system could detect and allow a response to outbreaks, as they are available to stakeholders at both national and international levels.

3.3.3. Completeness

As identified above, the term “completeness” was used to describe two distinct attributes: the proportion of cases of the condition that are picked up by the system (also referred to as sensitivity or coverage); and the extent to which the fields in the forms are completed. The findings in relation to these two attributes will be covered in turn.
Twenty studies used the first interpretation of the term in their assessment of surveillance systems [12,13,14,20,21,22,23,24,25,26,27,28,29,30,48,49,50,52,53,54]. They used retrospective analysis of routinely collected data [12,13,14,20,21,22,23,24,25,26,27,28,30,49,54] and retrospective analysis of routinely collected data combined with capture-recapture statistical methods [29,50,52,53], with only one study [48] opting to collect collected data prospectively. The proportion of cases captured by the different surveillance systems was compared with a variety of alternative sources of data, or estimated using the capture-recapture method, and ranged from 45% of cases in a TB surveillance system in Saudi Arabia [20] to 99.9% for a salmonellosis surveillance system in Sweden [52].
Most studies concluded that the completeness of the surveillance systems could be improved and made suggestions for how this could be achieved. This included switching to electronic reporting and training of personnel [27]; raising awareness among healthcare practitioners about the importance of reporting the conditions under surveillance [24]; incorporating data from other data collection systems (from primary care, hospitals, and pharmacies) [12,14,26,28,53]; making the notification process simpler [54].
The studies also identified characteristics that were associated with a higher or lower likelihood of cases being reported. An evaluation of the Swedish statutory surveillance system for communicable disease, which monitors salmonellosis and penicillin-resistant pneumococci, alongside other conditions, observed that it had higher sensitivity for recording diseases with a longer tradition of reporting (such as salmonellosis) [52]. An evaluation of TB notification in the Republic of Korea [21] noted that the type of medical institution (e.g., clinic or general hospital) and the nationality of the patient influenced the likelihood of cases being reported, with patients from general hospitals and those who were Korean nationals being more likely to be reported than those from clinics and foreign nationals respectively. The evaluation of a TB surveillance system in Italy [22] observed that under-notification was significantly higher in female patients and those with extra-pulmonary TB than in male patients and those with pulmonary TB, respectively. The surveillance of Haemophilus influenza was reported to have lower completeness following the introduction of routine immunisation in 1992, possibly leading to an over-estimation of the effectiveness of the vaccination programme [30].
Thirteen studies [13,15,17,20,22,26,31,32,33,34,35,36,54] investigated the second interpretation of the term “completeness” (the extent to which fields are completed in forms). All studies used retrospective analysis of routinely collected data to determine completeness. There was variation in how studies calculated the completeness of the fields in the case forms, with different approaches to classifying fields being used. For example, one study split fields into demographic and clinical variables [17] whilst another focused on a number of critical fields, for example, the site of the disease [31].
Completeness ranged from 34% to 90%, but the extent to which this was considered acceptably varied depending on the nature of the condition under surveillance, with a general view expressed that action was needed to be taken to improve rates. A concerning finding of the national TB surveillance system in Afghanistan was that the audit uncovered fields that were completed entirely incorrectly, whereby facilities reported examining sputum smears when they did not have a microscope [13].

3.3.4. Concordance

Eight studies included this attribute [15,17,18,20,26,29,36,37], using retrospective analysis of routinely collected data [15,17,18,26,36,37] alongside one study that additionally combined this with capture-recapture statistical methods [29]. These studies established the accuracy of data within surveillance systems by exploring the level of agreement of data collected either at different organisational levels (e.g., facility and province) [18], from different sources (microbiologists or clinicians) [36,37] or in different formats (paper sources and electronic formats) [20,26]. In addition, data from surveillance systems were compared with mandatory notification systems [29]. Generally, the level of agreement between sources was found to be higher for patient demographic data (k > 0.80) than for clinical data [17,18,26,36,37]. Sprinson et al. [36] identified the importance of concordant data to enable programme planning and evaluations, particularly in areas such as policy development and programme advocacy. Podewils et al. [26] recommended unifying different systems (paper, electronic and laboratory) to reduce these discrepancies.

3.3.5. Timeliness

This attribute was included in 18 studies [13,15,16,21,27,31,34,40,41,42,43,44,45,46,47,54,55,56], all using the same method (retrospective analysis of routinely collected data) but different reference periods to measure timeliness. These included the interval between the onset of symptoms and the notification to health authorities [40,43]; specimen collection to case reporting [27,42,54]; isolation at the primary laboratory to reporting to the surveillance body [47]; onset of symptoms to completion of case investigation [44]; reporting within one week of starting treatment [31]; starting treatment to notification [21]; notification within one or two incubation periods [40]; reporting delays regularly more than three months [13]. With the exception of one [16], all evaluations reported that timeliness of reporting could be improved, and some recommended that future research should explore the bottlenecks in the reporting process [41,42]. Suggestions made for improving the timeliness of reporting included transmitting laboratory reports electronically [56], reducing the laboratory electronic reporting period [55], and sending a sample of isolates to the national reference laboratory while the isolate is being analysed at the primary laboratory to reduce the delays incurred by waiting for the results of partial analysis [47].

3.3.6. Positive Predictive Value (PPV)

This attribute was examined in five studies [13,14,24,35,36], using retrospective analysis of routinely collected data. There was wide variation, with the lowest value (1.1%) being reported for one variable in the TB surveillance system in California [36], whereas the TB Notification System of the United States Military scored 100% [14]. Stenhem et al. [35] observed that variation in the PPV (calculated by comparing the information in the first notification with the information in the study database) depended on the variable collected, Sprinson et al. [36] observed wider variations within their dataset when they compared the data on a sample of cases from a TB registry with medical records.

3.3.7. Representativeness

Representativeness was included in only three evaluations [13,16,38], all using retrospective analysis of routinely collected data. An evaluation of a methicillin-resistant Staphylococcus aureus (MRSA) surveillance system in Japan using health insurance claims [38] observed that the surveillance system was accurately representing the ages of patients. Saeed et al. [13] observed that the HMIS in Afghanistan only included public health facilities with no data from the private sector (which it was attempting to cover). Furthermore, the system favoured rural areas with little attention to urban settings and secondary hospitals. Conversely, the evaluation of the Australian Gonococcal Surveillance Programme [16] observed that the reliance on molecular methods to diagnose infections had reduced the number of isolates available for AMR testing, especially in rural areas where molecular methods were increasingly used due to their greater cost-effectiveness.

3.3.8. Acceptability

This attribute was examined in five studies through the use of surveys of users [16,55] and semi-structured interviews [13,15,54]. These found varying levels of acceptability related to awareness of the system and ease of use of the system. For example, the NTP in Afghanistan was found to have poor acceptability as health workers lacked awareness of the procedures to follow. Delayed reporting (and sometimes failure to report) indicated refusals to co-operate with the protocols [13]. The Australian Gonococcal Surveillance Programme had high acceptability for the reference laboratories that were contributors, even though some of the stakeholders identified the lack of feedback of surveillance data as an issue [16]. The acceptability of the salmonella surveillance system in London and South East England [55] was found to be related to the method used to follow up case reports, with telephone questionnaires being more successful than mailed ones.
An evaluation of a TB surveillance system in Brazil found that acceptability and timeliness were linked to each other when Brazil’s micro-regions were divided into two groups according to the relative performance of their systems [15].

3.3.9. Flexibility

Two studies measured this attribute [13,16], using a survey of stakeholders [16] and interviews [13]. Samaan et al. observed that the reduced availability of isolates for testing (a result of the introduction of molecular-based methods to diagnose gonococcal infections) had challenged the flexibility of the Australian Gonococcal Surveillance Programme. It was trying to adapt to this situation by communicating with the public and private laboratories and asking them to forward any isolates they had available. Another challenge to the Australian system’s flexibility was the introduction of additional antimicrobials for the treatment of gonococcal infections. The system adapted through changes such as modifying its quality assurance process to include new resistance testing.
The two TB surveillance systems in Afghanistan evaluated by Saeed et al. [13] received different ratings, with the NTP being seen as flexible and able to accommodate changes. The HMIS, however, was seen as less flexible (for example, when there was a need for additional information or modes of operation).

3.3.10. Simplicity

This was examined in three evaluations [13,16,54] that used questionnaire surveys [16] or interviews with stakeholders [13,54]. The evaluation of the Campylobacter Infection Surveillance Programme in Victoria (Australia) found it to be “cumbersome” to use for case referral and investigation [54]. Similarly, the Australian Gonococcal Surveillance Programme was observed by Samaan et al. [16] to have reduced the simplicity of the system as a result of duplication of data entry for the isolates received from the initial diagnostic laboratories. However, the survey of stakeholders undertaken as part of the same evaluation found that 88% of respondents felt that the system was sufficiently straightforward. Those who did not find it so cited the poorly defined terminology in reports as one of their concerns. Saeed et al. [13] also reported divergent findings in relation to the simplicity of two TB monitoring systems in Afghanistan (the HMIS and the NTP). The HMIS system was found to be simple as case definitions were followed and consistent forms were used. The NTP system, on the other hand, was found to be complex due to multiple complicated forms that need to be completed, the use of paper forms, a lack of integration with the HIMS system and difficulties in providing training for staff.

3.3.11. Stability

Stability, defined as the ability to collect, manage, and provide data properly without failure and ability to be operational when needed [3], was included as an attribute in only one study [13], which investigated two TB surveillance programmes in Afghanistan using interviews. The authors reported that the HMIS was able to collect, manage and provide data from its facilities and produce monthly reports. The only problem reported that affected stability related to extraction of data from the system as a result of interruptions to the electricity supply. However, the authors also identified computer viruses as a potential threat. The other TB surveillance system in that evaluation was the NTP which was given a lower rating for stability due to being paper-based, possibly resulting in reporting delays.

4. Discussion

This review sought to examine the effectiveness of aspects of the performance of surveillance systems that are considered to be important for AMR monitoring. It found that most surveillance systems evaluated were underperforming in the key attributes related to both effectiveness and acceptability.
Lewis [57] described the overall objective of surveillance of AMR as the facilitation of control of AMR through informing the need to improve prescribing and infection control practices. The author identified three essential attributes for an AMR surveillance system for human health: timeliness, reliability, and representativeness. Timeliness was identified as important for AMR trends at the local level and to assist clinicians in the rational choice of antibiotic. However, the author acknowledged that prescribing decisions also needed to be supported by evidence on what constitutes unacceptable levels of resistance. Reliability (referring to the consistency in the laboratory data production process) was seen as important to assess trends over time and for benchmarking of resistance rates. The geographic, demographic and socioeconomic representativeness of the populations served by the laboratories where samples are generated was also seen to be important in order to be able to produce generalisable results. Our results indicate that for all three characteristics, the systems evaluated were underperforming.
There was wide variation in the nature and number of attributes examined in studies. Only two out of 43 reported all of the CDC attributes in their evaluations [45,52], with the majority focusing on only one or two attributes. Two characteristics, ease of use and awareness of the system, were associated with both greater acceptability and completeness (percentage of cases of condition reported). None of the other system characteristics were linked to benefits across more than one attribute. However, an examination of the attributes identifies other areas where the system’s performance on one attribute could influence others, even where this was not explicitly tested in the study. For example, Jansson et al. [42] observed that the timeliness of the Swedish statutory surveillance system was influenced by whether the reporting system was computerised or not, with computerised reporting resulting in shorter delays. The authors also point out in their discussion that computerised reporting would also be easier for clinicians to report cases. Hence, even though the authors did not examine simplicity as part of their evaluation, it is likely that their observations were relevant to this attribute.
There are two main reasons for caution when generalising the results of this review. Firstly, most evaluations (22/44) were of TB surveillance systems. This condition is at the forefront of the AMR challenge, as it is estimated that in 2017 there were 558,000 new cases of TB worldwide that were resistant to Rifampicin (the most effective first-line antibiotic), and among those, 82% had multidrug-resistant TB [58]. However, it is also a condition that is caused by a single pathogen, whereas the majority of infections (such as those affecting the upper and lower respiratory tracts, the urinary tract, wound, and bloodstream infections) are caused by a range of pathogens, and hence empirical treatment decisions require both the knowledge of the likely organisms and their likely susceptibilities to antibiotics [59].
Secondly, eight studies in the review were published between 1990 and 1999, and a further 14 were published between 2000 and 2009. Older evaluations may be of less relevance to certain aspects of current systems due to technological changes that have been implemented to the design and operation of surveillance systems which would make any older evaluations out of date and hence of limited relevance. Taking the UK as an example, one of the four British studies looked at the timeliness of reporting of Salmonella infections and the acceptability of the follow-up process [55]. The system, which used electronic reporting in 2010 in London and South East England, may not be relevant anymore given that the CoSurv software for recording laboratory isolates and case notifications that was examined in that study has since been replaced in England by the Second Generation Surveillance System (SGSS) Communicable Disease Module CDR [60]. The generalisability of a second study was also limited as the evaluation [25] was conducted between 1991–1993. The third study by Teo et al. [48], which investigated the Enhanced Tuberculosis Surveillance Scheme across England, Wales and Northern Ireland using a prospective rather than retrospective study design, was less prone to biases that may result from missing or erroneous entries. However, similar to the previous two, the age of the study (conducted between 2003 and 2005) limits its relevance to the current surveillance system in the UK. The current COVID-19 pandemic has demonstrated the impact that severe stresses can have on health systems. While laboratories remained largely functional during the pandemic, the surveillance reporting attached to these laboratories was deescalated due to capacity and other constraints for syndromic surveillance in particular [61].
There are also marked differences in the context of healthcare systems in the 18 countries where the surveillance systems were based, which would reduce the direct applicability of some of the findings to other settings. For example, the study by Olowokure et al. [30] to examine whether the introduction of the Haemophilus influenza type b conjugate vaccine in the UK in 1992 was associated with decreased effectiveness of the routine surveillance system, was confined to children under five living in one specific part of the UK (West Midlands).

Strengths and Limitations of the Research

The main strength of this systematic review is that it analysed study findings at the level of the attributes of different systems derived from the most widely recognised guidelines for establishing the utility of the outputs of surveillance systems [3]. This had the advantage of permitting a more homogenous synthesis of findings across diverse evaluations in a manner that has not been reported previously and which enables the identification of the attributes that need to be more consistently assessed, reported and implemented.
A limitation of the study relates to the restriction of the database searches to studies published in English, so some relevant studies may have potentially been missed.

5. Conclusions and Recommendations

We conclude that most of the health surveillance systems relevant to AMR that we reviewed were under-performing in the main attributes that relate to effectiveness and acceptability. Given that all desirable attributes cannot be maximised simultaneously, policy-makers need to decide which are the priority features that they seek to include in health surveillance and monitoring systems. Ease of use and users’ awareness of surveillance systems have been shown to be linked to high levels of acceptability and higher levels of completeness of data collection and could be targeted as priority areas for improvement of existing systems.
In addition, we recommend consistent use of definitions of attributes of surveillance systems. This would ensure a more consistent approach to evaluations and facilitate the identification of the attributes that have the greatest impact on the utility of the data produced by these systems, and hence should be prioritised for monitoring, as well as those that are highly correlated with others, and could be given lower priority in evaluations.

Author Contributions

Conceptualisation, M.A.-H., R.E.G., E.E., M.P., N.B. and N.M.; methodology, M.A.-H., R.E.G., E.E., M.P., N.B. and N.M.; data collection, M.A.-H. and R.E.G.; writing—original draft preparation, M.A.-H.; writing—review and editing, R.E.G., E.E., M.P., N.B. and N.M. All authors have read and agreed to the published version of the manuscript.

Funding

This study was funded by the National Institute for Health Research (NIHR) Policy Research Programme (Project No: 102/0001). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.

Data Availability Statement

Data is contained within the article.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A. Definitions of Attributes Identified as Important for Determining the Usefulness of a Surveillance System

Simplicity: the structure and ease of operation of the system means a system is as simple as possible while still meeting its objectives.
Flexibility: can adapt to changing information needs or operating conditions with minimal additional time, personnel, or allocated funds.
Data Quality: the completeness and validity of the data recorded in the public health surveillance system.
Acceptability: the willingness of persons and organisations to participate in the surveillance system.
Sensitivity: at two different levels: case reporting, where it refers to the proportion of cases of a condition detected by the system, and the ability to detect outbreaks, including the ability to monitor changes in the number of cases over time.
PPV: the proportion of reported cases that have a health-related event under surveillance.
Representativeness: the extent to which the system accurately describes the occurrence of a health-related event over time and its distribution in the population by place and person.
Timeliness: the speed between steps, in particular the interval between the onset of the health-related event and its reporting to the public health agency responsible for initiating control and prevention actions.
Stability: the reliability (ability to collect, manage and provide information without failure) and availability of the system (the ability to be operational when needed).

Appendix B. Search Terms

GENERIC SEARCH TERMS: Antimicrobial resistance OR Drug resistance OR Escherichia coli OR Bacteremia OR Sepsis OR Enterobacteriaceae OR Klebsiella pneumoniae OR Neisseria gonorrhoeae OR Acinetobacter OR Campylobacter OR beta-Lactamases OR Cephalosporin resistance OR Vancomycin resistance OR Enterococcus OR Pseudomonas OR Carbapenem resistance OR Salmonella OR Fluoroquinolone resistance OR Tuberculosis OR TB OR Mycobacterium OR Shigella OR Methicillin-resistant Staphylococcus aureus OR MRSA OR Streptococcus pneumonia OR Clarithromycin resistance OR Helicobacter pylori OR Haemophilus influenzae OR Communicable Diseases OR Infectious disease
AND
Surveillance OR Reporting OR Monitoring OR Notification OR Warning OR System OR Network
Web of Science Search Terms
“antimicrobial resistance” OR “drug resistance” OR “Escherichia coli” OR Bacteremia OR Sepsis OR Enterobacteriaceae OR “Klebsiella pneumoniae” OR “Neisseria gonorrhoeae” OR Acinetobacter OR Campylobacter OR beta-Lactamases OR “cephalosporin resistance” OR “vancomycin resistance” OR Enterococcus OR Pseudomonas OR “carbapenem resistance” OR Salmonella OR “fluoroquinolone resistance” OR Tuberculosis OR TB OR Mycobacterium OR Shigella OR “methicillin-resistant Staphylococcus aureus” OR MRSA OR “Streptococcus pneumonia” OR “clarithromycin resistance” OR “helicobacter pylori” OR “Haemophilus influenzae” OR “communicable diseases” OR “infectious disease”
AND
Surveillance OR Reporting OR Monitoring OR Notification OR Warning OR System OR Network
Open Grey
(“antimicrobial resistance” OR “drug resistance” OR “Escherichia coli” OR Bacteremia OR Sepsis OR Enterobacteriaceae OR “Klebsiella pneumoniae” OR “Neisseria gonorrhoeae” OR Acinetobacter OR Campylobacter OR beta-Lactamases OR “cephalosporin resistance” OR “vancomycin resistance” OR Enterococcus OR Pseudomonas OR “carbapenem resistance” OR Salmonella OR “fluoroquinolone resistance” OR Tuberculosis OR TB OR Mycobacterium OR Shigella OR “methicillin-resistant Staphylococcus aureus” OR MRSA OR “Streptococcus pneumonia” OR “clarithromycin resistance” OR “Helicobacter pylori” OR “Haemophilus influenzae” OR “communicable diseases” OR “infectious disease”) AND (Surveillance OR Reporting OR Monitoring OR Notification OR Warning OR System OR Network)
Scopus
(antimicrobial resistance) OR (drug resistance) OR (Escherichia coli) OR Bacteremia OR Sepsis OR Enterobacteriaceae OR (Klebsiella pneumonia) OR (Neisseria gonorrhoeae) OR Acinetobacter OR Campylobacter OR beta-Lactamases OR (cephalosporin resistance)
OR
(vancomycin resistance) OR Enterococcus OR Pseudomonas OR (carbapenem resistance) OR Salmonella OR (fluoroquinolone resistance) OR Tuberculosis OR TB OR Mycobacterium OR Shigella OR (methicillin-resistant Staphylococcus aureus) OR MRSA
OR
(Streptococcus pneumonia) OR (clarithromycin resistance) OR (Helicobacter pylori) OR (Haemophilus influenzae) OR (communicable diseases) OR (infectious disease)
AND
Surveillance OR Reporting OR Monitoring OR Notification OR Warning OR System OR Network

Appendix C. Attributes Used in Evaluations

AuthorAcceptabilityCompleteness (Proportion of Cases)Completeness (Variables Collected for Each Case)ConcordanceFlexibilityPositive Predictive Value (PPV)RepresentativenessSimplicitySpecificityStabilityTimelinessUsefulness
Heunis
Reijn
Devine
Gimenez-Duran
Auld
Podewils
Trei
Lo
Nguyen
Cojocaru
San Gabriel
Lirio
Santos
Takahashi
Grills
Khue
Guerrin-Tran
da Silva
Mancuso
Teo
Saeed
Alkhalawi
Jansson
Marks
Jansson
Severi
Birkhead
Dominguez
Kirk
Altmann
Kang
Migliori
Trepka
Nicolay
Curtis
Driver
Tanihara
Samaan
Stenhem
Van Leth
Jones
Olowokure
Sprinson

Appendix D. Attributes Examined by Health Condition or Microorganism

Condition/MicroorganismAcceptabilityCompleteness (Proportion of Cases)Completeness (Variables Collected for Each Case)ConcordanceFlexibilityPositive Predictive Value (PPV)RepresentativenessSimplicitySpecificityStabilityTimelinessUsefulness
TB
Salmonella/salmonellosis
Infections with penicillin-resistant pneumococci
MRSA
Neisseria gonorrhoeae/Gonoccoal infections
Shiga-toxin producing or enter-haemorrhagic Escherichia Coli
Shigellosis
Campylobacter
Haemophilus influenzae

References

  1. World Health Organisation. Antimicrobial Resistance. Available online: https://www.who.int/en/news-room/fact-sheets/detail/antimicrobial-resistance (accessed on 1 April 2021).
  2. World Health Organisation. WHO Global Strategy for Containment of Antimicrobial Resistance; WHO: Geneva, Switzerland, 2001. [Google Scholar]
  3. Centers for Disease Control and Prevention. Updated Guidelines for Evaluating Public Health Surveillance Systems: Recommendations from the Guidelines Working Group. MMWR 2001, 50, 1–35. [Google Scholar]
  4. World Health Organisation. Global Priority List of Antibiotic-Resistant Bacteria to Guide Research, Discovery, and Development of New Antibiotics; WHO: Geneva, Switzerland, 2017. [Google Scholar]
  5. Centers for Disease Control and Prevention. Biggest Threats; Centers for Disease Control and Prevention: Atlanta, GA, USA, 2017. [Google Scholar]
  6. Cormican, M.; Hopkins, S.; Jarlier, V.; Reilly, J.; Simonsen, G.S.; Strauss, R.; Vandenberg, O.; Zabicka, D.; Zarb, P.; Catchpole, M.; et al. ECDC, EFSA and EMA Joint Scientific Opinion on a list of outcome indicators as regards surveillance of antimicrobial resistance and antimicrobial consumption in humans and food-producing animals. EFSA J. 2017, 15, 5017. [Google Scholar]
  7. EFSA. Scientific Report of EFSA. Technical specification on the harmonised monitoring and reporting of antimicrobial resistance in Salmonella, Campylobacter and indicator Escherichia coli and Enterococcus spp. Bacteria transmitted through food. EFSA J. 2012, 10, 2742. [Google Scholar]
  8. Department of Health and Department for Environment and Rural Affairs, UK Five Year Antimicrobial Resistance Strategy 2013 to 2018. 2013. Available online: https://www.gov.uk/government/publications/uk-5-year-antimicrobial-resistance-strategy-2013-to-2018 (accessed on 1 April 2021).
  9. Critical Appraisal Skills Programme. CASP Checklist: 10 Questions to Help You Make Sense of A Qualitative Research; Critical Appraisal Skills Programme: Oxford, UK, 2018. [Google Scholar]
  10. Evans, N.; Lasen, M.; Tsey, K. Effective Public Health Practice Project (EPHPP) Quality Assessment Tool for Quantitative Studies, in A Systematic Review of Rural Development Research; Springer: Berlin/Heidelberg, Germany, 2015. [Google Scholar]
  11. Popay, J.; Petticrew, H.R.A.S.M.; Arai, L.; Rodgers, M.; Britten, N.; Roen, K.; Duffy, S. Guidance on the Conduct of Narrative Synthesis in Systematic Reviews. A Product from the ESRC Methods Programme. Version 1. ESRC. 2006. Available online: https://www.researchgate.net/publication/233866356_Guidance_on_the_conduct_of_narrative_synthesis_in_systematic_reviews_A_product_from_the_ESRC_Methods_Programme (accessed on 1 April 2021).
  12. Nguyen, D.B.; See, I.; Gualandi, N.; Shugart, A.; Lines, C.; Bamberg, W.; Dumyati, G.; Harrison, L.H.; Lesher, L.; Nadle, J.; et al. Completeness of Methicillin-Resistant Staphylococcus aureus Bloodstream Infection Reporting from Outpatient Hemodialysis Facilities to the National Healthcare Safety Network, 2013. Infect. Control. Hosp. Epidemiol. 2015, 37, 205. [Google Scholar] [CrossRef] [PubMed]
  13. Saeed, K.; Bano, R.; Asghar, R. Evaluation of national tuberculosis surveillance system in Afghanistan. East. Mediterr. Health J. 2013, 19, 200–207. [Google Scholar] [CrossRef] [PubMed]
  14. Mancuso, J.D.; Tobler, S.K.; Eick, A.A.; Olsen, C.H. An evaluation of the completeness and accuracy of active tuberculosis reporting in the United States military. Int. J. Tuberc. Lung Dis. 2010, 14, 1310–1315. [Google Scholar] [PubMed]
  15. Silva, G.D.M.D.; Bartholomay, P.; Cruz, O.G.; Garcia, L.P. Evaluation of data quality, timeliness and acceptability of the tuberculosis surveillance system in Brazil’s micro-regions. Cienc. Saude Coletiva 2017, 22, 3307–3319. [Google Scholar] [CrossRef] [PubMed]
  16. Samaan, G.; Roche, P.W.; E Greig, J.; Tapsall, J.W.; Programme, A.G.S. Evaluation of the Australian Gonococcal Surveillance Programme. Commun. Dis. Intell. Q. Rep. 2005, 29, 143–149. [Google Scholar]
  17. Auld, S.C.; Kim, L.; Webb, E.K.; Podewils, L.J.; Uys, M. Completeness and concordance of TB and HIV surveillance systems for TB-HIV co-infected patients in South Africa. Int. J. Tuberc. Lung Dis. 2013, 17, 186–191. [Google Scholar] [CrossRef]
  18. Heunis, C.; Wouters, E.; Kigozi, G.; Engelbrecht, M.; Tsibolane, Y.; Van Der Merwe, S.; Motlhanke, S. Accuracy of Tuberculosis Routine Data and Nurses’ Views of the TB-HIV Information System in the Free State, South Africa. J. Assoc. Nurses AIDS Care 2011, 22, 67–73. [Google Scholar] [CrossRef]
  19. Effective Public Health Practice Project. Quality Assessment Tool for Quantitative Studies; Effective Public Health Practice Project: Hamilton, ON, USA, 1998. [Google Scholar]
  20. Alkhalawi, M.J.; McNabb, S.J.; Assiri, A.M.; Memish, Z.A. Evaluation of tuberculosis public health surveillance, Al-Madinah province, Kingdom of Saudi Arabia, 2012. J. Epidemiol. Glob. Health 2015, 6, 37. [Google Scholar] [CrossRef]
  21. Kang, H.-Y.; Yoo, H.; Park, W.; Go, U.; Jeong, E.; Jung, K.-S.; Son, H. Tuberculosis Notification Completeness and Timeliness in the Republic of Korea during 2012–2014. Osong Public Health Res. Perspect. 2016, 7, 320–326. [Google Scholar] [CrossRef] [PubMed]
  22. Migliori, G.B.; Spanevello, A.; Ballardini, L.; Neri, M.; Gambarini, C.; Moro, M.L.; Trnka, L.; Raviglione, M. Validation of the surveillance system for new cases of tuberculosis in a province of Northern Italy. Eur. Respir. J. 1995, 8, 1252–1258. [Google Scholar] [CrossRef] [PubMed]
  23. Trepka, M.J.; Beyer, T.O.; Proctor, M.E.; Davis, J.P. An evaluation of the completeness of tuberculosis case reporting using hospital billing and laboratory data; Wisconsin, 1995. Ann. Epidemiol. 1999, 9, 419–423. [Google Scholar] [CrossRef]
  24. Driver, C.R.; Braden, C.R.; Nieves, R.L.; Navarro, A.M.; Rullan, J.V.; Valway, S.E.; McCray, E. Completeness of tuberculosis case reporting, San Juan and Caguas Regions, Puerto Rico, 1992. Public Health Rep. 1996, 111, 157–161. [Google Scholar] [PubMed]
  25. Devine, M.J.; Aston, R. Assessing the completeness of tuberculosis notification in a health district. Commun. Dis. Rep. CDR Rev. 1995, 5, 137–140. [Google Scholar]
  26. Podewils, L.J.; Bantubani, N.; Bristow, C.; E Bronner, L.; Peters, A.; Pym, A.; Mametja, L.D. Completeness and Reliability of the Republic of South Africa National Tuberculosis (TB) Surveillance System. BMC Public Health 2015, 15, 765. [Google Scholar] [CrossRef] [PubMed]
  27. Trei, J.S.; Carvelli, K.M. Completeness and timeliness of Chlamydia trachomatis and Neisseria gonorrhoeae genital infection reporting in the U.S. Air Force. Mil. Med. 2008, 173, 313–317. [Google Scholar] [CrossRef]
  28. San Gabriel, P.; Saiman, L.; Kaye, K.; Silin, M.; Onorato, I.; Schulte, J. Completeness of pediatric TB reporting in New York City. Public Health Rep. 2003, 118, 144–153. [Google Scholar] [CrossRef]
  29. Guerrin-Tran, E.; Thiolet, J.-M.; Rousseau, C.; Henry, S.; Poirier, C.; Che, D.; Vinas, J.-M.; Jarlier, V.; Robert, J. An evaluation of data quality in a network for surveillance of Mycobacterium tuberculosis resistance to antituberculosis drugs in Ile-de-France region-2001–2002. Eur. J. Epidemiol. 2006, 21, 783–785. [Google Scholar] [CrossRef]
  30. Olowokure, B.; Hawker, J.; Blair, I.; Spencer, N. Decrease in effectiveness of routine surveillance of Haemophilus influenzae disease after introduction of conjugate vaccine: Comparison of routine reporting with active surveillance system. BMJ 2000, 321, 731–732. [Google Scholar] [CrossRef]
  31. Lo, H.-Y.; Yang, S.-L.; Chou, P.; Chuang, J.-H.; Chiang, C.-Y. Completeness and timeliness of tuberculosis notification in Taiwan. BMC Public Health 2011, 11, 915. [Google Scholar] [CrossRef] [PubMed]
  32. Lírio, M.; Dos Santos, N.P.; Passos, L.A.R.; Kritski, A.; Galvão-Castro, B.; Grassi, M.F.R. Completeness of tuberculosis reporting forms for disease control in individuals with HIV/AIDS in priority cities of Bahia state. Ciência Saúde Coletiva 2015, 20, 1143–1148. [Google Scholar] [CrossRef]
  33. Santos, N.P.D.; Lírio, M.; Passos, L.A.R.; Dias, J.P.; Kritski, A.L.; Galvão-Castro, B.; Grassi, M.F.R. Completeness of tuberculosis reporting forms in five Brazilian capitals with a high incidence of the disease. J. Bras. Pneumol. 2013, 39, 221–225. [Google Scholar] [CrossRef] [PubMed]
  34. Nicolay, N.; Garvey, P.; DeLappe, N.; Cormican, M.; McKeown, P. Completeness and timeliness of Salmonella notifications in Ireland in 2008: A cross sectional study. BMC Public Health 2010, 10, 568. [Google Scholar] [CrossRef]
  35. Stenhem, M.; Örtqvist, Å.; Ringberg, H.; Larsson, L.; Olsson-Liljequist, B.; Haeggman, S.; Kalin, M.; Ekdahl, K.; the Swedish Study Group on MRSA Epidemiology. Validity of routine surveillance data: A case study on Swedish notifications of methicillin-resistant Staphylococcus aureus. Eurosurveillance 2009, 14, 19281. [Google Scholar] [CrossRef]
  36. E Sprinson, J.; Lawton, E.S.; Porco, T.C.; Flood, J.M.; Westenhouse, J.L. Assessing the validity of tuberculosis surveillance data in California. BMC Public Health 2006, 6, 217. [Google Scholar] [CrossRef] [PubMed]
  37. Khuê, P.M.; Mallet, A.; Veziris, N.; Jarlier, V.; Robert, J. Evaluation of data quality in a laboratory-based surveillance of M. tuberculosis drug resistance and impact on the prevalence of resistance: France, 2004. Epidemiol. Infect. 2007, 136, 1172–1178. [Google Scholar] [CrossRef] [PubMed]
  38. Tanihara, S.; Suzuki, S. Estimation of the incidence of MRSA patients: Evaluation of a surveillance system using health insurance claim data. Epidemiol. Infect. 2016, 144, 2260–2267. [Google Scholar] [CrossRef] [PubMed]
  39. Bai, J.; Marks, G.B.; Stewart, G.J.; Simpson, S.E.; Sullivan, E.A. Specificity of notification for tuberculosis among screened refugees in NSW. Aust. N. Z. J. Public Health 1999, 23, 410–413. [Google Scholar] [CrossRef] [PubMed]
  40. Reijn, E.; Swaan, C.M.; Kretzschmar, M.E.; Van Steenbergen, J.E. Analysis of timeliness of infectious disease reporting in the Netherlands. BMC Public Health 2011, 11, 409. [Google Scholar] [CrossRef]
  41. Takahashi, T. Evaluation of a public health Salmonella surveillance system in King County, Washington. Am. J. Infect. Control. 2004, 32, 7–11. [Google Scholar] [CrossRef] [PubMed]
  42. Jansson, A.; Arneborn, M.; Skärlund, K.; Ekdahl, K. Timeliness of case reporting in the Swedish statutory surveillance of communicable diseases 1998–2002. Scand. J. Infect. Dis. 2004, 36, 865–872. [Google Scholar] [CrossRef] [PubMed]
  43. Birkhead, G.; Chorba, T.L.; Root, S.; Klaucke, D.N.; Gibbs, N.J. Timeliness of national reporting of communicable diseases: The experience of the National Electronic Telecommunications System for Surveillance. Am. J. Public Health 1991, 81, 1313–1315. [Google Scholar] [CrossRef] [PubMed]
  44. Domínguez, A.; Coll, J.J.; Fuentes, M.; Salleras, L. Timeliness of notification in infectious disease cases. Public Health Rep. 1992, 107, 474–476. [Google Scholar] [PubMed]
  45. Altmann, M.; Wadl, M.; Altmann, D.; Benzler, J.; Eckmanns, T.; Krause, G.; Spode, A.; Der Heiden, M.A. Timeliness of Surveillance during Outbreak of Shiga Toxin–producingEscherichia coliInfection, Germany, 2011. Emerg. Infect. Dis. 2011, 17, 1906–1909. [Google Scholar] [CrossRef]
  46. Curtis, A.B.; McCray, E.; McKenna, M.; Onorato, I.M. Completeness and timeliness of tuberculosis case reporting. A multistate study. Am. J. Prev. Med. 2001, 20, 108–112. [Google Scholar] [CrossRef]
  47. Jones, G.; Le Hello, S.; Silva, N.J.-D.; Vaillant, V.; De Valk, H.; Weill, F.X.; Le Strat, Y. The French human Salmonella surveillance system: Evaluation of timeliness of laboratory reporting and factors associated with delays, 2007 to 2011. Eurosurveillance 2014, 19, 20664. [Google Scholar] [CrossRef]
  48. Teo, S.S.S.; Alfaham, M.; Evans, M.R.; Watson, J.M.; Riordan, A.; Sonnenberg, P.; Clark, J.; Hayward, A.; Sharland, M.; Moore-Gillon, J.; et al. An evaluation of the completeness of reporting of childhood tuberculosis. Eur. Respir. J. 2009, 34, 176–179. [Google Scholar] [CrossRef] [PubMed]
  49. Cojocaru, C.; A Van Hest, N.; Mihaescu, T.; Davies, P.D. Completeness of notification of adult tuberculosis in Iasi County, Romania: A capture-recapture analysis. Int. J. Tuberc. Lung Dis. 2009, 13, 1094–1099. [Google Scholar] [PubMed]
  50. Van Leth, F.; Evenblij, K.; Wit, F.; Kiers, A.; Sprenger, H.; Verhagen, M.; Hillebregt, M.; Kalisvaart, N.; Schimmel, H.; Verbon, A. TB-HIV co-infection in the Netherlands: Estimating prevalence and under-reporting in national registration databases using a capture-recapture analysis. J. Epidemiol. Community Health 2016, 70, 556–560. [Google Scholar] [CrossRef]
  51. Hook, E.B.; Regal, R.R. Capture-Recapture Methods in Epidemiology: Methods and Limitations. Epidemiol. Rev. 1995, 17, 243–264. [Google Scholar] [CrossRef]
  52. Jansson, A.; Arneborn, M.; Ekdahl, K. Sensitivity of the Swedish statutory surveillance system for communicable diseases 1998–2002, assessed by the capture–recapture method. Epidemiol. Infect. 2005, 133, 401–407. [Google Scholar] [CrossRef]
  53. Gimenez-Duran, J.; Galmes-Truyols, A.; Luque-Fernández, M.Á.; Bonilla-Vargas, L.A.; Bosch-Isabel, C.; Nicolau-Riutort, A.; De Mateo-Ontañón, S. Assessment of tuberculosis surveillance by capture–recapture in the Balearic Islands, Spain, 2005–2007. Enferm. Infecc. Microbiol. Clín. 2015, 33, 613–616. [Google Scholar] [CrossRef] [PubMed]
  54. Grills, N.J.; Rowe, S.L.; Gregory, J.E.; Lester, R.A.; Fielding, J.E. Evaluation of Campylobacter infection surveillance in Victoria. Commun. Dis. Intell. Q. Rep. 2010, 34, 110–115. [Google Scholar] [PubMed]
  55. Severi, E.; Dabrera, G.; Boxall, N.; Harvey-Vince, L.; Booth, L.; Balasegaram, S. Timeliness of Electronic Reporting and Acceptability of Public Health Follow-Up of Routine Nonparatyphoidal and Nontyphoidal Salmonella Infections, London and South East England, 2010 to 2011. J. Food Prot. 2014, 77, 94–99. [Google Scholar] [CrossRef] [PubMed]
  56. Kirk, M.D.; Dalton, C.B.; Beers, M.Y.; Cameron, A.S.; Murray, C. Timeliness of Salmonella notifications in South Australia. Aust. N. Z. J. Public Health 1999, 23, 198–200. [Google Scholar] [CrossRef] [PubMed]
  57. Lewis, D. Antimicrobial resistance surveillance: Methods will depend on objectives. J. Antimicrob. Chemother. 2002, 49, 3–5. [Google Scholar] [CrossRef] [PubMed]
  58. World Health Organisation. Global Tuberculosis Report 2018; World Health Organisation: Geneva, Switzerland, 2018. [Google Scholar]
  59. Johnson, A.P. Surveillance of antibiotic resistance. Philos. Trans. R. Soc. B Biol. Sci. 2015, 370, 20140080. [Google Scholar] [CrossRef]
  60. Public Health England. Health Protection Report: Infection Report; Public Health England: London, UK, 2017.
  61. Elliot, A.J.; Harcourt, S.E.; Hughes, H.E.; Loveridge, P.; Morbey, R.A.; Smith, S.; Soriano, A.; Bains, A.; Smith, G.E.; Edeghere, O.; et al. The COVID-19 pandemic: A new challenge for syndromic surveillance. Epidemiol. Infect. 2020, 148, e122. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) diagram of studies included at each stage of the screening process.
Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) diagram of studies included at each stage of the screening process.
Antibiotics 10 00431 g001
Table 1. Number of studies included in the review by study design.
Table 1. Number of studies included in the review by study design.
Study Design and Analysis ApproachesNumber of Studies
Observational: Retrospective analysis of routinely collected data30
Mixed methods: Retrospective analysis of routinely collected data combined with semi-structured interviews5
Observational: Retrospective analysis of routinely collected data combined with capture-recapture statistical methods5
Observational: Retrospective analysis of routinely collected data combined with a questionnaire survey2
Observational: Prospective analysis of routinely collected data1
Total43
Table 2. Country where evaluation took place.
Table 2. Country where evaluation took place.
Country of StudyNumber of Studies
Afghanistan1
Australia4
Brazil3
France3
Germany1
Ireland1
Italy1
Japan1
Netherlands2
Republic of Korea1
Romania1
Saudi Arabia1
South Africa3
Spain2
Sweden3
Taiwan1
UK4
USA10
Total43
Table 3. Attributes identified and their definitions.
Table 3. Attributes identified and their definitions.
Attribute Name.Description of Attribute as Used by Evaluators
AcceptabilityAwareness of, and adherence to, the surveillance system protocol by staff.
Completeness (also described as sensitivity, coverage, validity)Either: The proportion of cases reported by the system (established by looking at other systems or by estimating using the capture-recapture method); also known as sensitivity or coverage.
Or: Extent (or proportion) of the fields that are completed in the forms. In some studies, critical categories to be completed were identified; also known as validity.
Concordance (also known as reliability or consistency)The level of agreement between the different systems on the data collected for each case.
FlexibilityThe degree to which a system can adapt to changing information needs or operating conditions with little additional time, personnel, or allocated funds (CDC Definition) [3].
Positive Predictive Value (also known as Predictive Value Positive) PPVThe proportion of reported cases that actually have the health-related event under surveillance (CDC Definition) [3].
RepresentativenessGeographic or population coverage of system.
SimplicityFeatures that make a system easy to use (including the method of notification).
SpecificityCorrectly identifying patients who are free of the condition.
StabilityAbility to collect, manage, and provide data properly without failure and ability to be operational when needed [3].
TimelinessPeriod between different time points in the notification process.
UsefulnessAbility of a system to provide information that can be (or is) acted on; also known as efficacy.
Table 4. Antimicrobial resistance (AMR)-related health conditions and microorganisms monitored in surveillance systems. TB: tuberculosis; MRSA: methicillin resistance Staphylococcus aureus.
Table 4. Antimicrobial resistance (AMR)-related health conditions and microorganisms monitored in surveillance systems. TB: tuberculosis; MRSA: methicillin resistance Staphylococcus aureus.
Health Condition/MicroorganismNumber of Included Evaluations
TB (Pulmonary or extra-pulmonary)22
Salmonella/salmonellosis8
Infections with penicillin-resistant pneumococci2
MRSA3
Neisseria gonorrhoeae/Gonococcal infections2
Shiga-toxin producing or enterhaemorrhagic Escherichia coli2
Shigellosis1
TB in HIV patients4
Campylobacter1
Haemophilus influenzae1
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Back to TopTop