The process of antimicrobial resistance (AMR), whereby microbes evolve over time to become less susceptible to medicines, has reduced the ability of healthcare providers to treat common infections. It is a naturally occurring process that takes place through genetic mutations. The main drivers for AMR include the misuse and overuse of antimicrobials; lack of access to clean water, sanitation and hygiene; poor infection and disease prevention and control; poor access to quality, affordable medicines, vaccines and diagnostics; and lack of enforcement of legislation [1
]. In the context of combating AMR, the World Health Organisation (WHO) views surveillance systems as being essential for providing information on the magnitude and trends in AMR and for monitoring the effects of interventions [2
The most widely recognised guidelines for establishing the utility of the outputs of a surveillance system are those developed by the US Centers for Disease Control and Prevention (CDC) [3
]. These describe the usefulness of a system in terms of its contribution to the prevention and control of adverse health events, including an understanding of the implications of those events, based on consideration of nine attributes: Simplicity, Flexibility, Data Quality, Acceptability, Sensitivity, Positive Predictive Value (PPV), Representativeness, Timeliness, and Stability (See Appendix A
for definitions). The overall usefulness of a surveillance system is considered to be influenced (to varying degrees depending on the aim of the system) by these attributes [3
]. For example, a public health surveillance system that is simple, flexible, acceptable, and stable is more likely to be useful for public health action. However, the CDC acknowledges that there is no perfect system and that focusing resources to improve one attribute might have an adverse effect on another; for example, as sensitivity increases, the PPV might decrease, and efforts to increase sensitivity and PPV could result in a more complex surveillance system with reduced acceptability and timeliness [3
]. There is uncertainty about what constitute the most effective characteristics of a system.
A rapid review of the literature identified no systematic reviews of evaluations of the ability of surveillance systems to provide information that can be used by health care professionals to combat AMR. Such a synthesis could contribute to the evidence base when deciding whether to invest or continue investing in these systems in the future. Hence, the aim of this systematic review was to synthesise the evidence from evaluations of the quality and utility of information produced by surveillance systems that monitor organisms and conditions important in the AMR context, with a focus on human rather than animal surveillance systems. The research questions were:
2. Materials and Methods
The study protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO), registration number: CRD42018085346.
In order to answer the first research question (the effectiveness of systems), the following study designs were eligible for inclusion:
Prospective observational studies (controlled and uncontrolled before and after studies).
Retrospective observational evaluations, including case-control studies, retrospective cohort studies, and audits. Data sources included primary data collected for research and secondary data (for example, health insurance claim data).
Interventions using an experimental design.
Evidence from qualitative research (such as interviews and focus groups) was considered eligible for answering the second research question on the acceptability of systems.
Evaluations of surveillance systems that monitored the following organisms were eligible for inclusion:
Bacteria whose antibiotic susceptibility status was recorded by the surveillance system.
Bacteria relevant to AMR. A list was collated from the key AMR threats that have been identified by the WHO [4
], the CDC [5
], European Centre for Disease Control (ECDC) [6
], European Food Safety Authority (EFSA) [7
], and the key drug-bug combinations identified by Public Health England in the UK AMR Strategy [8
The following types of evaluation study were excluded from the review:
Evaluations of public surveillance systems that monitor non-bacterial microorganisms (for example, viruses or fungi).
Evaluations of surveillance systems that monitor bacterial microorganisms that are not on any of the priority lists described in the inclusion criteria above.
Screening systems that are limited to a single or group of hospitals, and where the information is not shared outside the hospital system.
Studies published prior to 1988, when the first CDC guidelines for evaluating Public Health Surveillance systems were published.
Articles published in languages other than English.
The following databases were searched for relevant articles from 1 January 1988 until 1 June 2018: OVID Medline; EMBASE; Cochrane Central Register of Controlled Trials (CENTRAL); Global Health (OVID); Web of Science; Open Grey and Scopus.
The search terms used can be found in Appendix B
. They were adapted for databases in accordance with the repository’s interface and search options. All search strings were run in English, and all the records were exported to Endnote reference management software v 18.0.2 and Excel 2016. In addition to searching databases, we also performed reference searches of the identified.
The primary outcome for inclusion in the review, to answer the research question on effectiveness, were attributes of surveillance systems, as described by the authors. Aspects of acceptability, such as ease of use, reported in qualitative data, were the outcomes of interest for answering the second research question.
2.2. Quality Assessment of Studies
The Critical Appraisal Skills Programme (CASP) checklist [9
] was used for assessing the quality of reporting of the qualitative studies included in the review, and the Effective Public Health Practice Project (EPHPP) Quality Assessment Tool was used for those that included a quantitative component [10
Given the heterogeneity among study designs and contexts, a narrative review was conducted following guidelines for narrative synthesis [11
This review sought to examine the effectiveness of aspects of the performance of surveillance systems that are considered to be important for AMR monitoring. It found that most surveillance systems evaluated were underperforming in the key attributes related to both effectiveness and acceptability.
] described the overall objective of surveillance of AMR as the facilitation of control of AMR through informing the need to improve prescribing and infection control practices. The author identified three essential attributes for an AMR surveillance system for human health: timeliness, reliability, and representativeness. Timeliness was identified as important for AMR trends at the local level and to assist clinicians in the rational choice of antibiotic. However, the author acknowledged that prescribing decisions also needed to be supported by evidence on what constitutes unacceptable levels of resistance. Reliability (referring to the consistency in the laboratory data production process) was seen as important to assess trends over time and for benchmarking of resistance rates. The geographic, demographic and socioeconomic representativeness of the populations served by the laboratories where samples are generated was also seen to be important in order to be able to produce generalisable results. Our results indicate that for all three characteristics, the systems evaluated were underperforming.
There was wide variation in the nature and number of attributes examined in studies. Only two out of 43 reported all of the CDC attributes in their evaluations [45
], with the majority focusing on only one or two attributes. Two characteristics, ease of use and awareness of the system, were associated with both greater acceptability and completeness (percentage of cases of condition reported). None of the other system characteristics were linked to benefits across more than one attribute. However, an examination of the attributes identifies other areas where the system’s performance on one attribute could influence others, even where this was not explicitly tested in the study. For example, Jansson et al. [42
] observed that the timeliness of the Swedish statutory surveillance system was influenced by whether the reporting system was computerised or not, with computerised reporting resulting in shorter delays. The authors also point out in their discussion that computerised reporting would also be easier for clinicians to report cases. Hence, even though the authors did not examine simplicity as part of their evaluation, it is likely that their observations were relevant to this attribute.
There are two main reasons for caution when generalising the results of this review. Firstly, most evaluations (22/44) were of TB surveillance systems. This condition is at the forefront of the AMR challenge, as it is estimated that in 2017 there were 558,000 new cases of TB worldwide that were resistant to Rifampicin (the most effective first-line antibiotic), and among those, 82% had multidrug-resistant TB [58
]. However, it is also a condition that is caused by a single pathogen, whereas the majority of infections (such as those affecting the upper and lower respiratory tracts, the urinary tract, wound, and bloodstream infections) are caused by a range of pathogens, and hence empirical treatment decisions require both the knowledge of the likely organisms and their likely susceptibilities to antibiotics [59
Secondly, eight studies in the review were published between 1990 and 1999, and a further 14 were published between 2000 and 2009. Older evaluations may be of less relevance to certain aspects of current systems due to technological changes that have been implemented to the design and operation of surveillance systems which would make any older evaluations out of date and hence of limited relevance. Taking the UK as an example, one of the four British studies looked at the timeliness of reporting of Salmonella
infections and the acceptability of the follow-up process [55
]. The system, which used electronic reporting in 2010 in London and South East England, may not be relevant anymore given that the CoSurv software for recording laboratory isolates and case notifications that was examined in that study has since been replaced in England by the Second Generation Surveillance System (SGSS) Communicable Disease Module CDR [60
]. The generalisability of a second study was also limited as the evaluation [25
] was conducted between 1991–1993. The third study by Teo et al. [48
], which investigated the Enhanced Tuberculosis Surveillance Scheme across England, Wales and Northern Ireland using a prospective rather than retrospective study design, was less prone to biases that may result from missing or erroneous entries. However, similar to the previous two, the age of the study (conducted between 2003 and 2005) limits its relevance to the current surveillance system in the UK. The current COVID-19 pandemic has demonstrated the impact that severe stresses can have on health systems. While laboratories remained largely functional during the pandemic, the surveillance reporting attached to these laboratories was deescalated due to capacity and other constraints for syndromic surveillance in particular [61
There are also marked differences in the context of healthcare systems in the 18 countries where the surveillance systems were based, which would reduce the direct applicability of some of the findings to other settings. For example, the study by Olowokure et al. [30
] to examine whether the introduction of the Haemophilus
influenza type b conjugate vaccine in the UK in 1992 was associated with decreased effectiveness of the routine surveillance system, was confined to children under five living in one specific part of the UK (West Midlands).
Strengths and Limitations of the Research
The main strength of this systematic review is that it analysed study findings at the level of the attributes of different systems derived from the most widely recognised guidelines for establishing the utility of the outputs of surveillance systems [3
]. This had the advantage of permitting a more homogenous synthesis of findings across diverse evaluations in a manner that has not been reported previously and which enables the identification of the attributes that need to be more consistently assessed, reported and implemented.
A limitation of the study relates to the restriction of the database searches to studies published in English, so some relevant studies may have potentially been missed.
5. Conclusions and Recommendations
We conclude that most of the health surveillance systems relevant to AMR that we reviewed were under-performing in the main attributes that relate to effectiveness and acceptability. Given that all desirable attributes cannot be maximised simultaneously, policy-makers need to decide which are the priority features that they seek to include in health surveillance and monitoring systems. Ease of use and users’ awareness of surveillance systems have been shown to be linked to high levels of acceptability and higher levels of completeness of data collection and could be targeted as priority areas for improvement of existing systems.
In addition, we recommend consistent use of definitions of attributes of surveillance systems. This would ensure a more consistent approach to evaluations and facilitate the identification of the attributes that have the greatest impact on the utility of the data produced by these systems, and hence should be prioritised for monitoring, as well as those that are highly correlated with others, and could be given lower priority in evaluations.