Outcome scales—An observational study of the use of continuous clinical outcome assessment and therapy management in neurosurgery

Simple Summary

Question: Quality management and the assessment of patient outcome are gaining increasing importance in clinical medicine. Politicians, beneficiaries of medical treatment and the general public alike wish to have the best possible medical service at the lowest possible cost. We aimed to develop a system that would allow assessment of medical and surgical procedures by using a short, easy-to-use form documenting key outcomes on a daily basis during the normal clinical routine. Method: A standardised daily evaluation protocol was developed based on established clinical scales. To facilitate its use in daily routine, each item/ domain was scored on a 5-point scale and the form was structured on a standard A4 sheet of paper. Data were entered into a customised database integrated into the hospital’s existing administration IT-system. Results: The system allowed daily documentation of different aspects of patient care. It included original and modified clinical scales as well as information on patient satisfaction with treatment and general well-being. Data were processed digitally with standard software and used to analyse progress and outcome as well as adverse events occurring during treatment. Conclusion: The documentation system provides medical staff with a checklist for rounds and allows systematic and standardised evaluation of clinical progress, complications, neurological deficits and patient satisfaction throughout the treatment process.

Introduction

In 2008 the effective costs of health care in Switzerland were estimated at 10.7% of GNP compared with 3.5% in 1950 and 7.5% in 1975. This is equivalent to a cost of 7589 Swiss francs per inhabitant/year in 2008 and total costs of 58 453 million Swiss francs [1,2]. With an increasing number of sophisticated treatment options, the cost of medical treatment is likely to increase further. Tools to survey patient safety, clinical outcome, complications and adverse events are helpful for treatment evaluation and therefore need to be developed and used on a regular basis in daily clinical practice. In this way provision of medical care can be optimised.

Such evaluation tools should be sensitive enough to detect subtle changes in individual or collective well-being and reliable enough to allow multicentre comparability. Equally, however, they must be easy to use, so as to achieve a good acceptance level among users and allow consistent and systematic acquisition of data from all patients.

Ideally, assessment scales to be used in daily practice should make use of common communication and interaction pathways and exploit the information collected during routine clinical activities, such as consultations or visits, without the need for extra post-processing. It is suggested that the provision of such a documentation and evaluation tool will ultimately help to improve workflow and reduce errors in the management of neurosurgical patients [3].

The aim of this study was to develop an assessment tool for neurosurgical patients based on established clinical scales and suitable for use in daily clinical practice, and to describe its subsequent implementation within the neurosurgery department of a cantonal hospital. The reliability, validity and functionality of the tool in assessing the key outcome domains (pain, function and satisfaction with treatment) in a common intervention–lumbar microdiscectomy–was also investigated.

Material and methods

The assessment instrument

The instrument (fig. 1) was developed on the basis of established questionnaires, using simple adjectival scales typical of those used in other established outcome instruments. Some scales simply documented an event (e.g., imaging, complication) while others described the extent or severity of a symptom, clinical or mental status, or disability. All information fitted onto a DIN A4 documentation form (fig. 1). The check boxes were arranged in groups with an array of columns from left to right for each day. All check box groups were arranged on a 0–5 scale, with 0 indicating missing data. The same system of numbered codes was maintained throughout the form to facilitate completion and simplify the later process of digitalisation. Information was to be added on a daily basis, except for the Admission scale, which simply described the main reason for admission or consultation. The options for this item were chosen on the basis of the commonest diseases coded in the hospital’s database.

Individual scales used

Daily quality scale (DQS): Each patient was evaluated daily with respect to general condition (not only the main symptoms) compared with the last consultation, with response options of better, same, worse.

Pain outcome scale (POS): The pain level of the patient was evaluated using the following 5-point verbal rating scale: (1) maximum/unbearable pain; (2) pain that was severe and restricted/limited activities; (3) pain that was quite severe but did not restrict/limit activities, (4) slight pain and (5) no pain. Reports in the literature state that numerical rating scales or categorical scales with verbal descriptors are generally preferable to visual analogue scales, and that daily pain assessment is recommended [4,5].

Therapy score (TS): The quantity of pain medication taken was used to give an idea of the effectiveness of treatment. For simplicity’s sake we distinguished between standard analgesics such as opioids, paracetamol and NSAIDs. Any medication taken for seizures, oncological or antibiotic therapy, and hormone substitution was documented, as was radiotherapy and external stabilisation. Multiple answers were allowed.

Glasgow Coma Scale (GCS): The GCS is the most renowned score evaluating the clinical status of patients with central nervous system problems. Specific values for eye opening, speech and best motor response were documented in accordance with the initial description of these subdomains [6,7].

Mini Mental Test (MMS): All patients admitted with brain lesions were examined using the mini mental test [8], which was also repeated on discharge.

Functional Outcome Scale (FOS): To cover all neurosurgical patients a generic functional outcome scale was used based on WHO International classification of Impairment, Disability and Handicap (1980 ICIDH, WHO) and including a 6-point scale with response options of: (0) no function; (1) some function; (2) limited and disabling function; (3) limited but not disabling function; (4) slightly limited function; (5) full function.

Complication Scale (CS): Complications such as embolic or thrombotic events (to which neurosurgical patients are highly prone) [9] were documented. Infections were documented as either surgical and/or other infections (such as pneumonia, urinary infection, etc.) [10]. Newly acquired deficits (such as motor weakness, etc.) or bleeding in the postoperative phase were also recorded. The complication item was always completed for as long as it was relevant (e.g., ongoing antibiotic or anticoagulation therapy). Multiple answers were allowed.

Imaging (I): Imaging was documented on the day it was carried out; any suspected pathologies were documented, as were any proven occurrences or recurrences, classified as stable, progressive and recurrent.

Glasgow Outcome Scale (GOS): The GOS/Karnofsky outcome scale was based on that initially described by Jennet and Bond in 1975 [11,12,13] and modified for conformity to the 5-point scaling system.

Professional Outcome (POS): We used a scale with three work status options (working for >80%, 50–80%, <50% of the standard working week), disability pension and the need for retraining. This facilitated the processing of insurance claims.

Patient Satisfaction Scale (PSS): The patient’s overall satisfaction with the treatment was documented on a 5-point scale, ranging from very satisfied (5) to very dissatisfied (1).

Info Quality (IQ): To authenticate the data, the clinician filling in the chart signed the daily form with his unique code; this meant that the data fed into the database were traceable.

Patients

The patient group comprised all consecutive neurosurgical patients treated at the Neurosurgical Unit of Kantonsspital Aarau. Data recording was continuous, beginning in 2001, with a last and major modification of the appearance of the form and of data recording in 2005. All patients, including those who were hospitalised and those seen in outpatient clinics or emergency rooms, were included. The Dept. of Neurosurgery at the Kantonsspital Aarau administers neurosurgical treatment for the whole of the canton and surrounding areas. For example, in 2009 1373 patients underwent inpatient treatment and 4483 patients received outpatient care. Surgery was performed in 1544 cases. The patient collective included in the database comprised both men and women (almost equally represented) and all age groups (except paediatric patients), the main patient group being the 50–75-year-olds [14].

Data collection and processing procedures

Data were collected in paper form and completed daily by the physician in charge during hospitalisation, and also at preoperative and postoperative outpatient visits. The same outcome assessment system was used for all patients, regardless of their specific disease or gender. Integration of digitally stored images to simplify data collection for administration and scientific use was also possible. Data from the paper form were digitised using a standard Internet browser to a newly developed database hosted within the hospital IT system run by HINTAG (Health Information Technologies AG, Aarau, Switzerland), IT partner of Kantonsspital Aarau. All diseases (IDC-10) and surgeries (ICM-9) were automatically linked and coded into the hospital’s administration database. The database was based on Internet Information Server (IIS) and built by ASP (Active Server Pages). Data Management was provided by H-Webbuilder [15] using SQL-Database and Crystal Reports from Business Objects [16] as a reporting system. Access to the data was restricted by means of various user privileges.

Reliability and validity of the instruments used

The reliability and validity of the assessments for the key outcome domains (pain, function and satisfaction with treatment) in one common intervention, lumbar microdiscectomy, was investigated. Interrater reliability was examined by comparing the independent assessments of two staff members for 20 randomly-chosen patients (11 male / 9 female) during their hospital stay. The impact of reducing the response category options from the commonly used 11-point (0–10) visual analogue scale [5] to a 5-point adjectival scale was studied in a randomly-selected group of 12 patients (7 female / 5 male) who completed both scales.

Pilot investigation: the assessment of pain evolution in patients with lumbar disc herniation

A pilot outcome study was performed in all patients undergoing surgery for lumbar monosegmental and unilateral disc herniation (LDH). The only exclusion criterion was missing data at one or more time points. A retrospective analysis of the prospectively-collected data from January 2007 to June 2009 was carried out. The outcomes under investigation were pain and patient satisfaction.

Statistical analysis

Statistical comparisons of the pain levels at different time points (pre-op, 1st day post-op and discharge) in the pilot study were made using one way repeated measures ANOVA with posthoc Tukey tests to identify the significance of any paired differences observed. The relationship between scores on different scales (concurrent validity) was assessed using Spearman rank correlation coefficients. Interrater reliability was assessed using Kappa coefficients. Statistical analyses were carried out using SPSS V.16 (SPSS Inc., Chicago, IL, USA), the threshold for statistical significance (alpha level) for all tests being set at p <5%.

Results

Evaluation of reliability and validity of the pain outcome scale

The Spearman Rank correlation coefficient describing the relationship between the 5-point pain outcome scale ratings and the 0–10 VAS pain scores was –0.89 (p <0.0001), indicating adequate concurrent validity for the 5-point pain outcome scale. The Kappa value for the ratings on the pain outcome scale (interrater reliability), recorded by two independent observers, was 0.83 (weighted Kappa, 0.85, “very good”).

Pain outcome scale and patient satisfaction after lumbar microdiscectomy

Microdiscectomy was performed in a total of 348 patients with a mean (SD; range) age of 53 (18,9; 17–78) years. All surgery involved segments L3-S1. 35 patient datasets (10%) had to be discarded because incomplete. Hence 313 patients met the inclusion criteria. There was a significant reduction in pain level from admission to post-op, post-op to discharge, and admission to discharge (each p <0.05) (fig. 2). Corresponding significant differences (p <0.05) were also found for the functional outcome scale (data not shown). The change in patient satisfaction over the course of the hospital stay is shown in figure 3 . There was a significant improvement (p <0.05) from preoperatively to the first day postoperatively, from preoperatively to discharge, and also from postoperatively to discharge.

Figure 4a and b show the changes in pain levels after microdiscectomy, split by gender. There was a marked decrease in the number of patients in categories 2 (limiting pain) and 3 (nonlimiting pain) and a simultaneous increase in the proportion in categories 4 (slight pain) and 5 (no pain). There was no significant difference in the pattern of change between men and women.

Discussion

Evaluation and documentation of clinical and surgical procedures is important for reduction of error, improvement of clinical processes and cost reduction while maintaining and/or improving quality and safety of treatment [17]. Quality management is a continuous process of evaluation that also includes documentation and management of complications. Several aspects of care need to be evaluated, including feedback from patients. To obtain disease-specific measures of outcome for all pathologies seen in a neurosurgical unit is challenging [18]. Further, such detailed evaluations would exceed the hospital’s administrative capacity. However, assessment of the type used in the present study, which relied on more “generic” or “global” ratings in the main domains of interest (pain, function, satisfaction, etc.), allows sufficient data to be gathered in an easy-to-use form applicable to all relevant clinical entities. Such instruments can simplify information exchange between medical personnel, especially during the handover process between shifts, a problem that needs to be faced due to more stringent working regulations. In the present study we developed a simple, self-explanatory and userfriendly tool for documentation of the most relevant aspects of the patient’s course without a relevant cost increase.

Many different types of outcome documentation are available. Most are quite specific and only incompletely cover a neurosurgical patient collective. For example, a wellestablished tool such as the Spine Society of Europe’s “Spine Tango” Surgical Registry [19,20] is suitable for use in the documentation of patient history, surgical procedures used, and follow-up of the spinal patient, especially when there is interest in long-term follow-up. However, in contrast to the system described in the present study, it is not suitable for short-term evaluation or daily documentation during the hospital stay. Further, it does not cover cranial, vascular and other neurosurgical treatment groups.

Completion of a complex chart limits the feasibility of any quality management system. In the present study the simplicity of the form and optimisation of its layout was expected to reduce the amount of missing data and improve user compliance. Although this aspect was not formally examined, the feedback from users suggested that the finally chosen layout made for a highly acceptable system. Completion of the documentation chart required only approximately 20–30 seconds for each patient. After initial testing of earlier versions it was obvious that use of a short form was necessary to ensure user acceptance. Compared to our earlier prototypes the time for completion of the final version was approximately halved. Further, many errors were made in the earlier versions, due to the different scaling methods used; this was resolved by adapting all scales to a 5-point system.

Reduction or simplification does not necessarily result in unreliable information, as shown in a recent study comparing 10-point and 5-point scales for patient experiences [21]. Others have also shown almost identical responsiveness for a 0–10 VAS and a 5-point Likert scale, the latter being easier to interpret and apply [22]. The 5-point Pain Outcome Scale used in the present study showed good interobserver reliability and acceptable construct validity, in that its scores correlated well with those of the 0–10 VAS for pain.

It was considered useful to include functional and workcapacity evaluations in the assessment battery, in order to address insurance requests, invalidity considerations and decision-making regarding the timing and extent of return to work. Other scales were used to document a trend in clinical progress (well-being in daily quality scale), clinical events (complication scale) or activities (therapy scales, imaging) and patient satisfaction.

The results of the pain outcome scale and patient satisfaction scale for a subset of patients were evaluated to provide examples showing the functionality of the system. The statistical analysis supported confirmed the validity and reliability of the clinical data management tool: the known effectiveness of microdiscectomy in the treatment of lumbar disc herniation [23] was reflected by the results of our analysis of pain reduction and patient satisfaction with treatment.

Interpretation of the outcome data is influenced by the disease and the treatment administered. For example, the functional scale showed deficits in the hormone axis when evaluating patients with hypophyseal masses, whereas it showed motor function deficits in trauma cases (specific data not shown). The combination of different scales provides important information, e.g., stable disease seen on imaging during chemotherapy shows the positive therapeutic effect of medication. Another example is the dosage and quality of analgesics, which is an important tool for evaluation of the level of pain and therefore the success of conservative or surgical treatment. All these reflect the variety of complex entities that can be documented using a simple tool.

In an effort to obtain complete data sets and prevent errors in completion of the forms we conducted regular training sessions, especially for new users. This also aimed to improve the consistency of completion, i.e., using a common language in understanding each scale/item. Data produced and collected by only one team might raise concerns about bias, but costs prohibited us from including the option of external observation for verification of the data. The daily documentation was performed by the physician in charge, who in most cases was not the surgeon himself. Tracing of the data collected was possible, as each physician had a unique name code that was documented along with the corresponding patient data.

Conclusion and outlook

Outcome scales provide us with an easy-to-use but nonetheless detailed system to evaluate the quality of our work and our patients’ clinical course. The idea of using a checklist was taken from aviation [3]. Just as a pilot uses a checklist to complete his preflight checks, we propose the use of one for daily clinical visits, outpatient consultations, emergency appointments and telephone contacts. A standardised list allows systematic data acquisition and also serves as a reminder to the staff to perform all necessary tasks. Continuous documentation of a patient’s condition from admission to the end of treatment is now feasible without major effort from individual team members. Individual surveys as well as patient group evaluation and analysis can be performed. The system is inexpensive, minimally time consuming and easy to use. It should help us to detect and avoid redundancy or errors in our daily work and might lead to improvements in efficiency, patient safety and costs. A future aim is the development of a digital desktop- and palmtop-based system to enable online and onsite data acquisition. While the “outcome scales” of the Department of Neurosurgery at Kantonsspital Aarau were designed in terms of our own needs, the open structure lends itself to use in any other department.

We thank the IT support of the Kantonsspital Aarau (HINTAG) for help in developing this tool and qualified support. We thank PD Dr. Anne F. Mannion for her assistance with the statistical analyses and kind help in the revision and proofreading of the manuscript.