Validity and Reliability of the Staden Schizophrenia Anxiety Rating Scale

In schizophrenia, none of the standard anxiety measures exhibit strong psychometric properties, and all performed poorly against quality assessment criteria. Developed for the schizophrenia population, this study examined the validity and reliability of the Staden Schizophrenia Anxiety Rating Scale (S-SARS) that measures both specified and undifferentiated anxiety. Among 353 schizophrenia patients, strong correlations with anxiety parameters supported the S-SARS’s convergent validity. Criterion-related validity testing yielded accuracy, sensitivity, and specificity rates of around 95%. Its discriminant validity was observed for measures of depression, psychosis, akathisia, fatigue, vigour, procrastination, behavioural inhibition and activation, and personal growth and initiative. Structural validity was found in a single-factor unidimensional model with a 0.953 factor score. Excellent results were found for internal consistency (Cronbach’s alpha = 0.931; Spearman–Brown coefficient = 0.937; Guttman split-half coefficient = 0.928) and inter-rater reliability (Krippendorff’s alpha = 0.852). It incurred no more than a small error of measurement whereby the observed scores were within 1.54 to 3.58 of a true score on a zero to 50 scale. These strong psychometric properties suggest that the S-SARS is a valid and reliable instrument for measuring specified and undifferentiated anxiety in schizophrenia, providing the means for the accurate measurement of anxiolytic treatment effects.


Introduction
Anxiety in schizophrenia is common and may present concurrently both as a specified anxiety disorder and in an undifferentiated way [1]. The pooled prevalence rate of specified anxiety disorders in schizophrenia was 38.3% in a meta-analysis of 52 studies, which was higher than the 28.8% reported for the general population [2,3]. In another study, the prevalence was 45% in schizophrenia patients compared to 16% among controls [4]. Prevalence rates for the individual specified anxiety disorders in schizophrenia were previously reported as 14.9% for social phobia, 9.8% for panic disorder, 10.9% for generalised anxiety disorder, 12.1% for obsessive-compulsive disorder, and 12.4% for post-traumatic stress disorder.
Undifferentiated anxiety may be understood as a constellation of clinically significant anxiety features that do not meet diagnostic criteria for any of the specified anxiety syndromes, for which the DSM-IV and DSM-5 respectively provide an anxiety disorder not otherwise specified and an unspecified anxiety disorder [5,6]. Undifferentiated anxiety may also be understood as an anxiety feature that is not specific to any one specified anxiety disorder [1]. Palpitations, hypervigilance, restlessness, trembling, and feelings of tension are examples. In contrast, panic attacks, obsessions, compulsions, specific fears, and excessive worries are more specific to the specified anxiety disorders. Undifferentiated anxiety may feature in the presence or absence of the specified anxiety disorders. For In contrast, this article analyses the data from these studies, separately and pooled, in reporting on the validity and reliability of the S-SARS in both acute and residual phases of schizophrenia.

Participants
All participants were 18 years or older and recruited to research participation at a large public sector psychiatric hospital in South Africa contributing to three data sets between the years 2010 and 2019 [1,9,17]. Participants for the first two data sets were inpatients within 10 days of being hospitalized. They met DSM-IV diagnostic criteria for schizophrenia in the acute phase [5]. The acute phase was further defined by the requirement of 60 or more score on the Structured Clinical Interview for the Positive and Negative Syndrome Scale (SCI-PANSS) [18]. Additionally, a score of four (i.e., moderate) or more was required on any two of the SCI-PANSS items that constitute psychotic items, namely, those that measure delusions, hallucinatory behaviour, conceptual disorganisation, or suspiciousness; participants had to be capable of giving informed consent to participate in research, which was captured and affirmed in an informed consent document.
In averting potential confounding influences on anxiety, the following exclusion criteria were applied to the first two data sets: No patients were included who had taken a benzodiazepine during the 24 h preceding the application of the measuring instruments, or for whom zuclopenthixol acetate was administered less than 72 h prior to applying the measuring instruments. Further exclusion criteria were patients with neurological conditions affecting the central nervous system, a known other medical condition that influenced their mental state, or substance dependence (excluding nicotine) by DSM-IV criteria.
The third data set comprised of outpatients attended at the same hospital with residual avolitional schizophrenia [17]. For inclusion, participants had to be diagnosed with schizophrenia in partial remission as defined by the Fifth Edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) [6]. The partial remission qualifier was further specified by the DSM-IV definition of the residual phase whereby non-remitting features were restricted to "only negative symptoms or by two or more symptoms listed in Criterion A present in an attenuated form" [5]. Avolition was defined on the SCI-PANSS, requiring a rating of no less than 3 on the item labelled "disturbance of volition" (G13) as well as a minimum cumulative score of 10 for items G13, N4 (passive/apathetic social withdrawal) and N2 (emotional withdrawal) [18]. As the G13 item described as a "disturbance in the wilful initiation, sustenance, and control of one's thoughts, behaviour, movements and speech" had previously been shown to overlap with expressive deficits (reflecting a loss of initiative) and social amotivation [19], this composite inclusive criterion was adopted for defining an avolitional population notwithstanding an overlap with other negative symptoms. Adequate information to this end was obtained through an interview and observation guide and scrutiny of clinical records, and information from nursing personnel and family. For inclusion, patients were furthermore required to be in a stable condition without any changes to their medication or dosages during the preceding three months, which was verified through self-report and the clinical records.
Exclusion criteria that pertained for the third data set were patients in an acute phase of schizophrenia as defined by DSM-5, and if self-reported or noted in the clinical records that there was a positive substance history during the preceding three months. Other exclusion criteria were intellectual disability, unstable or significant other medical disorders, and a past head injury with neurological sequelae or a loss of consciousness.

Assessments and Measuring Instruments
The 10-item Staden Schizophrenia Anxiety Rating Scale (S-SARS) being the focus of investigation in this study is a clinician-rated instrument for the assessment and measurement of specific and general anxiety in schizophrenia (see Supplementary Materials). Five items of the specific anxiety subscale measure persecutory and nihilistic anxiety, perceptual anxiety, anxiety attacks, situational anxiety, and obsessive-compulsive anxiety. The five items of the general anxiety subscale measure somatic anxiety, psychomotor and cognitive agitation, worry and fear, control-related anxiety, and impairment from anxiety. Each item has six narrative anchor points to indicate severity of anxiety during the preceding week on a scale from 0 to 5, and is accompanied by guided questions for use during the interview as to inform the ratings.
Anxiety was also measured by the Hamilton Anxiety Rating Scale (HAM-A), which is one of the most established and widely used observer-rated scales for anxiety [20]. The HAM-A is a general measure of anxiety that is not specific to schizophrenia or any specified anxiety disorder. Its 14 items are each rated from zero (i.e., none) to four (i.e., severe, grossly disabling), which are mostly about somatic and behavioural manifestations of anxiety.
The Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I) was applied in clinician-administered interviews for diagnosing anxiety disorders [21,22].
The Structured Clinical Interview for the Positive and Negative Syndrome Scale (SCI-PANSS) served as measure of psychotic episode severity [18]. One of its items on anxiety, was analysed as an additional parameter of anxiety.
The Calgary Depression Scale for Schizophrenia (CDSS) comprises nine items with descriptive anchor points that are clinician-administered. Its validity is well-established in correlating strongly to very strongly with other measures of depression [23] and accurately distinguishing depressive features from negative symptoms and extrapyramidal side effects. Cronbach's alpha coefficients between 0.7 and 0.9 suggest good internal consistency [24].
The Vigour Assessment Scale (VAS) measures by self-report both positive (being present) and negative (being absent) vigour [17]. Each of its 27 items are rated on a four-point Likert scale according to the frequency of the experience during the preceding seven days (1 = None of the time, 2 = Sometimes, 3 = Often, 4 = Most of the time). To obtain the total score, the subtotal of Category A (absence of vigour) is subtracted from the subtotal of Category B (presence of vigour). To prevent acquiescence bias, its positive and negative items are interspersed. Its convergent and discriminant validity were good, with a Cronbach's alpha value of 0.82, a clear six-factor correlation structure and small error of measurement [17].
The Personal Growth and Initiative Scale (PGIS) measures the active involvement and development of an individual as a person. The 16-item version has four subscales, which are readiness for change (RC), planfulness (Plan), using resources (UR), and intentional behaviour (IB). It has a four-factor structure for which adequate goodness-of-fit indices were found. Adequate test-retest reliability indices ranged from 0.73 (UR) to 0.81 (Plan), and the internal consistency of the subscales was good (RC, α = 0.83; Plan, α = 0.84; UR, α = 0.80; IB, α = 0.89) [25].
The Behavioral Inhibition/Behavioral Activation Scales (BIS/BAS) measure an individual's sensitivity to two motivational systems, captured by 24 items in four subscales [26]. Behaviour inhibition subscale (BIS) is about the anticipation of punishment, non-reward, and novelty. The three behaviour activation subscales are about drive (BAS-D), fun-seeking (BAS-FS), and reward responsiveness (BAS-RS). The subscales accounted cumulatively for 49% of the overall variance in a sample of 732 college students. Cronbach's alpha values were as follows: BIS, α = 0.74; Reward Responsiveness, α = 0.73; Drive, α = 0.76; Fun Seeking, α = 0.66). Test-retest reliability after eight weeks in 113 subjects yielded moderately strong correlations. Schizophrenia patients in comparison with healthy controls showed no differences in BAS sensitivity but higher BIS sensitivity [27]. In a study among 151 schizophrenia patients regarding approach and avoidance tendencies, significant sensitivity patterns in behavioural inhibition and activation were found [28].
The 10-item Fatigue Assessment Scale (FAS) uses a five-point Likert scale ranging from 'never' to 'always'. It is not specifically designed for schizophrenia, but its validity and reliability have been shown for various other populations, including women with breast problems, construction workers, and mothers of infants and young children [32][33][34]. Internal consistency testing yielded alpha ratings between 0.88 and 0.90, and strong correlations in test-retest reliability measurements a month apart (r = 0.88). Discriminant validity testing was demonstrated for state anxiety, depressive features, and neuroticism [32].

Procedures and Ethics Approval
Patients and their clinical records were assessed for meeting the inclusion and exclusion criteria. The interview-based measures (these are the S-SARS, SCI-PANNS, SCID, HAM-A, and the CDSS) were administered by the authors, all suitably trained in their use. Each participant gave written informed consent to participate in the study, which was performed in accordance with the Declaration of Helsinki. Ethics approval was obtained from Faculty of Health Sciences Research Ethics Committee at the University of Pretoria, South Africa.

Statistical Analyses
Point-biserial correlation testing between S-SARS scores and gender was performed. Spearman's rho and Kendall's tau correlation coefficients were calculated for the correlation between the S-SARS and age, as the latter did not follow a normal distribution as is usually the case. Psychometric analyses were performed both for each data set and combined data sets insofar as measures were shared among the data sets, as the three data sets did not derive from an identical set of measures. In reporting the results, each table below indicates which measures were shared among the data sets and to which data sets the results pertain. For convergent validity, parametric (Pearson's) correlations of the total S-SARS scores with the HAM-A and the anxiety item on the SCI-PANSS were tested and hypothesised to be moderate to strong. For criterion-related validity, a multiple canonical discriminant analysis was performed in examining the ability of an S-SARS model to account for three diagnostic groups as yielded by the SCID-these were, respectively, the no anxiety, the anxiety disorder not otherwise specified, and the specified anxiety groups. In addition, the ability of S-SARS model to accurately diagnose the presence or absence of an anxiety disorder was calculated in terms of specificity, sensitivity, positive predictive value, negative predictive value, miss rate or false negative rate, fall out or false positive rate, false discovery rate, and false omission rate using standard formulas for these. For discriminant validity, Pearson's correlations between the S-SARS and each of the non-anxiety measures were tested and hypothesised to be no more than weak, or negative.
Reliability testing of the S-SARS comprises calculations of the Cronbach's alpha coefficient for internal consistency, Spearman-Brown and Guttman split-half coefficients, and the Standard Error of Measurement (SEM) (as the product of the standard deviation and the square root of one minus the reliability coefficient). For structural validity and consistency of measurement, an exploratory factor analysis using Principal Axis Factoring was performed. The S-SARS's inter-rater reliability was examined by calculating the extent of agreement among nine senior medical students who applied the S-SARS to an audio-visual recording of an interview with a patient who suffered from anxiety in the acute phase of schizophrenia. For the extent of agreement, the intra-class correlation coefficient and Krippendorff's alpha for an ordinal metric were calculated.
The probability threshold for a type I error was set at 5%. The strength of correlation coefficients was defined as follows: r < 0.20 is negligible; 0.20 < r < 0.40 is weak; 0.40 < r < 0.60 is moderate; 0.60 < r < 0.80 is strong; r > 0.80 is very strong [35]. The strength of each psychometric property was defined conventionally as follows. Cronbach's α > 0.9 is excellent, 0.8 < α < 0.9 is good, 0.7 < α < 0.8 is acceptable, 0.6 < α < 0.7 is questionable, 0.5 < α < 0.6 is poor, and α < 0.5 is unacceptable [8]. SPSS version 27 was used for the analyses. Table 1 presents the gender and age for three data sets comprising 343 participants, of whom three-quarters were male. The S-SARS did not correlate significantly with gender (r = 0.030; p = 0.580), but weakly and inversely with age (ρ = −0.209; p < 0.001; τ = −0.156; p < 0.001). Table 2 presents the mean scores, standard deviations, and 95% confidence intervals of the mean for the various measures in each of the data sets and when combined. Table 3 does the same, but for the measures that were only applied in the third data set. The mean S-SARS scores were higher for the in-patient data sets than for the outpatients (i.e., data set no 3), whereas the outpatients scored higher on the avolition items of the SCI-PANSS, which was consistent with the way in which the population for data set no 3 had been defined.

Convergent and Criterion-Related Validity
Supporting convergent validity, the S-SARS correlated strongly with the HAM-A (r = 0.711; see Table 4) and the anxiety item on the SCI-PANSS (r = 0.762). Furthermore, the S-SARS showed criterion-related validity with the diagnosis of an anxiety diagnosis yielded by the SCID. A multiple discriminant analysis on the first data set resulted in a two-function model for the S-SARS items that accounted for the three diagnostic groups, these are, respectively, the no anxiety, the anxiety disorder not otherwise specified, and the specified anxiety groups. With a high Eigenvalue of 3.7, Function 1 expressed 86.8% of the variance (p < 0.001; canonical correlation = 0.89), but Function 2 also contributed significantly (Eigenvalue = 0.56; 13.2% of the variance; canonical correlation = 0.6; p = 0.005). Figure 1 shows the diagnostic group centroids for the two functions, which depicts how well the two-function model of the S-SARS accounted for the three diagnostic groups. The structure matrix for the S-SARS items contributing to the standardised canonical functions is presented in Table 5. As seen in Table 6, this S-SARS model correctly classified the patients into the three groups as classified by the SCID in 88.3% of cases. In total, 7 of the 60 patients were found to be misclassified; that is, their actual group differed from the group predicted by the S-SARS model. These all shared a very low rating on the 10th item of the S-SARS, which means that for all the misclassified cases, there was very little if any impairment owing to anxiety. Hence, these cases may be considered as "noise" to the model by less constitutive cases, presuming that little impairment implies less constitutive anxiety.
Instead of the three diagnostic groups, the ability of the S-SARS model to accurately diagnose the presence or absence of an anxiety disorder is reflected in the calculations presented in Table 7, based on the one false positive and two false negative instances of anxiety disorder yielded by the S-SARS model among 60 participants (as seen in Table 6).

Discriminant Validity
Supportive of the discriminant validity of the S-SARS, the correlation coefficients seen in Tables 4 and 8 indicated at most a weak correlation (r < 0.5) of the S-SARS with each of the non-anxiety measures, except for the CDSS. The correlation coefficient for the CDSS (r = 0.722) when corrected for measurement error (using reliability coefficients of 0.93 and 0.8 for, respectively, the S-SARS and the CDSS) was reduced to 0.67, which suggests that discriminant validity likely exists between the two measuring instruments [36]. This was also supported by a statistical difference between the measures (t = 10.822, df = 352; p < 0.001).  As seen in Table 6, this S-SARS model correctly classified the patients into the three groups as classified by the SCID in 88.3% of cases. In total, 7 of the 60 patients were found to be misclassified; that is, their actual group differed from the group predicted by the S-SARS model. These all shared a very low rating on the 10th item of the S-SARS, which means that for all the misclassified cases, there was very little if any impairment owing to anxiety. Hence, these cases may be considered as "noise" to the model by less constitutive cases, presuming that little impairment implies less constitutive anxiety.

Internal Consistency and Split-Half Reliability
As presented in Table 9, Cronbach's alpha coefficients among the S-SARS items ranged from 0.875 to 0.931, indicating excellent internal consistency. Cronbach's alpha coefficients for each half of the S-SARS were slightly lower, which is in keeping with the theoretical expectation that instruments comprising fewer items result in lower Cronbach's alpha values. The Spearman-Brown and the Guttman Split-Half coefficients ranged from 0.864 to 0.937.

Standard Error of Measurement
As presented in Table 9, the standard error of measurement (SEM) for the total S-SARS score ranged from 1.54 to 2.58, subject to a 68% degree of certainty using one standard deviation, as is customary for SEM calculations. This means that within a theoretical range from 0 to 50, the observed total scores were within 1.54 to 3.58 points of the calculated true scores. In terms of internal consistency, this means that the total observed score consistently measured what it was supposed to measure, plus or minus 1.54 to 3.58 points.

Structural Validity and Consistency of Measurement
Preceding an exploratory factor analysis, the Kaiser-Meyer-Olkin test of sampling adequacy yielded a coefficient of 0.933, indicating the sample size was sufficient. A Bartlett's test of sphericity was statistically significant (approximate chi-square = 2569.647, df = 45, p < 0.001), meaning that the S-SARS items were significantly related and suited for factor analysis.
The Kaiser criterion, by which factors with an eigenvalue greater than 1.0 should be retained, was applied when using principal axis factoring. This extracted a single factor after four iterations, which accounted for 62.25% of the variance (thus a satisfactory explanation) [37] with a factor score of 0.953 in the covariance matrix. The factor matrix for the S-SARS items is shown in Table 10, which suggests the unidimensionality of the S-SARS.

Inter-Rater Reliability
Nine raters were in agreement on the most common rating for each of the S-SARS items in 82.2% of ratings, which increased to 97.7% when agreement on the second most common rating was added. The intra-class correlation coefficient was 0.987, with a 95% confidence interval between 0.969 and 0.996. The Krippendorff's alpha value was 0.852, which indicates strong agreement. Table 10. Factor matrix of the S-SARS items derived from a factor analysis (n = 353).

Discussion
In contrast with the findings of a recent review that no instrument measuring anxiety in non-affective psychoses exhibits strong psychometric properties and that all the evaluated instruments performed poorly against standardised quality assessment criteria [8], the results of the current study suggest that the S-SARS, as a clinician-administered measure of anxiety in schizophrenia, holds excellent psychometric properties. Criterion-related, convergent, discriminant, and structural validity testing showed that it measured what it was supposed to measure. The results also showed that it measured so consistently, as supported by good to excellent internal consistency and split-half reliability, substantial inter-rater reliability, and homogeneity of measurement.
Convergent validity of the S-SARS was found in its strong correlations with the anxiety item on the SCI-PANNS and another measure of anxiety (i.e., HAM-A), despite the HAM-A not being specific to the schizophrenia population and its narrower scope in focusing on somatic and behavioural manifestations of anxiety. Criterion-related validity modelling yielded accuracy, sensitivity, and specificity rates of around 95%. The discriminant validity of the S-SARS is supported by its correlating negatively and no more than weakly with behavioural drive, reward-responsiveness, fun-seeking behaviour, vigour, avolition, personal growth, and initiative. It correlated no more than weakly with behavioural inhibition, procrastination, and fatigue. The moderate correlations with the SCI-PANSS and akathisia and the strong correlation with depressive features are congruent with previous reports [9,11], but these were below the threshold required to indicate discriminant validity, supported by a statistically significant difference between the S-SARS scores and the measure of depressive features.
The S-SARS's structural validity and consistency of measurement were confirmed in a single-factor unidimensional correlational model with a high factor score of 0.953, and 5 of the 10 items correlating very strongly, 3 items strongly, and 2 items moderately with the single factor. These results of the factor analysis suggest that the S-SARS measured a single construct and so measured what it was supposed to measure. These results also support the structural internal consistency of the measurement, which was congruent with the internal consistency reflected by good to excellent Cronbach's alpha values and strong to very strong correlations between the halves of the S-SARS. The unidimensionality seen in the factor analysis reflects the internal structure of the S-SARS, whereas the two canonical discriminant functions reflect a two-dimensionality in the correlations of the S-SARS items with an external diagnostic criterion. Notwithstanding the S-SARS thus measuring a single unidimensional construct, the finding of its two-dimensional diagnostic property supports the premise that an anxiety measure of sufficient scope in the schizophrenia population should measure both specified and undifferentiated anxiety.
Homogeneity of the S-SARS's measurements was found in it incurring in no more than a small error of measurement, whereby observed scores were within 1.54 to 3.58 of a true score on a scale ranging from 0 to 50. Consistency of measurements among raters was found in 82.2% of identical ratings for the most common rating for each item, a very high intra-class correlation coefficient of 0.987, and strong agreement among raters for all items, indicated by a Krippendorff's alpha value of 0.852.
With these strong psychometric properties of the S-SARS, using it in research and clinical practice may improve the accuracy of measurement of anxiety in schizophrenia. The S-SARS measures anxiety as it features in both the various specified anxiety disorders and undifferentiated anxiety. It does not replace, however, measures with a narrower focus on one of the specified anxiety disorders. Measuring anxiety accurately in schizophrenia is important for various clinical and research reasons. Anxiety adds to the burden of schizophrenia by impacting negatively on quality of life [38], functioning [39,40], overall psychopathology, and the severity of comorbid medical conditions [41]. Increased rates of relapse, more frequent and longer duration of hospitalisations, poorer response to pharmacological treatments, substance abuse, negative attributional style, suicide, and suicide attempts have been associated with anxiety in schizophrenia [42][43][44][45].
Furthermore, claims have been made about the efficacy of antipsychotic medication in reducing anxiety in schizophrenia [46][47][48], but these are scientifically weak insofar as instruments measuring anxiety have been restricted in their scope and psychometric properties as described above. Using the S-SARS in efficacy studies may strengthen conclusions on the anxiolytic effects of anti-psychotic and other treatments in the schizophrenia population.
There are limitations pertaining to the results reported here, which may be addressed in further research. The inter-rater reliability parameters, although obtaining excellent values here, are rather tentative considering the small number of raters. This should be examined further among more raters that assess and rate several patients with varying degrees of anxiety. Furthermore, the outpatients in the study were limited to those with avolition. Although there were no more than weak correlations between anxiety and the avolitional parameter (see Table 4), anxiety qualities may be different in schizophrenia outpatients who are not avolitional. The male preponderance in this study is similar to that seen in probabilistic schizophrenia samples [49]. This study found no correlation between anxiety and gender, but the potential influence of females being under-represented needs to be studied further.

Conclusions
The strong psychometric properties reported here suggest that the S-SARS is a valid and reliable instrument for measuring specified and undifferentiated anxiety in the schizophrenia population. In contrast with existing anxiety measures not specific to schizophrenia, the S-SARS accounts for the anxiety that is expressed within delusional content and in disturbances of perceptions without conflating these symptoms. It provides the means in future studies for the accurate measurement of the anxiolytic effects of treatments, including antipsychotic medication, in the schizophrenia population. Informed Consent Statement: Informed consent was obtained from all participants captured in informed consent documents that had been approved by the Research Ethics Committee.

Data Availability Statement:
The Research Ethics Committee that approved the research determines the limits on the availability of raw and processed data based on the merits of an application to gain access, the interests of stakeholders, and the mandates of research regulatory authorities. No special computer code or syntax is needed to reproduce analyses other than provided standardly in the SPSS-software program (version 27). Regarding the availability of research materials, the S-SARS is available as a supplement to the article.