Validity of a New Administrative Measure of Psychiatric Severity in a Prospective Sample of Veterans Applying for PTSD Disability Benefits: The Manifestations of Psychiatric Severity Index (MoPSI)

Abstract

Background: Administrative data help managers monitor and manage health care enrollees’ health. Of the few available administrative measures of psychiatric illness severity, however, most either commingle sociodemographics and medical comorbidities or lack ordinal properties. Objective: To assess construct, concurrent, and predictive validity of a novel, 6-item, administrative measure of psychiatric severity, the Manifestations of Psychiatric Severity Index (MoPSI). Methods: A panel study of 960 gender-stratified, nationally representative, post-9/11 US Veterans with pending disability claims for posttraumatic stress disorder (PTSD). MoPSI scoring was based on the joint probability density (JPD) method and a JPD linear approximation. Results: The JPD MoPSI score and its linear approximated score had a correlation of 0.999. Relative to their counterparts, unmarried Veterans, Veterans with low income, and Veterans with serious mental illness or PTSD had higher MoPSI scores (Ps: <0.0001–0.03). Higher MoPSI scores predicted cigarette and street drug use and PTSD and depression/anxiety symptoms six months later, and disability award approximately 1 year later (Ps: 0.01–0.02). Conclusions: In this sample, the MoPSI had evidence of construct, concurrent, and predictive validity.

1. Introduction

When health care administrative data shifted from paper storage to searchable and linkable computer frameworks in the 1970s, outputs were repurposed from billing to characterizing populations’ health status; prognosticating patient outcomes and costs; and accounting for differences in outcomes according to various patient-, provider-, and system-level attributes [1]. Despite lacking granularity, administrative health care data offer low costs of acquisition, relatively easy access to needed information, and near-universal coverage of populations of interest. Consequently, administrative data are mainstays in managing and monitoring individuals served by health care systems.

Several scalars based on administrative data are available to measure medical comorbidities, including the Charlson Comorbidity Index [2], the Elixhauser [3], and RxRisk [4], as well as their respective updates and extensions, e.g., [5]. These indices predict mortality, health care utilization and expenditures, and functioning [5,6,7]. Psychiatric illness also influences mortality, health care utilization and costs, and other medical outcomes [8,9,10]; therefore, incorporating measures of psychiatric severity into tools for monitoring health care systems is at least as important as incorporating medical comorbidity measures.

Four measures of psychiatric severity that combine administrative data with variables collected bedside during an index hospitalization include the READMIT [11], the Rehospitalization Clinical Assessment Protocol (RECAP) [12], Ashcraft et al.’s [13] 1989 algorithm, and the Ontario System for the Classification of Inpatient Psychiatry (SCIPP) [14]. Among patients hospitalized for psychiatric reasons, the first two indices estimate risk of psychiatric rehospitalization within 30 and 90 days, respectively; the third describes variance in hospitalization costs. The SCIPP is intended as a reimbursement algorithm and estimates hospitalization costs relative to an average patient in Ontario. Each of these indices was developed for a specific, narrow purpose. Furthermore, because each relies on data collected bedside, it is difficult to apply them in settings where similar bedside information is not routinely obtained.

The remaining psychiatric severity indices were developed in the Department of Veterans Affairs (VA) and use only routine administrative data. McCarthy et al.’s [15] and Kessler et al.’s [16] Recovery, Engagement, and Coordination for Health–Veterans Enhanced Treatment (REACH-VET) was developed to identify Veterans at high risk of suicide, and the Stratification Tool for Opioid Risk Mitigation (STORM) [17], to predict opioid overdose or suicide risk. These are not purely measures of psychiatric severity, however, as demographics and medical conditions contribute to patients’ final score.

The Psychiatric Case-Mix System (PsyCMS) [18] comprises psychiatric variables only and has been shown to predict between 5% and 24% of the variance in psychiatric bed days, costs, and outpatient encounters at VA hospitals and clinics [18]. Because the PsyCMS conceptualizes psychiatric severity as the sum of individuals’ diagnostic burden, scores are susceptible to diagnostic persistence and diagnostic drop-off. With diagnostic persistence, individuals tentatively diagnosed with a severe diagnosis, such as schizophrenia, may find that diagnosis carried forward in their chart for years, even if it later proves false. With diagnostic drop-off, diagnoses become background noise no longer coded during encounters. After years of managing a patient with somatization disorder, for example, a clinician might start coding encounters only for the medical presentations and not the underlying mental disorder. Simple counts of numbers of diagnoses may also sometimes result in rank orders that do not map to usual notions of psychiatric severity: a former smoker complaining of insufficient sleep and stress related to recent job changes (three diagnoses) could receive a higher psychiatric severity score than a patient with poorly controlled schizoaffective disorder and active opioid dependence (two diagnoses). For these reasons, the PsyCMS is considered a grouping variable [19] without reliably ordinal properties.

REACH-VET, STORM, and PsyCMS also require substantial numbers of inputs—61 variables for REACH-VET, 52 variables and 49 interaction terms for STORM, and 64 variables with 4 nested hierarchical algorithms for PsyCMS. This complexity lends a black box feel to the indices. Clinicians may be aware that a patient has a high score, but they may not know why the score is high—or how to lower it. The large numbers of inputs and interaction terms also present challenges for adopting these scales to smaller health care systems, individual research projects, or point-of-care assessments. When using these indices as predictors, the large number of non-psychiatric inputs increases risk for multi-collinearity.

The Manifestations of Psychiatric Severity Index (MoPSI) is a new, brief measure of psychiatric severity we developed in a sample of US Veterans applying for PTSD disability benefits [20]. Instead of diagnostic counts, the six-item MoPSI captures selected, observable manifestations of severer versus less severe psychiatric symptoms in administrative data, including self-medication (alcohol and substance use), self-harm (e.g., cutting, suicidality), and help-seeking (emergency department visits, mental health clinic visits, and psychiatric hospitalizations). The MoPSI’s ordinal alpha coefficient was 0.923, and it had an omega of 0.75 on confirmatory factor analysis [20]. Item response analysis indicated that its six items were well calibrated and captured a wide range of psychiatric severity [20]. In the present paper, we extend our original report to describe the MoPSI’s construct, concurrent, and predictive validity using the same sample. We also examine the clinical characteristics of groups in our sample as defined by ordinal categories of MoPSI scores. We report a scoring algorithm to facilitate out-of-sample extension.

Hypotheses

In terms of construct validity, were the MoPSI a valid measure of psychiatric severity, we would expect it to correlate with other constructs, such as married status, low income, and cigarette smoking, as observed in other samples where there is a gradient of psychological severity [21,22,23]. Thus, unmarried Veterans, Veterans with low income, and Veterans who smoke should have higher MoPSI scores than their counterparts. In terms of concurrent validity, we would expect individuals diagnosed with frequently disabling psychiatric disorders, such as schizophrenia or schizoaffective disorder, bipolar disorder, and posttraumatic stress disorder (PTSD), to have higher MoPSI scores than individuals without these diagnoses. In terms of predictive validity, we would expect higher MoPSI scores to be associated with worse self-reported PTSD, depression and anxiety symptoms, and problematic drinking and substance use 6 months later. Also, in terms of predictive validity, we would expect that Veterans who received a VA disability award would have higher prior MoPSI scores compared to those who did not receive a disability award.

2. Materials and Methods

2.1. Study Design and Human Studies Oversight

The design is a prospective, nationally representative panel study. The panel was foundation for a gender-blocked, 3 × 2 × 2 factorial trial that examined the impact of cover letter modifications on survey non-response bias [24]. The Minneapolis VA Health Care System’s Internal Review Board for Human Studies reviewed and approved the study protocol (#4495-B). Analyses were pre-planned.

2.2. Participants and Setting

Operations Enduring Freedom (OEF), Iraqi Freedom (OIF), and New Dawn (OND) were part of the United States’ Global War on Terror and were in force from 2001 to 2014. Participants were United States Veterans who had served during OEF/OIF/OND and had a pending VA disability claim for PTSD as of 18 November 2015. From the national sampling frame of 14,630 men and 2945 women with pending claims, we used simple randomization without replacement to select 480 men and 480 women to receive mailed surveys. These 960 men and women comprise the panel, and 410 panel members returned completed surveys for a response rate of 42.7%.

Survey participants were recruited between February and August 2016. Administrative data were collected retrospectively for all panel members between 5 August 2015 and 1 February 2016. VA disability award status was assessed on 11 April 2017.

The panel’s median age was 33.0 years (mean = 35.2, SD = 8.9, range = 19–67); 58.1% of the men and 40.6% of the women were White, 19.6% of the men and 36.5% of women were Black or African American, and 13.3% of men and 10.2% of women were Hispanic. In addition, 71.6% of men and 61.1% of women had a chart diagnosis of PTSD; 7.1% of men and 10.3% of women had been diagnosed with bipolar disorder, schizophrenia, or schizoaffective disorder. Forty-eight panel members did not use any VA health care during the study and are excluded from these analyses.

2.3. Survey Protocol

We used a mailed survey to collect information about Veterans’ self-reported symptom severity and use of substances. As described elsewhere [24], Veterans received pre-notification letters about the survey, followed by a postcard reminder and up to three copies of a mailed questionnaire at 2-week intervals. Veterans were assured that the survey was confidential. They were told that a numbered survey code would be used to keep people from linking the respondents’ names to their surveys and that results would be published in aggregate. Returning a completed questionnaire signified consent to participate in the survey. All methods were carried out in accordance with relevant guidelines and regulations.

2.4. Measures

2.4.1. Variables for the Manifestations of Psychiatric Severity Index

All administrative data were extracted from the VA Corporate Data Warehouse. MoPSI scores were calculated for the 180 days prior to mailing Veterans their first survey. Appendix A shows the ICD-9 and ICD-10 scores used for self-harm—including suicidality—and for substance and alcohol use.

2.4.2. Demographics-Administrative Data

Concurrently with calculating MoPSI scores, we extracted administrative demographic variables, including age, race and ethnicity, and gender. Based on natural cut-points in the data, we trichotomized age as <30 years, 30 to 50 years, or more than 50 years. We used the race and ethnicity and the gender data fields from the Veterans Benefits Administration. These fields are based on Veterans’ self-report at the time they applied for disability benefits. The fields categorize Veterans’ self-report into seven mutually exclusive race and ethnicity categories (“Asian or Pacific Islander”, “Black or African American”, “Hispanic ethnicity”, “Other”, “Unknown”, or “White”) and into two mutually exclusive gender categories (“male” or “female”).

2.4.3. Validation Measures-Administrative Data

Concurrently with calculating MoPSI scores, we extracted Veterans’ married status and total annual household income. Married status was categorized as “currently married” versus “not”, and income was categorized as “low” (USD 0–20,000) versus “higher than low” (>USD 20,000). Income is not routinely collected for all Veterans, and data were missing for 421 panel members. We used ICD-9-CM and ICD-10-CM codes to determine whether Veterans had been diagnosed with PTSD or with schizoaffective disorder, schizophrenia, or bipolar disorder. The latter three were classified as serious mental illness. Slightly more than 1 year after MoPSI scores were calculated, we revisited the Corporate Data Warehouse to ascertain whether panel members had been granted a VA disability award.

2.4.4. Validation Measures-Survey Data

The mailed questionnaire asked if respondents were currently married and asked about yearly household income in the following categories: USD 0–20,000, USD 20,001 to 40,000, USD 40,001–60,000, or over USD 60,000. We used the Penn Inventory for PTSD [25] to assess Veterans’ PTSD symptom severity in the week prior to survey; the RAND Mental Health Battery-short form [26] to assess depression and anxiety symptoms in the prior month; the TWEAK [27] to assess current problem drinking; and two face valid “yes/no” items to assess cigarette or illicit substance use (“Do you currently smoke cigarettes?” and “Do you ever use ‘street’ drugs?”). Among Veterans, Penn Inventory scores ≥ 35 have 0.90 to 0.98 sensitivity and 0.94 to 1.00 specificity for PTSD diagnosis [25]. The RAND Mental Health Battery-short form has 48% sensitivity for major depression, but almost 95% specificity [26]. Using a cutoff score of 3 or more, the TWEAK’s pooled sensitivity is 0.85 and specificity is 0.86 [28].

2.5. Statistical Analysis

We used R 4.2.1, SAS version 9.4, and STATA version 18.0 for analyses.

2.5.1. JPD Scoring

As reported elsewhere [20], when developing the MoPSI, we applied confirmatory factor analysis, three item-response theory approaches, and a novel approach we call the joint probability density (JPD) method to create unidimensional scores of psychiatric severity based on the six input variables. The JPD method of index creation uses the joint probability density or probability mass function of interdependent multiple indicators to summarize conceptually related information into a scalar as long as the inputs target the same construct (unidimensionality), have approximately the same scoring direction (unidirectionality), and correlate or agree with the total scalar’s score (codirectionality) [20]. We used chain rule factorization to estimate the joint probability density of the six MoPSI variables:

$$f\left(x_1,x_2,\dots ,x_6\right)=f\left(x_1\right)f\left(x_2|x_1\right)\dots f\left(x_6\right|x_1,\dots ,x_5)$$

(1)

where $f\left(x_1,x_2,\dots ,x_6\right)$ is the joint probability density, and $x_1,x_2,\dots ,x_6$ represent the six MoPSI variables. The chain rule can be applied in 6! (=720) ways. Each of these 720 conditionally specified models approximately capture a dependency structure. Those terms provide an estimate of the joint model, $f\left(x_1,x_2,\dots ,x_6\right)$—or, equivalently—its $log\left(f(x_1, x_2, . . ., x_6\right))$. To utilize all the information captured by these 720 models, we used the average of the estimates:

$$log\left(f(x_1, x_2, . . ., x_6)\right)= \sum _{6!}logf\left(x_{i_1}, x_{i_2}, . . ., x_{i_6}\right)/6!$$

(2)

The averaged $log\left(f\left(x_1, x_2, . . ., x_6\right)\right)$ is the MoPSI-JPD score.

Scores derived from confirmatory factor analysis, the various item-response models, and the JPD method correlated 0.976 to 0.991. However, the JPD method resulted in the lowest Shannon entropy and highest Pearson divergence, as well as the highest granularity [20]. Thus, the JPD method was the most informative. For reasons of space, we report MoPSI-JPD scores and approximated MoPSI-JPD scores (see “JPD Approximation” below) here. Results based on confirmatory factor analysis and item response theory were equivalent to those based on the JPD method and are available in the Supplementary Files.

2.5.2. JPD Approximation

An R macro is available to estimate input variables’ log joint probability density and return a unidimensional scalar [20]. However, we anticipated that future users would prefer to estimate MoPSI-JPD scores using a linear approximator. Therefore, we randomly selected 70% of the sample to be a training set, then regressed MoPSI-JPD scores on the six input variables (substance use; alcohol use; self-harm events; number of mental health clinic visits, emergency department visits, and psychiatric hospitalizations) using ordinary least-squares regression. The resulting linear equation was then applied to the remaining 30% of the sample (the test set).

The regression equation for the approximated MoPSI-JPD score (MoPSI-approxJPD) based on the 70% training set was as follows:

$$MoPSI approxJPD=-17.436+0.016\left(MHvisits\right)+2.106\left(M{H}_{ED}\right)+2.457\left(MHhospital\right)+4.452\left(Selfharm\right)+4.072\left(SUD\right)+3.325\left(ETOH\right)$$

(3)

where −17.436 = the intercept, and the remaining parameters are explained in Table 1 below.

Table 1. MoPSI parameters.

	Explanation	How Scored	Regression Weight
MHvisits	Number of mental health clinic visits	Continuous count	0.016
MH_ED	Number of emergency department or urgent care visits for psychiatric issues	Continuous count	2.106
MHhospital	Number of psychiatric hospitalizations	Continuous count	2.457
Selfharm	Any self-harm events, including suicidality	1 = Yes 0 = No	4.452
SUD	Any substance use diagnosis	1 = Yes 0 = No	4.072
ETOH	Any alcohol use diagnosis	1 = Yes 0 = No	3.325

Concordance between original MoPSI-JPD scores and MoPSI-approxJPD scores, as measured by Spearman’s r, was 0.999 for both the training and test set. The residual standard errors from the regressions were very small, just 0.24 for the training set and 0.22 for the test set.

2.5.3. Validation Assessments

To account for skewness, we used the Kruskal–Wallis χ² to test for differences in MoPSI scores by grouping characteristics. Kendall’s tau was used to assess the association between MoPSI scores and continuous variables. We used Cliff’s delta (δ) as a non-parametric effect size estimator for contrasts across predictive, categorical variables. Cliff’s δ is similar to Cohen’s d, with cut points < |0.15| = negligible; |0.15| = small; |0.33| = medium; and |0.47| = large. Cliff’s δ was the same regardless of whether we used the MoPSI-JPD, MoPSI-ApproxJPD, or the other scoring algorithms; therefore, for space, we report δ only for the MoPSI-JPD. We use a traditional threshold of p ≤ 0.05 to denote statistical significance.

2.5.4. Power

Analysis was based on the number of panel members. No formal power analysis was conducted.

3. Results

Summary statistics for the two MoPSI-JPD scoring approaches are shown in Table 2. As can be seen, the scores’ distributions are highly similar. MoPSI scores were not associated with survey response status (Ps = 0.28); likewise, scores did not differ significantly by age, race, or gender (Table 3).

Table 2. Summary statistics.

		MoPSI Scoring Approach
		MoPSI-JPD	MoPSI-ApproxJPD
Summary Statistic	Mean	−16.51	−16.51
	Median	−17.47	−17.42
	Standard deviation	2.79	2.78
	Interquartile range	−0.30	−0.15
	Minimum	−17.51	−17.44
	Maximum	5.67	8.49

MoPSI = Manifestations of Psychiatric Severity Index. JPD = joint probability density. ApproxJPD = approximated joint probability density.

Table 3. Manifestations of Psychiatric Severity Index scores by demographic characteristics.

		N	df	MoPSI-JPD Scores				MoPSI-ApproxJPD
				Mean (SD)	Median (IQR)	χ2	p-Value	Mean (SD)	Median (IQR)	χ2	p-Value
Age	<30 years	282	2	−16.46 (3.13)	−17.47 (−0.27)	0.54	0.76	−16.44 (3.15)	−17.42 (−0.13)	0.83	0.50
	30 to 49 years	548		−16.51 (2.71)	−17.47 (−0.30)			−16.51 (2.70)	−17.42 (−0.15)
	≥50 years	82		−16.76 (1.91)	−17.47 (−0.27)			−16.78 (1.78)	−17.42 (−0.13)
Race or ethnicity	White	456	6	−16.47 (2.97)	−17.47 (−0.33)	6.06	0.42	−16.47 (2.97)	−17.42 (−0.17)	5.71	0.46
	Black	251		−16.67 (2.32)	−17.47 (−0.23)			−16.67 (2.17)	−17.42 (−0.12)
	Hispanic	109		−16.59 (2.59)	−17.47 (−0.30)			−16.55 (2.70)	−17.42 (−0.15)
	Asian/PacIsl	57		−16.65 (2.43)	−17.47 (−0.30)			−16.65 (2.39)	−17.42 (−0.15)
	Am. Indian	13		−14.84 (6.39)	−17.51 (−0.27)			−14.74 (6.50)	−17.44 (−0.13)
	Other	11		−17.46 (0.80)	−17.47 (−0.07)			−17.41 (0.04)	−17.42 (−0.04)
	Unknown	15		−15.18 (3.61)	−17.38 (−4.12)			−15.29 (3.35)	−17.37 (−4.11)
Sex	Male	459	1	−16.30 (3.22)	−14.47 (−0.30)	0.27	0.60	−16.29 (3.25)	−17.42 (−0.15)	0.28	0.60
	Female	453		−16.73 (2.26)	−17.47 (−0.27)			−16.75 (2.17)	−17.42 (−0.13)

df = degrees of freedom. MoPSI = Manifestations of Psychiatric Severity Index. JPD = joint probability density. ApproxJPD = approximated joint probability density. SD = standard deviation. IQR = interquartile range. PacIsl = Pacific Islander. Am = American. Kruskal–Wallis χ².

3.1. Construct Validity

As anticipated, unmarried Veterans had significantly higher average MoPSI scores than did married Veterans (Table 4). Overall, MoPSI scores were also significantly higher among those with low income compared to those with higher income.

Table 4. Construct validity: Manifestations of Psychiatric Severity Index scores by sociologic characteristics.

				MoPSI-JPD Scores					MoPSI-ApproxJPD
		N	df	Mean (SD)	Median (IQR)	δ	χ2	p-Value	Mean (SD)	Median (IQR)	χ2	p-Value
Married	Yes	358	1	−16.71 (2.09)	−17.47 (−0.27)	−0.09	4.82	0.03	−16.71 (2.02)	−17.42 (−0.13)	4.56	0.03
	No	445		−16.15 (3.46)	−17.44 (−0.43)				−16.16 (3.49)	−17.40 (−0.22)
Married (SR)	Yes	215	1	−16.86 (2.02)	−17.47 (−0.23)	−0.15	7.09	0.008	−16.86 (1.91)	−17.42 (−0.12)	6.78	0.009
	No	175		−16.12 (3.42)	−17.44 (−0.59)				−16.12 (3.51)	−17.40 (−0.30)
Low Income	Yes	190	1	−15.52 (3.96)	−17.34 (−3.26)	−0.26	25.79	0.0001	−15.57 (3.96)	−17.35 (−2.32)	25.60	0.0001
	No	349		−16.68 (2.32)	−17.47 (−0.27)				−16.67 (2.34)	−17.42 (−0.13)
Low Income (SR)	Yes	98	1	−15.50 (4.26)	−17.41 (−0.97)	−0.15	5.29	0.02	−15.50 (4.38)	−17.39 (−0.49)	5.02	0.03
	No	289		−16.86 (1.92)	−17.47 (−0.23)				−16.86 (1.84)	−17.42 (−0.12)

df = degrees of freedom. MoPSI = Manifestations of Psychiatric Severity Index. JPD = joint probability density. ApproxJPD = approximated joint probability density. SD = standard deviation. IQR = interquartile range. SR = self-reported survey data. Kruskal–Wallis χ².

3.2. Concurrent Validity

Overall, Veterans diagnosed with serious mental illness and with PTSD had significantly higher concurrent MoPSI scores than did Veterans without these disorders (Table 5).

Table 5. Concurrent validity: Manifestations of Psychiatric Severity Index scores by mental health diagnosis.

				MoPSI-JPD Scores					MoPSI-ApproxJPD Scores
		N	df	Mean (SD)	Median (IQR)	δ	χ2	p-Value	Mean (SD)	Median (IQR)	χ2	p-Value
Serious mental illness	Yes	61	1	−13.89 (5.56)	−16.73 (−4.14)	−0.57	58.18	<0.0001	−14.02 (5.67)	−17.04 (−4.07)	54.63	0.0001
	No	851		−16.70 (2.37)	−17.47 (−0.23)				−16.69 (2.35)	−17.42 (−0.12)
PTSD	Yes	298	1	−15.13 (4.24)	−17.18 (−3.33)	−0.70	311.65	<0.0001	−15.21 (4.28)	−17.27 (−3.36)	294.24	<0.0001
	No	614		−17.18 (1.20)	−17.51 (−0.07)				−17.15 (1.16)	−17.44 (−0.04)

df = degrees of freedom. MoPSI = Manifestations of Psychiatric Severity Index. JPD = joint probability density. ApproxJPD = approximated joint probability density. SD = standard deviation. IQR = interquartile range. PTSD = posttraumatic stress disorder. Kruskal–Wallis χ².

3.3. Predictive Validity

Overall, survey respondents who reported smoking cigarettes and using street drugs had significantly higher prior MoPSI scores than those who did not report using those substances (Table 6). Although effects were in the expected direction, differences in MoPSI scores by problem alcohol drinking were not statistically significant. MoPSI scores were significantly higher for Veterans who received a VA disability award 1 year later. Kendall’s tau between MoPSI scores and self-reported PTSD and depression/anxiety symptoms 6 months later was 0.10 (Ps ≤ 0.009) (Table 7).

Table 6. Predictive validity: Manifestations of Psychiatric Severity Index scores by substance use and disability award.

				MoPSI-JPD Scores					MoPSI-ApproxJPD Scores
		N	df	Mean (SD)	Median (IQR)	δ	χ2	p-Value	Mean (SD)	Median (IQR)	χ2	p-Value
Cigarettes (SR)	Yes	90	1	−15.73 (3.76)	−17.41 (−0.91)	−0.17	6.03	0.01	−15.73 (3.78)	−17.39 (−0.46)	5.71	0.02
	No	297		−16.75 (2.35)	−17.47 (−0.27)				−16.75 (2.35)	−17.42 (−0.13)
Street drugs (SR)	Yes	46	1	−14.94 (4.49)	−17.38 (−3.62)	−0.22	6.18	0.01	−14.92 (4.61)	−17.37 (−3.51)	5.88	0.02
	No	326		−16.77 (2.37)	−17.47 (−0.23)				−16.76 (2.35)	−17.42 (−0.12)
Problem drinking (SR)	Yes	114		−16.00 (3.30)	−17.44 (−0.59)	−0.09	1.82	0.18	−15.98 (3.34)	−17.40 (−0.30)	1.82	0.18
	No	258		−16.75 (2.47)	−17.47 (−0.27)				−16.76 (2.47)	−17.42 (−0.13)
Disability award	Yes	764		−16.51 (2.80)	−17.47 (−0.30)	−0.12	5.84	0.02	−16.51 (2.80)	−17.42 (−0.14)	5.50	0.02
	No	148		−16.53 (2.70)	−17.47 (−0.22)				−16.53 (2.66)	−17.42 (−0.11)

df = degrees of freedom. MoPSI = Manifestations of Psychiatric Severity Index. JPD = joint probability density. ApproxJPD = approximated joint probability density. SD = standard deviation. IQR = interquartile range. SR = self-reported survey data. Kruskal–Wallis χ².

Table 7. Predictive validity: Correlation between Manifestations of Psychiatric Severity Index scores and self-reported PTSD and depression and anxiety symptoms 6 months later.

	MoPSI-JPD Score				MoPSI-ApproxJPD Score
	N	df	Kendall’s tau	p-Value	N	df	Kendall’s tau	p-Value
PTSD	358	356	0.10	0.009	358	356	0.10	0.009
Anxiety and depression	385	383	0.10	0.008	385	383	0.10	0.008

MoPSI = Manifestations of Psychiatric Severity Index. PTSD = posttraumatic stress disorder. JPD = joint probability density. ApproxJPD = approximated joint probability density. df = degrees of freedom.

4. Discussion

Findings showed that MoPSI scores corresponded to other constructs in expected directions and usually at statistically significant levels. Results support evidence for the MoPSI’s construct, concurrent, and predictive validity in this sample. Approximated JPD scores derived through regression corresponded almost perfectly to those derived directly through the joint probability density method, providing the easiest option for calculating scores, particularly for out-of-sample extensions or new enrollees (see Appendix B for calculation instructions). Against expectations, we did not see a statistically significant association between higher MoPSI scores and self-reported problematic drinking 6 months later. This was possibly a Type II error, as the effect was in the direction expected.

Although randomly selected and nationally representative, the sampling frame described here is unique. Because these Veterans were seeking disability benefits, we would expect them to have a higher net degree of psychiatric severity than individuals not seeking disability benefits. Veterans seeking disability benefits are a very large group: between 2020 and 2024 alone, the VA added approximately 1.7 million new disability recipients to their rolls [29,30,31,32]. There are now almost 1.6 million Veterans receiving VA disability benefits for PTSD, of whom almost two-fifths are OEF/OIF/OND Veterans [33]. The MoPSI’s generalizability to non-Veterans and to Veterans not seeking disability benefits cannot be established in the present study and should be investigated in future work. Findings here suggest that the MoPSI could be useful for examining psychiatric severity administratively among the large and clinically important group of Veterans who do seek disability benefits.

The MoPSI’s brevity is both a feature and a detriment. As a feature, its brevity lends itself to point-of-care assessments and to self-report tools. The inputs are intuitive and understandable. Its brevity also reduces risk of missingness and of collinearity with sociodemographic and medical conditions. Further, with just 6 inputs, we were able to calculate all 720 dependency factorizations to estimate the MoPSI’s joint probability density. Chain rule factorization with more inputs becomes infeasible very quickly. A 20-variable scalar would require more than 2.43 × 10¹⁸ factorizations, for example. We have recently shown that a much smaller, random selection of dependency factorizations or the backward selection of a single factorization may be sufficient to estimate a scalar’s jpd [34]. Such simplifications should facilitate greater use of JPD index construction [20], allowing psychometricians to preserve more information in a single, summarizing scalar.

In terms of detriment, the MoPSI’s brevity may contribute to its relatively low granularity. Future researchers might consider expanding the MoPSI’s item bank to address this limitation; however, low granularity commonly limits administrative measures, even those with much larger numbers of inputs. In a sample of middle-aged or older individuals, 58% had Charlson Comorbidity scores of zero, despite all participants being hospitalized with ischemic heart disease [35]. Estimated risk of suicide death in the top 0.10% of REACH-VET scorers is increased 30–40-fold relative to all scorers, but—fortunately—99.4% of all top scorers do not attempt suicide [15]. The Charlson Comorbidity Index has 17 inputs, and REACH-VET, as previously mentioned, has 52. As with other administrative measures, the MoPSI also risks potential systematic differences in coding or in patient access across regions [36,37,38].

Case-mix measures are usually designed to be stable across years [19], whereas our hope was to create an index that waxed and waned with patients’ symptom severity. Because we collected MoPSI scores at only a single time point, our success in this regard is unknown. Future studies should assess the index’s trajectory and sensitivity to changes in patient’s symptoms. We did not identify statistically significant differences in MoPSI scores by age, race, ethnicity, or gender. However, future work might consider testing more formally for measurement invariance by these or other demographic groups. It must also be acknowledged that the MoPSI is only a proxy for—not a direct measure of—psychiatric severity.

Throwing Brick to Attract Jade

There are very few available administrative measures to assess psychiatric severity, and those available either commingle psychiatric with non-psychiatric variables or lack consistent ordinality. They also require substantial inputs. Despite its limitations, the MoPSI offers some promise in addressing these gaps and could serve as a building block (the “brick”) for a future, superior instrument (the “jade”) or, if nothing else, as inspiration to develop a completely new, future, superior instrument.

5. Conclusions

Researchers and clinicians need more options for assessing psychiatric severity via administrative data. In the present sample, the MoPSI had evidence of construct, concurrent, and predictive validity. Its interval and ordinal properties make it unique among presently available administrative psychiatric measures. Because the JPD version captures the whole of its inputs’ underlying distribution, the scalar is more informative than alternatives based on total variance (i.e., those based on factor analysis). Given its ordinal and interval properties, the MoPSI may be used as either a dependent or independent variable in regression equations and, thus, would be suitable for researchers exploring causes or consequences of severer versus less severe psychiatric illness. It might also be helpful in clinical settings as a case-mix adjuster. However, the MoPSI’s generalizability beyond Veterans who are seeking disability benefits for PTSD is unknown. Future research should address this and other limitations.