Development of the PREMIUM Computerized Adaptive Testing for Measuring the Quality of Information Delivered to Patients with Severe Mental Illnesses

Measuring the quality of information delivered to patients with severe mental illness (SMI), i.e., schizophrenia, bipolar disorders, and major depressive disorders, is essential to improve their quality of care. In this work, we described the different steps of the validation of the PREMIUM computerized adaptive testing (CAT) for measuring the quality of information delivered to patients with SMI. The PREMIUM item bank regarding information included 25 items. A total of 499 patients with schizophrenia (53%), bipolar disorders (26%), and major depressive disorders (22%) were recruited from numerous in- and outpatient settings in France. Unidimensionality, local independence, and monotonicity were verified for 19 items of the item bank. The psychometric properties were satisfactory for both internal (RMSEA = 0.069, CFI = 0.969, TLI = 0.963) and external validity (in particular, significant associations were found with age, educational level, and social functioning). The CAT exhibited satisfactory accuracy and precision (standard error of measurement <0.55 and root mean square error <0.3), with an average administration of eight items. This CAT may be used by healthcare professionals in psychiatric settings to accurately assess the patients’ experience with information.


Introduction
Schizophrenia, bipolar disorders, and major depressive disorders are severe mental illnesses (SMI) that are characterized by poor quality of care [1][2][3][4]. Previous studies have reported substantial unmet information needs in patients with SMI [5,6]. For example, the need for information has been reported by more than half of patients with schizophrenia in a recent Brazilian study [7]. Sharing information is particularly challenging in patients with SMI because the nature of mental illness is often difficult to explain, and the treatment options and prognosis may vary considerably [8]. Yet, obtaining information about illness, available treatment and care options, prognosis, are strategies for self-management is an essential issue regarding the quality of care because it can affect the process of shared decision making (i.e., "a decision support model allowing information sharing between doctors and patients concerning different care options and taking into account patient preferences") [9,10] and treatment adherence [11], leading to improved prognosis, including the patients' quality of life [8]. Poor adherence to treatment may be partly explained by a patient's belief that the effect of medications is harmful or that medications are unnecessary [12]. It is therefore essential to provide understandable and appropriate information to patients to improve their treatment adherence [13], which remains a major public health problem.
Measuring the quality of information delivered to patients with SMI is an essential step to improve communication strategies for patients with SMI, as well as their quality of 2 of 13 care. The French group Patient-Reported Experience Measure for Improving qUality of care in Mental health (PREMIUM) has developed computerized adaptive tests (CATs) for seven domains of quality of care experienced by patients with SMI [14,15]: interpersonal relationships, care environment, drug therapy, access and care coordination, respect and dignity, psychological care, and information. The patient-reported experience measures (PREMs) developed by PREMIUM are expected to optimize measurement precision, to reduce the length of the questionnaire and completion time, and to improve the relevance and the acceptability of these questionnaires in psychiatric settings [16].
In this work, we described the different steps that led to the validation of the PRE-MIUM CAT for measuring the quality of information delivered to patients with SMI.

Study Setting and Population
We performed a national, multicenter, cross-sectional study between January 2016 and December 2021. Patients were recruited from in-and outpatient psychiatric settings in a French teaching hospital (Assistance Publique-Hôpitaux de Marseille), in the 39 FondaMental academic center of expertise for schizophrenia, bipolar disorders and depression [17] and through social networks, including patients' associations and with an online web survey. The study protocol was described in more detail elsewhere [14]. All participants gave their informed consent, and the study was carried out in accordance with the Declaration of Helsinki and approved by the local ethics committee (authorization number: 2014-A01152-45).
The inclusion criteria were patients over the age of 18 and under the age of 65, with a DSM-5 diagnosis of schizophrenia, bipolar disorders, or major depressive disorders [18] and having in-or outpatient psychiatric care, informed consent, and speaking/reading French.

Items Measuring the Quality of Information Delivered to Patients with SMI (PREMIUM Item Bank Regarding Information)
The PREMIUM item bank regarding information included 25 items that were selfadministered to patients with SMI. The development of the items was detailed in previous works and is based on a mixed method approach associating qualitative and quantitative methodologies [14,15]. All items are presented in Table 1 and scored on a 5-point Likert scale ("strongly disagree", "disagree", "neither agree nor disagree", "agree", "strongly agree") with a "not applicable" response option. In addition to these 25 items, we collected an overall satisfaction item ("Overall, were you satisfied with the information about your illness and your management?") and a visual analogue scale (VAS) (0 minimum to 10 maximum).

Collected Data
The following sociodemographic and clinical data were collected: sex, age, educational level, marital status, occupational status, diagnosis (schizophrenia, bipolar disorders, or major depressive disorders), duration of illness, the Global Assessment of Functioning scale [19] (GAF, ranging from 0 to 100, with a higher score indicating better functioning), quality of life (QoL) using the medical outcome study 12-item Short Form (SF-12) [20], which describes eight dimensions-physical functioning, social functioning, physical role, emotional role, mental health, vitality, bodily pain, general health-and two composite scores for physical and mental quality of life (ranging from 0 to 100, with higher scores indicating a better quality of life).

Statistical Analysis
Descriptive statistics of the sample included frequencies and percentages of categorical variables and the means and standard deviations of continuous variables.
All the items of the PREMIUM item bank regarding information were analyzed using descriptive indicators to select the high performing items.
The item response theory (IRT) assumptions were assessed [21]: unidimensionality using confirmatory factor analysis (CFA) or exploratory factor analysis (EFA), followed by bifactor structure (i.e., one general factor and several group factors), local independence using residual correlations from the CFA model, and monotonicity using visual inspection of characteristic item curves. The generalized partial credit model (GPCM) was used to calibrate the responses to the items [22] after having verified its superiority over other models, such as the partial credit model (PCM) [23]. Item parameters (discrimination and thresholds) were estimated using the maximum marginal likelihood estimation (MMLE) and the expectation-maximization (EM) algorithm [24], and then the model fit to the data was evaluated.
Convergent validity was investigated by examining the relationships between item bank scores and psychosocial functioning (GAF), quality of life (SF-12), and satisfaction (overall satisfaction item and VAS). Discriminant validity was investigated by comparing the mean scores of the item bank scores, according to sociodemographic and clinical characteristics, using t tests, analysis of variance (ANOVA), and Pearson's correlation coefficients.
Finally, we developed the item administration algorithm of the CAT. The starting item was the most informative item based on the maximum Fisher information (MFI) criterion [46]. The CAT algorithm then selected the next item with the highest information for the current θ estimate. The CAT algorithm ended when the stopping rule based on the standard error of measurement (SEM) was reached (acceptable range between 0.33 to 0.55 [47]). Three scenarios with different stopping rules corresponding to SEM values of 0.33, 0.44, and 0.55 were simulated and compared to select the best adaptive administration algorithm. The indicators used to assess the performance of the results at each stage of the statistical analyses are provided in Supplementary Table S1 [28][29][30][31][32][33][34][35][36][37][38][39][40][41][42][43][44][45].

Sample Characteristics
A total of 499 patients with schizophrenia (53%), bipolar disorders (26%) and major depressive disorders (22%) were recruited. The characteristics of the participants are presented in Table 2.

Descriptive Indicators of the PREMIUM Item Bank Regarding Information
All the indicators are presented in Supplementary Table S2 and were satisfactory, except that six items (i.e., I2, I4, I10, I15, I17, and I21) exhibited inter-item correlations that were too high (>0.70), reflecting redundancy between items, and they were thus removed. The lowest scores were reported for item 24 (information about the possibility to access your medical file), item 21 (information about the possibilities of arranging your working time) and item 25 (information about associations and self-help groups.

Assumptions Assessment and Calibration of the IRT Model
Unidimensionality, local independence, and monotonicity were inspected for the remaining 19 items. The fit indices of the 1-factor CFA model were not adequate (RMSEA = 0.126, 95% CI [0.117-0.135], CFI = 0.911, and TLI = 0.900) and were improved with a bifactor structure using one general factor and two group factors (RMSEA = 0.076, 95% CI [0.065-0.086], CFI = 0.975, TLI = 0.968). In the EFA, three factors had eigenvalues greater than 1 (9.0, 1.5, and 1.2, respectively), and the ratio of the first to second eigenvalue was 5.9. The scree plot revealed two predominant factors, while the parallel analysis revealed four predominant factors. All items had higher factor loadings for the general factor than for the group factors. The coefficient ωh for the general factor was 0.93, and those on the two group factors were 0.03 and 0.21, respectively. The percentage of ECV attributable to the general factor was 85.4%, whereas the remaining 14.6% was attributable to the two group factors (7.3% and 10.6%, respectively). All items in the bank were then recoded after examination of the item characteristic curves, which led to an improved model fit (Akaike information criterion, AIC = −8726.88 and Bayes information criterion, BIC = −8886.95). Finally, all residuals were lower than 0.20, and the Cronbach's alpha was 0.94.

Calibration and Fitting an IRT Model
The GPCM showed a better fit to the data compared to PCM (14, The test information curve of the item bank confirmed that the items have a high measurement precision over a broad spectrum of the latent trait (61.0% of total information is included in the [−2,2] range of the latent continuum values) (Figure 1). Item 13 was the most informative of the bank-"the course of your care (how to take your drug therapy, the frequency of your appointments, etc.", whereas item 20 was the least informative-"existing medical-social aid (reimbursement of health care costs, access to home help, obtaining the disabled adult allowance, etc.").

Calibration and Fitting an IRT Model
The GPCM showed a better fit to the data compared to PCM (14,433
The test information curve of the item bank confirmed that the items have a h measurement precision over a broad spectrum of the latent trait (61.0% of total in mation is included in the [−2,2] range of the latent continuum values) (Figure 1). Item was the most informative of the bank-"the course of your care (how to take your d therapy, the frequency of your appointments, etc.," whereas item 20 was the least infor tive-"existing medical-social aid (reimbursement of health care costs, access to h help, obtaining the disabled adult allowance, etc.").

External Validity of the PREMIUM Item Bank Regarding Information
The mean score was 46.97 ± 19.82. The scores were strongly correlated with ov satisfaction and the corresponding VAS, confirming the convergent validity. All the 12 dimensions were significantly correlated with information experience, but the relat ships were stronger regarding social and emotional functioning.
Concerning discriminant validity, younger age, higher educational level, and on survey were associated with poorer information experience. No significant differe were found by sex, marital status, employment status, care setting, diagnosis, duratio illness, and psychosocial functioning. The results regarding the external validation presented in Table 3.

External Validity of the PREMIUM Item Bank Regarding Information
The mean score was 46.97 ± 19.82. The scores were strongly correlated with overall satisfaction and the corresponding VAS, confirming the convergent validity. All the SF-12 dimensions were significantly correlated with information experience, but the relationships were stronger regarding social and emotional functioning.
Concerning discriminant validity, younger age, higher educational level, and online survey were associated with poorer information experience. No significant differences were found by sex, marital status, employment status, care setting, diagnosis, duration of illness, and psychosocial functioning. The results regarding the external validation are presented in Table 3.

Elaboration of the Item Administration Algorithm
Among the three scenarios tested, the CAT simulation with a level of precision of SEM < 0.33 was the most efficient, with the highest levels of accuracy (r = 0.97) and precision (RMSE = 0.23) while administering eight items, on average. Table 4 provides the results of the CAT simulations.

Discussion
In this study, we demonstrated the validity of the PREMIUM-CAT for measuring the quality of information delivered to patients with SMI. This CAT presents interesting characteristics for widespread use in psychiatric settings.
The PREMIUM item bank has been developed based on a literature review and patient interviews, ensuring its content validity [15,55]. Similar to a previous study on information needs in mental health [56], the PREMIUM item bank regarding information explores a broad facet of the information delivered to patients, including: basic facts (e.g., diagnosis, symptoms, etc.), treatment (e.g., side effects, etc.), coping (e.g., lifestyle advice, etc.), medical systems (e.g., possibility of accessing medical file, etc.), social aids, professional accommodations, and peer support. The basic facts are the information needs most frequently reported in the literature in contrast to self-help and peer support [56]. These findings are corroborated by our study in which the item regarding the course of care (I13) was the most informative of the bank, while the item on social aids (I20) was the least informative of the bank. Measurement invariance was demonstrated by the few indications of negligible magnitude DIF, which implies that the psychometric properties of the items are invariant across gender, age, care setting, and diagnosis, and that the item bank scores can be compared between respondents, regardless of their characteristics. The different sampling strategies may also have impacted participants' information experience at different levels. Response bias is known to affect self-reported data, leading to results biased in a positive direction, as in traditional satisfaction surveys [57]. Participants recruited from healthcare facilities are more likely to provide biased responses due to concerns about the impact on their care and relationships with the healthcare team. Participants recruited from healthcare were invited to participate in the study by a psychologist external to the healthcare team in order to limit this bias. However, the scores were significantly higher for participants from the healthcare facilities than for participants from the online survey. External validity, explored by the use of socio-demographic characteristics and established psychiatric and functioning measures, globally confirmed our assumptions. In particular, we confirmed that information experience was slightly and positively correlated to quality of life, which has been shown in other chronic severe illnesses, such as cancer [58]. Conversely, our results showed no statistical difference in information experience according to psychosocial functioning, even though most participants had poor psychosocial functioning. Our results also showed that participants with high educational level and young age reported a poorer information experience. Patients' preference about information may vary according to age and education [59]. Health literacy and internet use may also explain these results, in part [60][61][62][63]. Innovative information strategies could be implemented in novel digital adherence-assessment devices [64] and telemedicine interventions [65]. Finally, the item bank offers better precision and accuracy measurement over the [−2;2] range of theta values, while reducing response burden on participants, with an average of 8 items administered.
The analysis of the item scores has yielded three points of information that should be targeted with high priority. First, participants reported poor information experience on the right to access her/his medical file. All physicians are informed during their training of the 2003 Kouchner Law obliging institutions to provide medical records within 8 days to any patient on demand. The reasons for this poorer experience should be explored in future studies. However, it seems necessary to reinforce the information of this right to all patients, perhaps by automatic mailing or phone texting, for instance. Second, participants reported poor information experience regarding social aids and professional accommodations. This information is not directly related to medical care, but rather to medico-social/social/rehabilitation care and occupational medicine. Poor access to medicosocial and social care and lack of knowledge regarding these services may explain why patients report poorer information experience regarding this information. All patients could be delivered systematic written information regarding social aids and professional accommodations when receiving her/his first diagnosis. Third, participants reported poorer information experience regarding associations or programs aimed at helping patients. This poor experience may be due to a lack of knowledge. Physicians are themselves poorly informed about all associations, programs, and initiatives concerning all mental illnesses. This information also may vary across time and geographical areas. Caregivers may also estimate that their role is limited to medical care. Future qualitative studies should explore the reasons obehind this poorer experience and how to address this issue.

Futures Directions
The item bank and its CAT version provide a brief, accurate, and flexible assessment of the information experience of patients with SMI, suitable for research and clinical practice. The systematic and continuous collection of PREMs through a digital platform has great potential to inform quality improvement actions in mental health care [66]. However, the CAT version requires an appropriate information technology (IT) infrastructure to collect data, at the hospital and/or at home, requiring that patients have internet access. Future dissemination and implementation work will need to determine the feasibility of such a device for routine use and its effectiveness in meeting specific research or clinical needs.

Limitations
Our sample size is large compared to traditional psychiatric studies, but may be limited in the field of psychometric studies. However, our sample size was sufficient, since sample size requirements recommend approximately 500 observations for accurate parameter estimates with multiparameter models (e.g., GPCM) [67,68]. In addition, the sample included a diverse patient population, both inpatient and outpatient, from several facilities in different geographic regions of the country. Future work with a larger and more diverse sample will allow for additional psychometric testing to develop a better understanding of the metric properties, but also to examine the generalizability of this item bank to other populations, such as patients with common mental disorders (e.g., anxiety disorders). Second, in this study, as in previous PREMIUM studies [12,69], we used the GPCM, which showed an adequate fit to the data and whose fit can be compared to the PCM because these are nested models. The graded response model (GRM), which tends to provide similar results, could have been an alternative (e.g., in case of non-convergence) [70]. Third, criterion validity could not be assessed because, to the best of our knowledge, no gold standard was available. In addition, given the cross-sectional study design, responsiveness could not be studied. Fourth, CAT precision depends on the quality of the related item bank, which showed the highest measurement precision in the [−2;2] range of theta values. Future studies should evaluate whether additional items are needed to cover the ends of the latent continuum. Fifth, we did not evaluate health literacy, which may have influenced information experience [71]. Likewise, symptom severity was not measured using psychiatric scales (e.g., PANSS, MADRS, YMRS) due to diagnostic heterogeneity and different recruitment strategies that did not allow for hetero-assessment. By using the GAF, which provides an approximate measure of the severity of these symptoms, we examined the impact of the severity of mental illness on the information experience. Future studies should address these questions. Finally, the precision and accuracy of the CAT should be re-evaluated in an independent sample and under real-world conditions to assess whether the responses may be affected by other factors [72].

Conclusions
The PREMIUM item bank regarding information and its CAT version may be used by healthcare professionals in psychiatric settings to accurately assess the experience of patients with information. Particular attention should be paid to informing patients about their right to access their medical files, social aids, professional accommodations, and peer support services.
Supplementary Materials: The following supporting information can be downloaded at: https:// www.mdpi.com/article/10.3390/jcm11226687/s1, Table S1: Indicators of psychometric performance; Table S2: Descriptive statistics of the item bank; Table S3: Parameter estimates (discrimination and threshold) and fit statistics for the 19 items in the final item bank; Table S4: DIF results. Funding: This work was supported by an institutional grant from The French National Program on the Performance of the Health Care System (PREPS, financed by Direction Générale de l'Offre de Soins, 14, avenue Duquesne, 75350 Paris, France) and the Agence Technique de l'Information sur l'Hospitalisation (ATIH). The sponsors had no role in the study design; collection, analysis, and interpretation of data; report writing; or the decision to submit the article for publication.

Institutional Review Board Statement:
The trial registration is NCT02491866. The assessment protocol was approved by the relevant ethical review board (CPP-Sud Méditerranée V, n • 2014-A01152-45). All data were collected anonymously.
Informed Consent Statement: Informed consent was given by all participants.

Data Availability Statement:
The data are available on demand from the PREMIUM Scientific Committee.