Assessing Impacts on Unplanned Hospitalisations of Care Quality and Access Using a Structural Equation Method: With a Case Study of Diabetes

Background: Enhanced quality of care and improved access are central to effective primary care management of long term conditions. However, research evidence is inconclusive in establishing a link between quality of primary care, or access, and adverse outcomes, such as unplanned hospitalisation. Methods: This paper proposes a structural equation model for quality and access as latent variables affecting adverse outcomes, such as unplanned hospitalisations. In a case study application, quality of care (QOC) is defined in relation to diabetes, and the aim is to assess impacts of care quality and access on unplanned hospital admissions for diabetes, while allowing also for socio-economic deprivation, diabetes morbidity, and supply effects. The study involves 90 general practitioner (GP) practices in two London Clinical Commissioning Groups, using clinical quality of care indicators, and patient survey data on perceived access. Results: As a single predictor, quality of care has a significant negative impact on emergency admissions, and this significant effect remains when socio-economic deprivation and morbidity are allowed. In a full structural equation model including access, the probability that QOC negatively impacts on unplanned admissions exceeds 0.9. Furthermore, poor access is linked to deprivation, diminished QOC, and larger list sizes. Conclusions: Using a Bayesian inference methodology, the evidence from the analysis is weighted towards negative impacts of higher primary care quality and improved access on unplanned admissions. The methodology of the paper is potentially applicable to other long term conditions, and relevant when care quality and access cannot be measured directly and are better regarded as latent variables.


Introduction
Improving quality of, and access to, primary care are major strategic foci in transforming primary care in the face of growing demand and the increasing burden of long term conditions [1]. For example, in the English National Health Service (NHS), a primary care incentive scheme known as the Quality Outcomes Framework (or QOF) was introduced in 2004/2005 to promote improved chronic disease management [2]. The impact of this scheme on performance and care quality is measured by a set of indicators within various clinical domains (disease types or groupings).
Although findings are not consistent, quality of primary care may reduce unplanned emergency hospital admissions, which have major cost implications [3][4][5]. There is debate about impacts on adverse outcomes (e.g., premature mortality, avoidable hospitalisations) of quality as measured by the QOF indicators [6,7].
Unplanned admissions may also be reduced by improved access [8]. Access to care can be defined in various ways [9,10]. Utilisation-based measures of access provide a partial picture [11], and many UK studies now focus on patient-reported access to primary care, namely patient experience or perception of access: how easy patients believe it is to access their GP practice [12]. This includes factors such as convenience of GP surgery opening hours and ease of making appointments [13]. Policies to improve access to health care have included schemes to extend operating hours in GP practices, including GP appointments in the evenings or at weekends, measures to reduce inequity in access, and a target that every patient should be able to see a GP within 48 h [14,15].
Perceived access and quality of care are interrelated as higher patient satisfaction and engagement are linked to better self-care and clinical outcomes [16,17]. Indeed patient experience can be conceived as an aspect of quality of care, with the 2008 NHS Next Stage Review identifying patient experience as one dimension of quality, together with clinical effectiveness and patient safety [18]. More specifically in terms of avoidable hospitalisations, recent studies (e.g., [19]) find a negative relation between such hospitalisations and improved access, as measured by responses to the GP Patient Survey.
Impacts of primary quality of care and access on adverse outcomes, such as emergency admissions, may, however, be confounded by socioeconomic differences and health behaviours [20][21][22]. Thus, [23] mention that positive associations between ambulatory sensitive admissions and area deprivation may be related to socio-economic inequalities in health care provision; for example, see [24] regarding diabetes.
Impacts of deprivation on adverse outcomes may be partly mediated by variations in quality of care and access linked to socioeconomic deprivation. Thus, [25] found that poorly performing practices (in terms of diabetes care) tended to have deprived populations, and variations in access have been linked to characteristics of patient populations [11]. Links between access and deprivation may be partly an indirect expression of inequities in the supply of GPs with several studies showing mismatch between GP supply and health need (e.g., [26]).
Impacts of quality of care and access, as measured in regression or other statistical analysis, may also be affected by the methods used to derive summary indices. Quality of care and access cannot be measured by any single indicator and may, instead, be better regarded as latent quantities indirectly measured by a set of manifest observed indicators. Whereas the official method used to obtain quality scores under the QOF scheme does not adjust for overlapping correlation between input variables, multivariate techniques, such as factor analysis and structural equation methods, may seek instead to derive scores adjusted for multicollinearity [27,28].
The particular focus in the present paper is on the measurement of quality of care for diabetes, and assessing impacts of care quality and access on diabetes related emergency admissions. A factor score (latent indicator) method for measuring care quality and perceived access is proposed, with impacts on diabetes related emergency admissions assessed as part of a broader structural equation model (SEM). Manifest indicators of diabetes care (from which the latent indicator is derived) are taken from the Quality Outcomes Framework, while manifest access indicators are from a national survey of patients. A Bayesian inference approach is adopted, using Markov Chain Monte Carlo (MCMC) sampling [29], via the rjags program in R [30,31].

Case Study
A case study involves 90 GP practices in two London Clinical Commissioning Groups (Havering CCG, and Barking and Dagenham CCG), which are agencies coordinating provision of primary care. Patient populations are drawn predominantly from two north London boroughs-Havering, and Barking and Dagenham. Around 96% of the residents of these two boroughs have primary care provided by GP practices affiliated to the two CCGs.
As observed measures of care quality, the study uses four clinical indicators of diabetes care from the Quality Outcomes Framework for the financial year 2013-2014. The choice of indicators (see Table 1) is motivated by relevance in predicting risk of hospitalisation or complication, and also to avoid the duplication present in the official QOF indicators, such as overlapping measures of HbA1c and blood pressure. Regarding blood pressure, cholesterol, and blood glucose control (the first three indicators in Table 1), several studies confirm their importance in preventing complications and hospitalisation [21,32,33]. The National Audit Office [34] (Section 2.9) mentions that "the risk of developing diabetic complications can be minimised by early detection and management of high levels of blood glucose (measured using HbA1c), blood pressure and cholesterol". Regarding influenza immunisation, U.S. data [35] show patients with diabetes are six times more likely to be hospitalized from complications of influenza or pneumonia than those in the general population, while a UK study [36] demonstrated that influenza vaccination significantly reduced hospital admissions by diabetic patients.
As discussed in the Introduction, access to primary care is defined in terms of patient experience or perception of access, as drawn from NHS Patient Surveys of primary care experience and satisfaction. The indicators used in the present study (see Table 1) are from the 2014 patient survey and refer to patients (a) dissatisfied with surgery opening hours; (b) stating that appointments were not convenient; (c) stating their overall experience of the surgery as poor; and (d) reporting that waiting times at surgery were too long.
Studies such as [13] find that access to care may depend on GP supply or GP practice list size. These are incorporated in the analysis as causes of, as opposed to indicators of, varying access-following the multiple indicators-multiple causes terminology in the SEM literature (e.g., [37]).
As well as deriving composite quality and access scores, we seek to assess how far quality of care and access affect emergency hospital admissions by GP practices in 2013/2014, where diabetes is among the contributing diagnoses, either as the primary diagnosis or as the leading subsidiary diagnosis [38,39].
Hospital use may be affected by morbidity and demographic factors, including variations in socio-economic status [40]. Levels of diabetes morbidity are measured by practice level age standardised rates for diagnosed diabetes. Regarding socio-economic conditions, the study region shows wide differences: 10 of its 260 neighbourhoods (Census lower super output areas, abbreviated as LSOAs) are in the most affluent decile of such neighbourhoods across England, while at the other extreme, 13 neighbourhoods are among the most deprived 10% of all English neighbourhoods.

Methods
Some studies have used official QOF achievement scores (intended to measure quality of care) in assessing impacts on patient outcomes, for example in regression analysis of emergency admission rates [41,42]. However, these scores do not adjust for overlapping correlations between the input indicators used to obtain overall achievement scores within clinical domains such as diabetes. The official scores also take percentage indicators of achievement as known without error, regardless of the size of the patient population denominator (which may vary considerably between indicators and GP practices) and, therefore, do not allow for sampling variability (i.e., varying precision) in the percentage indicators based on binomial data.

Quality Scores in a Structural Equation Model
We wish to take account of sampling variability and control for correlations between measured input indicators, and obtain a composite measure of care quality with loadings on input indicators which are optimal in relation to potential outcomes (e.g., emergency admissions). For example, some indicators may have greater impact on emergency admissions.
Accordingly we define a structural equation model with a measurement model in which quality of care as a latent variable, denoted F, is measured by a set of observed input indicators Z, and an outcome model relating diabetes related emergency admissions Y to quality of care F (see Appendix A for a formal development). There are I = 90 GP practices and J = 4 manifest (observed) clinical indicators, Z ij (i = 1, . . . , I; j = 1, . . . , J), to define the latent quality scale. These are in the form of patient totals achieving a given performance target, and are from the Quality Outcomes Framework for 2013-2014.
We wish to represent stochastic uncertainty in the clinical indicators. Thus, let Z ij denote the number of diabetic patients in practice i for whom a particular clinical threshold is attained (e.g., diabetic patients with blood pressure reading below 150/90 mmHg), and N ij denote the relevant patient denominator (e.g., number of registered diabetes patients). A binomial likelihood model with unknown performance attainment probabilities is adopted for the observations (Z ij , N ij ) (see Appendix A), with the likelihood allowing for overdispersion.
It is assumed that variability in performance attainment probabilities is explained by normally-distributed latent quality scores F i . Regarding these scores, alternative distribution assumptions might be considered, for example, assuming F i to be Student's t-distributed rather than as normal. This might be relevant if there were distinct outlier practices with unduly high/low QOC or access.
The goal is not only to summarize quality of care in a composite index, but assess how far quality of care affects adverse hospital outcomes. In principle there may be more than one such outcome, but here there is a single outcome, the total of diabetes related emergency admissions, Y i . In the baseline model (model 1) relative risks of emergency admission ν i are predicted from a log-link regression including only quality of care: As F i is a positive measure of quality, one would expect γ 1 to be significantly negative if high QOC significantly reduces unplanned admissions. From the Bayesian estimation process, one can the estimated probability Pr(γ 1 < 0|Y,Z) that the coefficient is negative. This provides an alternative method of assessing the direction of effect to the 95% credible interval [43].
A second model (model 2) allows for the impact on emergency admissions of a score measuring area socioeconomic deprivation, D i . This is the 2015 index of income deprivation [44], calculated for GP practice populations. This is used instead of the main Index of Multiple Deprivation (IMD), which includes standardised emergency admission rates in the constituent variables [40].
A subsidiary analysis, not reported on in detail, assessed sensitivity to deprivation impacts of using the employment deprivation score [44] rather than income deprivation, throughout models 2 and 3, as considered below. This shows a slight diminution in goodness of fit as the employment domain had a lower impact on emergency diabetes admissions.
We further consider the impact on emergency admissions of diabetes morbidity (M i ), namely, the age-standardised rate at mid-2013 of diagnosed diabetes-see Equation (A8). Model 2 also extends the model for the quality scores so that deprivation can also potentially affect quality of care, as indicated by several studies [45][46][47].

Access to Care
As mentioned above, unplanned hospitalisation may also be linked to various aspects of access to primary care. In common with a number of UK recent studies, perceived patient access to primary care is based on indicators from the GP Patient Survey [48]. Just as for quality of care, access is conceived as a latent quality, not measurable by any single indicator.
Thus, the structural equation model (model 3) now has a measurement model in which quality of care as a latent variable F is measured by a set of observed input indicators Z, another measurement model in which poor access is a latent variable G measured by a set of observed indicators W, and an outcome model relating diabetes related emergency admissions Y to quality of care F and access G.
Let W ik (k = 1, . . . , K) be patient totals rating access as poor, and M ik be denominator populations (i.e., all patients surveyed, regardless of their rating). The indicators used are as in Table 1. Since access and quality of care may potentially be interrelated [13], it is assumed that F i and G i are bivariate normal, with covariance ∑ and correlation ρ.
Furthermore, we allow care quality and care access to both depend on deprivation (e.g., [49]), with respective coefficients β 1 and β 2 . Access to care may also depend on GP supply, namely full time equivalent GPs per 1000 patients, S 1i , and GP practice list size, S 2i [13]. These are included as additional potential causal influences on access, with respective coefficients β 3 and β 4 ; see Equation (A20).

Analysis and Results
Analyses are carried out using the rjags package in R, with detailed assumptions, model checks and fit measures described in Appendix B. Table 2 shows satisfactory predictive checks for all three models, all between 0.1 and 0.9. The WAIC fit criteria (Appendix B) show a gain in moving from model 1 to model 2, but no change in fit between models 2 and 3. Table 3 shows estimated regression coefficients under the three models for the emergency admissions regression. Table 4 sets out summaries for λ j and κ k , the loadings in the measurement model(s) for care quality and care access, together with β coefficients relating these constructs to deprivation and supply variables.   Table 3 shows that under model 1, the impact of care quality on emergency admissions has 95% credible interval (−0.50, 0.01) [50] (p. 1063). casts doubt on converting such an interval (which just straddles zero) into evidence for a null effect. Under a Bayesian approach, whether quality has a negative impact can be assessed via the posterior probability Pr(γ 1 < 0|Y,Z) [43] (Chapter 5). We find that there is a 97% probability that the coefficient is negative.

Regression Findings
The measurement model coefficients (Table 4) for model 1 show broadly similar relevance of each of the observed indicators (see Table 1) in defining quality of care. Posterior means on the loadings λ j range from 0.59 to 1.29.
Under model 2, the impacts of deprivation and morbidity on emergency admissions are both highly significant. This model also allows part of the impact of deprivation to be indirect, in that quality of care depends on deprivation; see Equation (A12) in Appendix A. In fact, this indirect effect is negative, as expected, but not pronounced: we find a probability of 0.73 that Pr(β 1 < 0|Y,Z), where a negative coefficient β 1 means that higher practice deprivation is associated with lower quality of care.
The impact of care quality itself on emergency admissions remains significant in model 2, as assessed by a probability Pr(γ 1 < 0|Y,Z) of 0.974. Figure 1 plots the posterior means (under model 2) of the quality score F i against those for emergency admission relative risk ν i , together with a LOWESS plot (locally weighted scatterplot smoothing). This plot suggests a threshold effect, with greater impacts of high care quality in reducing emergency admissions.  Model 3, the full structural model, involves access as well as quality of care. Models 2 and 3 have very similar fit measures, and statistical criteria may be supplemented by substantive considerations, since on the basis of accumulated evidence from other studies, model 3 may be regarded as providing a more complete description of the interrelated processes. Regarding the measurement model for access, the highest loading κ k (with posterior mean approaching 2) is for the variable overall poor experience of the surgery.
Of particular interest in the estimates for model 3 are positive impacts of deprivation on perceived poor access to care (the β 2 coefficient), and a negative correlation ρ between poor access and quality of care. The respective 95% credible intervals are (0.23, 0.68) and (−0.61, −0.12). There is also a significant positive impact on poor access of larger GP list sizes (the population total served by a GP practice), and a mostly negative impact of GP supply, with 95% interval (−0.82, 0.07). A list size effect on access (i.e., perceived access better in small practices) is also reported by [13].
The impact of poor access on emergency admissions has a posterior density concentrated on positive values, with a 95% credible interval (−0.10, 0.46), and a 90% probability of a positive effect. Figure 2 plots the posterior means of the poor access score G i against those for emergency admission relative risk ν i , with LOWESS smooth included.  It is of interest to assess how the QOC scores (the Fi) compare with the official diabetes QOF attainment scores. The correlation between the posterior mean F from model 3 and the QOF attainment scores is 0.84, and their interrelationship is shown in Figure 3. It is apparent that the modelled QOC scores provide extra discrimination in measuring quality of care at the highest levels of the official attainment score, namely, 23 practices with official attainment scores between 105 and the maximum possible 107. It is of interest to assess how the QOC scores (the F i ) compare with the official diabetes QOF attainment scores. The correlation between the posterior mean F from model 3 and the QOF attainment scores is 0.84, and their interrelationship is shown in Figure 3. It is apparent that the modelled QOC scores provide extra discrimination in measuring quality of care at the highest levels of the official attainment score, namely, 23 practices with official attainment scores between 105 and the maximum possible 107.

Sensitivity Analysis
The above analysis may be affected by distributional assumptions. The derivation of latent factors for care quality and access is central to the proposed approach, and so effects were assessed of adopting a Student's t distribution of factor scores rather than a normal density, as this heavier tailed density may be more robust to any outlier practices with unduly high/low quality of care or

Sensitivity Analysis
The above analysis may be affected by distributional assumptions. The derivation of latent factors for care quality and access is central to the proposed approach, and so effects were assessed of adopting a Student's t distribution of factor scores rather than a normal density, as this heavier tailed density may be more robust to any outlier practices with unduly high/low quality of care or access.
A bivariate Student's t density for the factor scores in model 3 was achieved using a scale mixture approach [51] (p. 138), with a default degrees of freedom set at four, as advocated by [52] (p. 449). This provides a fourth model, results from which are also included in Tables 2-4. This shows no marked change in fit, but slightly enhances the impact of quality of care on emergency admissions, with a 93% probability that the impact is negative.

Implications for Geographic Differences in Access
As noted above, around 96% of residents in the case study region have their primary care provided by GP practices in the two CCGs (which have the same names as the boroughs, namely Barking and Dagenham CCG, and Havering CCG). Hence, GP practice differences in access and care quality (measured by the G and F scores) can be translated into implied geographic differences in access and care quality. These are obtained using cross-reference files of populations classified both by LSOA and by GP practice (hence of dimension 260 by 90). LSOA weighted averages for access or care quality are based on proportions of each neighbourhood population cared for by each of the 90 GP practices. Figure 4 accordingly maps out access scores for the 260 neighbourhoods (LSOAs) using the posterior mean G scores for GP practices, and shows clear geographic clustering of poor access. An accompanying map ( Figure 5) shows LSOA scores for income deprivation, and associations between the geographic patterns of deprivation and poor access are apparent.

Conclusions
This paper has developed a structural equation model for quality of primary care and perceived access to primary care, and assessed their impacts on unplanned hospital admissions for diabetes, Score under 0.09 0.09 to 0.14 0.14 to 0.21 0.21 to 0.27 over 0.27 Figure 5. LSOA income deprivation scores.

Conclusions
This paper has developed a structural equation model for quality of primary care and perceived access to primary care, and assessed their impacts on unplanned hospital admissions for diabetes, while allowing also for socio-economic deprivation, morbidity, and supply indicators. This methodology is potentially applicable to other long term conditions, and relevant when care quality and access cannot be measured directly and are better regarded as latent variables. Methodology for measuring quality of care and access may affect findings regarding their impacts on outcomes, such as unplanned admissions.
In that regard, a number of studies report insignificant or negligible effects of quality of care on adverse hospital outcomes, such as avoidable or ambulatory sensitive hospital admissions. For example [22] report small associations between official QOF scores and emergency admissions, whilst impacts of socio-economic deprivation were much stronger. [40] also obtain a strong effect of deprivation on potentially avoidable emergency admissions, and mention that this is in part because deprivation is correlated with morbidity.
In the case study region of the current paper, primary care population register data for diagnosed diabetes (standardized prevalence rates in 2013) show wide variation between GP practices, and there is a 0.70 correlation between diabetes prevalence and income deprivation. This inter-correlation is controlled for in models 2 and 3, but a strong deprivation effect remains.
The analysis also suggests a strong indirect effect of deprivation on emergency admissions through access, though not a strong indirect impact through quality of care. The positive association between income deprivation and perceived poor access under model 3 can be represented in terms of implied neighbourhood access variations (as shown by Figure 4). This pattern is consistent with evidence that care access in deprived, as compared to affluent areas, may be related to variations in primary care provision, especially provision that matches health care need, with continuing evidence of an inverse care law [53,54].
Some potential extensions of the model framework and analysis of this paper may be proposed. Thus, an additional dimension that has been shown in the literature to have a substantial effect on quality of care is continuity of care, "in the sense of a patient repeatedly consulting the same doctor and forming a therapeutic relationship" [55]. For example, [56] in a Korean study derive different indices of continuity from detailed patient consultation histories. One could potentially form a composite index of continuity based on combining different indices, but a UK application is impeded by non-availability of patient primary care consultation data.
Interpretation of quality and access effects on adverse outcomes has preoccupied several recent studies. After accounting for the impact of deprivation and morbidity in model 2-which fits as well as the full model to the hospitalisation and quality indicators-the effect of quality of care on unplanned admissions remains significant: a 97% probability for a negative impact. For the full model, quality of care has a 91.8% probability of a negative impact, and poor access has a 90% probability of a positive impact. The latter two impacts are increased slightly under model 4 with a bivariate Student's t model for correlated quality and access.
These probabilities may be interpreted from a Bayesian perspective in terms of marginal Bayes factors [57]. The prior on the regression coefficients is neutral with regard to the direction (positive or negative) of the coefficient, so a 91.8% posterior probability of a negative impact of quality on emergency admissions implies a marginal Bayes factor of 11.2. Using standard guidelines [58], this counts as strong evidence of an effect.
In such terms, the analysis of the present study is not consistent with a null finding regarding impacts of quality of care and access on potentially avoidable emergency admissions. Rather the regression evidence is weighted towards negative impacts of higher primary care quality and improved access on unplanned admissions. Other aspects of the model are also important in healthcare terms, such as poor access being greater for deprived practice populations.

Conflicts of Interest:
The author declares no conflict of interest.

Appendix A. Formal Aspects of Statistical Models
For quality of care indices, let Z ij denote the number of diabetic patients in practice i for whom a particular clinical threshold is attained (e.g., diabetic patients with blood pressure reading below 150/90 mmHg), and N ij denote the relevant patient denominator (e.g., number of registered diabetes patients). One has binomial sampling: with performance attainment probabilities π ij that are unknown parameters to be estimated. In the simplest model (model 1) it is assumed that variability in attainment probabilities π ij is explained by a normally-distributed common quality factor F i , and random residual effects u ij , accounting for over-dispersion, which are also normally distributed. Thus with a logit link, and intercepts σ j , one has: For identifiability, the loading λ 1 = 1 and so the variance θ of the factor scores is an unknown [59]. The remaining loadings are constrained to be positive in line with a quality of care interpretation for the F scores, and for computational stability, to avoid label switching during MCMC updating.
In model 1, hospitalisation risks are also related to the latent QOC score. Denoting observed emergency admissions as Y i and expected admissions as E i , it is assumed that the Y i are negative binomial, with overdispersion parameter α, Y i~N B(E i ν i ,α), and likelihood: where µ i = E i ν i are predicted emergency admission totals, and conditional variances are var(Y i ) = µ i + αµ 2 i . Relative risks of emergency admission ν i are predicted from a log-link regression including only quality of care: Model 1 is a joint likelihood over Y and Z data, represented by Equations (A1)-(A6).
In model 2, we additionally consider the impacts on emergency admissions of a socioeconomic deprivation index, D i , and also of diabetes morbidity (M i ). Thus: Model 2 also allows for an indirect impact of deprivation on hospitalisations, via quality of care. This leads to a multiple indicator, multiple cause form of structural equation model. Thus with deprivation scores D i , centred for identifiability, one has: The joint likelihood under the second model is represented by Equations (A7)-(A12). In model 3, the latent variable model encompasses both poor access (denoted G) and quality of care (denoted F). Then: with N 2 denoting a bivariate normal density. Causes of poor access are deprivation and supply indicators, as in (A20). For identifiability, κ 1 = 1, with remaining κ loadings constrained to be positive in line with a poor access interpretation for the G scores. The model for emergency admissions includes access, as well as deprivation and morbidity, in view of evidence such as [8]: The joint likelihood under the third model is defined by Equations (A13)-(A22). Appendix C contains the code for the bivariate normal model 3.
A bivariate Student t density for quality and access factors is obtained by a scale mixture, namely: where δ is the Student's t degrees of freedom. Following [52] (p. 449), a robust analysis is obtained by taking a preset degrees of freedom δ = 4.

Appendix B. Prior Densities and Fit Measures
We assume gamma priors, with shape 1 and index 0.01, on inverse variance parameters (1/ψ j , 1/ϑ k , 1/θ) and the negative binomial parameter 1/α, and Normal priors with mean zero and precision 0.001 on fixed effects (such as regression intercepts and slopes). The precision matrix ∑ in model 3 is assigned a Wishart prior, with identity scale matrix and two degrees of freedom. The unknown loadings λ j and κ k in the factor score models are assigned exponential priors with mean 1. Estimates are based on the second halves of two chain runs of 20,000 iterations, with convergence assessed using Brooks-Gelman-Rubin diagnostics [60].
Fit is assessed using the WAIC information criterion [61], with WAIC obtained only on the Y and Z data in models 1 and model 2, and for the Y, Z, and W data in model 3. Posterior predictive checks [62,63] are also applied. For the emergency admissions model, these are based on predicted emergency admissions Y new,i sampled from the relevant posterior predictive density, namely p(Y new |Y,Z) in models 1 and 2, and p(Y new |Y,Z,W) in models 3 and 4. Posterior predictive checks for the clinical and access indicators are based on predicted clinical responses Z new , and predicted access indicators W new .
With R and R new denoting fit measures using observations and predictions respectively, posterior predictive p-values are estimated as Pr(R new > R|Y,Z) or Pr(R new > R|Y,Z,W), the proportion of iterations where R new > R. Extreme p-values (under 0.05 or over 0.95) indicate model discrepancies. A chi-square fit measure is used, with R Y = ∑ i (Y i −µ i ) 2 /µ i , and R Y,new = ∑ i (Y new,i −µ i ) 2 /µ i for the Y data.

Appendix C
The core of the rjags code for the bivariate normal factor model 3 is as follows, with terms to extract WAIC elements and predictive checks excluded for brevity.