A Bayes Inference for Ordinal Response with Latent Variable Approach

Abstract

In this paper, we propose a Bayesian model for the analysis of categorical data with an ordered outcome. The method provides a latent variable approach with an informative prior transformed from a Dirichlet distribution for the boundary parameters. A simulation study is carried out to assess the performance of the methods under various settings of the data structure. Our method produces predictive accuracy over the conventional classification procedures. Real data are analyzed to demonstrate the efficiency of the proposed method.

1. Introduction

Over the past few decades, modeling and predicting of ordinal outcomes have become essential to study in various fields especially in social and economic sciences where the natural ordering data very commonly appear. For example, socioeconomic status is typically broken into three levels (high, middle, and low) to describe the three places into which a family or an individual may fall. The level of education consists of high school, bachelor’s, master’s, and doctoral degrees. These levels or classes can be viewed as ordinal variables, but with no scale or magnitude available between each order or category. Many such ordinal data and numerous methods of analysis have been introduced and discussed by a number of researchers: [1,2,3], just to name a few. One well-known method is polytomous ordinal logistic regression (POLR) or the cumulative logit model initially proposed by [4], later called the proportional odds model [2] since the same proportionality constant applies for all cumulative logits. For a multinomial response variable Z with possible J ordered categorical outcomes and the associated p-dimensional vector of covariates $\mathit{x}$, the cumulative probability for Z on $\mathit{x}$ is given by:

$$\begin{array}{c}\hfill P(Z\le j|\mathit{x})=\frac{\exp ({\alpha }_j+{\mathit{x}}^{\prime }\mathit{\beta })}{1+\exp ({\alpha }_j+{\mathit{x}}^{\prime }\mathit{\beta })},j=1,2,\dots ,J-1,\end{array}$$

(1)

or the cumulative logit form as:

$$\begin{array}{c}\hfill \log \left(\frac{P(Z\le j|\mathit{x})}{P(Z>j|\mathit{x})}\right)={\alpha }_j+{\mathit{x}}^{\prime }\mathit{\beta },j=1,2,\dots ,J-1,\end{array}$$

(2)

where ${\alpha }_j$ is an unknown intercept parameter associated with the $j\mathrm{th}$ category and $\mathit{\beta }={({\beta }_1,{\beta }_2,\dots {\beta }_p)}^{\prime }$ is the common vector of effect coefficients across the categories. The POLR models the cumulative probabilities $P(Z\le j)$ rather than the specific category probabilities $P(Z=j)$ as in the nominal logistic regression. Implementing nominal analysis on the ordinal data will lead to underestimation of the variation and considerable loss of information [1]. Other relevant approaches dealing with ordinal outcomes include adjacent-category logit and sequential logit models. A number of researchers have proposed various inference processes for the logistic regression or other methods by classical approaches in [2,5] or from Bayesian perspectives in [6,7]. A comprehensive review about the analysis of ordered category data can be found in [1]. Another commonly-adopted way of modeling ordinal data is via an underlying continuous latent variable. This regards the observed ordinal response as a crude measurement of a continuous variable falling into an interval on the real line. The work in [3] applied a Bayesian analysis approach for a latent variable model to investigate the effect of a binary treatment on an ordinal outcome of interest. A fully-Bayesian method for modeling polychotomous ordinal categories was developed in [6] by using the data augmentation approach. Some issues arise with these approaches, one of which is the estimation of the cutpoint parameters of the interval boundary. Incorporating a vague prior on these parameters, the work in [6] proposed a probit model in a Bayesian framework, which leads to a slow convergence under large sample sizes because of inefficient sampling of the cutoff point parameters. The work in [7] suggested a hybrid Gibbs/Metropolis–Hasting (MH) sampling scheme, which updates these parameters jointly with other parameters. Although the approach reduces the high auto-correlations in the sampling, it may require the computation of joint cumulative probabilities of multivariate distributions, which is generally intractable, for computing the acceptance probability in MH samplers. To avoid these difficulties, the work in [8] proposed a probit model with latent variables following a mixture model, which can successfully characterize the ordinality of data without the need to estimate these parameters.

In this paper, we propose informative priors on the parameters in the probit model, especially for the boundaries associated with category responses based on a Dirichlet distribution via a parameter transformation. The resulting posterior distributions with appropriate hyperparameter values chosen makes the sampling algorithm have fast convergence and efficient estimations. The rest of the article is arranged as follows. Section 2 presents our Bayesian inference procedure for the probit model with the the latent variable approach. Subsequently, we carry out simulation studies to investigate the performance of the proposed methods in Section 3. For illustrative purposes, two real datasets are analyzed in Section 4, followed by some concluding remarks in Section 5.

2. Bayesian Method for the Probit Model

We first briefly introduce the probit model with a latent variable and then present a Bayesian procedure for the analysis. Let $(\mathit{Z},\mathit{X})$ denote the observed data, where the n-vector $\mathit{Z}={(Z_1,Z_2,\dots ,Z_n)}^{\prime }$ is the n-ordered categorical outcomes and $n\times p$ matrix of covariates $\mathit{X}={({\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_n)}^{\prime }$ with ${\mathit{x}}_i$ being the $i\mathrm{th}$ vector of covariates containing 1 in the first element. The responses $Z_i$ take one of the J values, $1,\dots ,J$, and are associated with the covariate vector ${\mathit{x}}_i$ through a latent continuous variable $y_i$ in the following linear regression model:

$$y_i={\mathit{x}}_i^{\prime }\mathit{\beta }+{\epsilon }_i,{\epsilon }_i\sim \mathcal{N}(0,{\sigma }^2),i=1,\dots ,n,$$

(3)

where $\mathit{\beta }={({\beta }_1,{\beta }_2,\dots ,{\beta }_p)}^{\prime }$ is a $p\times 1$ vector of regression coefficients and ${\sigma }^2$ is set to 1 to make the model identifiable. The correspondence between the observed outcome $z_i$ and the latent variable $y_i$ is defined by:

$$z_i=j\mathrm{if}{\delta }_{j-1}<y_i\le {\delta }_j,j=1,\dots ,J,$$

(4)

where the boundaries ${\delta }_j$ are unknown and $-\infty ={\delta }_0<{\delta }_1<\dots <{\delta }_{J-1}<{\delta }_J=\infty$.

2.1. Prior Specification

We chose an independent prior for the parameters $\mathit{\beta }$ and $\mathit{\delta }$, respectively. First, we specify a conjugate prior, multivariate normal, for the regression coefficients:

$$\begin{array}{c}\hfill \mathit{\beta }\sim \mathcal{N}({\mathit{\beta }}_0,{\mathsf{\Sigma }}_0),\end{array}$$

(5)

with the p-vector mean ${\mathit{\beta }}_0$ and $p\times p$ variance-covariance matrix ${\mathsf{\Sigma }}_0$ being hyperparameters. For setting a prior for the boundaries $\mathit{\delta }=({\delta }_1,{\delta }_2,\cdots ,{\delta }_{J-1})$, first, suppose that there is a continuous distribution function $F(·)$ whose domain lies in $(-\infty ,\infty )$, for example, a normal function, such that the coverage probability at each interval is $p_j=P({\delta }_{j-1}<\Delta <{\delta }_j)=F\left({\delta }_j\right)-F\left({\delta }_{j-1}\right),j=1.2,\dots ,J$, and so, ${\sum }_{j=1}^J{p}_j=1$. It follows that:

$$\begin{array}{c}\left\{\begin{array}{ccc}{p}_1& =& F\left({\delta }_1\right)\hfill \\ p_2& =& F\left({\delta }_2\right)-F\left({\delta }_1\right)\hfill \\ \cdots \\ p_{J-1}& =& F\left({\delta }_{J-1}\right)-F\left({\delta }_{J-2}\right)\hfill \end{array}\right.⟺\left\{\begin{array}{ccc}{\delta }_1& =& F^{-1}\left(p_1\right)\hfill \\ {\delta }_2& =& F^{-1}(p_1+p_2)\hfill \\ \cdots \\ {\delta }_{J-1}& =& F^{-1}(p_1+p_2+\cdots +p_{J-1}).\hfill \end{array}\right.\hfill \end{array}$$

(6)

Secondly, a Dirichlet prior distribution is set for $(p_1,p_2,\cdots ,p_J)$, that is, the prior density $\pi (p_1,p_2,\cdots ,p_J|\mathit{\gamma })=1/\mathit{B}\left(\mathit{\gamma }\right)\times {\prod }_{j=1}^J{p}_j^{{\gamma }_j-1}$ with the positive hyperparameters $\mathit{\gamma }=({\gamma }_1,{\gamma }_2,\cdots ,{\gamma }_J)$ and the Beta coefficient $\mathit{B}\left(\mathit{\gamma }\right)={\prod }_{j=1}^J\Gamma \left({\gamma }_j\right)/\Gamma \left({\sum }_{j=1}^J{\gamma }_j\right)$ with the gamma function $\Gamma (·)$. Thus, by the transformation in (6), the prior of $\mathit{\delta }=({\delta }_1,{\delta }_2,\cdots ,{\delta }_{J-1})$ becomes:

$$\begin{array}{c}\hfill \pi ({\delta }_1,{\delta }_2,\cdots ,{\delta }_{J-1}|F,\mathit{\gamma })=\frac{1}{\mathit{B}\left(\mathit{\gamma }\right)}\prod _{j=1}^J{[F\left({\delta }_j\right)-F\left({\delta }_{j-1}\right)]}^{{\gamma }_j-1}\prod _{j=1}^{J-1}f\left({\delta }_j\right),\end{array}$$

(7)

where $f(·)$ is the density function of $F(·)$. Clearly, the expression (7) can be regarded as the joint density of order statistics $({\delta }_1,{\delta }_2,\cdots ,{\delta }_{J-1})$ with the rank index $({\gamma }_1,{\gamma }_2,\cdots ,{\gamma }_J)$ drawn from the distribution function $F(·)$ (see, for example, [9]).

2.2. Posterior Inference

The prior beliefs are then updated with information from the data to lead to the following joint posterior distribution:

$$\begin{array}{c}\hfill \pi (\mathit{\beta },\mathit{\delta }|\mathit{X},\mathit{Y},\mathit{Z})\propto L(\mathit{\beta },\mathit{\delta }|\mathit{X},\mathit{Y},\mathit{Z})\pi \left(\mathit{\beta }\right)\pi \left(\mathit{\delta }\right),\end{array}$$

(8)

where $\mathbf{Y}={(y_1,y_2,\dots ,y_n)}^{\prime }$ is the vector of latent variables and the likelihood function $L(\mathit{\beta },\mathit{\delta }|\mathit{X},\mathit{Y},\mathit{Z})={\prod }_{i=1}^n{f}_Y\left(y_i\right|{\mathit{x}}_i,\mathit{\beta })I({\delta }_{j-1}<y_i<{\delta }_j,z_i=j)$ with $f_Y(·)$ being the density function of $\mathcal{N}({\mathit{x}}_i^{\prime }\mathit{\beta },1)$ and $I(·)$ being the indicator function. Then, the conditional posteriors are:

$$\begin{array}{}\left(9\right)& \pi \left(\mathit{\beta }\right|\mathit{X},\mathit{Y},\mathit{Z},\mathit{\delta })\hfill & \propto & L(\mathit{\beta },\mathit{\delta }|\mathit{X},\mathit{Y},\mathit{Z})\pi \left(\mathit{\beta }\right),\mathrm{resulting} \mathrm{in}\mathit{\beta }|(\mathit{X},\mathit{Y},\mathit{Z},\mathit{\delta })\sim \mathcal{N}(\tilde{\mathit{\beta }},\tilde{\mathsf{\Sigma }}),\hfill \left(10\right)& \pi \left(\mathit{\delta }\right|\mathit{X},\mathit{Y},\mathit{Z},\mathit{\beta })\hfill & \propto & \pi \left(\mathit{\delta }\right)\times \prod _{i=1}^{n}I({\delta }_{j-1}<y_i<{\delta }_j,z_i=j),\hfill \end{array}$$

with the posterior mean vector $\tilde{\mathit{\beta }}={({\mathit{X}}^{\prime }\mathit{X}+{\mathsf{\Sigma }}_0^{-1})}^{-1}({\mathit{X}}^{\prime }\mathit{Y}+{\mathsf{\Sigma }}_0^{-1}{\mathit{\beta }}_0)$ and variance-covariance matrix $\tilde{\mathsf{\Sigma }}={({\mathit{X}}^{\prime }\mathit{X}+{\mathsf{\Sigma }}_0^{-1})}^{-1}$. The conditional posterior of $\mathit{\delta }$ in (10) is a truncated joint density of order statistics $({\delta }_1,{\delta }_2,\cdots ,{\delta }_{J-1})$. To draw its posterior samples conveniently, we explore the conditional posterior distribution of each component ${\delta }_j$. Let ${\mathit{\delta }}_{(-j)}$ be the vector $\mathit{\delta }$ without the $j\mathrm{th}$ element, that is, ${\mathit{\delta }}_{(-j)}=({\delta }_1,\dots ,{\delta }_{j-1},{\delta }_{j+1},\dots ,{\delta }_{J-1})$, and it follows that the conditional posterior of ${\delta }_j$ is:

$$\begin{array}{ccc}\pi \left({\delta }_j\right|{\mathit{X},\mathit{Y},\mathit{Z},\mathit{\delta }}_{(-j)})\hfill & \propto & {[F\left({\delta }_j\right)-F\left({\delta }_{j-1}\right)]}^{{\gamma }_j-1}{[F\left({\delta }_{j+1}\right)-F\left({\delta }_j\right)]}^{{\gamma }_{j+1}-1}f\left({\delta }_j\right)\hfill \\ & \times & I(c_{j,1}<{\delta }_j<c_{j,2}),j=1,2,\dots ,J-1,\hfill \end{array}$$

(11)

where $c_{j,1}=\max \{y_i,i=1,2,\dots ,n:z_i=j\},c_{j,2}=\min \{y_i,i=1,2,\dots ,n:z_i=j+1\},j=1,2,\cdots ,J-1$, and $F\left({\delta }_0\right)=0,F\left({\delta }_J\right)=1$. Therefore, conditionally, ${\delta }_j$ is a random variable whose transformed $F\left({\delta }_j\right)$ is distributed as the shifted $F\left({\delta }_{j-1}\right)$ and scaled [$F\left({\delta }_{j+1}\right)-F\left({\delta }_{j-1}\right)$] Betadistribution Beta $({\gamma }_j,{\gamma }_{j+1})$ truncated at interval $[F\left(c_{j,1}\right),F\left(c_{j,2}\right)]$, or equivalently:

$$\begin{array}{c}\hfill \left.\frac{F\left({\delta }_j\right)-F\left({\delta }_{j-1}\right)}{F\left({\delta }_{j+1}\right)-F\left({\delta }_{j-1}\right)}\right|({\delta }_{j-1},{\delta }_{j+1})\sim \mathrm{Beta}({\gamma }_j,{\gamma }_{j+1})\\ \hfill \mathrm{truncated} \mathrm{at}\left[\frac{F\left(c_{j,1}\right)-F\left({\delta }_{j-1}\right)}{F\left({\delta }_{j+1}\right)-F\left({\delta }_{j-1}\right)},\frac{F\left(c_{j,2}\right)-F\left({\delta }_{j-1}\right)}{F\left({\delta }_{j+1}\right)-F\left({\delta }_{j-1}\right)}\right].\end{array}$$

(12)

The method that we propose here is closely related to the approach presented in [10] for multinomial probit models. In this context, however, the correspondence between $Z_i$ and $Y_i$ uses different boundaries that account for the natural ordering of the outcome. We performed posterior inference using Markov chain Monte Carlo (MCMC) techniques. Specifically, a Gibbs sampling based on the above conditional posteriors was adopted starting from a set of initial values of parameters, and then, the following step was repeated M times, among which, given the values of ${\mathit{Y}}^{\left(k\right)},{\mathit{\beta }}^{\left(k\right)}$ and ${\mathit{\delta }}^{\left(k\right)}$ at the $k\mathrm{th}$ iteration, the $(k+1)\mathrm{th}$ iteration is as follows:

(1)

Update the latent vector $\mathit{Y}$ from its posterior distribution given $(\mathit{\beta },\mathit{\delta },\mathit{X},\mathit{Z})$, each element of which is a truncated normal under the constraints defined in Equation (4):

$$\begin{array}{cc}\hfill Y_i^{(k+1)}|({\mathit{\beta }}^{\left(k\right)},{\mathit{\delta }}^{\left(k\right)},\mathit{X},\mathit{Z})\sim \mathcal{N}({\mathit{x}}_i^{\prime }{\mathit{\beta }}^{\left(k\right)},1),& \hfill {\delta }_{j-1}^{\left(k\right)}<Y_i^{(k+1)}\le {\delta }_j^{\left(k\right)}\mathrm{if}{Z}_i=j,\\ \hfill j=1,\dots ,J,i=1,2,\dots ,n.\end{array}$$

(13)

(2)

Update the regression coefficients $\mathit{\beta }$ from their posterior distribution in Equation (9) under the updated ${\mathit{Y}}^{(k+1)}$.

(3)

Update the boundary parameters ${\delta }_j$ from their posterior densities given in Equation (12) with $c_{j,1}$ and $c_{j,2}$ being evaluated at ${\mathit{Y}}^{(k+1)}$. Specifically, first draw a $d_j^{(k+1)}$ from the truncated Beta in Equation (12), and then, get ${\delta }_j^{(k+1)}=F^{-1}\left(u_j^{(k+1)}\right)$ where $u_j^{(k+1)}=F\left({\delta }_{j-1}^{(k+1)}\right)+[F\left({\delta }_{j+1}^{\left(k\right)}\right)-F\left({\delta }_{j-1}^{(k+1)}\right)]d_j^{(k+1)},j=1,2,\dots ,J-1$.

2.3. Hyperparameter Settings

The prior on $\mathit{\beta }$ depends on the mean ${\mathit{\beta }}_0$ and covariance matrix ${\mathsf{\Sigma }}_0$, and the work in [11] discussed the relative merits and drawbacks of different specifications. Here, we set ${\mathit{\beta }}_0=\mathbf{0}$ and ${\mathsf{\Sigma }}_0=c\mathbf{I}$, which is easier to calibrate. The parameter c regulates the amount of shrinkage in the model. In general, we want to avoid very small values of c, which cause too much regularization, and large values, which can induce nonlinear shrinkage as a result of Lindley’s paradox [12]. In the context of probit models for classification into nominal groups, the work in [10] provided some guidelines on how to choose this hyperparameter value. We used similar guidelines here. In practice, the values of c that provide good mixing of the MCMC sampler, with 25–50% distinct visited models, are appropriate [13]. An informative prior of $\mathit{\delta }$ in (7) can be specified by using the number of categorical data to set the component ${\gamma }_j$ in the parameter $\mathit{\gamma }$ to be the counts in the $j\mathrm{th}$ category among $z_i$’s. Alternatively, a diffuse prior can be made by assigning all ${\gamma }_j=1$ to express no prior belief for ${\delta }_j$, namely, a uniform distribution. For the transformed distribution function $F(·)$, we chose a normal distribution with zero mean and large-scale ${\sigma }_0$ (for example ${\sigma }_0=50)$ to cover a fairly wide range in $(-\infty ,\infty )$, and so, in this case, $F\left(\delta \right)=\Phi (\delta /{\sigma }_0)$ with $\Phi (·)$ being the cumulative distribution function of the standard normal.

2.4. Posterior Prediction

The MCMC procedure results in a list of sampled $\mathit{Y},\mathit{\beta }$, and $\mathit{\delta }$ vectors. In order to draw posterior inference, we first need to impute the latent vector $\mathbf{Y}$, which can be viewed as missing data. Let $\widehat{\mathbf{Y}},\widehat{\mathit{\beta }}$ and $\widehat{\mathit{\delta }}$ be the estimates obtained by averaging respectively over the sampled $\mathit{Y},\mathit{\beta }$, and $\mathit{\delta }$ vectors.

Inference on class prediction can be done in various ways. If a further future vector of covariates ${\mathit{x}}_f$ is available for validation, the least squares prediction based on a single value of $\mathit{\beta }$ can be computed as:

$$\begin{array}{c}\hfill {\widehat{y}}_f={\mathit{x}}_f^{\prime }\widehat{\mathit{\beta }}\mathrm{or}{\widehat{y}}_f={\mathit{x}}_f^{\prime }\widehat{\tilde{\mathit{\beta }}},\end{array}$$

(14)

where the posterior mean $\widehat{\tilde{\mathit{\beta }}}={({\mathit{X}}^{\prime }\mathit{X}+{\mathsf{\Sigma }}_0^{-1})}^{-1}{\mathit{X}}^{\prime }\widehat{\mathit{Y}}$. Alternatively, we can use the Bayesian model averaging over a set of a posteriori likely models of posterior samples ${\mathit{\beta }}^{\left(k\right)}$ to estimate $y_f$ as:

$$\begin{array}{c}\hfill {\widehat{y}}_f=\sum _{k=1}^M{\mathit{x}}_f^{\prime }{\mathit{\beta }}^{\left(k\right)}\pi \left({\mathit{\beta }}^{\left(k\right)}\right|\mathit{X},\widehat{\mathit{Y}},\mathit{Z},\widehat{\mathit{\delta }}).\end{array}$$

(15)

The ordered categorical outcomes can then be predicted using the boundary correspondence:

$$\begin{array}{c}\hfill {\widehat{z}}_f=j\mathrm{if}{\widehat{\delta }}_{j-1}<{\widehat{y}}_f\le {\widehat{\delta }}_j,j=1,2,\dots ,J.\end{array}$$

(16)

In addition, by the fact that $y_f\sim \mathcal{N}({\mathit{x}}_f^{\prime }\mathit{\beta },1)$, the prediction probability that it falls in each class can be computed through the model averaging over the posterior samples ${\mathit{\beta }}^{\left(k\right)}$:

$$\begin{array}{ccc}\hfill P(Z_f=j)& \approx & \sum _{k=1}^{M}P({\widehat{\delta }}_{j-1}<Y_f\le {\widehat{\delta }}_j|{\mathit{\beta }}^{\left(k\right)})\pi \left({\mathit{\beta }}^{\left(k\right)}\right|\mathit{X},\widehat{\mathit{Y}},\mathit{Z},\widehat{\mathit{\delta }})\hfill \\ & =& \left[\Phi \left({\widehat{\delta }}_j-{\mathit{x}}_f^{\prime }{\widehat{\mathit{\beta }}}^{\left(k\right)}\right)-\Phi \left({\widehat{\delta }}_{j-1}-{\mathit{x}}_f^{\prime }{\widehat{\mathit{\beta }}}^{\left(k\right)}\right)\right]\pi \left({\mathit{\beta }}^{\left(k\right)}\right|\mathit{X},\widehat{\mathit{Y}},\mathit{Z},\widehat{\mathit{\delta }}),\hfill \end{array}$$

(17)

where $\Phi (·)$ is the distribution function of the standard normal distribution. The class membership can then be predicted by the mode of the predictive distribution:

$$\begin{array}{c}\hfill {\widehat{z}}_f=\underset{1\le j\le J}{\mathrm{argmax}}P(Z_f=j).\end{array}$$

(18)

Furthermore, a less varied estimate can be obtained by the nearest rounded integer computed through membership averaging over predictive probabilities,

$$\begin{array}{c}\hfill {\widehat{z}}_f=\left[{\mu }_Z\right],\mathrm{where}{\mu }_Z=\sum _{j=1}^{J}jP(Z_f=j).\end{array}$$

(19)

3. Simulation Study

We conducted a simulation study to assess the performance of the proposed Bayesian method. The simulated datasets were from three multivariate normal distributions whose dimension was four and the means ${\mathit{\mu }}_1={[3,2,4,1]}^{\prime },{\mathit{\mu }}_2={[3,-2,4-,1]}^{\prime },{\mathit{\mu }}_3={[-3,-2,-4,-1]}^{\prime }$ and equal correlation variance-covariance matrices $\mathsf{\Sigma }={\sigma }^2[(1-\rho )\mathbf{I}+\rho {\mathbf{1}1}^{\prime }]$, where $\sigma =2$, the increasingly-ordered correlations $\rho =0.1,0.5,0.9$ for three structures, corresponding to the ordered response $z=1,2,3$, respectively, $\mathbf{I}$ is the identity matrix, and $\mathbf{1}$ is the vector of ones. We simulated 30 observational data $\mathit{x}={(x_1,x_2,x_3,x_4)}^{\prime }$ from each multivariate normal to make a total sample size of 90. Considering the structure of data may be an essential aspect for efficient classification. In order not to make erroneous conclusions from the predicted error rates, we critically examined the nature of the data to explore the variations existing among groups. In a model for predicting ordinal outcomes, it is obvious to believe that, remarkably, the least variations and far apart locations among groups of variables could make it easy to classify or predict correctly into which category a particular observation falls. In this vein, we visualized the data structure by using the tool of displaying data concentration in [14], who proposed the p-dimensional ellipsoid, $E_d$, of size (“radius”) d, which is defined as the set of all points $\mathit{X}$ in a contour:

$$\begin{array}{c}\hfill E_d(\overline{\mathit{X}},\mathit{S})=\{\mathit{X}:{(\mathit{X}-\overline{\mathit{X}})}^T{\mathit{S}}^{-1}(\mathit{X}-\overline{\mathit{X}})\le d^2\}.\end{array}$$

(20)

Clearly, $E_d$ corresponds to the set of points whose Mahalanobis distances $D^2={(\mathit{X}-\overline{\mathit{X}})}^{\prime }{\mathit{S}}^{-1}(\mathit{X}-\overline{\mathit{X}})$ with sample covariance matrix $\mathit{S}$, from the centroid of the sample, $\overline{\mathit{X}}={({\overline{X}}_1,{\overline{X}}_2,\cdots ,{\overline{X}}_p)}^{\prime }$, are less than or equal to $d^2$. For multivariate normal variables, the data ellipsoid approximates a contour with constant density in their joint distribution, and $D^2$ is the asymptotic chi-squared distribution with p degrees of freedom, ${\chi }_p^2$. The work in [15] elaborated more on the properties of data ellipsoids and their use in a wide variety of problems and applications. Taking $d^2={\chi }_{0.05,2}^2=5.99$, the two-dimensional pairwise ellipsoid is shown in Figure 1, where the ellipse encloses approximately 95% of the data points under the normal theory. The plot indicates that there were some overlaps between Groups 1 and 2, and Group 3 was relatively well-separated away from the other two. Thus, we expected a slightly high classification error rate where the most misclassified cases would probably occur among Groups 1 and 2.

Next, we performed the analysis by our method. To obtain accurate results and avoid over-fitting, we applied the Bayesian model on the simulated data (training data), and generated a test data with 300 observations for each category for validation. We ran four MCMC chains with widely different starting values for 10,000 iterations each and discarded the first 2000 as burn-in to eliminate dependence on the starting points. We also considered several hyperparameter values for the covariance of the regression coefficients ${\mathsf{\Sigma }}_0=c\mathbf{I}$, with c ranging between 5 and 20. There was minimal effect on the overall results, and here, we report the results for $c=10$. An informative prior for the boundary parameter vector $\mathit{\gamma }$ was specified by setting all the components ${\gamma }_j=30$, the counts in the $j\mathrm{th}$ category among ordinal responses. We note that despite the widely different starting values, there was a good agreement between the results. The error rate for the test data was recorded from the predictions to the misclassification rate tabulated in

Table 1. Simulated data: test data prediction misclassification rates. POLR, polytomous ordinal logistic regression; QDA, quadratic discriminant analysis.

Bayesian Method			POLR	LDA QDA KNN SVM
Boundary	Probability	Average
0.308	0.307	0.294	0.348	0.362   0.357   0.356   0.371

, containing the three prediction approaches in Equations (16) (where $y_f$ is estimated in Equations (14)), (18) and (19). The classification result showed that there were in total about 270 subjects misclassified, among which, 90 out of 300 observations in Group 1 were misclassified into Group 2, while 100 obsin Group 2 were incorrectly assigned to Group 1. The outcome validated our previous judgment from the pairwise ellipses in Figure 1, where Groups 1 and 2 shared quite a bit of common area. As shown in Table 1, the three types of predictions approximately yielded the same error rates, and the polytomous ordinal logistic regression (POLR) model resulted in a higher classification error rate.

Finally, for comparison purposes, we analyzed the data using common classification methods, such as linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), k-nearest neighbor (KNN), and support vector machines (SVM), which build multi-class classifiers without taking the natural ordering of the response into account. For KNN, we considered values of group number k ranging from 2–8, and we report the results for $k=3$, which gave the lowest overall misclassification rate. All these approaches produced a higher error rate as summarized in Table 1.

4. Real Data Analysis

We applied our model for discrimination analysis on two real datasets. The first was the well-known Irisflower data, and the second was about the measurements of male Egyptian skull data. The purpose of adopting these two datasets was to examine how the model predictions were affected by different data structures and measurement variations among groups.

4.1. Iris Data

We used the dataset as a benchmark for the analysis. It was introduced by the British statistician Ronald [16]. It is also sometimes called Anderson’s Iris data [17], which was collected to quantify the morphological variation of Iris flowers of three related species pictured in Figure 2. The dataset consisted of 50 samples from each of three species (Iris setosa, Iris versicolor, and Iris virginica) with four measured features for each sample: the length and width of the sepals and petals, in centimeters. These are generally viewed as nominal category data and have been analyzed in a vast literature, such as [18,19], and many others. Here, we explore it as ordinal outcomes of three species instead, ranked by the magnitude of measurement variations. The total variances of the four morphological measurements for each species were: 0.3292 (setosa), 0.6248 (versicolor), and 0.8883 (virginica). These numbers, to some extent, represent the size and spread of flowers, which is consistent with the image in Figure 2, where the flower of setosa has the least size in terms of sepal and petal, versicolor larger, and virginica largest.

These findings can also be observed from the visual display of the ellipses for all pairs of measurements shown in Figure 3, where one may further notice that, except a little overlap between groups of versicolor and virginica, the three species of data were overall well-separated; especially, Iris setosa was far apart from the other two species. To obtain an accurate error rate of classification, we applied the cross-validation approach to partition the whole dataset into a training set containing 120 observations with 40 for each category and a testing data containing 30 observations with 10 for each category. The partitioning was repeated five times until all the samples from each species were exhausted. The small error rate of classification produced by our method with cross-validation prediction, along with POLR regression, is summarized in

Table 2. Real data: cross-validated prediction misclassification rates.

	Bayesian Method			POLR	LDA QDA KNN SVM	Null Model
	Boundary	Probability	Average
Iris	0.013	0.013	0.013	0.026	0.036  0.034  0.035  0.033	0.667
Skull	0.553	0.546	0.533	0.607	0.646  0.633  0.653  0.653	0.800

, where all the cases of setosa were classified correctly by both methods, two (or three) cases from versicolor were misclassified to the virginica species by the Bayesian (or POLR) method, and one from virginica was incorrectly assigned to versicolor by the POLR model. Other common classification methods dealing with nominal data yielded a little larger error rate, shown in Table 2. To see how much the reduction of the error rates for each prediction approach were, we simply computed the error rate of the null model where no covariates was used, that is, $1-\max \left(R{F}_j\right)$, as listed in Table 2, where $R{F}_j$ is the relative frequency for the $j\mathrm{th}$ category in the dataset.

4.2. Skull Data

The dataset was obtained from the R embedded package “HSAUR”. The data consisted of four physical measurements in millimeters of 30 male Egyptian skulls from each of five epochs (periods) [20]: Period 1 (4000 BC), Period 2 (3300 BC), Period 3 (1850 BC), Period 4 (200BC), and Period 5 (150 AD). The measures were the maximal breadth (mb), basibregmatic height (bh), basialveolar length (bl), and nasal height (nh) of each skull. Figure 4 gives an illustrative labeled image for these four measurements of a typical skull. The researchers claimed that a change in skull measurement was as a result of the time duration. Systematic changes over time could indicate interbreeding among migrant populations (or the influence of other factors) [21]. The interest in this analysis, however, lies in the ability to predict which period these measurements fall within well.

Figure 5 displays the ellipses for all pairs of measurements, and it clearly shows that the five groups of data were pretty much overlapping each other so that one can hardly distinguish one class from the others, and a high classification error rate can be expected. For our Bayesian method, an MCMC chain with 10,000 iterations was run, and the first half was discarded to eliminate dependence on the starting points. Several hyperparameter values were chosen for the covariance of the regression coefficients, with c ranging between 5 and 15. We note that different hyperparameter values resulted in a similar classification. The error rates of cross-validated prediction are listed in Table 2, where, not surprisingly, the high error rates (83, 82, and 80 skulls were misclassified respectively by the three prediction approaches of the Bayesian method, while 92 skulls were incorrectly assigned by POLR) of classification produced by our method validated our previous judgment. In contrast to the Iris data, the skull dataset provided an example where the measurements tended to cluster around a common centroid and overlapped one another with little evidence of separation in location. The classification results of these two different structural datasets justified the reliability of our model in exploration and prediction for ordinal outcomes’ data.

We also noticed that all other common procedures even yielded higher error rates compared to our method summarized in Table 2. As a result of all these classification outcomes, we could logically infer that the measurements of these male Egyptian skulls did not change significantly across the given time periods.

5. Concluding Remarks

We have proposed a Bayesian approach for predicting ordinal outcomes. Introducing latent variables underlying the ordinal outcomes, the problem reduces to a linear model setting. While MCMC techniques are generally computationally intensive, with an informative prior posed on the boundary parameters, our MCMC algorithm showed practical implementation and fast convergence. The simulation study demonstrated the efficient and impressive performance of the proposed method. We have illustrated that our approach, with applications to two real datasets, provided an efficient, reliable, and precise analysis for ordinal categorical response data.