Bayesian Estimation of Simultaneous Regression Quantiles Using Hamiltonian Monte Carlo

Abstract

The simultaneous estimation of multiple quantiles is a crucial statistical task that enables a thorough understanding of data distribution for robust analysis and decision-making. In this study, we adopt a Bayesian approach to tackle this critical task, employing the asymmetric Laplace distribution (ALD) as a flexible framework for quantile modeling. Our methodology implementation involves the Hamiltonian Monte Carlo (HMC) algorithm, building on the foundation laid in prior work, where the error term is assumed to follow an ALD. Capitalizing on the interplay between two distinct quantiles of this distribution, we endorse a straightforward and fully Bayesian method that adheres to the non-crossing property of quantiles. Illustrated through simulated scenarios, we showcase the effectiveness of our approach in quantile estimation, enhancing precision via the HMC algorithm. The proposed method proves versatile, finding application in finance, environmental science, healthcare, and manufacturing, and contributing to sustainable development goals by fostering innovation and enhancing decision-making in diverse fields.

1. Introduction

Quantile regression, which is garnering increased attention from both theoretical and empirical perspectives [1,2,3,4,5], serves as a statistical cornerstone for modeling the intricate relationship between a set of predictor variables and a response variable [6,7]. This method excels in estimating the conditional quantiles of the response variable, going beyond conventional mean estimation. By shedding light on the varying impacts of predictor variables across different segments of a distribution, quantile regression offers a nuanced understanding of complex relationships in data analysis. As researchers delve into unraveling the intricacies of variable interplay, the application of quantile regression emerges as an invaluable approach for capturing and interpreting diverse patterns. For instance, we can enumerate examples in various fields such as the economy [8,9], ecology [10], epidemiology [11], survival analysis [12], and environmental economics [13,14]. Furthermore, quantile regression is a form of statistical and econometric regression analysis that adds a particular method to estimating families of conditional quantile functions and supplements the exclusive focus of least squares techniques based on the estimation of conditional mean functions [15]. Originating from the seminal work of [16], quantile regression introduces a versatile method for estimating conditional quantiles.

The authors demonstrated that any linear quantile of order $\tau$ could be derived from minimizing a check function, ${\rho }_{\tau }\left(u\right)=u(\tau -{\mathbb{1}}_{u<0})$, where $\mathbb{1}$ represents the indicator function. In exclusively linear regression models, Koenker et al. [16] suggest a straightforward method to estimating conditional quantiles, consider the regression linear normal model, as the solution of the minimization problem that follows:

$$\begin{array}{c}\hfill \underset{\mathit{\beta }}{min}\sum _t{\rho }_{\tau }(y_t-{\mathit{x}}_t^{\prime }\mathit{\beta }),\end{array}$$

(1)

where ${\mathit{x}}_t, t=1,\dots ,T$ denote a sequence of (row), ${\mathit{x}}^{\prime }$ is the transposed of $\mathit{x}$, K-vector of a known design matrix, and $y_t, t=1,\dots ,T$ are a random sample on the regression process $u=y-{\mathit{x}}_t^{\prime }\mathit{\beta }$ having distribution function F. Then, the $\tau$-th sample quantile, $0<\tau <1$, may be defined as any solution of the minimization problem (1).

In the framework of statistical research, Bayesian approaches have emerged as effective tools for investigating quantile regression [17,18,19,20] after the notable developments presented in [21]. The authors utilize a likelihood function grounded in the asymmetric distribution of Laplace to advance the understanding of quantile regression. While both frequentist and Bayesian perspectives prove suitable for examining individual quantiles [22], the complexities of real-world applications often demand more nuanced exploration. To intricately capture the relationship between the response variable and explanatory variables, it becomes imperative to estimate regression quantiles at various orders. Although it is conceivable to perform separate estimations for these quantiles, this approach proves unsatisfactory; particularly in scenarios involving multiple explanatory variables, it provides estimators that do not respect the property of the non-crossing of the regression quantiles.

To circumvent this limitation, a more favorable approach involves the simultaneous estimation of multiple regression quantiles [23,24,25,26]. This ensures a coherent and consistent estimation procedure, particularly in situations characterized by the presence of multiple explanatory variables.

In pursuit of refining the Bayesian quantile estimation paradigm, this article directs its attention to the groundwork laid by Merhi Bleik in the seminal work presented in [27]. The author’s utilization of the Metropolis–Hastings within Gibbs algorithm proved instrumental in sampling unknown parameters from their full conditional distribution, offering a new methodological framework. The proposed estimation procedure, SBQR, for simultaneous Bayesian quantile regression, as presented in [27], guarantees the fundamental property of non-crossing, under the assumption that the Asymmetric Laplace distribution (ALD) is the underlying data distribution. In extending and enhancing this foundational approach, our primary objective is to present a unified computational methodology. This methodology is designed to facilitate the simultaneous estimation of regression quantiles, employing the advanced Hamiltonian Monte Carlo (HMC) algorithm. Building on the achievements of [27], our proposed methodology aims to overcome potential limitations and further streamline the quantile estimation process. By integrating the HMC algorithm, we seek to harness its efficiency in exploring parameter spaces, thus enhancing the precision of quantile estimates. To evaluate the performance of our algorithm, we compare the mean of the Bayesian estimators with the theoretical quantile values using the root mean square error. For this reason, our objective is to simultaneously estimate the Bayesian regression quantile using the HMC algorithm. The main contributions of the paper are then the following:

• Conduct a simultaneous quantile estimation embedded with the HMC algorithm through Bayesian inference;
• Conduct a sensitivity analysis on the performance of the HMC algorithm in handling univariate linear quantiles and multivariate linear quantiles.

This article serves as a comprehensive guide to the unified computational framework, providing researchers and practitioners with a robust tool for simultaneous regression quantile estimation. By addressing challenges associated with multiple quantile modeling in a cohesive manner, our methodology proves versatile across diverse fields when used with ALD data. Notably, applications in finance, where risk assessment demands a nuanced understanding of the distribution, benefit from the precision offered by simultaneous regression quantile estimation. In environmental science, the ability to capture extreme events accurately is crucial, while in healthcare, predicting patient outcomes relies on a comprehensive grasp of varying response scenarios. Additionally, in manufacturing, ensuring quality control is facilitated by an in-depth understanding of distributional patterns. This versatility aligns with the broader societal goals, contributing to Sustainable Development Goal 9: Industry, Innovation, and Infrastructure. By fostering innovation in statistical methodologies and decision-making processes, our approach seeks to enhance the well-being of societies and align with broader sustainability objectives.

This paper introduces a novel approach by combining existing research with the innovative application of the HMC algorithm to simultaneous quantile estimation, offering significant advancements in the field. The paper is organized as follows: The ALD and its Gaussian mixture are presented in Section 2, in addition to the method proposed by the author in [27] for simultaneous Bayesian quantiles. A brief overview of HMC is presented in Section 3. The performance of the HMC algorithm is examined in cases of univariate linear quantiles and multivariate linear quantiles on a simulated data set in Section 4. Finally, Section 5 highlights the paper’s contributions and provides a discussion of perspectives.

2. Simultaneous Quantile Regression

2.1. Asymmetric Laplace Distribution

The ALD is a probability distribution that is used to model data with asymmetric tails and skewness. It is a continuous probability distribution defined on the real line. The distribution is characterized by three parameters: location, scale, and skewness parameters. Indeed, the ALD is often used in Bayesian statistics, particularly in the context of Bayesian regression models that involve quantile regression.

A random variable U follows an ALD($\mu$, $\sigma$, $\tau$) if its probability density is given by

$$\begin{array}{cc}\hfill & f(x;\mu ,\sigma ,\tau )={\displaystyle \frac{\tau (\tau -1)}{\sigma }}exp\left(-{\displaystyle \frac{(x-\mu )}{\sigma }}[\tau -\mathbb{1}(x⩽\mu )]\right),x\in \mathbb{R}.\hfill \end{array}$$

(2)

where $0<\tau <1$ is the skew parameter, $\sigma >0$ is the scale parameter, and $-\infty <\mu <\infty$ is the location parameter. The mean and variance of the ALD with density (2) are

$$\begin{array}{c}\hfill E\left[U\right]=\mu -{\displaystyle \frac{\tau -{\displaystyle \frac{1}{\tau }}}{\sigma }}\mathrm{and}Var\left[U\right]=\frac{1+{\tau }^4}{{\tau }^2{\sigma }^2}\end{array}$$

Some other ALD properties can be found in [28]. In order to simplify the posterior inference of the quantiles, it is convenient to use the location scale mixture representation. In particular, we are obliged to use it in HMC because ALD is not differentiable and to calculate the full conditional distributions (for more details, see [20]). The location-scale mixture of the ALD distribution will be used. Let $\omega$ be an exponential latent variable with parameter $1/\sigma$ denoted by $exp(1/\sigma )$ and let Z be a standard normal variable so that $\omega$ and Z are independent, then $ϵ\sim \mathrm{ALD}(0,\sigma ,\tau )$ has the following representation:

$$ϵ={\gamma }_{\tau }\omega +{\delta }_{\tau }\sqrt{\sigma \omega }Z,$$

where ${\displaystyle {\gamma }_{\tau }=\frac{1-2\tau }{\tau (1-\tau )}}$ and ${\displaystyle {\delta }_{\tau }^2=\frac{2}{\tau (1-\tau )}}$.

2.2. Regression Quantile

We begin with a fundamental definition of sample quantiles, a concept readily extended to the linear regression model without the typical need for an ordered set of sample observations. Consider the set $y_t:t=1,\dots ,T$ of independent and identically distributed (i.i.d.) observations on a random variable Y governed by the distribution function F. The $\tau$-th sample quantile, where $0<\tau <1$, can then be characterized as any solution to the minimization problem:

$$\begin{array}{c}\hfill \underset{\mathit{\beta }}{min}\sum _t{\rho }_{\tau }(y_t-\mathit{\beta }),\end{array}$$

where ${\rho }_{\tau }$ is the check loss function, defined as

$$\begin{array}{c}\hfill {\rho }_{\tau }\left(u\right)=u(\tau -\mathbb{1}(u<0)).\end{array}$$

(3)

Equivalently, we may write the minimization problem as

$$\begin{array}{c}\hfill \underset{\mathit{\beta }\in \mathbb{R}}{min}[\sum _{t\in \{t:y_t⩾\beta \}}\tau |y_t-\mathit{\beta }|+\sum _{t\in \{t:y_t<\mathit{\beta }\}}(1-\tau )|y_t-\mathit{\beta }\left|\right].\end{array}$$

Huber et al. [29] observed the difficulty in identifying outliers within the context of regression, emphasizing the inherent confusion in extending standard notions of sample quantiles to linear models based on the ordering of sample observations. This confusion is effectively addressed by a direct generalization of the earlier-described minimization problem. Consider a sequence of k-vectors from a known design matrix denoted by $\{x_t,t=1,\dots ,T\}$. Assume that $\{y_t,t=1,\dots ,T\}$ constitutes a random sample from the regression process, where $u_t=y_t-{\mathit{x}}_t^{\prime }\beta$ follows a distribution function F such that $F\left(0\right)=\tau$. The $\tau$-th quantile of the linear regression model, with $0<\tau <1$, is defined as any solution to the minimization problem:

$$\underset{\mathit{\beta }\in {\mathbb{R}}^K}{min}[\sum _{t\in \{t:y_t⩾{\mathit{x}}_t^{\prime }\mathit{\beta }\}}\tau |y_t-{\mathit{x}}_t^{\prime }\mathit{\beta }|+\sum _{t\in \{t:y_t<\mathit{\beta }\}}(1-\tau )|y_t-{\mathit{x}}_t^{\prime }\mathit{\beta }\left|\right]$$

(4)

2.3. Bayesian Linear Quantile Regression

Consider the following quantile regression model:

$$Y={\mathit{x}}^{\prime }{\mathit{\beta }}_{\mathit{\tau }}+ϵ.$$

(5)

We assume that the error $ϵ$ and $\mathit{x}$ are independent, and $ϵ\sim F$, F is unknown, but $F\left(0\right)=\tau$. Hence, ${\mathit{x}}^{\prime }{\mathit{\beta }}_{\mathit{\tau }}$ denotes the $\tau$-th quantile of $Y|\mathit{x}$. For all ${\mathit{x}}_{\mathit{i}}\in \mathbb{R}$, suppose the relationship between ${\mathit{\iota }}_{\tau }\left({\mathit{x}}_{\mathit{i}}\right)$ and $x_i$ could be modeled as ${\mathit{\iota }}_{\tau }\left(x_i\right)=x_i{\mathit{\beta }}_{\mathit{\tau }}$, where ${\mathit{\beta }}_{\tau }$ is a vector with unknown interest parameters. From i.i.d ${({\mathit{x}}_t,Y_t)}_{t=1,\dots ,T}$ issued from the model defined in Equation (5), we are interested in estimating quantiles of the $Y|\mathit{x}$ law via the Bayesian approach. It then suffices to estimate ${\mathit{\beta }}_{\mathit{\tau }}$.

If F is unknown, the likelihood is not available, but the Bayesian approach requires some likelihood; hence, we use the “working likelihood” defined as the ALD:

$$l\left(Y\right|\mu ,\sigma ,\tau )={\displaystyle \frac{\tau (\tau -1)}{\sigma }}exp(-{\rho }_{\tau }({\displaystyle \frac{Y-\mu }{\sigma }})),$$

where $\mu =x_t^{\prime }{\beta }_{\tau }$ and ${\rho }_{\tau }\left(u\right)$ is as defined in (3). The $\tau$-th quantile of this distribution is zero; hence, the pseudo posterior is

$$\pi \left(\mathit{\beta }\right|{({\mathit{x}}_t,Y_t)}_{t=1,\dots ,T})\propto \prod _{t=1}^T\mathrm{ALD}(y_t-{\mathit{x}}_t^{\prime }{\mathit{\beta }}_{\tau },\sigma ,\tau )\times \pi ({\mathit{\beta }}_{\tau },\sigma ),$$

(6)

Equation (6) describes the ALD probability distribution, where $y_t$ is the observed data point, $x_t^{\prime }$ is the design matrix for observation t, $\beta$ is a set of parameters, $\sigma$ is a scale parameter, and $\tau$ is a quantile parameter. This equation provides a natural way to solve the problem of Bayesian estimation of the quantile $q_{\tau }$ of order $\tau$. Moreover, the relationship between minimizing the loss function (3) and maximizing a likelihood function is demonstrable. This equivalence arises when the likelihood function is constructed by combining independently distributed asymmetric densities of the Laplace distribution. Furthermore, it is noteworthy to emphasize that the simple Bayesian quantile can be regarded as a specific instance of a Bayesian regression quantile. In this particular case, the simplicity is characterized by setting $k=1$ and considering only the variable x, with the absence of additional co-variables. This distinction highlights a specialized scenario within the broader framework of Bayesian quantile regression.

2.4. Simultaneous Quantile Regression

Many approaches have been proposed to overcome the crossing problem. See, among others, in the frequentist context [30,31,32]. In the simultaneous Bayesian context, on the other hand, Ref. [33] suggested a two-stage semi-parametric regression method. Steven G Xu and Brian J Reich [34] have also proposed a novel treatment to nonlinear simultaneous quantile regression. They have achieved this by specifying a Bayesian nonparametric model for the conditional distribution. In a similar vein, Das and Ghosal [35] represent the quantile process by articulating it as a weighted sum of B-spline basis functions corresponding to the quantile level. In this study, the method proposed in [27] is considered, which assumes the error term has an ALD. It is noteworthy that imposing a given distribution on $u_t$ is restrictive but provides a fully Bayesian approach to handling simultaneous regression quantiles of different orders. This is performed first by partitioning the entire sample into s sub-samples, where s is the number of quantiles of interest; second, by using the relationship between any two ALD quantiles; and third, by rewriting the entire likelihood into s terms where the j-th term depends only on $q_{{\tau }_j}$, where $q_{{\tau }_j}$ is the j-th term of the vector of quantiles. Let us explain how it works:

• Consider partitioning the entire sample into the sub-samples below:

  $$\begin{array}{c}\hfill ({\mathit{x}}_{I_j},Y_{I_j})=\{{({\mathit{x}}_i,Y_i)}_{i\in I_j}\},\forall j\in \{1,\dots ,s\},\end{array}$$

  where $I_j=\{(j-1)r+1,\dots ,jr\}$ and with $I_j\bigcap I_k=\varnothing$$\forall j\ne k$ and $j,k\in \{1,\dots ,s\}$; assume that $r={\displaystyle \frac{n}{s}}$ is an integer without loss of generality.
• In order to characterize the likelihood through all quantiles of interest, we use the relationship that connects any $\tau$-th quantile to the p-th quantile of the $\mathrm{ALD}(q_p,\sigma ,p)$:

  $$q_{\tau }\left(x\right)=q_p\left(x\right)+\sigma g(\tau ,p)$$

  (7)

  where ${\displaystyle g(\tau ,p)={\displaystyle \frac{1}{1-p}}log\left({\displaystyle \frac{\tau }{p}}\right){\mathbb{1}}_{0<\tau ⩽p}+{\displaystyle \frac{1}{p}}log\left({\displaystyle \frac{1-\tau }{1-p}}\right){\mathbb{1}}_{p<\tau ⩽1}}$.
• We rewrite the model given by (5) from Equation (7) as follows:

  $$Y_{I_j}=q_{{\tau }_j},I_j-\sigma g({\tau }_j,p)+{ϵ}_{I_j},\forall j=1,\dots ,s$$

  where

  $$\begin{array}{cc}\hfill & q_{{\tau }_j,I_j}={\left(q_{{\tau }_j}\left({\mathit{x}}_{i_j}\right)\right)}_{i_j\in I_j},\hfill \\ \hfill & g({\tau }_j,p)={\left(g({\tau }_j,p){\mathbb{1}}_{i_j\in I_j}\right)}_{i_j\in I_j},\hfill \\ \hfill & {ϵ}_{I_j}={\left({ϵ}_{i_j}\right)}_{i_j\in I_j}.\hfill \end{array}$$

  The likelihood connected with the model provided by Equation (5) is then rewritten as the product of s likelihoods, each depending on only one sub-sample:

  $${\displaystyle L(q_{{\tau }_1,I_1},\dots ,q_{{\tau }_s,I_s},\sigma ,p;x_{I_1},y_{I_1},\dots ,y_{I_s})=\prod _{j=1}^s\prod _{i_j\in I_j}{\displaystyle \frac{p(1-p)}{\sigma }}exp\{-{\displaystyle \frac{{\rho }_p(y_{i_j}-(q_{{\tau }_j}\left(x_{i_j}\right)-\sigma g({\tau }_j,p)))}{\sigma }}\}}$$

3. Hamiltonian Monte Carlo

Markov chain Monte Carlo (MCMC) was employed to simulate state distributions for an idealized molecular system. An alternative molecular dynamics approach was implemented, featuring deterministic molecular movement adhering to Newton’s law of motion elegantly formalized as Hamiltonian dynamics. The inaugural MCMC algorithm incorporating Hamiltonian dynamics was the static HMC [36]. HMC [37,38] has favorable properties in complex models. The sampling from Metropolis–Hastings [39] and Gibbs [40] relies on random samples from an easy-to-sample distribution of proposals or conditional densities. In complex problems, these algorithms may not give good results; for example, random walks have difficulty “exploring” distributions. The purpose of the HMC algorithm is to use a different kind of sampling that is expected to more effectively “explore” the posterior distribution. Moreover, the HMC requires the differentiability of the density function. According to [41], HMC’s strength is its ability to explore high-dimensional parameter spaces more effectively. By utilizing gradient information from the posterior distribution, HMC avoids the random-walk nature of traditional MCMC methods. This results in a faster convergence, lower autocorrelation, and enhanced mixing, which provides more accurate estimates of the posterior distribution using fewer samples (a simple example is presented in Appendix B). In addition, HMC outperforms Gibbs sampling and Metropolis–Hastings by efficiently exploring high-dimensional target distributions without succumbing to random walk behavior or sensitivity to correlated parameters [42]. Beyond its ability to manage complex and multimodal distributions, HMC’s versatility extends to diverse fields, making it a useful tool in areas such as machine learning, physics, and more.

HMC, essentially a variant of Metropolis–Hastings, draws inspiration from a physical analogy with a distinct approach to proposal generation. The dynamics can be visualized in two dimensions, such as that of a frictionless puck that slides across a variable-height surface. The state of this system comprises the puck position, provided by a two-dimensional vector z, and the puck momentum, provided by a two-dimensional vector p (its mass times its speed). The puck’s potential energy, $U\left(z\right)$, is proportional to its present position’s ground height, while its kinetic energy, $K\left(p\right)$, is equal to $p^2/\left(2m\right)$, where m is the puck’s mass. The puck passes at a steady velocity, equivalent to $p/m$, at a level part of the surface. The potential energy for these factors forms the negative logarithm of the probability density.

This physical analogy serves as a conceptual tool for understanding the dynamic interplay between position and momentum in the system, providing valuable insight into the foundational principles of the HMC algorithm. Subsequently, in HMC applications, Hamiltonian functions are commonly employed and can be represented as follows:

$$H(z,p)=U\left(z\right)+K\left(p\right)$$

(8)

Here, $U\left(z\right)$ is called potential energy and is defined as minus the distribution log probability density for z$\left(U\right(z)=-log(l\left(x\right|z)\times g(z\left)\right)$ that we want to sample, plus any useful constant. $K\left(p\right)$ is called kinetic energy and is generally described as

$$K\left(p\right)=p^T{M}^{-1}p/2$$

(9)

Here, M is a symmetric, positive-definite “mass matrix”.

After identifying our potential and kinetic energy functions and introducing momentum variables that are relevant to our position variables, we can simulate a Markov chain in which each iteration takes place:

• Randomly sample $K\left(p\right)$ momentum variables, regardless of present position;
• Update Metropolis using simulated Hamiltonian dynamics (implemented by the leapfrog method, for example).

More specifically, the momentum variable determines where the position z passes during this second dynamic proposal, and the gradient of $U\left(z\right)$ determines how the momentum p changes in accordance with the equations of Hamilton:

$$\begin{array}{c}\hfill \frac{d{z}_i}{dt}=\frac{\partial H}{\partial p_i},\forall i\in \{1,\dots ,k\},\\ \hfill \frac{d{p}_i}{dt}=\frac{\partial H}{\partial z_i},\forall i\in \{1,\dots ,k\}.\end{array}$$

Some properties of Hamiltonian dynamics are reversibility, conservation of the Hamiltonian, volume preservation, and symplecticness (for more details, see [38]).

• Leapfrog Method:

Hamilton’s equations need to be approximated by discretizing time for software execution using some tiny step size, $\mathsf{\epsilon }$. We do this by assuming that the Hamiltonian has the form $H(z,p)=U\left(z\right)+K\left(p\right)$, as in Equation (8), and by assuming that kinetic energy has the form $K\left(p\right)=p^T{M}^{-1}p/2$, as in Equation (9); moreover, M is diagonal, with $m_1,\dots ,m_d$ diagonal components, so that

$$K\left(p\right)=\sum _{i=1}^d\frac{p_i^2}{2m_i}$$

The leapfrog method works as follows:

•  

  $$\begin{array}{c}\hfill p_i(t+\mathsf{\epsilon }/2)=p_i\left(t\right)-(\mathsf{\epsilon }/2)\frac{\partial U}{\partial z_i}\left(z\left(t\right)\right)\end{array}$$
•  

  $$\begin{array}{c}\hfill z_i(t+\mathsf{\epsilon })=z_i\left(t\right)+\mathsf{\epsilon }\frac{p_i(t+\mathsf{\epsilon }/2)}{m_i}\end{array}$$
•  

  $$\begin{array}{c}\hfill p_i(t+\mathsf{\epsilon })=p_i(t+\mathsf{\epsilon }/2)-(\mathsf{\epsilon }/2)\frac{\partial U}{\partial z_i}\left(z(t+\mathsf{\epsilon })\right)\end{array}$$

For the momentum variables, we begin with a half step, then we perform a complete step for the position variables using the current momentum variables’ values, and lastly, we perform another half step for the momentum variables using the current position variables’ values.

• HMC Algorithm:

Each iteration of the HMC algorithm has two steps:

• New values are drawn randomly from their Gaussian distribution for the momentum variables, irrespective of the current position variables values.
• Updating Metropolis is performed using Hamiltonian dynamics to suggest a new state. Starting with the current state $(z,p)$, Hamiltonian dynamics is simulated using the leapfrog method (or some other reversible method that preserves volume) for L steps, using a step size of $\mathsf{\epsilon }$. At the end of this L-step trajectory, the momentum variables are then negated, resulting in a suggested state $(z^{*},p^{*})$. This proposed state is accepted as the next state in the Markov chain with probability:

  $$\begin{array}{c}\hfill min[1,exp(-H(z^{*},p^{*})+H(z,p))]=min[1,exp(-U\left(z^{*}\right)+U\left(z\right)-K\left(p^{*}\right)+K\left(p\right))].\end{array}$$

  If the proposed state is not approved (i.e., dismissed), the following state will be the same as the present state.

4. Numerical Results

In this section, we will perform a Bayesian quantile regression procedure to estimate simultaneous regression quantiles using HMC. HMC will be used through the probabilistic programming language STAN [43,44,45] program to estimate simultaneous linear quantiles. STAN provides a flexible framework for specifying and fitting a wide range of Bayesian models. Numerous probability densities, matrix operations, and numerical ODE solvers are supported by the expressive language STAN [46].

Two different models will be considered for the $\tau$-th quantile for the model provided by (5):

• Univariate linear quantile: $q_{\tau }\left(X\right)=1+2\mathit{X}$ with $X\sim {\mathcal{U}}_{[0,1]}$.
• Multivariate linear quantile: $q_{\tau }\left(X\right)=1+X_l^{\prime }{\beta }_{\tau }$, with ${\mathit{\beta }}_{\tau }\in {\mathbb{R}}^d$, $d=10$ and either $X_l\sim {\mathcal{U}}_{[-1,1]}$ or ${\mathit{X}}_l\sim \mathcal{N}(0,1)$, $\forall l\in \{1,\dots ,d\}$.

Our methodology involves the utilization of the state-of-the-art STAN program, leveraging Hamiltonian Monte Carlo for statistical modeling and high-performance statistical computing. This platform has been extensively employed across diverse disciplines, including the social, biological, and physical sciences, engineering, and industry, for tasks such as statistical modeling, data analysis, and prediction. Given the inherent relationship between any two ALD quantiles, we recognize that distinct quantiles vary solely in terms of their intercepts ${\beta }_{{\tau }_j}$. Consequently, all quantiles of interest have the same slope $({\beta }_{{\tau }_1}=\cdots ={\beta }_{{\tau }_s}={\beta }_p)$; then, the unknown vector parameter is $\mathit{\beta }$. The prior of $\mathit{\beta }$ is Gaussian, with mean 0 and variance 1. For further details about the STAN code, refer to the Appendix A.

4.1. Univariate Linear Quantile

Suppose the error terms ${ϵ}_{i=1,\dots ,T}\stackrel{i.i.d}{\sim }\mathrm{ALD}(0,0.1,0.75)$. As the Bayesian approach requires a likelihood, we use the ALD to make the Bayesian inference, and our quantiles of interest are $\tau =0.25$ and $\tau =0.5$; hence, we use the relationship between the two ALDs mentioned in Section 2.

We independently generate 300 observations, using R, issued from the model defined in Equation (5). Since we are using simulated data, the theoretical quantiles are ${\beta }_{0,0.25}=0.56$, ${\beta }_{0,0.5}=0.83$, and ${\beta }_{1,0.25}={\beta }_{1,0.5}=2$.

To study the quality of the estimate of the quantiles, we calculate the root mean square error:

$$RMSE=\sqrt{\sum _{i=1}^n{(q_{\tau }-{\widehat{q}}_{\tau })}^2},$$

where $q_{\tau }$ is the theoretical quantile and ${\widehat{q}}_{\tau }$ is the posterior mean quantile regression.

To assess the impact of data variance on estimation, we assumed that the error terms ${ϵ}_{i=1,\dots ,T}\stackrel{i.i.d}{\sim }\mathrm{ALD}(0,0.3,0.75)$. The obtained results are presented in

Table 1. Mean, standard deviation, theoretical quantile, and RMSE of $\beta$ for the univariate model with ${ϵ}_{i=1,\dots ,T}\stackrel{i.i.d}{\sim }\mathit{ALD}(0,0.1,0.75)$ and ${ϵ}_{i=1,\dots ,T}\stackrel{i.i.d}{\sim }\mathit{ALD}(0,0.3,0.75)$.

$\mathit{\beta }$	Mean	SD	Theoretical Quantile	RMSE
${\mathit{ϵ}}_{\mathit{i}=\mathit{1},\dots ,\mathit{T}}\stackrel{\mathit{i}.\mathit{i}.\mathit{d}}{\sim }\mathrm{ALD}(\mathbf{0},\mathbf{0}.\mathbf{1},\mathbf{0}.\mathbf{75})$
${\beta }_{0,0.25}$	0.44	0.05	0.56	0.12
${\beta }_{0,0.5}$	0.75	0.04	0.83	0.08
${\beta }_1$	2.1	0.01	2	0.1
${\mathit{ϵ}}_{\mathit{i}=\mathit{1},\dots ,\mathit{T}}\stackrel{\mathit{i}.\mathit{i}.\mathit{d}}{\sim }\mathrm{ALD}(\mathbf{0},\mathbf{0}.\mathbf{3},\mathbf{0}.\mathbf{75})$
${\beta }_{0,0.25}$	−0.63	0.15	0.56	1.19
${\beta }_{0,0.5}$	−0.39	0.14	0.83	1.22
${\beta }_1$	2.22	0.02	2	0.22

.

We run our algorithm for 20,000 iterations, with half of them being burn-in. Gelman–Rubin diagnostic (R-hat) values are depicted in Figure 1, with $\widehat{R}$ approaching 1, indicating the convergence of multiple chains. The densities of the estimated $\beta$s are depicted in Figure 2, providing insight into the distribution patterns.

Furthermore, Figure 3 illustrates that the autocorrelation of each chain converges to 0. While comparing the HMC method used in this paper with the Metropolis–Hastings within Gibbs algorithm used in [27], we find that using HMC exhibits lower autocorrelation at earlier lags (around 0 after 10 lags) compared to the Metropolis–Hastings within Gibbs algorithm, where autocorrelation remains around 0 only after 50 lags. This suggests that HMC achieves greater independence between samples sooner, indicating potentially more efficient sampling and better exploration of the distribution space compared to the algorithm used in [27].

Additionally, the examination of the trace plots in Figure 4 reveals consistent and stable behavior in the MCMC chains, showcasing no discernible oscillation, drift, or other abnormalities. This pattern suggests a thorough exploration of the parameter space, indicating successful convergence to the target distribution.

The small variance observed in Table 1 further supports the reliability of the results. Furthermore, the Monte Carlo standard error (MCSE) is 0 for all $\beta$s, indicating that the estimated values are likely close to the true values, as shown in Table 1. In addition, when the variance of the data is larger, the estimation worsens (higher RMSE and higher variance). This is normal, as a larger variance makes it harder to estimate the parameters.

It should be noted that diagnostics can only reliably be used to determine a lack of convergence and not detect convergence [47]. Therefore, from the obtained results, the trace plots do not seem to indicate non-stationary behavior.

4.2. Multivariate Linear Quantile

Suppose the error terms ${ϵ}_{i=1,\dots ,T}\stackrel{i.i.d}{\sim }\mathrm{ALD}(0,0.1,0.75)$. As the Bayesian approach requires a likelihood, we use the ALD to make the Bayesian inference, and the quantiles of interest are $\tau =0.1$ and $\tau =0.12$; hence, we use the relationship between the two ALDs mentioned in Section 2.

We independently generate 300 and 3000 observations, using R, issued from the model defined in Equation (5) with $X^{\prime }\beta =1+2x_1+1.5\ast x_2+x_3+6x_4+2x_5+3x_6+2x_7+x_8+6x_9+4x_{10}$.

Since we are using simulated data, the theoretical quantiles are: ${\beta }_{0,0.1}=0.1940388$, ${\beta }_{0,0.12}=0.2669674$, ${\beta }_{1,0.1}={\beta }_{1,0.12}=2$, ${\beta }_{2,0.1}={\beta }_{2,0.12}=1.5$, ${\beta }_{3,0.1}={\beta }_{3,0.12}=1$, ${\beta }_{4,0.1}={\beta }_{4,0.12}=6$, ${\beta }_{5,0.1}={\beta }_{5,0.12}=2$, ${\beta }_{5,0.1}={\beta }_{5,0.12}=2$, ${\beta }_{6,0.1}={\beta }_{6,0.12}=3$, ${\beta }_{7,0.1}={\beta }_{7,0.12}=2$, ${\beta }_{8,0.1}={\beta }_{8,0.12}=1$, ${\beta }_{9,0.1}={\beta }_{9,0.12}=6$, and ${\beta }_{10,0.1}={\beta }_{10,0.12}=4$. We run our algorithm for 20,000 iterations, with half of them being burn-in. Despite Y being univariate, our model is multivariate as we estimate quantiles at different levels ($\tau =0.1$ and $\tau =0.12$), treating each as a distinct response variable.

The Gelman–Rubin diagnostic (R-hat) values are depicted in Figure 5, with $\widehat{R}$ approaching 1, indicating the convergence of multiple chains. The histograms of the estimated $\beta$s are depicted in Figure 6, providing insights into the distribution patterns.

Furthermore, Figure 7 illustrates that the autocorrelation of each chain converges to 0. While comparing the HMC method used in this paper with the Metropolis–Hastings within Gibbs algorithm used in [27], we find that using HMC exhibits lower autocorrelation at earlier lags (around 0 before 10 lags) compared to the Metropolis–Hastings within Gibbs algorithm, where autocorrelation remains around 0 only after 10 lags. This suggests that HMC achieves greater independence between samples sooner, indicating potentially more efficient sampling and better exploration of the distribution space compared to the algorithm used in [27].

Additionally, an examination of the trace plots in Figure 8 reveals consistent and stable behavior in the MCMC chains, showcasing no discernible oscillation, drift, or other abnormalities. This pattern suggests a thorough exploration of the parameter space, indicating successful convergence to the target distribution.

The small variance observed in Table 1 further supports the reliability of the results. Furthermore, the MCSE is 0 for all $\beta$s, indicating that the estimated values are likely close to the true values, as shown in

Table 2. Mean, standard deviation, theoretical quantile, and RMSE of $\beta$ for the multivariate model with 300 and 3000 observations.

$\mathit{\beta }$	Mean	SD	Theoretical Quantile	RMSE
300 Observations
${\beta }_{0,0.1}$	0.15	0.05	0.19	0.4
${\beta }_{0,0.12}$	0.24	0.04	0.27	0.03
${\beta }_1$	1.98	0.02	2	0.02
${\beta }_2$	1.51	0.02	1.5	0.01
${\beta }_3$	0.98	0.02	1	0.02
${\beta }_4$	6.02	0.02	6	0.01
${\beta }_5$	2.01	0.02	2	0.01
${\beta }_6$	2.97	0.01	3	0.03
${\beta }_7$	2	0.01	2	0
${\beta }_8$	1	0.01	1	0
${\beta }_9$	6.01	0.01	6	0.1
${\beta }_{10}$	3.98	0.01	4	0.02
3000 Observations
${\beta }_{0,0.1}$	0.17	0.01	0.19	0.02
${\beta }_{0,0.12}$	0.25	0.02	0.27	0.02
${\beta }_1$	1.99	0.02	2	0.01
${\beta }_2$	1.5	0.01	1.5	0
${\beta }_3$	0.99	0.01	1	0.01
${\beta }_4$	6.01	0.01	6	0.01
${\beta }_5$	2	0.01	2	0
${\beta }_6$	3	0.01	3	0.01
${\beta }_7$	2.99	0.01	2	0.01
${\beta }_8$	1	0.01	1	0
${\beta }_9$	5.99	0.01	6	0.1
${\beta }_{10}$	4	0.01	4	0

. The sampler parameters for our model were configured with a $n_{leapfrog}$ value of approximately 50 for all chains, indicating that each iteration of the sampler took around 50 leapfrog steps to explore the parameter space.

From the obtained results, the trace plots do not seem to indicate non-stationary behavior.

Furthermore, upon comparing the results presented in Table 2 for 300 observations with those for 3000 observations, we observe an improvement in estimation accuracy with higher observation counts, indicated by lower RMSE values and reduced variance across most quantiles. This phenomenon is expected, as larger observation sizes generally lead to improved estimations.

5. Conclusions

This article introduces a Bayesian methodology for simultaneous regression quantile estimation. The framework assumes an ALD for the error term in the quantile regression model and leverages the inter-quantile relationships within the ALD. This approach allows for a comprehensive characterization of the model likelihood across a range of quantiles, providing a nuanced understanding of the estimation process.

Diverging from the methodology presented in [21], which addresses simple quantiles and regression quantiles across diverse data types, our proposed approach is tailored specifically to ALD data in the context of simultaneous regression quantiles. The utilization of HMC for estimation yields promising results, as demonstrated by our RMSE interpretation. However, it is important to note that the current formulation of the HMC method, as described in this study, is not explicitly designed to handle categorical predictors, making it an interesting area for future investigation.

Our estimation procedure, facilitated by HMC, enables the fitting of parametric quantile functions in both univariate linear and multivariate linear quantile settings. The simulation study validates the convergence of HMC algorithms, supported by diagnostic results. Our study demonstrates that HMC achieves greater independence between samples sooner compared to the algorithm used in [27], suggesting potentially more efficient sampling and better exploration of the distribution space. This finding highlights the effectiveness of HMC in Bayesian analysis and underscores its potential for improving computational efficiency in future research. This work represents a substantial advancement towards achieving a fully Bayesian estimation of multiple quantiles.

An exciting perspective to explore for future research involves extending the methodology introduced in [27] to accommodate any conditional distribution of $Y|\mathit{x}$. Such an extension would broaden the applicability of the proposed approach, enhancing its adaptability to various modeling scenarios. Another perspective within this framework involves conducting sensitivity analyses on key parameters and assumptions related to the HMC algorithm, such as step size and the number of leapfrog steps, to provide further insights into its robustness and effectiveness across various scenarios.