LLM-Guided Ensemble Learning for Contextual Bandits with Copula and Gaussian Process Models

Abstract

Contextual multi-armed bandits (CMABs) are vital for sequential decision-making in areas such as recommendation systems, clinical trials, and finance. We propose a simulation framework integrating Gaussian Process (GP)-based CMABs with vine copulas to model dependent contexts and GARCH processes to capture reward volatility. Rewards are generated via copula-transformed Beta distributions to reflect complex joint dependencies and skewness. We evaluate four policies—ensemble, Epsilon-greedy, Thompson, and Upper Confidence Bound (UCB)—over 10,000 replications, assessing cumulative regret, observed reward, and cumulative reward. While Thompson sampling and LLM-guided policies consistently minimize regret and maximize rewards under varied reward distributions, Epsilon-greedy shows instability, and UCB exhibits moderate performance. Enhancing the ensemble with copula features, GP models, and dynamic policy selection driven by a large language model (LLM) yields superior adaptability and performance. Our results highlight the effectiveness of combining structured probabilistic models with LLM-based guidance for robust, adaptive decision-making in skewed, high-variance environments.

1. Introduction

CMABs provide a principled framework for sequential decision-making under uncertainty, where an agent repeatedly selects actions (arms) based on observed context to maximize cumulative reward [1,2]. CMABs have been successfully applied in diverse fields including online advertising, recommendation systems, clinical trials, and financial trading [3].

The MAB framework balances exploration and exploitation to optimize decisions under uncertainty [4,5]. When side information or covariates are available at each decision step, contextual bandits exploit this additional information to improve learning and decision quality [2].

GP regression has emerged as a powerful tool for modeling unknown reward functions in CMABs [6,7,8]. GPs offer a nonparametric and probabilistic approach, capturing uncertainty naturally and enabling effective policies like Thompson Sampling (TS) and UCB. However, many practical scenarios involve complex, heteroskedastic, and dependent context variables that violate standard GP assumptions of independent and homoskedastic inputs.

To better reflect these complexities, we simulate covariates exhibiting volatility and nonlinear dependencies. Volatility is modeled using GARCH processes [9,10], capturing time-varying variance patterns common in finance and other domains. Dependence structures among features are represented with vine copulas [11,12], which flexibly model complex, high-dimensional dependencies beyond Gaussian assumptions. After applying the probability integral transform to normalize covariates, these enriched contexts create realistic and challenging bandit environments. Reward generation uses Beta distributions parameterized by the simulated volatility to encode heterogeneity and skewness [13]. Ref. [14] considered a novel generative-adversarial network algorithm, which is a deep learning algorithm, meticulously trained for the seismic inversion process. Recent breakthroughs in LLMs, such as GPT-4 [15], demonstrate remarkable ability in contextual understanding, summarization, and reasoning. Motivated by this, our approach integrates LLMs to assist both reward prediction and dynamic policy selection. The LLM reviews recent bandit performance and recommends strategies—including TS, UCB, Epsilon-Greedy, or a hybrid ensemble—to adaptively improve decision-making, especially in nonstationary or uncertain settings.

Building on prior work that combines copula-based context generation with GP reward modeling [16], our framework uniquely integrates LLM-driven guidance, yielding a flexible and interpretable model that leverages both structured probabilistic inference and deep language understanding. Simulation experiments with 10,000 independent runs demonstrate improved cumulative rewards and reduced regret compared to standard baselines, underscoring the benefits of this hybrid approach.

This paper addresses existing gaps by proposing a comprehensive simulation and modeling framework that functions as follows:

• Employs vine copulas to capture rich, nonlinear dependencies among contextual variables;
• Incorporates GARCH processes to simulate heteroskedastic, non-Gaussian reward noise;
• Utilizes GP-based bandit policies enhanced with dynamic, LLM-guided policy selection;
• Demonstrates through extensive experiments the superiority of this integrated approach in managing skewed, heavy-tailed, and volatile reward distributions.

Our contributions advance the state of the art by bridging structured probabilistic modeling with powerful LLM-based reasoning to improve adaptability and robustness in complex CMAB scenarios.

The rest of the paper is organized as follows. Section 2 introduces our model and reviews related literature on bandits, GPs, copulas, GARCH models, and LLMs. Section 3 details the simulation setup and empirical results. Section 4 concludes with final remarks and directions for future work.

2. Methods

2.1. Overview

We propose a hybrid simulation framework that integrates heteroskedastic contextual generation, copula-based dependence modeling, GP reward estimation, and LLM-guided policy selection. This framework enables evaluation and comparison of adaptive contextual bandit algorithms under realistic, complex environments.

2.2. Context Generation via GARCH and Copula Modeling

2.2.1. GARCH(1,1) Volatility Modeling

To capture time-varying volatility and heteroskedasticity in the context features ${\mathbf{x}}_t=(x_{t1},\dots ,x_{t,d})$, we simulate each dimension $j=1,\dots ,d$ independently using a standard GARCH(1,1) process [9,13]:

$${\sigma }_{t,j}^2=\omega +\alpha {ϵ}_{t-1,j}^2+\beta {\sigma }_{t-1,j}^2,$$

(1)

where

• ${\sigma }_{t,j}^2$ is the conditional variance of dimension j at time t;
• $\omega >0$ is the long-term variance constant;
• $\alpha \ge 0$ measures sensitivity to recent squared innovations ${ϵ}_{t-1,j}^2$;
• $\beta \ge 0$ captures the persistence of past variance ${\sigma }_{t-1,j}^2$,

with the stationarity condition $\alpha +\beta <1$ ensuring that variance remains bounded over time [13].

The innovations ${ϵ}_{t,j}$ are assumed to be independent and identically distributed standard normal variables:

$${ϵ}_{t,j}\sim \mathcal{N}(0,1).$$

The observed contextual feature for dimension j at time t is generated as:

$$x_{t,j}={\sigma }_{t,j}·z_{t,j},z_{t,j}\sim \mathcal{N}(0,1),$$

(2)

where $z_{t,j}$ is an independent noise term. This standard GARCH(1,1) setup captures typical volatility clustering found in financial and many real-world time series [9,17].

2.2.2. Copula-Based Dependence Modeling

To model nonlinear and flexible dependencies among the d contextual features, we apply a vine copula approach [11,12]. After transforming the marginals to uniform variables via the probability integral transform (PIT), the vine copula captures complex joint dependence structures beyond Gaussian correlation assumptions.

This integration of GARCH-driven heteroskedastic marginals with vine copula dependence provides a realistic and challenging environment for contextual bandit algorithms. While the GARCH model and vine copula construction follow established methodologies [11,13], our contribution lies in coupling these components within a unified simulation framework to generate realistic, time-varying, and dependent contexts for evaluating adaptive bandit policies.

2.2.3. Copula-Based Dependence Structure

To model cross-sectional dependence among the d contextual features at each time step t, we employ a copula-based approach [11,18]. This separates marginal behavior from the dependence structure, offering greater flexibility in simulating realistic multivariate contexts.

First, each contextual variable $x_{t,j}$ is transformed into a uniform score using the empirical cumulative distribution function (ECDF), which serves as a rank-based approximation:

$$u_{t,j}={\displaystyle \frac{\mathrm{rank}\left(x_{t,j}\right)}{T+1}},$$

(3)

where

• $u_{t,j}\in (0,1)$ is the pseudo-observation of the j-th variable at time t;
• $\mathrm{rank}\left(x_{t,j}\right)$ is the rank of $x_{t,j}$ among $\{x_{1,j},\dots ,x_{T,j}\}$;
• T is the total number of observations.

This yields the vector ${\mathbf{u}}_t=(u_{t,1},\dots ,u_{t,d})\in {(0,1)}^d$, which represents the transformed context in the copula space.

To model dependencies among these uniform variables, we fit a regular vine copula (R-vine) to the observed ${\left\{{\mathbf{u}}_t\right\}}_{t=1}^T$. An R-vine decomposes the joint copula density into a cascade of bivariate (pair) copulas organized in a sequence of linked trees, allowing rich and flexible modeling of high-dimensional dependencies [11,12].

Once the R-vine copula C is estimated, we apply the inverse copula transform as follows:

$${\mathbf{v}}_t=C^{-1}\left({\mathbf{u}}_t\right),$$

(4)

where $C^{-1}$ denotes the inverse transform (sampling from the copula distribution). The resulting ${\mathbf{v}}_t\in {\mathbb{R}}^d$ encodes dependencies among contextual features and is used as input to the contextual bandit policy.

This copula-based preprocessing allows the simulation of dependent and potentially non-Gaussian context vectors with realistic structure and tail dependence. Such features are often found in domains like finance, environment, and health care, where multivariate covariates are rarely independent.

2.3. Reward Simulation

For a contextual bandit problem with K arms, we simulate bounded stochastic rewards $r_t^{\left(k\right)}\in [0,1]$ at each time step t using the inverse cumulative distribution function (CDF) of the Beta distribution. The reward for each arm is generated based on its associated copula-transformed contextual variable $v_{t,k}\in (0,1)$, as follows:

$$r_t^{\left(k\right)}=F_{\mathrm{Beta}}^{-1}(v_{t,k};\alpha ,\beta ),$$

(5)

where

• $r_t^{\left(k\right)}$ denotes the reward for arm $k\in \{1,\dots ,K\}$ at time t;
• $v_{t,k}$ is the PIT-transformed contextual input for arm k, obtained through a copula model to capture dependence across arms;
• $F_{\mathrm{Beta}}^{-1}(·;\alpha ,\beta )$ is the inverse CDF (quantile function) of the Beta distribution;
• $\alpha >0$ and $\beta >0$ are shape parameters controlling the distribution’s skewness and variance.

The Beta distribution is flexible and supported on the interval $[0,1]$, making it well-suited for modeling bounded rewards. By adjusting the following shape parameters:

• Higher values of $\alpha$ relative to $\beta$ result in right-skewed distributions (more probability mass near 1);
• Higher values of $\beta$ relative to $\alpha$ lead to left-skewed distributions (more mass near 0);
• Equal values of $\alpha$ and $\beta$ yield symmetric distributions (e.g., the uniform distribution when $\alpha =\beta =1$).

This reward mechanism allows context-dependent simulation of non-Gaussian, heteroskedastic, and skewed reward profiles, reflecting many real-world environments such as treatment effects, conversion probabilities, or user responses.

2.4. GP Reward Estimation

To model the unknown reward function $f_k(·)$ for each arm $k\in \{1,\dots ,K\}$, we use independent GP regressions. Each GP is trained on the PIT-transformed contexts ${\mathbf{v}}_t\in {[0,1]}^d$ and the corresponding rewards $r_t^{\left(k\right)}$, using the training dataset ${\left\{({\mathbf{v}}_t,r_t^{\left(k\right)})\right\}}_{t=1}^{T_{\mathrm{train}}}$. This approach follows the probabilistic regression framework outlined in [8]:

$$f_k\left(\mathbf{v}\right)\sim \mathcal{GP}(m_k\left(\mathbf{v}\right),k_k(\mathbf{v},{\mathbf{v}}^{\prime })),$$

(6)

where

• $f_k\left(\mathbf{v}\right)$ is the latent reward function for arm k evaluated at input $\mathbf{v}\in {[0,1]}^d$;
• $\mathcal{GP}(·,·)$ denotes a GP prior;
• $m_k\left(\mathbf{v}\right)=\mathbb{E}\left[f_k\left(\mathbf{v}\right)\right]$ is the mean function, often assumed to be zero;
• $k_k(\mathbf{v},{\mathbf{v}}^{\prime })=\mathrm{Cov}(f_k\left(\mathbf{v}\right),f_k\left({\mathbf{v}}^{\prime }\right))$ is the covariance (kernel) function, typically chosen as the squared exponential (RBF) kernel, as follows:

  $$k_k(\mathbf{v},{\mathbf{v}}^{\prime })={\sigma }_f^2\exp \left(-{\displaystyle \frac{1}{2{\mathsf{\ell }}^2}}{\parallel \mathbf{v}-{\mathbf{v}}^{\prime }\parallel }^2\right),$$

  with hyperparameters ℓ (length-scale) and ${\sigma }_f^2$ (signal variance).

After training, the GP provides closed-form posterior predictive distributions for unseen inputs ${\mathbf{v}}_{*}$, yielding the following:

• Predictive mean ${\mu }_k\left({\mathbf{v}}_{*}\right)=\mathbb{E}[f_k\left({\mathbf{v}}_{*}\right)\mid {\mathcal{D}}_k]$;
• Predictive variance ${\sigma }_k^2\left({\mathbf{v}}_{*}\right)=\mathrm{Var}[f_k\left({\mathbf{v}}_{*}\right)\mid {\mathcal{D}}_k]$,

where ${\mathcal{D}}_k={\left\{({\mathbf{v}}_t,r_t^{\left(k\right)})\right\}}_{t=1}^{T_{\mathrm{train}}}$ is the training data for arm k.

These predictive statistics are used in downstream bandit policies—e.g., UCB and TS—allowing principled exploration under uncertainty.

2.5. Bandit Policies and LLM Integration

2.5.1. Classical Policies

We compare three classical MAB policies that trade off between exploration (trying new arms) and exploitation (selecting the current best arm), all using GP posterior reward models.

• TS: For each round t, the algorithm draws a sample from the GP posterior predictive distribution for each arm k given the current context ${\mathbf{v}}_t$, and selects the arm with the highest sampled reward:

  $$a_t=\arg \underset{k}{\max }{\tilde{r}}_t^{\left(k\right)},{\tilde{r}}_t^{\left(k\right)}\sim \mathcal{N}({\mu }_k\left({\mathbf{v}}_t\right),{\sigma }_k^2\left({\mathbf{v}}_t\right)),$$

  where ${\mu }_k\left({\mathbf{v}}_t\right)$ and ${\sigma }_k^2\left({\mathbf{v}}_t\right)$ are the GP posterior predictive mean and variance for arm k. This method balances exploration and exploitation through posterior sampling [7].
• UCB: At each time t, the algorithm selects the arm that maximizes an upper confidence bound:

  $$a_t=\arg \underset{k}{\max }\left[{\mu }_k\left({\mathbf{v}}_t\right)+\kappa {\sigma }_k\left({\mathbf{v}}_t\right)\right],$$

  where

  –

  ${\mu }_k\left({\mathbf{v}}_t\right)$ is the GP posterior predictive mean;

  –

  ${\sigma }_k\left({\mathbf{v}}_t\right)$ is the posterior predictive standard deviation;

  –

  $\kappa >0$ is a tunable parameter that controls the degree of exploration.

  Larger values of $\kappa$ favor more exploration. This approach encourages arms with either high expected reward or high uncertainty [1].
• Epsilon-Greedy: This strategy selects the empirically best arm (highest predicted mean reward) with probability $1-\epsilon$, and a random arm with probability $\epsilon$ as follows:

  $$a_t=\left\{\begin{array}{cc}\arg \underset{k}{\max }{\mu }_k\left({\mathbf{v}}_t\right),\hfill & \mathrm{with}\mathrm{probability}1-\epsilon ,\hfill \\ \mathrm{random}\mathrm{arm}k,\hfill & \mathrm{with}\mathrm{probability}\epsilon ,\hfill \end{array}\right.$$

  where $\epsilon \in [0,1]$ is a hyperparameter controlling exploration [5].

2.5.2. Ensemble Policy with LLM Guidance

We propose an ensemble bandit policy that integrates structured statistical modeling via GPs with unstructured, semantic reasoning from a LLM. For a given context vector ${\mathbf{v}}_t\in {\mathbb{R}}^d$, arm index $k\in \{1,\dots ,K\}$, and historical regret summary $h_t$, the ensemble reward prediction is defined as follows:

$${\widehat{r}}_{t,k}=(1-w)·{\mu }_k\left({\mathbf{v}}_t\right)+w·\mathrm{LLM}\_\mathrm{pred}({\mathbf{v}}_t,k,h_t),$$

(7)

where

• ${\mu }_k\left({\mathbf{v}}_t\right)$ is the posterior predictive mean reward for arm k under the GP model, given the PIT-transformed context ${\mathbf{v}}_t$;
• $\mathrm{LLM}\_\mathrm{pred}({\mathbf{v}}_t,k,h_t)$ is the reward estimate from an LLM (e.g., GPT-4 [15]) conditioned on contextual information and a summary of recent regret or reward history $h_t$;
• $w\in [0,1]$ is a tunable weighting hyperparameter that balances the contribution of model-based statistical prediction and LLM-based semantic inference.

The ensemble decision rule selects the arm maximizing ${\widehat{r}}_{t,k}$ across all k as follows:

$$a_t=\arg \underset{k}{\max }{\widehat{r}}_{t,k}.$$

The parameter w can be:

• Fixed: chosen via offline validation (e.g., cross-validation or grid search) to minimize cumulative regret;
• Adaptive: dynamically updated during online learning, for instance by tracking the relative predictive accuracy or uncertainty of the GP and LLM components.

This flexible ensemble allows the policy to leverage the statistical rigor of GPs (which are effective when sufficient data is available) while integrating high-level semantic reasoning from the LLM, which may capture patterns not explicitly encoded in the structured data. Empirically, moderate values of w (e.g., $w\in [0.3,0.5]$) often achieve a strong balance between interpretability and adaptivity.

2.6. Adaptive Policy Selection via LLM

To enhance flexibility and responsiveness in sequential decision-making, we introduce an adaptive policy selection mechanism that leverages a LLM to choose a policy at each time step $t>t_0$. The candidate policy set includes the following:

$$\mathcal{P}=\left\{TS,UCB,\mathrm{Epsilon-greedy},\mathrm{ensemble}\right\}.$$

At each decision point t, the LLM is prompted with a natural language summary of recent performance metrics—such as cumulative reward, rolling regret, arm selection diversity, and historical policy usage. The model then recommends a policy ${\pi }_t\in \mathcal{P}$ to be applied at time t. This adaptive strategy enables the agent to dynamically shift between exploration and exploitation, or between structured and semantic reasoning, depending on evolving environmental conditions.

To mitigate premature reliance on the LLM and ensure a stable foundation of training data, we define a **delay threshold** $t_0\in \mathbb{N}$, such that:

• For $t\le t_0$, actions are selected using classical model-based policies (e.g., TS or UCB), relying exclusively on GP reward estimates ${\mu }_k\left({\mathbf{v}}_t\right)$.
• For $t>t_0$, the LLM is queried to determine either the next policy ${\pi }_t$ or to provide reward guidance through the ensemble formulation (see Equation (7)).

In our empirical setup, we fix $t_0=100$, which provides a sufficient warm-up phase for GPs to form reliable posterior distributions. This threshold can also be made **adaptive** in future work, e.g., by monitoring, as follows:

• convergence of posterior uncertainty (${\sigma }_k\left({\mathbf{v}}_t\right)$);
• stabilization of cumulative regret slopes;
• entropy of policy selection history.

Overall, this framework empowers the bandit agent to transition from purely statistical decision-making to a hybrid regime incorporating external semantic reasoning from foundation models like GPT-4 [15].

2.7. LLM Prompt Engineering

We integrate a LLM, specifically GPT-4 [15], into our contextual bandit simulation to support both reward prediction and adaptive policy selection. The LLM is queried using structured prompts that encapsulate the current context, reward history, and policy performance summaries. This allows the model to inject high-level reasoning and learned patterns into the decision-making process.

2.7.1. Reward Prediction Prompt

To estimate the expected reward for a given arm k under context ${\mathbf{v}}_t=(v_{t,1},v_{t,2},\dots ,v_{t,d})$$\in {\mathbb{R}}^d$, the LLM is queried with the following prompt structure:

Given context vector: $v_{t,1},v_{t,2},\dots ,v_{t,d}$, and arm index: k, along with recent reward history: [summary of last m rewards], predict the reward in the range [0, 1].

This prompt instructs the LLM to return a scalar estimate ${\widehat{r}}_{t,k}^{\mathrm{LLM}}\in [0,1]$, interpreted as the predicted reward for arm k at time t. The recent reward history provides grounding information to contextualize predictions. To ensure stability, the decoding temperature is set to a low value (e.g., 0.2–0.3), promoting deterministic outputs.

2.7.2. Policy Selection Prompt

For adaptive policy selection, the LLM receives a summary of recent performance statistics:

“Given recent bandit performance metrics: [tabular summary of last n rounds including rewards, regrets, and selected policies], choose one policy from: TS, UCB, Epsilon-greedy, ensemble. Respond with only the policy name.”

This prompt yields a discrete choice ${\pi }_t^{\mathrm{LLM}}\in \mathcal{P}$, where $\mathcal{P}=\{\mathrm{TS},\mathrm{UCB},\mathrm{Epsilon-greedy},$$\mathrm{ensemble}\}$. The LLM leverages observed patterns in regret and reward trends to recommend the most contextually appropriate policy. By restricting the output format, we reduce parsing complexity and enhance reliability.

2.7.3. Caching and Robustness

To reduce API latency and ensure deterministic behavior, we implement query-level caching. Identical prompts yield reused outputs. If the LLM returns invalid or missing results, the system defaults to a safe fallback strategy (e.g., Thompson Sampling), preserving robustness throughout the simulation pipeline.

2.8. Evaluation Metrics

We employ a suite of quantitative metrics to evaluate the performance of contextual bandit policies, focusing on learning efficiency, stability, and adaptivity.

2.8.1. Cumulative Regret

The primary metric is the cumulative regret $R_t$, defined as the total difference between the reward of the optimal arm and that of the selected arm up to time t as follows:

$$R_t=\sum _{s=1}^t\left(\underset{k\in \{1,\dots ,K\}}{\max }{r}_s^{\left(k\right)}-r_s^{\left(a_s\right)}\right),$$

where $r_s^{\left(k\right)}\in [0,1]$ is the reward that would have been received if arm k had been selected at time s, $a_s\in \{1,\dots ,K\}$ is the arm actually selected at time s.

Lower values of $R_t$ indicate more effective exploration and faster convergence to optimal behavior.

2.8.2. Cumulative and Average Reward

We also track the cumulative reward and average reward:

$${\mathrm{CumulativeReward}}_t=\sum _{s=1}^t{r}_s^{\left(a_s\right)},{\mathrm{AverageReward}}_t={\displaystyle \frac{1}{t}}\sum _{s=1}^t{r}_s^{\left(a_s\right)}.$$

These metrics reflect how well the algorithm exploits learned knowledge to obtain high rewards over time.

2.8.3. Confidence Intervals and Replications

To assess variability and statistical reliability, all simulations are repeated independently N times. At each time step t, we compute the sample mean ${\overline{m}}_t$ and standard deviation $s_t$ for a metric of interest (e.g., regret or reward). The associated 95% confidence interval is:

$${\mathrm{CI}}_t={\overline{m}}_t\pm 1.96·{\displaystyle \frac{s_t}{\sqrt{N}}}.$$

These intervals are used to visualize uncertainty and support statistical comparisons across policies.

2.8.4. Policy Switching Dynamics

For adaptive systems (e.g., LLM-guided policies), we track the frequency with which each policy $p\in \mathcal{P}$ is selected at time t. Let ${\pi }_t^{\left(p\right)}\in [0,1]$ denote the proportion of times policy p was selected at time t across replications. These usage patterns are visualized with stacked area plots to characterize dynamic policy adaptation.

2.8.5. Per-Arm Performance

We summarize per-arm statistics such as the following:

-

Mean reward received when each arm k was selected;

-

Selection frequency per arm under each policy p.

This breakdown reveals how well different policies identify and exploit specific arms, highlighting arm-specific strengths and weaknesses.

The integration of LLM prompt engineering with classical bandit policies, along with robust evaluation metrics, enables comprehensive benchmarking of adaptive bandit algorithms under complex contextual conditions, including heteroskedasticity and temporal dependencies.

3. Simulation Setup

We conduct a comprehensive simulation study to evaluate the performance of several contextual bandit policies under structured, time-varying reward heterogeneity. Our framework integrates functional volatility dynamics, copula-based dependency modeling, and GP regression for reward prediction. We also incorporate LLM-guided decision-making policies in a hybrid setup.

3.1. Context Generation via Functional Volatility and Copula Transformation

To simulate heteroskedastic, non-Gaussian, and dependent contextual variables, we construct a matrix of contextual features

$${\mathbf{X}}_t\in {\mathbb{R}}^{T\times d}$$

driven by volatility dynamics and copula-based dependence modeling. Here:

• ${\mathbf{X}}_t$ is the matrix of contextual features with T time points and d feature dimensions;
• Each row ${\mathbf{x}}_t\in {\mathbb{R}}^d$ represents the context vector at time t.

• Heteroskedastic Context Generation

Each contextual dimension follows an sGARCH(1,1) process, as follows:

$${\sigma }_{t,j}^2=\omega +{\alpha }_1{\epsilon }_{t-1,j}^2+{\beta }_1{\sigma }_{t-1,j}^2,$$

(8)

where

• ${\sigma }_{t,j}^2$ is the conditional variance of dimension j at time t;
• $\omega =0.1$ is the constant base volatility;
• ${\alpha }_1=0.15$ controls sensitivity to past shocks ${\epsilon }_{t-1,j}^2$;
• ${\beta }_1=0.7$ governs persistence of volatility;
• ${\epsilon }_{t,j}\sim \mathcal{N}(0,1)$ are independent standard normal innovations;
• Stationarity is ensured by ${\alpha }_1+{\beta }_1<1$.

Each dimension $j=1,\dots ,d$ evolves independently according to this process.

• Copula-Based Dependency Modeling

To capture dependencies among the d contextual features,

• We transform the GARCH-generated contexts into pseudo-observations $U_{t,j}\in [0,1]$ using empirical cumulative distribution function (CDF) ranks as follows:

  $$U_{t,j}={\displaystyle \frac{1}{T+1}}\sum _{s=1}^T\mathbf{1}(x_{s,j}\le x_{t,j}),$$

  ensuring uniform marginals suitable for copula modeling;
• A high-dimensional Vine Copula is fitted to the data ${\mathbf{U}}_t=(U_{t,1},\dots ,U_{t,d})$ using the RVineStructureSelect algorithm;
• This copula captures flexible nonlinear and tail dependencies between features;
• The probability integral transform (PIT) from the fitted copula yields the final transformed contextual inputs ${\tilde{\mathbf{U}}}_t$, which serve as bandit model inputs.

• Time-Varying Reward Volatility

To model nonstationary reward noise, we simulate GARCH(1,1) volatility for each arm’s reward noise:

$${\sigma }_{t,k}^2=\omega +\alpha {ϵ}_{t-1,k}^2+\beta {\sigma }_{t-1,k}^2,$$

(9)

where

• ${\sigma }_{t,k}^2$ is the conditional reward variance for arm k at time t;
• $\omega =0.1$, $\alpha =0.1$, and $\beta =0.8$ are fixed parameters;
• ${ϵ}_{t,k}$ represents reward noise innovations;
• These parameters model volatility persistence ($\alpha +\beta <1$) and responsiveness to recent shocks;
• This setup reflects financial-style volatility clustering to stress-test policy robustness under realistic, time-varying uncertainty.

We have the following attentions:

• Distinguish volatility variables: ${\sigma }_{t,j}^2$ for context dimension j, ${\sigma }_{t,k}^2$ for arm k’s reward volatility;
• GARCH introduces temporal dependence, while Vine Copula models cross-sectional dependence among contexts;
• Each arm’s reward noise is modeled separately to allow arm-specific volatility dynamics.

3.2. Reward Generation

To simulate nonlinear, context-dependent reward functions, we define the arm-level reward distributions via a monotonic mapping from the copula-transformed contextual variable ${\tilde{U}}_t$ to the inverse CDF of a Beta distribution as follows:

$$Y_{t,a}\sim {\mathrm{Beta}}^{-1}({\tilde{U}}_{t,a};\alpha ,\beta ),a=1,\dots ,A,$$

where

• $Y_{t,a}$ is the reward at time t for arm a;
• A is the total number of arms;
• ${\tilde{U}}_{t,a}\in [0,1]$ is the probability integral transform (PIT) value corresponding to arm a at time t, obtained from copula-based transformations of context;
• ${\mathrm{Beta}}^{-1}(·;\alpha ,\beta )$ is the quantile function (inverse cumulative distribution function) of the Beta distribution parameterized by positive shape parameters $\alpha ,\beta$.

The shape parameters $\alpha$ and $\beta$ control the skewness and concentration of the reward distribution as follows:

• $\alpha =2,\beta =5$ produces a left-skewed distribution (mass near 0), modeling sparse or conservative rewards;
• $\alpha =5,\beta =2$ produces a right-skewed distribution (mass near 1), modeling optimistic rewards with heavier tails.

This construction ensures that rewards are nonlinear and heterogeneous functions of the context and arm.

3.3. Reward Modeling via Gaussian Processes

Each arm-specific reward function $f_a(·)$ is modeled with a GP regression as follows:

$$f_a\left(\mathbf{x}\right)\sim \mathcal{GP}(0,k(\mathbf{x},{\mathbf{x}}^{\prime })),$$

where

• $f_a\left(\mathbf{x}\right)$ is the unknown reward function for arm a evaluated at context $\mathbf{x}\in {\mathbb{R}}^d$;
• The mean function is assumed zero for simplicity;
• The covariance kernel $k(\mathbf{x},{\mathbf{x}}^{\prime })$ is the squared exponential (RBF) kernel, given by

  $$k(\mathbf{x},{\mathbf{x}}^{\prime })={\sigma }_f^2\exp \left(-{\displaystyle \frac{\parallel \mathbf{x}-{\mathbf{x}}^{\prime }{\parallel }^2}{2l^2}}\right)+g{\delta }_{\mathbf{x}{\mathbf{x}}^{\prime }},$$

  where

  –

  ${\sigma }_f^2$ is the signal variance;

  –

  l is the length-scale controlling smoothness;

  –

  $g={10}^{-6}$ is a small noise term (nugget) added for numerical stability;

  –

  ${\delta }_{\mathbf{x}{\mathbf{x}}^{\prime }}$ is the Kronecker delta function (1 if $\mathbf{x}={\mathbf{x}}^{\prime }$, else 0).

Posterior means ${\mu }_{t,a}$ and variances ${\sigma }_{t,a}^2$ from the GP are used to guide uncertainty-aware exploration policies.

3.4. Bandit Policies Evaluated

We compare several contextual bandit policies. For each time t, given posterior mean ${\mu }_{t,a}$ and variance ${\sigma }_{t,a}^2$ for arm a, the policies are:

• TS: Sample a reward

  $${\tilde{Y}}_{t,a}\sim \mathcal{N}({\mu }_{t,a},{\sigma }_{t,a}^2),$$

  then select the arm with the highest sampled reward:

  $$a_t=\arg \underset{a}{\max }{\tilde{Y}}_{t,a}.$$
• UCB: Select the arm maximizing the UCB score:

  $$a_t=\arg \underset{a}{\max }\left({\mu }_{t,a}+\sqrt{2\log \left(t\right)·{\sigma }_{t,a}^2}\right),$$

  balancing exploitation and exploration.
• Epsilon-Greedy: With probability $\epsilon =0.1$, select a random arm; otherwise, select the arm with the highest posterior mean:

  $$a_t=\arg \underset{a}{\max }{\mu }_{t,a}.$$
• LLM-Guided Policies:

  –

  LLM-Reward: Uses predicted reward estimates from an LLM (mocked in simulation) to select the arm with highest predicted reward;

  –

  LLM-Policy: Dynamically selects among TS, UCB, Epsilon-greedy, or LLM-Reward policies by querying an LLM based on recent performance summaries.

3.5. Simulation Protocol

• Total time steps: $T=100$;
• Context dimension: $d=10$;
• Number of arms: $A=3$.

The first 70% of data is used to train arm-specific GP models, and the last 30% is reserved for bandit simulation. At each time t, given context ${\mathbf{x}}_t$, the selected arm $a_t$ receives reward $Y_{t,a_t}$.

3.6. Evaluation Metrics and Replications

We evaluate each policy over R = 10,000 Monte Carlo replications, computing the following:

• Mean Reward:

  $${\displaystyle \frac{1}{R}}\sum _{r=1}^R{Y}_{t,a_t}^{\left(r\right)},$$

  where $Y_{t,a_t}^{\left(r\right)}$ is the reward at time t for replication r.
• Cumulative Regret:

  $${\mathrm{Regret}}_t^{\left(r\right)}=\sum _{s=1}^t\left(\underset{a}{\max }{Y}_{s,a}^{\left(r\right)}-Y_{s,a_s}^{\left(r\right)}\right),$$

  representing the cumulative loss compared to always playing the best arm.
• Confidence Intervals (95%):

  $${\overline{x}}_t\pm 1.96{\displaystyle \frac{s_t}{\sqrt{R}}},$$

  where ${\overline{x}}_t$ is the empirical mean and $s_t$ is the sample standard deviation over replications.

3.7. Empirical Results

Table 1. Comparison of policy performance under different reward distributions.

Policy	Mean R	SD R	Mean Reg	SD Reg	Mean R	SD R	Mean Reg	SD Reg
	${\mathit{Y}}_{\mathit{t}\mathbf{,}\mathit{a}}\mathbf{\sim }{\mathbf{Beta}}^{-\mathbf{1}}\mathbf{(}\mathbf{2}\mathbf{,}\mathbf{5}\mathbf{)}$				${\mathit{Y}}_{\mathit{t}\mathbf{,}\mathit{a}}\mathbf{\sim }{\mathbf{Beta}}^{\mathbf{-}\mathbf{1}}\mathbf{(}\mathbf{5}\mathbf{,}\mathbf{2}\mathbf{)}$
LLM-Reward	0.4440	0.0000	2.7300	0.1190	0.8730	0.0000	2.5400	0.1010
Epsilon-Greedy	0.4140	0.0992	2.9100	0.2700	0.8400	0.1110	2.5500	0.2600
UCB	0.0707	0.0000	3.4800	0.1830	0.4390	0.0000	3.1000	0.1700
Thompson	0.0707	0.0000	3.5800	0.0911	0.4390	0.0000	3.2600	0.1450

compares the performance of four contextual bandit policies:

• LLM-Reward: A policy guided by LLM reward prediction;
• Epsilon-Greedy: Selects a random arm with probability $\epsilon =0.1$, otherwise exploits the best-known arm;
• UCB: Selects arms by maximizing upper confidence bounds to balance exploration and exploitation;
• TS: Samples from posterior reward distributions for probabilistic action selection.

The table reports the following metrics under two different Beta-distributed reward regimes, $Y_{t,a}\sim {\mathrm{Beta}}^{-1}(\alpha ,\beta )$:

• Mean R: Average reward obtained by the policy; higher values indicate better performance;
• SD R: Standard deviation of the reward across replications; lower values indicate more stable results;
• Mean Reg: Mean cumulative regret, defined as the total difference between the optimal arm’s reward and the chosen arm’s reward; lower values indicate more efficient learning;
• SD Reg: Standard deviation of cumulative regret, reflecting variability in performance across replications.

The reward regimes are:

• $\alpha =2,\beta =5$: Left-skewed Beta distribution modeling sparse or conservative rewards.
• $\alpha =5,\beta =2$: Right-skewed Beta distribution modeling optimistic rewards with heavier tails.

The results demonstrate that the LLM-Reward policy consistently attains the highest mean reward and lowest regret, indicating superior decision-making. The Epsilon-Greedy policy shows competitive mean reward but suffers from higher variability. Meanwhile, UCB and TS underperform under both regimes, suggesting limited adaptability to complex or asymmetric reward landscapes.

Figure 1 (Cumulative Regret Over Time) plots the cumulative regret

$$R_t=\sum _{s=1}^t\underset{a}{\max }{r}_{s,a}-r_{s,a_s},$$

where $r_{s,a}$ is the reward of arm a at time s, and $a_s$ is the selected arm. Lower $R_t$ indicates more efficient learning. The figure shows that TS and the LLM-guided policy maintain the lowest regret growth, reflecting better exploration-exploitation trade-offs, whereas UCB and $ϵ$-greedy accumulate more regret over time.

Figure 2 (Average Reward Over Time) displays the average reward trajectory, highlighting that TS and LLM-guided policies generally achieve higher and more stable rewards. The $ϵ$-greedy policy exhibits high variance, reflecting less reliable learning.

Figure 3 (Regret Standard Deviation Over Time) assesses the consistency of policies by plotting the standard deviation of cumulative regret across simulation replications. Lower values indicate more stable and reliable learning behavior. Thompson and LLM policies show more stability early in learning, though all policies experience increased variability after around time step 20, potentially due to the heavy-tailed nature of the Beta reward distributions causing occasional outlier outcomes.

The notation $Y_{t,a}\sim {\mathrm{Beta}}^{-1}(\alpha ,\beta )$ indicates that rewards for arm a at time t are generated via the inverse cumulative distribution function (quantile function) of the Beta distribution with shape parameters $\alpha$ and $\beta$. This modeling induces skewness and heteroskedasticity in rewards, challenging standard bandit algorithms.

The empirical results illustrate that LLM-guided policies effectively leverage contextual and historical information to achieve robust performance across different reward distribution shapes and volatility, while classical policies like UCB and TS can struggle under these complex settings.

The use of 95% confidence intervals in the figures supports rigorous statistical interpretation, while regret metrics emphasize the fundamental exploration-exploitation balance inherent in sequential decision-making.

Figure 4, Figure 5 and Figure 6 illustrate the performance of contextual bandit policies under a left-skewed inverse Beta reward distribution $Y_{t,a}\sim {\mathrm{Beta}}^{-1}(5,2)$. This setup introduces significant variability and a greater likelihood of extreme reward values concentrated near the upper bound.

The notation $Y_{t,a}\sim {\mathrm{Beta}}^{-1}(\alpha ,\beta )$ indicates that the reward for arm a at time t is generated by the inverse cumulative distribution function (quantile function) of a Beta distribution with shape parameters $\alpha$ and $\beta$. Here, $\alpha =5$ and $\beta =2$ specify a left-skewed distribution. Cumulative regret $R_t={\sum }_{s=1}^t{\max }_a{Y}_{s,a}-Y_{s,a_s}$ measures the accumulated loss due to suboptimal arm selections up to time t. The plotted confidence intervals represent 95% uncertainty bounds around the mean metrics.

• Cumulative Regret (Figure 4)

This plot compares the accumulation of regret over time across policies. The TS and LLM-guided policies achieve the lowest cumulative regret, indicating efficient exploration-exploitation trade-offs and rapid adaptation to arms with higher rewards despite the skewed distribution. In contrast, the $ϵ$-greedy policy exhibits a significantly steeper regret curve, reflecting slower learning caused by excessive random exploration.

• Average Reward (Figure 5)

The average reward trajectories demonstrate that TS and LLM policies quickly converge to high-reward arms, effectively capitalizing on the tail behavior of the skewed reward distribution. The UCB policy performs moderately well but is consistently outperformed by the LLM and Thompson approaches. The $ϵ$-greedy policy shows lagging and erratic performance, likely due to over-exploration.

• Regret Variability (Figure 6)

This figure depicts the standard deviation of regret across repeated simulations, reflecting policy stability. Early in the simulation, TS and LLM-guided policies maintain lower variability, indicating more stable and reliable learning. However, all policies exhibit increasing variability in later stages, attributable to the heavy-tailed reward distribution producing occasional extreme outcomes. The Epsilon-greedy strategy remains the most unstable throughout.

The results collectively highlight that distribution-aware exploration strategies, such as TS and LLM-informed policies, handle skewed, high-variance reward environments more effectively than heuristic approaches like UCB or Epsilon-greedy. Incorporating LLM-based guidance alongside flexible probabilistic context modeling offers tangible advantages in complex sequential decision-making scenarios.

4. Conclusions

This study evaluates the performance of four contextual bandit policies—Ensemble, Epsilon-greedy, Thompson Sampling, and UCB—through extensive simulation over 10,000 replications, focusing on cumulative regret, reward stability, and adaptive effectiveness under heavy-tailed and skewed reward distributions. In scenarios where rewards follow complex, asymmetric distributions, the Thompson Sampling and LLM-guided policies consistently demonstrate superior performance, maintaining lower cumulative regret and higher observed rewards compared to baselines. The Epsilon-greedy policy, in contrast, exhibits unstable behavior and higher regret variability over time.

In both symmetric and skewed reward settings, the ensemble policy—enhanced with copula-transformed contextual features and Gaussian Process-based modeling—achieves the most robust performance, consistently yielding the lowest regret and highest cumulative rewards. These findings underscore the importance of flexible modeling techniques and the integration of language model guidance for real-time contextual decision-making in dynamic environments.

Looking forward, we plan to explore lightweight LLM surrogates, caching mechanisms, and asynchronous decision-making pipelines to address latency concerns associated with real-world deployment. Further directions include meta-bandit frameworks for strategy adaptation and extending our approach to high-dimensional, spatio-temporal applications. Balancing the intelligence of LLMs with real-time operational constraints will be central to advancing practical, deployable contextual bandit systems.