A Simulation Study on Adaptive Assignment Versus Randomizations in Clinical Trials

Abstract

This study investigates a sequential clinical trial comparing two treatments with dichotomous outcomes. We evaluate the effectiveness of five adaptive procedures and three randomization methods for assigning patients to different therapies. The primary objective is to identify an optimal treatment allocation policy that maximizes the proportion of successful outcomes in a trial. By comparing the performance of adaptive and randomized procedures, this research provides valuable insights for enhancing treatment allocation strategies in clinical trials, ultimately aiming to improve the overall success rates of therapeutic interventions.

1. Introduction

In clinical research, identifying the most effective treatment from multiple available options is crucial, particularly when treatment outcomes are binary—classified as either a success or a failure. In sequential clinical trials, where patients are treated one after another, and their outcomes are observed in real-time, there exists the opportunity to dynamically adjust the treatment strategy to maximize the overall success rate. Given two treatments in a clinical trial, with the objective of maximizing the proportion of successes among a fixed number of patients, n, the sequential nature of treatment allocation presents both a challenge and an opportunity: Can a simple/tractable procedure be developed to optimally allocate treatments and thereby maximize the proportion of successful outcomes?

This challenge lies at the heart of sequential trial design. The goal is not only to maximize immediate successes but also to ensure that the chosen procedure is both robust and efficient. Investigating such procedures is essential for improving clinical decision-making, as it directly influences the efficacy of treatments administered in practice. Thus, this study aims to address these challenges by developing and proposing a methodology designed to improve the overall success rate in clinical trials.

To address this problem, one might initially consider randomized designs. Randomization is a conventional approach used to mitigate systematic differences between treatment groups, which could otherwise bias the trial’s outcomes. By randomly assigning treatments to patients, researchers ensure that any observed differences in outcomes are more likely due to the treatments themselves, rather than confounding factors. However, while randomization reduces bias, it does not necessarily maximize the proportion of successes in the trial.

In contrast, adaptive procedures have gained considerable attention for their ability to treat patients more effectively within a trial. These methods allow treatment allocation to depend on accumulating information, adjusting the strategy as the trial progresses. Notable works in this area include those by Thompson [1], Feldman [2], Zelen [3], Sobel and Weiss [4], Berry [5], Bather [6], and Berry and Fristedt [7], who explored various adaptive methods in clinical trials. These procedures aim to enhance the likelihood of successful outcomes by making informed decisions based on the observed responses of previous patients.

Since 1995, significant advancements have been made in this field. Berry and Eick [8] compared randomization with adaptive allocation methods, demonstrating the advantages of adaptive designs for maximizing the expected number of successes. Their foundational work highlighted the benefits of adaptive methods over traditional randomization and provided a framework for comparing different treatment allocation strategies. Subsequently, Pocock and Simon [9] discussed sequential treatment assignment with balancing for prognostic factors, while Chaudhuri and Lo [10] explored hybrid randomization strategies. Additional applications and theoretical insights into randomization and adaptive designs can be found in Berry and Eick [11], Rosenberger and Lachin [12], Hu and Rosenberger [13], Chow and Chang [14], Porcher et al. [15], Laage et al. [16], Bhatt and Mehta [17], Chang and Balser [18], Pallmann et al. [19], Kelly et al. [20], Thorlund et al. [21], Afolabi and Kelly [22], Kaizer et al. [23], and Ben-Eltriki [24].

Building on this foundation, our study employs a simulation-based approach to optimizing treatment allocation strategies. Specifically, we aim to maximize the probability of achieving at least k successes out of n trials, given two treatments, A and B, with known success probabilities $\alpha$ and $\beta$. The optimization problem is formally described as follows:

Let $Z_j$ for $j=1,2,\dots ,n$ represent the response of patient j. The value of $Z_j$ is 1 if the response is positive and 0 if it is negative. Assume that the probability of success with treatment A is $\alpha$, and that with treatment B is $\beta$. Thus, if patient j receives treatment A,

$$E\left(Z_j\right|\alpha ,\beta )=P(Z_j=1|\alpha ,\beta )=\alpha ,$$

and if patient j receives treatment B,

$$E\left(Z_j\right|\alpha ,\beta )=P(Z_j=1|\alpha ,\beta )=\beta .$$

Let $\mathcal{D}$ denote the class of all treatment allocation procedures. For any given n and k ($1\le k\le n$), the conditional probability of achieving at least k successes (given $\alpha$ and $\beta$) under procedure $\tau \in \mathcal{D}$ is

$$P_{\tau }(k,n|\alpha ,\beta )=P_{\tau }\left(\sum _{i=1}^n{Z}_i\ge k\mid \alpha ,\beta \right),$$

(1)

where the distribution of the $Z_i$’s is determined by $\tau$. It is evident that the probability in (1) cannot exceed

$$P(k,n|\alpha ,\beta )=\sum _{i=k}^n\left({\displaystyle \genfrac{}{}{0pt}{}{n}{i}}\right){\gamma }^i{(1-\gamma )}^{n-i},$$

where $\gamma =max(\alpha ,\beta )$. Define

$$L_{\tau }(k,n|\alpha ,\beta )=P(k,n|\alpha ,\beta )-P_{\tau }(k,n|\alpha ,\beta ),$$

as the conditional probability lost (CPL) under procedure $\tau$. We evaluate the CPL values for various combinations of $\alpha$ and $\beta$ under three randomized procedures and five adaptive procedures studied by Berry and Eick [8]. Based on these numerical results, we aim to identify the most effective procedure.

The insights gained from our numerical analysis not only enhance our understanding of the performance characteristics of different allocation strategies but also offer practical recommendations for optimizing clinical trial designs. By integrating the CPL metric into the decision-making process, researchers can make informed choices about treatment allocation, ultimately improving the efficacy of clinical trials in real-world settings. The contributions of this study are particularly relevant in guiding future trials towards more effective treatment allocation strategies that consider the dynamic nature of sequential clinical trials.

Berry and Eick [8] introduced the conditional expected success loss (CESL) to evaluate and compare treatment allocation methods. The CESL quantifies the expected loss of treatment success due to an inefficient allocation strategy and is defined as

$$CESL=nmax(\alpha ,\beta )-E\left(\sum _{i=1}^n{Z}_i\mid \alpha ,\beta \right),$$

where $E\left({\sum }_{i=1}^n{Z}_i\mid \alpha ,\beta \right)$ represents the expected number of successes under the given allocation procedure. The CESL provides an ethical justification for adaptive methods by emphasizing the benefits to participants through more favorable treatment allocation. In contrast, the CPL metric focuses on optimizing the probability of achieving a pre-determined number of successes, thereby aligning with the trial’s objective of maximizing treatment effectiveness.

While CESL highlights the ethical benefits of adaptive methods, particularly in terms of benefiting participants, CPL prioritizes the achievement of trial objectives by optimizing the likelihood of achieving the desired success rate. A section of this paper will be dedicated to comparing these two metrics, providing a more comprehensive understanding of their respective advantages and applications in clinical trial design.

The remainder of this paper is organized as follows. Section 2 provides a detailed description of the eight discussed procedures, focusing on how they allocate treatments to n patients. Section 3 derives the formulas required to compute the probabilities of achieving at least k successes out of n trials for three randomized procedures and one adaptive procedure. Section 4 presents an extensive numerical analysis of these procedures, assessing their CPL and visualizing the results through heatmaps. The section concludes with observations on their application to the optimization problem at hand. Section 5 compares the CPL and CESL metrics, highlighting their respective strengths and applications in optimizing treatment allocation strategies. Finally, Section 6 provides concluding remarks, summarizing the main findings of this study and offering suggestions for future research in clinical trial design.

2. Procedures

There are eight potential procedures commonly used in bandit problems of this nature. For simplicity, we assume that n is an even positive integer. A comprehensive description of procedures ER, JB, PW, RB, and WT is provided in the work of Berry and Eick [8]. The references by Efron [25], Zelen [26], Wei [27], Pocock [28], and Rosenberger and Lachin [29] are related to procedure RR, while the references by Cornfield [30] and Fleiss [31] are associated with procedure SR. In this study, we propose a new procedure, PR, to enable a direct comparison with procedure JB.

• Procedure ER (equal randomized): Half of the n patients are randomly assigned to treatment A, while the other half receive treatment B. This approach ensures an equal representation of both treatments but may not maximize the overall success rate due to its lack of adaptability.
• Procedure RR (repeatedly randomized): Patients are randomized with equal probability at each assignment. This method provides flexibility by continuously adjusting treatment allocation but may introduce variability in treatment success.
• Procedure SR (single-randomized): Randomization occurs only at the first assignment, and the selected treatment is used for all subsequent patients. While this method simplifies the decision-making process, it risks poor outcomes if the initial randomization does not reflect the true effectiveness of the treatments.
• Procedure JB (J. Bather): Patients are randomized at each step using the following adaptive procedure. The first two patients are assigned treatments A and B randomly, ensuring that each receives a different therapy. Suppose that, by time t ($2\le t<n$), t patients have been treated, and the outcomes—successes and failures—on treatments A and B are recorded as $s_x, f_x, s_y, f_y$, respectively, such that $s_x+f_x+s_y+f_y=t$. Define

  $$\lambda \left(k\right)={\displaystyle \frac{4+\sqrt{k}}{15k}}.$$
  Let

  $${\lambda }_x=\lambda (s_x+f_x),{\lambda }_y=\lambda (s_y+f_y),$$

  and

  $$q={\displaystyle \frac{s_x}{s_x+f_x}}-{\displaystyle \frac{s_y}{s_y+f_y}}+2({\lambda }_x-{\lambda }_y).$$
  Under this procedure, the next patient (patient $t+1$) receives treatment A with probability

  $$\left\{\begin{array}{cc}{\displaystyle \frac{{\lambda }_x}{{\lambda }_x+{\lambda }_y}exp\left(\frac{q}{{\lambda }_x}\right)}\hfill & \mathrm{for}q\le 0,\hfill \\ \\ {\displaystyle 1-\frac{{\lambda }_y}{{\lambda }_x+{\lambda }_y}exp\left(-\frac{q}{{\lambda }_y}\right)}\hfill & \mathrm{for}q>0.\hfill \end{array}\right.$$
  This procedure dynamically adjusts treatment based on previous outcomes, potentially enhancing overall success rates by favoring treatments that have shown higher effectiveness.
• Procedure PW (play-the-winner/switch-from-a-loser): The first patient is randomly assigned to either treatment A or B with a probability of 0.5 for each. For patients 2 to n, the treatment given to the previous patient is repeated if it was successful; otherwise, the other treatment is administered. This strategy leverages successful outcomes to inform subsequent treatment choices, thus optimizing the allocation based on real-time feedback.
• Procedure RB (robust Bayes): This policy utilizes a randomization strategy based on a uniform prior density:

  $$\pi (\alpha ,\beta )=1\mathrm{on}(0, 1)\times (0, 1).$$
  Due to the symmetry of the uniform prior distribution, the treatments for the first patient are initially equivalent and can be chosen at random. If the first patient experiences success, the second patient receives the same treatment. Conversely, if the first patient experiences a failure, the second patient receives the alternative treatment. Therefore, procedure RB mimics procedure PW for the first two treatment assignments. The same treatment is continued as long as it is successful. However, after a failure, switching to the other treatment may or may not be optimal. If the data strongly support the treatment that has just failed, that treatment will be used again. Specifically, if the current probability of success of treatment A (the posterior expected value of $\alpha$, $(s_A+1)/(s_A+f_A+2)$) is greater than that of treatment B, then treatment A will be used. Additionally, if both treatments are judged to be equally effective at any stage, the next treatment assignment will be randomized. This approach balances robustness and adaptability, utilizing Bayesian updating to make informed decisions while maintaining some level of randomness.
• Procedure WT (W. Thompson): Similar to procedure JB, this procedure randomizes between treatments X and Y for patients 1 through n. The randomization is based on the current probability distribution of $(p_x,p_y)$, assuming a uniform prior density on $(p_x,p_y)$:

  $$\pi (p_x,p_y)=1\mathrm{on}(0,1)\times (0,1).$$
  The next patient receives treatment X with a probability equal to the current probability that $p_x>p_y$. This probability is given by

  $${\int }_0^1{\int }_{\nu }^1{u}^{s_x}{(1-u)}^{f_x}{\nu }^{s_y}{(1-\nu )}^{f_y}dud\nu {\left\{B(s_x+1,f_x+1)·B(s_y+1,f_y+1)\right\}}^{-1},$$

  where $B(·,·)$ denotes the complete beta function:

  $$B(a,b)={\int }_0^1{u}^{a-1}{(1-u)}^{b-1}du.$$
  By continuously updating the probability estimates, this procedure aims to maximize the chances of success based on the accumulated data.
• Procedure PR (posterior probability ratio): This newly proposed procedure selects treatment A or B based on posterior probabilities. In contrast to procedure JB, this procedure randomizes between treatments A and B for patients 1 through n. The randomization is based on the current expected values of $\alpha$ and $\beta$, assuming a uniform prior density on $(\alpha ,\beta )$. The next patient receives treatment A with a probability given by

  $${\displaystyle \frac{E\left(\alpha \right)}{E\left(\alpha \right)+E\left(\beta \right)}}={\displaystyle \frac{\frac{s_A+1}{s_A+f_A+2}}{\frac{s_A+1}{s_A+f_A+2}+\frac{s_B+1}{s_B+f_B+2}}}.$$
  This procedure provides a direct comparison of the posterior probabilities of success for each treatment, thus enabling more informed treatment decisions based on the most current data.

3. Probabilities

For procedures ER, SR, and RR, we can derive the formula for computing the probabilities of achieving at least k successes out of n trials, denoted as $P_{\tau }(k,n|\alpha ,\beta )$, for various values of $\alpha$ and $\beta$. Given that n is even, we let $n=2m$. This assumption simplifies calculations, allowing us to leverage symmetry in the analysis.

For procedure ER, we consider that the first m patients receive treatment A, while the remaining m patients receive treatment B. Consequently, we have the following:

$$\begin{array}{ccc}& & P_{ER}(k,2m|\alpha ,\beta )\hfill \\ & =& \sum _{j=0}^{m}P\left(\sum _{i=1}^m{Z}_i=j,\sum _{i=m+1}^{2m}{Z}_i=\ell ,wherej+\ell =k,k+1,\dots ,2m\right)\hfill \\ & =& \sum _{j=0}^{m}P\left(\sum _{i=1}^m{Z}_i=j\right)P\left(\sum _{i=m+1}^{2m}{Z}_i=\ell ,wherej+\ell =k,k+1,\dots ,2m|\sum _{i=1}^m{Z}_i=j\right)\hfill \\ & =& \sum _{j=0}^{m}P\left(\sum _{i=1}^m{Z}_i=j\right)P\left(\sum _{h=1}^m{Z}_h=\ell ,where\ell =k-j,k-j+1,\dots ,m|\sum _{i=1}^m{Z}_i=j\right)\hfill \\ & =& \sum _{j=0}^m\left({\displaystyle \genfrac{}{}{0pt}{}{m}{j}}\right){\alpha }^j{(1-\alpha )}^{m-j}\left[\sum _{\ell =k-j}^m\left({\displaystyle \genfrac{}{}{0pt}{}{m}{\ell }}\right){\beta }^{\ell }{(1-\beta )}^{m-\ell }\right].\hfill \end{array}$$

For procedures SR, we may assign treatment A or B to patient #1, each with a probability of $1/2$. If patient #1 receives treatment A, then the remaining $2m-1$ patients will also receive treatment A. Similarly, if patient #1 receives treatment B, then the remaining $2m-1$ patients will also receive treatment B. Hence,

$$\begin{array}{ccc}\hfill P_{SR}(k,2m|\alpha ,\beta )& =& {\displaystyle \frac{1}{2}}P\left(\sum _{i=1}^{2m}{Z}_i\ge k,\mathrm{all} \mathrm{patients} \mathrm{reveive} \mathrm{treatment} A\right)\hfill \\ & & +{\displaystyle \frac{1}{2}}P\left(\sum _{i=1}^{2m}{Z}_i\ge k,\mathrm{all} \mathrm{patients} \mathrm{reveive} \mathrm{treatment} B\right)\hfill \\ & =& {\displaystyle \frac{1}{2}}\sum _{j=k}^{2m}\left({\displaystyle \genfrac{}{}{0pt}{}{2m}{j}}\right){\alpha }^j{(1-\alpha )}^{2m-j}+{\displaystyle \frac{1}{2}}\sum _{j=k}^{2m}\left({\displaystyle \genfrac{}{}{0pt}{}{2m}{j}}\right){\beta }^j{(1-\beta )}^{2m-j}\hfill \\ & =& {\displaystyle \frac{1}{2}}\sum _{j=k}^{2m}\left({\displaystyle \genfrac{}{}{0pt}{}{2m}{j}}\right)\left\{{\alpha }^j{(1-\alpha )}^{2m-j}+{\beta }^j{(1-\beta )}^{2m-j}\right\}.\hfill \end{array}$$

For procedures RR, treatment A or B (each with probability 1/2) is assigned to $2m$ patients independently. Clearly, the probability that a patient receives a positive response or a negative response, respectively, is ${\displaystyle \frac{1}{2}}(\alpha +\beta )$ and $1-{\displaystyle \frac{1}{2}}(\alpha +\beta )$. Thus,

$$\begin{array}{ccc}\hfill P_{RR}(k,2m|\alpha ,\beta )& =& \sum _{j=k}^{2m}P\left(\sum _{i=1}^{2m}{Z}_i=j\right)\hfill \\ & =& \sum _{j=k}^{2m}\left({\displaystyle \genfrac{}{}{0pt}{}{2m}{j}}\right){\left({\displaystyle \frac{\alpha +\beta }{2}}\right)}^j{\left(1-{\displaystyle \frac{\alpha +\beta }{2}}\right)}^{2m-j}.\hfill \end{array}$$

Note that the difference between procedures ER and RR is that procedure ER assigns m patients with treatment A and the remaining patients with treatment B. In contrast, we are unable to know how many patients receive treatment A or B under procedure RR in advance. At the end of the experiment, there may be fewer m patients receiving treatment A and more than m patients receiving treatment B, and vice versa.

4. Numerical Studies

The derivation of recursive formulas to calculate the probability of achieving at least k successes out of n trials for procedures RB, JB, WT, and PR is complex. Consequently, simulation studies are employed to effectively evaluate and compare these procedures. The computer program designed for calculations and generating graphics to enhance reproducibility and transparency was written in R (R Core Team [32]), and it is available from the authors upon request.

To assess the performance of these procedures using the conditional probability lost (CPL) metric discussed in Section 2, we conducted a simulation study with 1000 iterations. This study estimated probabilities for selected values of $k=40, 50, 60, 70, 80$, $\alpha =0.25, 0.35, 0.45, 0.55, 0.65, 0.75$, $\beta =0.1, 0.3, 0.5, 0.7, 0.9$, and $n=100$. The diverse parameter set allows for a comprehensive evaluation of the procedures across various scenarios.

Based on the simulation results, we draw several key observations regarding the performance of the procedures in maximizing the proportion of successes in clinical trials:

Comparison of randomization procedures: 

Figure 1 and Figure 2 illustrate that the randomization procedures ER and RR perform similarly, with ER slightly outperforming RR. In certain cases, SR shows superior performance, indicating that the choice of randomization procedure significantly impacts outcomes. Notably, for each value of k, the range of $\beta$ over which SR outperforms ER and RR decreases as $\alpha$ increases. Conversely, as k increases, these ranges widen for a fixed $\alpha$. Overall, SR is less effective for larger values of $\beta$ when $\alpha$ or k is small to intermediate, underscoring the importance of tailoring randomization procedures to the specific trial conditions.

Performance of adaptive procedures: 

Figure 3 and Figure 4 reveal that, among the adaptive procedures, PR consistently performs the worst. Conversely, JB and WT demonstrate a similar performance, with JB slightly outperforming WT. For a small $\beta$, RB generally exhibits the best performance, followed by JB, WT, and PW. However, as $\beta$ increases, JB becomes the most effective procedure, with WT next in line, while PW and RB alternate in performance based on trial conditions.

For smaller $\beta$ values, RB tends to outperform JB when larger k values are combined with moderate to high $\alpha$ values, but it underperforms compared to PW when smaller k values and lower to moderate $\alpha$ values are involved. These performance shifts highlight the sensitivity of adaptive procedures to variations in k, $\alpha$, and $\beta$, emphasizing the necessity of tailoring the choice of procedure to the specific trial conditions, target success probability, and desired number of successes.

Comparison between SR, ER, JB, and RB: 

Figure 5 and Figure 6 indicate that, for small values of $\beta$, RB generally exhibits the best performance, followed by either JB or SR, while ER performs the least effectively. As $\beta$ increases, however, JB emerges as the most effective procedure, with the performance of RB, SR, and ER alternating based on trial conditions. Notably, ER outperforms RB when both $\alpha$ and $\beta$ are large.

Analysis of heatmaps: 

The series of heatmaps displayed in Figure 7, Figure 8 and Figure 9 illustrate how each allocation procedure performs across varying values of k and the parameters $\alpha$ and $\beta$. Key observations include the following.

Performance variation by procedure: 

• ER typically exhibits lighter shades of blue across various k values, indicating higher CPL and, thus, less favorable performance compared to the other procedures.
• JB shows darker blue regions in specific areas, particularly at higher k values, suggesting effective performance in minimizing CPL under those conditions.
• RB demonstrates dark blue regions, especially at a lower $\beta$, indicating effective performance; however, its effectiveness diminishes as $\beta$ increases.
• SR shows darker blue regions at higher $\beta$ values, indicating better performance in reducing CPL, although its effectiveness varies across all k values.

Influence of k on procedure performance: 

As k increases from 40 to 80, the distribution of darker blue regions varies across procedures. Notably, JB tends to achieve darker blue intensities (indicating better CPL performance) at higher k values, while RB displays a similar pattern but is more sensitive to changes in $\beta$.

Parameter sensitivity: 

The effectiveness of each procedure varies with $\alpha$ and $\beta$. For instance, both JB and RB exhibit darker blue regions (lower CPL) at lower $\beta$ values, while SR shows darker blue at higher $\beta$ values. This variability underscores the importance of selecting a procedure based on specific trial conditions, as some procedures yield better performance with low CPL in particular parameter settings.

In conclusion, this comparative analysis of randomization and adaptive procedures underscores the importance of aligning the chosen method with the specific conditions of the trial. The results presented in Figure 1, Figure 2, Figure 3 and Figure 4 reveal that, while randomization methods such as ER and RR exhibit comparable performance, procedure SR can excel in certain situations, particularly at lower values of $\beta$. However, its effectiveness diminishes as $\beta$ increases or when $\alpha$ and k are lower.

Among the adaptive procedures, JB consistently demonstrates superior performance, especially with an increasing $\beta$, while PR remains the least effective across all scenarios. The performance of RB is noteworthy, showing advantages under specific conditions with a smaller $\beta$, but it is highly sensitive to variations in the parameters k and $\alpha$.

To maximize effectiveness, the careful selection of randomization and adaptive procedures based on anticipated trial parameters is essential. The heatmap analysis further emphasizes the significant impact that the choice of procedure has on the CPL. Therefore, it is advisable for trial designers to consider the specific success target (k) and parameter values ($\alpha$ and $\beta$) when selecting procedures to optimize performance.

Moreover, the consistent performance of treatment allocation procedures for $\beta >0.3$ when $n=300$, with exceptions noted for SR in randomization methods and PR in adaptive procedures, suggests that the applicability of the CPL metric and the recommended procedures may be limited to smaller sample sizes. Thus, further investigations are warranted to enhance the applicability of these findings in diverse clinical trial designs.

5. Comparison of CESL and CPL: Methodological Perspectives on Treatment Allocation

This section compares the CESL metric introduced by Berry and Eick [8] and the CPL metric developed in our study. The objective is to evaluate treatment allocation procedures based on these two metrics and identify the most effective approaches to clinical trial designs, depending on the trial’s goals and constraints.

Berry and Eick [8] used the CESL metric to assess the effectiveness of adaptive treatment allocation procedures by measuring the expected success loss. They analyzed several adaptive designs, including RB, JB, WT, and PW, comparing each against the ER design. Their findings indicated that the RB procedure generally minimized the expected success loss more effectively than other adaptive procedures, especially when trial sizes (n) were small. However, they observed that RB may not consistently perform well across all sample sizes, n, and for some combinations of $(\alpha ,\beta )$, especially as these values moderate. Additionally, while RB demonstrated clear advantages over ER for small trials, its practical use diminished with larger trials, where it showed only marginal improvements over ER and was more challenging to implement due to its complexity.

In contrast, this study employed the CPL metric to evaluate treatment allocation procedures, focusing on the probability of achieving a predefined target of k successes within the trial. This alternative perspective enables a nuanced assessment of how well different procedures support success-driven trial goals. We examined three randomization-based procedures—ER, RR, and SR—alongside four adaptive approaches—RB, JB, PW, and PR. Our results indicate that the ER and RR designs perform similarly, with a slight edge for ER. Among the adaptive approaches, JB demonstrates consistently strong performance, particularly as $\beta$ increases, aligning well with trials that prioritize success maximization. Conversely, PR consistently shows the least efficacy across all parameter settings, highlighting its limitations in success-oriented applications. Notably, while RB maintains competitive performance under specific conditions, particularly with lower $\beta$, it exhibits considerable sensitivity to changes in k and $\alpha$, which could limit its generalizability across varied trial designs.

Overall, both the CESL and CPL metrics offer valuable insights into procedure performance but serve distinct trial objectives. The CPL metric is particularly beneficial for trials focused on achieving specific success thresholds, such as a target number of successes. In contrast, CESL better aligns with adaptive designs where minimizing expected success loss is a priority. Our findings, together with Berry and Eick’s observations, underscore the importance of selecting the appropriate metric and procedure based on the trial’s primary objectives. For trials emphasizing broader applicability and adaptability, CESL-guided adaptive designs may be preferable. However, for trials with a targeted success criterion, CPL-based assessments provide more relevant guidance for achieving within-trial success objectives.

6. Concluding Remarks

This study has introduced the conditional probability lost (CPL) metric, designed to evaluate and compare various treatment allocation procedures in sequential clinical trials with a focus on achieving specific success targets. Through a comprehensive simulation study, we assessed the performance of both randomization-based and adaptive procedures under diverse parameter settings, allowing for an analysis of each procedure’s strengths and limitations.

Our results reveal that randomization procedures, specifically ER and RR, perform similarly, with ER showing a slight advantage in general. Adaptive procedures, however, display more variability: while RB minimizes expected success loss in scenarios with small values of $\alpha$ and $\beta$, it becomes more sensitive to parameter changes, particularly with higher values of k and $\alpha$. In contrast, JB consistently outperforms other adaptive approaches, particularly as $\beta$ increases, making it suitable for trials that prioritize reaching defined success targets. The consistently low performance of PR across all settings highlights its limitations in success-oriented applications.

This study, together with the work of Berry and Eick (1995), emphasizes the importance of selecting metrics that are closely aligned with the goals of the trial. The CPL metric is especially useful for trials aiming to meet specific success thresholds, offering actionable insights for procedures designed to maximize within-trial success. Conversely, CESL’s focus on minimizing expected success loss is beneficial in adaptive designs where broader patient benefit and loss minimization are primary concerns. The choice of metric and procedure should, thus, align with the trial’s primary objectives, balancing between adaptability (CESL) and targeted success achievement (CPL).

Although our findings offer promising insights, certain limitations should be acknowledged. The recursive formulations used to compute success probabilities add complexity to practical applications, and while our simulation study is extensive, it remains bounded by specific parameter ranges and sample sizes.

Future research could further expand the CPL framework to accommodate multi-armed trials, broader parameter spaces, and additional trial objectives. Additionally, integrating real-world data may enhance the applicability of our results, providing a practical validation of CPL’s utility in diverse clinical settings.

Moreover, the variability of allocation procedures, which may accompany procedures closely aligned with ethical goals, should also be considered in making recommendations. Reporting this variability provides a more complete evaluation of each procedure, helping to balance ethical outcomes with allocation stability.

Finally, we can explore the integration of early-stop designs with the CPL framework. These designs, which allow for trial termination upon observing strong evidence of treatment efficacy, could further balance the power and success rate, optimizing trial outcomes while improving patient benefit and efficiency.

In conclusion, the CPL metric and our simulation findings contribute to informed decision-making in clinical trial design, supporting a more tailored approach to treatment allocation. By aligning the choice of procedure with trial-specific objectives, researchers can enhance trial effectiveness, ultimately advancing patient outcomes and medical knowledge.