Bayesian Binary Search

Abstract

We present Bayesian Binary Search (BBS), a novel framework that bridges statistical learning theory/probabilistic machine learning and binary search. BBS utilizes probabilistic methods to learn the underlying probability density of the search space. This learned distribution then informs a modified bisection strategy, where the split point is determined by probability density rather than the conventional midpoint. This learning process for search space density estimation can be achieved through various supervised probabilistic machine learning techniques (e.g., Gaussian Process Regression, Bayesian Neural Networks, and Quantile Regression) or unsupervised statistical learning algorithms (e.g., Gaussian Mixture Models, Kernel Density Estimation (KDE), and Maximum Likelihood Estimation (MLE)). Our results demonstrate substantial efficiency improvements using BBS on both synthetic data with diverse distributions and in a real-world scenario involving Bitcoin Lightning Network channel balance probing (3–6% efficiency gain), where BBS is currently in production.

1. Introduction

The concept of organizing data for efficient searching has ancient roots. One of the earliest known examples is the Inakibit-Anu tablet from Babylon (c. 200 BCE), which contained approximately 500 sorted sexagesimal numbers and their reciprocals, facilitating easier searches [1]. Similar sorting techniques were evident in name lists discovered on the Aegean Islands. The Catholicon, a Latin dictionary completed in 1286 CE, marked a significant advance by introducing rules for complete alphabetical classification [1].

The documented modern era of search algorithms began in 1946 when John Mauchly first mentioned binary search during the seminal Moore School Lectures [1]. This was followed by William Wesley Peterson’s introduction of interpolation search in 1957 [2]. A limitation of early binary search algorithms was their restriction to arrays with lengths one less than a power of two. This constraint was overcome in 1960 by Derrick Henry Lehmer, who published a generalized binary search algorithm applicable to arrays of any length [3]. A significant optimization came in 1962 when Hermann Bottenbruch presented an ALGOL 60 implementation of binary search that moved the equality comparison to the end of the process [4]. While this increased the average number of iterations by one, it reduced the number of comparisons per iteration to one, potentially improving efficiency. Further refinements to binary search continued, with A. K. Chandra of Stanford University developing the uniform binary search in 1971 [1]. This variant aimed to provide more consistent performance across different input distributions. In the realm of computational geometry, Bernard Chazelle and Leonidas J. Guibas introduced fractional cascading in 1986 [5]. This technique provided a method to efficiently solve various search problems in geometric contexts, broadening the application of search algorithms beyond simple sorted lists. More recently, hybrid and learning-based approaches have gained attention. For instance, ref. [6] proposed a hybrid search algorithm combining binary search and interpolated search, a theme also explored by [7] with an algorithm that performs interpolation once over a binary search. A newer related trend involves improving the binary search tree data structure using a frequency estimation oracle [8].

The bisection method, a fundamental technique in numerical analysis, shares conceptual similarities with binary search. This method, also known as the interval halving method, binary chopping, or Bolzano’s method, has origins of the bisection method that can be traced back to ancient mathematics. However, its formal description is often attributed to Bolzano’s work in the early 19th century [9]. Bolzano’s theorem, which guarantees the existence of a root in a continuous function that changes sign over an interval, provides the theoretical basis for the bisection method. In the context of root-finding algorithms, ref. [10] provides a comprehensive treatment of the bisection method, discussing its convergence properties and error bounds. They highlight the method’s robustness and guaranteed convergence, albeit at a relatively slow linear rate. Variants of the bisection method have been developed to improve efficiency and applicability. Ref. [11] first introduced probabilistic bisection. Ref. [12] employed interval analysis and bisection to develop reliable methods for finding global optima. These studies demonstrate how bisection principles can be extended beyond simple root-finding to more complex optimization problems.

The connection between bisection and binary search in computer science was explored by [1], who discussed both methods in the context of searching and optimization algorithms. This connection underlies our approach in leveraging probabilistic techniques to enhance binary search. Recent work has focused on adapting bisection methods to handle uncertainty and noise. Ref. [13] introduced a probabilistic bisection algorithm for noisy root-finding problems. Their method, which updates a probability distribution over the root location based on noisy measurements, shares conceptual similarities with our Bayesian Binary Search approach. In the context of stochastic optimization, ref. [14] developed a probabilistic bisection algorithm for binary classification problems. Their work demonstrates how Bayesian updating can be incorporated into the bisection process, providing a precedent for our probabilistic approach to binary search. The bisection method has also found applications in various fields beyond pure mathematics. For instance, ref. [15] discussed the use of parallel bisection methods in scientific computing, highlighting the potential for algorithmic improvements through parallelization.

Probabilistic machine learning, as described by Ghahramani [16], provides a framework for reasoning about uncertainty in data and models. Gaussian Processes (GPs), a cornerstone of many probabilistic machine learning methods, were extensively explored by Rasmussen andWilliams [17]. GPs offer a flexible, non-parametric approach to modeling distributions over functions, making them particularly suitable for tasks involving uncertainty quantification. In recent years, Bayesian deep learning has emerged as a powerful approach to combining the expressiveness of deep neural networks with principled uncertainty quantification. Wang and Yeung [18] provide a comprehensive survey of Bayesian deep learning techniques, discussing various methods for incorporating uncertainty into deep neural networks and their applications in complex, high-dimensional problems. The field of Bayesian Optimization, which shares conceptual similarities with our work, has seen substantial growth. Shahriari et al. [19] provide a comprehensive overview of Bayesian Optimization techniques, highlighting their effectiveness in optimizing expensive-to-evaluate functions. These methods typically use GPs, Bayesian Neural Networks (BNNs) or other probabilistic models to guide the optimization process, analogous to our use of probability density estimates in Bayesian Binary Search.

2. Methodology

2.1. Classical Binary Search/Bisection

Binary search is an efficient algorithm for finding a target value in a defined search space. It repeatedly divides the search interval in half, eliminating half of the remaining space at each step. The algorithm maintains two boundaries, low and high, and computes the midpoint as mid = (low + high)/2. By evaluating the midpoint and comparing it to the target value, it determines which half of the search space to explore next, updating either low or high accordingly. This process continues until the target is found or the search space is exhausted. The efficiency of binary search stems from its ability to halve the potential search space with each iteration, resulting in a logarithmic time complexity.

2.2. Bayesian Binary Search (BBS)

2.2.1. Binary Search as Bayesian Inference

The classical binary search/bisection algorithm can be formally framed within the context of Bayesian inference. Consider a sorted search space represented by the interval $[a_0,b_0]$. In the absence of prior knowledge about the location of the target value x, we define a uniform prior probability density function (PDF) over this initial interval:

$$P\left(x\right)=\left\{\begin{array}{cc}{\displaystyle \frac{1}{b_0-a_0}}\hfill & \mathrm{if}{a}_0\le x\le b_0\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

Let $[a_k,b_k]$ denote the search interval at the k-th step of the binary search. At each step, we probe the midpoint $m_k={\displaystyle \frac{a_k+b_k}{2}}$. The comparison of $m_k$ with the target value provides an observation $O_k$ which can be one of two outcomes: $x<m_k$ or $x>m_k$ (assuming the target is not exactly at $m_k$).

We can express the posterior probability density after the k-th probe using conditional probability. Let $P\left(x\right|O_k)$ denote the posterior PDF given the observation at step k.

If the observation is $x<m_k$, then the posterior PDF becomes

$$P\left(x\right|x<m_k)=\left\{\begin{array}{cc}{\displaystyle \frac{1}{m_k-a_k}}\hfill & \mathrm{if}{a}_k\le x<m_k\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

In this case, the new search interval for the next step is $[a_{k+1},b_{k+1}]=[a_k,m_k]$.

If the observation is $x>m_k$, then the posterior PDF becomes

$$P\left(x\right|x>m_k)=\left\{\begin{array}{cc}{\displaystyle \frac{1}{b_k-m_k}}\hfill & \mathrm{if}{m}_k<x\le b_k\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

Here, the new search interval is $[a_{k+1},b_{k+1}]=[m_k,b_k]$.

In both scenarios, the posterior distribution remains uniform over the reduced search interval, with the probability mass concentrated in half of the previous interval. This deterministic update of the probability distribution based on each probe is analogous to Bayesian inference, where the likelihood function effectively assigns zero probability to the half of the search space that is inconsistent with the observation.

Thus, standard binary search can be viewed as a specific instance of Bayesian inference employing a uniform prior and a deterministic observation model that halves the search space at each step. This provides a foundational perspective for understanding how Bayesian Binary Search generalizes this approach by incorporating learned priors and probabilistic probing strategies.

2.2.2. Problem Formulation

Given a search space S and a target value t, our aim is to locate t in S, or produce a specified range for t with a given tolerance. Unlike classical binary search, we assume that the distribution of potential target locations is not uniform across S. We represent this non-uniform distribution using a probability density function (PDF) $p\left(x\right)$, where $x\in S$. We formulate the binary search algorithm in two parts: (1) search space density estimation, which produces a PDF that is fed into a (2) modified binary search algorithm that begins at the median of the PDF and bisects in probability density space. BBS is equivalent to basic binary search when the search space PDF estimation process is replaced with the assumption of a uniformly distributed target space. This formulation maps to a binary search problem on a sorted array in the following way: the search space S corresponds to the sorted array, the target value t is the element being searched for, and the PDF p(x) represents the probability of finding the target at each index in the array using interpolation.

2.3. Bayesian Binary Search Framework

In contrast to the uniform prior that is assumed when looking at the classical binary search algorithm from the lens of Bayesian inference, BBS begins with a potentially non-uniform prior distribution, $P_{learned}\left(x\right)$, estimated from data using methods such as Gaussian Process Regression, Bayesian Neural Networks, Quantile Regression, Gaussian Mixture Models, Kernel Density Estimation, or Maximum Likelihood Estimation. This learned prior encapsulates prior knowledge or beliefs about the likely location of the target value.

The subsequent steps in BBS involve probing the search space based on this learned prior (or the evolving posterior distribution derived from it). Each probe and the resulting observation then lead to an update of the posterior distribution using Bayes’ Theorem:

$$P\left(x\right|O_k,P_{learned})\propto P\left(O_k\right|x)P_{learned}\left(x\right)$$

or, more generally, at step k:

$$P\left(x\right|O_k,\dots ,O_1,P_{learned})\propto P\left(O_k\right|x)P\left(x\right|O_{k-1},\dots ,O_1,P_{learned})$$

This iterative process, starting with a learned prior (Algorithm 1–5), allows BBS to focus its search on regions of higher probability, potentially leading to faster convergence and improved efficiency compared to standard binary search’s assumption of a uniform prior. The posterior update in this case is mathematically equivalent to bisecting in probability density space.

2.4. Proof of Posterior Update Equivalence to Bisection in Probability Density Space with a Non-Uniform Prior

In Bayesian Binary Search (BBS), the prior distribution over the search interval $[a_k,b_k]$ is not necessarily uniform. Let $P\left(x\right)$ be the prior PDF at the k-th step. The core principle of BBS is to choose a probe location $p_k$ within $[a_k,b_k]$ such that it effectively bisects the probability mass according to the current posterior belief.

At each step, BBS aims to find a probe location $p_k$ that satisfies

$${\int }_{a_k}^{p_k}P\left(x\right|{\mathcal{D}}_k)dx={\displaystyle \frac{1}{2}}{\int }_{a_k}^{b_k}P\left(x\right|{\mathcal{D}}_k)dx$$

where $P\left(x\right|{\mathcal{D}}_k)$ is the posterior PDF after k probes (or the learned prior at $k=0$). Assuming the posterior is properly normalized, ${\int }_{a_k}^{b_k}P\left(x\right|{\mathcal{D}}_k)dx=1$. Therefore, the probe location $p_k$ is chosen, such that

$${\int }_{a_k}^{p_k}P\left(x\right|{\mathcal{D}}_k)dx={\displaystyle \frac{1}{2}}$$

Now, let us consider the two possible outcomes of the probe at $p_k$.

2.5. Case 1: Observation $x<p_k$

The likelihood function for this observation is

$$P(O_k=\{x<p_k\}|x)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}x<p_k\hfill \\ 0\hfill & \mathrm{if}x\ge p_k\hfill \end{array}\right.$$

The posterior PDF after this observation is given by Bayes’ Theorem:

$$P\left(x\right|O_k=\{x<p_k\},{\mathcal{D}}_k)\propto P(O_k=\{x<p_k\}|x)P\left(x\right|{\mathcal{D}}_k)$$

This implies that the posterior PDF is non-zero only for $a_k\le x<p_k$:

$$P\left(x\right|x<p_k,{\mathcal{D}}_k)=\left\{\begin{array}{cc}{\displaystyle \frac{P\left(x\right|{\mathcal{D}}_k)}{{\int }_{a_k}^{p_k}P\left(t\right|{\mathcal{D}}_k)dt}}\hfill & \mathrm{if}{a}_k\le x<p_k\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

Since the probe $p_k$ was chosen such that ${\int }_{a_k}^{p_k}P\left(t\right|{\mathcal{D}}_k)dt={\displaystyle \frac{1}{2}}$, the posterior PDF over the new interval $[a_k,p_k]$ is proportional to the original posterior and normalized such that the total probability mass in this interval is now 1. The key point is that the probability mass is now concentrated in the interval $[a_k,p_k]$, which contained exactly half of the total probability mass before the observation.

2.6. Case 2: Observation $x>p_k$

The likelihood function for this observation is

$$P(O_k=\{x>p_k\}|x)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}x\le p_k\hfill \\ 1\hfill & \mathrm{if}x>p_k\hfill \end{array}\right.$$

The posterior PDF after this observation is

$$P\left(x\right|O_k=\{x>p_k\},{\mathcal{D}}_k)\propto P(O_k=\{x>p_k\}|x)P\left(x\right|{\mathcal{D}}_k)$$

This implies that the posterior PDF is non-zero only for $p_k<x\le b_k$:

$$P\left(x\right|x>p_k,{\mathcal{D}}_k)=\left\{\begin{array}{cc}{\displaystyle \frac{P\left(x\right|{\mathcal{D}}_k)}{{\int }_{p_k}^{b_k}P\left(t\right|{\mathcal{D}}_k)dt}}\hfill & \mathrm{if}{p}_k<x\le b_k\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

Since ${\int }_{a_k}^{p_k}P\left(t\right|{\mathcal{D}}_k)dt={\displaystyle \frac{1}{2}}$ and ${\int }_{a_k}^{b_k}P\left(t\right|{\mathcal{D}}_k)dt=1$, it follows that ${\int }_{p_k}^{b_k}P\left(t\right|{\mathcal{D}}_k)dt=1-{\displaystyle \frac{1}{2}}={\displaystyle \frac{1}{2}}$. Thus, the posterior PDF is now concentrated in the interval $[p_k,b_k]$, which also contained exactly half of the total probability mass before the observation.

In Bayesian Binary Search, the probe location $p_k$ is strategically chosen at each step to bisect the probability mass of the current posterior distribution. Regardless of whether the target is found to be less than or greater than $p_k$, the subsequent posterior distribution is concentrated over an interval that contained exactly half of the probability mass of the previous distribution. This demonstrates that the posterior update step in BBS, even with a non-uniform prior, is mathematically equivalent to a bisection in the probability density space, where the “bisection” refers to the halving of the probability mass rather than necessarily the physical midpoint of the search interval.

2.7. Search Space Probability Density Function Estimation

The effectiveness of BBS depends on the accuracy of the PDF estimation. Some methods for estimating $p\left(x\right)$ can include the following.

2.7.1. Supervised Learning Approaches

• Gaussian Process Regression (GPR): GPR provides a non-parametric way to model the PDF, offering both a mean prediction and uncertainty estimates.
• Bayesian Neural Networks (BNNs): BNNs combine the flexibility of neural networks with Bayesian inference, allowing for uncertainty quantification in the predictions.
• Quantile Regression: Quantile Regression estimates the conditional quantiles of a response variable, providing a more comprehensive view of the relationship between variables across different parts of the distribution, without assuming a particular parametric form for the underlying distribution.

2.7.2. Unsupervised Learning Approaches

• Gaussian Mixture Models (GMMs): GMMs (typically fit using the Expectation-Maximization (EM) algorithm) can model complex, multimodal distributions by representing the PDF as a weighted sum of Gaussian components.
• Kernel Density Estimation (KDE): KDE is a non-parametric method for estimating PDFs, which can capture complex shapes without assuming a specific functional form.
• Maximum Likelihood Estimation (MLE): MLE is a method of estimating the parameters of a probability distribution by maximizing a likelihood function. It can be used to fit various parametric distributions (e.g., normal, exponential, Poisson) to data, providing a PDF that best explains the observed samples according to the chosen distribution family.

2.8. Probabilistic Bisection Step

BBS leverages a probabilistic bisection step. Instead of dividing the search space at the midpoint, BBS divides it at the median of the current PDF. Formally, at each step, we find $x^{*}$, such that

$${\int }_{low}^{x^{*}}p\left(x\right)dx={\int }_{x^{*}}^{high}p\left(x\right)dx=0.5$$

(1)

After each comparison, we update the PDF based on the result. If the target is found to be in the lower half, we set $p\left(x\right)=0$ for $x>x^{*}$ and renormalize the PDF. Similarly, if the target is in the upper half, we set $p\left(x\right)=0$ for $x<x^{*}$ and renormalize. As proven above, this step is mathematically equivalent to the posterior update step when viewing the algorithm from the lens of Bayesian inference.

We evaluate BBS against a basic binary search on both simulated data across several distributions and a real-world example of binary search/bisection of probing channels in the Bitcoin Lightning Network, for which accessing the search space is an expensive operation (time and liquidity) [20]. We further perform a thorough theoretical analysis located in Appendix A.

Algorithm 1: Bayesian Binary Search

Algorithm 2: Estimate Search Space Probability Density Function (PDF)

Algorithm 3: FindMedian Function

Algorithm 4: UpdatePDF Function

Algorithm 5: Sign Function

3. Experiments

3.1. Experiments on Simulated Data

We evaluated the performance of BBS against binary search on simulated data from several distributions: normal, bimodal, and exponential. The results of the normal distribution experiments are shown below, while the results of the bimodal and exponential distributions can be found in Appendix A. We additionally show experiments for which the estimated search space density function does not match the target search space as measured by Kullback–Leibler (KL) divergence. We demonstrate how BBS performance degrades as the estimated search space density drifts further from the target distribution, and eventually can cause the BBS to perform worse than basic binary search.

3.1.1. Experimental Setup

To evaluate the performance of BBS compared to the standard binary search in the context of normal distributions, we conducted a series of experiments with the following setup.

3.1.2. Distribution Parameters

For the normal distribution experiments, we used the following parameters:

• Mean ($\mathit{\mu }$): 0;
• Standard Deviation ($\mathbf{\sigma }$): 10,000.

3.1.3. Target Value Generation

We generated 1000 target values by sampling from the specified normal distribution. Each target value was obtained by drawing a random sample x from the distribution and applying the floor function to ensure integer targets: $target=\mathrm{floor}\left(x\right)$.

3.1.4. Search Procedure

For each target value, both search algorithms attempt to locate the target within a specified precision ($ϵ$), set to 8. The search space is initialized based on the properties of the normal distribution:

• Lower Bound ($\mathit{lo}$): $\mathrm{floor}(\mu -4.2\times \sigma )$;
• Upper Bound ($\mathit{hi}$): $\mathrm{ceil}(\mu +4.2\times \sigma )$.

The algorithms iteratively narrow down the search space until the target is found within the acceptable error margin.

3.1.5. Metrics Collected

We collected the following performance metrics:

• Number of Steps: Total iterations required to find each target.
• Bracket Size: The range ($hi-lo$) at each step of the search.

3.2. Experiments on Lightning Network Channel Probing

Probing a channel in the Lightning Network is the construction of a fake payment to be sent through the network to obtain information on the balance of a channel. If the payment fails, then the balance on that side of the channel is lower than the payment amount; if it succeeds, then the balance is greater than or equal to the payment amount. Typically, it is currently performed using a basic binary search/bisection. The response of a given probe can be viewed as an oracle, which is a continuous and monotonic function, making it suitable for the bisection method as it guarantees that there is only one root of the function.

Each probe is computationally expensive and fundamentally constrained in this domain due to the max HTLC (Hashed Timelock Contract) limit (483 on Lightning Network Daemon (LND), the most popular Lightning Node implementation) on each channel in the Lightning Node software implementation for security purposes. Probes occupying this limit fundamentally constrain network throughput, which could otherwise process real payments. The Lightning Network can potentially be probed millions of times a week, underscoring the importance of designing an algorithm that optimizes efficiency.

To demonstrate the computational complexity of executing probes on the Lightning Network, we probed 1500 channels from our Lightning Node, with an average hop length of 2.26. The probe time average for each channel has a mean of 3.1 s and a standard deviation of 0.7 s. The density estimation step per channel here in comparison (random forest inference and KDE) takes 0.18 s, and only needs to be performed once per channel. The added overhead of density estimation here is significantly outweighed by the probing computational cost and the domain max HTLC constraint.

For search space density estimation in the Lightning Network channel probing experiment, we detail a random forest model to predict the search target (channel balance in this case). We constructed a PDF from the prediction of the random forest using Kernel Density Estimation (KDE) on the predictions of individual predictors (trees) in the ensemble. Alternatively, we could have used Bayesian Neural Networks or a Gaussian Process Regressor (results in Appendix A), but the random forest yielded superior predictive performance in our experiments [21].

3.2.1. Description of Channel Balance Prediction Task

The Lightning Network can be modeled as a directed graph $G=(V,E)$, where V represents the set of nodes and E represents the set of edges. Each node u and each edge $(u,v)$ have associated features, denoted by ${\mathbf{x}}_u\in {\mathbb{R}}^k$ for nodes and ${\mathbf{e}}_{(u,v)}$ for edges, which contain specific information about those nodes and edges. Additionally, each edge $(u,v)$ has a scalar value $y_{(u,v)}\ge 0$, representing the pre-allocated balance for transactions from u to v.

Graph G has the constraint that if an edge exists in one direction, it must also exist in the opposite direction, i.e., $(u,v)\in E\iff (v,u)\in E$. The set of two edges between any two nodes is called a channel, denoted as $\left\{\right(u,v),(v,u\left)\right\}$. For simplicity, we represent a channel by the set of its two nodes: $\{u,v\}$. The total capacity of the channel $\{u,v\}$ is defined as $c_{\{u,v\}}=y_{(u,v)}+y_{(v,u)}$.

We are provided with the total channel capacities $c_{\{u,v\}}$ for all channels in the graph, but we only know the individual values $y_{(u,v)}$ for a subset of edges. Note that knowing $y_{(u,v)}$ allows us to determine $y_{(v,u)}$, since $y_{(v,u)}=c_{\{u,v\}}-y_{(u,v)}$. Therefore, we can focus on predicting $y_{(u,v)}$.

Moreover, since we are given $c_{\{u,v\}}$ for all edges, and we know that $0\le y_{(u,v)}\le c_{\{u,v\}}$, we also have that $y_{(u,v)}=p_{(u,v)}{c}_{\{u,v\}}$, where $0\le p_{(u,v)}\le 1$. Intuitively, $p_{(u,v)}$ is the proportion of the channel capacity which belongs to the $(u,v)$ direction. From this, we see that we focus on predicting $p_{(u,v)}$, and then use it to obtain $y_{(u,v)}$.

Therefore, our primary task is to predict $p_{(u,v)}$ for all edges where it is not observed.

3.2.2. Data Collection and Preprocessing

The data used in this experiment are a combination of publicly available information from the Lightning Network and crowdsourced information from nodes in the network. A snapshot of the network from 15 December 2023 was used in this experiment. Balance information for each node is represented by its local balance, recorded at one-minute intervals over the preceding hour. These data were converted into a probability density function (PDF) through Kernel Density Estimation. Subsequently, a balance value was sampled from each PDF, serving as the representative local balance for the corresponding channel within the dataset.

3.3. Methodology

The following section outlines the details of our modeling approach.

3.3.1. Modeling

We will predict $p_{(u,v)}$ by learning a parametric function:

$${\widehat{p}}_{(u,v)}=f_{\mathsf{\Theta }}(u,v,G,{\mathbf{x}}_u,{\mathbf{x}}_v,{\mathbf{e}}_{(u,v)},c_{\{u,v\}})$$

where $\mathsf{\Theta }$ are learnable weights, ${\mathbf{x}}_u$, ${\mathbf{x}}_v$, and ${\mathbf{e}}_{(u,v)}$ are the node and edge features, respectively, while $c_{\{u,v\}}$ is the capacity of the channel. While several choices are possible for $f_{\mathsf{\Theta }}$, such as multi-layer perceptrons ([22]) or Graph Neural Networks, we focus on random forests for this work given their simplicity and efficacy. In particular, our random forest (RF) model operates on the concatenation of the features of the source and destination nodes as well as the edge features:

$${\widehat{p}}_{(u,v)}=\mathrm{RF}(x_u\left|\right|z_u\left|\right|x_v\left|\right|z_v\left|\right|e_{(u,v)})$$

The model was trained using a Mean Squared Error loss.

3.3.2. Node Features

• Node Feature Flags
  A vector of 0–1 indicating which of the features each node supports. For example, feature flag #19, the wumbo flag.
• Capacity Centrality
  The node’s capacity divided by the network’s capacity. This indicates how much of the network’s capacity is incident to a node.
• Fee Ratio
  Ratio of the mean cost of a node’s outgoing fees to the mean cost of its incoming fees.

3.3.3. Edge Features

• Time Lock Delta: The number of blocks a relayed payment is locked into an HTLC.
• Min HTLC: The minimum amount this edge will route (denominated in millisats).
• Max HTLC msat: The maximum amount this edge will route (denominated in millisats).
• Fee Rate millimsat: Proportional fee to route a payment along an edge (denominated in millimillisats).
• Fee Base msat: Fixed fee to route a payment along an edge (denominated in millisats).

3.3.4. Positional Encodings

Positional encoding is essential for capturing the structural context of nodes, as graphs lack inherent sequential order. Utilizing eigenvectors of the graph Laplacian matrix as positional encodings provides a robust solution to this challenge. These eigenvectors highlight key structural patterns, enriching node features with information about the overall topology of the graph [23]. By integrating these spectral properties, machine learning models can effectively recognize and utilize global characteristics, enhancing performance in tasks like node classification and community detection.

3.3.5. Concatenated Prediction ML Model

Our model predicts $p_{(u,v)}$ as a function of the concatenation of the features of the source and destination nodes as well as edge features:

$${\widehat{p}}_{(u,v)}=\mathrm{RF}(x_u\left|\right|x_v\left|\right|e_{(u,v)})$$

3.3.6. Model Training Details and Performance

We set aside $10\%$ of the observed $y_{(u,v)}$ as our test set and $10\%$ as our validation set. The RF model trained for balance prediction has an MAE of 1.08, an R of 0.612, and $R^2$ of 0.365. For the BBS experiments, we use the predictions on the validation set (89 channels). For validation set channels, we feed the predictions of each tree in the ensemble (100) into a Kernel Density Estimation (KDE) process to construct the search space PDF. Using these constructed PDFs, we compare BBS with basic binary search in measuring how many probes it would take to ascertain the balance of a given channel.

4. Discussion

Our experimental results demonstrate the potential of Bayesian Binary Search (BBS) as a promising alternative to classical binary search/bisection, particularly in scenarios where the search space exhibits non-uniform distributions and is costly to access. The performance improvements observed in both simulated environments and real-world applications, such as probing payment channels in the Bitcoin Lightning Network, highlight the promise of BBS (Figure 1 and Figure 2,

Table 1. $\mu =0.0,\sigma =\mathrm{10,000.0},N=500$.

$\mathit{ϵ}$	Percent Decrease	Basic Mean Steps	BBS Mean Steps
1	6.30%	16.47 ± 0.50	15.43 ± 1.19
2	6.36%	15.68 ± 0.47	14.68 ± 1.15
3	6.17%	15.00 ± 0.00	14.07 ± 1.15
4	8.76%	15.00 ± 0.00	13.69 ± 1.11
5	5.06%	14.15 ± 0.36	13.43 ± 1.02
6	6.16%	14.00 ± 0.00	13.14 ± 1.09
7	8.07%	14.00 ± 0.00	12.87 ± 1.16
8	9.39%	14.00 ± 0.00	12.69 ± 1.12
9	10.27%	14.00 ± 0.00	12.56 ± 1.08
10	6.14%	13.28 ± 0.45	12.47 ± 1.01
11	4.63%	13.00 ± 0.00	12.40 ± 0.99
12	5.88%	13.00 ± 0.00	12.24 ± 1.05
13	7.43%	13.00 ± 0.00	12.03 ± 1.13
14	8.51%	13.00 ± 0.00	11.89 ± 1.15
15	9.32%	13.00 ± 0.00	11.79 ± 1.15
16	9.89%	13.00 ± 0.00	11.71 ± 1.12
17	10.49%	13.00 ± 0.00	11.64 ± 1.11
18	11.03%	13.00 ± 0.00	11.57 ± 1.09
19	11.42%	13.00 ± 0.00	11.52 ± 1.05
20	8.38%	12.53 ± 0.50	11.48 ± 1.03
21	4.52%	12.00 ± 0.00	11.46 ± 1.01
22	4.80%	12.00 ± 0.00	11.42 ± 0.97
23	5.07%	12.00 ± 0.00	11.39 ± 0.95
24	5.83%	12.00 ± 0.00	11.30 ± 1.01
25	7.12%	12.00 ± 0.00	11.15 ± 1.10
26	8.03%	12.00 ± 0.00	11.04 ± 1.13
27	8.43%	12.00 ± 0.00	10.99 ± 1.14
28	9.17%	12.00 ± 0.00	10.90 ± 1.15
29	9.55%	12.00 ± 0.00	10.85 ± 1.15
30	10.02%	12.00 ± 0.00	10.80 ± 1.15
31	10.33%	12.00 ± 0.00	10.76 ± 1.14
32	10.57%	12.00 ± 0.00	10.73 ± 1.13

and

Table 2. Lightning Channel Probing Comparison.

$\mathit{ϵ}$	Percent Decrease	Basic Mean Steps	BBS Mean Steps
128	3.07%	14.26 ± 2.26	13.82 ± 2.35
256	3.31%	13.26 ± 2.26	12.82 ± 2.35
512	3.57%	12.26 ± 2.26	11.82 ± 2.35
1024	3.89%	11.26 ± 2.26	10.82 ± 2.35
2048	4.27%	10.26 ± 2.26	9.82 ± 2.35
4096	4.73%	9.26 ± 2.26	8.82 ± 2.35
8192	5.03%	8.26 ± 2.26	7.84 ± 2.30
16,384	5.73%	7.26 ± 2.26	6.84 ± 2.29

). However, it is important to consider these findings in a broader context and acknowledge the limitations and implications of our work. The results observed here are also consistent with some theoretical analysis from [24,25], and the BBS framework provides a novel perspective on how probabilistic machine learning techniques can be used to achieved these efficiencies.

4.1. Theoretical Alignment and Performance Gains

The superior performance of BBS over classical binary search in non-uniform distributions aligns with our theoretical expectations. By leveraging probabilistic information about the search space, BBS can make more informed decisions at each step, leading to faster convergence on average. This is particularly evident in our simulated experiments with skewed distributions, where BBS consistently required fewer steps to locate the target. The ability of BBS to adapt to the underlying distribution of the search space represents an advance in the design of search algorithms, and a fusion of search theory and probabilistic machine learning/statistical learning.

4.2. Real-World Application and Implications

In the context of the Bitcoin Lightning Network, the improvement in probing efficiency could have significant practical implications. Faster and more accurate channel capacity discovery can enhance routing algorithms, potentially leading to improved transaction success rates. Crucially, probes in the Lightning Network consume network-wide computational resources, as each probe requires processing by multiple nodes along potential payment routes. By using BBS to enhance probe efficiency, we effectively reduce unnecessary network load, akin to removing spam from the system. This reduction in network overhead could lead to improved overall network performance and scalability.

4.3. Limitations and Challenges

Despite the promising results, several limitations of our study should be acknowledged:

• Computational Overhead: The improved search efficiency of BBS comes at the cost of increased computational complexity, particularly in the PDF estimation step. This overhead may offset the reduction in search steps for certain applications, especially those dealing with small search spaces or requiring extremely fast operation.
• PDF Estimation Accuracy: The performance of BBS is heavily dependent on the accuracy of the PDF estimation. In scenarios where the underlying distribution is highly complex or rapidly changing, the chosen PDF estimation method may struggle to provide accurate probabilities, potentially leading to suboptimal performance.

4.4. Future Research Directions

Our findings open up several exciting avenues for future research:

• Adaptive PDF Estimation: Developing methods to dynamically adjust the PDF estimation technique based on the observed search space characteristics could further improve the robustness and efficiency of BBS.
• Theoretical Bounds: While we provide empirical evidence of BBS’s effectiveness, deriving tighter theoretical bounds on its performance under various distribution types would strengthen its theoretical foundation.
• Application to Other Domains: Exploring the applicability of BBS to other areas such as database indexing, computational biology, or optimization algorithms could reveal new use cases and challenges.

4.5. Conclusions

Bayesian Binary Search represents a significant step towards more adaptive and efficient search algorithms. By incorporating probabilistic information, BBS demonstrates the potential to outperform classical binary search in non-uniform distributions, as evidenced by our experiments in both simulated and real-world scenarios. In the context of the Lightning Network, BBS not only improves search efficiency but also contributes to reducing unnecessary network load, potentially enhancing the overall performance and scalability of the system. The promising results open up new possibilities for optimizing search processes in various domains. As we continue to navigate increasingly complex and data-rich environments, algorithms like BBS that can adapt to the underlying structure of the problem space will become increasingly valuable. Future work in this direction has the potential to not only improve search efficiency but also to deepen our understanding of how probabilistic approaches can enhance fundamental algorithms in computer science and their real-world applications.