A QUBO Formulation of the Stereo Matching Problem for D-Wave Quantum Annealers

Abstract

In this paper, we propose a methodology to solve the stereo matching problem through quantum annealing optimization. Our proposal takes advantage of the existing Min-Cut/Max-Flow network formulation of computer vision problems. Based on this network formulation, we construct a quadratic pseudo-Boolean function and then optimize it through the use of the D-Wave quantum annealing technology. Experimental validation using two kinds of stereo pair of images, random dot stereograms and gray-scale, shows that our methodology is effective.

1. Introduction

Computer vision is an interdisciplinary field of research with almost six decades of theoretical and algorithmic developments [1,2] that focuses on developing mathematical techniques and algorithms that aim at enabling computers to identify, analyze, and understand information from elements of imagery [3]. Computer vision has many links and common interests with Artificial Intelligence and Machine Learning.

Stereo vision refers to the ability to extract information about the 3-D structure and distance of a scene from two or more images taken from different viewpoints. The most basic stereo system consists of two cameras (left and right) and any stereo system must solve two problems:

• The stereo matching problem [4]: Which parts of the left and right images are projections of the same scene element?
• The reconstruction problem, which is stated as follows: given a number of corresponding parts of the left and right images, what can we say about the 3-D locations and structures of the observed objects?

In this paper, we focus on the stereo matching problem using a basic stereo system. This problem is difficult to solve because some parts of the scene are visible only by either the left or right camera but not by both; therefore, a stereo system must also be able to select the image parts to be matched [5]. Stereo vision has many applications, such as in photogrammetry [6], stereo-based head tracking [7], volumetric and 3-D surface reconstruction [8], video-based walkthroughs [9], and stereo-based autonomous navigation [10], among others.

Nowadays, there is an increasing amount of interest in applying quantum computing techniques on machine learning tasks. The research area of quantum machine learning (QML) [11,12] has as its main objective designing quantum algorithms and data representation using quantum states that outperform classical algorithms in machine learning problems. Examples of recent progress in QML are the proposals of quantum principal component analysis of classical data [13] and quantum support vector machine [14] (both papers use quantum states that encode classical data as input), as well as deep quantum learning [15] and sampling of a quantum Boltzman machine [16] (these two papers have classical data as input). More recent advances in QML can be found in [17,18].

Quantum hardware technologies such as the quantum circuit-based IBMQ computer [19] and the D-Wave [20] quantum annealer machine have motivated the development of applications in QML. In particular, the D-Wave computer has evolved and grown significantly with respect to the number of qubits, allowing the posing of problems of interest [21,22,23,24]. For instance, the current generation of D-Wave 2000Q processors has almost 2048 working superconducting qubits connected in a graph topology known as Chimera.

In this paper, we present a methodology for writing a Quadratic Unconstrained Binary Optimization (QUBO) formulation of the stereo matching problem via a graph-cut approach, followed by using this QUBO formulation to simulate a quantum annealing algorithm on D-Wave’s simulation software. This paper is meant to be a contribution to the nascent area of quantum algorithms to solve relevant problems in science, engineering and other fields, like finance.

Our method could also be used to implement NP-hard problem instances in novel architectures of classical hardware, like the Ising model architectures proposed by Fujitsu [25] and Hitachi [26] research laboratories (examples of NP-hard problems are combinatorial optimization problems. A concise introduction to NP-hardness can be found in [27]). Furthermore, since the stereo matching problem is of paramount importance in computer vision, several classical algorithms have been developed to solve it, among them, graph-cut-based methods (we provide a brief account of algorithms for stereo matching in Section 2). Although the stereo matching problem is in the $\mathsf{P}$ class, a minimum upper bound for the computational complexity is unknown; hence, the development of novel approaches is pertinent. In this sense, the proposal presented in this manuscript combined with present and future research on quantum annealing-based algorithms could be used in future work to develop new and less expensive algorithms.

Our work consists of two main goals: (a) to express a problem of interest as a QUBO problem and (b) to find a minor embedding of the QUBO problem into the Chimera graph architecture. Before applying our methodology, we represent the stereo matching problem as a graph-cut problem by constructing an edge weighted graph with two special vertices. This weighted graph has the following property: finding a subset of edges with minimum cost whose removal disconnects the special vertices is equivalent to solving the stereo matching problem. Therefore, for step (a), we construct a pseudo-Boolean expression of degree three for the graph-cut problem, and we use a reduction method to obtain a QUBO expression to be minimized. For step (b), a minor embedding is found by using the software tools provided by the D-Wave solver application programming interface (SAPI). To validate our methodology for the solution to the stereo matching problem, we present the results of the minimization of the QUBO expression using the classic solver qbsolv developed by D-Wave.

This paper is organized as follows: In Section 2, we succinctly introduce the basic notions of optimization via quantum annealing and we describe our methodology for solving the stereo matching problem using QUBO and D-Wave technology; in Section 3, we present our experiments and results; finally, we present our conclusions in Section 4.

2. Results

2.1. Quantum Annealing

Quantum annealing (QA) [28,29,30,31,32,33] can be seen as a quantum version of Classical Simulated Annealing (SA) [32,33,34,35]. SA uses the process of annealing in which a solid in a thermal bath is heated up by increasing the temperature, followed by a cooling by slowly lowering the temperature of the bath. The annealing allows the particles in the solid to go from a random configuration or high energy to a lattice configuration or low energy. This process was found to be equivalent to the problem of finding the configuration with the minimum amount of energy or the cost function of a given combinatorial optimization problem. QA employs quantum fluctuations to anneal the system down to its minimum energy state [33]. The most used physical system for QA is the Ising spin glass model on N particles described by

$$H_p=\sum _{1\le i\le N}{h}_i{\sigma }_i^z+\sum _{1\le i<j\le N}{J}_{ij}{\sigma }_i^z{\sigma }_j^z$$

(1)

where ${\sigma }_i^z$ is the Pauli matrix z acting on particle i, $h_i$ describes the magnetic field on particle i, and $J_{ij}$ is the coupling strength between particles i and j. The minimum energy state or ground state of the Hamiltonian $H_p$ corresponds to a configuration $\mathbf{s}=(s_1,\cdots ,s_N)\in {\{+1,-1\}}^N$ of spins that minimizes the following energy function

$$E(\mathbf{s})=\sum _{1\le i\le N}{h}_i{s}_i+\sum _{1\le i<j\le N}{J}_{ij}{s}_i{s}_j.$$

(2)

It is known that the problem of finding a configuration ${\mathbf{s}}^{*}$ with minimum energy $E({\mathbf{s}}^{*})$ is an NP-complete problem [36] (although some special cases of the Ising model can be solved in polynomial time. However, those cases are not part of our study). A related problem with the same complexity is obtained by changing the variable $x_i=(1+s_i)/2$ for $i=1,\cdots ,N$. In other words, finding a configuration $\mathbf{x}\in {\{0,1\}}^N$ such as $E(\mathbf{x})$ that is minimum also corresponds to an $\mathsf{NP}-\mathsf{complete}$ problem [37].

One of the schemes to realize QA is through adiabatic quantum evolution from the ground state of an initial Hamiltonian to the ground state of a final Hamiltonian which corresponds to the solution of a given problem [38]. According to this scheme, a time-dependent Hamiltonian takes the form

$$H(\tau )=A(\tau )H_0+B(\tau )H_p$$

(3)

where $\tau =t/t_a$ for $0\le t\le t_a$, is $t_a$ is the total annealing time, and the initial Hamiltonian $H_0=-{\sum }_{1\le i\le N}{\sigma }_i^x$ is responsible for quantum tunnelling among the localized classical states corresponding to the eigenstates of the Hamiltonian $H_p$. In the current generation of the D-Wave 2000Q processor, functions $A(\tau )$ and $B(\tau )$ are defined so that at time $\tau =0$, the influence of the Hamiltonian $H_0$ is predominant against $H_p$. As time evolves from $\tau =0$ to $\tau =1$, the influence of the Hamiltonian $H_p$ increases, while $H_0$ fades away.

One then can solve the time-dependent Schrödinger equation with the Hamiltonian $H(t)$ to obtain an approximate solution to the dynamics of the system. Consider the Hamiltonian $H(t)=\tilde{H}(t/t_a)=\tilde{H}(\tau )$ such that $0\le \tau \le 1$, and let us denote by $|l;\tau \rangle$ the instantaneous eigenvector of $\tilde{H}(\tau )$ corresponding to the instantaneous eigenvalue ${\lambda }_l(\tau )$. Then,

$$\tilde{H}(\tau )|l;\tau \rangle ={\lambda }_l(\tau )|l;\tau \rangle$$

(4)

with

$${\lambda }_0(\tau )\le {\lambda }_1(\tau )\le \cdots \le {\lambda }_{2^N-1}(\tau ).$$

(5)

The Adiabatic theorem asserts that for sufficiently large $t_a$,

$$\underset{t_a\to \infty }{lim}\left|\langle l=0;\tau =1|\psi (t_a)\rangle \right|=1$$

(6)

for some solution $\psi (t)$ to the Schrödinger equation with the Hamiltonian $H(t)$. Consequently, the state $\psi (t_a)$ will be very close to the ground state of the Hamiltonian $H_p$ with a high probability. A sufficient condition for the algorithm running time that is needed to satisfy the Adiabatic theorem is

$$t_a\gg \frac{{\epsilon }_{max}}{g_{min}^2}$$

(7)

where

$$g_{min}=\underset{0\le \tau \le 1}{min}({\lambda }_1(\tau )-{\lambda }_0(\tau ))$$

(8)

and

$${\epsilon }_{max}=\underset{0\le \tau \le 1}{max}|\langle l=1;\tau |\frac{d\tilde{H}}{d\tau }|l=0;\tau \rangle |.$$

(9)

A more rigorous formulation of the Adiabatic theorem can be found in [39]. It must be noted that in realistic physical settings, including the D-Wave processor, the QA dynamics of a system do not necessary evolve adiabatically. There are several reasons including thermal fluctuations and additive noise from the environment. Other considerations to take into account when solving problems on a D-Wave processor are the size of the device or number of qubits in the current architecture and the precision of the Ising parameters, such as the magnetic field values $h_j$ and the coupler strength $J_{ij}$ between pairs of qubits. A concise introduction to quantum annealing algorithms, their potential relevance for the advancement of key problems in theoretical computer science, as well as mathematical methods to implement quantum annealing algorithms in D-Wave quantum annealers can be found in  [27].

2.1.1. QUBO Formulation Approach to the Stereo Matching Problem

The stereo matching problem is of key importance in computer vision [5,40]. Without any loss of generality, a stereo setup consists of two spatially separated cameras located at the same height with parallel optical axes. A stereo pair of images, $I_l$ and $I_r$, are the recorded images by the left and right cameras, $c_l$ and $c_r$, respectively. For a point P in a scene observed by a stereo setup, it (point P) will be projected into the left and right images with intensities $I_l(x_1,y)$ and $I_r(x_2,y)$, respectively. Notice that, in a stereo setup, a point P will be projected at the same row y in both left and right images, whilst determining the columns $x_1$ and $x_2$ of projection is a difficult problem [4]. Let us now state the following problem.

Problem 1 (Stereo matching).

Given a stereo pair of images $(I_l,I_r)$, find, for every pixel $I_l(x_1,y)$ in the left image, its corresponding projection $I_r(x_2,y)$ in the right image (see Figure 1).

The solution to problem 1 involves calculating the disparity for every pixel in the left image. If the calibration parameters of the stereo setup are given [40], then it is possible to infer the depth at every point in the scene. Early approaches to solve problem 1 are those based on block matching techniques [4], dynamic programming [41], belief propagation [42] and graph-cut [43] based techniques, among others. In the following section, we formulate the stereo matching problems in terms of graph-cuts.

2.1.2. Energy Function

Let $\mathcal{P}$ be the image domain or set of pixels in the left image and let $\mathcal{L}$ be the set of labels or disparities; a labeling is a map

$$l:\mathcal{P}\to \mathcal{L}$$

(10)

such that for each pixel $p\in \mathcal{P}$, l assigns a label $l(p)=l_p$ to the pixel p.

Given a labeling l, its cost is defined as

$$\mathcal{E}(l)=\sum _{p\in \mathcal{P}}{D}_p(l_p)+\sum _{\{p,q\}\in \mathcal{N}}{V}_{pq}(l_p,l_q)$$

(11)

where $D_p$ models the cost of assigning label $l_p$ to the pixel p, $V_{pq}$ models the cost of assigning $l_p$ to p and $l_q$ to q, with p and q being adjacent pixels, and $\mathcal{N}$ is the set of neighbor pairs $\{p,q\}$.

Graph-cut techniques consist of constructing a graph with weighted edges and two special vertices, s and t. In this model, finding a cut with minimum cost that separates s and t is equivalent to minimizing function $\mathcal{E}(l)$ [44,45].

2.1.3. Graph Construction

The following graph construction is based on the Boykov and Kolmogorov approach for Min-Cut/Max-Flow Algorithms for Energy Minimization [44].

Let $G=(V,E,c)$ be an edge weighted graph where V is the set of vertices with two special vertices, the source s and the sink t, E is the set of undirected edges, and $c:E\to {\mathbb{Z}}^{+}$ is a function that assigns a positive value to each edge in E. A cut in G is a set of edges whose removal from E separates the source from the sink, and its cost is the sum of their weights. Let $\mathcal{L}=\{0,\cdots ,L\}$ be the set of possible disparities or labels. The graph G is constructed as follows:

• To each pixel p in the image, we associate a chain composed of $L+2$ vertices, say $p_0,p_1,\cdots ,p_{L+1}$. These vertices are connected by edges called t-links, $\{e_{p_0,p_1},e_{p_1,p_2},\cdots ,e_{p_L,p_{L+1}}\}$, where $e_{p_0,p_1}=\{p_0,p_1\},e_{p_1,p_2}=\{p_1,p_2\},\cdots ,e_{p_L,p_{L+1}}=\{p_L,p_{L+1}\}$. For a disparity range from 0 to L values, and for each pixel p, we will have $L+2$ vertices in the chain and $L+1$ t-links between these vertices; each edge defines a labeling with one specific value. Additionally, there are two t-links that connect the first vertex with the source and the last vertex with the sink. Hence, there exist $L+3$ such edges in the graph for each pixel p. The weights of those edges directly depend on the function $D_p$, which specifies the cost of applying a specific label to a pixel.
• To each pair of neighbor pixels, p and q, there will be links that connect the corresponding chains with edges called n-links. For instance, let $p_0,p_1,\cdots ,p_{L+1}$ and $q_0,q_1,\cdots ,q_{L+1}$ be the chain vertices for two neighbor pixels p and q; the n-links between chains are $e_{p_0,q_0}=\{p_0,q_0\},e_{p_1,q_1}=\{p_1,q_1\},\cdots ,e_{p_{L+1},q_{L+1}}=\{p_{L+1},q_{L+1}\}$. Usually a 4-neighborhood is assumed for every pixel. The weights of these edges should reflect the penalty when assigning different labels to neighboring pixels.

For a given stereo pair of images of size $N\times M$ pixels and a disparity range L, the total number of vertices in the constructed graph is $(L+2)NM+2$ and the number of edges is $NM(L+3)+((M-1)N+(N-1)M)(L+2)$. In Figure 2, we present an example of the graph topology for an image of $5\times 5$ pixels and $L=3$, in which we show the t-links and n-links (red edges) for the central pixel and its neighbor pixels (right). In this example, there are 127 vertices and 350 edges.

A cut separating the source from the sink severs t-links as well as n-links. The severed t-links directly define the labels assigned to each pixel. For instance, if a cut severs the t-link $t_{p_j}=\{p_j,p_{j+1}\}$ with $0\le j\le L$ for a pixel p, then the assigned label to p is j. Therefore, the cut must sever exactly one t-link at every pixel.

The cost of a t-link will correspond to the disparity of a given pixel. If the assumption that we have a certain disparity d at pixel $p=(x_1,y)$ is correct, the intensity $I_l(x,y)$ observed in the left camera at position $(x,y)$ should be similar to the intensity $I_r(x-d,y)$ in the right camera observed at position $(x-d,y)$, because both pixels depict the same scene object. Consequently, the cost of a given t-link $t_{p_d}=\{p_d,p_{d+1}\}$ with $0\le d\le L$ can be set to the $L_2$ norm of the intensity difference between corresponding pixels in both camera images:

$$c(t_{p_d})={|I_l(x,y)-I_r(x-d,y)|}^2+C$$

(12)

where C is a constant that is chosen to be sufficiently large in order to ensure that exactly one t-link is severed by the cut for each pixel. If the intensity difference between $I_l(x,y)$ and $I_r(x-d,y)$ is low enough, this indicates that the corresponding disparity is a good solution and consequently, $c(t_{p_d})$ is low. The constant C is set to be slightly larger than the sum of the weights of all penalty n-links belonging to pixel p,

$$C_p=1+(L-1)\sum _{q\in {\mathcal{N}}_p}{\lambda }_{pq}$$

(13)

where ${\mathcal{N}}_p$ defines the neighborhood of p.

The pairwise cost function $V_{pq}$ depends on the difference between labels $l_p$ and $l_q$. In the simplest case, $V_{pq}$ can be set to the absolute difference $|l_p-l_q|$. Therefore,

$$V_{pq}(l_p,l_q)={\lambda }_{pq}|l_p-l_q|$$

(14)

where ${\lambda }_{pq}$ is a weighting factor for setting the relative importance of the smoothness term, which can be chosen differently for each pair of pixels, but is often set to a constant $\lambda ={\lambda }_{pq}$ for all $p,q\in \mathcal{N}$. Given an n-link $n_{p_k{q}_k}$ for two neighbor pixels p and q, the cost of $n_{p_k{q}_k}$ is equal to $c(n_{p_k{q}_k})={\lambda }_{pq}$ for all $1\le k<L$.

2.1.4. QUBO Formulation of the Stereo Matching Problem via the Minimum Multicut Problem

The graph cut formulation of the stereo matching problem is a special case of the following general problem:

Problem 2 (Minimum multi-cut).

Given an edge weighted graph $G=(V,E,c)$ and a set $S=\{(s_1,t_1),\cdots ,(s_k,t_k)\}$ of k pairs of vertices, find a multi-cut with minimum cost, i.e., a set $E^{\prime }\subseteq E$ such that the removal of $E^{\prime }$ from E disconnects $s_i$ from $t_i$ for every pair $(s_i,t_i)\in S$, where the cost of $E^{\prime }$ is given as ${\sum }_{\{u,v\}\in E^{\prime }}c(\{u,v\})$ (see Figure 3).

When $k=1,2$, the minimum multi-cut problem can be solved in polynomial time [46], and it becomes NP-hard for $k\ge 3$ for general graphs [47]. In the special case where G is restricted to trees and for arbitrary k, Problem 2 remains NP-hard [48].

Let us formulate Problem 2 for $k=1$ as a QUBO expression. Let $G=(V,E,c)$ be a weighted graph and $(s,t)$ be the pair of vertices to be disconnected. For each edge $\{u,v\}\in E$, let us associate a Boolean variable $y_{uv}$ such that $y_{uv}=1$ if the edge $\{u,v\}$ is selected for a cut and $y_{uv}=0$ otherwise. Similarly, for each vertex $v\in V$, let us associate a Boolean variable $x_v$ such that $x_v=1$ if $v\in U$ and $x_v=0$ in the other cases where U is a subset of V.

Let $H_{\mathrm{problem}}$ be defined as

$$H_{\mathrm{problem}}=H_{\mathrm{cost}}+H_{\mathrm{penalty}}.$$

(15)

The first term, $H_{\mathrm{cost}}$, in Equation (15) expresses the cost of the selected edges to be removed from the graph G, and the second term, $H_{\mathrm{penalty}}$, has the purpose of introducing a penalization if the selected edges do not correspond to a cut in G.

The term $H_{\mathrm{cost}}$ can be easily defined as

$$H_{\mathrm{cost}}=\sum _{\{u,v\}\in E}{y}_{uv}·c(\{u,v\})$$

(16)

which gives us the cost of the selected edges to be removed.

The second term, $H_{\mathrm{penalty}}$, is constructed as

$$\begin{array}{ccc}\hfill H_{\mathrm{penalty}}& =& \alpha [1-x_s-x_t+2x_s{x}_t+\hfill \\ & & \sum _{\{u,v\}\in E}(1-y_{uv})(x_u+x_v-2x_u{x}_v)].\hfill \end{array}$$

(17)

The penalty term satisfies the criterion that if the selected edges form a cut, then $H_{\mathrm{penalty}}=0$, and $H_{\mathrm{penalty}}>0$ in other cases. The penalty term is constructed based on the observation that the selected edges form a cut if there exists a partition $(U,\overline{U})$ with $U\subseteq V$ such that $s\in U$ and $t\in \overline{U}$, and every edge $e\in E$ with $y_e=0$, has its extreme vertices on only one side of the partition, either in U or in $\overline{U}$. The coefficient $\alpha$ is a positive value which is used to ensure that solutions that do not constitute a cut are avoided.

The expression $H_{\mathrm{penalty}}$ given in (17) corresponds to a pseudo-Boolean function of the third degree which can be converted into a QUBO expression using a reduction method. Among reduction methods we find the Boros algorithms [37,49,50] and the Freedman [51] and Ishikawa [52] methods. A review of quadratization methods in pseudo-Boolean optimization can be found in [53].

The Ishikawa method says that every positive cubic term $x_1{x}_2{x}_3$ can be expressed as a quadratic term with

$$\begin{array}{ccc}\hfill x_1{x}_2{x}_3& =& x_1{x}_2+x_1{x}_3+x_2{x}_3+\underset{z\in \{0,1\}}{min}z(1-x_1-x_2-x_3)\hfill \\ & \le & x_1{x}_2+x_1{x}_3+x_2{x}_3+z(1-x_1-x_2-x_3)\hfill \end{array}$$

(18)

where z is an ancilla Boolean variable. By applying the Ishikawa method to Equation (17) (which is the method that uses the smallest number of ancilla variables to obtain a quadratic expression) we have

$$\begin{array}{ccc}\hfill H_{\mathrm{penalty}}^{\mathrm{qubo}}& =& \alpha [1-x_s-x_t+2x_s{x}_t+\sum _{\{u,v\}\in E}(x_u+\hfill \\ & & x_v+2w_{uv}+x_u{y}_{uv}+x_v{y}_{uv}-2x_u{w}_{uv}-\hfill \\ & & 2x_v{w}_{uv}-2y_{uv}{w}_{uv})].\hfill \end{array}$$

(19)

where $w_{uv}$ for each $\{u,v\}\in E$ are ancilla variables.

The final quadratic cost function to be minimized can be written as

$$H_{\mathrm{problem}}^{\mathrm{qubo}}=H_{\mathrm{cost}}+H_{\mathrm{penalty}}^{\mathrm{qubo}}$$

(20)

which requires a number $2\left|E\right|+\left|V\right|$ of variables to represent the minimum multi-cut problem for a single pair of vertices.

3. Experiments and Discussion

In this section, we prove the concept of the solution to the stereo matching problem using the methodology described in Section 2. The following steps summarise our methodology:

(i)

Construct the weighted graph described in Section 2.1.3 for the given stereo pair of images.

(ii)

Formulate the stereo matching problem as the minimum multi-cut problem for the pair $(s,t)$ using the weighted graph from the previous step.

(iii)

Construct the QUBO expression given in Equation (20) from the weighted graph in step (ii) for a single pair.

(iv)

Embed into the Chimera graph topology of the D-Wave computer.

(v)

Find the minimum energy solution to the QUBO/Ising problem by quantum annealing.

We consider two case studies of stereo pair images: (a) a binary random dot stereogram (BRDS) and (b) a gray scale imaged object, both with a resolution of 15 × 15 pixels. At this stage of our research, we are unable to study bigger size images because of the large resources needed to represent the stereo matching problem with the current limitations of quantum hardware. In case (a), we generate a BRDS as follows: a left image is created by setting each pixel randomly to either black or white. Then, a copy of this image is made and it is called the right image. Take a region in the right image and shift that region horizontally by d pixels to the left. The pixels vacated by shifting the region are filled in with random values. The shifted region simulates an object with a disparity of d pixels. In Figure 4 (top), we show a BRDS pair of images with a centered square region of size 7 × 7 pixels shifted by $d=3$ pixels.

For case (b), we crop a region of size 15 × 15 pixels from a stereo pair of images of a larger size. The cropped images have a maximum disparity range of five pixels. Figure 5 (top) shows a stereo pair showing a corner of an object on a background. In cases (a) and (b), the regions of interest (ROIs) to be matched are of sizes $15\times 12$ and $15\times 10$ pixels, respectively. In

Table 1. For case studies (a) and (b) in Figure 4 and Figure 5, respectively, the region of interest (ROI), the disparity range, the dimension of the weighted graph described in Section 2.1.3, and the number of logical variables used to formulate the minimum multi-cut problem given in Equation (19) are shown.

Case	ROI	L	$\left|\mathbf{V}\right|$	$\left|\mathbf{E}\right|$	${|\mathbf{H}}_{\mathbf{problem}}^{\mathbf{qubo}}|$
(a)	$15\times 12$	3	902	2745	6392
(b)	$15\times 10$	5	1052	3125	7302

, the size of the weighted graph described in Section 2.1.3 for step (i) in our methodology is shown. Despite the large number of vertices and edges needed to formulate the stereo matching as a graph-cut problem, it is possible to use simplification techniques to limit the volume of the graph [45,54,55].

Figure 6 shows the weighted graphs for cases (a) and (b) where the vertices $s,t$ and their incident edges are omitted for better visualization. The t-links in the graph, represented by wide lines, are colored according to their weights, and the n-links are colored in black as thinner lines. As it can be seen in Figure 6a, we have only two possible weights for a t-link because the cost given in (12) for a BRDS can be zero or one plus a constant value.

In the center vertices of the weighted graph, we can see the square region to be matched. In Figure 6a, we show the weighted graph for the gray scale stereo pair given in Figure 5. In this case, there are many possible weights or disparity options. The weights of the omitted t-links in Figure 6 are set as high as possible so that they can never be chosen in the multi-cut. In cases (a) and (b), the constant weight $\lambda$ is chosen in (14) which is equal to L for the n-links since the disparity is, at most, $L-1$, and the constant value C in (12) is set to be equal to the maximum value of $|I_l(x,y)-I_r{(x-d,y)|}^2$ for all possible values from $d=0,\cdots ,L-1$ for all pixels.

In step (iii), the stereo matching problem is formulated as a graph-cut problem for a single pair $(s,t)$. For this purpose, a QUBO function is constructed as given in (20). Table 1 presents the number of logical Boolean variables needed to construct the QUBO function for cases (a) and (b). The QUBO function $H_{\mathrm{problem}}^{\mathrm{qubo}}$ can be constructed for the given weighted graphs in Figure 6 as follows:

$$\begin{array}{ccc}\hfill H_{\mathrm{problem}}^{\mathrm{qubo}}& =& \sum _{\{u,v\}\in E_{tlink}}{y}_{uv}c(\{u,v\})+\sum _{\{u,v\}\in E_{nlink}}{y}_{uv}c(\{u,v\})+\alpha (1-x_s-x_t+2x_s{x}_t)+\hfill \\ & & \alpha \sum _{\{u,v\}\in E_{tlink}}(x_u+x_v+2w_{uv}+x_u{y}_{uv}-2x_u{w}_{uv}-2x_v{w}_{uv}-2y_{uv}{w}_{uv})+\hfill \\ & & \alpha \sum _{\{u,v\}\in E_{nlink}}(x_u+x_v+2w_{uv}+x_u{y}_{uv}-2x_u{w}_{uv}-2x_v{w}_{uv}-2y_{uv}{w}_{uv})\hfill \end{array}$$

(21)

where the set of edges $E=E_{tlink}\cup E_{nlink}$ such that $E_{tlink}$ is the set of t-links and $E_{nlink}$ is the set of n-links. For the weighted graphs in Figure 6a,b, the number of n-links and t-links are $|E_{nlink}|=1080$, $|E_{nlink}|=1665$, and $|E_{nlink}|=1200$, $|E_{nlink}|=1925$, respectively.

The coefficient $\alpha$ in (19) that is used to penalize sets of edges that do not correspond to a multi-cut is

$$\alpha =\sum _{\{u,v\}\in E}c(\{u,v\})-\sum _{u\in V}\left[c(\{s,u\})+c(\{t,u\})\right]$$

(22)

where the second term corresponds to the sum of the cost of all t-links that are adjacent to the source and sink vertices s and t. We omit the contribution of these t-links since they can never be selected for a multi-cut due to their high assigned cost.

The embedding procedure in step (iv) of the methodology consists of matching the connectivity of a given general graph into the D-Wave graph topology (see Figure 7). Translating a problem expressed as a QUBO function to an identical problem as a subgraph of the graph topology involves a process called minor embedding [56,57]. Commonly, the embedding process requires a number of physical qubits that is greater than the number of logical variables used in a given QUBO expression. Also, finding a minor embedding with the smallest number of physical qubits is an NP-hard problem [57].

The minor embedding problem can be stated as follows. Given the graph topology $\mathcal{G}$ of the quantum hardware and a QUBO problem $H_{\mathrm{problem}}^{\mathrm{qubo}}$, represented as a logical graph $G=(V,E)$, where V corresponds to the set of Boolean variables in $H_{\mathrm{problem}}^{\mathrm{qubo}}$ and E corresponds to the set of edges consisting of a pair of Boolean variables for each quadratic term in $H_{\mathrm{problem}}^{\mathrm{qubo}}$, find a subgraph in $\mathcal{G}$ such that

• Each vertex $j\in V$ is mapped to a connected subtree $T_j$ in $\mathcal{G}$.
• Each edge $\{i,j\}\in E$ must be mapped to at least one edge in $\mathcal{G}$.

Finding a minor embedding with the smallest number of physical qubits is an $\mathsf{NP}-\mathsf{hard}$ problem [57]. Figure 7a shows a logical graph with 26 vertices corresponding a QUBO expression, and Figure 7b shows an embedding into the hardware topology of the given logical graph.

We have simulated our algorithm using the software toolbox qbsolv that solves large QUBO problems by partitioning into subproblems targeted for execution on a D-Wave system [58]. Indeed, in addition to qbsolv, there are several QUBO solvers available, among them is [59,60] as well as those methods and software packages mentioned in [61]. We selected qbsolv because we are interested in testing our algorithm on D-Wave’s technology. Source code and full documentation of qbsolv can be found in  [62,63].

Our experiments were run on qbsolv tool using a desktop computer MacBook Air with a 1.3 GHz Intel Core i5 processor and 4 GB of RAM. The motivation of using qbsolv is that if the D-Wave quantum hardware is available, then the problem is partitioned into smaller subproblems that can be minimized by executing a quantum search on the quantum hardware. On the other hand, if no hardware is available, then the qbsolv tool executes the classical Tabu search algorithm to solve each subproblem.

We compare the solutions obtained using qbsolv with a traditional block matching technique. In Figure 4c, we present the truth disparity map which consists of a square region of $7\times 7$ pixels with a depth of 3 pixels. For the BRDS stereo pair, the result using a block matching algorithm can be seen in Figure 4d, and the result obtained using qbsolv can be seen in Figure 4e. In the first case, the square region is partially reconstructed with high disparity values, and in the second case it is almost completely reconstructed with low disparity values. In the latter case, it takes 91.67 s of classic cpu time and 1288 calls using qbsolv. For the gray-scale stereo pair images in Figure 5, we compare the result of a block matching algorithm with sub-pixel accuracy shown in Figure 5c with the result using qbsolv shown in Figure 5d. Figure 5e shows the absolute difference error between the disparity maps in Figure 5c,d, where the mean absolute error is 0.7359 pixels. In this case, it takes 148.31 s of classic cpu time and 1426 calls using qbsolv.

The examples presented in this section were selected with the purpose of showing how our proposal could be implemented. Due to hardware constraints, images used in this section are composed of a limited number of bits. We leave for future work a scalability analysis of our algorithm as well as a comparison study of our algorithm with its classical counterparts.

4. Conclusions

We have presented a method for rewriting the stereo matching problem as QUBO expressions based on the graph-cut and minimum multi-cut problems and the Ishikawa reduction method, followed by using QUBO expressions to simulate quantum annealing algorithms on D-Wave’s simulation software. This method has been validated using BRDS and gray scale stereo pairs of images. Future research will be focused on reducing the number of vertices in the graph-cut formulation of the stereo matching problem. This can be achieved by doing an initial disparity search to bound the limits of the disparity range for every pixel of the reference image as well as by incorporating an occlusion term in the cost model given in (11) to penalize occluded regions. An important contribution of this research is the quantum formulation of the minimum multi-cut problem for the case of a single pair of vertices. This formulation consists of the construction of a QUBO expression that is equivalent to the graph-cut problem; hence, no reduction algorithms are needed.

Although the proposed problem can be efficiently solved in a classical way, our methodology provides us with a new way of dealing with problems using quantum-annealing based algorithms and quantum technologies, such as the D-Wave annealer processor, to solve tasks within the realm of machine learning and related areas.