We compare here the plain correlation and multiresolution matching approaches. Both algorithms have as common attribute the window size. Although some authors recommend the use of a 7 × 7 window for plain correlation (see, for instance, the work of Hirshmuller [22]), we opted for testing several window sizes in order to compare the relative performances of both approaches.

Traditional plain correlation calculates the normalized, linear cross correlation between all possible windows of both images. For each point in one image, the matching point is chosen in the other image such as to maximize the correlation coefficient.

When matching square images of side w, this algorithm calculates w³ correlations, but when a search interval w_s < w is available, the number of correlations drops down to w_sw². Of course, in the worst case, we should assume that the plain correlation approach would have O(w³) complexity.

Multi-resolution stereo matching uses several pairs of images of the same scene, sampled with different levels of detail, as a double pyramidal representation of the scene [17]. As in any scale space, images at the base of the pyramid have higher resolution and, therefore, more detail of the scene than those at the top. The credit for using this idea in visual tasks can be given to Uhr [27]. The scale space theory is formalized by Witkin [24], and further by Lindeberg [25]. A variation, the Laplacian pyramid, was introduced by Burt and Adelson [26]. Tsotsos [56, 57] integrated multi-resolution into visual attention, implemented as such by Burt [58], and used in several visual models [28, 29, 59, 60]. Based on multi-resolution, Lindeberg [61] detected features using an automatic scale selection algorithm, while Lowe [62] dealt with detection of scale-invariant features.

Multiresolution algorithms in stereo matching calculate the disparity of all pixels (or blocks of pixels) of a coarse level image and refine them, matching the pixels of finer level images with a small number of pixels around the coarser match. We refer to the interval that contains those pixels as the “refining interval”.

For example, a multiresolution algorithm with fixed depth that matches the points of two 256 × 256 pixels images, say x₀ and y₀, may use three pairs of images having, thus, level 3 of sizes 128 × 128, 64 × 64 and 32 × 32; we denote these pairs of images (x_ℓ, y_ℓ), 1 ≤ ℓ ≤ 3 respectively. Note that usually x_ℓ(i, j) = (x_ℓ−1(2i, 2j) + x_ℓ−1(2i + 1, 2j) + x_ℓ−1(2i, 2j + 1) + x_ℓ−1(2i + 1, 2j + 1))/4, for every 1 ≤ ℓ ≤ 3, but other operators are also possible as will be seen in Section 4. In this case the window size is w = 2. The same transformation is recursively applied to y₀ in order to obtain y₁, y₂ and y₃. We omit the dependence of the coordinates (i, j) on the level ℓ for the sake of simplicity.

The classical approach would attempt to match all the 32 × 32 pixels of the pair (x₃, y₃) to, then, proceed to their refinement. The refinement of pixel x₃(i, j) consists of correlating the values x₂(2i, 2j), x₂(2i + 1, 2j), x₂(2i, 2j + 1) and x₂(2i + 1, 2j + 1) with the pixels within the refining interval around the matching point of y₂. This is repeated until the matching is done on the (x₀, y₀) pair, obtaining the final result.

This approach is known to be faster than the brute force search on (x₀, y₀) (plain correlation). In fact, on the extreme case, where the images are squares and the smallest ones are single pixels, it requires w² log(w) correlations, were w is the window size, thus its complexity is O(w² log(w)). Of course, there is the time used for building the pyramid. So, to determine final algorithm complexity, one must add the complexity for building the pyramid, which is O(w²) + O(w²/4) + ⋯ + O(w²/w²), with the complexity of the matching, given above, which results anyway in O(w² log(w)).

Reducing the search interval is not very efficient at improving this algorithm, since the gain in operations comes at the expense of more errors. Often, important characteristics are lost in the smaller images, reducing correlation precision. Those errors can sometimes be alleviated by a larger refining interval, which increases the execution time.

In practice, some implementations relate that the processing time used for building the multiresolution pyramid often compensates for the time gained on optimizing the correlations [22]. This basic multiresolution matching is seldom used in current applications [21].

We used a discrete wavelet transform to build the pyramids. With this approach, in a given level i, the scale image of the pyramid (I_i) is obtained by applying a low pass filter (L) to the scale image of level i − 1 followed by a decimation (↓). Detail images D_i (with vertical, horizontal and diagonal details) are calculated using high-pass filters applied to the scale image of level i − 1 followed by a decimation. Figure 1 shows the schema for calculating a wavelet pyramid of level 2. We used the Daubechies and Haar bases [63].

We build two multiresolution pyramids by successively convolving the previous images with the low-pass Gaussian (ϒ_G) and high-pass Laplacian masks (ϒ_L) defined in Equations (2), and then decimating: (2) ϒ G = 1 16 [ 1 2 1 2 4 2 1 2 1 ] ,         ϒ L = [ − 1 1 − 1 1 4 1 − 1 1 − 1 ] .

With this, we generate a pyramid of images and another of details. Figure 2 illustrates this filtering process used for the creation of a pyramid with three levels. By convolving the original image I₀ with the high-pass filter (H mask), image D₀ is generated. I₀ is then convolved with the low-pass filter defined by the mask L, and decimated by a factor 2, which generates I₁. This last image is then convolved again with the high-pass filter defined by the mask H, generating D₁. A second low-pass filter (L) followed by a decimation, applied to I₁, generates image I₂, which is finally filtered by H generating D₂.

These two pyramids are able to retain enough information in order to allow an efficient search for matching points.

The use of a sub-band filtering makes this algorithm much faster than the one proposed by Hoff and Ahuja [37], by removing the bottle-neck which is filtering. This fact, plus a lower error rate, allows to use a smaller refinement interval, which makes the multiresolution matching with variable depth much faster than the one with fixed depth and than the simple correlation approach in the original images.

Due to decimation, the construction of the scale images of the pyramid cannot be made shift-invariant. However, the detail images can be shift-invariant and this is a key difference between the two techniques. In the case of wavelets, the detail images are sensitive to shifts, but with 2D filtering they are invariant.

The wavelet transform is invertible. 2D filtering based transform is invertible only if both the high-pass and low-pass filters are ideal filters [64], which amounts to using convolution masks of the size of the original image. In order to be economic, small masks are employed and, therefore, this transformation is not invertible.

We use a propositional logic based on fuzzy evidence to derive a heuristic for calculating the desired level from which the matching will be performed. Such level is the coarsest one that can be labeled as “reliable”, in the sense that it provides enough information for the matching.

Fuzzy logic is composed of propositions P with continuous rather than binary truth values μ(P) ∈ [0, 1]. We used the following operators on those propositions: “¬”, where μ(¬P) = 1 − μ(P), “∧”, where μ(A ∧ B) = min(μ(A), μ(B)), “∨”, where μ(A ∨ B) = max(μ(A), μ(B)), “⇒”, where (A ⇒ B) ⇔ (μ(B) ≥ μ(A)) and “⇏”, where (A ⇏ B) ⇔ (μ(A) > μ(B)).

We define a predicate σ_ℓ(i, j) meaning “the classification of the block at position (i, j) and level ℓ is not reliable”. This predicate must satisfy the following conditions:

If the detail at (i, j) is zero, the classification is reliable: D(i, j) ≠ 0 ⇒ σ_ℓ(i, j), where D is the amount of detail available.

Because short execution time is our main objective, the heuristic has to be easy to compute by general purpose computers, leading to Equation (3): (3) σ ℓ   ( i , j ) = ( ∨ ( i , j ) ∈ K σ ℓ − 1   ( i , j ) ) ∨ D ( i , j ) ≠ 0.

We define, for any a ∈ [−1, 1], μ(a ≠ 0) = |a|, completely specifying the heuristic. Defining a dependability threshold δ ∈ [0, 1], our desired level for each pixel is the maximum level ℓ for which δ ⇒ σ_ℓ.

The ideal values of δ depend on the amount of detail in the image and, in principle, different values of δ should be associated to each pixel. For example, an image with substantial detail (texture) would be better treated at highest resolution, i.e., it should have values of δ very close to zero. Flat images with little detail could be dealt with at very coarse resolution without loosing information, i.e., with δ close to 1. Figure 3 illustrates this with a 5 × 5 image, where each pixel has a different δ associated to it; notice that the smallest values are associated to the border, where there is detail that would be lost if treated at a coarse resolution.

However, the amount of texture is not known a priori. So, in this work, an empirically value is assigned for δ and kept constant for the whole image. In practice, we found that values greater than 0.2, cause the algorithm not to perform well, as it will be seen in the experiments.

The fuzzy heuristic presented above is able to assign a proper level to every pixel of an image, identifying detailed and flat areas. A successful technique for our purposes should be able to detect the level of detail of each image region based on texture. Flat regions should be treated at coarser, i.e., higher levels of the pyramid (at the pyramid top) since they carry less information than detailed regions, which should be treated at lower levels (at the pyramid basis).

As it will be shown, at the coarsest level, the variable depth multiresolution matching also makes less mistakes than the fixed depth approaches. Because of that, we were able to obtain good results even with a refining interval as small as four pixels wide, leading to very fast execution.

The implementation of our proposal requires complex memory management that allocates and frees amounts of memory equivalent of several pages of the most common processors. Most operating systems lose performance on such conditions. So, also as a contribution of this work, we implemented a secondary memory management strategy that uses a buffer allocated only once at the beginning of execution. This pre-allocated memory is then managed by our procedure avoiding several calls to the operating system to perform this task. This approach alleviates the execution time, rendering a still faster procedure.

This module contains the library ops.h, that implements operations which are required in almost every stage. It also has the classes Position, that stores a position of type “(row, column)”, Window, that defines a rectangular area of interest, and Interval, that defines a connected subset of integer values. Classes Window and Interval also store some pre-calculated values used to accelerate the matching.

This module contains the template Image, and classes that specialize Pixel: ColorPixel, BWPixel, PositionPixel, BWLabel and ColorLabel. Image<PixType> has an array of elements of type PixType that represents the pixels. This template implements operations for image reading and writing images in PGM and PPM formats, and also guarantees access to operations in pixels and the wavelet transform.

Pixel provides arithmetic operators used in transformations and convolutions, besides methods for extracting data. Types ColorPixel and BWPixel implement pixels for color and monochromatic images. Types ColorLabel and BWLabel implement color and monochromatic pixels also, but with an integer identification code (id). PositionPixel implements a gray level pixel with integer value; it stores the final disparity map values and an integer id.

The data structures that store pyramids of images, regardless the technique (wavelets or 2D filtering), are created by the classes ImgPair and LowHigh. The former returns the first pair of images in the pyramid, while the latter builds the remaining pairs. Classes ImgSet and ImgListSet implement the data structure that contains the four images generated by wavelets transform and the lists of the images generated in a sequence of transformations, respectively.

Class DWT has values and methods used by the Daubechies wavelet transform. An object of class DWT has filters of a transformation implemented in another class; this strategy is adopted to avoid the use of a virtual class. Classes Haar and Daub4 implement the two types of wavelets used in this work, namely Daubechies and Haar.

The result of the heuristic that calculates the desired depth for each pixel requires a complex data structure. We implemented linked lists that contain objects of class Position. These lists have different formats in each execution of the matching requiring, thus, dynamical allocation of memory. A problem is that a list may use a large region of memory that may, sometimes, grow up to several megabytes. This is beyond the size of the memory page of most modern computer architectures, which is usually 16 Kb. As current operational systems usually lose performance as they allocate and free, repeatedly, such amounts of memory, we developed a memory managing system for our library. To do that, we created the class MemoryBuffer containing a buffer, which is allocated at the initialization, and resources for managing it.

By using the class MemoryBuffer, tailored to the needs of our library, program execution is much faster than by using the memory management provided by the operating system. The directive FAST_MEMORY, available at compiling time, makes memory management still faster by disabling the checking of buffer limit. When used through this library, all data stored in these buffers are calculated locally and not brought from other programs. We remark that this strategy presents low risk for the system security.

The class List implements a low-level list that can deal with allocated memory, with or without the aid of an object of the type MemoryBuffer. The other classes of this module are LinkedList and Stack, that implement high-level data structures (linked list and stack, respectively), useful for other modules of the library.

Class Fuzzy represents the fuzzy hypotheses, with the following operators: ¬ (!), ⊕ (+), . (*), ∨ (|), ∧ (&), ⇒ (<), and ⇏ (>).

Class FuzzyImax also composes this module. It is responsible for calculating the desired depth for each pixel. The return value of this method is of type LinkedList<LinkedList<Position>>, where Position stores a position in image. The output is a list of depth levels. For each depth, there is a list of pixels where disparity calculations start from that depth.

Note that each image pixel can be represented in more than a depth. In such case, matching must be performed at the least resolution depth in which the pixel is found. For example, if the sixth element of the returned list has position (1, 1), this means that for all pixels in the original image that lie in positions (x, y), x, y < 2⁶, the greater level that can be used is 5 (starting from zero). It is possible for a pixel to appear twice in the list, for instance if position (2, 3) appears at the fourth list, for all pixels of the interval (x,y), 2 × 2⁴ ≤ x <3 ×2⁴, 3 × 2⁴ ≤ y <4 × 2⁴, that is, in the interval (x, y), x, y < 2⁶, the depth must be up to 3, and not 5 anymore.

The easiest way of obtaining depth for each level is, thus, by traveling this list starting from the less coarse level and marking positions already visited. For that, pixels of the type ColorLabel and BWLabel are used.

The main classes for our application are LeftImax and PlainCorr, both derived from Vision. These classes implement the multiresolution with variable depth matching and the simple correlation methods. Objects of both classes are created using as parameters the left and right images, and the resulting image were disparity will be stored. Images can be created through allocation of a memory area or using an already allocated area. Image data are stored linewise as one-dimensional arrays.

Objects of classes LeftImax and PlainCorr can then be initialized with setWindow. For simple correlation, arguments are setWindow (Window C, Interval B), where C is the comparison window and B is the search interval. In this implementation, arguments are setWindow (Window C, Interval B, Interval R), where C and B are the same and R is the refining interval.

Classes Window and Interval define windows and intervals, respectively, as integer numbers. Windows can be created at any position, using Window (int rmin, int rmax, int cmin, int cmax), where rmin and rmax are the extreme lines that the window contains, and cmin and cmax the extreme columns. Intervals can be created in arbitrary positions; Interval (int min, int max) creates the interval [ min; max].

After windows are initialized, the matching is performed using match of LeftImax or PlainCorr. For plain correlation, this method does not receive arguments, and in multiresolution matching with variable depth it receives match ( Fuzzy δ) as argument, where δ is as defined in Equation (3). After matching is performed, disparities can be read at the resulting image.

Memory allocation is always done in a transparent way to the programmer. All necessary memory is allocated at the creation of the objects of classes LeftImax and PlainCorr. Garbage collection, however, is not supported. This is not a problem in most applications, but might be an issue when dealing with images from several pairs of different cameras. The constructor of class Fuzzy receives only an argument of type double that represents, in this case, μ(δ).

We performed tests with two versions of our multiresolution matching. The first uses only scale images in all levels based on correlation measures. The second uses the detail images in each level and the scale images at the coarsest level, since at this level there is less detail.

Disparity maps generated by both versions of our multiresolution algorithm are shown in Figure 8. These results are obtained with a correlation window of size 3 and a threshold δ = 0.3. Note that borders and edges obtained by the algorithm that uses detail coefficients are sharper and better defined than the ones produced by the other technique, which only uses scale images. Besides that, the overall aspect of the former disparity map is better than the latter. Figure 9 shows average measures of the errors obtained with several thresholds for both versions, keeping the correlation window at size 3. The minimum in both lines near the origin indicates that the threshold δ = 0.3 produced less errors. The use of scale images at all levels produces results with less errors, what is represented by the bottom lines in both graphs.

With the new fuzzy heuristic, multi-resolution matching is likely to start at the lowest level where there is a border adjacent to the pixel under assessment. The correlation of the images at the coarsest depth is, thus, highly prone to errors due to occlusions. Matching the details, instead of the raw images, should, in principle, lead to higher resistance to occlusions. That behavior was confirmed in our experiments, as the results obtained matching the scale images at each level were consistently better than those that employed detail information.

Here we contrast plain correlation with multiresolution algorithm. Disparity maps obtained by both algorithms are shown in Figure 10.

We made experiments with both approaches for window sizes of 3, 5, 7, 9 and 11. Standard deviation and mean distance of the measured errors for multiresolution approach with variable depth are shown in Figure 11. The same error measures produced by the technique without search interval are shown in Figure 12 for the same window sizes.

We observe that larger windows generate smaller errors in both approaches. Multiresolution incurred in smaller errors than plain correlation in most cases, and it made mistakes as often as the plain correlation. Plain correlation produces errors distributed on bigger areas than our algorithm, which is hard to visualize in the disparity figures. By the results, on the overall, our approach performed better than plain correlation.

Figure 13 shows a comparison between the matching using the two algorithms (plain correlation and ours, with threshold δ = 0.1, 0.2) for the Tsukuba images, while Figure 14 shows the same comparison applied to the Corridor images.

Figure 15 shows results of varying δ, with a search interval of 6 pixels wide.

We tested both algorithms also in the Corridor image, and the results are shown in Figure 16. In this case, a search interval of 10 pixels was imposed, a refinement interval of 4 and 6 pixels and square search window sizes of 5, 7, and 11 pixels. We tried with several limits (δ). Figure 17 shows the time necessary for running this experiment. The best result of the matching is achieved for δ = 0.05 and the best times start at δ = 0.1. So, one has to weight between precision and time. The result of the matching is still better than plain correlation for δ = 0.05, whose error and standard deviation are shown in Figure 18.

The time needed for the matching processes is shown in Figure 19 as a function of the threshold (δ). Multiresolution matching was consistently faster than plain correlation. It should be remarked that the execution time of our algorithm is much shorter than the plain correlation, on all thresholds, and it is even faster at small thresholds. Note that smaller correlation windows need less time. One has to weight between precision and available time when deciding the size to be used. Plain correlation errors usually increase a little from δ = 0, but they fall at near the same or smaller values near δ = 0.3, which seems to be an optimum threshold.