Next Article in Journal
Geometric and Optic Characterization of a Hemispherical Dome Port for Underwater Photogrammetry
Previous Article in Journal
Wireless Sensors Grouping Proofs for Medical Care and Ambient Assisted-Living Deployment
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

PIMR: Parallel and Integrated Matching for Raw Data

1
Key Laboratory for Optoelectronic Technology and Systems of Ministry of Education, College of Optoelectronic Engineering, Chongqing University, Chongqing 400044, China
2
Chongqing Academy of Science and Technology, Chongqing 401123, China
3
Beijing Institute of Environmental Features, Beijing 100854, China
*
Author to whom correspondence should be addressed.
Sensors 2016, 16(1), 54; https://doi.org/10.3390/s16010054
Submission received: 14 November 2015 / Revised: 29 December 2015 / Accepted: 30 December 2015 / Published: 2 January 2016
(This article belongs to the Section Physical Sensors)

Abstract

:
With the trend of high-resolution imaging, computational costs of image matching have substantially increased. In order to find the compromise between accuracy and computation in real-time applications, we bring forward a fast and robust matching algorithm, named parallel and integrated matching for raw data (PIMR). This algorithm not only effectively utilizes the color information of raw data, but also designs a parallel and integrated framework to shorten the time-cost in the demosaicing stage. Experiments show that compared to existing state-of-the-art methods, the proposed algorithm yields a comparable recognition rate, while the total time-cost of imaging and matching is significantly reduced.

1. Introduction

Image matching is a crucial technique with many practical applications in computer vision, including panorama stitching [1], remote sensing [2], intelligent video surveillance [3] and pathological disease detection [4].
Hu’s moment invariants as a shape feature has been widely used for image description due to its scaling and rotation invariance [5]. Wang et al. further proposed a two-step approach used for pathological brain detection by employing this feature combined with wavelet entropy [6]. Lowe’s scale-invariant feature transform (SIFT) is a de facto standard for matching, on account of its excellent performance, which is invariant to a variety of common image transformations [7]. Bay’s speeded up robust features (SURF) is another outstanding method performing approximately as well as SIFT with lower computational cost [8]. We proposed a lightweight approach with the name of region-restricted rapid keypoint registration (R3KR), which makes use of a 12-dimensional orientation descriptor and a two-stage strategy to further reduce the computational cost [9]. However, it is still computationally expensive for real-time applications.
Recently, many efforts have been made to enhance the efficiency of matching by employing binary descriptors instead of floating-point ones. Binary robust independent elementary features (BRIEF) is a representative example which directly computes the descriptor bit-stream quite fast, based on simple intensity difference tests in a smoothed patch [10]. When combined with a fast keypoint detector, such as features from accelerated segment test (FAST) [11] or center surround extrema (CenSurE) [12], the method provides a better alternative for real-time applications. Despite the efficiency and robustness to image blur and illumination change, the approach is very sensitive to rotation and scale changes. Rublee et al. further proposed oriented FAST and rotated BRIEF (ORB) on the basis of BRIEF [13]. The approach acquires multi-scale FAST keypoints using a pyramid scheme, and computes the orientation of the keypoints utilizing intensity centroid [14]; thus, the descriptor is rotation- and scale-invariant. In addition, it also uses a learning method to obtain binary tests with lower correlation, so that the descriptor becomes more discriminative accordingly. Some researchers also try to increase the robustness of matching by improving the sampling pattern for descriptors. The binary robust invariant scalable keypoints (BRISK) method proposed by Leutenegger adopts a circular pattern with 60 sampling points, of which the long-distance pairs are used for computing the orientation and the short-distance ones for building descriptors [15]. Alahi’s fast retina keypoint (FREAK) is another typical one leveraging a novel retina sampling pattern inspired by the human visual system [16]. Leutenegger uses a scale-space FAST-based detector in BRISK to cope with the scale invariance, which is employed by FREAK as well.
However, several inherent problems remain in existing methods. The digital image which consists of 24 bit blue/green/red (BGR) data is color-interpolated, which is known as demosaicing from raw data, including adjustment for saturation, sharpness and contrast, and sometimes compression for transmission [17,18]. This operation leads to irreversible information loss and quality degeneration. Furthermore, up until now, conventional image matching has been implemented after demosaicing, and such sequential operation restricts its application to general tasks. To address these limitations, in this paper we introduce an ultra-fast and robust algorithm, coined parallel and integrated matching for raw data (PIMR). The approach takes raw data instead of a digital image as the object for analysis, which is efficient for preventing information from being tampered with artificially. It is crucial to obtain high-quality features with the result that the approach can achieve comparable precision. Meanwhile, a parallel and integrated framework is employed to accelerate the entire image matching, in which two cores are used to respectively process the matching and demosaicing stages in parallel. Our experiments demonstrate that the proposed method can acquire more robust matches in most cases, even though it is much less time-consuming than traditional sequential image matching algorithms, such as BRIEF, ORB, BRISK and FREAK.
The rest of the paper is organized as follows. Section 2 gives the implementation details of the proposed method, which mainly includes the raw data reconstruction, and the parallel and integrated framework. In Section 3, we evaluate the performance of PIMR. Lastly, in Section 4, conclusions are presented.

2. Parallel and Integrated Matching for Raw Data

In Section 2.1, we give a brief account of raw data to make the reader understand our work more clearly. The key steps in PIMR are explained in Section 2.2 and Section 2.3, namely the reconstruction step for raw data and the details of the parallel and integrated framework.

2.1. Raw Data

Raw data, which is the unprocessed digital output of an image sensor, represents an amount of electrical charges accumulated in each photographic unit of the sensor. The notable features of raw data are nondestructive white balance, lossless compression, and high bit depth (e.g., 16 bits), providing considerably wider dynamic range than the JPEG file [19,20,21]. Hence, raw data maximally retains the real information of the scene compared to the digital image. By directly analyzing raw data, not only does it prevent the original information from being tampered with artificially, which contributes to an increase in precision, but it also shortens the time-cost in the demosaicing part.
The most popular format of raw data is Bayer Tile Pattern [22,23], typically in GBRG mode. It is widely used in industrial digital cameras. As illustrated in Figure 1, it is basically a series of band pass filters that only allow certain colors of light through, which are red (R), green (G) and blue (B). Green detail, to which the human visual system is more responsive than blue and red, is sampled more frequently, and the number of G is twice that of B and R.
Figure 1. Bayer pattern of GBRG mode.
Figure 1. Bayer pattern of GBRG mode.
Sensors 16 00054 g001

2.2. Reconstruction for Raw Data

At each pixel location, the sensor measures either the red, green, or blue value, and the values of other two are highly related to the neighbors around the pixel. On this basis, in order to make raw data more appropriate for further processing, we reconstruct it as follows. Initially, we define the 2 × 2 set of pixels as a cell. It starts with the target pixel C(i, j), and the other three pixels belonging to the cell are C(i + 1, j), C(i, j + 1) and C(i + 1, j + 1), respectively, where C represents an arbitrary color in G, B and R. Next, we utilize the color information in each cell to reconstruct the intensity I for the target pixel:
I MAX = max ( G + G , B + R )
I MIN = min ( G + G , B + R )
I = ( w 1 I MAX + w 2 I MIN ) / 2
where G, G’, B and R are the luminance values of each pixel in the defined cell, IMAX and IMIN are the maximum and minimum of the diagonal values in the cell, respectively, and w1 and w2 are the weight coefficients, which represent the contribution of diagonal color components to the cell. The greater the contribution, the larger the weight that will be assigned to the components. According to the results of multiple tests, w1 = 0.6, w2 = 0.4 in our method. All the cells in raw data are processed in this way.
In this procedure, the diagonal elements in each cell, one of which is two values of G and the others are B and R values, are combined to generate the target intensity which contains the essential information of the cell. Thus, the reconstruction operation making full use of abundant color information in raw data is conducive to enhancing the matching accuracy. Moreover, it reduces time complexity effectively as well, without the procedure of demosaicing for acquiring a full color image with traditional methods.

2.3. Parallel and Integrated Framework

With the focus on efficiency of computation, in our methodology, two cores in a multi-core processor are used to parallel the procedure of matching and imaging. As shown in Figure 2, thread 1 is responsible for handling the raw data matching, and thread 2 performs demosaicing to get a full color image with high quality. Eventually, the final results are obtained by merging the outcomes from the two different threads.
When handling the raw data matching, first of all, we preprocess it using the raw data reconstruction step whose detailed implementation is in Section 2.2. Since the intensity information of the pixels after reconstruction has a similar data form to gray images, we adopt the multi-scale FAST detector and the rotated BRIEF descriptor in ORB to complete feature detection and description for raw data, which achieves high robustness to general image transformations, including image blur, rotation, scale and illuminance change, and is of fast speed as well [13]. For the part of feature matching, we first find the k-nearest neighbors in the sensed image to the keypoints in the reference image via brute-force matching, and then employ the ratio test explained by Lowe [7] to select the best one from the k matches. The three parts mentioned above, i.e., the feature detection, description and matching, are collectively referred to as the overall matching stage in this paper. Moreover, note that the similarity between descriptors is measured by Hamming distance [10], which can be computed rapidly via a bitwise exclusive or (XOR) operation followed by a bit count. However, as the development of the multimedia instruction set of the central processing unit (CPU) is fast, it is more effective to calculate the number of bits set to 1 using the newer instructions on modern CPUs compared with the previous bit count operation. Hence, the population count (POPCNT) instruction, which is part of SSE4.2, is applied to our method for speeding up feature matching [24].
Figure 2. Flow chart of PIMR.
Figure 2. Flow chart of PIMR.
Sensors 16 00054 g002
We use Malvar’s algorithm for raw data demosacing [25]. The method computes an interpolation using a bilinear technique, computes a gradient correction term, and linearly combines the interpolation and the correction term to produce a corrected, high-quality interpolation of a missing color value at a pixel. A gradient-correction gain is also used to control how much correction is applied, and the gain parameters are computed by a Wiener approach and they contribute to the coefficients of the linear filter in the method. The approach outperforms most nonlinear and linear demosaicing algorithms with a reduced computational complexity. Due to the high image quality and low computational cost, the method is considered the major demosaicing approach in industrial digital cameras [26].

3. Performance Evaluation

3.1. Experimental Details

We evaluate PIMR using a well-known dataset, i.e., the Affine Covariant Features dataset introduced by Mikolajczyk and Schimid [27].
It should be noted that our approach directly processes raw data captured by the digital camera without demosaicing. For this reason, we need to extract the corresponding color information at each pixel location from the original images in the chosen dataset, in accordance with the GBRG mode of the Bayer pattern. The new dataset, being made up of a series of raw data, mainly contains the following six sequences: wall (view point change), bikes and trees (image blur), leuven (illumination change), University of British Columbia (UBC) (JPEG compression), and graffiti (rotation). Each sequence consists of six sets of raw data just as six images, sorted in order of a gradually increasing degree of distortions with respect to the first image with the exception of the graffiti sequence. We take the raw data of image 1 in each sequence as the reference data, and match the reference one against the remaining ones, yielding five matching pairs per sequence (1|2, 1|3, 1|4, 1|5 and 1|6). Figure 3 shows the first image in the original six sequences from the standard dataset.
Figure 3. Image sequences from the Affine Covariant Features dataset. (a) wall; (b) bikes; (c) trees; (d) leuven; (e) UBC; (f) graffiti.
Figure 3. Image sequences from the Affine Covariant Features dataset. (a) wall; (b) bikes; (c) trees; (d) leuven; (e) UBC; (f) graffiti.
Sensors 16 00054 g003
Since our work aims at realizing robust and fast feature detection, description and matching for raw data, we assess not only the accuracy but also the time-cost, comparing our PIMR with the state-of-the-art algorithms, including BRIEF, ORB, BRISK, and FREAK.
We compute 1000 keypoints on five scales per raw data with a scaling factor of 1.3 using PIMR. In order to ensure a valid and fair assessment, the full color image processed by the BRIEF, ORB, BRISK and FREAK algorithms is demosaiced from raw data using the same interpolation approach as PIMR. It must be emphasized that we combine the BRIEF descriptor with the CenSurE detector, and the multi-scale adaptive and generic corner detection based on the accelerated segment test (AGAST) detector proposed in BRISK with the FREAK descriptor, based on the settings in reference [10,16]. The implementation of these methods is built with OpenCV 2.4.8 which provides a common two-dimensional (2D) feature interface.

3.2. Accuracy

We use the recognition rate, namely the number of correct matches versus the number of total good matches, as the evaluation criterion. The results are shown in Figure 4. Each group of five bars with different colors represents the recognition rates of PIMR, ORB, BRIEF, BRISK and FREAK, respectively.
On the basis of these plots, we make the following observations. In general, our algorithm performs well for all test sequences, yielding comparable recognition accuracy with the state-of-the-art algorithms, and even outperforms them in most cases. There are only a few exceptions. For example, in the wall sequence, BRIEF achieves slightly higher precision than our methods, but in the other five sequences, PIMR outperforms the other algorithms.
Figure 4. Recognition rates achieved by PIMR, ORB, BRIEF, BRISK and FREAK.
Figure 4. Recognition rates achieved by PIMR, ORB, BRIEF, BRISK and FREAK.
Sensors 16 00054 g004

3.3. Time-Cost

Table 1 shows the time-cost of the bikes 1|3 employing different algorithms, measured on an Intel Core i7 processor at 3.4 GHz, in seconds. Since there is no raw data construction step in traditional methods, and the demosaicing is processed in parallel with the other stages within the proposed approach, the procedures marked with dashes in the table involve no extra time. Note that the time is averaged over 50 runs.
Considering the total time listed in the last column of Table 1, PIMR is about 1.4× faster than ORB and 7.1× faster than FREAK. In PIMR, the time-cost of demosaicing is excluded from the total time, as its demosaicing stage is faster than the sum of the other two stages. One potential reason for the high computational cost in BRISK and FREAK is the application of the scale-space FAST-based detector [15] which promises coping with the scale invariance.
Table 1. Times of matching for the bikes image 1 and 3.
Table 1. Times of matching for the bikes image 1 and 3.
MethodsDemosaicing (s)Raw Data Reconstruction (s)Overall Matching (s)Total (s)
ORB0.009-0.0400.049
BRIEF0.009-0.0490.058
BRISK0.009-0.2250.234
FREAK0.009-0.2310.240
PIMR-0.0040.0300.034

3.4. Matching Samples with PIMR

Extensive evaluations have been made for PIMR above, and we also provide some matching samples with the chosen dataset in this part. It is considered that the invariance of rotation, scale and illuminance is the overriding concern in image matching algorithms. Thereby, we present the matching results of the graffiti (rotation and scale change) and leuven (illumination change) sequences to further demonstrate the robustness of the proposed approach. Figure 5a shows the matching result of the graffiti sequence (1|2), and Figure 5b shows the matching result of the leuven sequence (1|2). The keypoints connected by green lines indicate keypoint correspondences, namely valid matches. It can be seen that the method PIMR can acquire sufficient robust matches with few outliers.
Figure 5. Matching samples using the PIMR. (a) The matching result of the graffiti sequence; (b) The matching result of the leuven sequence.
Figure 5. Matching samples using the PIMR. (a) The matching result of the graffiti sequence; (b) The matching result of the leuven sequence.
Sensors 16 00054 g005

4. Conclusions

We have presented a parallel and integrated matching algorithm for raw data named PIMR, mainly to speed up whole image matching and to enhance the robustness as well. In most cases, it achieves better performance for most image transformations than current approaches, with a fairly low computational cost. Furthermore, our work is not only basic research in the field of rapid image matching, but also a preliminary study with respect to a method for processing raw data. In the future, an ultra-light-weight local binary descriptor will be studied further.

Acknowledgments

This research was supported by the National Natural Science Foundation of China (61105093), the Fundamental Research Funds for the Central Universities (106112013CDJZR120013), the Chongqing Postdoctoral Science Foundation (Xm2015014), the Zhejiang Provincial Natural Science Foundation of China (LY15F020042), the Opening Fund of Key Laboratory of Inland Waterway Regulation Engineering (Chongqing Jiaotong University), Ministry of Communications (NHHD-201503), and the Visiting Scholar Foundation of Key Laboratory of Optoelectronic Technology & Systems (Chongqing University), Ministry of Education.

Author Contributions

The work presented in this paper is a collaborative development by five authors. Li defined the research theme, designed the methods and experiments. Li and Yang performed the experiments. Yang developed the data collection modules and guided the data analysis. Zhao, Han and Chai performed the data collection and data analysis. Yang wrote this paper, Li, Zhao, Han and Chai reviewed and edited the manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Zaragoza, J.; Tat-Jun, C.; Tran, Q.H.; Brown, M.S.; Suter, D. As-projective-as-possible image stitching with moving dlt. IEEE Trans. Pattern Anal. Mach. Intell. 2014, 36, 1285–1298. [Google Scholar] [PubMed]
  2. Mei, X.; Ma, Y.; Li, C.; Fan, F.; Huang, J.; Ma, J. A real-time infrared ultra-spectral signature classification method via spatial pyramid matching. Sensors 2015, 15, 15868–15887. [Google Scholar] [CrossRef] [PubMed]
  3. Xu, X.; Tang, J.; Zhang, X.; Liu, X.; Zhang, H.; Qiu, Y. Exploring techniques for vision based human activity recognition: Methods, systems, and evaluation. Sensors 2013, 13, 1635–1650. [Google Scholar] [CrossRef] [PubMed]
  4. Zhang, Y.D.; Dong, Z.C.; Wang, S.H.; Ji, G.L.; Yang, J.Q. Preclinical diagnosis of magnetic resonance (MR) brain images via discrete wavelet packet transform with tsallis entropy and generalized eigenvalue proximal support vector machine (GEPSVM). Entropy 2015, 17, 1795–1813. [Google Scholar] [CrossRef]
  5. Hu, M. Visual-pattern recognition by moment invariants. Ire Trans. Inf. Theory 1962, 8, 179–187. [Google Scholar]
  6. Zhang, Y.D.; Wang, S.H.; Sun, P.; Phillips, P. Pathological brain detection based on wavelet entropy and hu moment invariants. Bio-Med. Mater. Eng. 2015, 26, S1283–S1290. [Google Scholar] [CrossRef] [PubMed]
  7. Lowe, D. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 2004, 60, 91–110. [Google Scholar] [CrossRef]
  8. Bay, H.; Ess, A.; Tuytelaars, T.; van Gool, L. Speeded-up robust features (SURF). Comput. Vis. Image Underst. 2008, 110, 346–359. [Google Scholar] [CrossRef]
  9. Li, Z.; Gong, W.; Nee, A.Y.C.; Ong, S.K. Region-restricted rapid keypoint registration. Opt. Express 2009, 17, 22096–22101. [Google Scholar] [CrossRef] [PubMed]
  10. Calonder, M.; Lepetit, V.; Ozuysal, M.; Trzcinski, T.; Strecha, C.; Fua, P. Brief: Computing a local binary descriptor very fast. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 34, 1281–1298. [Google Scholar] [CrossRef] [PubMed]
  11. Rosten, E.; Drummond, T. Machine learning for high-speed corner detection. In Computer Vision—Eccv 2006; Leonardis, A., Bischof, H., Pinz, A., Eds.; Springer: Berlin/Heidelberg, Germany, 2006; Volume 3951, pp. 430–443. [Google Scholar]
  12. Agrawal, M.; Konolige, K.; Blas, M. Censure: Center surround extremas for realtime feature detection and matching. In Computer Vision—Eccv 2008; Forsyth, D., Torr, P., Zisserman, A., Eds.; Springer: Berlin/Heidelberg, Germany, 2008; Volume 5305, pp. 102–115. [Google Scholar]
  13. Rublee, E.; Rabaud, V.; Konolige, K.; Bradski, G. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain, 6–13 November 2011; pp. 2564–2571.
  14. Rosin, P.L. Measuring corner properties. Comput. Vis. Image Underst. 1999, 73, 291–307. [Google Scholar] [CrossRef]
  15. Leutenegger, S.; Chli, M.; Siegwart, R.Y. Brisk: Binary robust invariant scalable keypoints. In Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain, 6–13 November 2011; pp. 2548–2555.
  16. Alahi, A.; Ortiz, R.; Vandergheynst, P. Freak: Fast retina keypoint. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence Rhode Island Convention Center Providence, RI, USA, 16–21 June 2012; pp. 510–517.
  17. Schafer, R.W.; Mersereau, R.M. Demosaicking: Color filter array interpolation. IEEE Signal Process. Mag. 2005, 22, 44–54. [Google Scholar]
  18. Li, X.; Gunturk, B.; Zhang, L. Image demosaicing: A systematic survey. Proc. SPIE 2008, 6822. [Google Scholar] [CrossRef]
  19. Lu, Y.; Jian, S. High quality image reconstruction from raw and jpeg image pair. In Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain, 6–13 November 2011; pp. 2158–2165.
  20. Macfarlane, C.; Ryu, Y.; Ogden, G.N.; Sonnentag, O. Digital canopy photography: Exposed and in the raw. Agric. For. Meteorol. 2014, 197, 244–253. [Google Scholar] [CrossRef]
  21. Verhoeven, G.J.J. It’s all about the format—Unleashing the power of raw aerial photography. Int. J. Remote Sens. 2010, 31, 2009–2042. [Google Scholar] [CrossRef]
  22. Bayer, B.E. Color Imaging Array. U.S. Patent US3971065, 20 July 1976. [Google Scholar]
  23. Maschal, R.A.; Young, S.S.; Reynolds, J.P.; Krapels, K.; Fanning, J.; Corbin, T. New image quality assessment algorithms for cfa demosaicing. IEEE Sens. J. 2013, 13, 371–378. [Google Scholar] [CrossRef]
  24. Intel’s Software Network, Sofwareprojects. Intel. com/avx; programming reference; SSE Intel.: Santa Clara, CA, USA, 2007; Volume 2.
  25. Malvar, H.S.; He, L.-W.; Cutler, R. High-quality linear interpolation for demosaicing of bayer-patterned color images. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’04), Montreal, QC, Canada, 17–21 May 2004; pp. 485–488.
  26. Point Grey Research. Different Color Processing Algorithms. Available online: http://www.ptgrey.com/KB/10141 (accessed on 14 December 2015).
  27. Mikolajczyk, K.; Schmid, C. A performance evaluation of local descriptors. IEEE Trans. Pattern Anal. Mach. Intell. 2005, 27, 1615–1630. [Google Scholar] [CrossRef] [PubMed]

Share and Cite

MDPI and ACS Style

Li, Z.; Yang, J.; Zhao, J.; Han, P.; Chai, Z. PIMR: Parallel and Integrated Matching for Raw Data. Sensors 2016, 16, 54. https://doi.org/10.3390/s16010054

AMA Style

Li Z, Yang J, Zhao J, Han P, Chai Z. PIMR: Parallel and Integrated Matching for Raw Data. Sensors. 2016; 16(1):54. https://doi.org/10.3390/s16010054

Chicago/Turabian Style

Li, Zhenghao, Junying Yang, Jiaduo Zhao, Peng Han, and Zhi Chai. 2016. "PIMR: Parallel and Integrated Matching for Raw Data" Sensors 16, no. 1: 54. https://doi.org/10.3390/s16010054

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop