# Compressive Sensing Hyperspectral Imaging by Spectral Multiplexing with Liquid Crystal

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Spectral Modulation for CS with LC

## 3. Reconstruction Process

#### Dictionary for Sparse Representation

**S**:

**f**to its sparse representation, $f={\mathsf{\Psi}}_{d}\mathsf{\alpha}$. Each column of ${\mathsf{\Psi}}_{d}$ is referred to as an atom of the dictionary. Therefore, the spectrum

**f**can be viewed as a linear combination of atoms in ${\mathsf{\Psi}}_{d}$ according to weights in $\mathsf{\alpha}$. Based on Equation (7), a corresponding system dictionary ${\mathsf{\Omega}}_{d}\in {\Re}^{M\times {N}_{d}}$ is created by the inner products of the spectral dictionary with the CS-MUSI sensing matrix, $\mathsf{\Phi}$:

## 4. Compressive Hyperspectral and Ultra-Spectral Imaging

#### 4.1. Camera Calibration

#### 4.2. Staring Mode

#### 4.3. Scanning Mode

## 5. 4D Imaging

## 6. Target Detection

**x**is the pixel signature,

**t**is the target spectral signature and

**m**is the estimated background. $\Gamma $ is the covariance matrix, which holds the statistics of the background and can be approximated using:

**x**’ is a pixel that contains the target and p is the ratio of the target present in the pixel.

## 7. Discussion

## 8. Patents

## Supplementary Materials

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Schott, J.R. Remote Sensing: The Image Chain Approach; Oxford University Press: Oxford, UK, 2007. [Google Scholar]
- Borengasser, M.; Hungate, W.S.; Watkins, R. Hyperspectral Remote Sensing: Principles and Applications; CRC Press: Boca Raton, FL, USA, 2007. [Google Scholar]
- Eismann, M.T. Hyperspectral Remote Sensing; SPIE PRESS: Bellingham, WA, USA, 2012. [Google Scholar]
- Bioucas-Dias, J.M.; Plaza, A.; Camps-Valls, G.; Scheunders, P.; Nasrabadi, N.; Chanussot, J. Hyperspectral remote sensing data analysis and future challenges. IEEE Geosci. Remote Sens. Mag.
**2013**, 1, 6–36. [Google Scholar] [CrossRef] - Akbari, H.; Halig, L.; Schuster, D.M.; Fei, B.; Osunkoya, A.; Master, V.; Nieh, P.; Chen, G. Hyperspectral imaging and quantitative analysis for prostate cancer detection. J. Biomed. Opt.
**2012**, 17, 076005. [Google Scholar] [CrossRef] [PubMed] - Lu, G.; Fei, B. Medical hyperspectral imaging: A review. J. Biomed. Opt.
**2014**, 19, 010901. [Google Scholar] [CrossRef] [PubMed] - Calin, M.A.; Parasca, S.V.; Savastru, D.; Manea, D. Hyperspectral imaging in the medical field: Present and future. Appl. Spectrosc. Rev.
**2014**, 49, 435–447. [Google Scholar] [CrossRef] - Sun, D.W. Hyperspectral Imaging for Food Quality Analysis and Control; Academic Press/Elsevier: San Diego, CA, USA, 2010. [Google Scholar]
- Kamruzzaman, M.; ElMasry, G.; Sun, D.; Allen, P. Non-destructive prediction and visualization of chemical composition in lamb meat using NIR hyperspectral imaging and multivariate regression. Innov. Food Sci. Emerg. Technol.
**2012**, 16, 218–226. [Google Scholar] [CrossRef] - ElMasry, G.; Kamruzzaman, M.; Sun, D.; Allen, P. Principles and applications of hyperspectral imaging in quality evaluation of agro-food products: A review. Crit. Rev. Food Sci. Nutr.
**2012**, 52, 999–1023. [Google Scholar] [CrossRef] [PubMed] - Li, B.; Beveridge, P.; O’Hare, W.T.; Islam, M. The application of visible wavelength reflectance hyperspectral imaging for the detection and identification of blood stains. Sci. Justice
**2014**, 54, 432–438. [Google Scholar] [CrossRef] [PubMed] - Yang, J.; Messinger, D.W.; Dube, R.R. Bloodstain detection and discrimination impacted by spectral shift when using an interference filter-based visible and near-infrared multispectral crime scene imaging system. Opt. Eng.
**2018**, 57, 033101. [Google Scholar] [CrossRef] - Brook, A.; Ben-Dor, E. A spatial/spectral protocol for quality assurance of decompressed hyperspectral data for practical applications. In Proceedings of the 2010 2nd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), Reykjavik, Iceland, 14–16 June 2010; pp. 1–4. [Google Scholar] [CrossRef]
- Li, C.; Sun, T.; Kelly, K.F.; Zhang, Y. A compressive sensing and unmixing scheme for hyperspectral data processing. IEEE Trans. Image Process.
**2012**, 21, 1200–1210. [Google Scholar] [CrossRef] [PubMed] - August, Y.; Vachman, C.; Stern, A. Spatial versus spectral compression ratio in compressive sensing of hyperspectral imaging. In Compressive Sensing II, Proceedings of the SPIE Defense, Security, and Sensing 2013, Baltimore, MD, USA, 29 April–3 May 2013; SPIE: Bellingham, WA, USA, 2013; Volume 8717. [Google Scholar]
- Willett, R.M.; Duarte, M.F.; Davenport, M.; Baraniuk, R.G. Sparsity and structure in hyperspectral imaging: Sensing, reconstruction, and target detection. IEEE Signal Process Mag.
**2014**, 31, 116–126. [Google Scholar] [CrossRef] - Parkinnen, J.; Hallikainen, J.; Jaaskelainen, T. Characteristic spectra of surface Munsell colors. J. Opt. Soc. Am. A
**1989**, 6, 318–322. [Google Scholar] [CrossRef] - August, Y.; Vachman, C.; Rivenson, Y.; Stern, A. Compressive hyperspectral imaging by random separable projections in both the spatial and the spectral domains. Appl. Opt.
**2013**, 52, D46–D54. [Google Scholar] [CrossRef] [PubMed] - Stern, A.; Yitzhak, A.; Farber, V.; Oiknine, Y.; Rivenson, Y. Hyperspectral Compressive Imaging. In Proceedings of the 2013 12th Workshop on Information Optics (WIO), Puerto de la Cruz, Spain, 15–19 July 2013; pp. 1–3. [Google Scholar] [CrossRef]
- Lin, X.; Wetzstein, G.; Liu, Y.; Dai, Q. Dual-coded compressive hyperspectral imaging. Opt. Lett.
**2014**, 39, 2044–2047. [Google Scholar] [CrossRef] [PubMed] - Arce, G.R.; Brady, D.J.; Carin, L.; Arguello, H.; Kittle, D.S. Compressive coded aperture spectral imaging: An introduction. IEEE Signal Process Mag.
**2014**, 31, 105–115. [Google Scholar] [CrossRef] - Stern, A. Optical Compressive Imaging; CRC Press: Boca Raton, FL, USA, 2016. [Google Scholar]
- Golub, M.A.; Averbuch, A.; Nathan, M.; Zheludev, V.A.; Hauser, J.; Gurevitch, S.; Malinsky, R.; Kagan, A. Compressed sensing snapshot spectral imaging by a regular digital camera with an added optical diffuser. Appl. Opt.
**2016**, 55, 432–443. [Google Scholar] [CrossRef] [PubMed] - Arce, G.R.; Rueda, H.; Correa, C.V.; Ramirez, A.; Arguello, H. Snapshot compressive multispectral cameras. In Wiley Encyclopedia of Electrical and Electronics Engineering; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2017; pp. 1–22. [Google Scholar] [CrossRef]
- Saragadam, V.; Wang, J.; Li, X.; Sankaranarayanan, A.C. Compressive spectral anomaly detection. In Proceedings of the 2017 IEEE International Conference on Computational Photography (ICCP), Stanford, CA, USA, 12–14 May 2017; pp. 1–9. [Google Scholar] [CrossRef]
- Wang, X.; Zhang, Y.; Ma, X.; Xu, T.; Arce, G.R. Compressive spectral imaging system based on liquid crystal tunable filter. Opt. Express
**2018**, 26, 25226–25243. [Google Scholar] [CrossRef] - August, Y.; Stern, A. Compressive sensing spectrometry based on liquid crystal devices. Opt. Lett.
**2013**, 38, 4996–4999. [Google Scholar] [CrossRef] - August, I.; Oiknine, Y.; AbuLeil, M.; Abdulhalim, I.; Stern, A. Miniature Compressive Ultra-spectral Imaging System Utilizing a Single Liquid Crystal Phase Retarder. Sci. Rep.
**2016**, 6, 23524. [Google Scholar] [CrossRef][Green Version] - Yariv, A.; Yeh, P. Optical Waves in Crystals; Wiley: New York, NY, USA, 1984. [Google Scholar]
- Candès, E.J.; Romberg, J.; Tao, T. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory
**2006**, 52, 489–509. [Google Scholar] [CrossRef] - Donoho, D.L. Compressed sensing. IEEE Trans. Inf. Theory
**2006**, 52, 1289–1306. [Google Scholar] [CrossRef] - Eldar, Y.C.; Kutyniok, G. Compressed Sensing: Theory and Applications; Cambridge University Press: Cambridge, UK, 2012; ISBN 9781107005587. [Google Scholar]
- Bioucas-Dias, J.M.; Figueiredo, M.A. A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process.
**2007**, 16, 2992–3004. [Google Scholar] [CrossRef] [PubMed] - Figueiredo, M.A.; Nowak, R.D.; Wright, S.J. Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process.
**2007**, 1, 586–597. [Google Scholar] [CrossRef] - Wright, S.J.; Nowak, R.D.; Figueiredo, M.A. Sparse reconstruction by separable approximation. IEEE Trans. Signal Process.
**2009**, 57, 2479–2493. [Google Scholar] [CrossRef] - Li, C.; Yin, W.; Zhang, Y. User’s guide for TVAL3: TV minimization by augmented lagrangian and alternating direction algorithms. CAAM Rep.
**2009**, 20, 46–47. [Google Scholar] - Elad, M. Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing; Springer Science & Business Media: New York, NY, USA, 2010. [Google Scholar]
- Aharon, M.; Elad, M.; Bruckstein, A. K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process.
**2006**, 54, 4311–4322. [Google Scholar] [CrossRef] - Chakrabarti, A.; Zickler, T. Statistics of real-world hyperspectral images. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, USA, 20–25 June 2011; pp. 193–200. [Google Scholar] [CrossRef]
- Oiknine, Y.; Arad, B.; August, I.; Ben-Shahar, O.; Stern, A. Dictionary based hyperspectral image reconstruction captured with CS-MUSI. In Proceedings of the 2018 9nd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), Amsterdam, The Netherlands, 23–26 September 2018. [Google Scholar]
- Pudil, P.; Novovičová, J.; Kittler, J. Floating search methods in feature selection. Pattern Recognit. Lett.
**1994**, 15, 1119–1125. [Google Scholar] [CrossRef] - Arad, B.; Ben-Shahar, O. Sparse Recovery of Hyperspectral Signal from Natural RGB Images. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 8–16 October 2016; pp. 19–34. [Google Scholar] [CrossRef]
- Kokaly, R.F.; Clark, R.N.; Swayze, G.A.; Livo, K.E.; Hoefen, T.M.; Pearson, N.C.; Wise, R.A.; Benzel, W.M.; Lowers, H.A.; Driscoll, R.L. USGS Spectral Library Version 7. USGS
**2017**, 1035, 61. [Google Scholar] [CrossRef] - Oiknine, Y.; August, I.; Stern, A. Along-track scanning using a liquid crystal compressive hyperspectral imager. Opt. Express
**2016**, 24, 8446–8457. [Google Scholar] [CrossRef] - Reddy, B.S.; Chatterji, B.N. An FFT-based technique for translation, rotation, and scale-invariant image registration. IEEE Trans. Image Process.
**1996**, 5, 1266–1271. [Google Scholar] [CrossRef] - Stern, A.; Kopeika, N.S. Motion-distorted composite-frame restoration. Appl. Opt.
**1999**, 38, 757–765. [Google Scholar] [CrossRef] - Usama, S.; Montaser, M.; Ahmed, O. A complexity and quality evaluation of block based motion estimation algorithms. Acta Polytech.
**2005**, 45, 29–41. [Google Scholar] - Oiknine, Y.; August, Y.I.; Revah, L.; Stern, A. Comparison between various patch wise strategies for reconstruction of ultra-spectral cubes captured with a compressive sensing system. In Compressive Sensing V: From Diverse Modalities to Big Data Analytics, Proceedings of the SPIE Commercial + Scientific Sensing and Imaging 2016, Baltimore, MD, USA, 17–21 April 2016; SPIE: Bellingham, WA, USA, 2016; Volume 985705. [Google Scholar] [CrossRef]
- Farber, V.; Oiknine, Y.; August, I.; Stern, A. Compressive 4D spectro-volumetric imaging. Opt. Lett.
**2016**, 41, 5174–5177. [Google Scholar] [CrossRef] [PubMed] - Stern, A.; Farber, V.; Oiknine, Y.; August, I. Compressive hyperspectral synthetic aperture integral imaging. In 3D Image Acquisition and Display: Technology, Perception and Applications; Paper DW1F. 1; Optical Society of America (OSA): Washington, DC, USA, 2017. [Google Scholar]
- Farber, V.; Oiknine, Y.; August, I.; Stern, A. 3D reconstructions from spectral light fields. In Three-Dimensional Imaging, Visualization, and Display 2018, Proceedings of the SPIE Commercial + Scientific Sensing and Imaging 2018, Orlando, Florida, USA, 15–19 April 2018; SPIE: Bellingham, WA, USA, 2018; Volume 10666. [Google Scholar] [CrossRef]
- Farber, V.; Oiknine, Y.; August, I.; Stern, A. Spectral light fields for improved three-dimensional profilometry. Opt. Eng.
**2018**, 57, 061609. [Google Scholar] [CrossRef] - Lippmann, G. Epreuves reversibles Photographies integrals. C. R. Acad. Sci
**1908**, 146, 446–451. [Google Scholar] - Arimoto, H.; Javidi, B. Integral three-dimensional imaging with digital reconstruction. Opt. Lett.
**2001**, 26, 157–159. [Google Scholar] [CrossRef] [PubMed] - Stern, A.; Javidi, B. Three-dimensional image sensing, visualization, and processing using integral imaging. Proc. IEEE
**2006**, 94, 591–607. [Google Scholar] [CrossRef] - Hong, S.; Jang, J.; Javidi, B. Three-dimensional volumetric object reconstruction using computational integral imaging. Opt. Express
**2004**, 12, 483–491. [Google Scholar] [CrossRef] - Aloni, D.; Stern, A.; Javidi, B. Three-dimensional photon counting integral imaging reconstruction using penalized maximum likelihood expectation maximization. Opt. Express
**2011**, 19, 19681–19687. [Google Scholar] [CrossRef] - Llavador, A.; Sánchez-Ortiga, E.; Saavedra, G.; Javidi, B.; Martínez-Corral, M. Free-depths reconstruction with synthetic impulse response in integral imaging. Opt. Express
**2015**, 23, 30127–30135. [Google Scholar] [CrossRef] - Busuioceanu, M.; Messinger, D.W.; Greer, J.B.; Flake, J.C. Evaluation of the CASSI-DD hyperspectral compressive sensing imaging system. In Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XIX, Proceedings of the SPIE Defense, Security, and Sensing 2013, Baltimore, MD, USA, 29 April–3 May 2013; SPIE: Bellingham, WA, USA, 2013; Volume 8743. [Google Scholar] [CrossRef]
- Gedalin, D.; Oiknine, Y.; August, I.; Blumberg, D.G.; Rotman, S.R.; Stern, A. Performance of target detection algorithm in compressive sensing miniature ultraspectral imaging compressed sensing system. Opt. Eng.
**2017**, 56, 041312. [Google Scholar] [CrossRef] - Oiknine, Y.; Gedalin, D.; August, I.; Blumberg, D.G.; Rotman, S.R.; Stern, A. Target detection with compressive sensing hyperspectral images. In Image and Signal Processing for Remote Sensing XXIII, Proceedings of the SPIE Remote Sensing, 2017, Warsaw, Poland, 11–14 September 2017; SPIE: Bellingham, WA, USA, 2017; Volume 10427. [Google Scholar]
- Caefer, C.E.; Stefanou, M.S.; Nielsen, E.D.; Rizzuto, A.P.; Raviv, O.; Rotman, S.R. Analysis of false alarm distributions in the development and evaluation of hyperspectral point target detection algorithms. Opt. Eng.
**2007**, 46, 076402. [Google Scholar] [CrossRef] - Bar-Tal, M.; Rotman, S.R. Performance measurement in point source target detection. In Proceedings of the Eighteenth Convention of Electrical and Electronics Engineers in Israel, Tel Aviv, Israel, 7–8 March 1995; pp. 3.4.6/1–3.4.6/5. [Google Scholar] [CrossRef]
- Skauli, T.; Farrell, J. A collection of hyperspectral images for imaging systems research. In Digital Photography IX, Proceedings of the IS&T/SPIE Electronic Imaging, Burlingame, CA, USA, 3–7 February 2013; SPIE: Bellingham, WA, USA, 2013; Volume 8660. [Google Scholar] [CrossRef]
- Hyperspectral Remote Sensing Scenes. Available online: http://www.ehu.eus/ccwintco/index.php?title=Hyperspectral_Remote_Sensing_Scenes (accessed on 26 October 2018).
- Fellgett, P. The Multiplex Advantage. Ph.D. Thesis, University of Cambridge, Cambridge, UK, 1951. [Google Scholar]
- Oiknine, Y.; August, I.; Stern, A. Compressive spectroscopy by spectral modulation. In Optical Sensors 2017, Proceedings of the SPIE Optics + Optoelectronics, 2017, Prague, Czech Republic, 24–27 April 2017; SPIE: Bellingham, WA, USA, 2017; Volume 10231. [Google Scholar] [CrossRef]
- Oiknine, Y.; August, I.; Blumberg, D.G.; Stern, A. Compressive sensing resonator spectroscopy. Opt. Lett.
**2017**, 42, 25–28. [Google Scholar] [CrossRef] [PubMed] - Oiknine, Y.; August, I.; Blumberg, D.G.; Stern, A. NIR hyperspectral compressive imager based on a modified Fabry–Perot resonator. J. Opt.
**2018**, 20, 044011. [Google Scholar] [CrossRef][Green Version] - Oiknine, Y.; August, I.; Stern, A. Multi-aperture snapshot compressive hyperspectral camera. Opt. Lett.
**2018**, 43, 5042–5045. [Google Scholar] [CrossRef]

**Figure 1.**Spectral multiplexing. The figure represents three different examples of spectral multiplexing. Each sub-figure illustrates multiplexing of a few spectral bands onto a FPA.

**Figure 2.**LC cell phase retarder. The LC phase retarder is made of a Nematic LC layer (blue arrow) sandwiched between two glass plates and two linear polarizers (green layers). The glass plates are coated with Indium Tin Oxide (ITO, pink layers) and a polymer alignment layer (purple layers).

**Figure 3.**Measured spectral responses (intensity transmission vs. wavelength in nm) of the fabricated LC phase retarder. Each graph represents the spectral modulation with a different voltage applied on the LC cell (15 different voltages).

**Figure 4.**(

**a**) CS-MUSI acquisition process. (

**b**) CS-MUSI optical scheme diagram. The HS object $F(x,y,\lambda )$ is modulated according to ${\varphi}_{\mathrm{LC}}\left(\lambda ,{V}_{i}\right)$, yielding the multiplexed measurement ${G}_{i}(x,y)$.

**Figure 6.**CS-MUSI spectral response map for voltages from 0 V to 10 V (

**left map**) and a zoom in on the area where the voltages are from 1.3 V to 3.5 V (

**right map**).

**Figure 7.**Staring mode reconstruction result of three LED arrays. (

**a**) RGB color image of three LED arrays that were used as objects to be imaged with CS-MUSI. (

**b**–

**e**) Representative single exposure images for LC cell voltage of 0 V, 5.8373 V, 7.6301 V and 8.6552 V, respectively. (

**f**) RGB representation of the reconstructed HS image (700 × 700 pixels× 391 bands). (

**g**–

**i**) Reconstructed images at 460 nm, 520 nm and 650 nm, respectively. (

**k**–

**m**) Spectrum reconstruction for three points in the HS datacube and comparison to the measured spectra of the three respective LEDs with a commercial grating-based spectrometer.

**Figure 8.**Staring mode reconstruction result of six different markers. (

**a**) RGB representation of the reconstructed HS image (800 × 900 pixels × 1171 bands). (

**b**) Four reconstructed images at four different wavelengths (470 nm, 530 nm, 580 nm, and 630 nm).

**Figure 9.**Staring mode reconstruction results with the dictionary of (

**a**) outdoor and (

**b**) indoor HS images taken with CS-MUSI camera. The figures show RGB representation of the reconstructed HS datacube.

**Figure 10.**CS-MUSI camera along-track scanning. Each shot of the CS-MUSI camera, ${G}_{i}$, captures a shifted scene with a different LC spectral transmission (which depends on the voltage ${v}_{i}$).

**Figure 11.**Scanning mode (Figure 10) reconstruction result. (

**a**) RGB color image of three LED arrays. (

**b**–

**e**) representative single exposure images (frame #30, #90, #150 and #300, respectively) and (

**f**–

**i**) the RGB representation of the reconstructed HS image up to the appropriate column.

**Figure 13.**(

**a**) 4D Spectro-Volumetric imaging. (

**b**) Grayscale representation of HS images at three different depths (225 cm, 254 cm and 270 cm).

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Oiknine, Y.; August, I.; Farber, V.; Gedalin, D.; Stern, A. Compressive Sensing Hyperspectral Imaging by Spectral Multiplexing with Liquid Crystal. *J. Imaging* **2019**, *5*, 3.
https://doi.org/10.3390/jimaging5010003

**AMA Style**

Oiknine Y, August I, Farber V, Gedalin D, Stern A. Compressive Sensing Hyperspectral Imaging by Spectral Multiplexing with Liquid Crystal. *Journal of Imaging*. 2019; 5(1):3.
https://doi.org/10.3390/jimaging5010003

**Chicago/Turabian Style**

Oiknine, Yaniv, Isaac August, Vladimir Farber, Daniel Gedalin, and Adrian Stern. 2019. "Compressive Sensing Hyperspectral Imaging by Spectral Multiplexing with Liquid Crystal" *Journal of Imaging* 5, no. 1: 3.
https://doi.org/10.3390/jimaging5010003