# An Entropy-Based Algorithm with Nonlocal Residual Learning for Image Compressive Sensing Recovery

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Background

#### 2.1. Compressed Sensing

_{0}counting “norm” for sparsity), and ${\Vert x\Vert}_{TV}$ respectively).

#### 2.2. CS Recovery Based on Nonlocal Sparsity

#### 2.3. Denoising-Based Approximate Message Passing

## 3. Image CS Recovery via Nonlocal Residual Learning and D-AMP

#### 3.1. Residual Learning

#### 3.2. LSM Prior Modeling

#### 3.3. Entropy-Based Algorithm for CS Recovery

**Algorithm 1**, named as the D-AMP algorithm with residual learning (RL-DAMP).

Algorithm 1. The D-AMP Algorithm with Residual Learning. |

Input:y, A, T, $\alpha $, $\beta $, x^{0} = 0, z^{0} = y.For t = 1 to T do(a) Approximate the Onsager correction term via the MC method. (b) Update the residual ${z}^{(t)}$ with Equation (6). (c) Obtain the intermediate noisy image ${h}^{(t)}={x}^{(t)}+{A}^{\ast}{z}^{(t)}$, and estimate the noise variance ${\sigma}^{2}$. (d) Perform the denoising operator based on residual learning: For i = 1 to G do(I) Perform the BM3D denoising operator on ${h}^{(t)}$ to get the image $Db$. (II) Construct the low-rank matrix ${H}_{i}^{(t)}$ and perform the SVD on ${H}_{i}^{(t)}$ to get the dictionary ${D}_{i}^{(t)}$. (III) Estimate the expectations of scale parameters ${\tau}_{i,j}^{(t+1)}$ via Equation (19). (IV) Compute the global optimums of coefficients ${s}_{i}^{(t+1)}$ via Equation (21), and then obtain ${A}_{i}^{(t+1)}$ via Equation (22). (V) Obtain the matrix constructed by similar patches, therefore, ${X}_{i}^{(t+1)}={D}_{i}^{(t+1)}{A}_{i}^{(t+1)}$. (VI) If i = G, recover the whole image ${x}^{(t+1)}$ by aggregating all recovered pixels. |

## 4. Experiments

#### 4.1. Parameter Settings

#### 4.2. Experiments on Noiseless Data

#### 4.3. Experiments on Noisy Data

#### 4.4. Experiments on Small-Size Images

## 5. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Donoho, D.L. Compressed Sensing. IEEE Trans. Inf. Theory
**2006**, 52, 1289–1306. [Google Scholar] [CrossRef] - Zhang, J.; Yang, K. Informational Analysis for Compressive Sampling in Radar Imaging. Sensors
**2015**, 15, 7136–7155. [Google Scholar] [CrossRef] [Green Version] - Tropp, J.; Gilbert, A.C. Signal Recovery from Random Measurements Via Orthogonal Matching Pursuit. IEEE Trans. Inf. Theory
**2007**, 53, 4655–4666. [Google Scholar] [CrossRef] - Fu, L.; Zhang, M.; Liu, Z.; Li, H. Robust Frequency Estimation of Multi-sinusoidal Signals Using Orthogonal Matching Pursuit with Weak Derivatives Criterion. Circuits Syst. Signal Process.
**2019**, 38, 1194–1205. [Google Scholar] [CrossRef] - Beck, A.; Teboulle, M. A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems. SIAM J. Imaging Sci.
**2009**, 2, 183–202. [Google Scholar] [CrossRef] [Green Version] - Becker, S.; Bobin, J.; Candes, E.J. NESTA: A Fast and Accurate First Order Method for Sparse Recovery. SIAM J. Imaging Sci.
**2009**, 4, 1–39. [Google Scholar] [CrossRef] - He, L.; Chen, H.; Carin, L. Tree-Structured Compressive Sensing with Variational Bayesian Analysis. IEEE Signal Process. Lett.
**2010**, 17, 233–236. [Google Scholar] - Shekaramiz, M.; Moon, T.K.; Gunther, J.H. Bayesian Compressive Sensing of Sparse Signals with Unknown Clustering Patterns. Entropy
**2019**, 21, 247. [Google Scholar] [CrossRef] - Chen, Z.; Ma, S.; Wen, G. Image Compressive-Sensing Recovery Using Structured Laplacian Sparsity in DCT Domain and Multi-Hypothesis Prediction. In Proceedings of the 2014 IEEE International Conference on Multimedia & Expo (ICME), Chengdu, China, 14–18 July 2014; pp. 1–6. [Google Scholar]
- Wu, J.; Liu, F.; Jiao, L.C.; Wang, X.; Hou, B. Multivariate Compressive Sensing for Image Reconstruction in the Wavelet Domain: Using Scale Mixture Models. IEEE Trans. Image Process.
**2011**, 20, 3483–3494. [Google Scholar] [CrossRef] - Rudin, L.; Osher, S.; Fatemi, E. Nonlinear Total Variation Based Noise Removal Algorithms. Phys. D Nonlinear Phenom.
**1992**, 60, 259–268. [Google Scholar] [CrossRef] - Huang, J.; Zhang, S.; Metaxas, D. Efficient MR Image Reconstruction for Compressed MR Imaging. Med. Image Anal.
**2011**, 15, 670–679. [Google Scholar] [CrossRef] - Ashwini, K.; Amutha, R. Compressive Sensing Based Simultaneous Fusion and Compression of Multi-Focus Images Using Learned Dictionary. Multimed. Tools Appl.
**2018**, 77, 25889–25904. [Google Scholar] [CrossRef] - Smith, L.N.; Elad, M. Improving Dictionary Learning: Multiple Dictionary Updates and Coefficient Reuse. IEEE Signal Process. Lett.
**2012**, 20, 79–82. [Google Scholar] [CrossRef] - Guerrero-Colon, J.A.; Mancera, L.; Portilla, J. Image Restoration Using Space-Variant Gaussian Scale Mixtures in Overcomplete Pyramids. IEEE Trans. Image Process.
**2008**, 17, 27–41. [Google Scholar] [CrossRef] - Eldar, Y.C.; Kuppinger, P.; Bolcskei, H. Block-Sparse Signals: Uncertainty Relations and Efficient Recovery. IEEE Trans. Signal Process.
**2010**, 58, 3042–3054. [Google Scholar] [CrossRef] [Green Version] - Chen, C.; Huang, J. Exploiting the Wavelet Structure in Compressed Sensing MRI. Magn. Reson. Imaging
**2014**, 32, 1377–1389. [Google Scholar] [CrossRef] - Som, S.; Schniter, P. Compressive Imaging Using Approximate Message Passing and A Markov-Tree Prior. IEEE Trans. Signal Process.
**2012**, 60, 3439–3448. [Google Scholar] [CrossRef] - Zhang, X.; Bai, T.; Meng, H.; Chen, J. Compressive Sensing-Based ISAR Imaging via the Combination of the Sparsity and Nonlocal Total Variation. IEEE Geosci. Remote Sens. Lett.
**2014**, 11, 990–994. [Google Scholar] [CrossRef] - Dong, W.; Shi, G.; Wu, X.; Zhang, L. A learning-based method for compressive image recovery. J. Vis. Commun. Image Represent
**2013**, 24, 1055–1063. [Google Scholar] [CrossRef] - Seitzer, M.; Yang, G.; Schlemper, J.; Oktay, O.; Würfl, T.; Christlein, V.; Wong, T.; Mohiaddin, R.; Firmin, D.; Keegan, J.; et al. Adversarial and Perceptual Refinement for Compressed Sensing MRI Reconstruction. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Granada, Spain, 16–20 September 2018; pp. 232–240. [Google Scholar]
- Mousavi, A.; Baraniuk, R.G. Learning to Invert: Signal Recovery via Deep Convolutional Networks. In Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA, 5–9 March 2017; pp. 2272–2276. [Google Scholar]
- Yao, H.; Dai, F.; Zhang, D.; Ma, Y.; Zhang, S.; Zhang, Y. DR
^{2}-net: Deep Residual Reconstruction Network for Image Compressive Sensing. arXiv**2017**, arXiv:1702.05743. [Google Scholar] [CrossRef] - Lohit, S.; Kulkarni, K.; Kerviche, R.; Turaga, P.; Ashok, A. Convolutional Neural Networks for Non-iterative Reconstruction of Compressively Sensed Images. IEEE Trans. Comput. Imaging
**2018**, 4, 326–340. [Google Scholar] [CrossRef] - Yang, Y.; Sun, J.; Li, H.; Xu, Z. ADMM-CS Net: A Deep Learning Approach for Image Compressive Sensing. IEEE Trans. Pattern Anal. Mach. Intell.
**2018**, 1. [Google Scholar] [CrossRef] - Metzler, C.A.; Mousavi, A.; Baraniuk, R.G. Learned D-AMP: Principled Neural Network based Compressive Image Recovery. arXiv
**2017**, arXiv:1704.06625. [Google Scholar] - Yang, G.; Yu, S.; Dong, H.; Slabaugh, G.; Dragotti, P.L.; Ye, X.; Liu, F.; Arridge, S.; Keegan, J.; Guo, Y.; et al. DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction. IEEE Trans. Med. Imaging
**2018**, 37, 1310–1321. [Google Scholar] [CrossRef] - Yu, S.; Dong, H.; Yang, G.; Slabaugh, G.; Dragotti, P.L.; Ye, X.; Liu, F.; Arridge, S.; Keegan, J.; Firmin, D.; et al. Deep De-Aliasing for Fast Compressive Sensing MRI. arXiv
**2017**, arXiv:1705.07137. [Google Scholar] - Schlemper, J.; Yang, G.; Ferreira, P.; Scott, A.; McGill, L.A.; Khalique, Z.; Gorodezky, M.; Roehl, M.; Keegan, J.; Pennell, D.; et al. Stochastic Deep Compressive Sensing for the Reconstruction of Diffusion Tensor Cardiac MRI. In Proceedings of the 2018 International Conference on Medical Image Computing and Computer-Assisted Intervention, Granada, Spain, 16–20 September 2018; pp. 295–303. [Google Scholar]
- Dong, W.; Shi, G.; Li, X.; Ma, Y.; Huang, F. Compressive Sensing via Nonlocal Low-rank Regularization. IEEE Trans. Image Process.
**2014**, 23, 3618–3632. [Google Scholar] [CrossRef] - Metzler, C.A.; Maleki, A.; Baraniuk, R.G. From Denoising to Compressed Sensing. IEEE Trans. Inf. Theory
**2016**, 62, 5117–5144. [Google Scholar] [CrossRef] - Metzler, C.A.; Maleki, A.; Baraniuk, R.G. BM3D-AMP: A New Image Recovery Algorithm Based on BM3D Denoising. In Proceedings of the 2015 IEEE International Conference Image Processing (ICIP), Quebec City, QC, Canada, 27–30 September 2015; pp. 3116–3120. [Google Scholar]
- Dabov, K.; Foi, A.; Katkovnik, V.; Egiazarian, K. Image Denoising By Sparse 3-D Transform-Domain Collaborative Filtering. IEEE Trans. Image Process.
**2007**, 16, 2080–2095. [Google Scholar] [CrossRef] - Zhang, J.; Zhao, D.; Zhao, C.; Xiong, R.; Ma, S.; Gao, W. Image Compressive Sensing Recovery via Collaborative Sparsity. IEEE J. Emerg. Sel. Top. Circuits Syst.
**2012**, 2, 380–391. [Google Scholar] [CrossRef] - Dong, W.; Shi, G.; Li, X.; Zhang, L.; Wu, X. Image Reconstruction with Locally Adaptive Sparsity and Nonlocal Robust Regularization. Signal Process. Image Commun.
**2012**, 27, 1109–1122. [Google Scholar] [CrossRef] - Xie, Z.; Ma, L.; Zhong, F. An Approximate Message Passing Algorithm with Composite Sparse Constraint for CS Reconstruction. J. S. China Univ. Technol.
**2017**, 45, 18–25. [Google Scholar] - Shi, F.; Cheng, J.; Wang, L.; Yap, P.T.; Shen, D. LRTV: MR Image Super-Resolution with Low-Rank and Total Variation Regularizations. IEEE Trans. Med. Imaging
**2015**, 34, 2459–2466. [Google Scholar] [CrossRef] - Garrigues, P.; Olshausen, B.A. Group Sparse Coding with A Laplacian Scale Mixture Prior. In Proceedings of the Advances in Neural Information Processing Systems (NIPS), Vancouver, BC, Canada, 6–9 December 2010; pp. 676–684. [Google Scholar]
- Dempster, A.; Laird, N.M.; Rubin, D.B. Maximum-likelihood from Incomplete Data via the EM Algorithm. J. R. Stat. Soc.
**1977**, 39, 1–22. [Google Scholar] [CrossRef] - Donoho, D.L.; Maleki, A.; Montanari, A. Message Passing Algorithms for Compressed Sensing. Proc. Natl. Acad. Sci. USA
**2009**, 106, 18914–18919. [Google Scholar] [CrossRef] - Daubechies, I.; Defrise, M.; DeMol, C. An Iterative Thresholding Algorithm for Linear Inverse Problems with A Sparsity Constraint. Commun. Pure Appl. Math.
**2004**, 57, 1413–1457. [Google Scholar] [CrossRef] - Hosseini, M.S.; Plataniotis, K.N. High-Accuracy Total Variation with Application to Compressed Video Sensing. IEEE Trans. Image Process.
**2014**, 23, 3869–3884. [Google Scholar] [CrossRef] - Yin, W.; Osher, S.; Goldfarb, D.; Darbon, J. Bregman Iterative Algorithms For l1-minimization with Applications to Compressed Sensing. SIAM J. Imaging Sci.
**2008**, 1, 143–168. [Google Scholar] [CrossRef] - Donoho, D.L.; Johnstone, I.; Montanari, A. Accurate Prediction of Phase Transitions in Compressed Sensing via A Connection to Minimax Denoising. IEEE Trans. Inf. Theory
**2013**, 59, 3396–3433. [Google Scholar] [CrossRef] - Wang, X.; Liang, J. Side Information-aided Compressed Sensing Reconstruction Via Approximate Message Passing. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vancouver, BC, Canada, 4–9 May 2014; pp. 3330–3334. [Google Scholar]
- Hill, P.R.; Kim, J.H.; Basarab, A.; Kouamé, D.; Bull, D.R.; Achim, A. Compressive Imaging Using Approximate Message Passing and A Cauchy Prior in The Wavelet Domain. In Proceedings of the IEEE International Conference Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016; pp. 2514–2518. [Google Scholar]
- Ramani, S.; Blu, T.; Unser, M. Monte-Carlo Sure: A Black-Box Optimization of Regularization Parameters for General Denoising Algorithms. IEEE Trans. Image Process.
**2008**, 17, 1540–1554. [Google Scholar] [CrossRef] [Green Version] - Gu, S.; Xie, Q.; Meng, D.; Zuo, W.; Feng, X.; Zhang, L. Weighted Nuclear Norm Minimization and Its Applications to Low Level Vision. Int. J. Comput. Vis.
**2017**, 121, 183–208. [Google Scholar] [CrossRef] - Dong, W.; Shi, G.; Li, X. Nonlocal Image Restoration with Bilateral Variance Estimation: A Low-Rank Approach. IEEE Trans. Image Process.
**2013**, 22, 700–711. [Google Scholar] [CrossRef] - Rodger, J.A. QuantumIS: A Qualia Consciousness Awareness and Information Theory Quale Approach to Reducing Strategic Decision-Making Entropy. Entropy
**2019**, 21, 125. [Google Scholar] [CrossRef]

**Figure 4.**Average Peak Signal-to-Noise Ratio (PSNR) (

**a**) and Structural Similarity Index (SSIM) (

**b**) at different sample ratios.

**Figure 6.**Visual comparisons on Boat image at 20% sampling ratio. (

**a**) The original; (

**b**) NLR-CS (PSNR: 32.08 dB, SSIM: 0.9068); (

**c**) BM3D-AMP (PSNR: 32.29 dB, SSIM: 0.9387); (

**d**) ADMM-Net (PSNR: 32.71 dB, SSIM: 0.9226); (

**e**) LR-AMP (PSNR: 31.92 dB, SSIM: 0.9435); (

**f**) RL-DAMP (PSNR: 33.92 dB, SSIM: 0.9473).

**Figure 7.**Visual comparisons on Chest image at 18% sampling ratio. (

**a**) The original; (

**b**) NLR-CS (PSNR: 30.80 dB, SSIM: 0.9065); (

**c**) BM3D-AMP (PSNR: 31.70 dB, SSIM: 0.9353); (

**d**) ADMM-Net (PSNR: 32.33 dB, SSIM: 0.9187); (

**e**) LR-AMP (PSNR: 31.05 dB, SSIM: 0.9353); (

**f**) RL-DAMP (PSNR: 32.57 dB, SSIM: 0.9435).

**Figure 8.**Iterative curves on Boat image at 20% sampling ratio. (

**a**) Average PSNR to iterations; (

**b**) Average PSNR to CPU (central processing unit) time.

**Figure 9.**Iterative curves on Chest image at 20% sampling ratio. (

**a**) Average PSNR to iterations; (

**b**) Average PSNR to CPU time.

**Figure 10.**Average PSNR at different sample ratios with measurement noise with standard deviation 8 and 15. (

**a**) The standard deviation is 8; (

**b**) the standard deviation is 15.

**Figure 11.**Iterative curves on Barbara image at 20% sampling ratio with measurement noise with standard deviation 15. (

**a**) Average PSNR to iterations; (

**b**) average PSNR to CPU time.

**Figure 12.**Iterative curves on Brain image at 20% sampling ratio with measurement noise with standard deviation 8. (

**a**) Average PSNR to iterations; (

**b**) average PSNR to CPU time.

**Figure 13.**Visual comparisons on Barbara image at 20% sampling ratio with measurement noise with standard deviation 15. (

**a**) The original; (

**b**) NLR-CS (PSNR: 27.16 dB, SSIM: 0.8061); (

**c**) BM3D-AMP (PSNR: 28.65 dB, SSIM: 0.8465); (

**d**) ADMM-Net (PSNR: 28.13 dB, SSIM: 0.8270); (

**e**) LR-AMP (PSNR: 29.69 dB, SSIM: 0.8703); (

**f**) RL-DAMP (PSNR: 29.89 dB, SSIM: 0.8794).

**Figure 14.**Visual comparisons on Brain image at 20% sampling ratio with measurement noise with standard deviation 8. (

**a**) The original; (

**b**) NLR-CS (PSNR: 32.65 dB, SSIM: 0.8714); (

**c**) BM3D-AMP (PSNR: 33.63 dB, SSIM: 0.8962); (

**d**) ADMM-Net (PSNR: 31.21 dB, SSIM: 0.8107); (

**e**) LR-AMP (PSNR: 32.93 dB, SSIM: 0.8743); (

**f**) RL-DAMP (PSNR: 34.05 dB, SSIM: 0.8976).

**Figure 15.**Visual comparisons on Barbara image at 20% sampling ratio with size of 128 × 128. (

**a**) The original; (

**b**) NLR-CS (PSNR: 29.18 dB, SSIM: 0.8890); (

**c**) BM3D-AMP (PSNR: 28.49 dB, SSIM: 0.8997); (

**d**) LR-AMP (PSNR: 29.73 dB, SSIM: 0.9233); (

**e**) RL-DAMP (PSNR: 29.94 dB, SSIM: 0.9240).

**Figure 16.**Visual comparisons on Barbara image at 20% sampling ratio with size of 128 × 128 with measurement noise with standard deviation 8. (

**a**) The original; (

**b**) NLR-CS (PSNR: 27.17 dB, SSIM: 0.8369); (

**c**) BM3D-AMP (PSNR: 27.01 dB, SSIM: 0.8366); (

**d**) LR-AMP (PSNR: 27.74 dB, SSIM: 0.8486); (

**e**) RL-DAMP (PSNR: 27.93 dB, SSIM: 0.8495).

**Figure 17.**The sampling mask at 20% sampling ratio with different sizes. (

**a**) 256 × 256; (

**b**) 128 × 128.

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Xie, Z.; Liu, L.; Yang, C.
An Entropy-Based Algorithm with Nonlocal Residual Learning for Image Compressive Sensing Recovery. *Entropy* **2019**, *21*, 900.
https://doi.org/10.3390/e21090900

**AMA Style**

Xie Z, Liu L, Yang C.
An Entropy-Based Algorithm with Nonlocal Residual Learning for Image Compressive Sensing Recovery. *Entropy*. 2019; 21(9):900.
https://doi.org/10.3390/e21090900

**Chicago/Turabian Style**

Xie, Zhonghua, Lingjun Liu, and Cui Yang.
2019. "An Entropy-Based Algorithm with Nonlocal Residual Learning for Image Compressive Sensing Recovery" *Entropy* 21, no. 9: 900.
https://doi.org/10.3390/e21090900