Multi-Scale Shape Adaptive Network for Raindrop Detection and Removal from a Single Image
Abstract
:1. Introduction
- First, we extend an automatic raindrop rendering method and construct a large-scale synthetic raindrop dataset including 11,583 high-resolution raindrop and raindrop-free image pairs covering a wide variety of raindrop characteristics and background scenarios.
- Second, we propose a novel end-to-end raindrop removal network called Multi-scale Shape Adaptive Network (MSANet), which is composed of raindrop detection and removal branches. The MSANet can remove raindrops effectively while preserving more image details.
- Third, in the raindrop detection branch, the receptive field block (RFB) is used to strengthen the raindrop feature discriminability for accurately generating a raindrop map. Meanwhile, in the removal branch, the multi-scale dilated convolution module (MDCM) and multi-scale densely enhanced deformable module (DEDM) are adopted to effectively extract semantic information and adaptively remove diverse raindrops, respectively. The final derained result is obtained via a fusion between the two branches for better deraining.
- Lastly, we perform extensive experiments to evaluate the proposed method on both synthetic and real-world raindrop images. The results demonstrate that our proposed method outperforms the recent state-of-the-art methods. Furthermore, the extension of this model to rainy image segmentation and detection can benefit outdoor applications.
2. Related Work
2.1. Rain Streak and Rain Mist Removal
2.2. Raindrop Removal
2.2.1. Multi-Image Based Raindrop Removal
2.2.2. Single-Image Based Raindrop Removal
2.3. Deformable Convolution
3. RaindropCityscapes Dataset
4. Proposed Method
4.1. Raindrop Detection Branch
4.2. Raindrop Removal Branch
4.2.1. Multi-Scale Dilated Convolution Module
4.2.2. Multi-Scale Densely Enhanced Deformable Module
4.3. Loss Function
5. Experiments
5.1. Implementation Details
5.2. Results and Comparisons
5.2.1. Comparison Results on the Synthetic Dataset
5.2.2. Comparison Results on a Real-World Dataset
5.3. Ablation Study
5.3.1. Effectiveness of Modules in the Raindrop Removal Branch
5.3.2. Effectiveness of the Raindrop Detection Branch
5.4. Extension for High-Level Applications
5.5. Discussion
6. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Zhu, C.; Yin, X.C. Detecting multi-resolution pedestrians using group cost-sensitive boosting with channel features. Sensors 2019, 19, 780. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Arshad, S.; Sualeh, M.; Kim, D.; Nam, D.V.; Kim, G.W. Clothoid: an integrated hierarchical framework for autonomous driving in a dynamic urban environment. Sensors 2020, 20, 5053. [Google Scholar] [CrossRef] [PubMed]
- Sindagi, V.A.; Patel, V.M. Multi-level bottom-top and top-bottom feature fusion for crowd counting. In Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea, 27 October–2 November 2019; pp. 1002–1012. [Google Scholar]
- Sun, R.; Huang, Q.; Xia, M.; Zhang, J. Video-based person re-identification by an end-to-end learning architecture with hybrid deep appearance-temporal feature. Sensors 2018, 18, 3669. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Li, X.; Wu, J.; Lin, Z.; Liu, H.; Zha, H. Recurrent squeeze-and-excitation context aggregation net for single image deraining. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 254–269. [Google Scholar]
- Fu, X.; Huang, J.; Ding, X.; Liao, Y.; Paisley, J. Clearing the skies: A deep network architecture for single-image rain removal. IEEE Trans. Image Process. 2017, 26, 2944–2956. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhang, H.; Patel, V.M. Density-aware single image de-raining using a multi-stream dense network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 695–704. [Google Scholar]
- Wang, T.; Yang, X.; Xu, K.; Chen, S.; Zhang, Q.; Lau, R.W. Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 12270–12279. [Google Scholar]
- Wang, C.; Zhang, M.; Su, Z.; Yao, G.; Wang, Y.; Sun, X.; Luo, X. From coarse to fine: A stage-wise deraining net. IEEE Access 2019, 7, 84420–84428. [Google Scholar] [CrossRef]
- Ren, Y.; Li, S.; Nie, M.; Li, C. Single Image De-Raining via Improved Generative Adversarial Nets. Sensors 2020, 20, 1591. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Li, R.; Cheong, L.F.; Tan, R.T. Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 1633–1642. [Google Scholar]
- Hu, X.; Fu, C.W.; Zhu, L.; Heng, P.A. Depth-attentional Features for Single-image Rain Removal. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 8022–8031. [Google Scholar]
- Chen, K.C.; Chen, P.S.; Wong, S.K. A heuristic approach to the simulation of water drops and flows on glass panes. Comput. Graph. 2013, 37, 963–973. [Google Scholar] [CrossRef]
- Fu, X.; Huang, J.; Zeng, D.; Huang, Y.; Ding, X.; Paisley, J. Removing rain from single images via a deep detail network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 3855–3863. [Google Scholar]
- Roser, M.; Geiger, A. Video-based raindrop detection for improved image registration. In Proceedings of the 2009 IEEE 12th International Conference on Computer Vision Workshops (ICCV Workshops), Kyoto, Japan, 27 September–4 October 2009; pp. 570–577. [Google Scholar]
- You, S.; Tan, R.T.; Kawakami, R.; Ikeuchi, K. Adherent raindrop detection and removal in video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 1035–1042. [Google Scholar]
- You, S.; Tan, R.T.; Kawakami, R.; Mukaigawa, Y.; Ikeuchi, K. Raindrop detection and removal from long range trajectories. In Proceedings of the Asian Conference on Computer Vision, Singapore, 1–5 November 2014; Springer: Berlin/Heidelberg, Germany, 2014; pp. 569–585. [Google Scholar]
- Eigen, D.; Krishnan, D.; Fergus, R. Restoring an image taken through a window covered with dirt or rain. In Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013; pp. 633–640. [Google Scholar]
- Voulodimos, A.; Doulamis, N.; Doulamis, A.; Protopapadakis, E. Deep learning for computer vision: A brief review. Comput. Intell. Neurosci. 2018, 2018. [Google Scholar] [CrossRef] [PubMed]
- Guo, Y.; Liu, Y.; Oerlemans, A.; Lao, S.; Wu, S.; Lew, M.S. Deep learning for visual understanding: A review. Neurocomputing 2016, 187, 27–48. [Google Scholar] [CrossRef]
- Qian, R.; Tan, R.T.; Yang, W.; Su, J.; Liu, J. Attentive generative adversarial network for raindrop removal from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 2482–2491. [Google Scholar]
- Wu, Q.; Zhang, W.; Kumar, B.V. Raindrop detection and removal using salient visual features. In Proceedings of the 2012 19th IEEE International Conference on Image Processing, Orlando, FL, USA, 30 September–3 October 2012; pp. 941–944. [Google Scholar]
- Huang, D.A.; Kang, L.W.; Wang, Y.C.F.; Lin, C.W. Self-learning based image decomposition with applications to single image denoising. IEEE Trans. Multimed. 2013, 16, 83–93. [Google Scholar] [CrossRef]
- Li, Y.; Tan, R.T.; Guo, X.; Lu, J.; Brown, M.S. Rain streak removal using layer priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 2736–2744. [Google Scholar]
- Luo, Y.; Xu, Y.; Ji, H. Removing rain from a single image via discriminative sparse coding. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 3397–3405. [Google Scholar]
- Zhu, L.; Fu, C.W.; Lischinski, D.; Heng, P.A. Joint bi-layer optimization for single-image rain streak removal. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2526–2534. [Google Scholar]
- Dai, J.; Qi, H.; Xiong, Y.; Li, Y.; Zhang, G.; Hu, H.; Wei, Y. Deformable convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 764–773. [Google Scholar]
- Zhang, C.; Kim, J. Object detection with location-aware deformable convolution and backward attention filtering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 9452–9461. [Google Scholar]
- Guo, D.; Li, K.; Zha, Z.J.; Wang, M. Dadnet: Dilated-attention-deformable convnet for crowd counting. In Proceedings of the 27th ACM International Conference on Multimedia, Nice, France, 21–25 October 2019; pp. 1823–1832. [Google Scholar]
- Wang, X.; Chan, K.C.; Yu, K.; Dong, C.; Change Loy, C. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA, 16–17 June 2019. [Google Scholar]
- Tian, Y.; Zhang, Y.; Fu, Y.; Xu, C. TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 3360–3369. [Google Scholar]
- Wang, H.; Su, D.; Liu, C.; Jin, L.; Sun, X.; Peng, X. Deformable Non-Local Network for Video Super-Resolution. IEEE Access 2019, 7, 177734–177744. [Google Scholar] [CrossRef]
- Cordts, M.; Omran, M.; Ramos, S.; Rehfeld, T.; Enzweiler, M.; Benenson, R.; Franke, U.; Roth, S.; Schiele, B. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 3213–3223. [Google Scholar]
- Liu, S.; Huang, D.; Wang, Y. Receptive field block net for accurate and fast object detection. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 385–400. [Google Scholar]
- Wu, Z.; Su, L.; Huang, Q. Cascaded partial decoder for fast and accurate salient object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 3907–3916. [Google Scholar]
- Zhu, X.; Hu, H.; Lin, S.; Dai, J. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 9308–9316. [Google Scholar]
- Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K.Q. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 4700–4708. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–30 July 2016; pp. 770–778. [Google Scholar]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. Pytorch: An imperative style, high-performance deep learning library. In Proceedings of the Advances in Neural Information Processing Systems 33 (NIPS 2019), Vancouver, BC, Canada, 8–14 December 2019; pp. 8026–8037. [Google Scholar]
- Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
- Isola, P.; Zhu, J.Y.; Zhou, T.; Efros, A.A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1125–1134. [Google Scholar]
- Tang, H.; Xu, D.; Sebe, N.; Wang, Y.; Corso, J.J.; Yan, Y. Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 2417–2426. [Google Scholar]
- Huynh-Thu, Q.; Ghanbari, M. Scope of validity of PSNR in image/video quality assessment. Electron. Lett. 2008, 44, 800–801. [Google Scholar] [CrossRef]
- Jiang, K.; Wang, Z.; Yi, P.; Chen, C.; Huang, B.; Luo, Y.; Ma, J.; Jiang, J. Multi-scale progressive fusion network for single image deraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 8346–8355. [Google Scholar]
- Zhao, H.; Shi, J.; Qi, X.; Wang, X.; Jia, J. Pyramid scene parsing network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 2881–2890. [Google Scholar]
- Ren, S.; He, K.; Girshick, R.; Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proceedings of the Advances in Neural Information Processing Systems 28 (NIPS 2015), Montreal, QC, Canada, 7–12 December 2015; pp. 91–99. [Google Scholar]
Rainy Image | Eigen [18] | Pix2Pix [42] | SelectGAN [43] | AGAN [21] | Ours | |
---|---|---|---|---|---|---|
PSNR | 30.61 | 25.00 | 31.33 | 30.46 | 38.32 | 40.45 |
SSIM | 0.9514 | 0.9013 | 0.9302 | 0.9463 | 0.9809 | 0.9857 |
Rainy Image | Eigen [18] | Pix2Pix [42] | AGAN [21] | Ours | |
---|---|---|---|---|---|
PSNR | 21.41 | 17.64 | 21.24 | 24.43 | 25.32 |
SSIM | 0.7502 | 0.6128 | 0.6707 | 0.7975 | 0.8270 |
Module | ||||||
---|---|---|---|---|---|---|
ED | √ | √ | √ | √ | √ | √ |
MDCM | √ | √ | √ | √ | ||
DCB | √ | √ | ||||
MDCB | √ | √ | ||||
DCE | √ | |||||
PSNR | 38.45 | 39.41 | 39.63 | 39.87 | 40.12 | 40.45 |
SSIM | 0.9814 | 0.9834 | 0.9839 | 0.9846 | 0.9850 | 0.9857 |
Our MSANet | PSNR | SSIM |
---|---|---|
w/o raindrop detection | 39.91 | 0.9844 |
w/raindrop detection, w/o RFB | 40.21 | 0.9850 |
w/raindrop detection, w/RFB | 40.45 | 0.9857 |
Semantic Segmentation; Algorithm: PSPNet [46] | ||||||
---|---|---|---|---|---|---|
Rainy Image | Eigen [18] | Pix2Pix [42] | SelectGAN [43] | AGAN [21] | Ours | |
mIoU (%) | 67.1 | 57.7 | 57.6 | 66.5 | 72.3 | 73.0 |
mAcc (%) | 76.9 | 65.6 | 67.3 | 78.4 | 79.8 | 80.6 |
Object Detection; Algorithm: Faster R-CNN [47] | ||||||
Rainy Image | Eigen [18] | Pix2Pix [42] | SelectGAN [43] | AGAN [21] | Ours | |
mAP (%) | 34.9 | 26.5 | 35.1 | 37.5 | 43.4 | 43.8 |
AP (%) | 58.1 | 45.9 | 57.3 | 61.4 | 67.2 | 67.7 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Luo, H.; Wu, Q.; Ngan, K.N.; Luo, H.; Wei, H.; Li, H.; Meng, F.; Xu, L. Multi-Scale Shape Adaptive Network for Raindrop Detection and Removal from a Single Image. Sensors 2020, 20, 6733. https://doi.org/10.3390/s20236733
Luo H, Wu Q, Ngan KN, Luo H, Wei H, Li H, Meng F, Xu L. Multi-Scale Shape Adaptive Network for Raindrop Detection and Removal from a Single Image. Sensors. 2020; 20(23):6733. https://doi.org/10.3390/s20236733
Chicago/Turabian StyleLuo, Hao, Qingbo Wu, King Ngi Ngan, Hanxiao Luo, Haoran Wei, Hongliang Li, Fanman Meng, and Linfeng Xu. 2020. "Multi-Scale Shape Adaptive Network for Raindrop Detection and Removal from a Single Image" Sensors 20, no. 23: 6733. https://doi.org/10.3390/s20236733
APA StyleLuo, H., Wu, Q., Ngan, K. N., Luo, H., Wei, H., Li, H., Meng, F., & Xu, L. (2020). Multi-Scale Shape Adaptive Network for Raindrop Detection and Removal from a Single Image. Sensors, 20(23), 6733. https://doi.org/10.3390/s20236733