SaliencyAided Online RPCA for Moving Target Detection in Infrared Maritime Scenarios
Abstract
:1. Introduction
2. Theoretical Framework and Related Works
2.1. Notation and Pixel Model
2.2. RPCA for Background Subtraction
Algorithm 1: RPCA by ALM  
1  Input:

2  Initialize: ${S}_{\left(0\right)}\leftarrow 0\in {\mathbb{R}}^{{n}_{p}\times {n}_{f}}$, ${Y}_{\left(0\right)}\leftarrow 0\in {\mathbb{R}}^{{n}_{p}\times {n}_{f}}$ 
3  while not converged do 
4 

5 

6 

7  return:

 ${\mathcal{S}}_{\raisebox{1ex}{$\lambda $}\!\left/ \!\raisebox{1ex}{$\rho $}\right.}\left(Y\right)=\mathrm{s}\mathrm{g}\mathrm{n}\left(Y\right)\mathrm{m}\mathrm{a}\mathrm{x}\left(\leftY\right\frac{\lambda}{\rho},0\right)$ denotes the shrinkage operator applied on the matrix $Y$, which is the proximal operator for the l1norm minimization problem $\underset{X}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\left({\frac{\rho}{2}{\Vert XY\Vert}_{2}^{2}+\lambda \Vert \left(X\right)\Vert}_{1}\right)$[45];
 >${\mathcal{D}}_{\raisebox{1ex}{$\lambda $}\!\left/ \!\raisebox{1ex}{$\rho $}\right.}\left(Y\right)=U{\mathcal{S}}_{\raisebox{1ex}{$\lambda $}\!\left/ \!\raisebox{1ex}{$\rho $}\right.}\left(\Sigma \right){V}^{\mathrm{T}}$ denotes the singular value thresholding operator applied on the matrix $Y$, whose singular value decomposition (SVD) is $Y=U\Sigma {V}^{\mathrm{T}}$, which is the proximal operator for the nuclearnorm minimization problem $\underset{X}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\left({\frac{\rho}{2}{\Vert XY\Vert}_{2}^{2}+\lambda \Vert X\Vert}_{\ast}\right)$ [45].
2.3. Online Moving Window RPCA
Algorithm 2: Online Moving Window RPCA  
1  Input:

2  Initialize:

3  for $k=1$ to ${n}_{b}$ do 
4  ${A}_{k}\leftarrow {A}_{k1}+{\mathit{v}}_{k}{{\mathit{v}}_{k}}^{T},$ ${B}_{k}\leftarrow {B}_{k1}+\left({\mathit{x}}_{k}{\mathit{s}}_{k}\right){{\mathit{v}}_{k}}^{T}$ 
5  for $k={n}_{b}+1$ to ${n}_{f}$ do 
6 

7 

8 

9 

10  return:

 The parameter would become much more dependent on the specific input matrix $X$, while, in the practice, it is usually set as $\frac{1}{\sqrt{max\left({n}_{p},{n}_{f}\right)}}$;
 Along with the ghost pixels, a higher ${\lambda}_{2}$ would also cause erosion of target associated pixels, affecting the detection probability as well.
2.4. SaliencyAided RPCA
 inverts the polarity of each element of $P$, in the sense that a low value should address high objectness confidence, and vice versa;
 scales the resulting matrix in a wider modulation range (e.g., between 0 and 20).
Algorithm 3: Saliency aided RPCA  
1  Input:

2  Initialize: $P=\left\{{\mathit{p}}_{1},\dots ,{\mathit{p}}_{{n}_{f}}\right\}=0$ (empty matrix of size $\left({n}_{p}\times {n}_{f}\right)$) 
3  for $k=1$ to ${n}_{f}$ do 
4  Reshape ${\mathit{x}}_{k}$ in the frame form to get the matrix ${X}_{k}$ of size $\left(h\times w\right)$ 
5  Compute the saliency algorithm on the frame ${X}_{k}$ to get ${P}_{k}$ 
6  Put ${P}_{k}$ in lexicographic order to get ${\mathit{p}}_{k}$ and update $P$ 
7  while not converged do 
8 

9 

10 

11  return:

3. Proposed Method: SaliencyAided OMWRPCA
 First, Equation (9) is solved in $\left({\mathit{x}}_{k},{U}_{k1},{\mathit{p}}_{k}\right)$, to find ${\mathit{v}}_{k}$ and ${\mathit{s}}_{k}$;
 Then, $U$ is updated by blockcoordinate descent with warm restart.
Algorithm 4: Saliency aided OMWRPCA  
1  Input:

2  Initialize:

3  for $k=1$ to ${n}_{b}$ do 
4  ${A}_{k}\leftarrow {A}_{k1}+{\mathit{v}}_{k}{{\mathit{v}}_{k}}^{T}$, ${B}_{k}\leftarrow {B}_{k1}+\left({\mathit{x}}_{k}{\mathit{s}}_{k}\right){{\mathit{v}}_{k}}^{T}$ 
5  for $k=1$ to ${n}_{f}$ do 
6 

7 

8 

9 

10 

11 

13 

14  return:

4. Results
4.1. Dataset and Qualitative Evaluation
 The kayak is a very small target which covers 0.17% of the whole picture. The speed is such that the average permanence of the target on a single pixel is about 2 s. The background is quite hot and the waves on the sea are particularly evident.
 The speed boat is medium size and covers 0.72% of the picture. The average permanence is about 10 s. The background is colder, and the waves are less evident but still present.
 The sailing ship is an extended target which covers 2.21% of the picture. The average permanence is about 3 s. The background presents some hot spots near the horizon, while the sea is calm.
 After some experiments, we noted that the ghosts of the target were effectively deleted by increasing ${\lambda}_{2}$ by a factor higher than 10;
 The original dynamic of the saliency map is in the range $\left[0,1\right]$ and we can assume that the lowest values (i.e., $\left[0,3.3\right]$) address the background, the middle values (i.e., $\left[3.3,6.6\right]$) indicate an uncertainty area, while the highest values (i.e., $\left[6.6,1\right]$) address the targets.
4.2. PrecisionRecall Curves
4.3. Execution Time
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
 Prasad, D.K.; Rajan, D.; Rachmawati, L.; Rajabally, E.; Quek, C. Video Processing from ElectroOptical Sensors for Object Detection and Tracking in a Maritime Environment: A Survey. IEEE Trans. Intell. Transp. Syst. 2017, 18, 1993–2016. [Google Scholar] [CrossRef] [Green Version]
 Chapel, M.N.; Bouwmans, T. Moving Objects Detection with a Moving Camera: A Comprehensive Review. Comput. Sci. Rev. 2020, 38, 100310. [Google Scholar] [CrossRef]
 Acito, N.; Corsini, G.; Diani, M.; Pennucci, G. Comparative Analysis of Clutter Removal Techniques over Experimental IR Images. Opt. Eng. 2005, 44, 106401. [Google Scholar] [CrossRef]
 Cheng, M.M.; Mitra, N.J.; Huang, X.; Torr, P.H.S.; Hu, S.M. Global Contrast Based Salient Region Detection. IEEE Trans. Pattern Anal. Mach. Intell. 2015, 37, 569–582. [Google Scholar] [CrossRef] [Green Version]
 Pulpito, O.; Acito, N.; Diani, M.; Ceglie, S.U.D.; Corsini, G. Infrared Saliency Enhancement Techniques for Extended Naval Target Detection in Open Sea Scenario. In Proceedings of the ElectroOptical Remote Sensing XVI, Berlin, Germany, 5–8 September 2022; SPIE: Bellingham, WA, USA, 2022; Volume 12272, pp. 18–27. [Google Scholar]
 Itti, L.; Koch, C.; Niebur, E. A Model of SaliencyBased Visual Attention for Rapid Scene Analysis. IEEE Trans. Pattern Anal. Mach. Intell. 1998, 20, 1254–1259. [Google Scholar] [CrossRef] [Green Version]
 Montabone, S.; Soto, A. Human Detection Using a Mobile Platform and Novel Features Derived from a Visual Saliency Mechanism. Image Vis. Comput. 2010, 28, 391–402. [Google Scholar] [CrossRef]
 Hou, X.; Zhang, L. Saliency Detection: A Spectral Residual Approach. In Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 17–22 June 2007; pp. 1–8. [Google Scholar]
 Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You Only Look Once: Unified, RealTime Object Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 779–788. [Google Scholar]
 Hu, W.C.; Yang, C.Y.; Huang, D.Y. Robust RealTime Ship Detection and Tracking for Visual Surveillance of Cage Aquaculture. J. Vis. Commun. Image Represent. 2011, 22, 543–556. [Google Scholar] [CrossRef]
 Smith, A.A.; Teal, M.K. Identification and Tracking of Maritime Objects in NearInfrared Image Sequences for Collision Avoidance. In Proceedings of the 7th International Conference on Image Processing and Its Applications, Manchester, UK, 13–15 July 1999; pp. 250–254. [Google Scholar] [CrossRef]
 Sang, N.; Zhang, T. Segmentation of FLIR Images by Target Enhancement and Image Model. In Proceedings of the International Symposium on Multispectral Image Processing (ISMIP’98), Wuhan, China, 21–23 October 1998; SPIE: Bellingham, WA, USA, 1998; Volume 3545, pp. 274–277. [Google Scholar]
 Barnett, J. Statistical Analysis of Median Subtraction Filtering with Application To Point Target Detection In Infrared Backgrounds. In Proceedings of the Infrared Systems and Components III, Los Angeles, CA, USA, 16–17 January 1989; SPIE: Bellingham, WA, USA, 1989; Volume 1050, pp. 10–18. [Google Scholar]
 Voles, P.; Teal, M.; Sanderson, J. Target Identification in a Complex Maritime Scene. In Proceedings of the IEE Colloquium on Motion Analysis and Tracking (Ref. No. 1999/103), London, UK, 10 May 1999; p. 15. [Google Scholar] [CrossRef]
 Pulpito, O.; Acito, N.; Diani, M.; Corsini, G.; Grasso, R.; Ferri, G.; Grati, A.; Le Page, K.; Bereta, K.; Zissis, D. RealTime Moving Target Detection in Infrared Maritime Scenarios. In Proceedings of the 2022 IEEE International Workshop on Metrology for the Sea; Learning to Measure Sea Health Parameters (MetroSea), Milazzo, Italy, 3–5 October 2022; pp. 456–461. [Google Scholar]
 Wren, C.R.; Azarbayejani, A.; Darrell, T.; Pentland, A.P. Pfinder: RealTime Tracking of the Human Body. IEEE Trans. Pattern Anal. Mach. Intell. 1997, 19, 780–785. [Google Scholar] [CrossRef] [Green Version]
 Gupta, K.M.; Aha, D.W.; Hartley, R.; Moore, P.G. Adaptive Maritime Video Surveillance. SPIE Proc. 2009, 7346, 81–92. [Google Scholar]
 Zivkovic, Z. Improved Adaptive Gaussian Mixture Model for Background Subtraction. In Proceedings of the 17th International Conference on Pattern Recognition, 2004 (ICPR 2004), Cambridge, UK, 26 August 2004; Volume 2, pp. 28–31. [Google Scholar]
 Chen, Z.; Wang, X.; Sun, Z.; Wang, Z. Motion Saliency Detection Using a Temporal Fourier Transform. Opt. Laser Technol. 2016, 80, 1–15. [Google Scholar] [CrossRef]
 Strickland, R.N.; Hahn, H.I. Wavelet Transform Methods for Object Detection and Recovery. IEEE Trans. Image Process. 1997, 6, 724–735. [Google Scholar] [CrossRef] [PubMed] [Green Version]
 Infrared Image Segmentation by Combining Fractal Geometry with Wavelet Transformation. Available online: https://www.sensorsportal.com/HTML/DIGEST/P_2529.htm (accessed on 30 January 2023).
 Seki, M.; Wada, T.; Fujiwara, H.; Sumi, K. Background Subtraction Based on Cooccurrence of Image Variations. In Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Madison, WI, USA, 18–20 June 2003; Volume 2, p. II. [Google Scholar]
 Power, P.; Schoonees, J. Understanding Background Mixture Models for Foreground Segmentation. In Proceedings of the Image and Vision Computing New Zealand 2002, Auckland, New Zealand, 26–28 November 2002. [Google Scholar]
 Oliver, N.M.; Rosario, B.; Pentland, A.P. A Bayesian Computer Vision System for Modeling Human Interactions. IEEE Trans. Pattern Anal. Mach. Intell. 2000, 22, 831–843. [Google Scholar] [CrossRef] [Green Version]
 Candès, E.J.; Li, X.; Ma, Y.; Wright, J. Robust Principal Component Analysis? J. ACM 2011, 58, 1–37. [Google Scholar] [CrossRef]
 De la Torre, F.; Black, M.J. Robust Principal Component Analysis for Computer Vision. In Proceedings of the 8th IEEE International Conference on Computer Vision, ICCV 2001, Vancouver, BC, Canada, 7–14 July 2001; Volume 1, pp. 362–369. [Google Scholar]
 Cao, X.; Yang, L.; Guo, X. Total Variation Regularized RPCA for Irregularly Moving Object Detection Under Dynamic Background. IEEE Trans. Cybern. 2016, 46, 1014–1027. [Google Scholar] [CrossRef]
 Li, Z.; Wang, Y.; Zhao, Q.; Zhang, S.; Meng, D. A TensorBased Online RPCA Model for Compressive Background Subtraction. IEEE Trans. Neural Netw. Learn. Syst. 2022, 1–15, early access. [Google Scholar] [CrossRef]
 Zhang, J.; Zhou, D.; Chen, M. Adaptive Cointegration Analysis and Modified RPCA With Continual Learning Ability for Monitoring Multimode Nonstationary Processes. IEEE Trans. Cybern. 2022, 1–14, early access. [Google Scholar] [CrossRef] [PubMed]
 Yang, Y.; Yang, Z.; Li, J. Novel RPCA with Nonconvex Logarithm and Truncated Fraction Norms for Moving Object Detection. Digit. Signal Process. 2023, 133, 103892. [Google Scholar] [CrossRef]
 Ablavsky, V. Background Models for Tracking Objects in Water. In Proceedings of the 2003 International Conference on Image Processing (Cat. No.03CH37429), Barcelona, Spain, 14–17 September 2003; Volume 3, p. III125. [Google Scholar]
 Schofield, A.J.; Mehta, P.A.; Stonham, T.J. A System for Counting People in Video Images Using Neural Networks to Identify the Background Scene. Pattern Recognit. 1996, 29, 1421–1428. [Google Scholar] [CrossRef]
 Lim, L.A.; Yalim Keles, H. Foreground Segmentation Using Convolutional Neural Networks for Multiscale Feature Encoding. Pattern Recognit. Lett. 2018, 112, 256–262. [Google Scholar] [CrossRef] [Green Version]
 Maddalena, L.; Petrosino, A. A SelfOrganizing Approach to Detection of Moving Patterns for RealTime Applications. In Proceedings of the Advances in Brain, Vision, and Artificial Intelligence, Naples, Italy, 10–12 October 2007; Mele, F., Ramella, G., Santillo, S., Ventriglia, F., Eds.; Springer: Berlin/Heidelberg, Germany, 2007; pp. 181–190. [Google Scholar]
 Lim, K.; Jang, W.D.; Kim, C.S. Background Subtraction Using EncoderDecoder Structured Convolutional Neural Network. In Proceedings of the 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Lecce, Italy, 29 August–1 September 2017; pp. 1–6. [Google Scholar]
 Sharma, T.; Debaque, B.; Duclos, N.; Chehri, A.; Kinder, B.; Fortier, P. Deep LearningBased Object Detection and Scene Perception under Bad Weather Conditions. Electronics 2022, 11, 563. [Google Scholar] [CrossRef]
 Yang, J.; Liu, S.; Li, Z.; Li, X.; Sun, J. RealTime Object Detection for Streaming Perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 5385–5395. [Google Scholar]
 He, J.; Balzano, L.; Lui, J.C.S. Online Robust Subspace Tracking from Partial Information. arXiv 2011, arXiv:1109.3827. [Google Scholar]
 Feng, J.; Xu, H.; Yan, S. Online Robust PCA via Stochastic Optimization. In Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NE, USA, 5–8 December 2013; Curran Associates, Inc.: Red Hook, NY, USA, 2013; Volume 26. [Google Scholar]
 Xiao, W.; Huang, X.; He, F.; Silva, J.; Emrani, S.; Chaudhuri, A. Online Robust Principal Component Analysis with Change Point Detection. IEEE Trans. Multimed. 2020, 22, 59–68. [Google Scholar] [CrossRef] [Green Version]
 Oreifej, O.; Li, X.; Shah, M. Simultaneous Video Stabilization and Moving Object Detection in Turbulence. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 450–462. [Google Scholar] [CrossRef] [PubMed] [Green Version]
 Ferri, G.; Grasso, R.; Faggiani, A.; de Rosa, F.; Camossi, E.; Grati, A.; Stinco, P.; Tesei, A.; Been, R.; LePage, K.D.; et al. A Hybrid Robotic Network for Maritime Situational Awareness: Results from the INFORE22 Sea Trial. In Proceedings of the OCEANS 2022, Hampton Roads, VA, USA, 17–20 October 2022; pp. 1–10. [Google Scholar]
 Pearson, K. LIII. On Lines and Planes of Closest Fit to Systems of Points in Space. Lond. Edinb. Dublin Philos. Mag. J. Sci. 1901, 2, 559–572. [Google Scholar] [CrossRef] [Green Version]
 Brunton, S.L.; Kutz, J.N. DataDriven Science and Engineering: Machine Learning, Dynamical Systems, and Control; Cambridge University Press: Cambridge, UK, 2019. [Google Scholar]
 Parikh, N.; Boyd, S. Proximal Algorithms. Found. Trends Optim. 2014, 1, 127–239. [Google Scholar] [CrossRef]
 Burer, S.; Monteiro, R.D.C. A Nonlinear Programming Algorithm for Solving Semidefinite Programs via LowRank Factorization. Math. Program. Ser. B 2003, 95, 329–357. [Google Scholar] [CrossRef]
 Recht, B.; Fazel, M.; Parrilo, P.A. Guaranteed MinimumRank Solutions of Linear Matrix Equations via Nuclear Norm Minimization. SIAM Rev. 2010, 52, 471–501. [Google Scholar] [CrossRef] [Green Version]
 Sobral, A.; Bouwmans, T.; ZahZah, E. DoubleConstrained RPCA Based on Saliency Maps for Foreground Detection in Automated Maritime Surveillance. In Proceedings of the 2015 12th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Karlsruhe, Germany, 25–28 August 2015; pp. 1–6. [Google Scholar]
 Li, Y.; Liu, G.; Liu, Q.; Sun, Y.; Chen, S. Moving Object Detection via Segmentation and Saliency Constrained RPCA. Neurocomputing 2019, 323, 352–362. [Google Scholar] [CrossRef]
Detector Type  MWIR Indium Antimonide (InSb) or Mercury Cadmium Telluride (MCT) 

Spectral range  3 to 5 μm 
FOV  1.1(H)° × 0.84°(V) to 14.1°(H) × 10.5°(V) 
IFOV  0.383 mrad to 0.031 mrad 
Array format  640 × 512 pixel 
Frame rate  25 fps 
Thermal sensitivity  25 mK 
$\mu =\frac{1}{4\ast \mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}\left(\leftX\right\right)}$ (Default [40]) 
$\lambda =\frac{1}{\sqrt{\mathrm{m}\mathrm{a}\mathrm{x}\left({n}_{p},{n}_{f}\right)}}$ (Default [40]) 
${\lambda}_{1}=\frac{1}{\sqrt{\mathrm{m}\mathrm{a}\mathrm{x}\left({n}_{p},{n}_{f}\right)}}$ (Default [40]) 
${\lambda}_{2}=\frac{1}{\sqrt{\mathrm{m}\mathrm{a}\mathrm{x}\left({n}_{p},{n}_{f}\right)}}$ (Default [40]) 
${n}_{b}=50$ 
$\alpha =5$ 
$\beta =20$ 
Kayak  Speed Boat  Sailing Ship  Inflatable Boat  Speed Boat 2  Fishing Boat  Average  

GMM  0.67  0.77  0.87  0.65  0.74  0.93  0.78 
OMWRPCA  0.47  0.66  0.67  0.24  0.36  0.47  0.49 
SOMWRPCA with FG  0.77  0.71  0.82  0.52  0.40  0.53  0.63 
SOMWRPCA with SR  0.85  0.86  0.89  0.59  0.64  0.91  0.80 
Processor  Intel CPU Core I911900 

RAM  4 × 16 GB 3600 MHz 
GPU  NO 
Kayak  Speed Boat  Sailing Ship  Inflatable Boat  Speed Boat 2  Fishing Boat  

Burnin phase  OMWRPCA  0.0249  0.0208  0.0558  0.0235  0.0342  0.0258 
SOMWRPCA with SR  0.0072  0.0032  0.0240  0.0117  0.0196  0.0099  
SOMWRPCA with FG  0.0074  0.0033  0.0277  0.0112  0.0232  0.0101  
Online phase  OMWRPCA  0.0901  0.071  0.0754  0.0699  0.0573  0.0749 
SOMWRPCA with SR  0.1942  0.1944  0.1954  0.1977  0.192  0.1963  
SOMWRPCA with FG  0.1954  0.1939  0.1959  0.1952  0.1946  0.1978 
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. 
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Pulpito, O.; Acito, N.; Diani, M.; Ferri, G.; Grasso, R.; Zissis, D. SaliencyAided Online RPCA for Moving Target Detection in Infrared Maritime Scenarios. Sensors 2023, 23, 6334. https://doi.org/10.3390/s23146334
Pulpito O, Acito N, Diani M, Ferri G, Grasso R, Zissis D. SaliencyAided Online RPCA for Moving Target Detection in Infrared Maritime Scenarios. Sensors. 2023; 23(14):6334. https://doi.org/10.3390/s23146334
Chicago/Turabian StylePulpito, Osvaldo, Nicola Acito, Marco Diani, Gabriele Ferri, Raffaele Grasso, and Dimitris Zissis. 2023. "SaliencyAided Online RPCA for Moving Target Detection in Infrared Maritime Scenarios" Sensors 23, no. 14: 6334. https://doi.org/10.3390/s23146334