Review Reports
by
- Landan Zhang 1 and
- Zhenming Peng 1,2,*
Reviewer 1: Anonymous Reviewer 2: Anonymous Reviewer 3: Anonymous
Round 1
Reviewer 1 Report
The problem addressed by this paper is the detection of small targets in infrared images. It is a challenging problem given the very different backgrounds that can be observed. The core of the algorithm is a decomposition of the image as a sum of two images, one containing the background and the other the target. The input image is transformed to a tensor encoding local and non-local correlations, and rank-related priors are used to do the decomposition. Part of the optimization is an ADMM algorithm. The method is tested on 24 infrared images, some very difficult even for a human observer, and compared to 9 other methods along several criteria. The results show an excellent performance of the algorithm.
This is a strong paper, with detailed technical explanations, extensive citations, insightful discussions, and convincing experiments.
The main drawback is that the difference with the best competing method, RIPT, is not quite clear from reading the paper. Some weights are treated differently, but it is probably not the only difference, and the original methodological contribution of the article could be better highlighted.
The discussion of the other methods and of the results of the algorithm is very rich and extensive, but it is easy to get lost in all the different acronyms. Table 2 recapitulates those for the competing methods (but their reference should be recalled in the table), but many other acronyms are present: PSTNN, PSVT, RPCA... I suggest the addition of a table of acronyms.
The way to transform the image into a tensor should come earlier in the paper. This is only explained in the experimental section, which lets the reader wonder until then where the tensor comes from.
In step 2 of Algorithm 1, A is actually X.
In Figure 5, the result of the proposed modified weights by Equation (16) could be shown for comparison. Also, the caption of this Figure should remind that (b) relates to (13) and (c) to (14).
The technical content is sound and the explanations are interesting. There are still a few mistakes.
The solution to (4) should be X = S_\tau(Y), and there is a missing 0 in the max operator in (5).
When (6) is introduced, the PSSV pseudo-norm is not et introduced, so the solution does not really make sense for the reader.
In Algorithm 2, the update of B involves the P operator. It should remind the reader that this operator is defined in (7). In Equations (24) and (25), the subscripts k for B and T are missing.
I find the paper quite long and its attentive reading is an undertaking enterprise, but I do not have any specific suggestion on how to shorten it.
The English usage could be improved, though it does not impend the comprehension. Some of the required corrections follow, but a complete proofreading should be performed:
- 16: "was proposed" -> "is proposed"
- 21: "of A reweighting scheme"
- 48: "are difficult" -> "are unable"
- 54: "THE most"
- 60: "It's": many contractions are used in the paper, this is not proper usage for a scientific paper.
- 65: "utilized" -> "using" (twice)
- 80: "from the Fourier domain" -> "in the Fourier domain"
- 96: "assumption THAT"
- 100: "is that the imbalance" -> "is the imbalance"
- 113: "corelationships" -> "correlations"
- 117: "resulting from that" -> "due to the fact that"
- 120: "as illustrated" -> "is illustrated"
- 129: "superiority"
- 154: the last one, frontal slices, should be X_::i
- 157: "define THE operator unfold THAT maps"
- 189: "Overall given" -> "Overall"
- 201: "solve aN RPCA"
- 218: "could be seemed" -> "could be seen"
- 228 "number of small targetS"
- 244: "lies IN THE FACT THAT"
- 248: "The two HIGHEST eigenvalues"
- 277: strictly speaking, w is *half* the harmonic mean.
- 303: "high correlations"
- 313: "than SNN" -> "compared to SNN"
- 324: "an adaptiveLY predicted"
- 324: "in consideration of" -> "considering that"
- 333: "represents all the elements of W are the reciprocal" -> "is the tensor corresponding to elementwise reciprocals"
- 338: "many literatures" -> "many publications"
- 340: "of THE whole procedure"
- 348: "is WHEN THE relative error"
- 354: "higher accuracy" -> than what?
- 382: "as the same as in" -> "as in"
- 386: "ARE conducted"
- 397: "placing" -> "placed"
- 398: "from THE twenty-four"
- 401: "to access [...] quantificationally" -> "to quantify"
- 512: "size of THE small target"
- 553: "remainder" -> "the remaining"
Author Response
Response to Reviewer 1 Comments
Point 1: The main drawback is that the difference with the best competing method, RIPT, is not quite clear from reading the paper. Some weights are treated differently, but it is probably not the only difference, and the original methodological contribution of the article could be better highlighted.
Response 1: I am very grateful to your comments for the manuscript. Your suggestion is very useful for improving the paper. Of course, prior map is not the only difference, the surrogate of tensor rank is another main difference, which is described in subsection 3.3.1. More specifically, in RIPT model, the authors utilized the sum of nuclear norms (SNN) (which is convex) to relax the non-smooth and discrete tensor rank; while in the proposed model, we utilize the partial sum of the tensor nuclear norm (PSTNN) (which is non-convex) to replace the tensor rank. Furthermore, the difference between SNN and PSTNN leads to different solution ideas. RIPT is carried out based on the flattening matrices along three modes, while our method performs mainly relying on tensor singular value decomposition (t-SVD) in Fourier transform domain. But according with your advice, we still amend the relevant part in manuscript to better highlight the difference.
Point 2: The discussion of the other methods and of the results of the algorithm is very rich and extensive, but it is easy to get lost in all the different acronyms. Table 2 recapitulates those for the competing methods (but their reference should be recalled in the table), but many other acronyms are present: PSTNN, PSVT, RPCA... I suggest the addition of a table of acronyms.
Response 2: Thank you for pointing this out. We have added a table of acronyms to the paper in Section 2.
Point 3: The way to transform the image into a tensor should come earlier in the paper. This is only explained in the experimental section, which lets the reader wonder until then where the tensor comes from.
Response 3: Thank you for your suggestion. In fact, we had already mentioned the way to transform the image into a tensor at the beginning of subsection 3.1. Considering your suggestion, we revised the corresponding content and added an illustrative figure.
Point 4: In step 2 of Algorithm 1, A is actually X.
Response 4: We are very sorry for our carelessness. We have made correction according to your comments.
Point 5: In Figure 5, the result of the proposed modified weights by Equation (16) could be shown for comparison. Also, the caption of this Figure should remind that (b) relates to (13) and (c) to (14).
Response 5: Thank you for your comments. We have made corresponding modifications and additions. Now, Figure 5 has changed to Figure 6.
Point 6: The technical content is sound and the explanations are interesting. There are still a few mistakes. The solution to (4) should be X = S_\tau(Y), and there is a missing 0 in the max operator in (5). When (6) is introduced, the PSSV pseudo-norm is not yet introduced, so the solution does not really make sense for the reader.
Response 6: We are very sorry for our carelessness. We have made correction according to your comments. Furthermore, we have added Definition 5 in subsection 2.2 to introduce the PSSV pseudo-norm.
Point 7: In Algorithm 2, the update of B involves the P operator. It should remind the reader that this operator is defined in (7). In Equations (24) and (25), the subscripts k for B and T are missing.
Response 7: Considering your suggestion, we have added a hint in Algorithm 2 that the operator P is defined in Equation (7). It is worth noting that B and T in Equations (24) and (25) don’t have superscripts k because they are unknown variables to be solved.
Point 8: I find the paper quite long and its attentive reading is an undertaking enterprise, but I do not have any specific suggestion on how to shorten it.
Response 8: Thank you for this suggestion! We have tried to shorten our paper.
Point 9: The English usage could be improved, though it does not impend the comprehension. Some of the required corrections follow, but a complete proofreading should be performed.
Response 9: Thank you very much for pointing this out. We have revised them according to your suggestion and highlighted them in the paper. Furthermore, the manuscript is carefully edited now.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
This manuscript presented an infrared small target detection based on partial sum of the tensor nuclear norm method. It can be observed the proposed method has obtained better detection results and the algorithm is efficient theoretically.
But unfortunately, the description is a little hard to understand, and just from the description of this whole methodology, the proposed method should be time-consuming and have a huge computational burden, especially in the process of tensor decomposition.
In addition, how to determine the values of parameters is also a big task and may be sensitive to the final results. Please give more information and discuss.
Besides, the type of noise in reality maybe a little complex and may not be easy to remove as Gaussian type. If the contamination noise is just Gaussian white noise or salt-pepper noise, how to guarantee the infrared small target still being there without any harm?
Last but not least, I still have a question. How about using some special unmixing method if these infrared small targets are subpixel scales? Does it make sense?
Author Response
Response to Reviewer 2 Comments
Point 1: But unfortunately, the description is a little hard to understand, and just from the description of this whole methodology, the proposed method should be time-consuming and have a huge computational burden, especially in the process of tensor decomposition.
Response 1: Thank you very much for your review report. These comments are all valuable and very helpful for revising and improving our paper, as well as the important guiding significance to our researches. We have studied comments carefully and our answer to this point is as follows:
Background (patch-image) Low-rankness and target (patch-image) sparsity are two basic assumptions of our method, which are commonly used in the field of infrared small target detection. Firstly, we have to explain what a patch image is. Traversing an image with a sliding window from top left to bottom right, patch image is constructed by vectorizing the matrix within the sliding window, as shown in Figure 1 below. Secondly, the reason why background patch-image is low-rank is that the infrared background is regarded as slowly transitional, that means many local patches are approximately linearly correlated with each other. When vectorizing these patches into vectors, the obtained background patch-image is obviously low-rank. Thirdly, target sparsity is well understood because the small target only occupies a few pixels with respect to the whole image. The similar explanation can be found at the beginning of Section 3.
Figure 1. Illustration of patch image construction.
On the whole, an infrared image with small target can be regarded as
(1) |
where , , , are the original image, the background image, the target image and the noise image, respectively. To utilize the above two properties, Gao et al. [1] presented a more general model to describe original infrared image named infrared patch-image (IPI) model with the form of :
(2) |
where , , and are the corresponding patch-images of the original infrared image, the background image, the target image and the random noise image, respectively. Then, with the low-rank property of and the sparsity of , the conventional small target detection problem is converted to a robust principle component analysis (RPCA) optimization problem (RPCA is also called sparse and low rank matrix decomposition) as follows:
(3) |
where denotes the nuclear norm of a matrix (i.e. the sum of singular values), is the l1 norm (i.e. ), and is a positive weight constant.
IPI model shows superiority in comparison with conventional methods of infrared small target detection, but still exists some drawbacks, namely, background residuals in target image and target over-shrinking. To dig out more correlations among different patches and overcome the defects of IPI model, Dai et al. [2] generalized IPI model into the infrared patch-tensor (IPT) model by directly stacking the patches into a 3-D cube, as shown in Figure 2 below. Moreover, IPI model could be viewed as a special case of the IPT model, as the patch-image is just the mode-3 flattening matrix of the corresponding patch-tensor.
Figure 2. Illustration of patch tensor construction.
Hence, Equation (2) is transferred to the patch space:
(4) |
where , , , are the input patch-tensor, background patch-tensor, target patch-tensor , and noise patch-tensor, respectively. m and n are the patch height and width, k is the patch number. Then, on the basis of IPT model, we can obtain the following tensor robust principle component analysis (TRPCA) problem which attempts to separate the low-rank and sparse tensors (i.e. background tensor and target tensor respectively):
(5) |
where is a compromising parameter which controls the tradeoff between the target patch-tensor and the background patch-tensor, denotes l0 norm, which counts the number of nonzero entries. The detailed description can be found in subsection 3.1 in our paper.
To further improve the detecting performance of IPT model, the authors utilized reweighted scheme to enhance the target sparsity, and that is RIPT method. As we analyzed in subsections 1.2, 3.2 and 3.3.1, RIPT has two major defects that (i) sum of nuclear norms (SNN) would lead to a sub-optimal solution, and (ii) target over-shrinking, or even target totally lost. Figure 11 in our paper can also demonstrates this.
So, let’s come back to the analysis of the proposed method. Our method is on the basis of IPT model, and is designed to achieve better detection results. The main differences between our method and RIPT is that (i) we utilize the non-convex partial sum of the tensor nuclear norm (PSTNN) to replace the convex SNN, and (ii) we employ the target-related-prior and background-related-prior simultaneously to extract useful information from original image (detailed description is in Section 3). Finally, the objective function to be solved becomes
(6) |
where is the weight tensor related to priors, denotes the Hadamard product, and is the partial sum of the tensor nuclear norm. Then Equation (6) is solved by alternating direction method of multipliers (ADMM). Once we obtain the , tensor is first restored to the target matrix T; then, small targets are detected easily via performing adaptive threshold segmentation on T. This is the origin and construction process of the proposed method.
As for the computational burden of whole procedure, just as you said, the proposed method should be time-consuming and have a huge computational burden, but several key points are used to reduce the algorithm computational time. First, we utilize the accelerated version of tensor singular value decomposition (t-SVD) proposed in [3], which reduces computing time by half. Meanwhile, t-SVD is performed in Fourier transform domain, compared with CP decomposition, Tucker decomposition, and performing SVD in flattening matrix along each mode, this is much faster. Secondly, even though SVD is a time-consuming step, the patch size and sliding step used in our model are 40×40 and 40, respectively. In other words, the time cost of SVD on each frontal slice is actually not high. Thirdly, as we pointed out in subsection 3.3.2, we utilize an extra stop condition, namely, the algorithm stops iteration once the number of nonzero entries ceases to decrease. Because we observe that the number of nonzero entries in target patch-tensor stops changing after a few iterations, which is just little proportion of the entire procedure if the stop condition is only relative error smaller than a given threshold. For instance, this is a typical trend of the number of non-zero elements in the target tensor: 961, 21, 20, 19, 19, 19, 19…. If we don't stop iterating, the number of iterations will be 140. But in the fourth iteration of the algorithm, the target tensor has not changed. Therefore, the extra stop condition can reduce the computational time dramatically. With these key points, the actual computational burden of whole algorithm is not high.
References:
[1] Gao, Chenqiang, et al. "Infrared patch-image model for small target detection in a single image." IEEE Trans. Image Processing 22.12 (2013): 4996-5009.
[2] Dai, Yimian, and Yiquan Wu. "Reweighted infrared patch-tensor model with both nonlocal and local priors for single-frame small target detection." IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 10.8 (2017): 3752-3767.
[3] Kilmer, Misha E., and Carla D. Martin. "Factorization strategies for third-order tensors." Linear Algebra and its Applications 435.3 (2011): 641-658.
Point 2: In addition, how to determine the values of parameters is also a big task and may be sensitive to the final results. Please give more information and discuss.
Response 2: Thank you for pointing this out. Indeed, the values of parameters play an important role in whether the detection performance is good or not. So in subsection 4.3, we have already discussed the selection of three most vital parameters, and corresponding experiments have been carried out to determine proper values. At beginning, according to the Ref. [1], the way to determine the value of is much easy, thus we don’t give experiments to find the best value. Now, we have added experiments to determine in subsection 4.3.
References:
[1] Lu, Canyi, et al. "Tensor robust principal component analysis: Exact recovery of corrupted low-rank tensors via convex optimization." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.
Point 3: The type of noise in reality maybe a little complex and may not be easy to remove as Gaussian type. If the contamination noise is just Gaussian white noise or salt-pepper noise, how to guarantee the infrared small target still being there without any harm?
Response 3: Based on the experimental results in subsection 4.4.2, we can arrive at a conclusion that whatever the type of noise is, as long as the target in contaminative image can maintain a weak contrast like Figures 12(a)-(f), the proposed method is still able to detect the small target. And this is the so-called the ability to be robust to noise (in many papers). To be honest, if the noise is so dense that the target is totally overwhelmed, our method would also fail. I think in that case, there is almost no method to be effective. In other words, under the premise that the target is still salient, our method can work even though the contamination noise is not just Gaussian white noise or salt-pepper noise. Meanwhile, it should be noticed that the target shape would be harmed to some extent.
As shown in Figure 3 below, the infrared image in Figure 3(a) is polluted by Gaussian noise and Poisson noise, the infrared image in Figure 3(c) is polluted by three types of noise, including Gaussian noise, Poisson noise and multiplicative noise. The separated target images (Figure 3(b) and 3(d)) confirm the aforementioned explanation.
(a) (b) (c) (d)
Figure 3. Illustration of obtained target images when dealing with different noise.
Point 4: How about using some special unmixing method if these infrared small targets are subpixel scales? Does it make sense?
Response 4: They are not subpixel scales, and the targets detected in our experiments are generally larger than 2 × 2 pixels. (Please see the figures and equations in the PDF file)
Author Response File: Author Response.pdf
Reviewer 3 Report
Under some mild conditions, the nuclear norm is provable equvilent to F-norm in Connections Between Nuclear Norm and Frobenius Norm Based Representation which is with lower complexsity. So, is it possible to replace the nuclear norm in your method by F-norm? Moreover, some recent developments are also encouraged to review, which are close to this work, Structured AutoEncoders for Subspace Clustering; Multiple Marginal Fisher Analysis; Embedding Structured Contour and Location Prior in Siamesed Fully Convolutional Networks for Road Detection.
Author Response
Response to Reviewer 3 Comments
Point 1: Under some mild conditions, the nuclear norm is provable equvilent to F-norm in Connections Between Nuclear Norm and Frobenius Norm Based Representation which is with lower complexity. So, is it possible to replace the nuclear norm in your method by F-norm?
Response 1: I am very grateful to your comments for the manuscript. Your comments are all valuable and very helpful for revising and improving our paper. Recently, more and more infrared small target detection methods focus on the separation of background (low-rank matrix/tensor) and target (sparse matrix/tensor) [1-7]. The basic idea of this type of methods is based on robust principle component analysis (RPCA). Although many surrogates of rank function are not nuclear norms, I think they can be classified as nuclear-norm based representation (NNR) methods. For example, in Ref. [1], the objective function is
where D, B, and T are original image, background image, and target image, respectively, denotes the intensity of noise. In Ref. [2], the objective function is
where is the weighted nuclear norm, is the weighted l1 norm. It’s obvious that they can be viewed as NNR methods.
Hence, our research is mainly on the improvement of existing NNR methods, and we haven't studied Frobenius-norm-based representation (FNR) yet. We don’t sure whether F-norm could replace the nuclear norm in our method or not. Furthermore, as far as we know, FNR is not introduced into the field of infrared small target detection. Considering your comments, we consulted the relevant references, such as Ref. [8]. It’s so interesting and instructive, and we think FNR is a very good research direction. Just as you mentioned, FNR has lower complexity which is very important. We will study this issue in the future.
References:
[1] Gao, Chenqiang, et al. "Infrared patch-image model for small target detection in a single image." IEEE Trans. Image Processing 22.12 (2013): 4996-5009.
[2] Guo, Jun, Yiquan Wu, and Yimian Dai. "Small target detection based on reweighted infrared patch-image model." IET Image Processing 12.1 (2017): 70-79.
[3] Dai, Yimian, et al. "Non-negative infrared patch-image model: Robust target-background separation via partial sum minimization of singular values." Infrared Physics & Technology 81 (2017): 182-194.
[4] Wang, Xiaoyang, et al. "Infrared dim target detection based on total variation regularization and principal component pursuit." Image and Vision Computing 63 (2017): 1-9.
[5] Dai, Yimian, Yiquan Wu, and Yu Song. "Infrared small target and background separation via column-wise weighted robust principal component analysis." Infrared Physics & Technology 77 (2016): 421-430.
[6] Dai, Yimian, and Yiquan Wu. "Reweighted infrared patch-tensor model with both nonlocal and local priors for single-frame small target detection." IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 10.8 (2017): 3752-3767.
[7] Zhang, Landan, et al. "Infrared Small Target Detection via Non-Convex Rank Approximation Minimization Joint l2, 1 Norm." Remote Sensing 10.11 (2018): 1821.
[8] Peng, Xi, et al. "Connections between nuclear-norm and frobenius-norm-based representations." IEEE transactions on neural networks and learning systems 29.1 (2018): 218-224.
Point 2: Moreover, some recent developments are also encouraged to review, which are close to this work, Structured autoEncoders for Subspace Clustering; Multiple Marginal Fisher Analysis; Embedding Structured Contour and Location Prior in Siamesed Fully Convolutional Networks for Road Detection.
Response 2: Thank you very much for this suggestion! We have read these references carefully, and they really broaden our horizons. Generally speaking, infrared small target detection can be regarded as a binary classification problem, with some pixels belonging to the background and the remaining pixels belonging to the target. However, due to the lack of obvious texture and shape features, it is difficult to detect a target by directly classifying pixels in an infrared image using a classifier so far. Meanwhile, due to the complexity of the scene, it is hard to directly determine whether a pixel belongs to the background or target. That is, pixel-level approaches (classifier) are not applicable in the field of infrared small target detection. In addition, hard-to-get infrared images also limit machine (deep) learning methods for detection and recognition.
Until Gao et al. [1] first proposed the (matrix-level) assumption of low rank of infrared background, the idea of optimal separation of low-rank background and sparse target in single frame is introduced into this field. Thanks to the robust principle component analysis (RPCA), such idea achieves great success in infrared small target detection. Based on Gao’s model, many relevant methods sprang up in the past decade. Although the proposed method is based on tensor RPCA (TRPCA), the basic idea is the same as that of Gao’s model. More specially, different from those methods mentioned in Point 2 (i.e., subspace clustering, multiple marginal fisher analysis, and siamesed fully convolutional networks), our model focuses on how to best separate background and target in a single image, rather than how to extract features and recognize targets.
But there are many interesting and inspiring points in [2-4] which helps to diverge our thinking. Thus, to enrich our paper, we have already cited them.
References:
[1] Gao, Chenqiang, et al. "Infrared patch-image model for small target detection in a single image." IEEE Trans. Image Processing 22.12 (2013): 4996-5009.
[2] Peng, Xi, et al. "Structured AutoEncoders for Subspace Clustering." IEEE Transactions on Image Processing 27.10(2018): 5076-5086.
[3] Wang, Qi, Junyu Gao, and Yuan Yuan. "Embedding structured contour and location prior in siamesed fully convolutional networks for road detection." IEEE Transactions on Intelligent Transportation Systems 19.1 (2018): 230-241.
[4] Huang, Zhenyu, et al. " Multiple Marginal Fisher Analysis." IEEE Transactions on Industrial Electronics (2018): 1-1.
Point 3: Although some time costs are reported. Is it ok to evaluate the computational cost using different scale dataset? How many frames (with the image size) could be handled per second?
Response 3: Of course, it’s ok to evaluate the computational cost using different scale datasets. In fact, in our paper, the number of images in Sequences 1-4 is different, and Sequences 1-3 have different sizes (the size of Sequence 4 is the same as Sequence 3, please see Table 4 in the paper). However, since infrared images are often associated with the military, to the best of our knowledge, there is no public data set. In other words, we have limited data and are unable to conduct larger experiments. More specifically, on my laptop, about 11 frames could be handled per second with the image size of 256 × 200 pixels. The algorithm can be processed faster on desktop computer or better instruments. Perhaps this is not particularly fast, but compared with similar methods, there is a significant improvement. For example, the speed of IPI model is about 8.2s per image (please see Table 7). Considering the excellent detection performance, I think it is acceptable.
(Please see the equations in the PDF file)
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
After a quick overview of the revised version, I am satisfied with the paper. My observations on the first version were taken into account. The added figure for the tensor construction is helpful and the modifications are sound. There are only two very minor modifications I could advise:
- Caption of Table 2 is wrong.
- In Table 7, you could remind the reference for each method (this is just a suggestion).