Search Results (16)

Hyperspectral object tracking has emerged as a promising task in visual object tracking. The rich spectral information within hyperspectral images benefits the accurate tracking in challenging scenarios. The performances of existing hyperspectral object tracking networks are constrained by neglecting the interactive information among bands within hyperspectral images. Moreover, designing an accurate deep learning-based algorithm for hyperspectral object tracking poses challenges because of the substantial amount of training data required. In order to address these challenges, a new mixed multi-head attention-based feature fusion tracking (MMFT) algorithm for hyperspectral videos is proposed. Firstly, MMFT introduces a feature-level fusion module, mixed multi-head attention feature fusion (MMFF), which fuses false-color features and augments the fused feature with one mixed multi-head attention (MMA) block with interactive information, which increases the representational ability of the features for tracking. Specifically, MMA learns the interactive information across the bands in the false-color images and incorporates the learned interactive information into the fused feature, which is obtained by combining the features of the false-color images. Secondly, a new training procedure is introduced, in which the modules designed for hyperspectral object tracking are first pre-trained on a sufficient amount of modified RGB data to enhance generalization, and then fine-tuned on a limited amount of HS data for task adaption. Extensive experiments verify the effectiveness of MMFT, demonstrating its SOTA performance. Full article

In the field of hyperspectral video tracking (HVT), occclusion poses a challenging issue without a satisfactory solution. To address this challenge, the current study explores the application of capsule networks in HVT and proposes an approach based on octave convolution and a spatial–spectral capsule network (OCSCNet). Specifically, the spatial–spectral octave convolution module is designed to learn features from hyperspectral images by integrating spatial and spectral information. Hence, unlike traditional convolution, which is limited to learning spatial features, the proposed strategy also focuses on learning and modeling the spectral features. The proposed spatial–spectral capsule network integrates spectral information to distinguish among underlying capsule categories based on their spectral similarity. The approach enhances separability and establishes relationships between different components and targets at various scales. Finally, a confidence threshold judgment module utilizes the information from the initial and adjacent frames for relocating the lost target. Experiments conducted on the HOT2023 dataset illustrate that the proposed model outperforms state-of-the-art methods, achieving a success rate of 65.2% and a precision of 89.3%. In addition, extensive experimental results and visualizations further demonstrate the effectiveness and interpretability of the proposed OCSCNet. Full article

With the rapid advancement of hyperspectral imaging technology, hyperspectral object tracking (HOT) has become a research hotspot in the field of remote sensing. Advanced HOT methods have been continuously proposed and validated on scarce datasets in recent years, which can be roughly divided into handcrafted feature-based methods and deep feature-based methods. Compared with methods via handcrafted features, deep feature-based methods can extract highly discriminative semantic features from hyperspectral images (HSIs) and achieve excellent tracking performance, making them more favored by the hyperspectral tracking community. However, deep feature-based HOT still faces challenges such as data-hungry, band gap, low tracking efficiency, etc. Therefore, it is necessary to conduct a thorough review of current trackers and unresolved problems in the HOT field. In this survey, we systematically classify and conduct a comprehensive analysis of 13 state-of-the-art deep feature-based hyperspectral trackers. First, we classify and analyze the trackers based on the framework and tracking process. Second, the trackers are compared and analyzed in terms of tracking accuracy and speed on two datasets for cross-validation. Finally, we design a specialized experiment for small object tracking (SOT) to further validate the tracking performance. Through in-depth investigation, the advantages and weaknesses of current HOT technology based on deep features are clearly demonstrated, which also points out the directions for future development. Full article

Hyperspectral video provides rich spatial and spectral information, which is crucial for object tracking in complex scenarios. Despite extensive research, existing methods often face an inherent trade-off between rich spectral information and redundant noisy information. This dilemma arises from the efficient utilization of hyperspectral image data channels. To alleviate this problem, this paper introduces a hierarchical spectral attention network for hyperspectral object tracking. We employ a spectral band attention mechanism with adaptive soft threshold to examine the correlations across spectral bands, which integrates the information available in various spectral bands and eliminates redundant information. Moreover, we integrate spectral attention into a hierarchical tracking network to improve the integration of spectral and spatial information. The experimental results on entire public hyperspectral competition dataset WHISPER2020 show the superior performance of our proposed method compared with that of several related methods in visual effects and objective evaluation. Full article

Compared to hyperspectral trackers that adopt the “pre-training then fine-tuning” training paradigm, those using the “pre-training then prompt-tuning” training paradigm can inherit the expressive capabilities of the pre-trained model with fewer training parameters. Existing hyperspectral trackers utilizing prompt learning lack an adequate prompt template design, thus failing to bridge the domain gap between hyperspectral data and pre-trained models. Consequently, their tracking performance suffers. Additionally, these networks have a poor generalization ability and require re-training for the different spectral bands of hyperspectral data, leading to the inefficient use of computational resources. In order to address the aforementioned problems, we propose a spectral similarity prompt learning approach for hyperspectral object tracking (SPTrack). First, we introduce a spectral matching map based on spectral similarity, which converts 3D hyperspectral data with different spectral bands into single-channel hotmaps, thus enabling cross-spectral domain generalization. Then, we design a channel and position attention-based feature complementary prompter to learn blended prompts from spectral matching maps and three-channel images. Extensive experiments are conducted on the HOT2023 and IMEC25 data sets, and SPTrack is found to achieve state-of-the-art performance with minimal computational effort. Additionally, we verify the cross-spectral domain generalization ability of SPTrack on the HOT2023 data set, which includes data from three spectral bands. Full article

Hyperspectral images provide a wealth of spectral and spatial information, offering significant advantages for the purpose of tracking objects. However, Siamese trackers are unable to fully exploit spectral features due to the limited number of hyperspectral videos. The high-dimensional nature of hyperspectral images complicates the model training process. In order to address the aforementioned issues, this article proposes a hyperspectral object tracking (HOT) algorithm callled SiamPKHT, which leverages the SiamCAR model by incorporating pyramid shuffle attention (PSA) and knowledge distillation (KD). First, the PSA module employs pyramid convolutions to extract multiscale features. In addition, shuffle attention is adopted to capture relationships between different channels and spatial positions, thereby obtaining good features with a stronger classification performance. Second, KD is introduced under the guidance of a pre-trained RGB tracking model, which deals with the problem of overfitting in HOT. Experiments using HOT2022 data indicate that the designed SiamPKHT achieves better performance compared to the baseline method (SiamCAR) and other state-of-the-art HOT algorithms. It also achieves real-time requirements at 43 frames per second. Full article

Hyperspectral videos (HSVs) can record more adequate detail clues than other videos, which is especially beneficial in cases of abundant spectral information. Although traditional methods based on correlation filters (CFs) employed to explore spectral information locally achieve promising results, their performances are limited by ignoring global information. In this paper, a joint spectral–spatial information method, named spectral–spatial transformer-based feature fusion tracker (SSTFT), is proposed for hyperspectral video tracking, which is capable of utilizing spectral–spatial features and considering global interactions. Specifically, the feature extraction module employs two parallel branches to extract multiple-level coarse-grained and fine-grained spectral–spatial features, which are fused with adaptive weights. The extracted features are further fused with the context fusion module based on a transformer with the hyperspectral self-attention (HSA) and hyperspectral cross-attention (HCA), which are designed to capture the self-context feature interaction and the cross-context feature interaction, respectively. Furthermore, an adaptive dynamic template updating strategy is used to update the template bounding box based on the prediction score. The extensive experimental results on benchmark hyperspectral video tracking datasets demonstrated that the proposed SSTFT outperforms the state-of-the-art methods in both precision and speed. Full article

Object tracking using Hyperspectral Images (HSIs) obtains satisfactory result in distinguishing objects with similar colors. Yet, the tracking algorithm tends to fail when the target undergoes deformation. In this paper, a SiamRPN based hyperspectral tracker is proposed to deal with this problem. Firstly, a band selection method based on a genetic optimization method is designed for rapidly reducing the redundancy of information in HSIs. Specifically, three bands with highest joint entropy are selected. To solve the problem that the information of the template in the SiamRPN model decays over time, an update network is trained on the dataset from general objective tracking benchmark, which can obtain effective cumulative templates. The use of cumulative templates with spectral information makes it easier to track the deformed target. In addition, transfer learning of the pre-trained SiamRPN is designed to obtain a better model for HSIs. The experimental results show that the proposed tracker can obtain good tracking results over the entire public dataset, and that it is better than the other popular trackers when the target’s deformation is qualitatively and quantitatively compared, achieving an overall success rate of 57.5% and a deformation challenge success rate of 70.8%. Full article

The advent of hyperspectral cameras has popularized the study of hyperspectral video trackers. Although hyperspectral images can better distinguish the targets compared to their RGB counterparts, the occlusion and rotation of the target affect the effectiveness of the target. For instance, occlusion obscures the target, reducing the tracking accuracy and even causing tracking failure. In this regard, this paper proposes a novel hyperspectral video tracker where the double Siamese network (D-Siam) forms the basis of the framework. Moreover, AlexNet serves as the backbone of D-Siam. The current study also adopts a novel spectral–deviation-based dimensionality reduction approach on the learned features to match the input requirements of the AlexNet. It should be noted that the proposed dimensionality reduction method increases the distinction between the target and background. The two response maps, namely the initial response map and the adjacent response map, obtained using the D-Siam network, were fused using an adaptive weight estimation strategy. Finally, a confidence judgment module is proposed to regulate the update for the whole framework. A comparative analysis of the proposed approach with state-of-the-art trackers and an extensive ablation study were conducted on a publicly available benchmark hyperspectral dataset. The results show that the proposed tracker outperforms the existing state-of-the-art approaches against most of the challenges. Full article

With the rapid development of hyperspectral imaging technology, object tracking in hyperspectral video has become a research hotspot. Real-time object tracking for hyperspectral video is a great challenge. We propose a fast hyperspectral object tracking method via a channel selection strategy to improve the tracking speed significantly. First, we design a strategy of channel selection to select few candidate channels from many hyperspectral video channels, and then send the candidates to the subsequent background-aware correlation filter (BACF) tracking framework. In addition, we consider the importance of local and global spectral information in feature extraction, and further improve the BACF tracker to ensure high tracking accuracy. In the experiments carried out in this study, the proposed method was verified and the best performance was achieved on the publicly available hyperspectral dataset of the WHISPERS Hyperspectral Objecting Tracking Challenge. Our method was superior to state-of-the-art RGB-based and hyperspectral trackers, in terms of both the area under the curve (AUC) and DP@20pixels. The tracking speed of our method reached 21.9 FPS, which is much faster than that of the current most advanced hyperspectral trackers. Full article

Hyperspectral video with spatial and spectral information has great potential to improve object tracking performance. However, the limited hyperspectral training samples hinder the development of hyperspectral object tracking. Since hyperspectral data has multiple bands, from which any three bands can be extracted to form pseudocolor images, we propose a Transformer-based multimodality information transfer network (TMTNet), aiming to improve the tracking performance by efficiently transferring the information of multimodality data composed of RGB and hyperspectral in the hyperspectral tracking process. The multimodality information needed to be transferred mainly includes the RGB and hyperspectral multimodality fusion information and the RGB modality information. Specifically, we construct two subnetworks to transfer the multimodality fusion information and the robust RGB visual information, respectively. Among them, the multimodality fusion information transfer subnetwork is designed based on the dual Siamese branch structure. The subnetwork employs the pretrained RGB tracking model as the RGB branch to guide the training of the hyperspectral branch with little training samples. The RGB modality information transfer subnetwork is designed based on a pretrained RGB tracking model with good performance to improve the tracking network’s generalization and accuracy in unknown complex scenes. In addition, we design an information interaction module based on Transformer in the multimodality fusion information transfer subnetwork. The module can fuse multimodality information by capturing the potential interaction between different modalities. We also add a spatial optimization module to TMTNet, which further optimizes the object position predicted by the subject network by fully retaining and utilizing detailed spatial information. Experimental results on the only available hyperspectral tracking benchmark dataset show that the proposed TMTNet tracker outperforms the advanced trackers, demonstrating the effectiveness of this method. Full article

To address the problem that the performance of hyperspectral target tracking will be degraded when facing background clutter, this paper proposes a novel hyperspectral target tracking algorithm based on the deep edge convolution feature (DECF) and an improved context filter (ICF). DECF is a fusion feature via deep features convolving 3D edge features, which makes targets easier to distinguish under complex backgrounds. In order to reduce background clutter interference, an ICF is proposed. The ICF selects eight neighborhoods around the target as the context areas. Then the first four areas that have a greater interference in the context areas are regarded as negative samples to train the ICF. To reduce the tracking drift caused by target deformation, an adaptive scale estimation module, named the region proposal module, is proposed for the adaptive estimation of the target box. Experimental results show that the proposed algorithm has satisfactory tracking performance against background clutter challenges. Full article

Hyperspectral video target tracking is generally challenging when the scale of the target varies. In this paper, a novel algorithm is proposed to address the challenges prevalent in the existing hyperspectral video target tracking approaches. The proposed approach employs deep features along with spectral matching reduction and adaptive-scale 3D hog features to track the objects even when the scale is varying. Spectral matching reduction is adopted to estimate the spectral curve of the selected target region using a weighted combination of the global and local spectral curves. In addition to the deep features, adaptive-scale 3D hog features are extracted using cube-level features at three different scales. The four weak response maps thus obtained are then combined using adaptive weights to yield a strong response map. Finally, the region proposal module is utilized to estimate the target box. The proposed strategies make the approach robust against scale variations of the target. A comparative study on different hyperspectral video sequences illustrate the superior performance of the proposed algorithm as compared to the state-of-the-art approaches. Full article

Hyperspectral (HS) videos can describe objects at the material level due to their rich spectral bands, which are more conducive to object tracking compared with color videos. However, the existing HS object trackers cannot make good use of deep-learning models to mine their semantic information due to limited annotation data samples. Moreover, the high-dimensional characteristics of HS videos makes the training of a deep-learning model challenging. To address the above problems, this paper proposes a spatial–spectral cross-correlation embedded dual-transfer network (SSDT-Net). Specifically, first, we propose to use transfer learning to transfer the knowledge of traditional color videos to the HS tracking task and develop a dual-transfer strategy to gauge the similarity between the source and target domain. In addition, a spectral weighted fusion method is introduced to obtain the inputs of the Siamese network, and we propose a spatial–spectral cross-correlation module to better embed the spatial and material information between the two branches of the Siamese network for classification and regression. The experimental results demonstrate that, compared to the state of the art, the proposed SSDT-Net tracker offers more satisfactory performance based on a similar speed to the traditional color trackers. Full article

This paper presents a correlation filter object tracker based on fast spatial-spectral features (FSSF) to realize robust, real-time object tracking in hyperspectral surveillance video. Traditional object tracking in surveillance video based only on appearance information often fails in the presence of background clutter, low resolution, and appearance changes. Hyperspectral imaging uses unique spectral properties as well as spatial information to improve tracking accuracy in such challenging environments. However, the high-dimensionality of hyperspectral images causes high computational costs and difficulties for discriminative feature extraction. In FSSF, the real-time spatial-spectral convolution (RSSC) kernel is updated in real time in the Fourier transform domain without offline training to quickly extract discriminative spatial-spectral features. The spatial-spectral features are integrated into correlation filters to complete the hyperspectral tracking. To validate the proposed scheme, we collected a hyperspectral surveillance video (HSSV) dataset consisting of 70 sequences in 25 bands. Extensive experiments confirm the advantages and the efficiency of the proposed FSSF for object tracking in hyperspectral video tracking in challenging conditions of background clutter, low resolution, and appearance changes. Full article