SAR-to-Infrared Domain Adaptation for Maritime Surveillance with Limited Data ^†

Abstract

Deep Learning (DL) algorithms need extensive amounts of data for classification tasks, which can be costly in specialized fields like maritime monitoring. To address data scarcity, we propose a fine-tuning approach leveraging complementary Infrared (IR) and Synthetic Aperture Radar (SAR) datasets. We evaluated our method using the ISDD, HRSID, and FuSAR datasets, employing VGG16 as a shared backbone integrated with Faster R-CNN (for ship detection) and a three-layer classifier (for ship classification). The results showed significant improvements in IR ship detection (mAP: +20%; Recall: +17%) and modest but consistent gains in SAR ship detection tasks (F1-score: +3%, Recall: +1%, mAP: +1%). Our findings highlight the effectiveness of domain adaptation in improving DL’s performance under limited data conditions.

1. Introduction

Effective maritime surveillance aids environmental sustainability by preventing illegal fishing, pollution, and illegal trafficking. Maritime traffic monitoring is particularly challenging due to the vast oceanic coverage, with maritime transport responsible for approximately 80% of the world’s trade [1]. To efficiently monitor such extensive areas, satellite-based surveillance using Synthetic Aperture Radar (SAR) and Infrared (IR) imaging has proven effective.

Classical Machine Learning techniques, when applied to satellite imagery, require extensive manual effort for feature extraction, consuming significant amounts of time and resources. Deep Learning (DL) algorithms offer a substantial advantage by automatically identifying and extracting useful features directly from data, significantly reducing the need for manual intervention and improving task-specific outcomes such as classification, detection, and segmentation.

However, DL algorithms typically require large datasets to iteratively learn and effectively identify discriminative features. In maritime surveillance, acquiring substantial amounts of SAR and IR data is costly, limiting the potential performance of DL methods. Several researchers have employed DL for maritime tasks, such as ship detection [2,3,4] and ship classification [5,6,7], highlighting both its potential and current limitations due to data scarcity.

To address data limitations, techniques such as data augmentation, transfer learning, and domain adaptation have been proposed. In this study, we specifically propose a domain adaptation approach using complementary SAR and IR datasets to improve the effectiveness of DL models in maritime traffic monitoring. The main contributions of this work are as follows:

1.

Improved IR detection performance, achieving more than a 17% increase in the Recall and Mean Average Precision (mAP).

2.

Enhanced SAR classification performance, achieving a 3% increase in the F1-score compared to the baseline.

In the following sections, we briefly discuss the relevant background, outline our methodology, present and analyze our experimental results, and conclude the study.

2. Methodology

Our methodology comprised two main stages: pretraining and fine-tuning. The complete workflow is illustrated in Figure 1.

2.1. Training Pipelines and Models

We explored three distinct fine-tuning pipelines:

1.

Detection-to-Detection (DET-DET): A VGG16 backbone shared with Faster R-CNN is initially trained on SAR detection data and subsequently fine-tuned on IR detection data, and vice versa.

2.

Classification-to-Detection (CLS-DET): A VGG16-based classifier is first trained on SAR classification data. Its trained backbone is then integrated into Faster R-CNN and fine-tuned for IR detection tasks.

3.

Detection-to-Classification (DET-CLS): A Faster R-CNN model with a VGG16 backbone is initially trained on IR detection data. The backbone is then extracted and fine-tuned for SAR classification tasks using a three-layer classifier.

2.2. Datasets

We utilized three publicly available datasets for our experiments:

1.

FuSAR-Ship [8]: A high-resolution SAR ship classification dataset comprising 15 main classes and 98 subclasses, with image dimensions of 512 × 512. For our experiments, we selected four primary classes: Bulk, Cargo, Tanker, and Fishing.

2.

HRSID [9]: A high-resolution SAR ship detection dataset containing 16,951 images, each with a resolution of $800\times 800$.

3.

ISDD [10]: An IR ship detection dataset containing 3061 ship instances, with images sized $500\times 500$.

2.3. Evaluation and Training Parameters

The classification performance was evaluated using the F1-score, while the detection performance was measured using the mAP and Recall. All the models were trained with a batch size of 32 and a learning rate of $1\times {10}^{-4}$ and optimized using the Adam optimizer.

3. Results

We evaluated our method across two experimental setups: (1) same-task (Detection-to-Detection) and (2) cross-task (Classification-to-Detection and vice versa) setups.

3.1. Same-Task Adaptation: Detection-to-Detection (DET-DET)

Table 1. Detection-to-Detection results; bold indicates target dataset. Improvements over baseline is shown in parentheses (%▲).

Adaptation Scenario	Recall (%)		mAP (%)
	Baseline	Ours	Baseline	Ours
SAR → IR (ISDD)	21.73	41 (+19.3)▲	3.53	27.26 (+23.7%▲)
IR → SAR (HRSID)	39.97	40.91 (+1.0)▲	32.64	33.52 (+1.0%▲)

shows the results for the DET-DET pipeline. For IR detection (ISDD), the Faster R-CNN and VGG model, when trained and tested exclusively on IR data (baseline), severely underfitted the data, achieving a low Recall (21.73%) and mAP (3.53%). After integrating SAR features (HRSID), substantial improvements were observed, with the Recall increasing by approximately 19.3% and the mAP by approximately 23.7%. In contrast, using IR data to improve SAR detection resulted in only a modest increase of around 1%, as the models trained and tested solely on SAR data (baseline) were not underfitting the data.

3.2. Cross-Task Adaptation: Classification-to-Detection (CLS-DET) and Vice Versa (DET-CLS)

Table 2. Cross-task results. Bold = target dataset, improvements (%) over baseline = ▲.

Training Approach	Baseline (%)	Ours (%)	Metric
SAR Classification → IR Detection (ISDD)	21.73	↑ 38.95 (+17▲)	Recall
SAR Classification → IR Detection (ISDD)	3.532	↑ 23.521 (+20▲)	mAP
IR Detection → SAR Classification (Fusar)	56.35	↑ 59.63 (+3▲)	F1-Score

shows the results for the cross-task experiments. For CLS-DET, initially training the VGG16 backbone on SAR classification data (FuSAR) and then fine-tuning it for IR detection (ISDD) resulted in significant performance gains, with the Recall improving by approximately 17% and the mAP by approximately 20%. Conversely, in DET-CLS, when initially training the backbone on IR detection data and then fine-tuning it for SAR classification, the F1-score increased by 3%, demonstrating beneficial but smaller cross-domain improvements.

4. Discussion

We investigated two primary domain adaptation scenarios: same-task (Detection-to-Detection) and cross-task (transferring features between classification and detection tasks) scenarios. Our results clearly illustrate that integrating SAR data significantly enhances the IR detection performance. Specifically, the SAR data provided robust features, effectively addressing the underfitting observed in the IR-only models, as evidenced by the marked improvements in the mAP and Recall metrics.

In the Detection-to-Detection scenario, substantial gains in the IR performance highlight the advantage of transferring feature representations from SAR to IR detection. The minimal improvement from IR to SAR detection suggests that SAR datasets inherently contain richer and more diverse features, making them benefit less from complementary/different domain information.

The cross-task experiments also provided valuable insights. The transfer of features from SAR classification to IR detection tasks considerably improved the IR performance, demonstrating effective feature generalization across the tasks. Similarly, utilizing IR detection data for SAR classification, although beneficial, resulted in smaller improvements (approximately 3%), reflecting the limited feature complexity in IR data relative to that in SAR data.

Overall, our findings emphasize the interoperability and mutual benefits between the SAR and IR domains, particularly in scenarios with limited data. Future work will explore different DL architectures and additional SAR and IR datasets and investigate the impact of the geographic and temporal similarity of data sources on the model performance.

5. Conclusions

Our experiments demonstrated that integrating SAR data significantly enhanced the IR ship detection performance, increasing the mAP by over 20% and the Recall by more than 17%. For SAR-based tasks, the improvements were modest but consistent, with increases of 3%, 1%, and 1% in the F1-score, Recall, and mAP, respectively. These findings highlight the complementary relationship between SAR and IR data, confirming that domain adaptation effectively mitigates challenges related to data scarcity. Future research will explore additional SAR and IR datasets collected under geographically and temporally similar conditions to further validate and refine our approach. Overall, this study contributes to the development of more robust DL solutions in maritime surveillance applications facing limited data availability.