Beyond RGB: Early Stage Fusion of Thermal and Visual Modalities for Robust Maritime Perception

Abstract

In maritime environments, reliable object detection and semantic segmentation are essential for navigation and collision avoidance, especially under adverse conditions. This paper benchmarks early stage RGB–thermal (RGBT) fusion architectures for these tasks using a novel, pixel-aligned maritime dataset. We evaluate transformer-based, attention-driven, and lightweight convolutional models, analyzing trade-offs between accuracy and efficiency for edge deployment. Our results show that RGBT fusion significantly improved detection robustness, with transformer models achieving the top accuracy and lightweight models like WNet-S offering strong performance with lower computational costs. We also introduce a modular, open-source fusion framework to support reproducible research and practical deployment in maritime and other safety-critical domains.

1. Introduction

Safe navigation and collision avoidance in the maritime domain are heavily dependent on automatic identification system (AIS) and radar technologies. The AIS is a vessel tracking system that broadcasts a ship’s position, identity, and navigation status through VHF radio to improve situational awareness [1]. However, the AIS is only mandatory for vessels exceeding a gross tonnage of 300 and for all passenger ships, which means that many smaller boats remain untracked. Radar is another widely used modality, but its performance is sensitive to meteorological conditions, as well as the shape, size, and material of targets. As a result, radar systems may fail to detect small floating objects or small boats. Effective collision avoidance is therefore critical in maritime environments, where high traffic density, limited maneuverability, and poor visibility can significantly increase the risk of accidents. This creates a pressing need for automated, vision-based perception systems capable of operating reliably across diverse and adverse environmental conditions.

Traditionally, human vision has been used to augment radar and AIS, but it presents inherent limitations. Humans cannot continuously monitor the entire environment, and their perception degrades under fatigue, darkness, fog, heavy rain, or even strong sunlight. Small vessels and floating debris often go unnoticed during nighttime navigation or in dense fog, while heavy rain can obscure even large objects. These limitations underscore the need for automated perception systems that provide reliable detection across diverse and adverse conditions. Consequently, maritime safety increasingly relies on computer vision technologies capable of automatically detecting and segmenting obstacles, regardless of visibility or weather.

Over the past decade, deep learning has transformed computer vision, enabling remarkable advances in image classification, semantic segmentation, and object detection. Convolutional neural networks (CNNs) such as U-Net [2] and, more recently, transformer-based models [3,4] have become the state of the art across domains ranging from autonomous driving to medical imaging [5]. However, models trained solely on RGB imagery often fail under poor illumination or adverse weather, making them insufficient for safety-critical maritime applications.

Multimodal fusion is increasingly vital in safety-critical domains such as autonomous driving, surveillance, and maritime robotics, where combining complementary sensor modalities enhances robustness under adverse conditions. RGB–thermal (RGBT) fusion leverages the strengths of both modalities; thermal cameras, operating in the long-wave infrared (LWIR) spectrum (8–12 $\mu$m), detect heat signatures regardless of lighting, while RGB sensors provide rich spatial and color detail [6]. Prior works in autonomous driving and surveillance have demonstrated the benefits of RGBT fusion.

For instance, in autonomous driving, RGBT fusion has been shown to enhance pedestrian detection in low-light environments [7], as well as enhancing perception during nighttime and in adverse weather conditions [8]. Similarly, in surveillance, RGBT fusion improved intruder recognition and activity monitoring [9]. Recently, Ying et al. introduced RGBT-Tiny, a large scale benchmark for visible-thermal tiny object detection [10]. While their focus was specialized object detection algorithms, our work emphasizes semantic segmentation as the primary task, with object detection only considered as a secondary post-processing step. Similarly, Zhou et al. presented MaDiNet, a Gamma diffusion-based approach for SAR target detection using RGB and radar modalities [11]. Although the sensing modalities differ from our RGBT set-up, this work highlights the growing interest in multimodal fusion for safety-critical environments. These insights underscore the importance of robust multimodal research for small object detection, which is directly relevant to maritime collision avoidance, characterized by small, distant targets.

Despite its success in other domains, RGBT fusion remains underexplored in maritime environments, where variable lighting, reflective surfaces, and limited annotated data pose unique challenges. In maritime perception, recent datasets such as those in [12,13] have enabled benchmarking of multimodal fusion approaches using RGB, thermal, radar, and AIS data. However, systematic evaluation of fusion architectures tailored to maritime conditions—characterized by small, distant objects and dynamic backgrounds—remains limited.

Within computer vision, semantic segmentation and object detection are key tasks enabling detailed scene understanding. A pivotal milestone in RGBT segmentation research was MFNet, which introduced a benchmark dataset of aligned street scenes and a symmetric double-encoder fusion architecture [8]. Subsequent works, such as RTFNet [14] and FuseSeg [15], refined feature fusion strategies, while GMNet introduced graded shallow and deep fusion modules with multiscale supervision [16]. These architectures share many features with other multimodal domains, such as RGB and depth (RGBD). The RGBD architectures follow a similar double-encoder paradigm, which can be seen in FuseNet [17] or ACNet [18]. More recently, attention-based models such as SA-Gate [19] and transformer-based approaches like CMX [20] have pushed the boundaries of accuracy, with CMX achieving state-of-the-art results across multiple modality pairs. At the same time, lightweight architectures with efficient encoders and separable convolutions [21] have emerged to address real-time deployment needs on resource-constrained platforms.

Despite these advances, several challenges remain. First, most multimodal fusion methods have been developed for autonomous driving or indoor robotics and have rarely been evaluated in maritime settings, where object scales, environmental variability, and sensor perspectives differ substantially. Second, a persistent trade-off exists between accuracy and deployability; transformer-based approaches deliver excellent accuracy but incur high computational costs, making them impractical for edge devices commonly used onboard vessels. This trade-off is particularly evident when detecting small objects such as buoys or wooden debris. Transformer-based models like CMX achieve superior accuracy in these cases due to their global attention mechanisms, which capture long-range dependencies and subtle thermal cues. However, their computational demands limit real-time deployment. In contrast, lightweight CNN architectures offer competitive performance for small-object detection while maintaining low latency and minimal parameter counts, making them suitable for resource-constrained onboard systems. This balance remains insufficiently benchmarked in maritime contexts. Finally, while multimodal datasets for driving and surveillance exist (e.g., PST900 [22]), aligned RGBT maritime datasets suitable for segmentation and detection have only recently become available [23].

To address these gaps, our work focuses on challenges unique to maritime perception and builds upon prior advances in multimodal fusion. Building on our previous work to create the RGBT maritime dataset [23] we expanded its size and diversity and improved the alignment quality to support robust benchmarking. Using this enhanced dataset, we trained and evaluated a diverse set of models, including CNN baselines, transformer-based networks, and novel fusion designs. Central to this study is the introduction of WNet, a modular double-encoder architecture that allows systematic benchmarking of different encoder and decoder configurations as well as alternative fusion strategies. This modularity enabled the development of hybrid variants such as WNet-FFM and WNet-CMX, combining elements from state-of-the-art fusion paradigms. Models were assessed not only for segmentation accuracy but also for computational efficiency and inference speed, providing practical insights into optimal design choices for real-time deployment in embedded maritime systems.

To further close these gaps in maritime perception, this paper investigates how multimodal fusion architectures can be optimized to balance accuracy, robustness, and computational efficiency under real-world conditions. Specifically, our contributions are as follows:

• Comprehensive Benchmarking: We present a systematic evaluation of multimodal (RGBT) early fusion architectures for maritime semantic segmentation and object detection. The benchmark spans transformer-based, attention-based, and lightweight CNN models, all retrained on a rigorously aligned RGBT maritime dataset.
• Modular Fusion Architecture: We introduce the WNet architecture, a flexible fusion framework that enables interchangeable encoders, fusion modules, and decoders. This modular design supports systematic exploration of architecture trade-offs and achieves a superior performance-to-cost ratio, making it suitable for edge deployment.
• Performance Efficiency Analysis: We provide practical insights into fusion depth, model complexity, and real-time deployability through a detailed analysis of the inference speed and parameter count. This helps identify architectures that are both accurate and resource-efficient.

Finally, we unveil WNet, an open-source multimodal fusion framework which promotes reproducibility and transparent benchmarking. The framework supports modular experimentation and is available at https://github.com/SEA-AI/beyond-RGB (accessed on 28 November 2025).

2. Materials and Methods

2.1. Dataset

The dataset used in this study consists of precisely aligned RGBT image pairs based on a recently published pixel-aligned maritime panoptic dataset collected in open-water scenarios using SEA.AI devices capable of streaming both modalities. Alignment was performed as described in [23] using a homography-based pipeline built on machine learning-driven correspondence search, enabling accurate annotation propagation and a unified ground truth for masks and bounding boxes.

A key advantage of this dataset is its panoptic nature, which combines semantic and instance-level annotations to support benchmarking of both segmentation and object detection tasks within a single framework. This ensures a consistent evaluation of multimodal fusion architectures in complementary perception objectives.

The collection spans diverse geographic regions throughout the Atlantic Ocean (e.g., Greenland, the Caribbean, La Manche, Gibraltar, and Biscay Bay), varying illumination (daytime and nighttime), and multiple weather conditions, including clear skies, fog, rain, and overcast scenarios. These conditions significantly impact RGB image quality—fog and rain reduce contrast, and nighttime introduces severe illumination challenges—while thermal imagery remains largely unaffected. Representative examples of these scenarios are shown in Figure 1. The dataset also covers a range of maritime environments, from open ocean to harbors and near-shore scenes, with variable traffic density and background complexity. It includes 18 annotated object categories, such as (1) vessel classes (e.g., sailboats and cargo ships), (2) navigation objects (e.g., buoys), and (3) other floating objects (e.g., containers and wooden debris). Following the original protocol, augmented training data were generated to reach approximately 6000 training and 2000 validation samples using semi-automated annotation. A statistical summary of the dataset is provided in

Table 1. Summary statistics of the RGBT maritime panoptic dataset.

Property	Count or Range	Description
Total image pairs	∼8000	6000 training, 2000 validation
Image resolution	640 × 512 px	Pixel-aligned RGB and thermal pairs
Geographic regions	20+	Throughout Atlantic ocean
Illumination conditions	2	Daytime or nighttime (80:20) percentage
Weather conditions	4	Clear, foggy, rainy, overcast
Object categories	18	Vessels, buoys, containers, wooden logs, etc.
Sensor platform	SEA.AI embedded system	RGB + LWIR cameras

.

2.2. Architectures and Fusion Strategies

This work explores a range of early stage fusion architectures for multimodal semantic segmentation and object detection, building on both well-established convolutional neural network (CNN) models and more recent transformer-based approaches.

Early fusion strategies have been widely adopted in multimodal research because they allow for feature representations from RGB and thermal (or, for example, depth) modalities to be combined at multiple stages of the network, rather than relying solely on late decision-level fusion. This is particularly beneficial in maritime perception, where small objects must be detected under challenging conditions, and complementary cues from both modalities can improve feature robustness.

In general, three principal fusion paradigms are used in multimodal computer vision: early fusion, late fusion, and proposal- or decision-level fusion. Late fusion combines the outputs of independent unimodal networks (typically by merging or weighting their detection or segmentation results that can even be learned [24]), which offers modularity and robustness to sensor failure but limits fine-grained feature interaction [25]. Proposal-level fusion, commonly employed in multimodal detection frameworks such as AVOD [26] or MMF [27], merges modality-specific region proposals, improving object-level confidence estimation while remaining limited in pixel-level precision. By contrast, early fusion enables rich cross-modal feature interactions throughout the encoder-decoder hierarchy, which is particularly advantageous for dense prediction tasks such as semantic segmentation or maritime obstacle detection. This justifies our focus on early fusion architectures in this study, as they best capture the complementary nature of RGB and thermal information at multiple feature scales.

A recurring design pattern in multimodal segmentation research is the double-encoder architecture. Consequently, most existing RGBT segmentation models can be abstracted into a generalized form, which is illustrated schematically in Figure 2. In this framework, RGB and thermal inputs are processed by separate encoders, whose feature representations are subsequently combined through a multilevel fusion strategy (denoted as the F block). Similar to the original single-encoder U-Net segmentation architecture [2], skip connections propagate fused encoder features to the decoder at each depth level, facilitating fine-grained spatial reconstruction and preserving detailed information across scales.

This general formulation is highly flexible; the encoders may be based on standard CNN backbones such as ResNet [28], DenseNet [29], or even visual transformers [3,4]. Likewise, the decoder can vary in depth and resolution, while the specific contribution of each architecture typically lies in its chosen fusion strategy. For example, GMNet [16] employs multi-level supervision, CAINet [30] integrates context-aware interactions, and transformer-based CMX [20] introduces cross-modal attention blocks. Thus, the schematic provides a unifying framework for comparing both classical and state-of-the-art RGBT fusion architectures. We benchmarked a diverse set of multimodal architectures, covering both convolutional and transformer-based approaches.

The architectures in the published framework, which were trained and benchmarked on the maritime dataset, include the following:

• Single-modality baselines: RGB-only and thermal-only U-Net models are used to quantify the contribution of each modality individually. Additionally, U-Net variants with separable convolutions, such as those proposed in MobileNet [31], were benchmarked, as the ultimate goal is deployment on edge devices.
• Early fusion baseline: A four-channel U-Net (RGB + thermal concatenated at the input) served as the simplest early fusion strategy.
• Established RGBT double-encoder fusion networks: (1) RTFNet [14], one of the first architectures to introduce a multilevel fusion strategy for RGB and thermal images, performs simple addition-based one-way feature propagation between modalities. (2) GMNet [16] integrates both shallow and deep feature fusion, depending on the encoder depth. It also employs multiple training losses, including boundary loss, binary mask loss, and semantic mask loss.
• Attention-based fusion: SA-Gate [19], which has shown promising results in the RGB depth domain, was benchmarked here as well. It employs bidirectional feature propagation with attention gates to selectively combine features from both modalities.
• Transformer-based fusion: CMX [20], which leverages cross-modal attention mechanisms to fuse features effectively across modalities, was evaluated together with its SegFormer MiT backbone [4].
• Proposed modular network: WNet corresponds to the general schematic shown in Figure 2. It was implemented as a double-encoder U-Net baseline, but it was designed modularly to allow configuration of the encoder (e.g., SegFormer or ResNet), fusion strategy, decoder structure, and depth. This modular design enables systematic experiments across numerous architecture variants, allowing for a balance between feature interaction, depth, and computational cost.

2.3. Training Set-Up

All experiments were implemented in PyTorch (2.2.2+cu121) and executed on an NVIDIA RTX 4070 Ti GPU (Nvidia Corporation, Santa Clara, CA, USA). The models were trained for 60 epochs with a batch size of 6, being constrained by the GPU memory. The Adam optimizer was used with an initial learning rate of $5\times {10}^{-4}$, which decayed stepwise at epochs 40 and 50 to $1\times {10}^{-4}$ and $5\times {10}^{-5}$, respectively.

The training/validation split was performed with entire trips (images collected throughout one voyage) rather than individual frames to avoid data leakage. Data augmentation was applied using the Albumentations library [32], including horizontal flips and affine transformations.

The loss functions varied depending on the architecture. Binary cross-entropy with logits was used for most models due to its numerical stability compared with plain BCE. For the selected networks, Lovász-Softmax [33] and Lovász hinge [34] losses were retained in order to be consistent with their original publications.

2.4. Evaluation Metrics

Performance was evaluated using established metrics for semantic segmentation and object detection, as both pixel-level accuracy and detection reliability are critical in maritime perception tasks. All evaluations were conducted in a class-agnostic manner for the following, since the primary objective was robust detection and accurate object masks, while class differentiation played a secondary role:

• Segmentation metrics: (1) The intersection over union (IoU) [35] measures the overlap between the predicted and ground truth masks. (2) The Dice similarity coefficient (equivalent to the F1 score) [36] emphasizes the balance between precision and recall. (3) The Matthews correlation coefficient [37] provides a balanced evaluation, even under class imbalance. The binarization threshold was selected for the validation set to maximize the mean IoU.
• Detection: Bounding boxes were derived from segmentation masks and evaluated using precision, recall, and the F1 score [38]. True positives, false positives, and false negatives were determined using an IoU threshold of 0.5 between the predicted and ground truth boxes [35]. This threshold aligns with widely adopted standards (e.g., PASCAL VOC, and COCO) and offers a practical balance between localization precision and recall. Higher thresholds would disproportionately penalize small object detections, which are critical in maritime scenarios. To further analyze robustness, the results were stratified by object size into four categories: small, medium, large, and valid. The valid category included all objects larger than 1.75 × 1.75 px, sharing its lower bound with the small category, while the thresholds for the medium and large categories were set at 8 × 8 px and 20 × 20 px. This stratification highlights performance differences across scales, which is particularly relevant for detecting small and distant objects, which are common in maritime environments.
• Efficiency: The inference time (milliseconds per frame) and model parameter count were measured to quantify the trade-off between accuracy and practical deployability on resource-constrained edge devices. These metrics provide insight into whether an architecture can support real-time operation onboard vessels without exceeding hardware limitations.

3. Results

3.1. Semantic Segmentation

We evaluated the model variants described in the previous section, and the corresponding results are summarized in

Table 2. Segmentation benchmark results. The time is in milliseconds, the size is in millions of parameters, and the binarization threshold is the selected confidence threshold. Bold values highlight the best performance within each metric category (e.g., lowest inference time, smallest model size, highest IoU, Dice, and Matthews coefficients), indicating architectures that achieve optimal trade-offs between accuracy and efficiency. Models are grouped by sensor fusion approaches and U-Net-based baseline architectures, as indicated by the horizontal lines, to facilitate comparison within similar design families.

Model	Time	Size	Thr	IoU	Dice	Matthews
RTFNet-R18	4.13	31.00	0.28	31.74	39.76	41.18
GMNet-R18	6.70	31.98	0.25	33.17	41.11	42.39
GMNet-R18-1L *	6.66	31.87	0.28	32.76	40.57	41.81
GMNet-R34	8.14	52.20	0.22	32.04	39.88	41.16
SAGate-R18	4.28	9.82	0.35	32.93	40.80	42.11
SAGate-R18-1L	4.38	9.67	0.30	32.99	40.95	42.33
SAGate-R34	5.71	14.99	0.32	34.46	42.46	43.74
WNet	2.94	5.17	0.32	32.69	40.57	41.97
WNet-S **	2.52	0.62	0.28	33.13	41.12	42.46
WNet-S-Deeper	3.67	2.25	0.28	32.82	40.89	42.29
WNet-MLP	2.24	0.88	0.32	32.96	41.07	42.41
WNet-CMX	5.06	4.95	0.25	32.41	40.51	41.97
WNet-FFM	5.18	4.51	0.25	32.67	40.46	41.76
CMX-b0	6.98	12.11	0.30	33.99	41.84	43.05
CMX-b2	11.68	66.56	0.38	36.67	44.61	45.76
UNetS-S-4i	1.99	0.35	0.28	31.99	39.89	41.29
UNet-S-4i	2.40	0.29	0.25	31.68	39.68	41.19
UNetS-S-THR	2.02	0.35	0.22	30.67	38.36	39.87
UNet-S-THR	2.31	0.29	0.22	30.40	38.20	39.58
UNetS-S-RGB	2.04	0.35	0.15	16.52	23.00	24.57
UNet-S-RGB	2.28	0.29	0.10	16.17	22.57	24.28

* 1L = only one loss used instead of the original. ** S = separable convolution was used.

. The confidence thresholds for segmentation mask binarization were determined on a validation set by maximizing the mean intersection over union (mIoU). The selected threshold for each model is reported in Table 2.

The benchmarked architectures included double-encoder models, single-encoder models with four-channel input (denoted by the suffix -4i), and single-encoder models using only thermal (THR in the table) or RGB input. When a ResNet backbone was employed, the suffix R18 or R34 indicates the specific version of the ResNet encoder used. For the transformer-based CMX architectures, the suffixes b0 and b2 denote the transformer backbones SegFormer MiT-b0 and MiT-b2, respectively. The default U-Net and WNet configurations use encoder channel dimensions of $32,64,64,128,256$ to preserve a low parameter number. There is also U-NetS, which is a bit simplified as it works with bilinear interpolation instead of the last convoluted upsampling.

Additionally, we included several hybrid architectures that combine elements of WNet and CMX; WNet-MLP adopts CMX’s multilayer perceptron (MLP) decoder, WNet-FFM incorporates CMX’s feature fusion module, and WNet-CMX integrates both components.

Multimodal fusion consistently outperformed the single-modality baseline in all IoU values and Dice and Matthews coefficients. RGB-only and thermal-only U-Net achieved limited performance, with the thermal-only models performing more robustly.

Transformer-based CMX achieved the highest values for all metrics across the dataset, confirming the strong capacity of transformer-based feature aggregation. However, this came at a significant computational cost. Lightweight networks such as WNet-S or SAGate-R18 reached competitive IoU values while maintaining far lower parameter counts, demonstrating the potential of efficiency-focused designs.

To complement these quantitative results, Figure 3 presents the representative segmentation outputs for three architectures across different environmental conditions: clear daytime, nighttime, and overcast scenes. Transformer-based CMX-b2 achieved the most accurate segmentation, closely matching the ground truth even in challenging nighttime scenarios. WNet-FFM demonstrated competitive performance with fewer parameters, successfully detecting small objects in low-contrast conditions. In contrast, the thermal-only UNet-S-THR baseline model struggled with spatial detail and occasionally missed distant objects, such as two light-emitting targets in the nighttime example that were visible only in the RGB image, underscoring the importance of RGBT fusion for robust maritime perception. These qualitative examples reinforce the quantitative findings in Table 2, where multimodal fusion consistently outperformed the single-modality baselines, particularly for small-object detection under adverse conditions.

3.2. Object Detection

The detection results, derived from the segmentation masks, mirrored the segmentation findings, as shown in

Table 3. Detection benchmark precision, recall, and F1 score results for various detection size classes, all expressed as percentages. Bold values highlight the best performance within each metric category (e.g., lowest inference time, smallest model size, highest IoU, Dice, and Matthews coefficients), indicating architectures that achieve optimal trade-offs between accuracy and efficiency. Models are grouped by sensor fusion approaches and U-Net-based baseline architectures, as indicated by the horizontal lines, to facilitate comparison within similar design families.

Model	Valid			Small			Large
	prec	rec	f1	prec	rec	f1	prec	rec	f1
RTFNet-R18	45.41	42.88	44.11	28.62	23.54	25.83	78.36	76.06	77.19
GMNet-R18	46.47	44.77	45.60	28.77	24.56	26.50	76.07	75.57	75.82
GMNet-R18-1L	49.80	45.19	47.39	31.64	24.02	27.31	76.82	75.57	76.19
GMNet-R34	46.87	43.63	45.19	28.86	23.18	25.71	78.31	74.10	76.15
SAGate-R18	50.56	47.44	48.95	35.06	29.32	31.93	69.67	76.71	73.02
SAGate-R18-1L	49.18	47.70	48.43	34.51	30.52	32.40	74.38	78.01	76.15
SAGate-R34	50.82	50.70	50.76	34.89	31.91	33.33	72.96	83.06	77.68
WNet-S	50.66	51.61	51.13	37.27	36.48	36.87	70.02	71.50	70.75
WNet	49.82	49.95	49.89	38.39	36.85	37.60	62.95	68.89	65.79
WNet-S-Deeper	48.28	51.58	49.87	35.32	38.11	36.66	69.42	72.48	70.92
WNet-MLP	44.39	50.64	47.31	32.96	37.09	34.90	61.24	72.31	66.32
WNet-CMX	48.86	50.11	49.48	35.98	36.30	36.14	67.37	73.29	70.20
WNet-FFM	51.65	51.12	51.38	37.63	35.82	36.71	69.27	73.78	71.45
CMX-b0	53.47	50.67	52.03	37.70	32.39	34.84	76.12	80.46	78.23
CMX-b2	56.06	54.22	55.13	40.01	35.34	37.53	81.48	82.41	81.94
UNetS-S-4i	48.38	50.64	49.48	35.11	36.54	35.81	68.95	70.52	69.73
UNet-S-4i	48.19	49.79	48.97	36.42	35.76	36.09	61.73	68.57	64.97
UNetS-S-THR	47.84	48.26	48.05	35.99	35.34	35.66	66.49	61.40	63.84
UNet-S-THR	45.91	48.06	46.96	34.35	36.54	35.41	61.97	62.38	62.18

. We did not compare them against detection architectures as this work focuses on segmentation, and the detection metrics provide a different view on their performance.

Across all evaluated object size categories, CMX consistently achieved the highest detection F1 scores, confirming its strong performance in multimodal feature integration. The CMX-b2 variant led the benchmark tests, with superior results in the valid, small, and large object categories, reflecting the benefits of transformer-based cross-modal attention for robust object localization. WNet-S and SA-Gate [19] followed closely, delivering competitive accuracy with significantly lower computational complexity. These findings reinforce the hypothesis that multimodal fusion architectures outperform single-modality baselines, highlighting the advantage of combining RGB and thermal modalities for comprehensive detection in challenging maritime environments.

3.3. Efficiency Analysis

To evaluate the trade-off between accuracy and deployability, we analyzed the inference time and model size alongside detection performance. Figure 4 summarizes these relationships across three subplots.

• (a) Inference time vs. F1 detection score: This plot illustrates the balance between computational speed and detection accuracy. The transformer-based CMX-b2 achieved the highest F1 score but at the cost of significantly longer inference times, making it less suitable for real-time edge deployment. In contrast, lightweight architectures such as WNet-S and SA-Gate delivered competitive F1 scores while maintaining inference times below 3 ms per frame, highlighting their suitability for embedded systems.
• (b) Inference time vs. false positives: This subplot shows how efficiency correlates with detection reliability. The models with longer inference times (e.g., CMX-b2) generally exhibited fewer false positives, while faster models like WNet-S maintained a reasonable balance, suggesting that efficiency-focused designs do not necessarily compromise robustness.
• (c) Inference time vs. Matthews correlation coefficient: This metric provides a holistic view of the binary classification quality. CMX-b2 again led in overall performance, but WNet-S and SA-Gate achieved strong Matthews coefficients with minimal computational overhead, reinforcing their practicality for real-time maritime applications.

Overall, these results confirm that while transformer-based models offer state-of-the-art accuracy, lightweight CNN-based architectures provide an optimal compromise between performance and efficiency, making them ideal for deployment on resource-constrained platforms.

4. Discussion

The results presented in this work provide clear evidence of the benefits of multimodal fusion for object detection and semantic segmentation in maritime environments. Across all evaluated architectures, introducing thermal and RGB modalities jointly led to consistent performance gains compared with single-modality models. Even the simple early fusion variants, such as four-channel U-Net models, outperformed their single-input counterparts. However, the best results were achieved with double-encoder fusion networks, confirming that modality-specific feature extraction followed by learned fusion enables more robust multimodal representations.

These findings align with prior work in terrestrial RGBT segmentation [8,14,16,19,20], where multimodal architectures consistently outperformed the unimodal baselines. The present results extend these observations to the maritime domain, which is characterized by high scene variability, reflections, and frequent low-contrast conditions. The improvement due to multimodal fusion highlights that thermal imagery compensates for illumination-related weaknesses of RGB, while RGB contributes spatial and color information absent in thermal channels.

Among the tested models, transformer-based CMX [20] achieved the highest overall segmentation and detection accuracy, confirming (1) the strong capability of cross-modal attention mechanisms to integrate features effectively and (2) transformers’ capability of nuanced feature extraction. However, this came at a considerable computational cost, as CMX exhibited the slowest inference times and highest parameter count. Its smaller variant, CMX-b0, demonstrated a more favorable balance between accuracy and efficiency, suggesting that transformer fusion can scale down effectively while retaining competitive performance.

In contrast, lightweight architectures such as WNet-S and SA-Gate provided strong alternatives for edge deployment. Both achieved high IoU values and F1 scores while maintaining significantly lower inference times. The proposed modular WNet implementation demonstrated that the general double-encoder U-Net scheme is flexible enough to accommodate different backbones, fusion modules, and decoder configurations while maintaining high performance. The results further indicate that the model complexity and parameter count are not always reliable predictors of performance. For instance, larger architectures such as GMNet [16] and RTFNet [14] did not outperform simpler designs like WNet-S in either the segmentation or detection tasks.

From the object detection results, which were derived via extraction of the bounding box from the segmentation masks, similar conclusions can be drawn. The models leveraging both modalities consistently achieved higher precision and recall values. False positives and false negatives were concentrated primarily in the small object category, which remains a persistent challenge in maritime vision tasks.

Beyond maritime applications, the insights from this study can be generalized to other safety-critical domains where multimodal fusion is essential, such as autonomous driving, intelligent video surveillance, and industrial robotics. These fields share common challenges, including variable illumination, adverse weather, and the need for real-time inference on constrained hardware platforms. The modular fusion strategies proposed here, along with the accompanying performance-efficiency analysis, offer a transferable foundation for designing deployable multimodal perception systems in these domains.

For example, early fusion techniques that improve low-light visibility in thermal RGB maritime imagery can similarly improve pedestrian detection in nighttime driving or intruder recognition in infrared surveillance systems. By demonstrating how lightweight fusion mechanisms and dual-encoder architectures balance accuracy with computational cost, this work contributes generalizable design principles for sensor fusion. These principles are adaptable to the differing technical constraints of maritime, automotive, and surveillance platforms—such as sensor placement, latency requirements, and object scale variability—making the proposed framework broadly applicable.

In the maritime context specifically, improved multimodal perception directly supports collision avoidance by enhancing detection reliability under poor visibility, fog, or nighttime conditions, situations where radar or an AIS often underperform. Integrating RGBT fusion models into onboard vision systems can therefore significantly strengthen situational awareness, enabling earlier and more dependable identification of nearby vessels and floating obstacles and ultimately reducing the risk of maritime accidents.

Limitations and Future Work. Despite these advances, several limitations remain, and thus there is significant potential for future research. First, this study focused solely on RGB and thermal modalities without considering other potential sources of information such as radar or LiDAR. Second, all object detection results were derived from segmentation masks rather than using dedicated detection networks such as YOLO [39] or Mask R-CNN [40], since our evaluation focused on segmentation-derived bounding boxes to maintain consistency across architectures. In future work, we plan to train and evaluate multimodal object detection architectures, such as YOLO and Mask R-CNN variants, to directly compare their performance with segmentation-derived detection and assess the impact of multi-modality on end-to-end detection accuracy. Additionally, while the inference time and model size were measured on a standard GPU set-up, no hardware-optimized or compiled inference benchmarking (e.g., TensorRT or ONNX graph optimization) was performed in this study. Such optimizations can significantly accelerate inference by leveraging graph-level transformations, kernel fusion, and precision reduction (e.g., FP16 or INT8 quantization). Beyond TensorRT and ONNX, other techniques such as pruning, weight clustering, and mixed-precision training are widely used to reduce latency and the memory footprint without substantial accuracy loss. Architectural simplifications also offer potential gains. For example, Su et al. proposed the Pixel Difference Network (PiDiNet), which leverages pixel difference convolutions to capture high-order local information while maintaining low computational costs. Future work will include a systematic evaluation of these deployment-oriented optimizations to quantify their impact on the speed-accuracy trade-off, particularly for transformer-based architectures, where the computational overhead is most pronounced. This analysis will help determine whether lightweight CNN models remain the most practical choice after optimization or whether transformer models can achieve real-time performance under hardware-specific acceleration. Another finding was that precise RGBT alignment is critical for effective multimodal fusion, particularly for early stage architectures that rely on pixel-level correspondence. While we did not conduct a formal sensitivity analysis in this study, initial experiments using unaligned image pairs showed no improvement over single-modality baselines, indicating that misalignment can negate the benefits of fusion. Future work will include a systematic evaluation of alignment errors and their impact on segmentation and detection performance, with a focus on small objects, where pixel-level accuracy is most influential. We also plan to provide visual qualitative comparisons and integrate recently published architectures to strengthen benchmarking. Finally, while the dataset covers a wide range of marine conditions, it is still limited in size compared with large-scale terrestrial benchmarks such as COCO [41], and studying environmental effects such as day-versus-nighttime conditions or varying weather scenarios on model robustness would be a valuable direction for future work.

Generalization to Other Domains. While this study focused on maritime perception, the proposed modular fusion framework and benchmarking methodology have strong potential for adaptation to other safety-critical domains such as autonomous driving, aerial surveillance, and industrial robotics. These domains share similar challenges, including adverse weather, low visibility, and hardware constraints. Future research will explore how the insights gained here—particularly regarding lightweight architectures and multimodal fusion strategies—can be transferred and validated across these applications.

5. Conclusions

This study presents a comprehensive benchmark of early stage RGB–thermal fusion architectures tailored for maritime perception, addressing the challenges of object detection and semantic segmentation under adverse conditions. By leveraging a rigorously aligned multimodal dataset and evaluating a diverse set of architectures, including transformer-based, attention-driven, and lightweight CNN models, we demonstrated that multimodal fusion significantly enhances detection robustness and segmentation accuracy compared with single-modality baselines. For example, transformer-based CMX-b2 achieved the highest segmentation IoU value (36.67%) and detection F1 score (81.94%) but required 11.68 ms per frame and 66.56 M parameters, highlighting its computational cost. In contrast, lightweight designs such as WNet-S delivered a competitive IoU value (33.13%) and F1 score (70.75%) with only 0.62 M parameters and 2.52 ms for the inference time, offering a practical solution for real-time edge deployment. The proposed modular fusion framework further enables flexible experimentation and rapid prototyping, contributing a valuable tool for the research community.

Beyond maritime applications, the insights gained here can be generalized to other safety-critical domains such as autonomous driving and surveillance, where multimodal perception under hardware constraints is essential. By linking performance metrics to practical deployment considerations, this work establishes design guidelines for balancing accuracy and efficiency and provides a framework to accelerate future research in deployable multimodal vision systems.