Multimodal Instance Segmentation of Building Façade Damage with Frequency-Decoupled Feature Fusion

Abstract

Building façade damage segmentation remains challenging because cracks, peeling, hollow-like areas, stains, and erosion often exhibit weak contrast, irregular boundaries, discontinuous structures, and substantial scale variations under complex surface textures and illumination conditions. To address these challenges, this study proposes MMDF-Net, a multimodal instance segmentation network for paired RGB–IR façade damage images. The network adopts dual RGB and infrared branches and performs multi-scale cross-modal interaction across P2–P5 levels. A Frequency-Decoupled Cross-Modal Bridge is designed to separately model low-frequency material responses and high-frequency local damage details, thereby enabling gated bidirectional exchange between the two modalities. A Crack Topology-Aware Encoder further enhances the continuity of thin, curved, branched, and locally discontinuous cracks through directional strip convolution and bending-aware modeling. In addition, a Multi-Scale Damage Decoding Pyramid integrates high-resolution boundary details, mid-level structural cues, and high-level semantic context to support stable mask prediction across different damage scales. Experiments were conducted on a custom-built RGB–IR building façade damage dataset containing 1500 paired image samples across five damage categories. MMDF-Net achieved 82.1% mask precision, 78.6% mask recall, 72.3% mask mAP50, and 65.6% mask mAP50–95, with 7.8 M parameters and 23.5 GFLOPs, outperforming representative RGB and RGB–IR segmentation baselines. These results indicate that task-oriented multimodal feature organization can improve façade damage segmentation without relying on excessive model expansion.

1. Introduction

Building façades and exterior walls are continuously exposed to environmental loads, temperature variations, moisture migration, material aging, and in-service deterioration, making them susceptible to surface damage such as cracks, spalling, staining, erosion, hollowing, and detachment. These defects are important indicators for building maintenance assessment and may also reflect potential risks, including internal material deterioration, interfacial debonding, and localized degradation of in-service performance [1]. Therefore, the timely and accurate identification of exterior wall damage is essential for building safety assessment, maintenance decision-making, and whole-life-cycle management. However, conventional building façade inspection still relies primarily on manual visual assessment, which is labor-intensive, subjective, inefficient, and associated with substantial safety risks when performed at height. This reliance makes it difficult to meet the requirements of large-area, periodic, and detailed inspections [2]. In recent years, unmanned aerial vehicle platforms, infrared thermography, deep learning, and BIM-based digital workflows have increasingly been introduced into building inspection, providing new technical pathways for improving the efficiency of damage-data acquisition, localization accuracy, and maintenance management [3].

With the development of deep learning methods, including convolutional neural networks and Transformers, significant progress has been made in image-based crack detection, defect recognition, and damage segmentation [4]. Compared with conventional methods based on manually defined thresholds, edge operators, or handcrafted texture features, deep learning methods can automatically extract shape, texture, and semantic information from damaged regions through end-to-end feature learning and exhibit greater adaptability to complex backgrounds [5]. Within semantic and instance segmentation frameworks in particular, models can not only classify damage types but also delineate pixel-level contours, thereby supporting damage-area estimation, boundary analysis, and repair quantification [6]. However, segmenting damage on building exterior walls remains more challenging than general scene segmentation. Cracks are typically thin, low-contrast, locally discontinuous, and branched; spalling and erosion regions often exhibit irregular boundaries and fragmented textures; stains can be confused with shadows, joints, surface contamination, and natural aging marks; and regions affected by hollowing or detachment often lack clear boundaries in visible-light images [7]. Consequently, models are prone to missed detections, false detections, and incomplete boundary segmentation in practical inspections.

Most existing studies on building damage segmentation rely on visible-light red–green–blue (RGB) images. RGB images provide rich color, texture, edge, and geometric information, making them valuable for identifying visible cracks, spalling, and surface deterioration [8]. However, single-modality RGB imaging is susceptible to illumination variations, shadow occlusion, façade textures, wall joints, and low-contrast damage [9]. When damage features are inconspicuous or background textures are complex, visible-light information alone cannot reliably distinguish genuine defects from surface-related interference. By contrast, infrared thermography can capture differences in thermal responses caused by material discontinuities, moisture distribution, interfacial debonding, and potential hollowing, thereby providing physical information complementary to visible-light imagery for exterior wall damage detection [10]. Previous studies have demonstrated the potential of infrared imaging for identifying delamination, debonding, and latent defects in building façades, concrete structures, and infrastructure [11].

Accordingly, visible–infrared fusion has emerged as an important research direction in building exterior wall damage detection and concrete defect segmentation. Relevant studies have investigated multimodal complementarity at the input, feature, and decision levels [12]. For example, studies comparing early-, intermediate-, and late-stage RGB–thermal fusion for multi-defect detection on exterior walls have shown that these strategies differ in accuracy and efficiency [13]. Other studies have applied infrared, visible-light, and fused images to concrete damage segmentation, demonstrating that multimodal information helps reduce false detections in complex backgrounds [14]. Recent end-to-end multimodal frameworks for building façade damage segmentation have further integrated image registration, infrared–visible fusion, and damage segmentation into a unified pipeline, facilitating the application of multisource perception to building inspection [15].

Despite this progress, RGB–infrared (IR) instance segmentation of building damage in practical settings still faces several key challenges. First, existing fusion strategies often concatenate or weight visible and infrared features as if they were homogeneous, overlooking differences in physical imaging mechanisms and frequency representations between the two modalities [16]. Visible-light images are more sensitive to high-frequency crack edges, abrupt texture changes, and local geometric variations, whereas infrared images generally contain smoother, low-frequency material responses and thermal-anomaly distributions [17]. Simple fusion may weaken fine-crack boundaries or introduce irrelevant thermal responses and background interference into the damage predictions. Second, crack segmentation requires not only the identification of local grayscale or texture differences but also the preservation of spatial continuity in thin, curved, branched, and locally discontinuous structures [18]. Conventional convolutional features effectively capture local textures but are insufficient for modeling the directionality and topological connectivity of crack paths. Finally, building damage exhibits substantial scale variation, ranging from hairline microcracks and medium-scale spalling to large-area erosion and material deterioration. Different damage types also have distinct requirements for spatial resolution, structural context, and high-level semantics [19]. Therefore, models require a more effective multiscale coordination mechanism to balance high-resolution boundary details with high-level semantic constraints.

To address the aforementioned challenges, this study proposes a Multimodal Damage Fusion Network (MMDF-Net) for the instance segmentation of building exterior wall damage. The network employs a dual-branch architecture to separately extract multilevel features from registered RGB and IR images and performs cross-modal interaction at the P2–P5 scales. To account for differences between the two modalities in imaging mechanisms, frequency responses, and damage representations, a Frequency-Decoupled Cross-modal Bridge (FDCB) is designed to separately model low-frequency material responses and high-frequency damage details and to achieve complementary enhancement through a bidirectional gating mechanism. To address the fragmentation, weak responses, and poor preservation of branching structures associated with thin cracks, a Crack Topology-Aware Encoder (CTAE) is introduced to enhance the representation of crack-path continuity through directional strip convolutions and a bending-aware branch. Furthermore, a Multi-scale Damage Decoding Pyramid (MDDP) is constructed to integrate shallow details, intermediate-level structures, and high-level semantics, thereby improving the stability of mask prediction for damage regions at different scales. Finally, the network predicts damage classes, bounding boxes, and instance masks using the YOLO11 segmentation head.

The main contributions of this study are summarized as follows:

(1) A paired RGB–IR dataset was constructed for instance segmentation of building damage. The dataset comprises 1500 paired RGB–IR images acquired using a DJI Mavic 3T platform and covers five typical categories of exterior wall damage: Cracks, Peeling, Suspected Hollow, Stains, and Erosion.

(2) MMDF-Net was proposed for RGB–IR instance segmentation of building damage. Through dual-branch feature extraction and multiscale cross-modal interaction, the network jointly exploits RGB texture details and IR material-response information, thereby improving the stability of damage recognition and mask segmentation in complex exterior wall scenes.

(3) An FDCB was designed to separately model low-frequency material responses and high-frequency damage details. Through bidirectional gating, the module strengthens relevant damage evidence while suppressing interference from redundant textures and irrelevant thermal responses.

(4) A CTAE was designed to enhance the continuity representation of thin, curved, branched, and locally discontinuous cracks through directional strip convolutions and bending-aware modeling, while also accounting for irregular damage regions such as peeling and erosion.

(5) An MDDP was introduced to integrate microcrack boundaries, medium-scale damage structures, and high-level semantic information associated with large-area material deterioration. Experimental results show that the complete model, with 7.8 M parameters and 23.5 G FLOPs, achieves a mask Precision of 82.1%, a mask Recall of 78.6%, a mask mAP50 of 72.3%, and a mask mAP50–95 of 65.6%, outperforming representative RGB and RGB–IR segmentation baselines.

The remainder of this paper is organized as follows. Section 2 reviews related research on deep-learning-based damage segmentation and multimodal fusion. Section 3 presents MMDF-Net and its core modules. Section 4 describes the dataset, experimental settings, evaluation metrics, and comparative results. Section 5 discusses the main findings, limitations, and future research directions. Section 6 concludes the paper.

2. Related Work

2.1. Deep Learning-Based Detection and Segmentation of Building Damage

With advances in machine learning technologies, data-driven methods have been increasingly applied to complex engineering tasks [20,21,22], Building damage inspection has gradually transitioned from manual observation and handcrafted image-processing methods to data-driven visual recognition. Traditional approaches typically rely on grayscale thresholding, edge detection, texture descriptors, or morphological operations to extract cracks and other surface defects [23,24,25]. Although straightforward and readily interpretable, these methods are highly sensitive to variations in illumination, surface staining, wall joints, aging textures, and complex façade backgrounds [26]. In practical building inspections, damage patterns often exhibit low contrast, irregular boundaries, and substantial scale variation, making it difficult for rule-based methods to generalize reliably across buildings and acquisition conditions. In addition, machine learning has also demonstrated considerable potential for structural performance assessment [27,28].

With advances in unmanned aerial vehicle (UAV)-based image acquisition and deep learning, automated building damage detection has emerged as an important research direction in civil infrastructure inspection [29]. UAV platforms can acquire close-range façade images at different heights, viewpoints, and distances, thereby reducing the safety risks and labor costs associated with manual inspection [30]. Recent studies have further integrated UAV imagery with BIM, edge computing, and digital maintenance workflows to support defect localization, visualization, and management [31]. These studies demonstrate the engineering value of vision-based inspection; however, the reliability of subsequent damage assessment remains highly dependent on accurate pixel- or instance-level segmentation.

Deep learning methods have substantially improved the recognition of cracks and other forms of surface deterioration. Encoder–decoder networks such as U-Net have been widely used for defect segmentation because they recover spatial details through skip connections [32]. DeepLabV3+ enhances multiscale contextual modeling and boundary recovery through atrous spatial pyramid pooling and a lightweight decoder [33]. PSPNet employs a pyramid pooling module to aggregate global contextual information, providing an effective framework for pixel-level prediction in complex scenes [34]. SegFormer further combines a hierarchical Transformer encoder with a lightweight MLP decoder, thereby improving the global modeling capability and computational efficiency of semantic segmentation [35]. Although effective in delineating category-level damage regions, these semantic segmentation methods may still produce blurred boundaries, merged regions, or incomplete masks when damage instances are small, discontinuous, or affected by background interference.

Crack segmentation is particularly challenging in building façade inspection because façade components, wall joints, surface materials, shadows, and background textures can interfere with crack recognition. Chen et al. proposed a two-stage deep learning method for segmenting building façade cracks in UAV-acquired images, demonstrating that filtering irrelevant façade regions before pixel-level crack analysis can reduce background interference [4]. For multiple types of façade defects, Li et al. further addressed multi-defect segmentation and class imbalance by adopting a balanced copy–paste strategy to improve the recognition of minority classes and small defects [36]. These studies indicate that building damage segmentation is not merely a general segmentation task but is jointly constrained by complex façade backgrounds, variations in defect morphology, and data imbalance.

Instance segmentation provides a more suitable representation for building damage assessment because it distinguishes individual damaged regions while generating pixel-level masks. Two-stage methods such as Mask R-CNN offer strong instance-modeling capabilities [37], whereas Cascade R-CNN improves the discrimination of high-quality region proposals through cascaded detectors [38]. YOLO-based segmentation models provide efficient detection and segmentation solutions for practical inspection scenarios [39]. However, general-purpose instance segmentation models are not specifically designed for building surface damage. Fine cracks may be attenuated through repeated downsampling; the boundaries of hollow or debonded regions may be indistinct; and spalling, stains, and erosion often exhibit irregular shapes that can be confused with surface textures. Therefore, building damage segmentation requires not only object-level localization but also fine-grained boundary preservation, crack-continuity modeling, and multiscale feature representation.

Another important challenge arises from the structural characteristics of cracks. Cracks are typically thin, curved, branched, and locally discontinuous. Their recognition depends not only on local intensity or texture differences but also on the continuity of crack paths [18]. Methods such as DeepCrack have shown that multilevel deep features and edge-based supervision can enhance the structural representation of crack pixels [40]. Similarly, recent segmentation studies have demonstrated that attention-enhanced feature extraction and multiscale feature fusion can improve the representation of weak-edge and fragmented structures [41]. In addition, building damage spans multiple spatial scales: fine cracks require high-resolution shallow features; spalling and crack clusters require intermediate-level structural cues; whereas large-area erosion or material deterioration requires high-level semantic context. Feature Pyramid Networks have demonstrated the effectiveness of constructing multiscale representations through top-down pathways and lateral connections [19]. However, standard multiscale decoders may still struggle to balance fine-crack details with global damage semantics in complex façade damage segmentation. These limitations have motivated the development of damage-aware segmentation networks that simultaneously account for crack topology, irregular boundaries, and multiscale damage distributions.

2.2. Multimodal Fusion for Building Damage Assessment

Multimodal fusion has become an important research direction in visual perception because a single modality is often insufficient to characterize complex real-world scenes. Different modalities generally capture complementary information: optical images provide rich textural and geometric details; infrared images reveal thermal responses or material differences; depth images characterize spatial structures; and other sensing sources may reflect physical or environmental properties [42]. By integrating heterogeneous information at the image, feature, or decision level, multimodal methods can improve model robustness to illumination variations, occlusion, low contrast, and background interference. In recent years, multimodal fusion has been widely applied to scene understanding in autonomous driving [43,44,45] and robotic perception [46,47,48], where it can improve segmentation stability in complex environments. Multisource observational data have also been fused in remote sensing to enhance land-cover recognition and change detection [49,50,51], whereas multimodal information has been integrated in medical imaging to improve the reliability of diagnosis and quantitative analysis [52,53,54]. Collectively, these studies indicate that reliable perception benefits from the complementary modeling of visual appearance and modality-specific physical cues.

For building damage assessment, visible-light and infrared images provide particularly valuable complementary information. RGB images preserve color, texture, edge information, and geometric details, which are important for identifying visible cracks, spalling boundaries, stains, and erosion [8]. However, RGB images are susceptible to variations in illumination, shadows, surface contamination, façade joints, and material textures [9]. Infrared thermal imaging reflects physical responses associated with surface temperature, moisture distribution, material discontinuities, voids, and debonding anomalies [10]. Infrared thermography has been extensively evaluated and applied to detect delamination and debonding in buildings and infrastructure, demonstrating its value in identifying latent or weakly visible defects [11]. Therefore, RGB–IR multimodal imaging provides a promising approach to improving the robustness of building damage detection under complex surface conditions.

Existing multimodal fusion strategies can be broadly categorized into image-level, feature-level, and decision-level fusion. Image-level fusion combines visible-light and infrared images before feature extraction and is conceptually straightforward. However, it may introduce artifacts or suppress details if the fused representation is not well aligned with the downstream segmentation objective [55]. Decision-level fusion combines independent predictions from different modalities but cannot fully exploit cross-modal interactions during feature learning. Feature-level fusion offers greater flexibility and has been widely applied to RGB–thermal semantic segmentation and RGB-X scene understanding [56]. Studies of infrared–visible image fusion have further emphasized the importance of registration accuracy, complementary-information preservation, and task adaptability for reliable multimodal perception [57].

RTFNet was among the earlier methods to investigate dual-stream RGB–thermal feature fusion for urban semantic segmentation, demonstrating that thermal information can compensate for the limitations of visible-light images under complex illumination conditions [43]. FEANet exploits multilevel RGB–thermal features through feature-enhanced attention modules, highlighting the importance of high-resolution spatial details for real-time semantic segmentation [46]. CMX further introduces a unified RGB-X semantic segmentation framework that facilitates multisource information interaction through cross-modal feature rectification and fusion, demonstrating the potential of modality-agnostic cross-modal modeling [56]. However, these methods are primarily designed for general scene understanding, and their fusion mechanisms do not specifically address crack edges, material-related thermal anomalies, or irregular defect boundaries in building damage assessment.

In building and infrastructure inspection, several studies have shown that combining infrared and visible-light information can improve defect-detection performance relative to RGB imagery alone. Yang et al. compared early-, intermediate-, and late-stage RGB–thermal fusion strategies for multi-defect detection on exterior walls, demonstrating that the fusion stage significantly affects model accuracy and efficiency [13]. Pozzer et al. investigated the effects of infrared, visible-light, and fused images on concrete damage segmentation, showing that different imaging modalities contribute differently to concrete defect recognition [14]. Subsequent studies using multimodal Siamese networks further demonstrated that exploiting similarities between visible-light and thermal images can reduce false positives in concrete debonding segmentation [58]. For building façades, Wang et al. proposed an automated deterioration-detection system based on infrared–visible image fusion and deep learning and constructed a paired infrared–visible dataset of façade damage [15]. Furthermore, an end-to-end multimodal façade damage segmentation framework integrates image registration, infrared–visible fusion, and damage segmentation within a unified pipeline, thereby increasing the level of automation in building inspection tasks [59]. These studies confirm the value of multimodal sensing for building damage perception; however, they also indicate that fusion performance depends strongly on how information from different modalities is represented and exchanged.

A key issue is the inherent heterogeneity between visible-light and infrared features. Visible-light images generally contain high-frequency information, including crack edges, texture discontinuities, spalling boundaries, and local geometric variations; by contrast, infrared images typically exhibit smoother, low-frequency responses associated with thermal distributions, moisture variations, and regional material anomalies [16]. Research on infrared–visible image fusion has also shown that decomposing images into low-frequency base components and high-frequency detail components can facilitate the separate preservation of thermal responses and visible texture details [17]. Direct concatenation or generic feature fusion may weaken fine-crack boundaries, amplify irrelevant thermal responses, or introduce redundant background information. Therefore, multimodal building damage segmentation should not only fuse RGB and IR features but also account for differences between the modalities in their frequency characteristics and physical meaning.

In addition, different categories of building damage rely on the two modalities to varying degrees. Cracks rely strongly on visible edges and topological cues; hollow or debonded regions may rely more heavily on regional infrared responses; spalling and erosion require both boundary details and material context; and stains can be confused with background textures, necessitating stronger cross-modal discrimination [13]. These category-specific dependencies indicate that a single, simple fusion strategy is unlikely to fully accommodate complex multi-class damage segmentation tasks.

3. Proposed Method

3.1. Overall Framework of MMDF-Net

The proposed MMDF-Net adopts a dual-backbone multimodal instance segmentation framework to precisely segment cracks and other damaged regions in registered visible-light and infrared images of building façades. The overall architecture is illustrated in Figure 1. Given a pair of registered RGB–IR images, the network feeds the RGB and IR images into their respective backbone branches to extract modality-specific features.

Both encoding branches consist of Conv and C3k2 [60] modules and produce features at four scales: P2, P3, P4, and P5. Specifically, P2 retains a high spatial resolution to capture hairline cracks and fine boundaries; P3 represents crack groups and local connectivity; and P4 and P5 provide regional semantics and macro-scale damage context. Rather than directly concatenating the bimodal features, an FDCB is introduced at each scale to model complementary information in the corresponding RGB and IR features according to their material-related frequency characteristics, thereby enhancing informative modality-specific responses and suppressing redundant interference.

Following fusion by the FDCBs, the network obtains four cross-modal features: S2, S3, S4, and S5. Among these features, S5 is further processed by Spatial Pyramid Pooling–Fast (SPPF) to enhance its high-level semantic and global contextual representations. Subsequently, MDDP performs bidirectional multi-scale decoding. The top-down pathway progressively propagates high-level damage semantics to high-resolution features, strengthening the semantic constraints on fine-grained regions. In contrast, the bottom-up pathway feeds shallow boundary and texture information back into intermediate- and high-level features, improving the consistency between crack details and regional semantics.

To address the elongated, curved, branched, and locally discontinuous characteristics of cracks, CTAE is introduced into the crack-sensitive P2 and P3 layers to enhance the representation of crack-path continuity and weak boundaries. Finally, the decoded features D2, R3, R4, and R5 are fed into the YOLO11n-seg segmentation head to predict damage classes, bounding boxes, and instance masks. MMDF-Net is trained end-to-end using the default loss configuration of YOLO11n-seg [61], which includes the bounding-box regression loss, instance-mask segmentation loss, classification loss, and Distribution Focal Loss (DFL), with corresponding loss gains of 7.5, 7.5, 0.5, and 1.5, respectively. During training, all modules jointly contribute to the forward computation. PyTorch’s automatic differentiation mechanism [62] computes the gradients of the total loss with respect to all trainable parameters based on the computational graph and the chain rule. The network parameters are then jointly updated through backpropagation using the SGD optimizer.

3.2. Frequency-Decoupled Cross-Modal Bridge

Building damage exhibits distinct responses in the visible-light and infrared modalities. Visible-light images tend to preserve crack edges, surface textures, and local geometric discontinuities, which typically manifest as high-frequency responses. In contrast, infrared images tend to reflect material thermal responses, moisture differences, potential Suspected Hollow, and regional anomalies, generally exhibiting smoother feature distributions and distinct low-frequency characteristics. Therefore, directly concatenating the two modalities as homogeneous features may weaken crack boundaries and cause background textures to be misinterpreted as valid damage cues.

To mitigate information redundancy, detail attenuation, and noise interference that may arise from the direct fusion of visible-light and infrared features, this study introduces an FDCB to facilitate evidence exchange at corresponding scales between the two encoder streams. Based on the imaging characteristics of building cracks and material damage, the FDCB decomposes the features of each modality into low-frequency material responses and high-frequency local details and uses an adaptive gating mechanism to enable bidirectional information compensation between the two modalities.

As shown in Figure 2, the visible-light feature V and infrared feature I are used as inputs and independently processed by two-dimensional average pooling (AvgPool2d) with a 5 × 5 kernel, a stride of 1, and a padding of 2 to perform local spatial smoothing, yielding the corresponding low-frequency components:

$$V_l=AvgPool2d(V), I_l=AvgPool2d(I)$$

(1)

AvgPool2d suppresses rapidly varying feature responses through local averaging, so that its output primarily retains relatively smooth regional information, whereas the residual between the original feature and its smoothed component emphasizes local edges, textures, and fine-grained variations. It should be noted that this process constitutes a frequency-inspired spatial-domain low-pass–residual approximation rather than an explicit transform-domain frequency decomposition. The corresponding high-frequency residuals are then obtained by subtracting the low-frequency components from the original features:

$$V_h=V-V_l, I_h=I-I_l$$

(2)

The high-frequency components primarily capture crack edges, abrupt texture changes, local peeling boundaries, and fine-grained surface variations. Through the frequency decomposition described above, the FDCB separately models regional material responses and crack-detail cues, thereby reducing the interference that may arise when information at different frequencies is processed uniformly during fusion.

To further select effective cross-modal information, the FDCB constructs separate low- and high-frequency gating branches. The low-frequency gating branch generates cross-modal material-response weights from the low-frequency visible-light and infrared features, $V_l$ and $I_l$, to regulate the transfer of material-state and regional information. Similarly, the high-frequency gating branch generates cross-modal crack-texture weights from the high-frequency features,$V_h$ and $I_h$, to regulate the transfer of edge details and local anomalous responses. This gating process can be expressed as follows:

$$\begin{array}{c}[G_l^V,G_l^I]=\sigma \left({\varphi }_l(Concat(V_l,I_l))\right)\\ [G_h^V,G_h^I]=\sigma \left({\varphi }_h(Concat(V_h,I_h))\right)\end{array}$$

(3)

where ${\varphi }_l(\cdot )$ and ${\varphi }_h(\cdot )$ denote the low- and high-frequency gating functions, respectively, each implemented using a lightweight 1 × 1 convolution. In addition, $\sigma (\cdot )$ denotes the sigmoid activation function. $G_{l }^V$ and $G_h^V$ control the strength of information transfer from the infrared modality to the visible-light branch, whereas $G_{l }^I$ and $G_h^I$ control that from the visible-light modality to the infrared branch.

During cross-modal evidence exchange, the visible-light branch incorporates information on material conditions and thermal responses from the low-frequency infrared components, together with local anomalous responses from the high-frequency infrared components. Conversely, the infrared branch incorporates surface structure, crack-edge, and texture-detail information from the visible-light features. Specifically, this process is defined as follows:

$$\begin{array}{l}\widehat{V}=V+G_l^V\odot Conv(I_l)+G_h^V\odot DWConv(I_h)\\ \widehat{I}=I+G_l^I\odot Conv(V_l)+G_h^I\odot DWConv(V_h)\end{array}$$

(4)

The low-frequency information is projected through a 1 × 1 convolution to perform channel adaptation for the low-frequency material responses. In parallel, the high-frequency information is processed using a depthwise separable 3 × 3 convolution to extract local high-frequency structures, thereby enhancing crack edges and fine-grained damage cues. Here, ⊙ denotes element-wise multiplication. Through this bidirectional bridging scheme, the FDCB selectively incorporates relevant evidence from the other modality while preserving the original modality-specific features, thereby reducing the influence of irrelevant thermal responses, complex textures, and imaging noise on crack segmentation.

Finally, the fusion-enhanced visible-light and infrared features, V and I, respectively, are concatenated along the channel dimension and then passed through a 1 × 1 convolution to generate the cross-modal output feature at the current scale:

$$F=Concat(\widehat{V},\widehat{I})$$

(5)

The FDCB is deployed across the P2–P5 scales to accommodate differences in the damage information encoded by features at different levels. Specifically, P2 emphasizes microcrack boundaries, fine textures, and shallow surface variations; P3 better represents crack connectivity and local morphology; and P4 and P5 emphasize local peeling, large-area damage, material anomaly regions, and stronger semantic responses. By exchanging low- and high-frequency evidence across multiple scales, the FDCB improves the efficiency with which visible-light and infrared information is utilized without altering the basic structure of the backbone network, thereby allowing the multimodal fusion process to align more closely with the imaging mechanisms associated with building cracks and material damage.

3.3. Crack Topology-Aware Encoder

The challenge of crack segmentation lies not only in identifying local differences in grayscale, temperature, or texture, but also in preserving the spatial continuity of crack paths. Cracks on real building surfaces are typically elongated, curved, branched, and locally discontinuous. In the presence of interference from stains, shadows, wall joints, material textures, and surface peeling, local crack responses may be extremely weak, making the model prone to discontinuous crack predictions, boundary jitter, and localized segmentation omissions. Although conventional 3 × 3 square convolutions can extract neighborhood texture information, their receptive fields are biased toward local block-shaped regions and are insufficient for modeling the directional extension and path continuity of elongated cracks. Consequently, it is difficult to adequately represent the elongated structures and topological connectivity of cracks using conventional convolutions alone.

To enhance the model’s ability to represent continuous crack structures, a CTAE is designed for the fine-grained decoding stage. This module models crack responses across different orientations and morphologies, enabling the decoded features to better preserve the continuity of crack paths while recovering spatial details.

As shown in Figure 3, given an input feature X, the CTAE first applies a 1 × 1 convolution to reorganize the channel information. It then constructs three parallel depthwise convolution branches to capture crack structures with different orientations and morphologies. Specifically, the horizontal strip branch employs a 1 × k depthwise convolution to capture horizontally extending fine cracks; the vertical strip branch employs a k × 1 depthwise convolution to extract vertical cracks, vertical cracking on wall surfaces, and local vertical discontinuities; and the bending branch employs a 3 × 3 depthwise convolution with a dilation rate of d = 2 to expand the local receptive field and supplement the responses associated with oblique, curved, branched, and irregular crack regions. The three topology branches can be formulated as follows:

$$\begin{array}{l}H(X)=DWCon{v}_{1\times k}(X)\\ V(X)=DWCon{v}_{k\times 1}(X)\\ B(X)=DWCon{v}_{3\times 3}(X)\end{array}$$

(6)

where K is initially set to 7; $H(\cdot )$, $V(\cdot )$, and $B(\cdot )$ denote the horizontal strip, vertical strip, and bending-aware branches, respectively, and DWConv denotes depthwise convolution. Compared with conventional 3 × 3 convolutions, strip convolutions expand the receptive field along a single direction with low computational overhead, making them more suitable for modeling the directional extension of elongated cracks. Meanwhile, dilated depthwise convolutions provide complementary local structural information over a broader spatial range without substantially increasing the number of parameters.

The outputs of the three branches are subsequently processed by Batch Normalization (BN) followed by the Sigmoid Linear Unit (SiLU). SiLU was originally proposed as a sigmoid-weighted linear unit and is defined as SiLU(x) = xσ(x), where σ(⋅) denotes the sigmoid function. Compared with piecewise-linear activation functions such as ReLU, SiLU provides a smooth, continuously differentiable nonlinearity while retaining negative-valued responses. In this study, SiLU is retained as the activation function in CTAE to maintain consistency with the default activation configuration of the YOLO11 backbone.

Because crack orientations and morphologies vary across building surfaces and damage regions, the CTAE further introduces a dynamic topology routing mechanism. The module first aggregates the global response of the current feature map through global average pooling and then applies a 1 × 1 convolution to generate three sets of weights for the horizontal, vertical, and bending-aware branches:

$$[G_h,G_v,G_b]=\sigma (Wr(GAP(X)))$$

(7)

where $GAP(\cdot )$ denotes global average pooling, $Wr(\cdot )$ denotes the 1 × 1 convolutional routing function, and $\sigma (\cdot )$ denotes the sigmoid activation function. $G_h$, $G_v$, and $G_b$ regulate the response strengths of the horizontal, vertical, and bending-aware branches, respectively. Through this dynamic routing mechanism, the module adaptively emphasizes more reliable topology branches according to the current crack morphology rather than relying on a fixed convolutional direction.

Finally, the responses of the three topology branches are weighted by their corresponding coefficients, fused, aligned through a 1 × 1 convolution, and added to the original features through a residual connection:

$$X^{\prime }=X+Conv(G_h\odot H(X)+G_v\odot V(X)+G_b\odot B(X))$$

(8)

where $\odot$ denotes element-wise multiplication. The residual connection preserves the original damage semantics and prevents the topology enhancement process from excessively altering the original feature distribution, thereby supporting compatible representations of cracks, peeling, and other damaged regions.

From an engineering application perspective, the CTAE offers two principal advantages. First, strip convolutions expand the receptive field along the direction of crack extension with low computational overhead, making them more suitable than conventional square convolutions for modeling the continuous paths of elongated cracks. Second, dynamic topology routing avoids the assumption of a fixed crack orientation, allowing the module to adaptively adjust the contributions of different branches according to the current image content. For example, the module can emphasize the horizontal strip response in regions containing horizontal cracks and strengthen the vertical strip response in regions containing vertical cracks. For curved, branched, and irregular crack regions, the dilated bending-aware branch can more effectively supplement structural information.

3.4. Multi-Scale Damage Decoding Pyramid

Building damage exhibits substantial scale variation. Hairline cracks typically occupy only a few pixels and rely primarily on shallow, high-resolution features for boundary localization. Crack clusters, network-like cracks, and local peeling require intermediate-level structural information to characterize their connectivity and regional morphology. In contrast, large-area damage, post-disaster wall damage, and regional material anomalies rely more heavily on high-level semantic context for discrimination. Conventional YOLO segmentation architectures generally use P3–P5 features for prediction, achieving a balance between accuracy and efficiency in general object segmentation tasks. However, for extremely thin targets such as building cracks, features at P3 and above may already have lost some boundary details after repeated downsampling. Although the direct introduction of a P2 branch can improve the recall of microcracks and fine boundaries, insufficient high-level semantic constraints may cause wall textures, joints, stains, and shadows to be misidentified as cracks.

To address these issues, the MDDP is constructed to jointly model microcrack boundaries, intermediate-scale damage structures, and macroscale damage semantics. As shown in Figure 4, the MDDP takes the backbone outputs S2, S3 and S4, together with the high-level feature F5 processed by SPPF, as inputs. First, the high-level feature F5 is processed by Cross-Stage Partial Spatial Attention (C2PSA) to enhance its global semantic representation, yielding a stronger representation of the macroscale damage context: D5 = C2PSA(F5).

C2PSA adopts a cross-stage partial structure in which one branch is processed by PSABlock before being fused with the preserved branch. Each PSABlock contains a multi-head attention module and a Feed-Forward Network (FFN) with residual connections. The attention module models long-range dependencies among spatial features, thereby enhancing the global contextual representation of high-level damage features [63].

SPPF expands the receptive field of the high-level features, whereas C2PSA enhances global contextual interactions. Together, these operations enable the model to better capture large-area damage, regional material anomalies, and damage semantics at the building-component level.

Subsequently, the network progressively restores spatial resolution through a top-down pathway. The high-level semantic features are sequentially upsampled and concatenated with the corresponding lower-level features S4, S3, and S2, with C3k2 used for feature aggregation at each stage:

$$\begin{array}{l}{D}_4=C3k2(Concat(Up(D_5),S_4))\\ D_3=CTAE(C3k2(Concat(Up(D_4),S_3)))\\ D_2=CTAE(C3k2(Concat(Up(D_3),S_2)))\end{array}$$

(9)

where $Up(\cdot )$ denotes the upsampling operation. This pathway progressively propagates high-level damage semantics to the high-resolution feature layers, enabling P2 and P3 to preserve crack boundaries and fine-grained details while maintaining macroscale semantic constraints. Because P2 and P3 are more sensitive to microcracks, discontinuous cracks, and crack connectivity, CTAE is introduced at these two scales to enhance the representation of elongated crack paths and local topological structures.

Building on this process, the pyramid further establishes a bottom-up feedback pathway. Following downsampling, the high-resolution features are fused with the corresponding features from the top-down pathway at the same scale, thereby propagating fine-grained crack-boundary information back to the intermediate- and high-level layers:

$$\begin{array}{l}{R}_3=CTAE(C3k2(Concat[Conv(D_2),D_3]))\\ R_4=C3k2(Concat[Conv(R_3),D_4])\\ R_5=C3k2(Concat[Conv(R_4),D_5])\end{array}$$

(10)

This feedback process enables the microcrack details recovered at P2 to contribute to the scale-wise refinement of P3–P5, thereby reducing semantic inconsistencies among predictions across scales. Finally, the pyramid outputs a four-scale feature set, [D2,R3,R4,R5].

Through this design, the MDDP can adaptively organize multilevel features according to the scale variations and morphological characteristics of building damage. When an image contains hairline cracks or fine, discontinuous cracks, the model places greater emphasis on the high-resolution details and topological information provided by D2 and R3. When crack clusters, local peeling, or intermediate-scale damage are present, the model can use R3 and R4 to strengthen structural connectivity and regional consistency. For large-area damage and post-disaster wall damage, R5 provides more comprehensive global semantic constraints. Therefore, the pyramid balances microcrack details, intermediate-scale structures, and macroscale damage semantics, thereby improving segmentation stability in scenes containing coexisting damage at multiple scales and complex background interference.

4. Experimental Results and Analysis

4.1. Datasets

4.1.1. Data Acquisition Conditions

To evaluate the effectiveness of the proposed method for damage recognition and segmentation under complex building-surface conditions, a visible–infrared bimodal dataset for building damage was constructed. The images were collected from the façades of an old teaching building on campus at varying acquisition heights, viewing angles, distances, illumination conditions, and damage states. Representative samples from the dataset are shown in Figure 5.

Data were acquired using a DJI Mavic 3T unmanned aerial vehicle (UAV) equipped with a visible-light camera at a resolution of 3000 × 4000 pixels and a thermal infrared camera at a resolution of 512 × 640 pixels. The onboard thermal imager employs an uncooled vanadium oxide (VOx) microbolometer with a pixel pitch of 12 μm, an operating spectral range of 8–14 μm, and a noise-equivalent temperature difference (NETD) of ≤50 mK (F1.0).

To improve image quality and the distinguishability of surface thermal anomalies, data acquisition was conducted under the following conditions:

Camera orientation: The optical axis of the UAV camera was maintained as close to perpendicular to the building façade as possible to reduce perspective distortion and thermal-radiation errors associated with variations in the observation angle.

Acquisition distance: The flight distance was adjusted according to the façade dimensions and the camera field of view such that the vertical field of view of each image covered approximately 3–6 m of the façade.

Environmental and temporal conditions: Image acquisition was conducted, whenever possible, under clear or partly cloudy conditions, with low wind speeds and no rainfall. Acquisition periods were selected according to façade orientation to provide suitable solar irradiation and surface thermal gradients. East-facing façades were primarily imaged between 08:00 and 09:00, west-facing façades between 14:00 and 18:00, and north-facing façades between 11:00 and 13:00, whereas south-facing façades were imaged during periods of relatively high daytime temperatures. For façades with nonstandard orientations, the acquisition time was adjusted according to the on-site solar irradiation conditions and variations in surface temperature.

4.1.2. Dataset Registration and Processing

Differences in imaging resolution, field of view, and geometric distortion between the visible-light and thermal infrared cameras prevent the direct use of the original image pairs for training and inference. Therefore, image registration is required before model training and inference.

During registration, the transparency of the infrared image layer was adjusted, and the infrared image was overlaid on the corresponding RGB image. Structures that could be reliably identified in both modalities, including window-frame corners, wall joints, and component edges, were selected as registration control points. Through iterative overlay comparison and parameter adjustment, the horizontal and vertical scaling factors and translation parameters of the RGB image relative to the infrared image were determined. The resulting initial registration parameters were then batch-applied to images acquired from the same region and fine-tuned for individual samples when necessary.

Registration accuracy was evaluated based on the pixel distance between corresponding control points in the two modalities, with an acceptable error threshold of 2 pixels. Each registered image pair was further inspected using transparent overlays and edge-overlap checks, with particular attention to the alignment of window frames, wall joints, and structures adjacent to damaged regions. For image pairs with registration errors exceeding 2 pixels, the scaling and translation parameters were further adjusted. Samples that could not be reliably registered because of variations in viewing angle, local parallax, or poor image quality were excluded from the dataset. After registration, corresponding regions were cropped to the common valid field of view of the two modalities, and all images were uniformly saved at a resolution of 512 × 640 pixels.

The final dataset contains 1500 registered RGB–IR image pairs. RGB images primarily provide information on the color, texture, edges, and morphology of building surfaces, whereas infrared images provide information on surface-temperature distributions and thermal anomalies. The complementary information provided by the two modalities can enhance the model’s representation of low-contrast damage, suspected hollow regions, and the spatial distribution of surface anomalies.

Based on common damage characteristics of aging building façades, five damage categories were annotated: Crack, Peeling, Suspected Hollow, Stain, and Erosion. Cracks typically appear as elongated, low-contrast, or locally discontinuous linear structures; Peeling refers to regions of surface-material detachment with irregular boundaries; Suspected Hollow refers to regions in which internal voids or abnormalities may be present and whose surfaces may exhibit abnormal temperature responses under specific thermal conditions; Stain mainly includes wall discoloration and contamination traces that can be confused with background textures; and Erosion includes regions with complex boundaries associated with weathering, corrosion, and material deterioration. These categories encompass fine linear damage, regional damage, and surface-material deterioration, thereby enabling evaluation of the model’s instance-segmentation and cross-modal fusion capabilities under complex backgrounds and multi-scale damage conditions.

All samples were annotated for instance segmentation, with each damage instance represented by a class label and a polygonal contour. The original annotations were uniformly converted to the YOLO segmentation format, in which each line contains a class index followed by the normalized coordinates of the corresponding polygon vertices.

To prevent the same physical region or damage instance from appearing in both the training and test sets, a spatially disjoint train–test splitting strategy was adopted. Samples from the east-, west-, and south-facing façades were used for training and validation, whereas samples from the north-facing façade were reserved exclusively for testing. The final training, validation, and test sets contained 1200, 150, and 150 image pairs, respectively, corresponding to an 8:1:1 split. The training set was used for model parameter optimization, the validation set for model selection and hyperparameter monitoring, and the test set exclusively for final performance evaluation. The same data split was used for all comparative methods to ensure consistent and fair evaluation.

4.2. Experimental Setup

Unless otherwise stated, all models were trained and evaluated on the same experimental platform under identical training settings to ensure fair comparisons. The experiments were conducted on the Ubuntu 22.04 operating system using Python 3.10, PyTorch 2.2.2, CUDA 12.1, Ultralytics 8.3.75, and OpenCV 4.13.0. Stochastic Gradient Descent (SGD) was used as the optimizer for all experiments, with no dedicated hyperparameter tuning strategy applied. Detailed experimental configurations are provided in

Table 1. Experimental setup and training parameters.

Item	Configuration
Image size	640 × 640
Batch size	32
Training epochs	400
Warm-up strategy	3 epochs linear ramp-up
Initial learning rate	0.01
Optimizer	Stochastic Gradient Descent (SGD)
Hardware	NVIDIA RTX 3080 GPU

.

4.3. Evaluation Metrics

As this study addresses building damage instance segmentation, the model predicts not only class labels and bounding boxes but also a pixel-level mask for each damage instance. Accordingly, mask-based metrics are adopted as the primary evaluation criteria, including mask Precision, mask Recall, mask mAP50, and mask mAP50–95. Unlike box-based metrics, mask-based metrics are calculated from the pixel-level overlap between predicted and ground-truth masks, thereby providing a more direct assessment of the segmentation quality of cracks, peeling, Suspected Hollow, stains, and erosion.

Given a predicted mask Mp and its corresponding ground-truth mask Mg, the intersection over union mask (IoU_mask) is defined as follows:

$$Io{U}_{mask}={\displaystyle \frac{|M_p\cap M_g|}{|M_p\cup M_g|}}$$

(11)

A prediction is classified as a true positive (TP) when its predicted class is correct and its mask IoU with the matched ground-truth instance exceeds the specified threshold; otherwise, it is classified as a false positive (FP). Unmatched ground-truth instances are classified as false negatives (FNs). To ensure consistent evaluation conditions across different models, the reported Precision and Recall are both calculated at a fixed confidence threshold of 0.25 and are defined as follows:

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(12)

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(13)

Here, TP, FP, and FN denote the numbers of true positives, false positives, and false negatives, respectively.

Average precision (AP) represents the area under the Precision–Recall curve for a given class across confidence thresholds and is expressed as follows:

$$AP={\int }_0^{1}P(R)dR$$

(14)

For a task involving N damage classes, mean average precision (mAP) is calculated by averaging the AP values across all classes:

$$mAP={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^{n}A{P}_i}$$

(15)

Here, AP_i denotes the average precision for the i-th damage class. Mask mAP50 is calculated at a mask IoU threshold of 0.50, whereas mask mAP50–95 averages the mAP values obtained at mask IoU thresholds from 0.50 to 0.95 in increments of 0.05, providing a more stringent assessment of mask overlap and overall segmentation quality.

In addition, the number of parameters and floating-point operations (FLOPs) are reported to compare model size and computational cost.

4.4. Comparison with Representative Segmentation Methods

4.4.1. Overall Performance Comparison

As shown in

Table 2. Comparison with representative segmentation methods.

Model	Modality	Precisionmask	Recallmask	mAP50mask	mAP50-95mask	Params/M	FLOPs/G
Mask R-CNN	RGB	72.4 ± 0.44	65.8 ± 0.51	61.3 ± 0.39	52.9 ± 0.45	44.2	162.7
Cascade R-CNN	RGB	74.1 ± 0.41	67.2 ± 0.47	63.5 ± 0.36	55.3 ± 0.42	68.9	214.6
U-Net	RGB	68.6 ± 0.55	62.4 ± 0.61	56.1 ± 0.50	47.5 ± 0.58	31.0	54.8
DeepLabV3+	RGB	70.5 ± 0.48	65.1 ± 0.54	59.2 ± 0.43	50.8 ± 0.49	41.3	86.5
PSPNet	RGB	69.4 ± 0.52	63.7 ± 0.58	57.6 ± 0.47	49.5 ± 0.53	46.7	93.8
SegFormer	RGB	73.1 ± 0.39	67.9 ± 0.45	62.7 ± 0.34	54.2 ± 0.40	13.7	25.4
YOLOv8n-seg	RGB	73.8 ± 0.38	68.9 ± 0.43	64.8 ± 0.33	56.4 ± 0.39	3.4	12.6
YOLO11n-seg	RGB	75.2 ± 0.36	70.6 ± 0.42	66.7 ± 0.31	58.6 ± 0.37	2.9	10.7
FEANet	RGB–IR	77.6 ± 0.31	73.4 ± 0.36	68.5 ± 0.28	60.8 ± 0.33	28.6	47.9
CMX	RGB–IR	79.4 ± 0.28	75.1 ± 0.33	69.8 ± 0.25	62.7 ± 0.30	36.9	61.3
MMDF-Net	RGB–IR	82.1 ± 0.22	78.6 ± 0.27	72.3 ± 0.20	65.6 ± 0.24	7.8	23.5

, MMDF-Net achieves the best overall performance in building damage instance segmentation, demonstrating its effectiveness under complex building surface conditions. Compared with conventional instance segmentation models, such as Mask R-CNN and Cascade R-CNN, the proposed method achieves higher mask-based metrics with substantially lower model complexity. Although conventional multi-stage methods provide instance-level modeling capabilities, their region proposal and cascaded prediction processes introduce considerable parameter and computational overhead while remaining limited in delineating elongated cracks, localized peeling, and irregular damage boundaries.

Semantic segmentation models such as U-Net, DeepLabV3+, PSPNet, and SegFormer do not directly produce mutually independent damage instances. To enable a fair comparison of mask AP within a unified evaluation framework, instance-level post-processing is applied to their class-probability maps. Specifically, a fixed probability threshold of 0.5 is first applied to generate a binary prediction mask for each damage category. Connected-component analysis is then used to separate spatially disconnected regions into independent candidate instances, and the confidence score of each instance is derived from the predicted probabilities within the corresponding candidate region. Isolated regions with areas below a predefined threshold are removed as noise. The resulting candidate instances are evaluated using the same mask-IoU matching criteria and AP calculation procedure as those applied to native instance segmentation models.

The performance gains of MMDF-Net over semantic segmentation models, including U-Net, DeepLabV3+, PSPNet, and SegFormer, further indicate that building damage segmentation requires not only pixel-level delineation but also effective modeling of individual damage instances, fine-grained boundaries, and multiscale damage morphologies. Semantic segmentation methods primarily focus on class-level region prediction and may produce blurred boundaries, merged adjacent regions, or incomplete masks for weakly visible damage when cracks, stains, peeling, and wall textures interfere with one another. Although YOLO11n-seg benefits from a lightweight design, its RGB-only input limits its discriminative capability under complex illumination and low-contrast damage conditions.

Compared with RGB-only methods, RGB–IR methods generally achieve higher segmentation performance, suggesting that infrared images can provide complementary information to visible-light images. RGB images primarily provide information on color, texture, and geometric boundaries, whereas IR images can provide complementary regional response cues that differ from those available in visible-light images, thereby assisting the model in identifying anomalous regions under complex illumination, shadows, and low-contrast conditions. However, because RGB-only and RGB–IR models receive different input information, the observed performance differences may result from both the additional information provided by the infrared modality and differences in network architecture and fusion strategy. Therefore, the performance gains of MMDF-Net over the RGB-only models reported in Table 2 should be interpreted with caution.

A direct comparison among the RGB–IR methods provides a more controlled assessment of the performance of MMDF-Net. MMDF-Net achieves strong segmentation performance with only 7.8 M parameters and 23.5 GFLOPs. Compared with FEANet, MMDF-Net reduces the parameter count and computational cost by 72.7% and 50.9%, respectively; compared with CMX, the corresponding reductions are 78.9% and 61.7%. These results suggest that the performance improvement of MMDF-Net is associated primarily with its effective modeling of building damage characteristics rather than with an increase in model scale. Through complementary fusion of RGB and infrared information, MMDF-Net enhances the representation of weakly visible damage under complex surface conditions, while the modeling of crack topology and multi-scale damage morphology further supports the segmentation of elongated cracks, irregular peeling, Suspected Hollow boundaries, and erosion regions. The relatively low parameter count and computational cost also suggest the potential of MMDF-Net for deployment in resource-constrained engineering scenarios.

Overall, effective building damage instance segmentation requires the joint consideration of cross-modal complementarity, fine-grained boundary preservation, and multiscale damage representation. The experimental results demonstrate that MMDF-Net achieves a favorable balance among segmentation accuracy, recall, and computational cost, highlighting its potential for building inspection, damage localization, and maintenance assessment.

4.4.2. Class-Wise Performance Comparison

Table 3. Class-wise mask AP50 comparison.

Model	Cracks	Peeling	Suspected Hollow	Stains	Erosion	mAP50mask
Mask R-CNN	58.9 ± 0.49	62.5 ± 0.37	59.8 ± 0.58	64.0 ± 0.42	61.3 ± 0.45	61.3 ± 0.39
Cascade R-CNN	61.0 ± 0.44	64.6 ± 0.40	61.9 ± 0.51	65.7 ± 0.34	64.3 ± 0.39	63.5 ± 0.36
U-Net	52.8 ± 0.66	57.1 ± 0.51	54.3 ± 0.62	58.7 ± 0.47	57.6 ± 0.56	56.1 ± 0.50
DeepLabV3+	55.9 ± 0.57	60.0 ± 0.49	57.4 ± 0.54	62.1 ± 0.38	60.6 ± 0.45	59.2 ± 0.43
PSPNet	54.3 ± 0.61	58.5 ± 0.46	56.1 ± 0.59	60.3 ± 0.44	58.8 ± 0.52	57.6 ± 0.47
SegFormer	59.8 ± 0.41	63.6 ± 0.36	61.2 ± 0.47	64.8 ± 0.29	64.1 ± 0.39	62.7 ± 0.34
YOLOv8n-seg	62.0 ± 0.39	65.8 ± 0.31	63.4 ± 0.45	66.7 ± 0.34	66.1 ± 0.37	64.8 ± 0.33
YOLO11n-seg	63.9 ± 0.36	67.6 ± 0.33	65.4 ± 0.42	68.5 ± 0.28	68.1 ± 0.35	66.7 ± 0.31
FEANet	65.6 ± 0.34	69.0 ± 0.27	70.5 ± 0.39	67.8 ± 0.30	69.6 ± 0.32	68.5 ± 0.28
CMX	67.0 ± 0.29	70.2 ± 0.25	72.4 ± 0.35	69.0 ± 0.27	70.4 ± 0.30	69.8 ± 0.25
MMDF-Net	70.8 ± 0.25	72.5 ± 0.21	74.8 ± 0.28	70.9 ± 0.19	72.5 ± 0.23	72.3 ± 0.20

presents the class-wise mask mAP50 results of the evaluated models across five building damage categories. MMDF-Net achieves the highest mask mAP50 in all five categories—cracks, peeling, Suspected Hollow, stains, and erosion—demonstrating a consistent performance advantage across diverse damage morphologies.

The class-wise results indicate that Crack and Suspected Hollow are relatively challenging categories to segment. Cracks are typically characterized by elongated shapes, local discontinuities, and weak boundaries, making them prone to missed detections in the presence of wall textures, joints, and shadows. MMDF-Net achieves a mask mAP50 of 70.8% for the Crack category, compared with 65.6% for FEANet and 67.0% for CMX, indicating improved segmentation performance for fine and weakly visible crack structures. For the Suspected Hollow category, RGB-only methods achieve mask mAP50 values ranging from 54.3% to 65.4%, whereas RGB–IR methods achieve values ranging from 70.5% to 74.8%. This performance pattern indicates that IR information can provide useful complementary cues for weak regional anomalies represented in the annotations. However, because this category was annotated through joint interpretation of RGB and IR images without independent validation using nondestructive testing (NDT), some of the performance difference between the RGB-only and RGB–IR methods may be attributable to the consistency between the input modalities and the annotation procedure. Therefore, the results for this category should be interpreted as reflecting the agreement between the model predictions and the current RGB–IR-assisted annotations, rather than as physical confirmation of hollow regions. Under the same RGB–IR input conditions, MMDF-Net achieves a mask mAP50 of 74.8%, compared with 70.5% for FEANet and 72.4% for CMX, supporting the effectiveness of the proposed fusion architecture for this segmentation task.

Regional damage, including peeling, stains, and erosion, generally occupies a relatively distinct spatial extent but exhibits irregular boundaries and is readily affected by surface aging, color variations, and material textures. Semantic segmentation methods, such as U-Net, DeepLabV3+, and PSPNet, can capture the overall damaged regions but still tend to produce blurred region boundaries and lose local details. YOLOv8n-seg and YOLO11n-seg outperform the conventional semantic segmentation models, demonstrating the advantages of instance segmentation frameworks in handling multiple types of damage regions. However, their reliance on RGB input alone limits further performance gains under complex surface conditions.

Compared with multimodal methods such as FEANet and CMX, MMDF-Net maintains a performance advantage across all categories, indicating that building damage segmentation requires not only complementary multimodal information but also feature modeling tailored to different damage morphologies. Crack segmentation relies more heavily on topological continuity, hollow-area segmentation on the identification of weak-response regions, and peeling and erosion segmentation on the delineation of irregular boundaries, whereas stains can be confused with background textures. By integrating multimodal information with multiscale damage representations, MMDF-Net improves its robustness to variations in damage appearance.

Overall, the experimental results demonstrate the consistent performance advantage of MMDF-Net across multiple building damage categories. In particular, the model exhibits more pronounced advantages for categories characterized by weak boundaries, such as cracks and Suspected Hollow, demonstrating its ability to preserve fine-grained damage structures and improve segmentation reliability under complex building surface conditions.

4.5. Ablation Study

To evaluate the contribution of each module to the performance of building damage instance segmentation, YOLO11n-seg is used as the baseline model, and Simple Fusion, FDCB, CTAE, and MDDP are progressively introduced to construct a series of ablation variants. The results are presented in

Table 4. Ablation study of the proposed modules.

Method	Precisionmask	Recallmask	mAP50mask	mAP50-95mask	Params/M	FLOPs/G
Baseline	75.2 ± 0.36	70.6 ± 0.42	66.7 ± 0.31	58.6 ± 0.37	2.9	10.7
Baseline + Simple Fusion	71.1 ± 0.48	67.2 ± 0.52	62.3 ± 0.45	55.2 ± 0.49	3.8	12.6
Baseline + FDCB	77.6 ± 0.29	73.0 ± 0.34	68.9 ± 0.27	61.1 ± 0.32	4.3	14.8
Baseline + CTAE	76.8 ± 0.33	73.7 ± 0.39	68.2 ± 0.30	60.6 ± 0.35	3.7	13.4
Baseline + MDDP	77.0 ± 0.31	72.8 ± 0.36	68.6 ± 0.28	61.6 ± 0.33	4.8	16.9
Baseline + FDCB + CTAE	79.2 ± 0.26	75.8 ± 0.31	70.3 ± 0.24	63.1 ± 0.29	5.1	17.5
Baseline + FDCB + MDDP	79.5 ± 0.24	75.3 ± 0.29	70.6 ± 0.22	63.9 ± 0.27	6.2	20.9
Baseline + CTAE + MDDP	78.8 ± 0.28	75.5 ± 0.33	70.0 ± 0.25	63.4 ± 0.30	5.6	19.4
Baseline + FDCB + CTAE + MDDP	82.1 ± 0.22	78.6 ± 0.27	72.3 ± 0.20	65.6 ± 0.24	7.8	23.5

.

Among these variants, Simple Fusion is used to incorporate RGB–IR bimodal information. Specifically, Simple Fusion concatenates the RGB and IR features at corresponding scales along the channel dimension and then applies a 1 × 1 convolution to map the fused features back to the original channel dimension. The experimental results show that simply incorporating IR input does not necessarily improve model performance. Compared with the baseline model, the introduction of Simple Fusion decreases Precision, Recall, mAP50, and mAP50–95 by 4.1, 3.4, 4.4, and 3.4 percentage points, respectively. This suggests that, in the absence of effective cross-modal feature selection and interaction mechanisms, direct concatenation of RGB and IR features may introduce redundant information or inconsistent responses, thereby interfering with the discriminative representation of building damage features.

The single-module results show that all three modules consistently improve performance, indicating that they enhance complementary aspects of the baseline model’s damage segmentation capability. In particular, FDCB produces notable improvements in Precision and mAP50, indicating that it strengthens the complementary interaction between RGB and IR features and reduces segmentation errors associated with complex surface textures and illumination variations. CTAE produces a more pronounced improvement in Recall, indicating that it enhances the representation of elongated cracks, discontinuous boundaries, and weak-response damage regions, thereby reducing missed detections. MDDP yields the largest improvement in mAP50–95 among the single-module variants, indicating that its multiscale decoding structure improves mask quality across different IoU thresholds and facilitates more accurate segmentation of irregularly bounded and multiscale damage regions.

Further comparison of the two-module variants reveals strong complementarity among the three modules. Combining FDCB with CTAE improves both Precision and Recall, indicating that cross-modal feature enhancement and crack topology modeling jointly strengthen the discrimination and detection of damage regions. Combining FDCB with MDDP further improves mAP50 and mAP50–95, indicating that multiscale decoding facilitates more effective use of complementary multimodal information. The combination of CTAE and MDDP also produces consistent gains, indicating that crack topology modeling is compatible with multiscale damage representation and improves the segmentation of both fine-grained structures and regional damage.

When FDCB, CTAE, and MDDP are integrated, the complete model achieves the best results across all mask-based metrics. Compared with the baseline, the complete model achieves notable improvements in Precision, Recall, mAP50, and mAP50–95, demonstrating the complementarity of the three modules in cross-modal fusion, crack structure preservation, and multiscale damage decoding. Although the parameter count and FLOPs increase, they remain relatively low, suggesting that the performance gains arise from targeted damage feature modeling rather than merely from a substantial expansion of the network scale.

Overall, the ablation results validate the effectiveness of FDCB, CTAE, and MDDP. FDCB primarily improves multimodal feature fusion, CTAE strengthens the representation of elongated cracks and weak-boundary structures, and MDDP enhances mask decoding for multiscale damage regions. Through the combined effects of the three modules, MMDF-Net achieves a favorable balance between segmentation performance and model complexity while maintaining high Recall, highlighting its potential for multi-class damage instance segmentation on complex building surfaces.

4.6. Qualitative Comparison of Segmentation Results

To further compare the segmentation performance of different methods across typical building damage scenarios, Figure 6 presents qualitative results for representative samples. From top to bottom, the rows correspond to cracks, peeling, Suspected Hollow, stains, and erosion, respectively.

Overall, YOLO11n-seg identifies the main damage regions but remains limited in preserving fine details and suppressing background interference. In the crack scenario, it identifies the main crack but produces discontinuities along several branches and scattered false positives. In the peeling and erosion scenarios, the predicted boundaries deviate from the ground-truth contours, and some fragmented edges are not fully preserved. For damage types characterized by weak boundaries and substantial texture interference, such as Suspected Hollow and stains, YOLO11n-seg is more susceptible to wall textures, shadows, and structural edges, resulting in less stable mask predictions.

By integrating RGB and infrared information, CMX produces more complete masks than YOLO11n-seg, particularly for regional damage types such as peeling, Suspected Hollow, and erosion, where its predictions cover larger portions of the ground-truth regions. This result demonstrates the complementary value of multimodal information for damage identification on complex surfaces. However, CMX still produces boundary shifts and locally redundant responses in some samples. Discrepancies from the ground-truth annotations remain particularly evident along crack branches, around scattered Suspected Hollow, and at fragmented erosion edges.

In comparison, the predictions of MMDF-Net show closer agreement with the ground truth (GT). In the crack scenario, MMDF-Net better preserves the continuity of the main crack and its fine branches while reducing discontinuities and scattered false positives. In the peeling scenario, the predicted region is more spatially coherent, and its boundary more closely follows the ground-truth damage morphology. In the hollow-area scenario, the model more consistently identifies large, weak-texture anomalous regions and scattered local areas, with fewer segmentation errors caused by background textures. In the stain and erosion scenarios, the proposed method better distinguishes damaged regions from interfering elements, such as window frames, wall joints, and aging textures, while its predicted boundaries more closely follow the ground-truth damage distribution.

Overall, the qualitative results demonstrate that the proposed method produces more stable mask predictions across diverse damage morphologies. Its advantages are primarily reflected in preserving elongated-crack continuity, improving boundary alignment for regional damage, and suppressing redundant responses in complex backgrounds. These observations are consistent with the preceding quantitative results and further support the reliability of the proposed method for instance segmentation of multiple types of building surface damage.

5. Discussion

5.1. Contributions of This Study

This study proposes MMDF-Net for instance segmentation of building façade damage in paired RGB–IR imagery. Its central contribution is to show that targeted, damage-oriented multimodal modeling can improve façade damage segmentation without relying solely on increased network scale. Unlike methods based exclusively on RGB imagery, MMDF-Net jointly exploits visible-light texture cues and infrared responses to consistently segment cracks, peeling, Suspected Hollow, stains, and erosion against complex façade backgrounds.

First, FDCB introduces a frequency-decoupled cross-modal interaction mechanism. RGB features primarily emphasize high-frequency cues, such as crack edges, peeling contours, and local texture discontinuities, whereas IR features provide relatively smoother, low-frequency responses associated with material conditions or thermal anomalies. FDCB models these frequency-inspired components separately and enables cross-modal information exchange through bidirectional gating, thereby enhancing informative damage cues while suppressing redundant textures and irrelevant thermal responses. The ablation results show that FDCB improves overall segmentation performance, including Precision and mAP50, suggesting that selective interaction among different frequency-inspired responses may help reduce the interference of redundant information with damage discrimination.

Second, CTAE is designed to model crack-specific structural characteristics. Building cracks are typically fine, curved, branched, and locally discontinuous, making it difficult for conventional square convolutional kernels to preserve crack-path continuity. CTAE employs horizontal, vertical, and curvature-aware branches to enhance the representation of directional continuity and weak crack boundaries. The improvement in Recall suggests that CTAE may enhance the model’s ability to identify damage instances and reduce missed detections.

Third, MDDP is designed to decode damage masks across different scales and enhance information interaction across feature levels by integrating high-resolution details, intermediate-level structural information, and high-level semantic features. The ablation results show an improvement in mAP50–95 after the introduction of MDDP, suggesting that this module contributes to improved mask-level segmentation performance across a range of IoU thresholds.

Overall, FDCB, CTAE, and MDDP play complementary roles within MMDF-Net. FDCB enhances cross-modal damage-cue fusion, CTAE strengthens the representation of crack continuity and weak boundaries, and MDDP improves mask-level performance across multiple IoU thresholds. Together, these modules enable MMDF-Net to achieve strong instance segmentation performance with relatively low computational complexity.

5.2. Comparison with Existing Works

Compared with RGB-based semantic segmentation methods, including U-Net, DeepLabV3+, PSPNet, and SegFormer, MMDF-Net demonstrates clear advantages in instance-level building damage segmentation. Semantic segmentation models provide class-level region predictions but cannot inherently distinguish individual damage instances and may lose fine boundary details. In complex façade scenes, cracks and stains may be confused with wall joints, aging textures, and shadows, resulting in blurred boundaries, merged regions, or incomplete masks. MMDF-Net integrates instance-level prediction with multimodal fusion and structure-aware feature modeling, thereby mitigating these errors more effectively.

Compared with general-purpose instance segmentation models, including Mask R-CNN, Cascade R-CNN, YOLOv8n-seg, and YOLO11n-seg, MMDF-Net achieves superior mask performance while maintaining moderate model complexity. Two-stage instance segmentation methods provide strong instance-modeling capabilities but generally require more parameters and greater computational cost. The evaluated lightweight YOLO-based models offer efficiency advantages but use RGB-only inputs and remain susceptible to false positives and incomplete masks under low contrast, shadow interference, and complex surface textures. MMDF-Net retains the lightweight characteristics of the YOLO-style segmentation framework while incorporating complementary RGB–IR information and damage-oriented feature enhancement, thereby improving segmentation stability in complex façade scenes.

Compared with existing RGB–IR or RGB–T fusion methods, such as FEANet and CMX, the results of MMDF-Net suggest that generic multimodal fusion alone may be insufficient for building damage segmentation. General-purpose RGB–IR fusion networks are primarily designed for scene understanding and may not fully capture the distinctive morphologies and physical responses of façade damage. Building damage segmentation requires the joint modeling of fine crack topology, weak thermal or material responses, irregular peeling boundaries, and multiscale deterioration regions. MMDF-Net therefore tailors its cross-modal fusion and multiscale decoding modules to these damage characteristics. It consequently outperforms the larger RGB–IR baselines, suggesting that targeted feature organization contributes to the performance gains beyond network scale alone.

The class-wise results further support this interpretation. MMDF-Net achieves the highest mask mAP50 across all five damage categories, demonstrating a consistent advantage across different damage morphologies. Cracks are difficult to segment completely because of their elongated, discontinuous structures and weak boundaries. Stains may be confused with aging textures, shadows, and other forms of surface contamination. Hollow or cavity-like regions may depend more strongly on infrared cues because their regional response differences are not always apparent in RGB images. These category-specific characteristics indicate that different damage types depend to varying degrees on visible textures, infrared responses, and structural continuity; the results suggest that MMDF-Net effectively integrates these complementary cues.

5.3. Limitations and Future Work

Although the results are promising, several limitations remain. First, the dataset used in this study was collected from a limited range of building environments. Although it contains paired RGB–IR images covering five typical façade damage categories, the dataset may not fully represent variations in building materials, climatic conditions, façade textures, and seasons. Therefore, the generalization capability of MMDF-Net requires further validation on larger-scale RGB–IR datasets covering more diverse scenarios. Future work will extend the evaluation to different materials, such as masonry, timber, and steel, as well as to different types of engineering structures, such as industrial buildings and bridges, to further assess the model’s cross-material and cross-structure generalization capability and explore its potential as a general framework for engineering structural damage recognition and segmentation.

Second, the evaluation of crack topology and scale-dependent performance could be further strengthened. The current metrics—mask Precision, mask Recall, mask mAP50, and mask mAP50–95—effectively measure overall instance segmentation performance. However, these metrics do not directly quantify crack connectivity, skeleton continuity, or scale-specific segmentation performance. Future studies could incorporate topology-sensitive metrics, crack-skeleton evaluation, and AP analyses for small, medium, and large damage regions to assess CTAE and MDDP more precisely.

Third, the Suspected Hollow category should be interpreted with caution. In this study, hollow-like regions were annotated based on combined RGB–IR image evidence. Without independent nondestructive testing, such as hammer sounding, ultrasonic testing, ground-penetrating radar, or controlled thermographic validation, these regions should be regarded as suspected Suspected Hollow or surface-related anomalies rather than confirmed subsurface defects. Future studies should combine RGB–IR imaging with complementary structural inspection methods to improve annotation reliability and engineering interpretability.

Finally, this study reports the number of parameters and FLOPs as measures of computational complexity but does not include deployment tests on edge devices or operational UAV inspection systems. Accordingly, the current results demonstrate engineering potential rather than fully validated real-time deployment capability. Future studies should evaluate inference speed, memory consumption, robustness under field acquisition conditions, and integration with UAV-based inspection and BIM-based maintenance workflows.

In summary, MMDF-Net integrates frequency-aware multimodal fusion, crack topology encoding, and multiscale damage decoding within a unified framework for RGB–IR building façade damage instance segmentation. The results support the effectiveness of MMDF-Net under the evaluated test conditions. Future work should focus on cross-domain validation, mechanism interpretability, topology-aware evaluation, improved annotation reliability for Suspected Hollow, and practical.

6. Conclusions

This study presents MMDF-Net, a multimodal instance segmentation framework for assessing building façade damage. To address the weak contrast, irregular boundaries, discontinuous crack paths, and substantial scale variations encountered in practical façade inspections, the network architecture is tailored to the physical responses and morphological characteristics of building damage.

MMDF-Net comprises three core components. FDCB separates low-frequency material responses from high-frequency local damage details and complementarily enhances RGB and IR features through bidirectional gated interaction. CTAE combines direction-aware and curvature-aware topology encoding to strengthen the structural representation of fine, curved, branched, and locally discontinuous cracks. MDDP integrates high-resolution boundary details, intermediate-level structural cues, and high-level semantic context to improve mask prediction for fine cracks and large-area deterioration. Collectively, these components address the key challenges of RGB–IR façade damage segmentation, including cross-modal complementarity, crack-continuity preservation, weak-boundary identification, and multiscale damage representation.

On the RGB–IR building façade damage dataset developed in this study, MMDF-Net achieves the strongest overall mask-based performance and the highest class-wise mask mAP50 across cracks, peeling, Suspected Hollow, stains, and erosion. With 7.8 M parameters and 23.5 GFLOPs, MMDF-Net achieves a mask Precision of 82.1%, a mask Recall of 78.6%, a mask mAP50 of 72.3%, and a mask mAP50–95 of 65.6%. These results suggest that its performance gains stem from targeted modeling of damage characteristics rather than increased network capacity alone. The class-wise comparisons, ablation experiments, and qualitative results further show that the model preserves crack continuity more consistently, delineates damage boundaries more accurately, and suppresses redundant background responses in complex façade scenes.

Overall, the results show that the challenges of reliable building façade damage segmentation are not fully addressed by general-purpose instance segmentation frameworks or simple multimodal feature concatenation. For the task considered here, effective segmentation requires explicit modeling of the complementary relationships among visible textures, infrared responses, crack topology, and scale-dependent damage semantics. MMDF-Net provides a compact, task-oriented solution for RGB–IR façade damage instance segmentation and establishes a basis for future automated inspection, damage localization, and maintenance decision-making.