Transfer Learning Based on Multi-Branch Architecture Feature Extractor for Airborne LiDAR Point Cloud Semantic Segmentation with Few Samples

Abstract

The existing deep learning-based Airborne Laser Scanning (ALS) point cloud semantic segmentation methods require a large amount of labeled data for training, which is not always feasible in practice. Insufficient training data may lead to over-fitting. To address this issue, we propose a novel Multi-branch Feature Extractor (MFE) and a three-stage transfer learning strategy that conducts pre-training on multi-source ALS data and transfers the model to another dataset with few samples, thereby improving the model’s generalization ability and reducing the need for manual annotation. The proposed MFE is based on a novel multi-branch architecture integrating Neighborhood Embedding Block (NEB) and Point Transformer Block (PTB); it aims to extract heterogeneous features (e.g., geometric features, reflectance features, and internal structural features) by leveraging the parameters contained in ALS point clouds. To address model transfer, a three-stage strategy was developed: (1) A pre-training subtask was employed to pre-train the proposed MFE if the source domain consisted of multi-source ALS data, overcoming parameter differences. (2) A domain adaptation subtask was employed to align cross-domain feature distributions between source and target domains. (3) An incremental learning subtask was proposed for continuous learning of novel categories in the target domain, avoiding catastrophic forgetting. Experiments conducted on the source domain consisted of DALES and Dublin datasets and the target domain consists of ISPRS benchmark dataset. The experimental results show that the proposed method achieved the highest OA of 85.5% and an average F1 score of 74.0% using only 10% training samples, which means the proposed framework can reduce manual annotation by 90% while keeping competitive classification accuracy.

1. Introduction

Airborne Laser Scanning (ALS) point clouds acquired by an airborne Light Detection and Ranging (LiDAR) system have demonstrated widespread applications in high-accuracy Digital Elevation Model (DEM) production [1], three-dimensional (3D) city modeling [2], vegetation detection and forest parameters retrieval [3], power line detection [4], and many others. One of the prerequisites to apply a point cloud to the aforementioned fields is the extraction of the thematic information from the original laser scanning data. In recent years, deep learning-based methods have achieved superior results in mobile laser scanning (MLS) and terrestrial laser scanning (TLS) point cloud semantic segmentation [5]. These methods can be summarized into three main categories [6], which are shown in

Table 1. Summarization of point cloud semantic segmentation methods.

Methods	Brief Descriptions	Advantages and Limitations
MLP-based methods [12,13,14]	A shared multi-layer perceptron (MLP) is adopted as the core building block. Aggregating features are complemented by symmetric functions.	Simple and efficient, but underperforms when the point cloud contains complex local structures.
Convolution-based methods [15,16,17]	Inspired by the success of convolution kernels in image segmentation, they are adopted for point clouds in order to extract local geometries.	Mature deep learning models can be adopted, but this type of method requires converting the point cloud into a 2D/3D regular structure first by gridding/voxelization, and information loss is unavoidable during this process. Moreover, the input order of the point cloud may cause differences in the segmentation results.
Attention-based methods [18,19,20,21]	Long-range dependencies are modeled by attention mechanisms, which are suitable for handling point cloud irregularities.	Attention-based methods have achieved superior results, but this type of method has a high demand for computing resources, and a large amount of high-quality annotated data is required to ensure its high performance.

. A recent comprehensive survey on deep learning techniques for 3D point cloud analysis was provided by Ref. [7]. Though they originated from the semantic segmentation of MLS and TLS point clouds, these methods have been extended to ALS point cloud analysis with minor modifications [8,9,10,11].

While deep learning techniques outperform traditional classifiers in various applications, they rely heavily on large volumes of manually annotated data for training. However, annotation remains a labor-intensive task, particularly for ALS point clouds. ALS point cloud data is not only costly to collect but also requires specialized expertise for annotation, incurring substantial time and financial expenses. Furthermore, like other machine learning methods, deep learning demands consistency in data distribution and feature space between the training dataset and the target application dataset. Failure to meet this requirement leads to degraded model performance, making it challenging to extend pre-trained models to datasets with inconsistent distributions. This issue is prevalent in airborne LiDAR data acquisition projects, where different LiDAR systems are deployed across diverse topographies. Data from various sensors may share common parameters such as 3D coordinates (X, Y, Z) while differing in sensor-specific attributes like intensity and multi-echo information. Additionally, category definitions vary across projects, resulting in datasets with distinct distributions. Traditional machine learning classifiers address such distribution discrepancies by retraining models from scratch using new data that matches the testing dataset’s distribution [22]. In contrast, a more efficient approach is to transfer knowledge learned from one task to other similar tasks with limited training data—a concept at the core of transfer learning.

The origins of transfer learning can be traced back to Bozinovski’s work in the 1970s [23]. Since then, numerous studies have applied this framework across diverse domains [24], including natural language processing, audio processing, image processing, and multi-source data fusion. In these fields, transfer learning has consistently demonstrated advantages in both efficiency and performance [25].

Transfer learning has also been investigated for ALS point cloud semantic segmentation and classification, aiming at reducing the requirements of annotation. Ma H et al. [26] first introduced model transfer for ALS point cloud ground filtering, which is used to segment ground points and non-ground points. In this study, the trained model was transferred to new data without additional training. The effectiveness of traditional classifiers such as random forest was verified. However, the performance of deep learning methods is often superior to that of traditional classifiers and does not require manual feature design and selection [27]. Some works extend transfer learning to deep learning methods. Peng et al. [28] and Xie et al. [29] introduced Unsupervised Domain Adaptation (UDA), where models pre-trained in the source domain were transferred to a target domain through correlation alignment and domain adversarial learning. However, these methods assume that the source and target data share a consistent label and feature space, which is often not the reality. Zhao et al. [30] converted point clouds into 2D representations and utilize pre-trained 2D networks for classification. Due to the loss of complex 3D geometric information, the performance for large scenes was limited [31].

In order to address the issue of inconsistent labels between source and target domains while directly acting on 3D points to avoid information loss caused by projection, Dai et al. [32] proposed a cross-domain incremental learning method with few samples. In this study, a photogrammetric point cloud and ALS point clouds were selected as the source domain and target domain, respectively. A conditional domain adversarial network (CDAN) [33] was introduced for domain adaptation of base categories consisting of ground, vegetation, and building. Incremental learning was used to acquire knowledge of novel categories in the target domain. Although the authors provided a novel approach of transfer learning for ALS point cloud learning using few samples, the performance of the transferred model was not as perfect as expected, which may have been due to the significant discrepancies between the source and target domains. These discrepancies can be summarized as follows:

(1)

Different scenes from where a LiDAR system collects data. This affects the number of categories and the feature distribution of ground objects. More complex scenes contain more object categories, leading to a harder semantic segmentation task.

(2)

Inconsistent parameters recorded in the point clouds. In addition to the common XYZ values, photogrammetric point clouds generally include the colors RGB, while ALS data may contain intensity and echo-related parameters. When the parameters of the target domain and the source domain are different, the transferability of the features learned by the pre-trained model is poor, and directly applying the pre-trained model to the target domain may result in unsatisfactory performance.

(3)

Inconsistencies in point density. Due to the variety of flight altitudes and LiDAR systems used in practice, data collected can have significant differences in point density, ranging from a few points to several hundred points per square meter. The same ground object may exhibit various features under inconsistent point densities; ideally, learned features should be density-invariant.

To address these challenges and to utilize multi-source ALS data as the source domain, a novel Multi-branch Feature Extractor and a novel transfer learning strategy were proposed, aimed at improving the performance of the model and reducing the amount of annotated data. The proposed method addresses the following issues: (1) It fully utilizes the parameters contained in ALS point clouds to learn salient features for semantic segmentation. (2) It overcomes difficulties in model transfer caused by differences in types of parameters between source domain and target domain. (3) It deals with inconsistent categories between source domain and target domain during domain adaptation. The main contributions include:

(1)

A novel multi-branch architecture was proposed to learn heterogeneous features (such as geometric features, reflection features, and internal structural features). Based on this architecture, a feature extraction network named Multi-branch Feature Extractor (MFE) was proposed, where each branch consists of the proposed Neighborhood Embedding Block (NEB) and the existing Point Transformer Block (PTB). A pre-training strategy is expounded to address the difference in parameters between source and target domains: A specific branch was pre-trained and transferred through domain adaptation. The number of branches is adjustable to fit the scenarios where the source domain consists of multiple point cloud datasets with inconsistent parameters.

(2)

A Multi-class Domain Adversarial Network (MDAN) was proposed to transfer the pre-trained feature extractor from source domain to target domain. By setting up independent discriminators for each base category, it promotes positive transfer of the base categories between source domain and target domain, avoiding negative transfer caused by unique categories.

(3)

An incremental learning strategy consisting of a Knowledge Transfer Module (KTM) and a Feature Separation Module (FSM) was designed to enable the model to segment novel categories while avoiding catastrophic base category data loss. For base categories, a Knowledge Transfer Module (KTM) consisting of Kullback–Leibler (KL) divergence and feature similarity was used to transfer knowledge. For novel categories, an FSM that utilized orthogonal loss functions as constraints was proposed to learn unique features for distinguishing base categories.

The remainder of this paper is organized as follows: Section 2 describes the proposed Multi-branch Feature Extractor and transfer learning strategy for ALS point cloud semantic segmentation with few samples. Section 3 describes the experimental results, including datasets, evaluation metrics, segmentation results, and method comparison. Section 4 presents the ablation experiments and discussions. Section 5 provides the conclusions and future directions for our work.

2. Method

In this section, the proposed method is described from five aspects: problem definition, Multi-branch Feature Extractor (MFE), pre-training subtask, domain adaption subtask, and incremental learning subtask.

The workflow of the proposed method is illustrated in Figure 1. To provide a clearer description, the same categories in the source and target domains are defined as base categories, and the unique categories in the target domain are defined as novel categories. The detailed definition is presented in Section 2.1. Existing studies have shown that asymmetric mapping can better simulate the differences in low-level features than symmetric mapping [34]; therefore, the pre-training subtask was separated from the domain adaptation subtask. Another advantage is that when facing a new dataset, there is no need for additional pre-training. At the same time, due to the presence of novel categories in the target domain, the model should continue to learn and acquire the ability to segment novel categories. However, introducing novel categories into the model may lead to a sudden loss of knowledge regarding base categories—a phenomenon known as catastrophic forgetting in transfer learning [35]. To mitigate this issue, an incremental learning subtask is proposed to enable the model to segment novel categories while avoiding catastrophic forgetting.

Hence, the proposed method consists of three modules corresponding to three subtasks: pre-training subtask, domain adaptation subtask, and incremental learning subtask. In the pre-training subtask, multi-source ALS data was used to pre-train the Multi-branch Feature Extractor $F_s$, and semantic segmentation loss was used to constrain the learning process. In the domain adaptation subtask, target subdomain $D_{t\_b}$ with base categories were used to train $F_{t\_b}$, and weights of $F_{t\_b}$ were initialized by $F_s$. Multi-class Domain Adversarial Network (MDAN) was applied to learn domain invariant features. Semantic segmentation loss was used to learn the target domain’s unique features and train the classifier $G_{t\_b}$ of the target domain. In the incremental learning subtask, through KTM and FSM, target domain feature extractor $F_t$ and classifier $G_t$ were able to segment target domain data $D_{t\_n}$ with novel categories while retaining its ability to segment $D_{t\_b}$. The finally obtained $F_t$ and $G_t$ had the abilities of semantic segmentation for the whole target domain $D_t$.

2.1. Problem Definition

The benefits of transfer learning are generally believed to come from reusing the pre-trained feature hierarchy [36]. In the present study, features will be learned from a source domain that consists of inconsistent training datasets, and the learned features will be transferred to the downstream segmentation task, where there are novel categories. Therefore, the objective is to identify a feature representation that maximizes task relevance between the source and target domain while minimizing domain discrepancies. Considering the source domain $D_s=\left\{X_s, C_s\right\}$ and target domain $D_t=\left\{X_t, C_t\right\}$, both comprise the parameters X and labels C. The purpose of transfer learning is to transfer the pre-trained feature extractor$F_s$ to the target domain as $F_t$. The same categories in both source and target domains are regarded as base categories $C_b=C_s\cap C_t$, and the unique categories of the target domain are regarded as novel categories $C_n=C_t-C_b$. Therefore, the target domain can be decomposed into subdomains $D_{t\_b}=\left\{X_{t\_b}, C_b\right\}$ and $D_{t\_n}=\left\{X_{t\_n}, C_n\right\}$.

Armed with the above definition, the three subtasks shown in Figure 1 can be summarized and formulated as:

(1)

Pre-training subtask: Feature extractor $F_s$ was pre-trained in source domain $D_s$.

(2)

Domain adaptation subtask: For the base categories $C_b$, domain adaptation was employed to transfer the pre-trained $F_s$ adapted to the target domain as $F_{t\_b}$ in the subdomain $D_{t\_b}$: $F_s\stackrel{D_{t\_b}}{\to }{F}_{t\_b}$, and target classifier $G_{t\_b}$ was trained in the subdomain $D_{t\_b}$.

(3)

Incremental learning subtask: The feature extractor $F_t$ and classifier $G_t$ should be able to segment $C_n$ while retaining the ability to segment for $C_b$ in $D_t$: $F_{t\_b}\stackrel{D_t}{\to }{F}_t$, $G_{t\_b}\stackrel{D_t}{\to }{G}_t$.

2.2. Multi-Branch Feature Extractor (MFE)

Given that the source domain may comprise multi-source datasets with inconsistent categories, point densities, and other attributes, a Multi-branch Feature Extractor (MFE) based on the Point Transformer was designed, as illustrated in Figure 2. The input data was divided into different branches, such as a coordinate branch, an intensity branch, and an echo-related branch, according to the types of parameters they contained. Feature aggregation within each branch was conducted by extended Point Transformer with proposed Neighborhood Embedding Block (NEB). Finally, the features learned from multiple branches were concatenated and fed into subsequent processing, such as classifiers.

2.2.1. Neighborhood Embedding Block (NEB)

In the classical Point Transformer, embedded features are encoded via a shared-weight MLP before being fed into the Point Transformer Block, which is effective for the TLS and MLS point clouds. However, this encoding method is hard to extend to ALS point clouds. For instance, ground points may be distributed arbitrarily across a scene, meaning two points that are far apart in coordinate space could belong to the same category. In contrast, directly encoding coordinate parameters using a shared-weight MLP would lead to significantly different feature representations for points of the same category.

In traditional classifiers for airborne LiDAR point clouds, it is necessary to handcraft features from raw data and input them into a classifier [37]. Neighborhood-based features, such as eigenvalues, play a crucial role in point cloud classification. Drawing on this insight, we hypothesize that incorporating point neighborhood information into the embedding module can effectively enhance the overall performance of the network. As shown in Figure 3, the NEB includes three steps: Firstly, parameters of the current point were embedded using MLP1. Secondly, parameters of the K-nearest-neighboring points were embedded by MLP2. Finally, the two encoded features were aggregated to obtain the finally embedded features through MLP3.

The pseudo code for the NEB is shown in Algorithm 1:

Algorithm 1. Neighborhood Embedding Block

Input: ALS point cloud data $X^{N\times D_{in}}$, where N is the number of points, and$D_{in}$ is the dimension of the input. For instance, if the input is X, Y, and Z, then $D_{in}=3$. Output: Embedded features $F^{N\times D_{out}}$, where $D_{out}$ is the dimension of the embedded features. Initialization: P is point sets, K is the number of neighboring points. 1: For $p_i\left(i=1,2,3,\dots ,N\right)$ in P: 2:  Search for the K-nearest-neighboring points of the current point ($p_i$); the points within the neighborhood are denoted by $X_{p_i}^{K\times D_{in}}$. 3:  Obtain the original parameters of the current point ($p_i$) from point cloud data ($X^{N\times D_{in}}$) as $X_{p_i}^{D_{in}}$. 4:  Encode the current point using MLP1: $X_{p_i}^{D_{in}}\stackrel{MLP1}{\to }{F_o}_{p_i}^{D_{out}}$. 5:  Encode the neighborhood of the current point using MLP2: $X_{p_i}^{K\times D_{in}}\stackrel{MLP2}{\to }{F_n}_{p_i}^{D_{out}}$. 6:  Combine the encoded features to obtain the final features through MLP3: [${F_o}_{p_i}^{D_{out}}$ ${F_n}_{p_i}^{D_{out}}$]$\stackrel{MLP3}{\to }{F}_{p_i}^{D_{out}}$. 7: End for

2.2.2. Concept of Multi-Branch Architecture

In the existing deep learning methods for point cloud semantic segmentation, the most commonly selected parameters include coordinate values, intensity, echo-related parameters, etc. These parameters are fed to the network simultaneously for feature learning. Though simple in network design, due to the different characteristics of ground objects reflected by different parameters, it might lead the inconspicuous features weighing out the salient ones, thereby decreasing the final segmentation accuracy. Numerous studies explore the relationship between input parameters and segmentation accuracy [38,39]. However, these studies also adopt the simultaneous input of parameters, leading to severe coupling of the influences exerted by different parameters. This coupling hinders the ability to discern the relationship between learned features and ground objects.

Thus, to our knowledge, there is currently limited research on the impact of different parameters on ALS point cloud semantic segmentation. In order to more intuitively demonstrate the impact, t-Distributed Stochastic Neighbor Embedding (t-SNE) was employed to evaluate the effectiveness of the learned features. The experiments were conducted on the ISPRS benchmark dataset and Point Transformer. The preprocessing of input data was the same as that of existing deep learning methods. The network was trained using the training set and was applied to the testing set. The features input into the classifier were taken out for visualization, as shown in Figure 3. Four combinations were conducted separately:

(a)

Mixed parameters (including XYZ values, intensity, and echo-related parameters), which are also commonly input into current deep learning methods for ALS point cloud analysis;

(b)

Echo-related parameters, which reflect the internal structure of ground objects;

(c)

Intensity, which characterizes the backscattering coefficient of ground objects;

(d)

XYZ values, which record the three-dimensional coordinate information of ground objects and are also the most commonly used parameters in traditional machine learning methods.

From Figure 4, it can be seen that when mixed parameters are input, the learned features are somewhat separable among low_veg, imp_surf, tree, and roof. However, the boundaries between these categories remain indistinct, and the intra-class distances are large. For the other five categories, the learned features are not clearly distinguishable. When the echo-related parameter is used, low_veg and imp_surf can be distinguished from tree and roof, which may be due to differences in internal structure. When intensity is used, low_veg and imp_surf can be distinguished, mainly due to their different backscattering coefficients. When XYZ values are used, tree and roof can be distinguished, which may be due to the differences in their geometric structures. Additionally, advantageous features of cars and fac can be learned when intensity and XYZ values are input separately.

As revealed by the t-SNE, features learned from different parameters are capable of distinguishing between various ground objects. Furthermore, since source domain data originates from diverse datasets with inconsistent parameter configurations, the number of inputs for feature learning remains variable. To address this, a multi-branch architecture is introduced. A key characteristic of this structure is that each specific branch is assigned distinct inputs and tasked with learning corresponding specific features, with the final output features being utilized for subsequent processing.

2.2.3. Construction of MFE Based on Multi-Branch Architecture

With the aforementioned facts in mind, MFE was proposed based on extended Point Transformer and multi-branch architecture. MFE consists of three branches corresponding to three types of parameters. Each branch consists of NEB and PTB. The three branches are:

(a)

Coordinate branch: the input includes X, Y, and Z values. In the NEB, besides X, Y, and Z values, relative Z values of the K-nearest-neighboring points of the current point were input into the embedding features. Relative Z values were calculated because relative Z values are significant features to distinguish ground objects [40].

(b)

Intensity branch: The input is the intensity. In the NEB, the intensity of the current point and that of the K-nearest-neighboring points were input into the embedding features.

(c)

Echo-related branch: The input is the preprocessed echo-related parameters. In the NEB, echo-related parameters of the current point and those of the K-nearest-neighboring points were input into embedding features.

Preprocessing is necessary for the coordinate values, intensity and echo-related parameters, as was described in detail in Section 3.2. The value K was determined by trial and error.

2.3. Pre-Training Subtask

The purpose of the pre-training subtask is to pre-train the feature extractor with multi-source ALS data. The larger the amount of pre-training data, the better the generalization performance of the pre-trained model [41]. Thus, we aimed to utilize as much data as possible for pre-training. However, multi-source ALS data involves diverse types of parameters. To address this issue, we constructed a three-branch architecture for the feature extractor, specifically comprising a coordinate branch, an intensity branch, and an echo-related branch. Each branch consists of an MFE and an MLP-based classifier.

The overall framework of the pre-training subtask is illustrated in Figure 5. First, the source domain data is divided into square blocks. Given the potential impact of point density variations on model transferability, voxelization is performed prior to feature learning. During voxelization, one point is randomly selected from each voxel, and these selected points are fed into the MFE. After voxelization, the point cloud density primarily depends on the voxel size—an approach that effectively mitigates differences in point cloud density across multi-source ALS data. To achieve this, the voxel size is determined based on the lowest point density among the datasets in both the source and target domains.

The feature extractor was optimized through semantic segmentation loss function in each branch:

$$\underset{G_s, F_s}{argmin} L_{seg\_s}=-{{\displaystyle \sum }}_{i=1}^{C_b}{y}_{i}log\left(G_s\left(F_s\left(x_i\right)\right)\right),$$

(1)

where $y_i$ is the true label, $x_i$ is the input of each branch, $F_s$ is feature extractor, and $G_s$ is the MLP-based classifier.

The semantic segmentation loss in the pre-training subtask can be defined as:

$$L_{seg\_s}=L_{seg\_C}+L_{seg\_I}+L_{seg\_E},$$

(2)

Furthermore, appropriate branches are selected based on the shared parameter types between the target domain and source domain to learn transferable features. For instance, if the source domain lacks intensity data, only the coordinate branch and echo-related branch undergo pre-training in the subtask before being transferred to the target domain. As for the intensity branch, supervised training is conducted using the target domain training set.

2.4. Domain Adaption Subtask

In order to transfer the pre-trained $F_s$ to the target domain to obtain $F_{t\_b}$, two modules are proposed: (1) the Domain Adaptation Module, which aims to learn domain-invariant features, and (2) the Semantic Segmentation Module, which is designed to learn target domain-specific features and training target classifier $G_{t\_b}$. A detailed description of these two modules is provided below.

(1) Domain Adaptation Module

Domain adaptation is recognized as a special case of transfer learning [42]. In recent years, deep domain adaptation has been proposed to integrate the properties of deep networks with domain adaptation techniques. Domain-Adversarial Training of Neural Networks (DANN) [43] was the first to introduce adversarial learning into deep domain adaptation, which maps source and target domain data symmetrically into a new feature space to learn domain-invariant features. CDAN [33] introduced category information to allow the network to learn domain invariant features corresponding to categories. Dai [32] applied CDAN for ALS data to achieve domain adaptation. However, CDAN performs optimally when the source and target domains share identical categories. Thus, directly using CDAN may cause the following issues: (1) Classifying source unique categories and target novel categories into one category will reduce the ability of the feature extractor. (2) The source’s unique categories and target’s novel categories may lead to negative transfer. (3) It cannot be guaranteed that all samples used for domain adaptation obtain the correct category, and the misclassified samples may also lead to negative transfer.

To address the above issues, inspired by Multi-Adversarial Domain Adaptation (MADA) [44], a separate adversarial network named Multi-class Domain Adversarial Network (MDAN) was proposed, as shown in Figure 6 and Figure 7. From Figure 6, domain-adversarial training was applied to each branch of the pre-trained Multi-branch Feature Extractor, aligning the feature distribution through MDAN. As shown in Figure 7, in MDAN, one discriminator corresponds to one base category; hence, there is an equal number of discriminators and base categories. The advantage of setting multiple discriminators is to alleviate negative transfer by preventing false alignment of modes in different distributions across domains. In the implementation of MDAN, the output features $f_s$ of the source feature extractor and the output features $f_t$ of the target feature extractor were selected by labels separately to obtain $f_{s\_b}$, $f_{t\_b}$ as the input of MDAN. In MDAN, multiple domain discriminators were designed with the same number as $C_b$.$f_{s\_b}$, $f_{t\_b}$ were input into the MLP-based domain discriminator to obtain the domain output, and adversarial loss was calculated and backpropagated through Gradient Reverse Layer (GRL). Compared to MADA, our method has the following advantages: (1) freezing the pre-trained feature extractor helps to preserve the feature extraction abilities of the source domain and improve the ability to extract domain invariant features, and (2) using labels instead of category probabilities to determine which discriminator the sample should input can help with positive transfer, avoiding negative transfer caused by misclassification.

Based on the above analysis, the loss function within each domain discriminator is calculated as follows:

$$\underset{D}{argmin} \underset{F_{t\_b} }{argmax} L_{MDAN}^i={\displaystyle \sum }_{x_j\in D_s, x_k\in D_{t\_b}}{L}_D(D\left(F_{t_b}\left(x_j\right), d_j\right)+L_D(D\left(F_{t_b}\left(x_k\right),d_k\right)$$

(3)

where i represents the i-th domain discriminator, $L_D$ is the loss function, D is the MLP-based domain discriminator, and $d_j$ and $d_k$ are the domain labels.

Then, the total adversarial loss is:

$$\underset{D}{argmin} \underset{F_{t\_b} }{argmax} L_{adv\_MDAN}={{\displaystyle \sum }}_{j=1}^B{{\displaystyle \sum }}_{i\in C_b}{L_{MDAN}^i}_j,$$

(4)

where B is the number of branches.

To reduce computational complexity and avoid domain label imbalance caused by significant differences in data volume between the source and target domains, an equal number of blocks are randomly selected from both domains in each epoch for domain adversarial learning.

(2) Semantic Segmentation Module

After domain adaptation, $F_{t\_b}$ obtains the ability to extract domain invariant features. However, the target domain and the source domain are not exactly the same. In order to learn target domain specific features and train the target classifier $G_{t\_b}$, semantic segmentation loss is used, defined as follows:

$$\underset{G_{t\_b}, F_{t\_b}}{argmin} L_{seg\_base}=-{{\displaystyle \sum }}_{x_i\in D_{t\_b}}{y}_{i}log\left(G_{t\_b}\left(F_{t\_b}\left(x_i\right)\right)\right),$$

(5)

Therefore, the overall loss function of domain adaptation subtask can be defined as:

$$L_{DA}= L_{seg\_base}+ {\lambda }_1{L}_{adv\_MDAN},$$

(6)

where ${\lambda }_1$ is a learnable weight parameter.

2.5. Incremental Learning Subtask

After the domain adaptation subtask, $F_{t\_b}$ obtains the ability of feature extraction for $C_b$. Due to the presence of novel categories, the feature extractor should continue to learn for $C_n$. If only $C_n$ is used for learning, the feature extractor would inevitably fall into catastrophic forgetting of $C_b$. If $C_n$ and $C_b$ are used to train the $F_t$ and $G_t$ together, the model would lose its generalization ability to $C_b$ because the source domain is no longer visible. To address this issue, an incremental learning strategy was proposed that integrates knowledge transfer and feature separation. This strategy allows the model to learn the segmenting ability of $C_n$ while avoiding catastrophic forgetting for $C_b$ and retaining the generalization ability learned from the source domain. This subtask consists of two modules: (1) the Knowledge Transfer Module for $C_b$ and (2) the Feature Separation Module for $C_n$. Next, we will provide a detailed description of these two modules.

(1) Knowledge Transfer Module (KTM)

The purpose of knowledge transfer is to enable the student model (SM) to retain the feature extraction capability of the teacher model (TM). Dai [32] introduced KL divergence as $L_{KL}$ into incremental learning to avoid catastrophic forgetting. However, the dimension of the classifier’s output layer was expanded, which could lead to the two identical points having different category probability distributions, and it could reduce the performance of knowledge transfer. Therefore, in addition to $L_{KL}$, feature similarity was introduced as a constraint to overcome these differences. According to [45], Manhattan distance has better similarity measurement in prototype networks than Euclidean distance, cosine distance, and Chebyshev distance. At the same time, it is more stable. Therefore, for feature similarity measurement, Manhattan distance was chosen instead of other distances. Then, the loss function of feature similarity can be defined as:

$$\underset{F_t, G_t}{argmin} L_{feat}={\displaystyle \frac{1}{N}}{{\displaystyle \sum }}_{x_i\in D_{t\_b}}^N\left|F_t\left(x_i\right)-F_{t\_b}\left(x_i\right)\right|,$$

(7)

Then, the loss function in KTM consists of KL divergence $L_{KL}$ and feature similarity $L_{feat}$.

$$L_{KT}=L_{KL}+{\lambda }_2{L}_{feat},$$

(8)

where ${\lambda }_2$ is a learnable weight parameter.

(2) Feature Separation Module (FSM)

Due to the presence of novel categories $C_n$, Dai [32] introduced an adversarial learning module, hoping for the model to learn distinguished features between $C_b$ and $C_n$. However, the adversarial learning is not stable, and additional domain discriminators will increase the computational complexity. To solve this problem, the idea of feature separation was introduced using orthogonal constraint loss functions as a constraint to expect the network to learn unique and separable features for $C_n$. Assuming that the features of $C_b$ are orthogonal to $C_n$, the classifier can independently activate both types of features through linear transformation. Orthogonal constraints are established between the output features of $F_t$ and $F_{t\_b}$.

$$\underset{ F_t}{argmin} L_{Orth}=\Vert F_{t\_b}{\left(x_i\right)}^T{F}_t{\left(x_i\right)\Vert }_{Fro}^2, x_i\in D_{t\_n},$$

(9)

In order to make the learned features more effective for $C_n$, semantic segmentation loss was used as an auxiliary:

$$\underset{G_t, F_t}{argmin} L_{seg\_novel}=-{\displaystyle \sum }_{x_i\in D_{t\_n}}{y}_{i}log\left(G_t\left(F_t\left(x_i\right)\right)\right)$$

(10)

Then, the loss function in FSM consists of $L_{Orth}$ and $L_{seg\_novel}$:

$$L_{FS}=L_{Orth}+ {\lambda }_3{L}_{seg\_novel},$$

(11)

where ${\lambda }_3$ is a learnable weight parameter.

In summary, the total loss function of incremental learning subtask can be defined as:

$$L_{IL}=L_{KT}+{\lambda }_4{L}_{FS},$$

(12)

where ${\lambda }_4$ is a learnable weight parameter.

Pseudocodes of the three-stage transfer learning strategy are provided as shown in Algorithm 2:

Algorithm 2. Three-stage transfer learning strategy

Pre-training subtask

Input: Source domain $X^{N_s}$, where $N_s$ is the number of total blocks after gridding the source domain. Output: Pre-trained ${\mathrm{MFE}}_P$ consists of Multi-branch Feature Extractor $F_s$ and an MLP-based classification layer within each branch. The number of branches is set to 3, namely coordinate branch MFE-C, intensity branch MFE-I, and echo-related branch MFE-E. 1: For each block $X_i$ in $X^{N_s}$: 2:  Train ${\mathrm{MFE}}_P$-C, ${\mathrm{MFE}}_P$-I and ${\mathrm{MFE}}_P$-E using $X_i$ and calculate $L_{seg\_s}$. 3:  Backpropagate $L_{seg\_s}$ and update ${\mathrm{MFE}}_P$. 4: End for

Domain adaptation subtask

Input: Source domain $X^{N_s}$ and target domain with labeled base categories $X^{N_{t\_b}}$, where $N_s$ and $N_{t\_b}$are the number of blocks after gridding the source and target domain, respectively. Frozen pre-trained ${\mathrm{MFE}}_P$. MDAN consists of k MLP-based domain discriminators, where k is the number of base categories. Output: Domain-adapted ${\mathrm{MFE}}_{DA}$ consists of Multi-branch Feature Extractor $F_{t\_b}$ and MLP-based classification layer $G_{t\_b}$. 1: For $i$ in [1, 2, 3, …,$N_{t\_b}$]: 2:  Given a block $X_s^i$ from source domain and a block $X_t^i$ from target domain. 3:  For each branch $j$ in {MFE-C, MFE-I, MFE-E}: 4:   Features $f_t$, $f_s$ are obtained by: $X_t^i\stackrel{F_{t\_bj}}{\to }{f}_t$, $X_s^i\stackrel{F_{sj}}{\to }{f}_s$. 5:   For each base category k: 6:    Domain labels $d_k$ are obtained by $MDA{N}_k$: [$f_{t\_k}$, $f_{s\_k}$]$\stackrel{MDA{N}_k}{\to }$ $d_k$. 7:    Loss $L_{MDAN}^k-j$ is calculated. 8:   End for 9:  Loss $L_{adv\_MDAN}$ is calculated by: $L_{adv\_MDAN}={\displaystyle \sum }_j{\displaystyle \sum }_k{L}_{MDAN}^k-j$. 10: End for 11: Train ${\mathrm{MFE}}_{DA}$ using $X_t^i$ and calculate the segmentation loss $L_{seg\_base}$. 12: $L_{DA}= L_{seg\_base}+ {\lambda }_1{L}_{adv\_MDAN}$. 13: Backpropagate $L_{DA}$, updating ${\mathrm{MFE}}_D$ and MDAN. 14: End for

Incremental learning subtask

Input: Target domain $X^{N_t}$, where $N_t$ is the number of total blocks after gridding the target domain. Frozen domain adaptated ${\mathrm{MFE}}_{DA}$. Output: ${\mathrm{MFE}}_T$ consists of Multi-branch Feature Extractor $F_t$ and MLP-based classification layer $G_t$. 1: For each block $X_i$ in $X^{N_t}$: 2:  For points with labeled base categories: 3:   Features $f_{t\_b}^{’}$ and logists $g_{t\_b}^{’}$ are obtained by ${\mathrm{MFE}}_{DA}$: $X_i\stackrel{{\mathrm{MFE}}_{DA}}{\to }\left[f_{t\_b}^{’}, g_{t\_b}^{’}\right]$. 4:   Features $f_{t\_b}$ and logists $g_{t\_b}$ are obtained by ${\mathrm{MFE}}_T$: $X_i\stackrel{{\mathrm{MFE}}_T}{\to }\left[f_{t\_b}, g_{t\_b}\right]$. 5:   $L_{KL}$ is calculated by $g_{t\_b}$ and $g_{t\_b}^{’}$; $L_{feat}$ is calculated by $f_{t\_b}$ and $f_{t\_b}^{’}$. 6:   $L_{KT}=L_{KL}+ {\lambda }_2{L}_{feat}$. 7:  End for 8:  For points with labeled novel categories: 9:   Features $f_{t\_n}^{’}$ are obtained by ${\mathrm{MFE}}_{DA}$: $X_i\stackrel{F_{t\_b}}{\to }{f}_{t\_n}$. 10:   Features $f_{t\_n}$ and logists $g_{t\_b}$ are obtained by $MF{E}_T$: $X_t^i\stackrel{MF{E}_T}{\to }\left[f_{t\_b}, g_{t\_b}\right]$. 11:   $L_{Orth}$ is calculated by $f_{t\_n}$ and $f_{t\_n}^{’}$; segmentation loss $L_{seg\_novel}$ is calculated. 12:   $L_{FS}=L_{Orth}+ {\lambda }_3{L}_{seg\_novel}$. 13: End for 14:$L_{IL}=L_{KT}+{\lambda }_4{L}_{FS}$ 15: Backpropagate $L_{IL}$ and update ${\mathrm{MFE}}_T$. 16: End for

3. Experiments

In this chapter, we will introduce the experimental results from three aspects: datasets, evaluation metrics, and experimental results.

3.1. Datasets and Evaluation Metrics

Three datasets were used to evaluate the proposed model: the International Society for Photogrammetry and Remote Sensing (ISPRS) benchmark dataset [46,47], the Dublin City Annotated LiDAR Point Cloud dataset [48], and the Dayton Annotated LiDAR Earth Scan (DALES) dataset [49]. All these datasets were acquired via airborne LiDAR systems. Specifically, the DALES and Dublin datasets served as the source domains; the training set of the ISPRS benchmark dataset was used for domain adaptation and incremental learning, while its testing set was employed to evaluate the proposed method. Detailed information on these three datasets is presented in

Table 2. Detailed information of the three datasets.

Domain	Target Domain	Source Domain
Dataset	ISPRS benchmark dataset	DALES dataset	Dublin dataset
Density	Above 4 pts/m2	Above 50 pts/m2	>150 pts/m2
Parameter type	XYZ values Intensity Echo-related parameters	XYZ values Echo-related parameters	XYZ Values intensity Echo-related parameters
Categories	Powerlines, low_veg, imp_sur (ground), cars, fence/hedge (fences), roof (building), fac, shrub, tree (vegetation)	Ground, vegetation, cars, trucks, powerlines, poles, fences, building	Building, ground and vegetation
Base categories	Ground, vegetation, car, building, fence, powerlines
Novel categories	low_veg, fac, shrub	-	-

.

Three evaluation metrics were adopted to assess the segmentation results: F1 score (F1), average F1 (averF1), intersection over union (IoU), and overall accuracy (OA). Specifically, F1, averF1, and OA are used to compare with other fully supervised deep learning methods, while IoU is employed to compare with other few sample learning methods.

A paired t-test was used to examine significant differences between experimental results. In the ablation experiments, multiple trials were conducted, and the p-values of two-tailed tests were reported. If p < 0.05, it indicates that there is a statistically significant difference between the two compared models.

3.2. Experimental Results

3.2.1. Processing Details

1.

Data preprocessing

All three datasets were divided into blocks of 50 × 50 m, and coordinate values were centralized in each block. Due to the large range of the centralized coordinate values, intensity, and echo-related parameters, they were preprocessed by the following methods before being input into the network in order for the network to learn normally:

• Coordinate values: Normalized after centralization, after which the range of value is 0~1.
• Intensity: Normalization, after which the range of value is 0~1.
• Echo-related parameters: Existing deep learning methods normalize the return number (RN) and number of returns (NR) separately [50]. Other studies pointed out that NR is positively correlated with the complexity of the internal structure [38]. Therefore, by combining RN and NR, a novel normalization method for echo-related parameters was proposed as follow:

$$F_e=\left\{\begin{array}{c} 0 , when NR=1\\ {\displaystyle \frac{RN}{NR}}, otherwise \end{array}\right.,$$

(13)

The range of $F_e$ is [0, 1], where 0 indicates that the ground object cannot be penetrated. The closer the value is to 1, the more complexity there is of the internal structure of the object from where the laser beam reflected.

2.

Settings of pre-training subtask

Both the training set and testing set of the DALES dataset and Dublin dataset were used to pre-train MFE. A voxel size of 0.5 m was selected. The pre-training situation for each branch of MFE was as follows:

(1)

Coordinate branch: Pre-trained using the DALES dataset and Dublin dataset.

(2)

Echo-related branch: Pre-trained using the DALES dataset and Dublin dataset.

(3)

Intensity branch: Pre-trained using the Dublin dataset.

3.

Settings of domain adaptation subtask and incremental learning subtask

The training set of the ISPRS benchmark dataset was used in the domain adaptation subtask and incremental learning subtask. In order to evaluate the impact of different sampling proportions on the model, 10%, 1%, and 0.1% samples of the training set were selected through random sampling in each category, and the number of samples in each category was not less than 1. The number of samples is shown in

Table 3. Different sampling proportions in the training set of the ISPRS benchmark dataset.

Proportion	Base Categories						Novel Categories
	Power	Imp_Surf	Car	Fence	Roof	Tree	Low_Veg	Fac	Shrub
100%	546	193,723	4614	12,070	152,045	135,173	180,580	27,250	47,605
10%	55	19,383	462	1207	15,205	13,518	18,058	2725	4761
1%	6	1939	47	121	1521	1352	1806	273	477
0.1%	1	194	5	13	153	136	181	28	48

.

For the domain adaptation subtask, samples with base categories were used. For the incremental learning subtask, samples with base categories were used in the Knowledge Transfer Module, and samples with novel categories were used in the Feature Separation Module.

After the three subtasks, feature extractor $F_t$ and classifier $G_t$ were evaluated on the testing set of the ISPRS benchmark dataset.

4.

Other settings

In all three subtasks, the Adam optimizer with an initial learning rate of 0.0005 was used, the momentum value was 0.9, and the batch size was 4. The learning rate was iteratively reduced based on the current epoch by a factor of 0.7. In the pre-training subtask, the training epoch was 30. In the domain adaptation subtask and incremental learning subtask, the training epoch was 300.

3.2.2. Comparison of Experimental Results

The proposed method was evaluated from the following two aspects: (1) comparison with other fully supervised deep learning methods and (2) comparison with other few sample learning methods.

1.

Comparison with other fully supervised deep learning methods

Table 4. Comparison of segmentation results using the ISPRS benchmark dataset with other fully supervised deep learning methods. The segmentation accuracy of each category was evaluated by F1 and is expressed in percentage. The highest values of F1, OA, and averF1 are marked with bold text, and the second highest values are marked with underlines.

Method	Power	Low_Veg	Imp_Surf	Car	Fence	Roof	Fac	Shrub	Tree	OA	averF1
PointNet++ (2017) [14]	57.9	79.6	90.6	66.1	31.5	91.6	54.3	41.6	77.0	81.2	65.6
KPConv (2019) [51]	63.1	82.3	91.4	72.5	25.2	94.4	60.3	44.9	81.2	83.7	68.4
DANCE-NET (2020) [52]	68.4	81.6	92.8	77.2	38.6	93.9	60.2	47.2	81.4	83.9	71.2
GANet (2020) [53]	75.4	82.0	91.6	77.8	44.2	94.4	61.5	49.6	82.6	84.5	73.2
GraNet (2021) [18]	67.7	82.7	91.7	80.9	51.1	94.5	62.0	49.9	82.0	84.5	73.6
RFFS-NET (2022) [54]	75.5	80.0	90.5	78.5	45.5	92.7	57.9	48.3	75.7	82.1	71.6
DGCN-ED (2023) [55]	72.6	82.9	92.3	75.9	41.8	92.1	60.9	43.4	79.5	83.5	71.3
TDConvs (2024) [56]	67.0	82.4	91.6	84.7	48.7	94.2	63.3	46.9	81.7	84.5	73.4
MIA-Net (2024) [57]	65.8	79.5	89.7	71.1	26.2	94.0	63.5	48.1	82.8	83.3	71.8
Ours (10%)	72.9	83.1	91.9	83.4	42.1	95.4	63.9	51.7	82.8	85.5	74.0
Ours (1%)	65.2	81.6	90.3	66.1	31.6	94.1	60.1	43.5	82.6	83.7	68.4
Ours (0.1%)	59.6	80.2	89.7	62.9	26.1	92.5	58.3	36.1	80.7	81.8	65.1
Ours (100%)	76.3	83.4	91.8	83.2	37.6	95.5	68.4	52.8	83.9	85.7	74.7

shows the semantic segmentation results of the proposed method in comparison with other fully supervised deep learning methods published in recent literature. The thematic map of the segmentation result using 10% training samples and the ground truth are shown in Figure 8, while Figure 9 illustrates the difference between them. Figure 10 displays the zoomed-in map, which shows that some facades were misclassified as roofs, and some confusion also occurred between low_veg and fences, as well as between trees and roofs. These errors can be attributed to the close similarity in geometric and internal structural characteristics between these classes, making it difficult to learn distinctive features for their differentiation.

From Table 4, it can be seen that the proposed method achieved the highest OA and averF1, and achieved the highest F1 score in five out of nine categories. Among the six base categories, the segmentation performance was highly competitive, except for fence. This may have been due to the low point density of the ISPRS benchmark dataset, which resulted in incomplete morphology of the fence, and the proposed MFE relied on local features, leading to an inferior result for this category. Notably, the proposed method achieved the best performance in three novel categories (low_veg, fac, and shrub). This indicates that the method can effectively transfer knowledge learned from the source domain to the target domain, thereby enhancing the model’s generalization ability.

Meanwhile, the impact of various proportions of samples on semantic segmentation was compared. When using 1% of training samples, the model still achieved competitive results compared to other fully supervised deep learning methods. This was mainly due to the strong generalization ability of the model acquired from the source domain, which allowed the model to avoid overfitting, thereby maintaining the performance of the model with fewer target training samples. This greatly reduced the cost of manual annotation. This means that when using 0.1% of training samples, each category had only a few to less than 200 training samples, and the results deteriorated. Nevertheless, the method still achieved favorable performance for major ground objects such as imp_surf, tree, and roof, demonstrating its practical value in large-scale applications. By comparing the results of different sampling proportions, it can be seen that compared to using 100% training samples, using 10% training samples kept competitive classification accuracy while reducing manual annotations by 90%.

2.

Comparison with other few sample learning methods

To evaluate the proposed method for few sample learning, it was compared with Thr-MPRNet and Dai’s method [32]. In order to obtain more objective comparisons, the same dataset, SensatUrban [58], and sampling method as Dai [32] were adopted. A brief overview of the sampling method in Ref. [32] is as follows: (1) The training set is divided into grids with overlapping regions. (2) The class of each “representative block” is defined based on the proportion of points within each grid. If the number of points belonging to the selected category within a block exceeds a certain threshold, then the block is considered the “representative block” of this category. (3) In each “representative block,” N points are sampled and their annotation information is retained. A more detailed description of the sampling method can be found in the original paper.

Ground, vegetation, and building were taken as base categories. As the SensatUrban dataset contains coordinate values XYZ and color information RGB, in the pre-training subtask, only the coordinate branch was set up and pre-trained using coordinate values XYZ, while RGB information was abandoned. The sampled training set of the ISPRS benchmark dataset was used to train the intensity branch and echo-related branch in the domain adaptation subtask. The detailed settings are shown in

Table 5. Experimental settings for comparison with other few sample learning methods.

Domain		Target Domain	Source Domain
Dataset		ISPRS benchmark dataset	SensatUrban
Parameter type		XYZ values Intensity Echo-related parameters	XYZ values RGB
Number of categories		9	12
Base categories		Ground, vegetation, building
Novel categories		Power, low_veg, car, fence, fac, shrub	—
Settings of Multi-branch Feature Extractor	Components	(1) Coordinate branch, (2) intensity branch, (3) echo-related branch
	Pre-training subtask	_	Coordinate branch: pre-training using XYZ values of SensatUrban dataset
	Domain adaptation subtask	(1) Coordinate branch: domain adaptation using sampled training set with base categories (2) Intensity branch and echo-related branch: supervised learning using sampled training set with base categories	-
	Incremental learning subtask	Coordinate branch, intensity branch, and echo-related branch: incremental learning using sampled training set with base categories and novel categories	-

.

As shown in

Table 6. Comparison of segmentation results with other few sample learning methods. The segmentation accuracy of each category was evaluated by IoU. The highest values of IoU, OA, and mIoU are marked with bold text.

Method	Power	Low_Veg	Imp_Surf	Car	Fence	Roof	Fac	Shrub	Tree	OA	mIoU
Thr-MPRNet (2024) [37]	0.0	60.58	70.6	8.6	1.7	57.8	12.08	17.4	44.7	44.1	32.4
Dai (2025) [32]	0.0	43.9	84.0	12.0	1.5	47.2	15.6	18.2	59.6	48.1	37.3
Ours	51.8	70.3	84.6	58.8	27.7	89.1	44.3	28.0	67.9	84.2	58.1

, Thr-MPRNet and Dai’s methods achieved inferior results, which may have been due to (1) difficulty in transferring the pre-trained model to the target domain due to the difference in parameter type between the source and target domains. Simply embedding the parameters through the MLP-based embedding layer can lead to poor transferability of the learned features. (2) There was a significant difference in point density between the source and target domains, which may have resulted in significant differences in local features. This is not conducive to the transfer of features. The proposed Multi-branch Feature Extractor and transfer learning strategy seem likely to solve these two problems effectively.

4. Ablation Experiments and Discussions

The experimental results show the effectiveness of the proposed method. Ablation experiments were conducted to further study the impact of each component of the proposed method. We will separately discuss the proposed NEB, multi-branch architecture, and transfer learning strategy.

4.1. Discussions of Neighborhood Embedding Block

Experiments were performed to assess the effectiveness of NEB and compare the impact of different K-nearest-neighbor sizes on model performance. To eliminate interference from the multi-branch architecture, only the coordinate branch was adopted. The experimental results are presented in

Table 7. Effect of different K-nearest neighboring on the NEB measured by OA, averF1, and p-values.

Method	OA	p-Values (OA)	averF1	p-Value (averF1)
Base	80.1 (±0.18)	-	62.9 (±0.74)	-
NEB (K = 4)	80.9 (±0.26)	0.009	65.6 (±0.72)	<0.0001
NEB (K = 8)	81.2 (±0.24)	0.0005	66.7 (±0.83)	0.001
NEB (K = 16)	81.8 (±0.28)	0.0008	67.1 (±0.76)	0.0006
NEB (K = 32)	81.7 (±0.21)	0.0006	66.53 (±0.77)	0.005
NEB (K = 64)	80.5 (±0.22)	0.04	63.2 (±0.72)	0.49

, where “Base” denotes the original Point Transformer utilizing an MLP-based embedding block. For each K-nearest-neighbor setting, the experiment was repeated five times, with the averages and ranges of F1 scores, OA, and averF1 reported, along with p-values for OA and averF1 in comparison to the “Base”. As shown in the table, incorporating neighborhood information into NEB can effectively enhance model performance with increasing K values. However, when K exceeds 16, the model performance starts to decline. This phenomenon may be attributed to the fact that a larger K might encode features from other categories, leading to inconsistent embedded features among points of the same category and thus hindering subsequent feature learning and overall model performance. When K < 64, both p-values for OA and averF1 are less than 0.05, indicating statistically significant improvements. In contrast, when K = 64, the p-value for averF1 exceeds 0.05, suggesting that the improvement is not statistically significant.

4.2. Discussions of Multi-Branch Architecture

As mentioned before, the two purposes of multi-branch architecture in the feature extractor are (a) to learn more representative features and (b) to deal with variable parameters contained in a point cloud dataset. In this subsection, an ablation experiment was conducted to show the effectiveness of the first purpose. In the ablation experiment, the descriptions of the two models used are shown in

Table 8. Models with different architecture.

Model	Description
M1	Model consists of an MLP-based embedding layer, Point Transformer Block (PTB), and MLP-based classifier. The input data is a 5-dimensional vector that includes 3 coordinate values, intensity, and an echo-related parameter.
M2	Model consists of a three-branch feature extractor that includes a coordinate branch, an intensity branch, and an echo-related branch. Each branch consists of an MLP-based embedding layer and PTB. Features from each branch were concatenated and then input into an MLP-based classifier to obtain segmentation results.

. To avoid the impact of NEB, an MLP-based embedding layer was used for feature embedding. The flowchart of the model is shown in Figure 11.

The experimental results are presented in

Table 9. Ablation experiment of multi-branch architecture and single-branch architecture. The segmentation effectiveness of each category was evaluated by F1 score and is expressed in percentage.

Method	Power	Low_Veg	Imp_Surf	Car	Fence	Roof	Fac	Shrub	Tree	OA	averF1	p-Value (OA)	p-Value (averF1)
M1	50.2 (±5.41)	80.4 (±0.28)	91.0 (±0.25)	67.4 (±1.94)	21.6 (±5.15)	92.8 (±0.25)	56.7 (±0.48)	46.7 (±0.43)	80.5 (±0.28)	82.4 (±0.28)	65.3 (±0.82)	-	-
M2	67.1 (±4.82)	82.3 (±0.34)	91.2 (±0.23)	70.5 (±2.17)	33.6 (±3.24)	94.5 (±0.21)	60.7 (±0.52)	49.3 (±0.38)	80.8 (±0.31)	83.7 (±0.26)	69.8 (±0.78)	0.0008	0.0007

. Each experiment was conducted five times, with the averages and ranges of the F1 scores, OA, and averF1 reported, along with p-values for OA and averF1 comparing M1 and M2. Both p-values (for OA and averF1) were less than 0.05, indicating a statistically significant difference between M1 and M2. As observed, the multi-branch architecture significantly improved performance across all nine categories. Among them, the segmentation accuracy of powerlines and fences showed the most substantial increase, which can be attributed to the separate echo-related branch effectively extracting their internal structural features. There was also a noticeable improvement in low_veg segmentation, possibly due to the intensity branch’s capability to extract distinguishable features. For buildings, the enhanced segmentation accuracy may stem from the unique geometric features extracted by the coordinate branch. Combined with the t-SNE analysis in Section 2.3 and the experimental results, the multi-branch architecture is confirmed to be superior to the single-branch counterpart. This superiority may be attributed to the fact that different parameters correspond to distinct physical characteristics, enabling the multi-branch architecture to more effectively learn features that distinguish various ground objects.

To evaluate the time complexity of the proposed architecture, M1 and M2 were executed on the testing set of ISPRS benchmark dataset five times and the running time was separately recorded, as shown in

Table 10. Time complexity measured by running time of M1 and M2 on the testing set of ISPRS dataset.

Method	Maximum (Second)	Minimum (Second)	Average (Second)
M1	28.7	31.9	30.7
M2	55.3	61.4	58.6

. Though the multi-branch architecture almost doubles the running time in average compared to the single branch architecture, it is still in the acceptable range. But the most important is that it can effectively solve the issue of parameters differences between the source and target domains, therefore, greatly improve the performance of model transfer.

4.3. Discussion of Domain Adaptation Subtask and Incremental Learning Subtask

To demonstrate the effectiveness of domain adaptation subtask and incremental learning subtask, comparative experiment and ablation experiment were conducted. 10% samples of the target training set were used for domain adaptation and incremental learning. The trained model was evaluated on target testing set through segmentation performance.

(1) Discussion of Domain Adaptation Subtask

To verify the effectiveness of the proposed multi-class domain adversarial network (MDAN), comparative experiment was conducted where Fine-tuning (FT) method was used as baseline and MADA as comparison. In Fine-tuning, pre-trained Multi-branch Feature Extractor was frozen and only MLP-based classifier was trained in target subdomain $D_{t\_b}$. For MADA, due to the need for category probability to determine which domain discriminator the sample was input, the pre-trained $G_s$ in the pre-training subtask was connected to $F_{t\_b}$ as ${G_s}^{\prime }$ to obtain category probability, and ${G_s}^{\prime }$ was updated by semantic segmentation loss.

The experimental results are shown in

Table 11. Comparison of different domain adaptation methods. The segmentation results of each category were evaluated by F1 score and are expressed in percentage.

Method	Power	Imp_Surf	Car	Fence	Roof	Tree	OA	p-Value (OA)
FT	50.4 (±5.89)	97.9 (±0.14)	66.3 (±1.91)	48.9 (±5.41)	90.7 (±0.21)	84.1 (±0.25)	90.3 (±0.28)	<0.0001
MADA	57.1 (±4.76)	98.3 (±0.17)	78.6 (±2.41)	55.2 (±4.68)	97.5 (±0.12)	93.7 (±0.18)	95.8 (±0.24)	<0.0001
MDAN (Ours)	76.9 (±3.46)	98.5 (±0.11)	82.6 (±1.81)	61.4 (±4.32)	98.0 (±0.15)	93.9 (±0.2)	96.5 (±0.17)	-

. Since domain adaptation subtask was applied for $C_b$, only power, imp_surf (ground), car, fence, roof (building) and tree (vegetation) were considered. The experiments were conducted five times, the average and ranges of F1 scores and OA were reported, along with p-values (OA) compared to MDAN. The p-values (OA) of FT and MADA were both less than 0.05, indicating there were significant differences. From the table, it can be seen that MDAN could transfer learned knowledge more effectively compared with MADA, which likely contributed to the MDAN strategy of solely using points of the same category for domain adaptation, avoiding negative transfer. Specifically, for categories with fewer samples of power, fences, and cars, the improvement was more pronounced compared with FT. This may be due to the fact that the proposed method can avoid overfitting and improve the segmentation performance.

(2) Discussion of Incremental Learning Subtask

As for the incremental learning subtask, ablation experiments on KTM and FSM were carried out separately. To evaluate KTM, an MLP-base classifier was fine-tuned using the labeled samples from $C_n$ to obtain the ability to segment novel categories. The experimental results are shown in

Table 12. Ablation experiments of KTM and FSM. The segmentation effectiveness of each category was evaluated by F1 and is expressed in percentage.

Method	${\mathit{C}}_{\mathit{b}}$	${\mathit{C}}_{\mathit{n}}$				OA	p-Value (OA)
	OA	Low_Veg	Fac	Shrub	OA
Base	96.5	-	-	-	-	-	-
KTM + Fine-tuning	91.7 (±0.25)	74.5 (±0.37)	42.8 (±0.92)	41.6 (±1.27)	62.7 (±0.45)	82.2 (±0.34)	<0.0001
FSM	81.0 (±0.19)	84.0 (±0.28)	69.5 (±0.86)	65.1 (±0.93)	78.1 (±0.38)	80.1 (±0.23)	0.0001
Ours (KTM + FSM)	91.7 (±0.25)	83.2 (±0.19)	64.2 (±0.73)	52.1 (±0.84)	73.2 (±0.26)	85.5 (±0.25)	-

. “Base” represents the segmentation results of $C_b$ after the domain adaptation subtask. The experiments were conducted five times, and the average and ranges of F1 scores and OA were reported, along with the p-values (OA) compared to the proposed KTM + FSM. The p-values (OA) of FT and MADA were both less than 0.05, indicating there were significant differences. It can be seen that when only using FSM, compared to fine-tuning, the model performed better in segmenting novel categories $C_n$, but it would fall into catastrophic forgetting, resulting in poor performance for base categories $C_b$. The combination of the proposed KTM and FSM improved the segmentation performance of novel categories while avoiding catastrophic forgetting the knowledge learned from the base categories.

In order to better evaluate the proposed FSM, experiments were conducted five times to compare with the Semantic Adversarial Learning Module (SALM) proposed by Dai [25]. The average and ranges of F1 scores and OA were reported, along with p-values (OA) compared between SALM and FSM. The p-value (OA) was less than 0.05, indicating there was significant difference. The experimental results in

Table 13. Comparative experiments of SALM and the proposed FSM. The segmentation accuracy of each category was evaluated by F1 score and is expressed in percentage.

Method	${\mathit{C}}_{\mathit{n}}$
	Low_Veg	Fac	Shrub	OA	p-Value (OA)
SALM	80.4 (±0.81)	58.3 (±1.45)	56.8 (±0.96)	77.1 (±0.9)	-
FSM	84.0 (±0.28)	69.5 (±0.86)	65.1 (±0.93)	78.1 (±0.38)	0.008

show that the proposed FSM achieved better results in novel categories and was more stable.

4.4. Analysis of the Impact of Random Sampling

As mentioned in Section 3.2, few labeled samples were selected via random sampling. Experiments were conducted to evaluate the impact of this process. For proportions of 10%, 1%, and 0.1%, five random samplings were performed for each, with OA and averF1 recorded. The experimental results are presented in

Table 14. The impact of random sampling. OA and averF1 are expressed in percentage.

Proportion		Maximum	Minimum	Average
10%	OA	85.7	85.2	85.5
	averF1	74.3	72.1	73.8
1%	OA	84.1	83.4	83.8
	averF1	67.1	69.2	68.4
0.1%	OA	81.9	80.7	81.6
	averF1	65.6	62.1	64.1

. In general, both OA and averF1 exhibited stability, despite minor fluctuations in averF1. This indicates that random sampling had little impact on segmentation performance, thereby confirming the feasibility of this sampling method.

5. Conclusions

In this study, a novel Multi-branch Feature Extractor and transfer learning strategy were proposed, aiming at improving the performance of the model and reducing the amount of annotated data. MFE was based on multi-branch architecture, which is able to extract more representative features and deal with inconsistent parameter types between the source domain and target domain. A three-stage transfer learning strategy was proposed to cope with feature transfer and catastrophic forgetting: (1) The pre-training subtask effectively utilizes multi-source ALS data to obtain pre-trained feature extractor. The source domain consists of datasets collected by different LiDAR systems; therefore, there are data discrepancies in term of parameters recorded, point density, topography, and landforms. MFE plays a core role in this stage to learn features. (2) In the domain transfer subtask, MDAN aligns the distributions of the source and target domains for each category separately through multiple domain discriminators, avoiding negative transfer caused by unique categories of the source domain and novel categories of the target domain. (3) The incremental learning subtask, consisting of KTM and FSM, was designed to learn the knowledge of novel categories while avoiding catastrophic forgetting for base categories. The experiments conducted on the ISPRS benchmark dataset validated the effectiveness of the proposed method. Through the proposed method, using only 10% labeled training samples can achieve competitive segmentation results comparable to other fully supervised methods. Meanwhile, the proposed method is also superior to other few sample learning methods. By fully utilizing existing annotated ALS data, the need for manual annotation is greatly reduced when facing a new dataset.

Although the proposed method reduced the need for manual annotation, the cost was the increase in computational complexity, especially for the domain transfer subtask. The existing few sample learning methods require retraining for a new dataset. The proposed method solves this problem by separating the pre-training subtask from the domain adaptation subtask. However, in the domain adaptation subtask, it is still necessary to extract features from the source domain to align the features, which will increase the computational load. Reducing the computational load of the domain adaptation subtask is still a problem that needs further research. Airborne LiDAR data is difficult to collect compared to spectral images in Earth observation. Exploring the possibility of cross-modal transfer from spectral images to LiDAR point clouds is meaningful and listed as the future research direction of the authors’ R&D. Meanwhile, MFE has demonstrated its application value in semantic segmentation. Extending it to other 3D data processing tasks (e.g., object detection) is also a potential research direction.