IPCONV: Convolution with Multiple Different Kernels for Point Cloud Semantic Segmentation

Abstract

The segmentation of airborne laser scanning (ALS) point clouds remains a challenge in remote sensing and photogrammetry. Deep learning methods, such as KPCONV, have proven effective on various datasets. However, the rigid convolutional kernel strategy of KPCONV limits its potential use for 3D object segmentation due to its uniform approach. To address this issue, we propose an Integrated Point Convolution (IPCONV) based on KPCONV, which utilizes two different convolution kernel point generation strategies, one cylindrical and one a spherical cone, for more efficient learning of point cloud data features. We propose a customizable Multi-Shape Neighborhood System (MSNS) to balance the relationship between these convolution kernel point generations. Experiments on the ISPRS benchmark dataset, LASDU dataset, and DFC2019 dataset demonstrate the validity of our method.

1. Introduction

Point clouds provide precise 3D geometric information. They are used in digital twin cities [1,2], railways [3], cultural heritage [4], power line extraction and reconstruction [5], and autonomous driving [6]. However, they lack semantic information, hindering machine comprehension and usage. High-precision semantic segmentation is necessary for many applications.

Supervised machine learning methods like SVM [7] and Random Forest [8] are often used in early point cloud segmentation research and rely on hand-crafted features to extract low-dimensional data. However, with the increase in GPU computing power, deep learning methods have become more common in recent years for processing point cloud data and tackling complex semantic segmentation tasks [9].

Convolution is an important branch in the field of 3D deep learning, known for its advantages of weight sharing and local perception [10]. The deep learning method KPCONV [11] extends convolution to 3D point cloud processing, achieving competitive results. It uses convolution kernel points with fixed spatial generation, calculates the weight matrix through the kernel function, and performs calculations with points in the spherical neighborhood. KPCONV bypasses the problem of direct, high-complexity regular grid convolution by obtaining points in the spherical neighborhood through Poisson disks or random points to reduce operational complexity.

However, recognizing objects in 3D scenes can be challenging due to their complexity. For instance, buildings often have multiple sides and facades, while streets look flat. Additionally, trees have a tall vertical structure and a large crown. The rigid convolutional kernel point strategy of KPCONV inhibits its potential to recognize buildings, trees, and other objects from point clouds. For example, when dealing with large-scale airborne laser scanning (ALS) data, planar structures with multiple sides and facades are often present in the point cloud data of buildings, roads, and other similar categories, while structures with one side larger than the others, such as a spherical cone, are typically found in vegetation data. The convolution kernel used by KPCONV employs a system composed of attraction and repulsion and indiscriminately considers the characteristics of local neighborhood points. Pursuing more detailed semantic segmentation effects may slightly constrain the network effect due to this characteristic.

Based on the larger variance of ALS point cloud coordinates in the XY plane than along the Z-axis, LGENet [12] created a hybrid 2D and 3D convolutional module using the original KPCONV convolution. Similar to the principle of LGENet, with a better disparity in its 3D structure, the cylindrical convolutional kernel generation solves the inconsistency between the XY and Z planes often found in buildings and roads. Furthermore, the dataset often contains vegetation, and those features can be further explored. The spherical cone convolution kernel generation is designed to improve the recognition accuracy of trees and shrubs.

Optimizing the effectiveness of a network involves judiciously leveraging the diverse capabilities of convolutional kernels, a facet that often remains untapped when employing only a single kernel per downsampling layer. Drawing inspiration from the domain of image analysis, where many convolution kernels prove invaluable for robust feature extraction, we recognize the potential of a richer kernel ensemble. Building upon the conceptual foundation of KPCONV, we introduce a novel and streamlined architecture called Integrated Point Convolution (IPCONV), fusing multiple computational elements to enhance network performance.

We introduce the Multi-Shape Neighborhood System (MSNS) during the downsampling phase. This sophisticated construct synergizes the insights derived from two distinct convolution kernel types: cylindrical and spherical cone kernels. The resultant hybrid kernel framework is illustrated in Figure 1, juxtaposing the novel IPCONV with the traditional KPCONV, wherein IPCONV replaces the conventional single-layer spherical convolution kernels with an array of different convolution kernels. The significant contributions of this work are as follows:

(1)

Innovative Kernel Generation Methodology: Our study pioneers the development of two distinct methodologies for generating cylindrical convolution kernel points and spherical cone kernel points. Our proposed network optimally captures idiosyncrasies in ground object characteristics by meticulously tuning these parameters, enhancing feature learning.

(2)

Enhanced Local Category Differentiation: Addressing the need for precise discrimination between local categories, we introduce the MSNS. This system effectively concatenates knowledge acquired through diverse convolutional kernel point generation methods, elevating the proficiency of feature learning. This approach enhances the discernment capabilities of the network and augments its grasp on complex local features.

(3)

Benchmark Validation: The efficacy of our proposed model is demonstrated through comprehensive evaluations on multiple 3D benchmark datasets. Comparisons with established baseline methods consistently highlight the progress achieved. Specifically, on the ISPRS Vaihingen 3D dataset [13], our model achieves an impressive Avg.F1 score of 70.7%, with a corresponding overall accuracy (OA) index of 84.5%. Performance on the LASDU dataset [14] further underscores the prowess of our model, with an Avg.F1 score of 75.67% and an OA of 86.66%. On the DFC 2019 dataset [15], our model attains a remarkable Avg.F1 score of 87.9%, and OA score of 97.1%.

In essence, our work, IPCONV, builds upon the KP-FCNN architecture of KPCONV and harnesses the combined potential of various convolutional kernels and kernel generation methodologies. The network significantly enhances feature learning, local categorization, and benchmark performance. This study thereby contributes substantively to the domain of 3D benchmark dataset analysis and serves as a testament to the efficacy of our proposed IPCONV model.

2. Related Work

There are three main categories for 3D point cloud semantic segmentation using deep learning: projection-based, voxel-based, and point-based networks.

2.1. Projection-Based Networks

Initially, researchers used multi-view methods to classify point clouds. A projection-based method uses a 2D Convolutional Neural Network (CNN). Multi-view CNN (MVCNN) [16] pioneered using virtual cameras to render different 2D views is used to recognize 3D models. However, MAX-POOLING may cause a loss of detail, and multi-resolution filtering is employed to improve performance [17]. MHBN [18] works from the perspective of a patches-to-patches similarity measurement to preserve more details during projection. In addition to 2D CNN, a 2D Fully Convolutional Network (FCN) can accept input images of any size for semantic segmentation. After projecting a point cloud onto a 2D image, FCN networks can segment the image with high accuracy and low inference time [19].

Besides 2D CNN and 2D FCN variants, other works have incorporated concepts from different research fields [20,21]. MVTN [20] uses differentiable rendering to identify optimal viewpoints for 3D shape recognition through regression, while 2D&3DHNet [21] fuses 3D global and 2D local Hough features from CNN to improve classification accuracy.

2.2. Voxel-Based Networks

Voxelization is a pivotal transformation method, converting intricate point cloud data into a structured voxel grid format for subsequent processing through a 3D CNN and related techniques. Of the early adopters, VoxNet [22] utilized voxelization to handle point clouds, yet memory constraints posed a challenge. Consequently, various networks have emerged to enhance its computational efficiency and model efficacy [23,24,25,26,27,28,29]. In the pursuit of optimizing 3D data memory utilization, ingenious data structure algorithms have been harnessed to translate data into a sparse framework. OctNet [24] employs an octree, MVKd-net [25] employs a kd-tree, and EVBS constructs voxels using a binary octree. While conventional voxel representation using Boolean occupancy information falls short in efficiently encapsulating sparsely distributed points, VV-Net [26] introduces an innovative approach. They advocate for radial basis functions (RBF) to calculate a local, continuous representation within each voxel, overcoming the challenge of compactly representing sparsely scattered points.

However, the challenge of identifying small-sized objects escalates with sparse distribution. In the context of outdoor driving scenes, implementing cylindrical partitioning via 3D convolutional neural networks has been instrumental in overcoming sparsity and fluctuations in density [27]. To tackle the intricate task of learning from sparsely distributed points and encapsulating contextual details, SVASeg [28] has harnessed attention mechanisms. D-Net [29] has introduced an innovative density-based voxelization procedure to enhance the situation on a parallel front. This approach entails assigning each voxel a density value that properly characterizes the distribution of points within its designated location.

2.3. Point-Based Networks

While projection-based and voxel-based networks necessitate feature extraction or data transformation to adapt point cloud data into structured formats, point-based networks stand out for their capacity to directly handle irregular point cloud data. These networks possess the unique ability to perform feature extraction and processing directly on individual points. Notably, the introduction of PointNet [30] marked a milestone in this realm by enabling the direct processing of irregular point clouds, sidestepping the need for voxelization or two-dimensional conversion. PointNet retains the crucial permutation invariance of input points. Following the lead of PointNet, subsequent methodologies have emerged [31,32,33,34,35]. Parallel to this, graph convolution-based networks, including DGCNN [36] and other variants [37,38,39,40,41,42], have focused on autonomously assigning weights between points. The landscape also includes attention-based networks [43,44,45,46,47,48,49,50,51], which play a pivotal role in point-based networks by encoding point cloud positions directly or indirectly.

Inspired by the 2D FCN [19], which handles local correlations in densely represented data, 3D PointCNN [52] tackles the weighting of the input features associated with the points and the permutation of the points into a latent and potentially canonical order by employing an X-transformation. A-CNN [53] introduces Annular Convolutional as a solution to the issue of insufficient local information extraction observed in PointNet. This method leverages hand-crafted rings to encode local structures. ConvPoint [54] extends the principles of discrete convolutional neural networks (CNNs) to point cloud processing by replacing discrete kernels with continuous ones. DPC [55] innovates dilated point convolutions to tackle the receptive field constraint, effectively enlarging the receptive field size of point convolutions. Pointconvformer [56] combines point convolution with a transformer, which is a deep learning model architecture [9] that uses self-attention mechanisms to capture dependencies between elements in a sequence of tasks, harnessing the advantages of transformers in conjunction with convolutional methods.

3. Method

In this section, we outline our approach, beginning with two novel convolution kernel point generation strategies and introducing the fundamental principles of convolution in Section 3.1. These strategies encompass cylindrical convolution kernel point generation and spherical cone kernel point generation, aligned with the KPCONV-based convolution rules. Subsequently, Section 3.2 describes the MSNS, while Section 3.3 illustrates the comprehensive network architecture.

3.1. Convolution Kernel Point Generation

We propose a novel approach for distributing convolution kernel points to enhance the efficacy of capturing point cloud features. Departing from the rigid convolution generation operator of the traditional KPCONV, which optimizes kernel point spacing within a designated sphere by applying an attractive potential at the center of the sphere to prevent divergence (as shown in Figure 2a), we propose two supplementary strategies: cylindrical convolution kernel point generation and spherical cone convolution kernel point generation. These methods serve as a valuable augmentation to the original KPCONV, designed to broaden its capabilities.

3.1.1. Cylindrical Convolution Kernel Point Generation

The cylindrical convolution kernel point generation strategy proves highly effective in enhancing the discernment of planar and flat structures featuring multiple height layers, such as buildings, as depicted in Figure 2b. This strategy unfolds in two distinct phases.

Initially, a plane is established parallel to the base of the cylinder. The intersection between this plane and the cylinder creates a circle. By subsequently moving the plane along the cylindrical axis, adhering to a predetermined step, a series of $M$ circles akin to the initial one is generated. The position of each plane, denoted as $Z\left(i\right)$, is calculated using Equation (1).

$$Z\left(i\right)=Z_{min}+\left(Z_{max}-Z_{min}\right)\ast i/\left(M+1\right),i=1,2,\dots ,M$$

(1)

where $Z_{min}$ and $Z_{max}$ represent opposite values, with their absolute magnitudes equating to half the cylindrical height.

In the subsequent phase, a specified number of kernel points is extracted for each derived circle through Equation (2). These kernel points adopt a 2D version of the initial KPCONV, a technique employed to enhance the comprehension of planar entities [12]. The kernel points show a clear tendency to avoid each other, ensuring a level of separation while also possessing an attraction that draws them within a certain radius. The formulation for generating the 2D KPCONV is outlined in Equations (3) and (4).

$$\left\{{\tilde{x}}_j\left(x,y,z\right)\right|{\tilde{x}}_j\left(z\right)=Z\left(i\right),j=1,2,\dots ,G\}\subset N_i^{cyclinder}$$

(2)

$$E^{2Dtot}={\displaystyle \sum }_{n<G}(\Vert {\tilde{x}}_n{\left(x,y\right)\Vert }^2+{\displaystyle \sum }_{G\ne n}\frac{1}{\Vert {\tilde{x}}_g\left(x,y\right)-{\tilde{x}}_n\left(x,y\right)\Vert })$$

(3)

$$\left\{{\tilde{x}}_j\left(x,y,z\right)\right|{\tilde{x}}_j{\left(x\right)}^2+{\tilde{x}}_j{\left(y\right)}^2<r^2,j=1,2,\dots ,G\}\subset N_i^{cyclinder}$$

(4)

where the radius of the cylinder is denoted as $r$, the number of kernel points ${\tilde{x}}_j\left(x,y,z\right)$ per layer is $G$, $N^{cyclinder}$ signifies the overall count of kernel points, the kernel points for each layer are represented by $N_i^{cyclinder}$, and $E^{2Dtot}$ represents the combined constraint that encompasses both repulsion and attraction effects.

Finally, akin to KPCONV, the kernel points $N_i^{cyclinder}$ at each layer undergo random rotations around the Z-axis and are subject to random coordinate offsets.

3.1.2. Spherical Cone Convolution Kernel Point Generation

While cylindrical convolution kernel point generation excels in distributing points across height multilayers with plane symmetry, it may fall short in recognizing objects that do not fit this assumption, like trees. Trees often exhibit significant size discrepancies between their canopy and trunk regions, resulting in sparse trunk point representation. To address this, we propose the spherical cone convolution kernel point generation.

Spherical convolutional kernel point generation closely resembles the spherical cone approach. Additionally, the parameters can be adjusted to yield a dual spherical cone or accommodate a versatile complement within the spherical neighborhood, as described earlier. This configuration offers substantial flexibility to suit various scenarios.

Firstly, an initial set of candidate points $N_{init}^{\mathrm{sphericalcone}}$ is randomly generated within a sphere.

Secondly, as depicted in Figure 3, a line segment is traced from a point on the surface of the sphere to the center of the sphere, forming an angle of $\theta$ with the Z-axis. A spherical cone is formed by rotating this line segment around the Z-axis. This spherical cone can be represented as $\{{\tilde{x}}_i\left(\mathrm{X},\mathrm{Y},\mathrm{Z}\right)|\mathrm{i}<S\}\subset N_{init}^{\mathrm{sphericalcone}}$, as indicated below:

$$\{\frac{{\tilde{x}}_i\left(Z\right)}{\sqrt{{\tilde{x}}_i{\left(X\right)}^2+{\tilde{x}}_i{\left(Y\right)}^2}}>tan\left(\pi /2-\theta \right)| i=1,2,\dots ,S\}$$

(5)

where $\theta$ signifies the threshold angle between a point and the $Z$-axis, $S$ stands for the kernel points located within the spherical cone, and ${\tilde{x}}_i\left(\mathrm{X},\mathrm{Y},\mathrm{Z}\right)$ denotes the positions of kernel points.

Furthermore, Equation (5) incorporates a range of parameters that have been strategically devised to enhance the versatility of the convolution process. These parameters offer multiple configurations, such as concatenating them to derive dual-volume kernel points in the “both-side” mode and modifying the direction of the inequality to generate reverse points in the “complementary” mode. This adaptability allows the convolution process to cater to diverse scenarios and requirements.

Lastly, the resulting convolution kernel points ${\tilde{x}}_j$ are randomly selected to fulfill the specified count as the definitive kernel points. Analogous to KPCONV, these kernel points undergo random 3D rotations and are subject to random coordinate offsets to complete the process.

The point cloud generation process associated with spherical cone convolution yields denser results than cylindrical forms, closely mirroring the morphology of the tree. This convolutional kernel point generation relies on the concept of a sphere neighborhood. As illustrated in Figure 3, its base mode is instrumental in capturing tree features. This nuanced convolutional kernel point generation is anticipated to unlock additional layers of point cloud feature information.

3.1.3. Convolution Rules Based on KPCONV

After kernel point generation, the convolution process is the same as for KPCONV. The convolution procedure $Conv\left(p,P,F,N\right)$ for a given point $p$ can be delineated as follows:

$$Conv\left(p,P,F,N\right)={\displaystyle \sum }_{i=1}^{PNum}\left\{{\displaystyle \sum }_{j=1}^N\left[h\left(p_i-p,\widetilde{x_j}\right)\ast W_j\right]\ast f_i\right\}$$

(6)

here, $N$ signifies the previously acquired convolution kernel points, with their positions denoted as $\tilde{x}$. The points within the spherical neighborhood are represented as $p_i$ from $P\in R^{PNum\times 3}$, and their point features $f_i$ are represented as $F\in R^{PNum\times D_1}$.

The convolution commences by selecting an arbitrary neighboring point $p_i$, then iterating through each kernel point $\widetilde{x_j}$ to calculate the positional weight $h\left(p_i-p,\widetilde{x_j}\right)$. This involves subtracting the normalized position of the specific point $p$ from the neighboring point $p_i$. Subsequently, the weights for each kernel point are obtained by multiplying the positional weights $h\left(p_i-p,\widetilde{x_j}\right)$ with the feature weight mapping matrix $\left\{W_j\right|j<N\}\subset R^{D_{in}\times D_{out}}$, where the input dimension of the feature mapping matrix is $D_{in}$ and the output dimension is $D_{out}$, and the weight for neighborhood point $p_i$ is obtained by summing the weights of each kernel point. The outcome of multiplying the weight obtained from $p_i$ with the corresponding feature $f_i$ serves as the convolution feature of the neighboring point $p_i$. The convolutional features of all neighboring points for a particular point $p$ are aggregated to derive the final convolutional feature, which becomes the output feature for that specific point.

IPCONV builds upon the KP-FCNN framework and employs cylindrical convolution kernel point generation and spherical cone convolution kernel point generation for generating kernel points. These kernel points, utilizing geometric relationships between kernel points and the neighboring point $p_i$ through Equation (6) of the convolution rules, impact the efficacy of the feature learning process for each particular point p.

3.2. Multi-Shape Neighborhood System

After the preceding convolution process, we employ a singular convolution operator for downsampling to learn neighborhood points. Building upon these convolution kernels, we focus on refining convolutional intricacies by creating a highly adaptable MSNS. Our aim with the MSNS is to augment the features of the convolution kernels, mitigating the potential imbalance inherent in employing a solitary convolution kernel for a given category. To achieve this, we incorporate diverse convolutional kernel point generation methods for each layer, including cylindrical convolution and spherical cone point generation. The accumulated knowledge from these distinct convolution patterns results in $F^{cyclinder}$ and $F^{\mathrm{sphericalcone}}$ representing the acquired features. Subsequently, the MSNS process involves dimensionality reduction and feature concatenation. Illustrated in Figure 4, our proposed MSNS integrates multiple convolutional kernels of varying types, fostering enriched feature learning.

Before feeding the network with data, the MSNS initially applies a batch normalization and ReLU layer to the original features $D_{in}$.

Next, features are learned through the generated cylindrical and spherical convolutional kernels, designated as $\left\{F_i^{cyclinder}\right|i=1,2,\dots ,N_{cyclinder}\}$ and $\left\{F_i^{\mathrm{sphericalcone}}\right|i=1,2,\dots ,N_{\mathrm{sphericalcone}}\}, \mathrm{respectively}$. These features are concatenated to yield $F_{concat}^{diy}$, expressed as:

$$F_{concat}^{diy}=F_i^{cyclinder}{{\displaystyle \cup }}^{ }{F}_2^{cyclinder}{{\displaystyle \cup }}^{ }{F}_3^{cyclinder}{{\displaystyle \cup }}^{ }{F}_4^{cyclinder}{{\displaystyle \cup }}^{ }{F}_1^{Sphericalcone}{{\displaystyle \cup }}^{ }{F}_2^{Sphericalcone}$$

(7)

$F_{concat}^{diy}$ is normalized, resulting in a dimensionality reduction denoted as $D_{out}$. Concurrently, the processed total network features $D_{out}$ are integrated with the input network features. This integration entails combining the input network features with the processed total network features $D_{out}$ to establish short-circuit connections, thereby finalizing the module.

3.3. Network Architecture

The fundamental network structure utilized in this study is KP-FCNN, wherein the conventional KPCONV module is substituted with the MSNS. Figure 5 provides a visual depiction, highlighting that IPCONV predominantly enriches the feature learning process during downsampling. The network framework is an amalgamation of distinct elements, encompassing upsampling paired with feature concatenation, 1D convolution, skip link, the MSNS module, and the Strided Multi-Shape Neighborhood System module (Strided MSNS).

The encoder consists of 5 convolutional layers. IPCONV uses the MSNS module and the Strided MSNS module for all convolutional blocks for each layer. The MSNS module keeps both the input and output feature dimensions and points the same. The Strided MSNS can be likened to image stride convolution grounded in KPCONV, where the output feature dimension $D_{out}$ is twice as large as the input dimension $D_{in}$, and the output number of points $S_{out}$ is fewer than the input point count $S_{in}$.

In the decoder, nearest neighbor upsampling is used to obtain the final point-by-point features. There are four skip links that pass intermediate features from the encoder to the decoder. These features are concatenated using the upsampled features and point-by-point features, then passed through a 1D convolution block.

4. Experiments

4.1. Dataset

We conducted experiments on the ISPRS Vaihingen 3D Semantic Labeling benchmark dataset. This dataset encompasses an ALS point cloud collected using the Leica ALS50 system from an altitude of 500 m over Vaihingen, Germany, in August 2008. The dataset comprises nine categories: powerline, low vegetation, impervious surfaces, car, fence/hedge, roof, facade, shrub, and tree. Each point is characterized by five features: X, Y, Z, intensity, and return number. The dataset includes training and test sets, containing 53,876 points and 411,722 points, respectively. Refer to Figure 6 for an illustration.

In order to ascertain the applicability of the method on diverse datasets, we employ the LASDU dataset [14] to further validate our experimental results. This dataset originates from an ALS point cloud collection using the Leica ALS70 system at approximately 1200 m over the Heihe River Basin in Northwest China, in 2012. The dataset is categorized into five classes: buildings, trees, low vegetation, artifacts, and unclassified points. The dataset comprises around 3.12 million labeled points. As depicted in Figure 7, the labeled point cloud of the survey area is partitioned into four sections, containing approximately 770,000, 590,000, 1.13 million, and 620,000 points, respectively. Specifically, Section 2 and Section 3 serve as the training set, while Section 1 and Section 4 function as the test set.

Furthermore, we extend our exploration to the IEEE Geoscience and Remote Sensing Society (GRSS) Data Fusion Contest in 2019 (DFC2019) [20]. The DFC data include XYZ data, intensity, and echo number. The study areas cover Jacksonville, Florida, and Omaha, Nebraska, in the United States. The dataset encompasses five point cloud tag categories: ground, high vegetation, building, water, and bridge deck. We follow the evaluation method proposed by Wen [57], utilizing 100 tiles for training and 10 for testing.

4.2. Evaluation Metrics

We employ two metrics, namely Overall Accuracy (OA) and mean F1 score, to assess the effectiveness of the proposed method. Specifically, OA represents the correctness of point cloud segmentations in the test dataset, and the F1 score is the harmonic mean of Precision and Recall for each category, mathematically defined as:

$$OA=\left(TP+TN\right)/\left(TP+TN+FP+FN\right)$$

(8)

$$Precision=TP/\left(TP+FP\right)$$

(9)

$$Recall=TP/\left(TP+FN\right)$$

(10)

$$F_1 Score=2\times \left(Precision\times Recall\right)/\left(Precision+Recall\right)$$

(11)

here $TP$ signifies true positives, $TN$ is the true negatives, $FP$ represents false positives, and $FN$ denotes false negatives.

4.3. Model Hyperparameters

For the ISPRS benchmark dataset, the hyperparameters of the experiment are set as follows: a batch size of 2, an input sphere radius of 12.0 m. Grid sampling is a sampling method involves assigning points to their respective regular voxel grid, we set the first grid sampling parameter $G_0$ as 0.2 m. Then, their grid sampling parameters $G_i$ for the other 4 convolutional layers are defined as $\{G_i=G_{i-1}\ast 2|i=2,3,4,5\}$. The convolution radius is 2.5 times the grid size in the corresponding layer. The aggregation function of “sum” is employed, which signifies the aggregation of features by adding them together. No additional preprocessing is necessary for training. The original KPCONV employed 12–15 kernels, whereas we expanded the number to 20 to enhance geometric feature perception. In Figure 8, the point surface density of the three datasets is shown, indicating the variations in surface density among the different point cloud datasets. These chosen parameters can effectively handle the diverse point densities and are used in all experiments.

Regarding the number of MSNS cores, considering memory consumption, a cautious recommendation of 4–8 is proposed. For the ISPRS benchmark dataset, four cylindrical convolution kernels and two spherical cone convolution kernels are utilized within the MSNS, as detailed in

Table 1. Four cylindrical convolution kernel point generation parameters. On the first line, we give the different convolution kernel point generation parameters, which are order (Number), cylinder height (H), and multilayer layers (M).

Number	H	M
1	8	2
2	4	2
3	2	1
4	2	1

and

Table 2. Two spherical cone convolution kernel point generation parameters. On the first line, we give the different convolution kernel point generation parameters: order (Number) and angle ($\theta$).

Number	$\mathit{\theta }$
5	45
6	60

. This configuration enables practical feature processing. These MSNS parameters are applied across the ISPRS benchmark, LASDU, and DFC 2019 datasets in this study. Note that all experiments are conducted on the same machine with an Intel [email protected] CPU with 32GB RAM and a GPU card with 16GB memory, using the Python programming language.

5. Results

5.1. Semantic Segmentation Results for the ISPRS Benchmark Dataset

The error plot depicting the ISPRS prediction data is shown in Figure 9, affirming the commendable performance of IPCONV on this dataset. Figure 10 provides a comprehensive view of the overall semantic segmentation outcomes for the entire dataset and localized portions. Notably, satisfactory results for the roof and facade categories are attained, facilitated by the targeted cylindrical convolution kernel point generation strategy. The tree category receives notable benefits from the spherical cone convolution kernel point generation design. The synergistic integration of these strategies culminates in favorable outcomes across the entire dataset.

Figure 11 provides a comparison with the baseline methods. The first and second rows highlight the superior differentiation of IPCONV between building and roof categories, attributed to the cylindrical convolutional kernel point strategy. This strategy leverages layered cylindrical convolutional kernel points with distinct height differences for improved discrimination. In the third row, the efficacy of the spherical cone convolutional kernel point strategy is evident, enhancing tree feature recognition, especially in cases of shrub–tree confusion. The combined performances underscore the effectiveness of IPCONV and demonstrate the potential of the MSNS-generated convolutional kernel strategy for feature-specific recognition, particularly in mapping contexts.

Table 3. Quantitative comparisons between our IPCONV and other models on the ISPRS benchmark dataset. The F1 scores for different classes are shown in the first nine columns, and the average F1 score and the OA are shown in the last two columns.

Method	Power	Low Veg	Imp Surf	Car	Fence	Roof	Facade	Shrub	Tree	Avg.F1	OA
LUH [58]	59.6	77.5	91.1	73.1	34.0	94.2	56.3	46.6	83.1	68.4	81.6
WhuY4 [59]	42.5	82.7	91.4	74.7	53.7	94.3	53.1	47.9	82.8	69.2	84.9
RIT_1 [60]	37.5	77.9	91.5	73.4	18.0	94.0	49.3	45.9	82.5	63.3	81.6
alsNet [61]	70.1	80.5	90.2	45.7	7.6	93.1	47.3	34.7	74.5	60.4	80.6
A-XCRF [62]	63.0	82.6	91.9	74.9	39.9	94.5	59.3	50.7	82.7	71.1	85.0
D-FCN [57]	70.4	80.2	91.4	78.1	37.0	93.0	60.5	46.0	79.4	70.7	82.2
LGENet [12]	76.5	82.1	91.8	80.0	40.6	93.8	64.7	49.9	83.6	73.7	84.5
RFFS-Net [63]	75.5	80.0	90.5	78.5	45.5	92.7	57.9	48.3	75.7	71.6	82.1
KPCONV [12]	73.5	78.7	88.0	79.4	33.0	94.2	61.3	45.7	82.0	70.6	81.7
Ours	66.8	82.1	91.4	74.3	36.8	94.8	65.2	42.3	82.7	70.7	84.5

Bold represents the best performance in each column.

offers a comparison of the performance of our model with state-of-the-art ISPRS benchmark models, including LUH [58], WhuY4 [59], RIT_1 [60], alsNet [61], A-XCRF [62], D-FCN [57], LGENet [12], RFFS-Net [63], and KPCONV [12].

Among these, LGENet incorporates SegECC, an attention strategy, and a hybrid 2D3D convolution strategy into the KPCONV framework. Our method performs comparably to LGENet in terms of OA, slightly trailing WhuY4 with an OA of 84.5%. Our approach offers a more straightforward architecture, achieving a comparable OA performance to LGENet. However, the current mF1 metric performance of our network is lower than that of LGENet and RFFS-Net. This discrepancy can be attributed to our selective use of the MSNS, which focuses more on specific categories like roofs and trees, potentially neglecting others like power and shrub.

Our approach shows superior results when comparing our method to the benchmark KPCONV, with an improvement in OA by 2.8% compared to KPCONV. Its performance in identifying objects in categories such as power, shrub, and vehicles is relatively inferior due to the focus of IPCONV on categories like buildings and trees. The high similarity in height between shrubs and low vegetation may have caused confusion, and the explanation for misclassification of vehicles is that they are typically slightly higher than the street level in aerial data, which can cause misclassification. Sparse points in the point cloud data for power lines may be missed by the network. However, apart from these categories, the proposed method has significantly improved the accuracy of tree, roof, and other categories, as well as the overall accuracy of the dataset.

5.2. Semantic Segmentation Results on the LASDU Dataset

Comparing the performance of IPCONV with various networks such as PointNet++ [31], PointCNN [52], DensePoint [64], DGCNN [36], PosPool [65], HDA-PointNet++ [66], PointConv [67], VD-LAB [68], RFFS-Net [63], and KPCONV [12], as detailed in

Table 4. Quantitative comparisons between our IPCONV and other models on the LASDU dataset. The F1 scores for different classes are shown in the first nine columns, and the average F1 score and the OA are shown in the last two columns.

Method	Ground	Buil.	Trees	Low Veg	Artifacts	OA	Avg.F1
PointNet++ [14]	87.74	90.63	81.98	63.17	31.26	82.84	70.96
PointCNN [68]	89.3	92.83	84.08	62.77	31.65	85.04	72.13
DensePoint [68]	89.78	94.77	85.2	65.45	34.17	86.31	73.87
DGCNN [68]	90.52	93.21	81.55	63.26	37.08	85.51	73.12
PosPool [68]	88.25	93.67	83.92	61.00	38.34	83.52	73.03
HAD-PointNet++ [14]	88.74	93.16	82.24	65.24	36.89	84.37	73.25
PointConv [63]	89.57	94.31	84.59	67.51	36.41	85.91	74.48
VD-LAB [68]	91.19	95.53	87.26	73.49	44.64	88.01	78.42
RFFS-Net [63]	90.92	95.35	86.81	71.01	44.36	87.12	77.69
KPCONV [68]	89.12	93.43	83.22	59.70	31.85	83.71	71.47
IPCONV(Ours)	90.47	96.26	85.75	59.58	46.34	86.66	75.67

Bold represents the best performance in each column.

, reveals that the results in this study lag slightly behind VD-LAB and RFFS-Net. Nonetheless, notable achievements are recorded in specific categories, such as buildings and artifacts. This success underscores the effective role played by the distinct convolution kernel point generation strategies—cylindrical and spherical cone—along with the parallelization strategy.

Compared with the KPCONV benchmark experiments, the performance of IPCONV on the LASDU dataset is improved, elevating the OA to 86.66% (a 2.95% enhancement) and mF1 to 75.67% (a 4.2% improvement). Figure 12 visually illustrates the precision of IPCONV in the semantic segmentation results for the LASDU dataset, particularly in the previously mentioned categories of buildings and artifacts.

5.3. Semantic Segmentation Results on the DFC Dataset

Table 5. Quantitative comparisons between our IPCONV and other models on the DFC 2019 dataset. The F1 scores for different classes are shown in the first nine columns, and the average F1 score and the OA are shown in the last two columns.

Method	Ground	Trees	Buil.	Water	Bridge	OA	Avg.F1
PointNet++ [69]	98.3	95.8	79.7	4.40	7.30	92.7	57.1
PointSIFT [69]	98.6	97.0	85.5	46.4	60.4	94.0	77.6
PointCNN [69]	98.7	97.2	84.9	44.1	65.3	93.8	78.0
D-FCN [57]	99.1	98.1	89.9	45.0	73.0	95.6	81.0
DANCE-NET [69]	99.1	93.9	87.0	58.3	83.9	96.8	84.4
PointConv [63]	97.3	95.8	93.6	74.5	69.2	95.3	86.1
RFFS-Net [63]	96.6	96.1	88.7	77.8	80.1	94.3	88.0
KPCONV	98.5	97.3	89.5	87.9	39.5	95.7	82.4
IPCONV(Ours)	98.8	97.5	92.9	92.1	58.2	97.1	87.9

Bold represents the best performance in each column.

presents a comparison of test results, including PointNet++ [31], PointSIFT [32], PointCNN [52], D-FCN [57], DANCE-NET [69], PointConv [67], RFFS-Net [63], and KPCONV [12]. These results highlight the performance of the approach presented in this paper, achieving a leading OA of 97.1% and a mF1 of 87.9%. Notably, the mF1 score is slightly lower than that of RFFS-Net. Additionally, the benchmark experiment results are provided, showing that our OA surpasses KPCONV by 2.6% and mF1 by 5.5%, showcasing the effectiveness of our strategy across all benchmark network categories. Figure 13 visually underscores the enhanced ability of IPCONV to differentiate features like buildings, bridges, and water, compared to baseline (KPCONV) results.

6. Ablation Study

In this section, we discuss the impact of different convolutional kernel point generation methods and the MSNS on the ISPRS benchmark dataset.

6.1. Effect of Difference Convolution Kernel Point Generation

This section evaluates the impact of difference convolution kernel point generation methods when implemented individually within the same type.

Table 6. Ablation study on difference convolution kernel point generation. The F1 scores for different classes are shown in the first nine columns, and the average F1 score and the OA are shown in the last two columns.

Method	Power	Low Veg	Imp Surf	Car	Fence	Roof	Facade	Shrub	Tree	Avg.F1	OA
KPCONV [12]	73.5	78.7	88.0	79.4	33.0	94.2	61.3	45.7	82.0	70.6	81.7
Cylinder (1) 1	72.3	82.2	91.6	74.3	24.0	93.9	63.1	43.9	80.7	69.5	83.8
Spherical cone (1)	52.4	81.4	90.6	76.6	37.6	94.3	62.1	46.6	82.7	69.4	83.9
IPCONV (6)	66.8	82.1	91.4	74.3	36.8	94.8	65.2	42.3	82.7	70.7	84.5

¹ the number in parentheses, e.g., (1), indicates the convolution kernels in each downsampling layer. Bold represents the best performance in each column.

presents the results, demonstrating the efficacy of single cylindrical convolution generation and single spherical cone convolution kernel point generation over the baseline network.

For single cylindrical convolution generation, an Avg.F1 of 69.5% and an OA of 83.8% are achieved. Similarly, for single spherical cone convolution kernel point generation, the Avg.F1 is 69.4%, and the OA is 83.9%, both showing improvements.

Compared to the benchmark KPCONV, single cylindrical convolution generation focuses on enhancing the accuracy of the facade, impervious surfaces, and low vegetation categories, with F1 scores improved by 1.8%, 3.6%, and 3.5%, respectively. However, accuracy for the fence/hedge category decreases by 9.0%, possibly due to the insensitivity of the network to short, linear geometries like fences/hedges. Slight decreases in the accuracy of the tree, power, and car categories are attributed to the multilayer generation of kernel points in cylindrical convolutional generation struggling to learn narrow or low features in height space layers during feature recognition.

For single spherical cone convolution kernel generation, the F1 scores for low vegetation, fence/hedge, shrub, and tree improve by 2.7% compared to the benchmark network KPCONV. Notably, the strategy improves vegetation features while maintaining overall category accuracy. However, recognition accuracy for power decreases by 21.1%, likely due to the scarcity of power line categories leading to more misclassifications.

This study highlights the strengths of spherical cone convolution kernel generation for tree features and cylindrical convolution kernel generation for features extending along the XY plane, showcasing promising results.

6.2. Effect of the MSNS

An examination of the impact of the MSNS is presented in

Table 7. Ablation study on the effect of MSNS. The F1 scores for different classes are shown in the first nine columns, and the average F1 score and the OA are shown in the last two columns.

Method	Power	Low Veg	Imp Surf	Car	Fence	Roof	Facade	Shrub	Tree	Avg.F1	OA
Vanilla (6) 1	67.5	81.7	91.5	62.5	30.0	94.7	63.5	41.2	81.5	68.2	83.9
Cylinder (6)	65.2	81.8	91.1	72.4	31.5	94.9	63.2	45.8	81.9	69.8	84.3
Spherical cone (6)	64.2	81.9	91.5	62.4	31.9	94.8	62.9	44.3	82.9	68.5	84.3
IPCONV (6)	66.8	82.1	91.4	74.3	36.8	94.8	65.2	42.3	82.7	70.7	84.5

¹ the number in parentheses, e.g., (6), indicates the convolution kernels in each downsampling layer. Bold represents the best performance in each column.

, where the baseline for the ablation study is KPCONV. Various combinations of the original rigid convolution kernels are designed to showcase the effectiveness of the parallel algorithm in contrast to the vanilla version consisting of six rigid KPCONV convolution kernels.

Experimental results reveal that the accuracy has been moderately improved, reaching 83.9%. However, the extent of the improvement is limited, suggesting that the undifferentiated spherical convolutional kernel point generation approach encounters constraints in processing our point cloud data. Notably, its recognition accuracy for the “power” category is higher, achieving a 67.5% in F1 score. Nonetheless, the OA performance in other feature categories remains average. Nevertheless, the OA still demonstrates an enhancement of approximately 2.2% compared to a single cylindrical convolutional generation.

Meanwhile, an investigation was conducted by incorporating multiple instances of cylindrical convolution kernel point generations to analyze their impact on the results. As the cylindrical convolution kernel point generation is designed to specifically address building features, and given that the majority of categories in the ISPRS benchmark dataset pertain to buildings, commendable outcomes were achieved. With an Avg.F1 reaching 69.8% and an OA reaching 84.3%, it demonstrates promising performance, particularly on the roof, facade, and shrub categories. Nonetheless, it fell slightly short of the performance achieved by a single spherical cone convolution.

The parallel spherical cone convolution kernel point generation attained an OA result of 84.3%. However, we noted a slight decline in Avg.F1, indicating that it slightly underperforms compared to parallel cylindrical convolution kernel point generation in terms of overall accuracy. However, it achieved an impressive 82.9% classification accuracy on tree features, while the results for low vegetation and shrub categories were less effective than anticipated.

Ultimately, by considering various combinations of cylindrical convolution kernel point generations and spherical cone convolution kernel point generations, specifically focusing on building and tree categories, the optimal combination was selected from these permutations to achieve the current noteworthy semantic segmentation outcomes.

7. Conclusions

We have introduced IPCONV, a straightforward yet highly effective deep learning network tailored for semantic segmentation tasks on point cloud data, particularly in urban settings. Our proposed IPCONV demonstrates a superior performance compared to the KPCONV baseline, showcasing its enhanced capabilities in processing urban datasets. One of the key innovations is the integration of the MSNS, which empowers each convolution layer to incorporate a variety of convolution kernels through customizable configurations. This flexibility allows us to tailor the kernel combination according to specific feature recognition needs, such as using cylindrical convolution kernel point generation for architectural categories and spherical cone convolution kernel point generation for tree categories. The simplicity and straightforwardness of the MSNS parameters facilitate easy integration and experimentation.

The advantages of IPCONV are comprehensively validated through ablation experiments conducted on the ISPRS benchmark dataset. IPCONV showcases an improved OA and an average F1 score of 70.7% over the KPCONV baseline. Furthermore, compared to the LGENet strategy, which is also an enhancement based on KPCONV, IPCONV achieves a similar performance while offering a simpler approach, with an OA index of around 84.5%. Additionally, the efficacy of IPCONV is further demonstrated through evaluations of the LASDU and DFC 2019 datasets.

In the future, we hope to design a more practical network framework for point cloud semantic segmentation tasks. At the same time, we also need to consider the effectiveness of the model for large-scale point cloud datasets to improve network performance. We attempt a feature screening strategy to accomplish feature selection and reduce the dependence on unnecessary features.