Pepper-4D: Spatiotemporal 3D Pepper Crop Dataset for Phenotyping

Abstract

Pepper (Capsicum annuum) is a globally significant horticultural crop cultivated for its culinary, medicinal, and economic value. Traditional approaches for boosting the agricultural production of pepper, notably, expanding farmland, have become increasingly unsustainable. Recent advancements in artificial intelligence and 3D computer vision have started to transform crop cultivation and phenotyping, which has shed new light on increasing production by advanced breeding. However, currently, the field still lacks 3D pepper data that contains enough detail for organ-level analysis. Therefore, we propose Pepper-4D, a new, high-precision 4D point cloud dataset that records both the spatial structure and temporal development of pepper plants across various continuous growth stages. Our dataset is divided into three subsets, including a total of 916 individual point clouds from 29 indoor-cultivated pepper plant samples. Our dataset provides manual annotations at both the plant-level and organ-level, supporting phenotyping tasks such as pepper growth status classification, organ semantic segmentation, organ instance segmentation, organ growth tracking, new organ detection, and even the generation of synthetic 3D pepper plants.

1. Introduction

Pepper (Capsicum annuum) is a globally significant horticultural crop cultivated for its culinary, medicinal, and economic value. Originating in Central and South America, peppers have a long history of domestication [1,2], with archeological evidence indicating their use as early as 7500 BC. Through the Columbian Exchange, Capsicum peppers spread from the Americas across continents and were rapidly integrated into regional diets and agricultural systems [3]. Today, peppers enrich a wide range of cuisines and support a multi-billion-dollar global trade, underpinning the livelihoods of millions of smallholder farmers [4]. The presence of capsaicinoids, which exhibit antioxidant and anti-inflammatory activities, combined with high levels of vitamins A and C, underscores the significance of peppers in nutritional science and preventive healthcare. Together, these culinary, economic, and nutritional/health-related aspects highlight the strategic importance of pepper production [5].

In parallel with the expanding economic and nutritional role of pepper cultivation, global agriculture is facing significant challenges such as the pressure of global population growth, climate change, and the shrinking arable land area; these challenges have already negatively impacted crop yields, especially key staples such as wheat, maize, and rice [6,7]. Traditional approaches for boosting agricultural production, notably, expanding farmland, have become increasingly unsustainable due to biodiversity loss, soil degradation, and environmental instability [8,9]. Recent advancements in artificial intelligence (AI) and computer vision have started to transform crop cultivation and breeding methods. Techniques of plant phenotyping, such as Convolutional Neural Network (CNN) and YOLO-like detection networks, enable automated tasks such as disease detection [10], pest identification [11], fruit localization [12], and yield estimation [13] using two-dimensional (2D) images.

Although 2D-based methods have shown strong results in specific agricultural applications, they have fundamental limitations due to their flat perspective. Visual obstructions and viewpoint dependence often result in data loss, making it difficult to accurately reconstruct the plant’s total shape, volume, and spatial arrangement of organs. Furthermore, existing 2D plant datasets only record snapshots at one or several discrete time points, which limits their utility in analyzing continuous growth and structural changes over time. To address these shortcomings, several 3D crop datasets have been developed to capture detailed spatial information and enable more accurate organ-level analysis. These include three crop datasets introduced in [14] (tomato, tobacco, and sorghum), as well as datasets for rose [15], maize [16], and soybean [17], which collectively demonstrate the value of 3D data for advancing structural phenotyping across diverse species. However, these datasets generally only capture static snapshots at one or several discrete time points, which limits their utility in analyzing continuous growth and structural changes over time. This limitation reduces their effectiveness for studying dynamic developmental processes such as germination, rapid expansion, and senescence. To the best of our knowledge, currently, none of the public datasets capture the complete 3D structural and developmental dynamics of pepper plants in enough detail for organ-level analysis.

To overcome these challenges, we introduce Pepper-4D, a new, high-precision 4D point cloud dataset that records both the spatial structure and temporal development of pepper plants across various continuous growth stages. This dataset offers detailed insights into morphological changes from early vegetative stages to flowering, fruiting, and aging. By addressing the shortcoming of the temporal aspect on existing 3D datasets, Pepper-4D aims to serve as a benchmark for advancing dynamic 3D plant phenotyping, automated growth monitoring, and the structural analysis of plant organs at different developmental phases. Our contributions are three-fold.

(i)

We establish Pepper-4D, a large-scale spatiotemporal dataset comprising 916 individual point clouds from 29 indoor-cultivated pepper plant samples. The peppers were scanned daily across a period of more than 45 days with the Neural Radiance Field (NeRF) technique, which uses about 100 images from different viewing angles to generate a single pepper point cloud containing point numbers from 36,762 to 1,320,895. The total point number of the dataset is 322.72 million.

(ii)

Pepper-4D contains three different subsets. Subset 1 records a long period of growth for 11 potted peppers; Subset 2 contains eight pepper sequences for geotropism tests, during which the cultivation pots are overturned for a period and then recovered; and Subset 3 contains 10 pepper growth sequences. These subsets document developmental events such as budding, flowering, fruiting, organ disappearance, and withering, and contain plant-level and organ-level annotations for testing different phenotyping algorithms.

(iii)

Our dataset provides manual annotations such as point-level growth status label (healthy or withering), point-level semantic organ labels (e.g., stem, leaf) and instance organ annotations, and the point-level organ labels tracked in the timeline, as well as the point-level new organ labels. Based on these annotations, we successfully conducted experiments that cover the phenotyping tasks of pepper growth status classification, pepper organ semantic segmentation, pepper organ instance segmentation, pepper organ growth tracking, new organ detection, and even the generation of synthetic 3D pepper plants with existing strong baselines.

2. Related Work

2.1. 2D Pepper Datasets and Applications

Nowadays, the number and scale of crop-related image datasets are huge [18], and new open datasets are still sprouting up. Therefore, we will only review datasets that are closely related to pepper plants. Two-dimensional (2D) image datasets have played a foundational role in the development of deep learning models for automated pepper phenotyping. Spanning from real-world agricultural images to synthetic renderings, these datasets have enabled substantial progress in precision agriculture for tasks such as disease classification [19], fruit detection [2], and fruit grading [20].

Multiple 2D image datasets have been developed to support studies on peppers, particularly in disease recognition/classification. Barth et al. [21] introduced a Capsicum annuum dataset for image-based semantic segmentation in agricultural scenes, showing that learning-based models can effectively parse pepper plant imagery. In a complementary direction, Sosa-Herrera et al. [19] investigated Capsicum annuum crop health estimation from RGB imagery using deep learning, further demonstrating the potential of 2D vision models for pepper phenotyping and monitoring tasks. Pallewatta et al. [22] introduced the BellCrop dataset with 4860 leaf images covering three categories (healthy, powdery mildew-infected, and magnesium-deficient leaves), where EfficientNet-B7 achieved 96.67% accuracy. Bezabh et al. [23] proposed a concatenated model combining VGG16 and AlexNet for pepper leaf and fruit disease classification on a dataset containing 1596 images with six disease classes. Yin et al. [20] constructed a large-scale spicy pepper dataset with over 28,011 images and 34 classes of diseases and pests and used a ResNet50-based model to conduct classification. For external fruit grading, Ren et al. [24] developed a dataset of 776 images, including 400 images of normal peppers and 376 images of inferior peppers for pepper fruit grading. While these studies demonstrate a strong performance, most of these datasets remain unavailable to the public.

Beyond disease classification, several studies have focused on organ or disease/pest area detection on peppers to support robotic harvesting and yield estimation. Lopez-Barrios et al. [25] applied Mask R-CNN with a ResNet-101 backbone to detect fruits and peduncles of green sweet peppers in greenhouse conditions with 30,000 images under varying light intensity. Li et al. [26] applied an improved YOLOv4-tiny model for green pepper detection on a dataset of 1500 greenhouse images collected under different illumination conditions. Zheng et al. [27] proposed MSPB-YOLO, an improved YOLOv8-based model that integrates RVB-EMA attention, RepGFPN feature fusion, and DIoU loss to detect multi-site pepper blight disease on leaves, stems, and fruits with a mAP of 96.4% under complex conditions. Kapetas et al. [28] developed YOLOv11-Small for leaf segmentation with Transformer and RT-qPCR validation, achieving 87.42% accuracy and an F1-score of 81.13% for early detection of the Botrytis Cinerea pest on pepper leaves.

Despite these advances, existing pepper image datasets still fall short in representing the structural and developmental complexity of pepper plants, and applications mainly focus on detection or identification, lacking organ-level segmentation and analysis. Most of the datasets consist of static snapshots without temporal information, making them unsuitable for continuous growth monitoring and detailed organ-level analysis over time. These issues motivated us to bridge the research gap by establishing a comprehensive, annotated, spatiotemporal 3D dataset of the pepper plants.

2.2. 3D Crop Datasets

Three-dimensional (3D) crop datasets have become increasingly popular in advancing structural plant phenotyping [29,30], organ-level analysis [31], and growth monitoring [32]. Unlike traditional 2D datasets, 3D point clouds preserve volumetric and spatial integrity, which enables high-precision, organ-level trait extraction under both controlled and field conditions. One of the earliest efforts was made by Conn et al. [14]; they captured 558 high-resolution models of tomato, tobacco, and sorghum species across different growth conditions during a growth period. However, the lack of organ-level annotations restricts the applicability of the dataset for fine-grained phenotyping. Subsequent work [33,34] addressed this limitation by introducing manual organ annotations of this dataset. Dutagaci et al. [15] developed ROSE-X, an X-ray tomography-based dataset of 11 rosebushes with organ-level semantic labels for stems and leaves. This dataset enables the application of organ-level segmentation methods; however, its small scale may introduce scalability problems. Schunck et al. [16] established Pheno4D, a publicly available spatiotemporal dataset featuring point clouds of seven maize and seven tomato plants collected across a period of time. In total, the dataset provides 49 labeled maize point clouds and 77 labeled tomato point clouds to support tasks such as semantic and instance segmentation.

More recent contributions have expanded both feature diversity and scale. Sun et al. [17] introduced Soybean-MVS, which reconstructed 102 soybean point clouds with real colors across 13 growth stages using Multi-view Stereo (MVS), and the authors also provided both semantic and instance labels for leaves. Mertoglu et al. [35] presented PLANesT-3D, providing 34 annotated color point clouds of pepper, rose, and ribes plants reconstructed via MVS, with both semantic and instance annotations. However, the 10 pepper point clouds in PLANesT-3D only provide point clouds for a single growth stage without temporal coverage. Marks et al. [36] presented BonnBeetClouds3D, a large-scale 3D sugar beet plant dataset captured in real breeding plots. The dataset provides organ-level labels for 186 plants and 2661 leaves across 48 varieties, along with over 10,000 salient features such as leaf tips, leaf corners, and plant centers.

At a larger scale, Zhu et al. [37] released Crops3D, which compiled 1230 point clouds from eight crop species (maize, rice, wheat, potato, cabbage, tomato, cotton, and rapeseed), scanned with a combination of Terrestrial Leaser Scanning (TLS), Structured Light, and MVS techniques. With semantic and instance annotations for multiple organs, Crops3D supports applications of classification and segmentation. Most recently, Kimara et al. [38] introduced AgriField3D, an open dataset that provides a total of 1045 maize point clouds, with 520 inside being labeled. The dataset also offers multiple resolution versions (100k, 50k, and 10k points) to facilitate diversified downstream applications, making it an effective dataset for phenotypic analysis.

Despite publicly available 3D crop datasets increasing, the research community still lacks public, high-quality 3D datasets for pepper plants. Although temporal pepper datasets based on RGB image sequences and morphological measurements have been released [39,40], publicly available temporal 3D datasets for pepper remain scarce, which restricts dynamic growth monitoring and phenotyping across time. These shortcomings also explain why a dense, labeled, and spatiotemporal dataset of pepper is important.

2.3. Applications on 3D Crop Datasets

Three-dimensional (3D) crop datasets have enabled a wide range of advanced applications in plant phenotyping. The increasing availability of annotated point cloud datasets has accelerated progress in the semantic segmentation of crops, organ-level trait extraction, and growth monitoring.

3D crop datasets facilitate the tasks of organ semantic segmentation, organ instance segmentation, fruit detection, and trait analysis. Li et al. [33] introduced PlantNet, a dual-function deep learning network that simultaneously performs the semantic segmentation of stems and leaves and instance segmentation of individual leaves from 3D plant point clouds with multiple species. Zhang et al. [41] proposed an automatic organ semantic segmentation network SN-MGGE for cucumber seedling point cloud segmentation and phenotypic parameter extraction. SN-MGGE achieves high accuracy by integrating the Euclidean local geometric relationship and the HGS to enhance point embedding. Cui et al. [42] proposed PVSegNet to leverage voxel convolutions to enhance feature extraction for both the organ semantic and instance segmentations of mature soybean pods and stems on point clouds. Wu et al. [43] proposed a skeleton extraction approach for deriving phenotypic traits of maize plants directly from 3D point clouds. The method integrates point cloud de-noising, Laplacian-based contraction, adaptive sampling, and skeleton calibration to generate organ-level structural skeletons. Xie et al. [44] introduced Plant-MAE, a self-supervised 3D plant organ semantic segmentation framework trained on the large, unlabeled point cloud datasets of eight crop species, including maize, tomato, potato, cotton, cabbage, rapeseed, rice, and wheat. Dong et al. [45] proposed an automatic two-stage organ instance segmentation method that integrates PointNeXt for stem–leaf semantic segmentation and Quickshift++ for leaf instance segmentation on sugarcane, maize, and tomato point clouds. Gene-Mola et al. [46] developed a fruit detection and 3D localization framework that integrates Mask R-CNN instance segmentation with structure-from-motion (SfM) photogrammetry to identify and spatially locate fruits in orchard environments. The method was evaluated on 11 Fuji apple trees containing 1455 fruits and showed satisfactory results on 3D fruit detection and localization. Liu et al. [47] introduced FF3D, a 3D apple detection framework that integrates an anisotropic Gaussian-based next-best view (NBV) estimator to achieve robust fruit localization under occlusion for robotic harvesting.

Tracking the same organ across growth stages and detecting growth events (like budding, bifurcation, and decay) are also essential for dynamic plant trait analysis. Li et al. [48] presented a framework for analyzing plant growth from 4D point cloud data, enabling the robust detection of spatiotemporal growth events such as budding, bifurcation, and decay and also the tracking of individual organs through time. Magistri et al. [49] developed a registration framework that aligns plant point clouds across growth stages by integrating SVM-based stem–leaf classification, DBSCAN-based leaf instance segmentation, and self-organizing maps (SOMs). Li et al. [50] developed TrackPlant3D, a 3D organ growth tracking framework that enables dynamic organ-level phenotyping from time-series crop point clouds. The study introduced a spatiotemporal dataset of four plant species (maize, sorghum, tobacco, and tomato) with temporally consistent organ instance annotations, supporting the detection of growth events. Most recently, Li et al. [32] presented 3D-NOD, a spatiotemporal segmentation framework for detecting new organs from time-series point clouds. The method reached a mean F1 of 88.1% and mean IoU of 80.7% for new organ detection on species including tobacco, tomato, and sorghum.

Existing 3D crop datasets mainly focus on crops such as maize, tomato, cotton, and apple. Annotated 3D resources for pepper plants are still missing. Also, the evaluation of a broad range of algorithms on different phenotyping tasks on the existing 3D datasets is also rare to find, especially the spatiotemporal analysis at organ-level. All these reasons motivated us to build the Pepper-4D dataset.

3. Materials and Methods

3.1. Materials

The Pepper-4D dataset comprises three distinct subsets that collectively capture developmental diversity and experimental variability in pepper plants cultivated under controlled indoor conditions. Indoor lighting was provided for approximately 12 h per day, with an average ambient temperature of ~25 °C and average relative humidity of ~45%. Point clouds were scheduled to be acquired at two-day intervals once plants reached a minimum size suitable for reliable 3D scanning; the reported month ranges denote the overall cultivation period. All three subsets were collected under the same indoor cultivation conditions, and representative samples are shown in Figure 1. Subset 1 (Figure 1a) contains 460 point clouds from 11 potted plant sequences collected during the July–December 2024 cultivation period, covering the full developmental cycle from early vegetative growth to flowering, fruiting, and senescence. Subset 2 (Figure 1b) includes 238 point clouds from 8 plant sequences collected between September and December 2024 for geotropism testing: potted plants were placed on their side (approximately horizontal) for a controlled duration and then restored upright to observe recovery behavior. This subset captures structural reorientation and morphological adaptation under gravitational perturbation. Subset 3 (Figure 1c) consists of 218 point clouds from 10 plant sequences collected during the July–September 2025 cultivation period, focusing on early-to-mid developmental phases characterized by rapid organ emergence and canopy expansion. Together, these subsets span major growth events such as budding, flowering, fruiting, organ disappearance, and withering; therefore, our Pepper-4D dataset is a comprehensive benchmark for precise plant-level and organ-level dynamic spatiotemporal phenotyping of pepper plants.

3.2. Data Acquisition

The workflow of the data acquisition is illustrated in Figure 2, consisting of three sequential stages—image acquisition and preprocessing (Figure 2a), 3D reconstruction (Figure 2b), and preparation of plant-only point cloud. During image acquisition, each sample plant was photographed by an RGB camera from a moving cellphone along two circled trajectories, collecting approximately 100 multi-view images per scan under uniform indoor LED illumination to ensure consistent color and low reflection artifacts. The preprocessing sub-step uses the Pose Refinement (PR) method to obtain a sparse scene point cloud and estimate camera poses from captured multi-view images. PR uses the Scale-Invariant Feature Transformation (SIFT) to identify key points and their SIFT feature descriptors in each image. Subsequently, the SIFT feature descriptors are matched across different images. Then, the RANSAC algorithm is used to estimate the relative motion (i.e., relative rotation and translation) between each pair of images. Finally, through Bundle Adjustment (BA), the internal and external parameters of the camera are refined, and a sparse 3D point cloud of pepper (the rightmost of Figure 2a) is obtained. In the 3D reconstruction stage, we turn the sparse point cloud into a high-precision one. NeRFacto [51], an improved version of NeRF, is used as the neural backbone for the 3D reconstruction step (Figure 2b). NeRFacto takes rough scene point clouds and camera poses as input and outputs precise point clouds with adjustable density by Point Cloud Exporter (a part of NeRFacto). Finally, we prepare plant-only point cloud (shown in Figure 2c) by keeping the crop part above the cultivation pot and then applying the Statistical Outlier Removal (SOR) operation from CloudCompare [52] to reduce noise.

3.3. Data Annotation

Data annotations were carried out on both plant-level and organ-level, and we will explain the annotation tasks with visualized examples from Subset 1 and Subset 2. The data annotation examples on Subset 1 are shown in Figure 3, and annotations on Subset 2 are shown in Figure 4. All point clouds were manually annotated using the CloudCompare [52] tool. In Subset 1, we focus on four different types of labels spanning across organ-level labels to plant-level labels. One sequence from Subset 1 is visualized to show how we annotate organ instance labels [32,33], organ semantic labels (stem and leaf), temporally aligned organ labels (for organ tracking) [50], and plant-level health labels in Figure 3. In Subset 2, we focus on two other different types of labels spanning across organ-level labels to plant-level labels. One sequence from Subset 2 is visualized to show plant-level geotropism and organ-level new organ growth events in Figure 4. The organ-level new organ growth events were annotated using the Backward and Forward Labeling (BFL) technique proposed in 3D-NOD [32], in which each pepper point cloud is compared bidirectionally in the timeline to identify new buds and new leaves.

4. Tasks and Results

4.1. Health Assessment by Classification

Plant phenotyping fundamentally aims to monitor crop health, because the timely recognition of stress symptoms is essential for optimizing cultivation management and accelerating breeding decisions. Hereby, we show that the automatic crop health assessment can be carried out on Pepper-4D; the task is realized as a binary point cloud classification task on the pepper sequences, where each point cloud will be classified as either normal (being vigorous and healthy) or withering (undergoing stress-induced decline). Three representative deep architectures, PointNet [53], PointNet++ [54], and Dynamic Graph CNN (DGCNN) [55], were evaluated on Pepper-4D to evaluate the accuracy and robustness of plant health assessment.

4.1.1. Methodology

Point clouds provide a natural and compact representation of plant architecture, preserving the complete three-dimensional arrangement of stems, leaves, and fruits. PointNet [53] is a classical baseline model for point cloud classification and segmentation. PointNet applies shared Multi-layer Perceptrons (MLPs) to individual points and aggregates features through max pooling into a global descriptor, ensuring permutation invariance. PointNet++ (Figure 5a), an improved version of PointNet, extends the feature extraction capability of PointNet through hierarchical architecture design, where points are sampled, grouped into local neighborhoods, and processed to capture multi-scale geometric features. DGCNN, illustrated in (Figure 5b), employs Dynamic Graph learning with EdgeConv operations on the k-nearest neighbor of graphs of feature embeddings to model local geometry and higher-order structural relationships, producing a more adaptive representation of plant morphology.

4.1.2. Health Assessment Results

The health assessment experiment on the Pepper-4D dataset includes 460 point clouds from 11 growth sequences, from which nine sequences have withering events. During data augmentation, we enriched the training data using standard point cloud augmentation, with Farthest Point Sampling (FPS) applied to generate uniform point sets, following the augmentation protocol in the original implementation [54]. The normal pepper point clouds were augmented 10 times, and the withering (dying) point clouds were augmented 100 times to keep a balanced training set, since the dying samples are substantially fewer than the normal ones. After augmentation, the training set contains 2710 “Normal” samples and 2400 “Withering” samples, while the testing set contains 1540 “Normal” and 1100 “Withering” samples, and each point cloud contains exactly 2048 points. The quantitative results of our health assessment experiments are summarized in

Table 1. Quantitative performance of PointNet, PointNet++, and DGCNN on the Pepper-4D health assessment (normal vs. withering). Metrics are reported as class-wise precision, recall, and F1-score (%), with mean values averaged across both classes. The “↑” sign means the higher the better, and the bold means the best in comparison.

Metrics	Network	Normal	Withering	Mean
Prec (%) ↑	PointNet	99.95	94.84	97.39
	PointNet++	99.54	97.23	98.38
	DGCNN	99.98	93.22	96.60
Rec (%) ↑	PointNet	96.13	99.98	98.05
	PointNet++	97.43	99.44	98.43
	DGCNN	94.81	99.96	97.38
F1 (%) ↑	PointNet	98.04	97.34	97.69
	PointNet++	98.31	98.34	98.32
	DGCNN	97.33	96.49	96.91

, which includes comparative metrics from all three networks. In Table 1, three metrics—precision (Prec), recall (Rec), and F1 [33]—are listed to show the quantitative performances. All three networks achieve high accuracy with mean F1-scores above 96.00%. PointNet++ achieves the best result under all metrics, with a mean Prec at 98.38%, a mean Rec at 98.43%, and a mean F1-score of 98.32%.

The qualitative results of the health assessment are shown in Figure 6. PointNet++ shows the best prediction results compared with the other two models, having no misclassification in the ten test samples. Both quantitative and qualitative results implicate that the PointNet++ model (across the three models) is best suited for solving the pepper health assessment task, which achieves a stable and consistent performance by effectively capturing hierarchical geometric features and local structural information.

4.2. Organ Segmentation

Organ segmentation is a key component of 3D phenotyping, enabling fine-grained analysis of the plant structure. Organ segmentation usually comprises two subtasks—organ semantic segmentation, which assigns an organ semantic label (stem/leaf/fruit, etc.) to each point of the crop, and organ instance segmentation, e.g., distinguishing a dense canopy class of a crop into individual leaves. We apply two representative dual-branch networks—PlantNet [33] and PSegNet [34]—to carry out the organ segmentation on the Pepper-4D dataset. Both models are capable of performing organ semantic segmentation and organ instance segmentation simultaneously.

4.2.1. Methodology

PlantNet [33] is a dual-function segmentation network specifically developed for multi-species crop point cloud datasets. It is able to perform leaf–stem semantic segmentation and leaf instance segmentation simultaneously. As shown in Figure 7a, the PlantNet model adopts a dual-pathway architecture comprising a shared encoder and a dual-pathway decoder that connects to two segmentation tasks, respectively. To effectively capture fine-grained geometric characteristics, PlantNet incorporates multiple Local Feature Extraction Operations (LFEOs) constructed upon EdgeConv blocks [55]. A Feature Fusion Module (FFM) is subsequently employed to integrate multi-level representations, while a spatial attention mechanism in the rear of the architecture emphasizes spatially salient features to enhance segmentation. Similarly, PSegNet [34] is a biologically inspired dual-function deep network designed for the organ semantic and instance segmentations of crop point clouds. As illustrated in Figure 7b, its architecture begins with a shared encoder containing a series of Local Feature Extraction Modules (LFEMs) for efficient local feature aggregation. In the middle part, a biologically inspired dual-pathway decoder that resembles the dorsal flow and the ventral flow inside the human brain is first used to form two streams of features, and then a Dual-granularity Feature Fusion Module (DGFFM) integrates the two complementary feature streams to enrich feature representation. The final stage combines spatial and channel attention mechanisms to further improve feature extraction. Ultimately, one pathway carries out organ semantic segmentation, and the other pathway conducts organ instance segmentation.

4.2.2. Semantic Segmentation Results

In the organ semantic segmentation experiments done by PlantNet and PSegNet on the Pepper-4D dataset, a total of 187 point clouds from Subset 1 were used for training and 50 point clouds from Subset 1 for testing. During data preparation, each point cloud was down-sampled to 4096 points using the Farthest Point Sampling (FPS) method to ensure uniform spatial resolution and low point redundancy. Ten-fold data augmentation based on FPS was applied to all training point clouds, resulting in a total of 1870 training samples. The quantitative results of our organ semantic segmentation experiments are summarized in

Table 2. Quantitative performances of PlantNet and PSegNet on organ semantic (stem/leaf) segmentation in Pepper-4D Subset 1. The “↑” sign means the higher the better, and the bold means the best in comparison.

Metrics	Network	Stem	Leaf	Mean
Prec (%) ↑	PlantNet	92.89	97.60	95.25
	PSegNet	93.06	97.68	95.38
Rec (%) ↑	PlantNet	91.63	98.08	94.86
	PSegNet	92.13	98.21	95.17
F1 (%) ↑	PlantNet	92.26	97.84	95.05
	PSegNet	92.59	97.95	95.27
IoU (%) ↑	PlantNet	85.63	95.78	90.71
	PSegNet	86.21	95.98	91.10

. The evaluation metrics for semantics include precision (Prec), recall (Rec), F1-score (F1), and Intersection-over-Union (IoU) [33] and are computed separately for stems and leaves in Table 2. Both networks achieve high accuracy with mean F1-scores and mean IoU values above 90.00%. Among them, PSegNet demonstrates a slightly better overall quantitative performance against PlantNet.

The qualitative results of organ semantic segmentation by the two networks are shown in Figure 8. Both PlantNet and PSegNet accurately separate the stem and leaf regions across samples with different structural complexity and demonstrate similar results against the Ground Truth. PSegNet, in particular, produces more stable predictions in dense canopy regions (the last column of Figure 8) and effectively mitigates local misclassifications along the boundaries between the leaves and the stems. The superiority of PSegNet over PlantNet may be explained by its biologically inspired architecture that enhances the capability to capture hierarchical geometric cues and spatial information.

4.2.3. Instance Segmentation Results

Organ instance segmentation aims to separate individual organs (usually leaves and fruits) for detailed phenotypic analysis. In this task, we applied PlantNet and PSegNet on the Pepper-4D dataset, with 187 training and 50 testing point clouds from Subset 1 to segment leaf instances. During data preparation, each point cloud was down-sampled to 4096 points using the Farthest Point Sampling (FPS) method to ensure uniform spatial resolution and to reduce point redundancy across samples. To further improve the training process, ten-fold data augmentation based on FPS was applied to all training point clouds, resulting in a total of 1870 training samples for actual training. The quantitative results of our leaf instance segmentation experiments are summarized in

Table 3. Quantitative performances of PlantNet and PSegNet for leaf instance segmentation on Pepper-4D. The “↑” sign means the higher the better, and the bold means the best in comparison.

Metrics	Networks	Leaf Instance Segmentation
mPrec (%) ↑	PlantNet	73.86
	PSegNet	76.58
mRec (%) ↑	PlantNet	87.48
	PSegNet	86.34
mCov (%) ↑	PlantNet	81.62
	PSegNet	82.57
mWCov (%) ↑	PlantNet	85.79
	PSegNet	86.37

, including evaluation metrics of the mean precision (mPrec), mean recall (mRec), mean Coverage (mCov), and mean Weighted Coverage (mWCov) [33]. Both PlantNet and PSegNet achieved strong performances with all metrics exceeding 70.00%. PSegNet exhibited slightly superior results compared to PlantNet, showing higher mPrec, mCov, and mWCov values.

The qualitative results of leaf instance segmentation by the two networks are presented in Figure 9, showing individual leaf segmentations across different growth stages with varying canopy densities. Both networks accurately identified most leaf instances, even under partial occlusion. PlantNet effectively detected small or emerging leaves near the bottom of the main stem but occasionally misclassified stem regions. PSegNet, in contrast, predicted more stable leaf instances along the main stem; however, PSegNet tends to produce over-segmentation on several leaves. Overall, both quantitative and qualitative results show that PSegNet achieved more reliable organ instance segmentation than PlantNet in our task, which may be explained by the benefit from PSegNet’s capacity to effectively capture hierarchical geometric features through the biologically inspired architecture and the dual-granularity feature extraction.

4.3. Detection of New Organs

The detection of new plant organs on crop point clouds is a task focusing on identifying emergent budding/new leaves in the 3D growth sequence. This task is crucial for plant phenotyping, as accurately detecting these temporal growth events offers critical insights into the physiological and structural responses of plants under varying environmental conditions. The new organ detection task remains challenging due to the tiny scales of buds, the imbalanced classes of mature organs and new organs, and the drastic morphological state change from the “new” organ to the “old” organ in growth. To address this task in the Pepper-4D dataset, we adopt the 3D-NOD [32] framework, which detects new organs across the point cloud sequences of pepper plants.

4.3.1. Methodology

The 3D-NOD [32] framework is specifically designed for new organ detection in spatiotemporal point cloud datasets. The design of 3D-NOD drew inspiration from how a well-experienced human utilizes spatiotemporal information to identify buds on a plant across several adjacent growth stages in the timeline. This framework transforms the detection problem into a point cloud segmentation task by integrating three key components. The Backward and Forward Labeling (BFL) process ensures temporally consistent annotations by refining organ labels across consecutive growth stages. The Registration and Mix-up (RMU) module performs spatial alignment between adjacent crop point clouds, fusing the spatial information and temporal information. The Humanoid Data Augmentation (HDA) strategy further strengthens model robustness by simulating human annotator behaviors, such as viewpoint variation and panoramic visibility, which aids in capturing small, newly emerging organs. Additionally, a Dynamic Graph CNN (DGCNN) [55] backbone, enriched with temporal encoding, is employed for new organ segmentation. Together, these components allow 3D-NOD to capture the subtle geometric and structural variations caused by growth development, thereby facilitating accurate and temporally consistent new organ detection.

4.3.2. New Organ Detection Results

The new organ detection experiments were done on eight partial growth sequences from Subset 2 in our Pepper-4D dataset, including a total of 80 point clouds, 60 samples of which were used for training and 20 others were for testing. During data preparation, we followed the steps of 3D-NOD and first annotated the entire dataset at the point level using the Backward and Forward Labeling (BFL) strategy [32], defining two semantic classes—the old organ and new organ (some examples are given in Figure 4b). Each point cloud was then down-sampled to 2048 points using the Farthest Point Sampling (FPS) algorithm to ensure both uniform spatial resolution and computational efficiency. Subsequently, we applied the Registration and Mix-up (RMU) step [32], which aligned every two adjacent point clouds in the timeline and then merged them into a single mixture containing 4096 points. This process automatically aligned the same organ instances in 4D space, ensuring both spatial and temporal coherence. Finally, the Humanoid Data Augmentation (HDA) strategy was employed to generate 10 times the augmented data of each mixed point cloud to increase data diversity. After training on a modified DGCNN backbone [55], a Split and Refinement step was applied to separate the combined point cloud back into the two original point clouds, effectively reversing the earlier Mix-up process. This step enables the 3D-NOD framework to refine the spatial boundaries of detected new organs. The evaluation metrics included precision (Prec), recall (Rec), F1-score (F1), and Intersection-over-Union (IoU) [32]. The 3D-NOD framework achieved a strong quantitative performance on the testing dataset, with a mean precision of 77.64%, recall of 84.90%, F1-score of 80.82%, and IoU of 71.88%. The qualitative results of two growth sequences from the testing dataset, as well as the GTs, are illustrated in Figure 10, in which the red-circled regions highlight newly detected organs. The quantitative and qualitative results show that 3D-NOD identifies new organs in our Pepper-4D dataset with high sensitivity and satisfactory accuracy. Overall, 3D-NOD exhibits a robust and temporally consistent performance for new organ detection in Pepper-4D, and also proves the Pepper-4D dataset to be an ideal dataset for time-series phenotyping analysis under spatiotemporal conditions.

4.4. Organ Tracking

Dynamic 3D phenotyping is essential for understanding plant growth and morphological development, as it enables the quantitative analysis of temporal structural variations and organ-level traits. However, significant challenges remain due to non-rigid organ deformation, frequent growth events such as budding and withering that continuously alter organ numbers, and the difficulty of maintaining the spatial and temporal correspondences of the same organ. To overcome these challenges, we not only need good organ-level growth tracking algorithms but also a high-precision spatiotemporal crop dataset, such as Pepper-4D, that allows algorithm testing. In order to perform the organ tracking task on Pepper-4D, we adopted TrackPlant3D [50] to track individual leaves across the different growth stages of pepper plants.

4.4.1. Methodology

TrackPlant3D [50] is an automated organ-level growth tracking framework developed for time-series crop point clouds, aiming to establish temporally consistent organ labels throughout sequential growth stages. The framework comprises three main stages—point cloud processing, organ correspondence estimation, and per-organ growth tracking. In the first stage, adjacent plant point clouds in the timeline are spatially aligned through iterated non-rigid registrations to minimize structural discrepancies caused by non-rigid morphological changes in growth. The second stage associates the same organ instance across time by combining the geometric similarity and spatial proximity on both local and global aspects. Finally, the third stage propagates and refines organ labels throughout the sequence to produce tracked point clouds. By modeling spatiotemporal relationships between consecutive growth stages, TrackPlant3D is able to achieve accurate short- and long-term organ tracking performances across several different plant species.

4.4.2. Organ Tracking Results

In the organ tracking experiment, we conducted TrackPlant3D [50] on the Pepper-4D dataset. A total of 236 point clouds from Subset 1 were used for evaluation. Each point cloud contains temporally aligned organ instance labels, including the stem system and individual leaves (several examples are shown in Figure 3d). Considering the dense organ structure of pepper plants, each point cloud was down-sampled to 10,000 points using the Farthest Point Sampling (FPS) method to ensure uniform spatial resolution and to reduce point redundancy before calculation. In the organ tracking process, randomly arranged organ instance labels were used as the segmentation input to TrackPlant3D, and the framework successfully established temporal correspondences among independently segmented organs for most cases, achieving accurate qualitative tracking results in Figure 11. Despite bending and the emergence of new leaves, most organ outputs of TrackPlant3D have a unique and consistent color identity in the time-series sequence, clearly illustrating label consistency. The satisfactory tracking result of TrackPlant3D also underscores the Pepper-4D dataset’s significance for reliable organ-level analysis and dynamic phenotyping research.

4.5. Generating Natural and Vivid 3D Plants

Natural and vivid 3D reconstructions of a plant/crop are beneficial to the fields of plant physiology research, precision agriculture, plant phenotyping research, and even the game industry. The generation of 3D plants aims to accurately represent real plant morphology with coherent organ topology, realistic branching, and smooth geometric transitions among different organs. When serving the plant phenotyping purpose, 3D-generated plants not only support the development of data-driven growth models for physiological simulation and crop output prediction, but they can also augment limited real datasets for improving the robustness of deep learning models. Mature methodologies such as L-systems [56,57] in this domain have achieved the realistic visualization of some crop species; however, generating a structurally coherent 3D plant remains challenging due to non-rigid deformation, self-occlusion, and the continuous evolution of hierarchical organ development during growth. Generative Adversarial Networks (GANs) [58] offer a promising solution by learning the underlying distribution of training data to synthesize realistic samples. By generating additional data beyond the original dataset, GANs alleviate data scarcity through synthetic augmentation, enhancing model robustness and generalization in deep learning applications. Most existing GAN frameworks target two-dimensional images or generic geometric objects; however, synthesizing realistic 3D plant point clouds remains more difficult due to the intricate topology of stems, leaves, and branching architectures. To advance this direction, the Pepper-4D dataset provides a comprehensive dataset for plant structure generation, and two representative GAN architectures—TreeGAN [59] and WarpingGAN [60]—were employed to evaluate their generative performance on realistic pepper point clouds.

4.5.1. Methodology

Generative Adversarial Networks (GANs) [58] are a class of deep generative models that learn the underlying distribution of training data to synthesize new, realistic samples through an adversarial learning process. A Generative Adversarial Network (GAN) typically consists of two adversarial components—a generator and a discriminator—that are jointly optimized in opposition to each other, as illustrated in Figure 12. The generator receives random noise as the input and synthesizes realistic 3D point clouds that emulate authentic plant structures in the training dataset. The discriminator, in contrast, learns to distinguish real point clouds from all outputs generated by the generator, thereby driving the generator to progressively refine its outputs until they become similar to real crop data. Based on our Pepper-4D dataset, two representative GAN architectures, TreeGAN [59] and WarpingGAN [60], were employed for 3D pepper plant generation. TreeGAN utilizes tree-structured graph convolutions that keep structurally consistent point distributions and coherent organ geometries. WarpingGAN, on the other hand, incorporates a warping mechanism that deforms multiple uniform 3D priors into realistic plant shapes, achieving enhanced geometric coherence and structural uniformity across synthesized pepper samples.

4.5.2. Results of 3D Generation of Pepper Plants

In this subsection, we show the point cloud results from both TreeGAN and WarpingGAN on the Pepper-4D dataset and evaluate their respective generative performances. A total of 20 pepper plant samples were used for training, with each point cloud down-sampled to 2048 points using Farthest Point Sampling (FPS) and augmented 128-fold to ensure both uniform spatial resolution and robust adversarial training. As illustrated in Figure 13, the qualitative results demonstrate that both TreeGAN and WarpingGAN can generate plausible pepper plant morphologies that approximate the original samples but mix with diversities at the same time. It is also interesting to note that TreeGAN and WarpingGAN differ in detail. TreeGAN produces sharper organ-level geometries (such as leaf surface and branching hierarchies), owing to its tree-structured graph convolutions; however, several TreeGAN-generated outputs encounter incomplete reconstruction, with sparse or missing leaf regions. WarpingGAN generates overall (globally) coherent plants but sometimes exhibits distorted or partially deformed organs, particularly in the upper canopy. Together, TreeGAN and WarpingGAN demonstrate the effectiveness of adversarial learning for generating complex 3D plant architectures, and they also reveal the potential value of our Pepper-4D dataset to advance the research of 3D-generated plant models.

5. Conclusions

This study introduces Pepper-4D, an open spatiotemporal (4D) point cloud dataset that captures the developmental cycle of pepper plants from early growth to senescence under controlled indoor conditions. The dataset consists of 916 high-precision point clouds collected from 29 plants and supports a wide range of 3D plant phenotyping tasks, including health classification, organ semantic segmentation, organ instance segmentation, new organ detection, organ-level growth tracking, and 3D virtual plant generation.

Beyond benchmarking 3D plant-understanding models, Pepper-4D provides practical value for dynamic phenotyping and controlled-environment cultivation research. It enables the quantitative analysis of stage-dependent architectural changes (e.g., organ emergence, canopy expansion, and senescence progression) and supports algorithm development for applications such as new growth detection, health/stress assessment over time, and organ-level trait extraction. By combining detailed 3D structural information with spatiotemporal annotations, Pepper-4D helps fill an important need for the accurate, scalable, time-resolved analysis of growth dynamics from 3D data.

Our future work will focus on enhancing dataset completeness and usability by extending organ-level annotations to the full dataset and incorporating additional labels for new organ detection and growth tracking analysis. In parallel, we will explore advanced learning frameworks for organ segmentation and spatiotemporal plant growth modeling that better leverage the temporal richness of Pepper-4D. We believe Pepper-4D will continue to evolve as a reliable and comprehensive 4D resource for plant phenotyping and precision agriculture.