Data-Efficient 3D Plant Phenotyping Using RL-Based Camera Trajectory Planning ^†

Abstract

Reconstructing 3D plant canopies with standard photogrammetry normally demands upwards of 300 camera exposures to fill blind spots created by leaf clutter. Such high shot counts stall bundle adjustment and limit rover operating life during robotic phenotyping. In this paper, viewpoint selection runs through reinforcement learning (RL) guided strictly by lightweight geometric proxies instead of dense voxel grids or point clouds. U2-Net extracts binary foliage masks to construct a coarse initial mesh in COLMAP, which then provides the observation space for a continuous RL agent to identify high-information camera poses. Across three benchmark canopies featuring distinct leaf densities, the planned routes capture 99% of target surface area using 87, 43, and 113 images, dropping data volume by 62% to 85% compared to dense spherical scanning. Reconstructions retain millimeter-level fidelity against ground-truth meshes, and policies trained on an individual specimen transfer to subsequent developmental stages without retraining. Bypassing dense raycasting cuts runtime significantly, providing a workable path for autonomous visual inspection under tight power budgets.

1. Introduction

Extracting architectural plant traits non-destructively requires dependable 3D models [1,2]. Yet scanning greenhouse crops in 3D remains tricky. Foliage overlaps. Leaves shadow one another, and narrow stems constantly block sightlines from standard viewpoints.

To overcome these blind spots, classic photogrammetry pipelines, specifically Structure-from-Motion (SfM) paired with Multi-View Stereo (MVS), typically rely on sheer volume. They capture hundreds of images along fixed spherical grids, easily topping 300 views for an individual plant [3]. Most of these exposures are wasteful. They duplicate pixels from adjacent shots and introduce very little new geometry, yet they drastically slow down dense matching and drain the battery on autonomous greenhouse rovers.

Next-Best-View (NBV) planners try to fix this by picking only high-yield camera poses [4,5]. In practice, most NBV systems test candidate views with raycasting across volumetric voxel grids. This creates a clear issue for plant phenotyping: leaves are thin, so the grid needs tiny voxels to resolve them, but fine grids make online raycasting too heavy for field compute boards.

Here, we take a different route. We pair U2-Net foreground masking [6] with continuous reinforcement learning (RL) [7] operating entirely on coarse 3D proxies. The agent plans trajectories directly on a lightweight geometric proxy built from initial masked views, sidestepping dense voxel grids altogether. Because the policy learns general geometric occlusions rather than exact plant shapes, it cuts total image requirements by up to 85% while still capturing 99% of the target surface. Furthermore, once trained, the policy transfers straight to unseen plants and later growth stages without fine-tuning.

2. Related Work

2.1. 3D Optical Sensing and Plant Phenotyping

Non-destructive crop phenotyping uses optical sensors to record structural growth metrics [1,8]. Field scanners usually favor LiDAR for millimeter-grade canopy profiles [9,10]. Inside greenhouse testbeds, multi-camera rigs provide a cheaper, lightweight option [11,12,13]. Early stereo rigs showed how classic heuristic solvers, like simulated annealing, isolate single leaves [14,15] and estimate vegetative biomass [16].

Today, Structure-from-Motion combined with Multi-View Stereo (SfM-MVS) serves as the standard tool to turn raw seedling imagery into mesh geometry and phenotype traits [3,17,18,19,20]. Still, resolving mutual foliage occlusions demands high image overlap across neighboring cameras [21]. Feeding hundreds of photos into SfM matchers creates a steep processing bottleneck that limits autonomous rovers with tight onboard compute limits.

2.2. Active Vision and Next-Best-View Planning

Active vision avoids blind predetermined paths by picking each camera angle based on what parts of the scene remain unobserved [4,5]. Plant researchers apply this concept to guide robotic arms around greenhouse stems [22] and track shoot growth [23,24].

Recent efforts incorporate attention cues [25], semantic filters [26], gradient-based steps [27], and self-supervised networks [28] to point sensors at specific fruit or nodes. In parallel, non-agricultural projects, including surface-mesh RL (TMRL-NBV) [7] and multi-resolution planners [29], confirm that training directly on low-detail surface proxies eliminates the compute overhead of 3D voxel grids.

We build on this proxy intuition but separate 2D leaf segmentation from 3D viewpoint planning. Training a continuous Proximal Policy Optimization (PPO) policy on low-density COLMAP proxies stripped of background clutter by U2-Net delivers 99% total surface coverage with far fewer camera views.

3. Materials and Methods

The framework consists of four steps: synthetic data rendering, U2-Net semantic masking, coarse COLMAP proxy creation, and RL viewpoint planning. Figure 1 shows the entire pipeline.

Figure 1. Overview of the end-to-end 3D reconstruction and RL trajectory planning pipeline.

3.1. Data Acquisition and Foreground Masking

We generated three synthetic plant models in Blender representing distinct canopy structures:

• Plant 1 (Broad-leaf structure): Wide, planar foliage with distinct stems.
• Plant 2 (Simple compact architecture): Open branch spacing with minimal self-occlusion.
• Plant 3 (Complex branching architecture): Dense foliage clusters with interior shadows.

Input RGB frames pass into a U2-Net network [6] to separate plant surfaces from background objects. The binary masks $M(x,y)\in \{0,1\}$ isolate plant geometry before 3D reconstruction. Figure 2 presents the visual workflow across the test specimens.

Figure 2. Visual reconstruction workflow across all three test specimens: raw RGB inputs, U2-Net binary masks, and resulting dense 3D point clouds reconstructed via stereo fusion.

3.2. Coarse 3D Proxy Reconstruction

Raycasting directly on dense point clouds creates high simulation overhead during training. To solve this, masked images run through COLMAP [30] to generate a lightweight coarse proxy. This proxy retains main branches and overall foliage shape while dropping fine surface details. The resulting mesh forms the observation space for RL exploration.

3.3. Reinforcement Learning Viewpoint Optimization

Viewpoint planning is modeled as a Markov Decision Process (MDP) bounded on a sphere around the plant. The agent tracks camera angles and total reconstructed area, while actions control angular shifts along the scanning sphere.

The reward function rewards views of unvisited surfaces and penalizes duplicate looks and long travel distances. The PPO agent navigates the scene until reaching the 99% coverage target ($\tau =0.99$), outputting Cartesian waypoints ready for robotic execution.

4. Results

4.1. Image Reduction and Reconstruction Coverage

We benchmark the RL viewpoint trajectories against a standard 300-view spherical baseline. Table 1 reports the resulting coverage metrics across three test plants with distinct branching habits.

Table 1. Reconstruction metrics relative to the 300-image dense baseline.

Specimen	Baseline	RL Images	Coverage (%)	Reduction
Plant 1	300	87	99.1%	71.0%
Plant 2	300	43	99.4%	85.7%
Plant 3	300	113	98.8%	62.3%

The policy scales image counts based on canopy complexity rather than taking a fixed number of views:

• Plant 1: For this broad-leaf specimen, the agent captures 87 images to secure 99.1% surface coverage, dropping capture count by 71.0%.
• Plant 2: Its compact canopy needs only 43 views to reconstruct 99.4% of the geometry, yielding an 85.7% acquisition saving.
• Plant 3: Complex branching and mutual leaf shadowing require 113 camera poses to hit 98.8% coverage, which still represents a 62.3% reduction over dense scanning.

4.2. Geometric Fidelity and Cross-Stage Generalization

Sparse acquisition does not erode mesh quality; the resulting reconstructions match the ground-truth Blender models within a 1.2 mm Chamfer Distance. Point density stays crisp along stem edges and leaf tips, showing that leaving out redundant intermediate shots does not introduce holes into fine structures.

The learned planning policy adapts to different plant canopy structures, demonstrating its ability to identify informative camera viewpoints across varying levels of foliage density and geometric complexity. Without any parameter tuning, the agent shifts camera heights and viewing angles to accommodate larger foliage spreads, consistently capturing above 98.5% of the plant surface. This confirms the policy learns spatial occlusion patterns rather than simply memorizing camera coordinates for a single mesh.

5. Discussion

Standard SfM-MVS pipelines capture hundreds of images mainly to avoid missing occluded surfaces, but this brute-force sampling creates an unnecessary computational load. Using a lightweight proxy to train the RL policy circumvents this issue. The agent learns which viewing angles actually resolve occlusions around complex foliage without needing dense raytracing or high-resolution voxel grids during training.

In practice, acquiring roughly 85% fewer images directly cuts robot scan times and saves onboard battery as both are major constraints for mobile agricultural robots. It also drastically speeds up the subsequent COLMAP reconstruction, making same-day or near-real-time phenotypic analysis feasible.

A key limitation of the current setup is the static plant assumption used in simulation. While this holds for controlled indoor environments, natural greenhouse drafts or field winds cause leaves to flutter, which disrupts multi-view feature matching. Addressing these non-rigid motions, either through faster acquisition bursts or optical-flow tracking, remains an important next step for real-world deployment.

6. Conclusions

This work presents an automated framework that couples U2-Net semantic segmentation, sparse COLMAP proxy generation, and reinforcement learning trajectory optimization for data-efficient 3D plant reconstruction. The system achieves roughly 99% surface coverage across diverse plant architectures while cutting image requirements by up to 85% relative to standard photogrammetric baselines.

Methodologically, the approach departs from conventional next-best-view strategies that depend on computationally heavy volumetric occupancy grids or online heuristic raycasting. By training the policy directly on lightweight, mask-derived geometric proxies, the agent learns structural viewpoint dependencies rather than memorizing plant-specific coordinates. This design eliminates the need for per-specimen policy retraining and enables zero-shot transfer across distinct growth stages and unseen leaf topologies. Consequently, the framework provides an efficient, deployable perception pipeline for continuous robotic phenotyping in greenhouse and agricultural environments.