Reinforcement Learning for Stand Structure Optimization of Pinus yunnanensis Secondary Forests in Southwest China

Abstract

Aiming to enhance the efficiency and precision of multi-objective optimization in southwestern secondary growth of Pinus yunnanensis forests, this study integrated spatial and non-spatial structural indicators to establish objective functions and constraints for assessing forest structure. Felling decisions were made using the random selection method (RSM), Q-value method (QVM), and V-map method (VMM). Actions taken to optimize the forest stand structure (FSS) through tree selection were approached as decisions by a reinforcement learning (RL) agent. Leveraging RL’s trial-and-error strategy, we continually refined the agent’s decision-making process, applying it to multi-objective optimization. Simulated felling experiments conducted across circular sample plots (P1–P4) compared RL, Monte Carlo (MC), and particle swarm optimization (PSO) in FSS optimization. Notable enhancements in the values of the objective function (VOFs) were observed across all plots. RL-based strategies exhibited improvements, achieving VOF increases of 17.24%, 44.92%, 34.66%, and 17.10% for P1–P4, respectively, outperforming MC-based (10.73%, 41.54%, 30.39%, and 15.07%, respectively) and PSO-based (14.08%, 37.78%, 26.17%, and 16.23%, respectively) approaches. The hybrid M7 scheme, integrating RL with the RSM, consistently outperformed other schemes across all plots, yielding an average 26.81% increase in VOF compared to the average enhancement of all schemes (17.42%). This study significantly advances the efficacy and precision of multi-objective optimization strategies for Pinus yunnanensis secondary forests, emphasizing RL’s superior optimization performance, particularly when combined with the RSM, highlighting its potential for optimizing sustainable forest management strategies.

1. Introduction

Pinus yunnanensis holds a dominant position in the coniferous landscapes of southwestern China, establishing itself as the primary forest type within Yunnan Province. Widely cultivated for reforestation and ecological engineering endeavors, this species has garnered significant attention [1]. Stretching across the vast expanse of southwestern China, it swathes approximately 52% of forested terrain and contributes to 32% of the overall timber yield [2]. As a trailblazing tree species, Pinus yunnanensis thrives in rocky, nutrient-deprived soils. Its tolerance of shade and resilience in drought conditions, attributed to its deep-rooted system, further underscore its distinctive attributes [3]. Notably, its significance extends to regional economic advancement and ecological restoration pursuits [2,3,4]. Unraveling the inherent dynamics of Pinus yunnanensis plays a pivotal role in reinvigorating secondary forests, and in turn, contributes to China’s ambitious goal of achieving carbon neutrality by 2060. This prospect arises from the potential to cultivate effective forest carbon sinks within the region [5].

Secondary forests often grapple with challenges such as uneven forest stand structure (FSS), diminished biodiversity, heightened susceptibility to forest fires, and vulnerability to natural disasters like pests, diseases, and wildfires [6,7,8]. Presently, the majority of secondary forests harboring Pinus yunnanensis in Yunnan Province, China, are situated within the middle-aged and young forest stages. These forests exhibit a simplistic stand composition, unhealthy structures, severe quality degradation, and significant susceptibility to fires, pests, and diseases [9]. The local government has prioritized the advancement and safeguarding of ecological environments. Consequently, an imperative for research focused on the wholesome management of Pinus yunnanensis secondary forests has arisen. However, pertinent investigations remain scarce, particularly concerning the secondary forests of Pinus yunnanensis along the eastern slopes of Cangshan Mountain in Dali, Yunnan Province, which have yet to be addressed in the existing literature. As a result, embarking on comprehensive research dedicated to Pinus yunnanensis secondary forests not only holds notable theoretical scientific significance but also holds substantial practical implications.

Optimizing and regulating FSS is one of the most direct and effective technical approaches in forest management and planning, particularly in the management of secondary forests. The existing studies have predominantly focused on rectangular plots for adjustments, with a variety of optimization methods, yet research addressing the optimization of circular small plots regarding FSS remains scarce. The majority of studies related to FSS optimization primarily revolve around selective logging as the main optimization measure. Identifying the combination of trees that need selective logging rapidly and accurately is a critical and challenging issue within the optimization process. The key to resolving these issues lies in determining FSS indicators, the preliminary identification of trees for selective cutting, constructing optimization models, and designing solution algorithms. FSS indicators encompass both non-spatial and spatial structural indicators, including non-spatial indicators such as tree diameter distributions [10], species counts, and canopy density [11], alongside spatial indicators like the mingling index (M) [12,13], canopy competition index (CI) [14,15], angle index (W) [16,17], storey index (S) [18], and open comparison (OP) [19]. These indicators comprehensively evaluate various aspects of FSS from different perspectives.

Three primary methods are utilized for the initial determination of potential trees for felling: the random selection method (RSM) [20], Q-value method (QVM) [21], and V-map method (VMM) [22]. The RSM is straightforward, involving the random sampling of initial forest trees within a plot to assemble a collection of trees for potential felling. The QVM employs a multiplication and division strategy, considering larger-is-better metrics as the numerator and smaller-is-better metrics as the denominator when combining forest structure indicators. This process generates a comprehensive single-tree evaluation index, Q. By computing the Q-value for each central tree and sorting them in ascending order, trees ranked higher exhibit poorer performance within the spatial structure, thus making them more suitable for removal. The top ‘n’ central trees within the felling quota constitute the collection of trees identified by the QVM for potential felling. Similar to the QVM’s approach to tree selection, the VMM divides the forest trees within a plot into a Voronoi diagram. It calculates the angle index for each central tree and the average angle index for the plot. Utilizing the average angle index as the criterion for tree selection, trees with an angle index higher or lower than the average, based on optimization goals, are selected for the collection of trees identified for potential felling.

An enhanced understanding of the factors impacting forest ecosystems has led us to realize that a well-structured forest stand is not merely a singular objective. The improvement of multiple structural indicators interacts, presenting a challenge akin to multi-objective optimization, characterized by multi-objectives and non-linearity. In such scenarios, traditional simplistic mathematical planning methods struggle to suffice, whereas multi-objective optimization problems (MOPs) offer a more comprehensive solution. They permit us to concurrently balance and optimize multiple interrelated objectives, rather than solely pursuing a singular optimal solution. This approach enables a more holistic comprehension and management of the diversity and complexity within FSSs, thereby effectively enhancing the overall health of forest ecosystems. Commonly used intelligent optimization algorithms for solving MOPs include heuristic algorithms like Monte Carlo (MC) [23,24,25,26], bio-inspired algorithms such as genetic algorithms (GA) [27], simulated annealing (SA) [28,29], and particle swarm optimization (PSO) [30,31,32,33], among others. MC is an early heuristic algorithm applied in forest management due to its simplicity and high programmability, yet its results often lack accuracy due to inherent algorithmic shortcomings. Bio-inspired algorithms like GA and PSO frequently encounter issues like susceptibility to local optima. Throughout the solution process, these methods tend to exhibit high fluctuations and discreteness in the values of the objective function (VOFs) [34]. However, particle swarm optimization offers the advantage of fast search speed and finds relatively more applications in multi-objective optimization of forest stand structures (MOFSSs). Yet, due to the inherent complexity of forest ecosystems, using these algorithms for such problems often presents challenges such as extensive computational requirements, intricate solution processes, and difficulties in optimization [26,30,35]. Particularly in cases where forest stand structures are complex or involve numerous trees, the aforementioned algorithms often struggle with low solution efficiency and subpar optimization accuracy.

RL represents an intelligent algorithm rooted in the trial-and-error strategy, characterized by its conceptual simplicity, absence of strict modeling requirements, dynamic decision-making capabilities, and robust adaptability [36]. Leveraging continuous trial and learning, RL can progressively enhance its efficacy, rendering it well-suited for addressing dynamically evolving multi-objective optimization challenges. The inherent complexity of RL resides in formulating an effective reward function, a pivotal aspect that enables the algorithm to converge toward improved outcomes within a shorter time frame [37,38]. Furthermore, the efficiency of RL hinges on the ongoing interaction between the intelligent agent and its environment, necessitating a substantial volume of samples for training. As a result, RL has been widely applied in dynamic multi-objective optimization fields [39,40] such as optimal smart grid management, intelligent transportation systems, and unmanned aircraft control [41,42]. These domains not only provide ample training samples but also establish a conducive environment for effective interactions with intelligent agents.

RL has found extensive application in the field of forestry. Research endeavors have concentrated on various aspects such as simulated fire fighting [43], forest fire detection [44], optimal harvest modeling [45], and forest management [46], offering crucial insights into these domains. Despite their significant contributions to the efficient preservation and management of forests, these research areas differ significantly in their focus and methodologies. Each domain adopts distinct approaches and tools tailored to address specific challenges and requirements. However, there is currently a significant gap in research concerning the application of RL in multi-objective optimization within the field of forest stand structure. This study is focused on utilizing RL techniques to address the multi-objective optimization challenges associated with secondary forest stand structure of Pinus yunnanensis. We aim to bridge the existing gap in RL-based multi-objective optimization within the domain of FSS optimization. While other machine learning techniques have found widespread application in forestry [47,48], particularly in single-objective studies such as fire prediction [49], yield forecasting [50], and LiDAR-based forest structure identification [51], our extensive literature review has revealed a lack of these methods applied to address multi-objective optimization in FSSs. This highlights a deficiency in these techniques when tackling complex multi-objective optimization scenarios. By employing RL methods, our study focuses on the multi-objective optimization of secondary forests of Pinus yunnanensis. Differing from existing research, we concentrate on addressing the challenges posed by multiple objectives, an area not fully explored in the current literature. Our research carves out a new path in forest structure optimization, aiming to bridge the research gap in machine learning approaches to multi-objective optimization in FSSs. Simultaneously, while deep learning excels in managing large datasets and pattern recognition, its application in forest management encounters certain limitations. Challenges in environmental interaction and reward function design restrict its current primary application to forest inventory and planning [52,53], without fully addressing complex multi-objective optimization problems.

Hence, this study chose the MC algorithm for its simplicity, ease of programmability, and historical relevance, despite its potential for yielding less accurate results. As a supplementary method, we utilized the PSO algorithm, known for its rapid search speed and widespread application, albeit with a susceptibility to local optima, as our control group. Utilizing the forest stand survey database of Pinus yunnanensis secondary forests in Yunnan Province, southwestern China, we conducted simulated felling optimization across various sample plots. Through comparative analysis of the simulation outcomes, we assessed the impact of three methods for determining potential felling trees on the optimization of FSSs. Furthermore, this study evaluated the performance of three multi-objective optimization algorithms and explored the application of Q-learning, an RL method, in optimizing the structure of secondary forests dominated by Pinus yunnanensis. The research findings demonstrate the potential advantages of employing RL in addressing optimization challenges related to FSS. This approach holds promise in overcoming prevalent hurdles in widespread applications. Specifically focusing on the management and optimization of secondary Pinus yunnanensis forests in southwest China, this study provides valuable guidance and direction. Moreover, its adaptability and transferability offer valuable insights for similar forest ecosystem management issues in other regions.

2. Materials and Methods

2.1. Study Areas

The forest data collection was conducted in the vicinity of Cangshan Mountain (${25}^{\circ }{34}^{\prime }$–${26}^{\circ }{00}^{\prime }$ N, ${99}^{\circ }{55}^{\prime }$–${100}^{\circ }{12}^{\prime }$ E), Yunnan Province, southwestern China (Figure 1). The surveyed areas encompass Zhonghe Peak, Foding Peak, Malong Peak, Lan Peak, and others. These peaks are situated on the eastern slope of Cangshan Mountain. The eastern slope of Cangshan Mountain falls within the subtropical climate zone, characterized by an annual average temperature of 15 °C and prevailing southwest monsoon winds. Abundant annual rainfall exceeding 1000 mm is observed, with distinct dry and wet seasons, primarily concentrated from May to October, contributing to 84% of the total annual precipitation [9]. The prevalent soil type in this area is red soil.

The main tree species in the study areas include Pinus yunnanensis, Pinus armandii Franch., Quercus acutissima Carruth, Vaccinium bracteatum Thunb, Ternstroemia gymnanthera, Betula alnoides, Gaultheria griffithiana Wight, Eucalyptus robusta Smith, Quercus aliena, Q. varia bilis, Abies fabri, Keteleeriaevelyniana, Alnus nepalensis, Juglans regia L, and Populus tremula Linn, among others.

2.2. Study Site and Data Collection

The fundamental statistics of the sample plots are presented in

Table 1. Key information about sample plot attributes.

Sample Plots	East Long.	North Lat.	Elevation (m)	Slope (${}^{\circ }$)	Slope Dir.	Sample Plot Radius (m)	Tree Species Composition	Stand Density (Trees/ha)
P1	100°08.2149${}^{″}$	25°41.5280${}^{″}$	2254	13.45	East	35	8 PY-2 PA-BA-TG	1481
P2	100°10.9639${}^{″}$	25°38.1518${}^{″}$	2271	16.15	South	32	7 PY-3 PA	1822
P3	100°09.3947${}^{″}$	25°39.9506${}^{″}$	2195	17.70	NE	20	7 PY-3 PA-QA-VB GG-BA	1830
P4	100°07.1906${}^{″}$	25°43.5923${}^{″}$	2138	5.10	NE	19	10 PY-QA	1975

Note: NE, northeast; PY, Pinus yunnanensis; PA, Pinus armandii; QA, Quercus acutissima; VB, Vaccinium bracteatum; GG, Gaultheria griffithiana; BA, Betula alnoide; TG, Ternstroemia gymnanthera; DBH, diameter at breast height. The column ’Tree Species Composition’ represents the quantity of tree species within each plot. The numerical values preceding each tree species indicate the proportion of that species for every 10 trees within the plot.

. The predominant tree species, Pinus yunnanensis, was surveyed across four circular sample plots, each located at a distinct peak. The radii of these circular sample plots measure 35 m (Zhonghe Peak), 32 m (Foding Peak), 20 m (Malong Peak), and 19 m (Lan Peak).

Between July and December 2022, forestry operations were carried out within the aforementioned sample plots. Over this timeframe, the geographical coordinates, elevation, slope, slope direction, and radius of each sample plot were meticulously measured and recorded. A comprehensive examination of live standing trees with a diameter at breast height (DBH, the diameter of a tree measured at 1.3 m above the ground) equal to or greater than 5 cm (DBH ≥ 5 cm) was conducted within the sample plots. For each tree, essential forestry attributes, such as species, DBH, tree height (TH, the vertical distance from the tree base to its highest point), and crown width (CW, the average of the longest horizontal distances of the crown in the four cardinal directions), were documented with specialized altimeters and distance measurers. The accurate relative coordinates to the center of the sample plots for the base of each tree were determined using a Topcon GTS-2002 autofocus total station (Topcon, Tokyo, Japan).

2.3. Determination of Spatial Structure Units and Edge Correction

The Voronoi method was employed to delineate the spatial structure units [54]. This approach to constructing forest stand spatial structure units using the Voronoi diagram accounts for the unique characteristics of each individual tree, resulting in a more rational and comprehensive representation. Based on the measured relative positional coordinates of the forest trees, the Voronoi diagram was generated using R4.2.0. In this diagram, each polygon corresponds to a spatial structure unit formed by each forest tree and its neighboring trees [55,56]. The spatial structure units of edge stands can be impacted by the boundaries of the sample, potentially introducing calculation errors in the spatial structure index.

To mitigate these errors, this study suggests implementing the buffer zone method [57,58]. Specifically, the sample plot boundary was confined within a b-meter circular area to serve as a buffer zone. When calculating the spatial structure index for the spatial structure unit composed of the central tree and its adjacent trees, only the forest trees located within the buffer zone were considered in forming the spatial structure unit as adjacent trees [59], as depicted in Figure 2.

2.4. Stand Structure Indexes

(1) Tree Diameter Classes [10]

The categorization of trees into distinct diameter classes stands as a pivotal facet of our analysis. Elevation in the number of these classes distinctly correlates with enhanced stand growth. For this investigation, we embarked upon categorization based on DBH, commencing from 6 cm DBH and advancing in 2 cm increments. This systematic approach ensured the preservation of the number of diameter classes amidst stand optimization:

$$D=D_0$$

(1)

where $D_0$ denotes the count of diameter classes before selection cutting, and D signifies the count post-selection cutting.

(2) Number of Species

A paramount objective during selection cutting entails the safeguarding of tree species diversity, preventing any inadvertent eradication. We vigilantly upheld the assurance that the count of tree species would remain unaltered:

$$T=T_0$$

(2)

where $T_0$ signifies the initial count of tree species, and T denotes the count post-selection cutting.

(3) Cutting Intensity

The vigor of the stand post-optimization hinges upon the determination of cutting intensity. Ideally, the annual volume of cutting should remain below the annual growth rate of the stand. Previous research [61,62] has indicated that the optimal thinning intensity for secondary forests of Pinus yunnanensis should not surpass 35%:

$$N\ge N_0\left(1-35\%\right)$$

(3)

where $N_0$ signifies the total count of trees prior to selection cutting, and N represents the count post-selection cutting.

(4) $CD$ (canopy density) [11]

In practical forest management, the formation of a continuous canopy is a quintessential requirement. Generally, a canopy density of not less than 0.7 signifies an adeptly continuous forest floor cover:

$$CD\ge 0.7$$

(4)

Spatial Structure Indexes

(1) M (mingling index)

The quantification of spatial segregation of tree species, known as M, is ascertained by the ratio of neighboring tree j stem count, not belonging to the same species as the focal tree i, to the overall neighboring tree stem count. Its mathematical representation is given by [12,13]:

$$M_i=\frac{1}{n}\sum _{j=1}^n{v}_{ij}$$

(5)

where $M_i$ symbolizes the mingling degree of the focal tree i, and $v_{ij}$ is a binary variable. When the neighboring tree j is not of the same species as tree i, $v_{ij}$ = 1; otherwise, $v_{ij}$ = 0. This metric, $M_i$, ranges from 0 to 1, indicative of different degrees of mixing, with $M_i=0$ signifying zero-grade mixing and $M_i=1$ indicating extreme hybridization.

(2) $CI$ (canopy competition index)

To gauge the competitive pressures among trees, we designed a $CI$ that employs the overlap area of canopies. The index is calculated as follows [15]:

$$C{I}_i=\frac{1}{Z_i}\times \sum _{j=1}^{n}A{O}_{ij}\times \frac{L_j}{L_i}$$

(6)

Equation $C{I}_i$ is the canopy competition index of object tree i, $Z_i$ is the projected area of the canopy of object tree i, $L_i=H_i$ × $C{w}_i$ × $C{L}_i$ (height of object tree i × canopy width of object tree i × crown length of object tree i), $L_j=H_j$ × $C{w}_j$ × $C{L}_j$ (height of competing tree j × canopy width of competing tree j × crown length of competing tree j). $A{O}_{ij}$ is the canopy overlap area between object tree i and competitor tree j; when object tree i and competitor tree j do not overlap, $A{O}_{ij}=1$.

When object tree i overlaps with competing tree j, the canopy overlap area $A{O}_{ij}$ is calculated as follows.

The total area of shading produced by n competing trees j on object tree i is first calculated with the formula [14]:

$$\begin{array}{c}\hfill S_0=\frac{C{w}_i^2}{2}arccos\left(\frac{q_j^2}{2C{w}_i^2}-1\right)-\frac{1}{4}\sum _{j=1}^n{q}_j\sqrt{4C{w}_i^2-q_j^2}\end{array}$$

(7)

The total shading area of object tree i over n competing trees j is then calculated by the formula [14]:

$$\begin{array}{c}\hfill S_1=\frac{1}{2}\sum _{j=1}^n\left[C{w}_j^{2}arccos\left(1-\frac{4C{w}_i^2-q_j^2}{2C{w}_j^2}\right)-\frac{\sqrt{4C{w}_i^2-q_j^2}}{2}\sqrt{4C{w}_j^2-\left(4C{w}_i^2-q_j^2\right)}\right]\end{array}$$

(8)

Finally, the total area of the overlapping part of the object tree i and the competing tree j is calculated with the formula:

$$A{O}_{ij}=S_0+S_1$$

(9)

Equation $q_j=\frac{L_{ij}^2-\left(C{w}_j^2-C{w}_i^2\right)}{L_{ij}}$, where $L_{ij}$ is the distance between the competing tree j and the object tree i, $C{w}_i$ is the canopy radius of the subject tree i, $C{w}_j$ is the radius of the canopy of the competing tree j, n is the number of competing trees.

(3) W (uniform angle index)

The delineation of the stand’s horizontal distribution pattern finds expression through W. This index captures the proportion of angles $\alpha$ between the focal tree i and its nearest neighbors, smaller than a predefined standard angle ${\alpha }_0$. Mathematically, this is represented as [16,17]:

$$W_i=\frac{1}{n}\sum _{j=1}^n{z}_{ij}$$

(10)

where $W_i$ signifies the angle index of tree i, and $z_{ij}$ is binary. If the angle $\alpha$ between trees i and j is less than ${\alpha }_0$, $z_{ij}$ = 1; otherwise, $z_{ij}$ = 0. The index values of $W_i$ offer insights into the degree of distribution uniformity, ranging from an extremely uniform distribution ($W_i=0$) to a highly inhomogeneous distribution ($W_i=1$).

(4) S (storey index)

The vertical diversity and complexity within a stand are encapsulated by S, which quantifies the proportion of neighboring trees j at the same height level as the focal tree i. Mathematically, this index is represented as [18]:

$$S_i=\frac{1}{n}\sum _{j=1}^n{v}_{ij}$$

(11)

where $S_i$ represents the storey index of tree i, and $v_{ij}$ is a binary variable. If tree i shares the same height level as tree j, $v_{ij}$ = 0; otherwise, $v_{ij}$ = 1. The $S_i$ index provides insights into the vertical structural diversity of the stand.

(5) $OP$ (open comparison)

The $OP$ encapsulates the light environment and growing space size of tall trees within the stand. This metric quantifies the extent to which the focal tree i within a spatial structure unit is shaded by its neighboring tree j. Mathematically, it is expressed as [19]:

$$O{P}_i=\frac{1}{n}\sum _{j=1}^n{t}_{ij}$$

(12)

Within Equation $\left(9\right)$, $O{P}_i$ stands for the open comparison corresponding to object tree i, while $t_{ij}$ is characterized as a discrete variable. If the horizontal distance between the object tree i and its neighboring tree j surpasses the disparity in height between them, then $t_{ij}$ is assigned a value of 1; contrarily, it receives a value of 0. $O{P}_i$ assumes five distinct categories: 0 signifies that the luminous environment for object tree i is entirely shaded; $0.25$ implies that the light surrounding object tree i is somewhat obscured; $0.5$ suggests a moderately open luminous environment for object tree i; $0.75$ indicates a notably open luminous setting around object tree i; lastly, 1 indicates a luminous atmosphere of utmost openness, characterized by an abundant influx of direct light.

The quantification of stand structure is the basis of stand structure optimization. In this study, we selected the number of tree diameter classes, the number of tree species, and the canopy density as indexes to quantify the non-spatial structure of the stand and the uniform angle index, mingling index, crown competition index, storey index, and open comparison as indexes to quantify the spatial structure of the forest stand. The uniform angle index describes the horizontal distribution pattern of trees, the mingling index describes the degree of species isolation, the crown competition index describes the competitive pressure among trees, the storey index describes the vertical distribution pattern of trees, and the open comparison describes the intensity of understory illumination.

2.5. Methods for Selecting Trees for Cutting

2.5.1. RSM (Random Selection Method) [20]

A swift random sampling approach was employed from the original pool of retained trees (comprising all trees within the stand) to acquire the subset of trees designated for selection, ensuring that the cutting intensity remained within the prescribed limits (Figure 3).

2.5.2. QVM (Q-Value Method) [21]

In previous studies, a comprehensive index, Q, was formulated to assess the spatial structure of trees. This index integrated multiple spatial structural indicators using multiplication and division methods, with larger-is-better metrics as the numerator and smaller-is-better metrics as the denominator. These indicators, such as the mingling index (M), canopy competition index ($CI$), angle index (W), storey index (S), and open comparison ($OP$), collectively capture diverse aspects of tree spatial structure. Higher values of M, $OP$, and S, coupled with lower values of $CI$, typically indicate a superior overall spatial structure. The W tends to approximate standard random distribution values.

To ensure equal significance and influence of each structural index in the comprehensive evaluation, a normalization method with identical weights was employed. This approach effectively synthesizes various indices, ensuring their equitable contributions and mitigating the impact of different scales.

The final formula is represented as:

$$Q_i=\frac{\frac{1+S_i}{{\delta }_S}·\frac{1+O{P}_i}{{\delta }_{OP}}·\frac{1+M_i}{{\delta }_M}}{\frac{1+C{I}_i}{{\delta }_{CI}}·\frac{1+W_i}{{\delta }_W}}$$

(13)

In Equation (13), $W_i$ represents the angle index, $M_i$ denotes the mingling index, $S_i$ signifies the storey index, $O{P}_i$ characterizes the open comparison, and $C{I}_i$ quantifies the canopy competition index. The terms ${\delta }_W$, ${\delta }_M$, ${\delta }_S$, ${\delta }_{OP}$, and ${\delta }_{CI}$ correspond to the standard deviations of the respective indices. This formula provides a comprehensive evaluation for each tree, ranked in ascending order. Trees with higher rankings exhibit poorer spatial structure and are consequently more suitable for harvesting. Within a specific harvesting limit, the top n ranked trees form the set designated for felling.

2.5.3. VMM (V-Map Method) [22]

In random forests, the range of [0.475, 0.517] is identified as the optimal angle scale for an ideal stand [63], commonly centered around a midpoint value of 0.496. This range represents an optimal balance for the stand’s spatial structure. The choice of this interval is based on previous research or empirical evidence indicating that stands within this range tend to exhibit more favorable spatial characteristics. Additionally, the proximity of a stand’s horizontal pattern to a random distribution is determined by assessing the angle scale value and its deviation from the established midpoint of 0.496 ($|\overline{W}-0.496|$). This method offers insights into deviations from the expected random distribution, aiding in the assessment of a stand’s spatial arrangement relative to an ideal pattern. During the initial selection of trees for cutting, priority is given to the nearest neighbor of the reference tree with the highest $|\overline{W}-0.496|$ value. The closest neighbor of the reference tree is initially chosen for cutting, while stands with trees exhibiting weak, moderate, or suppressed competition are identified as potential candidates for retention. If necessary, further calculations are performed using the critical distance formula proposed by Johann (1982) [64]:

$$G_{iz}=\frac{h_z}{A}·\frac{d_i}{d_z}$$

(14)

where $G_{iz}$ represents the critical distance between the reference tree and the competing tree, $h_z$ is the tree height of the reference tree z, $d_i$ signifies the DBH of competing tree i, $d_z$ stands for the DBH of reference tree z, and A is the parameter defining the cutting weights, which is commonly assumed to be $A=5$ (Figure 4).

2.6. Optimization Model for Selection Cutting of Forest Stand Structure

2.6.1. Constraints

(1) Spatial Structure Constraints

Post-optimization and -adjustments, the values of each sub-objective should not degrade compared to the pre-optimization values. This ensures the conservation and improvement of the spatial diversity of the forest stand. Precisely, it guarantees the convergence of the stand’s horizontal distribution pattern towards a random distribution, enhancement in the level of mixing, reduction in competition within the stand, augmentation of vertical richness, and intensification of the stand’s openness and light penetration. Mathematically, it is expressed as:

$$\left\{\begin{array}{c}\left|\overline{W}-0.496\right|\le \left|\overline{W_0}-0.496\right|\\ \overline{M}\ge \overline{M_0}\\ \overline{CI}\le \overline{C{I}_0}\\ \overline{S}\ge \overline{S_0}\\ \overline{OP}\ge \overline{O{P}_0}\end{array}\right.$$

(15)

Here, $\overline{W}$ represents the post-selection cutting uniform angle index of the forest stand, $\overline{M}$ denotes the mingling index of the forest stand after cutting, $\overline{CI}$ signifies the canopy competition index of the forest stand after cutting, $\overline{S}$ reflects the storey index post-cutting, and $\overline{OP}$ captures the open comparison post-cutting. $\overline{W_0}$, $\overline{M_0}$, $\overline{C{I}_0}$, $\overline{S_0}$, and $\overline{O{P}_0}$ correspond to the values of the aforementioned indices before selection cutting.

(2) Non-spatial Structure Constraints

The selection cutting process must also adhere to non-spatial structure constraints. These constraints ensure the preservation of the number of diameter classes and tree species within the stand, maintain the cutting intensity under 35%, and uphold a canopy density greater than 0.7. They are formally represented as:

$$\left\{\begin{array}{c}D=D_0\\ T=T_0\\ N\ge N_0\left(1-35\%\right)\\ CD\ge 0.7\end{array}\right.$$

(16)

where $D_0$ signifies the number of diameter classes of trees within the stand before selection cutting, D represents the number of diameter classes after cutting; $T_0$ denotes the count of tree species before cutting, T reflects the count of tree species after cutting; $N_0$ stands for the total tree count within the stand before cutting, N corresponds to the total tree count after cutting; and $CD$ quantifies the canopy density post-cutting.

2.6.2. Construction of the Model

Addressing a MOP (multi-objective optimization problem) involves the quest for an optimal solution that balances multiple objectives within a constrained framework. Often, these objectives are interconnected and bound by constraints, making the pursuit of an ideal solution challenging. Hence, it becomes imperative to amalgamate and synthesize these sub-objectives, shaping an overall OF (objective function), and subsequently discovering the optimal solution within this OF. In the context of MOFSS (multi-objective optimization of forest stand structure), a larger VOF (value of the objective function) corresponds to a more favorable spatial structure of the corresponding forest stand.

In this study, we carefully selected five pivotal spatial structure indices: W (uniform angle index), M (mingling index), $CI$ (crown competition index), S (storey index), and $OP$ (open comparison). Employing a ‘multiplication and division’ strategy, we orchestrated the multi-objective planning and synchronization of these spatial structure indices. This orchestration culminated in the formulation of the OF for the multi-objective optimization model of FSS:  

$$maxOF=\frac{1}{N}\sum _{i=1}^N\frac{\frac{1+M_i}{{\delta }_M}·\frac{1+O{P}_i}{{\delta }_{OP}}·\frac{1+S_i}{{\delta }_S}}{\frac{1+C{I}_i}{{\delta }_{CI}}·\frac{1+\left|W_i-0.496\right|}{{\delta }_{\left|W-0.496\right|}}}$$

(17)

Here, $W_i$, $M_i$, $S_i$, $O{P}_i$, and $C{I}_i$ represent the uniform angle index, mingling index, storey index, open comparison, and canopy competition of the central tree i, respectively. Additionally, ${\delta }_W$, ${\delta }_M$, ${\delta }_S$, ${\delta }_{OP}$, and ${\delta }_{CI}$ denote the standard deviations of their respective structural parameters. The midpoint of the range [0.475, 0.517] is 0.496, where a smaller value of $\left|W_i-0.496\right|$ indicates a closer alignment of the forest stand’s horizontal distribution pattern with randomness.

2.7. Algorithms for Solving the Model

2.7.1. MC (Monte Carlo)

MC, a classical approach for addressing MOPs, has been extensively applied to tackle MOFSS problems [20]. Its fundamental principle involves calculating the VOF and related indices for the retained trees after each simulated logging iteration. By iteratively filtering out trees that do not meet the constraints, the MC algorithm generates an approximation of optimal solutions for the MOFSS problem.

The general procedure of the MC algorithm for the MOFSS is outlined as follows: (1) Set the initial control parameters and initialize $I=0$ (count of successive non-feasible solutions) and $U=0$ (count of non-improvement of feasible solutions). (2) Formulate the initial set of retained trees as $g*$ and calculate its corresponding VOF value. (3) Determine the proposed trees for felling using the method for selecting candidate cutting trees, resulting in the retained tree set g. (4) Verify whether the set g satisfies all constraints. If it does, calculate the VOF for set g; otherwise, increase I and continue if I is within the limit $I_{\max }$. (5) If VOF > VOF*, assign the set g and its VOF to $g*$ and VOF*, respectively, and reset $U=0$. Then, re-select felling trees and repeat the process; otherwise, increment U. (6) If $U>U_{\max }$, the algorithm has converged to optimal solutions g and the corresponding OF.

The MC flow chart for solving MOFSS is presented in Figure 5.

2.7.2. PSO (Particle Swarm Optimization)

The integration of PSO, renowned for its spatial search efficiency, with multi-objective problems has gained traction in addressing the multi-objective optimization model of FSS [54,65]. The fundamental concept involves representing each tree in the forest as a particle within the solution space of the PSO algorithm. The two-dimensional spatial layout of the entire forest corresponds to the objective solution space, and the positions of particles in this solution space mimic the spatial coordinates of trees. The VOF for stand structure optimization acts as the fitness value of particles, transforming the MOFSS problem into an iterative optimization process within the solution space of PSO.

The key steps encompass setting up the parameters of the PSO algorithm, mapping the Voronoi space of the forest as the objective solution space, and associating the spatial coordinates of trees with potential particle positions. Particles collectively explore the Voronoi space of the forest to identify trees that fulfill the optimization objectives. During each iteration, particles search for a Voronoi polygon that satisfies the objectives. The position of the best-performing particle (corresponding to a tree) updates the current position of the particle, leading it to ‘land’ on a tree that aligns with improved FSS. The flowchart outlining the PSO procedure for MOFSS is illustrated in Figure 6.

2.7.3. RL (Reinforcement Learning)

Acknowledging the inherent accuracy limitations of MC and the susceptibility of PSO to local optimization, RL presents an alternative with distinct advantages. RL is characterized by its trial-and-error approach, absence of a predefined model, dynamic decision making, and adaptive capabilities. Introducing RL into the realm of MOFSS is a novel endeavor that transforms the process of tree selection into actions guided by an intelligent agent. This agent makes decisions about which trees to fell based on a system of rewards and penalties, continually refining its decision-making strategies, approximating the optimal approach, and ultimately identifying relatively optimal solutions.

In the context of RL applied to MOFSS, tree selection is transformed into intelligent agent actions. The agent’s actions involve deciding whether to cut or not cut specific trees from the set designated for cutting. The environment responds by providing rewards or penalties to the agent based on a predefined reward function. This feedback influences the agent’s subsequent actions. Through the process of maximizing rewards, the agent aims to identify the actions that yield the highest cumulative rewards, thereby contributing to the enhancement of the forest stand structure. The flowchart depicting the RL approach to solving MOFSS is illustrated in Figure 7. The pseudocode for applying RL to MOFSS is detailed in Appendix A.

2.7.4. Optimization Schemes

As shown in

Table 2. Nine optimization schemes for FSS.

	RSM	QVM	VMM
MC	r-MC(M1)	Q-MC(M2)	V-MC(M3)
PSO	r-PSO(M4)	Q-PSO(M5)	V-PSO(M6)
RL	r-RL(M7)	Q-RL(M8)	V-RL(M9)

Note: The abbreviations correspond to the following methods and algorithms: “r” denotes random selection method (RSM), “Q” represents Q-value method (QVM), “V” signifies V-map method (VMM), “MC” stands for Monte Carlo algorithm, “PSO” refers to particle swarm optimization algorithm, and “RL” indicates reinforcement learning algorithm. These three methodologies for devising the proposed selection felling approach are paired with three distinct algorithms for solving the model, resulting in the generation of a total of nine schemes for MOFSSs.

, to evaluate the effectiveness of these algorithms in practical MOFSS applications and to investigate the impact of tree count on model performance, four real forest stands characterized by varying densities and conditions were selected for simulation experiments. The choice of methods for selecting proposed cutting trees leads to different sets of selected trees, consequently influencing the overall performance of the models. In this study, spatial structure units based on the V diagram were utilized, and the tree selection methods included RSM, QVM, and VMM. The optimization of the model was carried out using MC, PSO, and RL, resulting in a total of nine distinct stand structure optimization strategies. A uniform maximum loop iteration of 10,000 was set for all of these strategies.

3. Results

3.1. Algorithm Parameter Settings

The parameter configurations for each solution algorithm in the experimental setup are outlined in

Table 3. Optimizing algorithm parameter configuration.

Algorithms	Parameters and Parameter Values	Value Meaning
MC	$I=0$	Initial iteration
	$I_{max}$ = 10,000	Maximum number of iterations
	$U=0$	Initial number of consecutive iterations without OF improvement
	$U_{max}=500$	Maximum number of consecutive iterations without OF improvement
PSO	$I=0$	Initial iteration
	$I_{max}$ = 10,000	Maximum number of iterations
	$p=20$	Number of particles
	$w=0.5$	Inertia weights
	$c1=0.5$	Individual particle learning factor
	$c2=0.5$	Global learning factor
RL	$I=0$	Initial iteration
	$I_{max}$ = 10,000	Maximum number of iterations
	$state=0$	Initial position of the agent
	$stat{e}_{max}=100$	Farthest position the agent is allowed to move to
	$a=150$, $b=10$, $c=-1$, $d=1$	Reward and penalty values

.

For all algorithms, the initial number of iterations (I) is set to 0, while the maximum number of iterations ($I_{max}$) is set to 10,000, a conventional value commonly used in optimization studies. The variable U represents the initial number of consecutive iterations in which the objective function does not exhibit improvement for the MC algorithm, initialized as $U=0$, and $U_{max}$ is set to 500. In the context of MC, assuming that the nth feasible solution has a value of the objective function VOF(n), if none of the 500 consecutive feasible solutions found after the nth solution demonstrate superior quality compared to VOF(n), then VOF(n) is regarded as an optimal solution.

Parameters such as p, w, $c1$, and $c2$ are common across the PSO algorithm. Specifically, p denotes the number of particles, w stands for the inertia weight, $c1$ represents the individual particle learning factor, and $c2$ signifies the global learning factor. The values of these parameters adhere to conventional standards. To implement RL for MOFSS, this study abstracts the problem into a bidirectional movement scenario for the RL agent. At the initiation of each iteration, the agent’s position is at the ‘starting point’ ($state=0$). Should the agent choose to execute selective tree cutting to enhance FSS, it advances forward ($state=state+1$). On the other hand, the agent might remain stationary ($state=state+1$) or take a step backward ($state=state-1$) if it decides not to cut trees (refer to Figure 7 or the pseudo-code for precise agent movement rules). The ‘end point’ corresponds to a distance of 100 units from the ‘starting point’ ($stat{e}_{max}=100$). The parameters a, b, c, and d are meticulously adjusted reward and penalty values acquired through numerous trial runs, where rewards hold positive values and penalties carry negative values.

3.2. Results of Simulated Selective Cutting Optimization

The findings demonstrate that following the implementation of selective cutting adjustments across different scenarios, the counts of tree diameter classes and tree species within each sample site remained consistent. The spatial structure quality indices for each forest stand exhibited varying degrees of improvement. The mean angle index ($\left|W_i-0.496\right|$), which represents the horizontal distribution pattern, experienced a slight reduction, bringing it closer to a random distribution. The canopy competition index decreased, indicating a reduction in inter-tree competition. The mingling index showed improvement, with the most notable enhancement observed in sample site P4. The storey index and open comparison values increased, thereby enhancing vertical structural diversity and light penetration. Overall, VOF scores were enhanced, indicating the successful optimization of forest stand structure across the different sites (see Figure 8).

3.3. Algorithm Performance

The optimization strategies were simulated across the four sample plots.

The selection of VOF as the evaluation metric for MOFSSs is founded on its comprehensive nature, alignment with prior research [66,67], and interpretive capacity. VOF enables a comprehensive evaluation of forest stand spatial structures and serves as a reliable metric commonly used in related studies, ensuring comparability. The exclusive use of VOF as the sole metric streamlines the problem, facilitates interpretable outcomes, and aligns with the optimization model employed in this study, effectively addressing challenges associated with model complexity and compatibility.

We selected VOF as the primary evaluation metric for MOFSSs based on its extensive application in the research domain, comprehensiveness, and interpretative capabilities. Prior studies [66,67] have demonstrated the effectiveness of VOF in assessing forest structure by considering the spatial arrangement comprehensively, ensuring comparability across related studies. In this research, solely using VOF as the exclusive evaluation metric assists in simplifying the problem, generating results that are easier to interpret, and aligning with our adopted optimization model, effectively addressing challenges related to model complexity and compatibility. While other metrics also possess certain advantages, within the current research context, VOF, as a comprehensive and reliable metric, best assesses the multi-objective optimization effects on forest structure we are investigating. Moreover, despite the existence of other parameters, such as object value [20], comprehensive homogeneity index [66], and fitness value [67], utilized in assessing the optimization effects of forest structure, their design principles and computation methods closely resemble those of VOF. Most parameters have adopted the principles of multiplication and division, integrating multiple forest structure evaluation indices logically.

Figure 9 illustrates the most favorable optimization outcomes achieved with each scheme. On the horizontal axis, various forest stand structure optimization models are represented, each applied to the four distinct sample plots. The vertical axis showcases the optimal objective function values corresponding to the FSS optimization model of each plot. In the context of MOFSS across the four sample plots, both MC-type and PSO-type optimization strategies encountered suboptimal solutions on two occasions. Specifically, MC-type schemes produced suboptimal outcomes for sample plots P2 and P3 using M1 (r-MC), while PSO-type schemes yielded suboptimal solutions for sample plots P1 and P4 via M6 (V-PSO) and M5 (Q-PSO), respectively. The disparity in suboptimal solutions between these two categories of schemes is relatively minor, indicating the effective performance of both MC-type and PSO-type approaches in addressing MOFSS challenges. Remarkably, M7 (r-RL) consistently achieved the relative maximum VOF within the 35% selection threshold for each individual sample plot.

To evaluate the optimization performance of each scheme on the secondary forest stand structure of Pinus yunnanensis, this study compares their effectiveness in terms of algorithm convergence speed and the extent of stand structure optimization. The advantages and drawbacks of the stand structure are gauged through the value of the objective function. To ensure that sample site conditions do not bias the comparison results, the optimization schemes were simulated for logging selection in the four sample plots, and the outcomes are presented in

Table 4. The variation in VOFs before and after simulated cutting.

VOF		P1	P2	P3	P4	Average Value of Scheme	Improvement Magnitude of Scheme	Average Improvement Magnitude of Scheme
Before Cutting		0.3515	0.2814	0.3748	0.4812	0.3722
After Cutting	M1	0.3892	0.3983	0.4887	0.5537	0.4575	22.92%	17.42%
	M2	0.3858	0.3420	0.4588	0.5180	0.4261	14.48%
	M3	0.3676	0.3066	0.4144	0.5109	0.3999	7.44%
	M4	0.3975	0.3705	0.4560	0.5292	0.4383	17.76%
	M5	0.3933	0.3773	0.4729	0.5593	0.4507	21.09%
	M6	0.4009	0.3877	0.4675	0.5373	0.4484	20.47%
	M7	0.4121	0.4078	0.5047	0.5635	0.4720	26.81%
	M8	0.3916	0.3866	0.4610	0.5186	0.4395	18.08%
	M9	0.3663	0.3076	0.4144	0.5149	0.4008	7.684%
Average Value of Sample Plot		0.3894	0.3649	0.4598	0.5340

Note: The ‘Average Value of Sample Plot’ represents the average maximum objective function value achieved by the 9 optimization schemes within a specific sample plot. The ’Average Value of Scheme’ represents the mean of the maximum VOFs obtained by a particular optimization scheme across the simulated cutting experiments in the four sample plots. The ‘Improvement Magnitude of Scheme’ signifies the percentage increase in the mean of the maximum VOFs relative to the mean initial VOFs before cutting in the four sample plots for a specific optimization scheme. The ’Average Improvement Magnitude of Scheme’ denotes the percentage increase in the mean of the maximum VOFs achieved by the nine optimization schemes in all sample plots relative to the mean initial VOFs before cutting in the four sample plots.

. Similarly, the convergence progress of each optimization scheme in addressing the stand structure optimization challenge within the same plots is depicted in Figure 10.

As indicated in Table 4, the objective function value of the stand structure for sample plot P1 increased from 0.3515 to 0.3894, P2 increased from 0.2814 to 0.3649, P3 increased from 0.3748 to 0.4598, and P4 increased from 0.4812 to 0.5340. In the simulated logging selection experiments across these four sample plots, the highest objective function value achieved by the reinforcement learning scheme was 0.5983, and the optimal objective function value reached by the Monte Carlo scheme was 0.4887. The values obtained using the reinforcement learning algorithm (0.4121, 0.4078, 0.5047, 0.5635, respectively) clearly exceeded those of the MC (0.3892, 0.3983, 0.4887, 0.5537, respectively) and PSO (0.4010, 0.3877, 0.4729, 0.5593, respectively) schemes. Observing Figure 10, the experiments logged the iterations conducted during model runs, offering partial insight into the model’s runtime. The results reveal varying convergence times as the number of iterations increases. The corresponding runtime table (

Table 5. Running time taken to attain Mmax VOF across different sites for each strategy.

Running Time/(min)	M1	M2	M3	M4	M5	M6	M7	M8	M9
P1	7847.881	4997.854	9793.233	4198.420	3404.310	3401.141	3398.441	7394.313	3400.292
P2	4845.591	3450.688	2635.718	5053.131	4240.855	5058.874	7229.825	4242.174	2641.545
P3	7892.756	2645.365	6600.294	7424.275	5039.715	5039.071	7372.265	8207.307	5007.491
P4	12,947.977	3475.821	6615.058	6632.085	3459.199	5033.976	5035.914	4242.309	6626.676

Note: This table illustrates the time required for each optimization strategy to achieve the maximum VOF across four different sites. Each row represents a specific strategy, and each column denotes a different site. The values in the table represent the time (in minutes) taken by each strategy to reach the maximum VOF in the respective site. These findings serve as a reference for the time efficiency of each method in achieving optimization. To mitigate the complexity arising from an extended manuscript, we have relocated the detailed depiction of line graphs illustrating the runtime and VOFs of various optimization models across different simulated FSSs to Figure A2. This graph further emphasizes the advantages of RL in multi-objective optimization of FSSs.

) provides a comprehensive overview of the time required for each strategy to attain the maximum objective function values across different plots. The table demonstrates the practical runtime implications of each strategy, complementing the insights obtained from the iteration logs. Upon comprehensive analysis of iterations and runtime, it becomes evident that the RL algorithm demonstrates slightly faster convergence compared to the MC and PSO algorithms when achieving similar objective function values. Furthermore, it is evident that the MC algorithm converges more slowly in cases like M3 in sample plots P1 and P3 and M1 in sample plot P4. The PSO algorithm tends to encounter issues with local optimization, such as M6 in sample plots P1, P3, and P4.

3.4. Influence of Proposed Selective Felling Tree Determination Methods

As depicted in Figure A1, within the MC-class optimization schemes (M1, M2, M3), the optimization scheme utilizing random selection as the method for determining proposed logging (M1) significantly outperformed the optimization schemes employing the QVM or VMM (M2, M3) in enhancing the stand structure of the sampled areas.

In the PSO-class optimization schemes (M4, M5, M6), the optimal results achieved by the schemes utilizing the QVM or VMM for selecting trees to be felled (M5, M6) were generally comparable in optimizing the stand structure across each sample site, and slightly superior to the optimization scheme combining RSM and PSO (M4). Similar to the MC-class scheme, the scheme that combined the RL with the RSM for selecting trees to be felled (M7) demonstrated the best performance across the four sample sites (M7 > M8 > M9).

It is also evident from Table 4 that the average improvement in the objective function value of M7 across the four sample plots amounts to 26.81%, which is significantly higher than the average improvement of all the schemes (17.42%).

4. Discussion

In the domain of multi-objective optimization for forest stand structures (MOFSSs), the quest for effective solutions remains paramount. Although machine learning techniques like random forests and decision trees find applications in forest management, their research scope regarding dynamic multi-objective forest structure optimization remains limited. While deep learning excels in handling extensive data and complex pattern recognition, its practical application in forest management faces constraints related to environmental interaction and reward function design.

While certain machine learning methods such as random forests and decision trees have been applied in forest management scenarios, their exploration specifically concerning dynamic multi-objective forest structure optimization remains restricted in the literature. Our extensive review did not reveal substantial investigations employing these machine learning methods to address the intricate challenges of multi-objective forest stand structure optimization. This highlights a notable research gap in utilizing machine learning techniques for effectively handling complex multi-objective optimization issues within forest structures. Similarly, deep learning’s proficiency in managing extensive datasets and intricate pattern recognition has not translated effectively into practical applications within forest management due to inherent challenges in environmental interaction and reward function design.

As of now, a universally acknowledged, efficient approach to address MOFSSs remains elusive. To overcome the constraints of the current algorithms and leverage the inherent advantages of RL in multi-objective optimization, this study introduces a pioneering approach rooted in RL principles to address the complexities of optimizing multi-objective FSSs. Simulated felling experiments were conducted using data obtained from circular sample plots in the southwestern region of China within secondary forests of Pinus yunnanensis.

The findings underscore the promising potential of RL in optimizing FSSs, indicating its capability to mitigate prevailing challenges in widespread application. Particularly pertinent to the management and optimization of Pinus yunnanensis secondary forests in southwest China, this study offers crucial guidance and direction. The outcomes of this study are not confined to a specific region or tree species; their universality offers valuable insights for addressing similar forest ecosystem management issues in other regions or with different tree species.

4.1. Results of Simulated Selective Cutting Optimization

In terms of FSS, simulated selective cutting led to improved horizontal distribution patterns, enhanced the spatial segregation of tree species, reduced the competitive pressure among trees, enriched vertical tree structures, and improved understory light conditions [66,68,69]. While maintaining the number of tree diameter classes and species and adhering to cutting intensity and canopy density constraints, the post-cutting forest stand structure surpassed the pre-cutting condition [70,71,72]. This underscores the feasibility of applying RL to solve MOFSS and the potential of using RL to enhance FSS. The results also demonstrate the efficacy of both MC and PSO in addressing MOFSS, aligning with findings from previous research [20,54,65,73,74].

Due to the limited availability of current data, this study has relied on existing research findings and the insights of previous researchers [61,62]. In the simulated selection felling experiment with Pinus yunnanensis conducted in four sample plots, the selection intensity was uniformly set not to exceed 35% of the total tree count within each plot. However, it is important to note that varying site conditions and forest stand structures may lead to different optimal selection limits across different plots [67,75].

In the subsequent phases of our research, we intend to address this gap by conducting additional experiments to gather more relevant data. Our goal is to refine and tailor the selection intensity limits for each individual sample plot, taking into consideration the specific site characteristics and forest composition.

4.2. Algorithm Performance

Based on the results obtained from the simulated selective cutting experiments conducted in the four sample plots, the RL optimization schemes consistently yielded higher maximum VOFs (0.4121, 0.4078, 0.5047, 0.5635, respectively) compared to the MC class (0.3892, 0.3983, 0.4887, 0.5537, respectively) and the PSO class (0.4010, 0.3877, 0.4729, 0.5593). Notably, under similar VOFs, the RL algorithm exhibited slightly faster convergence compared to the MC and PSO algorithms. The optimization results for sample plots P1, P3, and M6 in P4 demonstrate that the MC scheme has a slower convergence speed, while the results for sample plots P1, P3, and P4 indicate that the PSO algorithm is prone to becoming stuck in ‘local optima’. In summary, the RL algorithm proves to be competitive with traditional approaches like the MC and PSO algorithms for solving the MOFSS.

In the simulated harvesting experiments, the RL optimization scheme did not exhibit significant advantages in terms of time complexity and space complexity [76,77,78] when compared with the MC and PSO classes. Our future research will focus on enhancing the efficiency of utilizing RL to solve the MOFSS. We aim to bridge the gap between RL and existing optimization methods for FSS in terms of time and space complexity.

4.3. Influence of Proposed Selective Felling Tree Determination Methods

To explore the impact of the proposed selection methods on experimental outcomes, this study simulates the selection of trees from the set of selected trees generated by the three proposed methods during the model-solving process. The experimental results indicate that optimization schemes combining the RSM with either the MC or RL algorithms (M1, M7) achieve superior results compared to schemes combining the QVM or VMM with the MC or RL algorithms (M1 > M2 > M3, M7 > M8 > M9). Moreover, the fitness level of schemes utilizing the QVM or VMM in combination with the PSO algorithm is higher (M5 > M4, M6 > M4).

Particularly, the M7 optimization scheme, which combines the RSM with the RL algorithm, exhibits an average enhancement of 26.81% in VOFs across the four sample plots. This improvement is significantly higher than the average enhancement observed in all schemes (17.42%). These results suggest that the RSM is highly adaptable and effective in conjunction with RL.

These findings may be attributed to the number of proposed felling trees generated by each method and the characteristics of the algorithms for solving the MOP. RSM generates the most proposed felling trees, while QVM and VMM yield relatively fewer. MC involves randomness in its estimations [79,80,81], while RL continuously improves effectiveness through experimentation and learning [42,82]. Larger sets of proposed felling trees provide more samples for training and enhancing experimental results in MC and RL [38,81,83,84]. PSO is more suitable for optimizing a relatively small number of trees chosen based on QVM or VMM, helping avoid local optimization [85,86].

The variation in proposed felling trees generated by each method and the characteristics of the algorithms for solving the MOP likely contribute to these findings. Relative to the others, the RSM generates a higher number of proposed felling trees, while the QVM and VMM produce comparatively fewer selections. It is worth noting that MC involves a stochastic element in its estimations, introducing randomness [80,81,87], whereas RL continually refines its performance through iterative experimentation and learning [38,42,88].

Larger sets of proposed felling trees inherently offer more training samples, thus potentially enhancing the experimental outcomes in both MC and RL [38,81,83]. On the other hand, PSO is better suited for optimizing a relatively modest number of trees chosen using QVM or VMM, which aids in avoiding issues related to local optimization [89,90]. The relatively favorable experimental outcomes (M5 > M4, M6 > M4) of the schemes that combine the PSO algorithm with the QVM or VMM provide indirect confirmation of this perspective. However, reducing the sample size might result in the omission of potential global optimal solutions and weaken the global search capability of the PSO algorithm [91]. Consequently, besides the quest for a more suitable method for determining proposed tree cutting within PSO, it is of paramount importance for us to comprehensively consider the characteristics inherent to the challenges of multi-objective optimization in the context of forest stand structural optimization. Further investigation is warranted to explore strategies for mitigating the issue of PSO becoming trapped in local optima when addressing such challenges.

Based on the optimal reinforcement learning scheme, we have also developed a harvested tree information table (

Table 6. Partial selective cutting information.

Sample Site	Optimal Scheme	Tree Number	DBH (cm)	TH (m)	CW (m)	Tree Specie
P1	M7	76	11.90	10.40	1.43	PY
	M7	234	7.80	7.20	1.78	PA
	M7	335	11.70	15.00	1.35	PY
	M7	519	14.80	12.40	1.85	PY
	M7	638	9.70	10.52	1.15	PY
P2	M7	66	15.50	7.80	1.95	PY
	M7	224	10.50	7.50	1.43	PY
	M7	319	8.50	5.70	2.08	PA
	M7	505	5.00	4.10	1.10	PY
	M7	670	19.40	12.75	2.25	PY
P3	M7	61	22.10	15.85	1.75	PY
	M7	143	16.00	13.27	1.45	PY
	M7	160	7.30	6.40	1.26	PY
	M7	275	9.40	8.67	1.24	PY
	M7	299	13.20	7.36	2.10	PY
P4	M7	19	8.32	7.67	0.93	PY
	M7	68	7.70	6.27	1.19	PY
	M7	145	22.80	11.20	1.93	PY
	M7	194	5.62	6.50	1.58	PY
	M7	247	16.15	11.56	2.00	PY

Note: PY, Pinus yunnanensis; PA, Pinus armandii. This table presents partial selective cutting information output from different selective cutting optimization schemes.

) that provides strong support for real-world forest stand management decisions [54,65,92]. This table integrates scientific algorithms while addressing the practical needs of multi-objective optimization for forest stand structure. By implementing these optimization results in practical scenarios, we effectively achieve sustainable forest stand management, highlighting the practical application value of this study in the field of forest resource management.

Compared to some existing methods, this approach demonstrates higher solution efficiency; however, there remain several areas for continuous improvement:

(1) This study exclusively employs the fundamental Q-learning algorithm in RL. Exploring and validating more advanced algorithms for solving the MOP of FSS that offer improved solution efficiency is a promising avenue for further research.

(2) Due to the constraints posed by sample site conditions, the current study’s sample size for each site primarily ranges between 200 and 500. In-depth investigation and research are needed to address the efficiency of solving the MOP of FSS using the RL algorithm with larger and even super-large sample sizes.

(3) Achieving an optimized FSS is not a one-step process; it requires multiple adjustments to converge to an ideal state. The optimization objective is dynamic and adaptable. During each adjustment, the direction and goal of optimization should be precisely defined based on the forest stand’s specific circumstances and operational objectives.

(4) Replanting is a relative strategy to selective cutting in the FSS optimization process. Sustainable forest management necessitates the integration of selective logging and replanting principles. Investigating an optimization model that incorporates replanting in FSS optimization using the RL algorithm is an important topic for future research.

5. Conclusions

The introduction of reinforcement learning (RL) to address multi-objective optimization for forest stand structure (MOFSS) is a pioneering step, marking the first application of RL in forest stand structure (FSS) optimization research. By translating the tree selection process in FSS into RL-based intelligent agent actions, this research has successfully revolutionized the approach to solving FSS multi-objective problems. This pioneering use of RL showcases its potential to transform conventional methodologies in FSS optimization practices.

Extensive experimental comparisons unequivocally confirm RL’s exceptional effectiveness in MOFSS. The RL scheme achieves significantly higher values of the objective function (VOFs) across various plots compared to the Monte Carlo (MC) and particle swarm optimization (PSO) methods, quantifying RL’s excellence and establishing its practical relevance in forest management.

Three RL schemes (M7, M8, and M9) based on different tree selection methods have been introduced and rigorously analyzed. Among these, the M7 scheme, combining the random selection method (RSM) with RL, emerges as a superior performer across the four sample plots. This comprehensive analysis provides valuable insights into practical MOFSS solutions, emphasizing RL’s compatibility with RSM.

The empirical validation of the optimal RL-based scheme for generating felling tables strongly reinforces its support for real-world stand structure optimization decisions. This emphasizes the practical significance of integrating RL into FSS strategies. The results of this study transcend specific regional or tree species boundaries; their universality provides invaluable insights for tackling analogous forest ecosystem management challenges in diverse regions or with varying tree species.