A Context-Conditioned Reinforcement Learning Framework for Space Frame Structure Optimization

Abstract

With the rapid advancement of artificial intelligence in recent years, its adoption in structural design has been increasing. However, supervised learning in engineering design is often limited by the availability and quality of labeled data. To address this issue, this study proposes a context-conditioned deep reinforcement learning (DRL) method for the automated optimization of space frame structures, termed the Space Frame Optimization Agent (SFO-Agent). Instead of learning a policy for a single fixed design task, the proposed agent is conditioned on task-related context variables and trained through direct interaction with a finite element analysis environment, thereby avoiding reliance on pre-collected training datasets. In addition, a piecewise reward function is further formulated using code-prescribed limit values to achieve both structural safety and economic efficiency. Numerical experiments demonstrate that the proposed agent effectively captures reusable design policies under varying task contexts and produces high-quality, code-compliant designs across the full parameter domain. Comparisons with conventional optimization algorithms further demonstrate its superior efficiency and competitive optimization performance, indicating strong potential for broad engineering applications.

1. Introduction

Space frame structures have been widely adopted in large public buildings due to their favorable mechanical performance and high design flexibility [1]. By connecting numerous members through joints arranged according to prescribed geometric rules, these systems form spatially distributed load-bearing structures [2]. In conventional practice, the design and optimization process of space frame structures typically comprises three major stages: shape optimization, mesh generation, and structural analysis [3,4]. However, these stages are often treated as relatively decoupled tasks, and an integrated optimization framework that coordinates them in a unified manner remains limited. Consequently, practical optimization still requires substantial manual effort and is time-consuming. To reduce dependence on engineers’ experience, a variety of optimization algorithms have been employed for automated design of space frame structures. Erbatur et al. [5] adopted the genetic algorithm (GA) to optimize frames and truss structures, while Delyová et al. [6] applied GA to optimize member sizing and topological relationships of truss systems. GA-based methods have also been specifically applied to the design optimization of gridshell structures [7,8]. In addition, simulated annealing and particle swarm optimization have also been explored for space-structure optimization [9,10,11,12]. In addition to optimization-oriented studies, related model-based computational methods have also been developed for the analysis and damage identification of steel truss structures [13,14]. Although these methods have achieved promising results, some limitations still remain with respect to computational cost and runtime. In addition, they are generally designed for a single instance in a one-shot manner. When the initial conditions change, the optimization process often needs to be carried out again, which may constrain their generalization capability.

In recent years, with the improvement of computational efficiency and the rapid development of machine learning techniques, an increasing number of learning-based methods have been introduced into structural engineering design, enabling efficient and accurate design of structural members [15,16,17] as well as prediction of member performance [18,19]. Machine learning methods have also been applied to the performance prediction and design of entire structural systems. de Lautour and Omenzetter [20] employed an artificial neural network (ANN) to overcome the efficiency limitations of nonlinear finite element analysis and achieved efficient prediction of seismic damage in reinforced concrete frame structures. Asgarkhani et al. [21] used 32 machine learning algorithms with hyperparameter tuning to predict the maximum residual drift ratio based on numerical simulation data from 384 steel frames, achieving an accuracy of 95%. Chang and Cheng [22] and Song et al. [23] proposed graph-based surrogate models for simple frames and elastic planar frames, providing feasible approaches for the application of machine learning to structural design and optimization. Zhao et al. [24] further employed a graph neural network (GNN) to perform the layout design of frame-structure beams. These studies indicate that machine learning methods have become capable of accomplishing certain specific engineering structural design tasks.

Driven by advances in convolutional neural networks (CNNs) [25], computers have gained the capability to recognize engineering drawings, making it feasible to use such drawings directly as training data. Pizarro et al. [26] compiled a database of 165 Chilean residential reinforced-concrete shear-wall building projects and trained a regression neural network to predict wall thickness and length from a 30-feature input vector describing the building. Huang and Zheng [27] trained a Pix2pixHD model on image data to automatically generate apartment floor plans, whereas Zheng et al. [28] trained a generative adversarial network (GAN) to generate architectural plans from input image boundaries. Liao et al. [29] further proposed a GAN-based shear-wall design method that learns from existing shear-wall design documents and enables rapid, intelligent structural design. Methodologically, these approaches are representative of supervised learning, where labeled data are used to learn an input–output mapping for prediction on unseen samples. Nevertheless, applying such approaches to the optimization of space frame structures poses several challenges: (1) the geometric forms of space frame structures are often highly complex, and drawing-based representations alone may be insufficient for accurate description; (2) supervised learning typically requires large volumes of labeled data, whereas datasets in the space frame domain are difficult to obtain and remain limited; (3) design variations across engineering projects are substantial, and predictive models tend to degrade when applied to cases outside the covered data distribution.

To address these limitations, reinforcement learning (RL) [30] offers a viable alternative. Unlike supervised learning, RL does not require pre-labeled datasets; instead, an agent improves its policy through interaction with an environment and the resulting feedback. It is widely applied to sequential decision-making and control tasks, such as robotic control, autonomous driving, and path planning. RL methods are broadly categorized into model-based [31,32] and model-free approaches. Model-free RL learns directly from data generated through interaction with the environment and is commonly categorized into two families: value learning and policy optimization. Value learning estimates a value function to evaluate states (or state–action pairs) and typically derives the policy implicitly via greedy action selection; representative algorithms include the Deep Q-Network [33] and the Dueling Deep Q-Network [34]. Policy optimization, by comparison, optimizes the policy explicitly, most commonly via gradient-based updates to improve expected return; representative algorithms include Trust Region Policy Optimization (TRPO) [35] and Proximal Policy Optimization (PPO) [36]. These two families offer complementary strengths: Value learning is typically more sample-efficient but can be sensitive to estimation error. Policy optimization is often more stable and better suited to continuous control, but more sensitive to data and hyperparameter tuning. To combine the advantages of value learning and policy optimization, the actor–critic approach was developed [37]. It learns a policy network (actor) while simultaneously training a value estimator (critic) to evaluate and guide policy updates, thereby improving overall performance. The Soft Actor–Critic (SAC) method [38], adopted in this study to construct the agent, is a representative actor–critic method. Deep reinforcement learning (DRL) is a class of methods that integrates deep neural networks into reinforcement learning for value function approximation and policy learning.

In structural design, the finite element method (FEM) serves as the environment by evaluating a given model and returning performance indicators. Accordingly, RL has been explored for tasks such as topology optimization of planar truss structures [39,40], design of reinforced-concrete (RC) beams [41], and optimization of simplified steel frames [42]. Du et al. [43] proposed a reinforcement learning environment specifically designed for the optimal design of steel members and structures, providing a platform for agents to learn efficient structural optimization strategies. Notably, most existing studies focus on member-level optimization or rely on simplified surrogate environments, and optimization of full structural models remains scarce. Moreover, conventional RL frameworks are typically developed for a fixed design setting, which limits their ability to generalize across varying design conditions. To address task variations, conditional RL and contextual MDP formulations have been developed, in which task-related variables are incorporated into the decision process so that the learned policy can adapt to different goals, environmental parameters, or task conditions [44,45]. However, the application of such context-conditioned formulations to structural design and optimization remains limited.

Motivated by the above considerations, this study proposes a context-conditioned DRL-based optimization method for space frame structures. A DRL environment is established by coupling parametric modeling with FEM, enabling performance improvement through the closed-loop interaction between the agent and the environment. In particular, the proposed framework introduces structural design conditions as context variables, allowing the agent to learn a policy that is adaptable to multiple design scenarios rather than a single fixed task. The main contributions are as follows:

• Based on the Soft Actor–Critic framework in DRL, the SFO-Agent is developed to interact directly with FEM and achieve automated optimization design of space frame structures through training.
• A DRL environment is constructed by integrating parametric modeling and FEM, providing an efficient state representation and enabling coordinated optimization of structural geometry and component engineering.
• A piecewise reward function is formulated in accordance with relevant design-code provisions for space frame structures, guiding the optimization to satisfy safety requirements while reducing material consumption.
• A context-conditioned off-policy training strategy is adopted, in which varying design conditions are incorporated into the training process, enabling the agent to learn reusable design policies and improve generalization across multiple structural design scenarios.

2. Methodology

The proposed SFO-Agent is developed within a context-conditioned DRL framework. As illustrated in Figure 1, the design conditions of each design case are first represented parametrically and mapped into a context, which is then concatenated with the environment state to form a complete augmented state. Subsequently, the SFO-Agent, consisting of an actor and critics, receives the state as input and outputs an action. The SFO-Environment, which integrates parametric modeling and finite element analysis, then performs automated structural modeling and analysis based on the given state and action, and obtains the corresponding structural performance indicators. Compliance is evaluated according to the code-prescribed limit values, and all resulting information is stored in the replay buffer as transitions. Thereafter, the SFO-Agent draws mini-batches of samples from the replay buffer to update its networks. By continuously iterating through this cycle, the performance of the agent is progressively improved.

2.1. SFO-Environment

In RL, the environment receives actions from the agent, updates its internal configuration accordingly, and returns the newly observed state together with a reward signal. In addition to widely used general-purpose environment toolkits [46], several studies have constructed RL environments for structural engineering design [39,43]. However, these environments have largely relied on surrogate models or simplified structural representations, such as planar frames and simple frame systems. By contrast, RL environments tailored to structural engineering problems that support the analysis of full three-dimensional structural models remain scarce. To enable training of the proposed optimization agent for space frame structures, it is therefore necessary to establish an RL environment that integrates structural modeling, finite element analysis, performance-index extraction, and reward evaluation.

By integrating parametric modeling [47] with FEM, this study develops an RL environment, termed SFO-Environment, that enables automated modeling and analysis of the entire structural system. The environment allows rapid modification of the model through a set of key parameters, while the effects of the agent’s actions are evaluated and returned through finite element analysis. As shown in Figure 2, the environment takes the state and action as inputs, and the combined tensor primarily consists of two components: Geometry data and simulation model data. The geometry data, which encode spatial information such as node coordinates and topology, are used to rapidly generate a geometric model. The simulation model data specify finite-element analysis inputs, including cross-sections, loads, and restraint conditions. By integrating these two data streams and transferring them to FEM software through an API, a complete FEM model can be automatically constructed. A reward function based on code-prescribed limits is then evaluated to compute the current reward, which is returned together with the next state. In the implementation, the parametric modeling module was developed using the Rhino 7.0–Grasshopper (version 1.0.0007) platform, while the finite element analysis was conducted in SAP2000 (version 21.0.2). The interaction between the parametric model and the FEM solver, including model generation, analysis execution, and result extraction, was implemented through the SAP2000 API.

2.2. State and Action

The state represents the environment condition and, for structural design, corresponds to the current set of design parameters, typically encoded as a tensor. RL training is conducted over multiple episodes. Within each episode, the agent explores the environment through a sequence of actions, after which the state is reset to start the next episode. In conventional RL frameworks, optimization is usually formulated for a fixed task, and the state is therefore reset to a fixed tensor at the beginning of each episode. In this study, however, the aim is not to optimize a single structure, but to learn a policy that generalizes across a family of design tasks within the same structural type. Inspired by the concept of contextual Markov decision processes [44], this study introduces a context variable into the state formulation to represent different design cases. As shown in Figure 3, the context is concatenated with the environment state to form an augmented state for training within each episode, while different contexts are randomly sampled from uniform distributions within their admissible ranges across episodes. In this way, the agent can learn optimization patterns under diverse design cases, thereby achieving improved generalization performance.

In structural applications, RL often faces varying state dimensionality due to changes in the number of structural members. To address this issue, the maximum considered structure strategy was proposed [42]. It fixes the state dimension to the maximum possible member layout, and assigns empty entries to positions where members are absent, thereby ensuring a consistent input size. This approach is effective for surrogate-based models with a limited number of members. However, for near-realistic full space frame models, it leads to excessively high-dimensional states, increasing agent complexity and making training difficult to converge. This study uses parametric representation to characterize structures within the same structural type. Specifically, the state of a given structure is represented by a small set of key parameters, rather than being encoded at the member level. As an illustrative example, the Kiewitt Dome can be represented by 9 parameters, whose definitions are summarized in

Table 1. Definitions of the parameters in the state.

No.	Parameter	Definition	Unit
1	D	Spherical shell span diameter	mm
2	$R_{HD}$	Rise-to-span ratio	-
3	DL	Dead load	N/mm2
4	LL	Live load	N/mm2
5	$l_1$	Radial grid size	mm
6	$R_{l1}$	Ratio of circumferential to radial grid size	-
7	$S_1$	Section index of radial members	-
8	$S_2$	Section index of circumferential members	-
9	$S_3$	Section index of diagonal members	-

. A typical state transition driven by these parameters is illustrated in Figure 4. When the action $a_t$ is applied, the environment updates from $s_t$ to $s_{t+1}$. Although the number of structural members changes, the state dimensionality remains invariant. The selected state parameters are the direct input variables of the parametric modeling program rather than post-extracted features from an existing FEM model. For each admissible state vector, the parametric modeling procedure deterministically reconstructs the corresponding structural geometry, member topology, section assignment, and load conditions, which are then transferred to the FEM software for analysis. This state-to-model mapping was verified by reconstructing FEM models from sampled state vectors and confirming that the generated models could be successfully analyzed and return the required performance indicators. However, this representation does not aim to cover the entire real design space of complex spatial structures, especially non-parametric geometries, irregular local configurations, or member-wise independent design variables. In this study, the low-dimensional parametric representation is adopted to avoid variable-length member-level states and to improve training efficiency.

2.3. Reward Design

The reward is the scalar feedback produced after the environment receives an action and evaluates the resulting new state. In this study, reward computation follows the code [48] for global-structure and member-level checks. For the Kiewitt dome case described above, the corresponding indicators and their definitions are summarized in

Table 2. FEM output metrics used in reward function.

No.	Parameter	Definition	Unit
1	M	Material usage	t
2	UZ	Maximum deflection	mm
3	BF	First-order buckling factor	-
4	$R_1$	$\mathrm{Maximum} \mathrm{stress} \mathrm{ratio} \mathrm{of} \mathrm{member} \mathrm{group} S_1$	-
5	$R_2$	$\mathrm{Maximum} \mathrm{stress} \mathrm{ratio} \mathrm{of} \mathrm{member} \mathrm{group} S_2$	-
6	$R_3$	$\mathrm{Maximum} \mathrm{stress} \mathrm{ratio} \mathrm{of} \mathrm{member} \mathrm{group} S_3$	-

. The optimization objective can be stated as minimizing material usage while satisfying all code requirements. Accordingly, the reward terms can be grouped into two categories: (1) criteria subject to strict code limits, such as global deformation as well as member strength and stability checks; (2) quantities without explicit code limits but desirable to reduce, such as the material usage M.

Following the code [48], the displacement of the structure under design loads is normalized by the global-structure limit of 1/400, as given in Equation (1). Also, the first-order buckling factor, which characterizes global stability, is normalized using a limit value of $B{F}_{lim}=7.0$, as shown in Equation (2).

$$U{Z}_{\mathrm{norm}}={\displaystyle \frac{abs(UZ/D)}{1/400}}$$

(1)

$$B{F}_{\mathrm{norm}}=BF/B{F}_{lim}$$

(2)

For code-limited criteria, satisfying the prescribed thresholds is sufficient; improvements beyond the limits should not further increase the reward. A clipping function defined in Equation (3) is applied to cap the reward once the requirements are met.

$$clip(x,a,b)=\left\{\begin{array}{c}a\\ x\\ b\end{array}\right.\begin{array}{c}if \mathrm{x}<\mathrm{a}\\ if \mathrm{a}\le \mathrm{x}\le \mathrm{b}\\ if \mathrm{x}>\mathrm{b}\end{array}$$

(3)

$$\begin{array}{l}{v}_{\mathrm{UZ}}=clip((U{Z}_{\mathrm{norm}}-1.0),0,5)\\ v_{\mathrm{BF}}=clip((1.0-B{F}_{\mathrm{norm}}),0,5)\\ v_{\mathrm{Ri}}=clip((R_i-1.0),0,5)\end{array}$$

(4)

$$v_{\mathrm{sum}}=v_{\mathrm{UZ}}+v_{\mathrm{BF}}+{\displaystyle \sum v_{\mathrm{Ri}}}$$

(5)

Equation (4) retains only the portions that exceed the code limits and quantifies the corresponding exceedance, which is then used to compute penalties. Equation (5) aggregates the exceedances from all criteria. If $v_{\mathrm{sum}}>0$, at least one requirement is violated; otherwise, all criteria satisfy the code limits. With a penalty shaping function defined in Equation (6), which maintains sensitivity to small exceedances and imposes stronger penalties on larger violations, the reward for code-infeasible actions can be expressed as Equation (7).

$$\varphi (v)=v+v^2$$

(6)

$$R=-{\omega }_{\mathrm{UZ}}\varphi (v_{\mathrm{UZ}})-{\omega }_{\mathrm{BF}}\varphi (v_{\mathrm{BF}})-{\displaystyle \sum {\omega }_{\mathrm{Ri}}\varphi (v_{\mathrm{Ri}})}$$

(7)

When all performance criteria satisfy the code limits, the design is evaluated primarily by its material usage, which is also normalized to stabilize training. For the Kiewitt Dome, in addition to steel weight per unit area, effects such as the rise-to-span ratio and loading conditions are incorporated into the normalization, as given in Equation (8).

$$M_{\mathrm{norm}}=R_{HD}\times {\displaystyle \frac{M}{Area\times (DL+LL)}}$$

(8)

Accordingly, the reward under code-compliant conditions is given by Equation (9), while the overall reward is given by Equation (10).

$$R=K-{\omega }_{\mathrm{M}}{M}_{\mathrm{norm}}$$

(9)

$$R=\left\{\begin{array}{ll}-{\omega }_{\mathrm{UZ}}\varphi (v_{\mathrm{UZ}})-{\omega }_{\mathrm{BF}}\varphi (v_{\mathrm{BF}})-{\displaystyle \sum {\omega }_{\mathrm{Ri}}\varphi (v_{\mathrm{Ri}})}& if\hspace{1em}{v}_{\mathrm{sum}}>0\\ K-{\omega }_{\mathrm{M}}{M}_{\mathrm{norm}}& if\hspace{1em}{v}_{\mathrm{sum}}\le 0\end{array}\right.$$

(10)

This piecewise reward design encourages the agent to prioritize code compliance during training, and then to pursue material reduction once feasibility is achieved. Moreover, the sign of the reward (i.e., whether it is greater than zero) provides an intuitive indicator of whether the current design satisfies the prescribed limits. The reward weights were selected according to a safety-priority principle. Since all code-related indicators are normalized by their corresponding limit values, ${\omega }_{UZ}$, ${\omega }_{BF}$, and ${\omega }_{Ri}$ are used to balance the penalties for stiffness, global stability, and member strength/stability violations, respectively, and are all set to 1.5. The material-related weight ${\omega }_M$ is set to 5.0 and controls the material-reduction term after code compliance is achieved. These weights were determined through preliminary empirical tuning guided by engineering judgment.

In the proposed FEM–RL interaction, the reward is computed from deterministic FEM outputs rather than from a stochastic surrogate model. For the same state–action pair, fixed modeling procedures and solver settings generate consistent analysis results; therefore, reward uncertainty is mainly limited to minor numerical errors caused by rounding and solution tolerances, which have only a limited influence on the reward evaluation in this study.

2.4. SFO-Agent Design

The agent built in this study is not developed based on the more commonly used TRPO [35] or PPO [36] frameworks. The primary reason is that these methods are on-policy algorithms, for which the data collected in a given episode are discarded immediately after use, resulting in relatively low sample efficiency. In structural design tasks, however, finite element analysis is computationally expensive, and the introduction of context variables into the state to represent different design tasks further reduces repeated visits to the same state, thereby increasing the variance of learning. Inspired by the Soft Actor–Critic (SAC) [38] framework, the proposed agent, termed SFO-Agent, is formulated as a maximum-entropy RL framework with off-policy training. All collected data are stored in a replay buffer and repeatedly sampled for training, which greatly improves sample efficiency. In addition, maximum-entropy RL incorporates the entropy term in Equation (11) into both the policy objective and the value function, thereby promoting more diverse exploration and reducing the risk of convergence to a local optimum.

$$H(X)=E_{x~p}[-\log p(x)]$$

(11)

As summarized in Figure 5, the SFO-Agent comprises five deep neural networks (DNNs): an actor implemented by a policy network $\pi (a_t|s_t)$, and critics implemented by two Q-networks $Q_1 \mathrm{and} Q_2$ together with their corresponding target networks $q_1 \mathrm{and} q_2$. Data flow and parameter updates across these networks can be summarized into four parts:

• Actor–environment interaction: Given the current state $s_t$, the policy outputs an action $a_t$, which is applied to the SFO-Environment to obtain the reward $r_t$ and the next state $s_{t+1}$. The transition tuple $〈s_t,a_t,r_t,s_{t+1}〉$ is then stored in the replay buffer for subsequent training.
• Critic update: Given $s_{t+1}$, the actor outputs the next action $a_{t+1}$ and the entropy-related term $\log (\pi (a_{t+1}|s_{t+1})$. Concatenating $s_{t+1}$ and $a_{t+1}$ forms the input to the target critics, which produce two target Q-values, $q_1$ and $q_2$. Similarly, concatenating the sampled $s_t$ and $a_t$ provides the input to the critics, yielding the action-value estimates $Q_1$ and $Q_2.$ The target temporal difference and temporal difference error used for network updates are given in Equations (12) and (13).

$$T{D}_{\mathrm{target}}=r_t+\gamma (\min (q_1,q_2)-\alpha \log (\pi (a_{t+1}|s_{t+1})))$$

(12)

$$T{D}_{\mathrm{error},i}={\displaystyle \frac{1}{2}}{(Q_i-T{D}_{\mathrm{target}})}^2$$

(13)

3.

Soft update of the target critics: The target critics share the same network architecture as the critics, and this study adopts a soft update controlled by $\tau$, which incrementally moves the target parameters toward the critic parameters, as given in Equation (14), where ${\omega }_i^{\prime }$ and ${\omega }_i$ denote the parameters of the target critics and critics, respectively.

$${{\omega }_i}^{\prime }\leftarrow \tau {\omega }_i+(1-\tau ){{\omega }_i}^{\prime }$$

(14)

4.

Actor update: The actor is updated under the guidance of the critics to favor higher-value actions. For sampled states $s_t$, the actor produces a new action $a_t^{\prime }$, which is evaluated by the critics to obtain $Q_1^{\prime }$ and $Q_2^{\prime }$. The actor objective encourages both high action value and high entropy, leading to the loss function in Equation (15).

$$loss=\alpha \log (\pi (a_t^{\prime }|s_t))-\min ({Q_1}^{\prime },{Q_2}^{\prime })$$

(15)

By incorporating Prioritized Experience Replay [49] and adaptive entropy regularization into the training pipeline, the overall training procedure can be summarized by the pseudocode in Appendix A.

3. Experiments and Results

3.1. Computational Setup

To validate the effectiveness of the proposed SFO-Agent, a representative benchmark is selected for agent construction and training. The single-layer Kiewitt dome exhibits relatively uniform force distribution and high material efficiency [50] and has been widely used in practice. It is also among the commonly recommended space frame structural forms in the code [48]. Therefore, a single-layer Kiewitt dome shown in Figure 6 is adopted as the test case to evaluate the proposed agent.

Following the procedure described in Section 2.4, the components of the SFO-Agent are implemented and coupled via off-policy training. A parametric model is first developed for the Kiewitt dome, whose state space is defined by 9 parameters; their definitions and admissible ranges are provided in

Table 3. Ranges of the state parameters.

No.	Parameter	Unit	Min	Max	Resolution	Description
1	D	mm	20,000	80,000	1000	Spherical shell span diameter
2	$R_{HD}$	-	1/10	1/4	0.01	Rise-to-span ratio
3	DL	N/mm2	0.0003	0.0015	0.0001	Dead load
4	LL	N/mm2	0.0005	0.0015	0.0001	Live load
5	$l_1$	mm	2000	6000	100	Radial grid size
6	$R_{l1}$	-	0.5	1.5	0.1	Ratio of circumferential to radial grid size
7	$S_1$	-	0	74	1	Section index of radial members
8	$S_2$	-	0	74	1	Section index of circumferential members
9	$S_3$	-	0	74	1	Section index of diagonal members

. Cross-section candidates are taken from the commonly used rectangular tube sections specified in the code [51], yielding 75 sections indexed from 0 to 74 in ascending size. Prior to being fed into the networks, all state parameters are linearly scaled and normalized to [−1, 1]. This linear rescaling is used only to transform the bounded actor output into feasible design variables and does not assume a linear relationship between the design variables and structural performance. The nonlinear effects of mesh size and section selection on stiffness, stability, stress ratio, and material usage are evaluated by the FEM analysis and reflected in the reward signal during training.

The first four dimensions of the state are treated as context variables to represent different design cases. The context variables were sampled within the admissible ranges listed in Table 3, which were selected to cover representative engineering design conditions of single-layer space frame structures, including different span scales, rise-to-span ratios, and roof load levels. The action is defined by the last five dimensions of the state, forming the tensor $\left[l_1,R_{l1},S_1,S_2,S_3\right]$, which specifies the adjustments applied to the Kiewitt dome mesh sizing and the cross-section of the member groups. Similar to state normalization, the actor outputs actions through a tanh activation to constrain them within [−1, 1], followed by dimension-wise rescaling to recover the corresponding physical values.

The reward follows the criteria defined in Section 2.3, aiming to guide the agent toward designs that satisfy the code [48] requirements on strength, stiffness, and global stability. The SFO-Agent comprises five DNNs, and the network hyperparameters are summarized in

Table 4. Network configuration.

Network	Hidden Layer	Hidden Layer Dimension	Activation Function	Learning Rate
Actor	1	512	ReLU	$1\times {10}^{-4}$
	2	512	Tanh
Critic 1	1	512	ReLU	$3\times {10}^{-4}$
	2	512	Tanh
Critic 2	1	512	ReLU	$3\times {10}^{-4}$
	2	512	Tanh
Target critic 1	1	512	ReLU	-
	2	512	Tanh
Target critic 2	1	512	ReLU	-
	2	512	Tanh

. For training, the number of episodes is set to 600, and the number of trial steps T per episode is 4, meaning that each randomly initialized structure is adjusted four times. The replay buffer warm-up size is 400; network updates start only after more than 400 transitions have been stored. The batch size for sampling from the replay buffer is 256. The target-critic soft update coefficient is $\tau$ = 0.003, the discount factor $\gamma$ = 0.99, and the target entropy is set to −5.

3.2. Training Results

After 600 training episodes (approximately 12 h), the agent’s performance improves substantially and reaches a stable level. The episode-averaged reward is smoothed using a moving average with a window size of 3 and plotted in Figure 7. According to the reward definition, the reward becomes negative in two situations: when any code requirement is violated, or when the design is code-compliant but exhibits excessive material usage. Both are regarded as undesirable solutions; therefore, reward = 0 is adopted as the code threshold for evaluation, as indicated by the red dashed line in Figure 7.

The off-policy training procedure consists of two stages. Before the replay buffer reaches its minimum size, the agent only interacts with the environment to collect sufficient transitions, and no network updates are performed; this is the replay buffer warm-up stage. The second stage is the gradient-updating phase, during which the networks are updated using minibatches sampled from the replay buffer while newly generated transitions are continuously appended.

Figure 7 shows that during the replay buffer warm-up stage, the untrained agent generates low-quality designs, and almost none satisfy the code requirements. After approximately 100 episodes, training enters the gradient-updating phase, where the networks begin learning from replayed data. A clear performance improvement emerges around episode 130, and by about episode 200 the agent is able to generate code-compliant designs. Thereafter, it further optimizes material usage to obtain higher rewards. From episodes 350 to 600, the mean reward fluctuates at a consistently high level and remains well above the code threshold.

To evaluate the training stability and reproducibility of the proposed method, five independent training runs were conducted using different random seeds while keeping the network architecture, hyperparameters, reward function, and FEM environment unchanged. Figure 8 shows the mean episode reward over the five runs, with the shaded region indicating one standard deviation. In the later training stage, the reward remains mostly above zero, while the reduced standard deviation indicates that different random-seed runs converge to a similar high-reward region. These results demonstrate that the proposed training process is reproducible and reasonably stable despite the stochastic nature of RL training.

Compared with conventional DRL framework, the mean reward of the SFO-Agent still exhibits noticeable fluctuations in the late training stage. This behavior is expected because the proposed context-conditioned framework optimizes a family of structures rather than a single fixed instance. Nevertheless, the trained agent produces designs with rewards predominantly above zero, indicating that it has learned effective layout patterns for space frame structures and can generate safe and material-efficient solutions under varying initial conditions, with favorable generalization capability.

3.3. Typical Case Analysis

In this section, the trained SFO-Agent is applied to a representative design case and the resulting solutions are discussed. A Kiewitt dome with fixed span, rise-to-span ratio, and loading is selected to be analyzed. The case considers a 50 m span, a rise-to-span ratio of 1/7, dead and live loads of 1.5 kN/m² and 0.5 kN/m², respectively; the initial model is shown in Figure 9. The trained agent performs a 10-step decision sequence for this case. After each step, the structural responses are evaluated and the reward is computed, and the design with the highest reward among the 10 steps is selected as the final solution. The stepwise actions and corresponding design results are reported in

Table 5. The 10-step design procedure for typical case.

Step	${\mathit{l}}_1$ (mm)	${\mathit{R}}_{\mathit{l}1}$	${\mathit{S}}_1$	${\mathit{S}}_2$	${\mathit{S}}_3$	Reward	Compliance	M (t)
1	3600	1.2	37	20	33	9.30	Pass	59.93
2	4400	1.2	56	33	32	10.20	Pass	54.96
3	3600	1.4	46	24	29	10.57	Pass	52.93
4	4400	1.4	51	29	28	12.44	Pass	42.67
5	3900	1.5	52	27	27	10.03	Pass	55.92
6	4200	1.5	49	29	26	10.28	Pass	54.55
7	4100	1.4	52	27	26	9.43	Pass	59.22
8	4200	1.4	52	27	26	8.95	Pass	61.83
9	4000	1.5	49	26	28	10.65	Pass	52.51
10	4300	1.5	52	29	27	10.08	Pass	56.64

.

Across the design process, the agent selects each action based on the previous state to seek higher-reward solutions. As shown in Table 5, all 10 steps satisfy the code limits. The best solution is obtained at Step 4 with a reward of 12.44, corresponding to $l_1=4.4 \mathrm{m}$ and $R_{l1}=1.4$. The selected section indices for radial, circumferential, and diagonal members are 51, 29, and 28, respectively, corresponding to rectangular tube sections 400 × 200 × 8, 200 × 150 × 5 and 220 × 140 × 4, with a total material usage of 42.67 t.

The associated finite element results are shown in Figure 10: the first-order buckling factor is 10.46 (above the 7.0 limit), the maximum vertical deflection is −19.45 mm, giving a deflection-to-span ratio of 1/2570 (well below 1/400), and the maximum member stress ratio is 0.77. Overall, the design approaches but does not exceed the limit, indicating a material-efficient solution. Notably, the agent does not merely refine designs near the best-scoring region; it also explores distant but competitive actions (e.g., Step 3), which adopts a denser mesh with smaller sections and remains code-compliant, albeit with slightly higher steel usage than Step 4. This behavior suggests that the agent, by encouraging higher action entropy, promotes broader exploration and helps avoid poor local optima.

4. Discussion of Agent Performance

4.1. Agent Evaluation

To further evaluate the agent, 100 design cases are randomly generated and tested. For each case, the highest-reward design among the 10 steps is taken as the final outcome. The results for all 100 cases are summarized in Figure 11.

Across the 100 randomly generated test cases, the agent achieves a mean reward of 10.16, and all cases yield rewards greater than zero, indicating code-compliant designs with relatively low material usage. This further suggests that the SFO-Agent learns effective design strategies through interaction with the finite element solver and can rapidly generate solutions under varied design conditions. Nevertheless, several cases exhibit noticeably lower rewards, such as Case 5 and Case 56 with rewards of 1.76 and 1.84, corresponding to states [69,000, 0.21, 0.0006, 0.0006, 3900, 1.0, 28, 21, 19] and [50,000, 0.16, 0.0003, 0.0006, 4200, 0.9, 23, 16, 22], respectively. A common feature is relatively small loads. Under such conditions, the design is no longer controlled by stress ratio but by slenderness limit, leading to low load-carrying efficiency. Thus, the low rewards are primarily attributable to the specific design conditions. Even under relatively demanding constraints, the agent still produces code-compliant designs, albeit with lower rewards.

The trained agent is evaluated on unseen Kiewitt dome cases with different spans, rise-to-span ratios, and loading conditions within the predefined parameter ranges. This is the generalization capability discussed in this study, namely the agent’s adaptability to different contexts within the same parametric structural type. It should be distinguished from topological generalization across different structural systems, which is not the focus of the present study.

The above results indicate that the agent can yield lower rewards under certain design conditions, leading to performance variability. To further assess its behavior across the full parameter domains, additional samples are generated by randomly sampling design conditions while sweeping the variable of context over its entire admissible range.

For each context dimension, 20 samples are generated, yielding 20 reward curves per dimension; the corresponding mean curve is also plotted, as shown in Figure 12. Overall, the agent consistently produces code-compliant designs, with no violations observed. The rewards are relatively stable and remain at high levels with respect to structural diameter and rise-to-span ratio. In contrast, higher dispersion is observed along the dead-load and live-load dimensions: rewards are lower at small load levels and tend to increase as the loads increase. This trend is attributed to a shift in governing constraints from detailing-driven requirements at low loads to stress-governed design at higher loads, which improves load-carrying efficiency and thus increases the reward. These results indicate that a well-trained agent achieves robust performance across the full parameter domains and satisfies typical design requirements for space frame structures.

4.2. Algorithm Comparison

To justify the selection of SAC, a comparative ablation study was conducted using PPO as a representative on-policy baseline. The PPO agent was trained under the same SFO-Environment, state–action definition, reward function, parameter ranges, and finite-element interaction budget as SAC. As shown in Figure 13, both algorithms show an increasing reward trend during training, but SAC reaches the code-compliant reward region earlier and maintains a higher reward level in the later training stage. By contrast, PPO exhibits slower reward improvement and larger fluctuations, with several drops below the code threshold even in the later stage. This comparison indicates that SAC provides better sample efficiency and training stability for the proposed FEM-interaction-based structural optimization task, mainly due to its off-policy replay-buffer reuse and entropy-regularized exploration mechanism.

Another way to assess the performance of the SFO-Agent is to compare it with a conventional optimization algorithm. Three representative Kiewitt single-layer dome cases are selected and designed using both a genetic algorithm (GA) and the trained agent. The case parameters are summarized in

Table 6. Design parameters of the comparative cases.

Case for Comparison	D (mm)	${\mathit{R}}_{\mathit{H}\mathit{D}}$	DL (N/mm2)	LL (N/mm2)
Case 1	25,000	1/8	0.0014	0.001
Case 2	50,000	1/7	0.0015	0.0005
Case 3	70,000	1/6	0.0008	0.0005

, corresponding to spans of 25 m (short), 50 m (medium), and 70 m (large).

The GA uses tournament selection with a tournament size of 4, a crossover rate of 0.9, and a mutation rate of 0.2, with a population size of 40. Fitness is evaluated using the same reward function as SFO-Agent. The search is terminated when the best fitness in the population remains unchanged for three consecutive generations, indicating convergence.

For the three cases, the runtimes and key outcomes of GA and the SFO-Agent are summarized in

Table 7. Results of algorithm comparison.

Case for Comparison	GA			SFO-Agent
	Time (s)	Reward	M (t)	Time (s)	Reward	M (t)
Case 1	2036	14.54	10.67	57	13.60	12.44
Case 2	3296	12.29	43.48	102	12.44	42.67
Case 3	5862	5.86	86.06	137	7.90	73.82

. The GA is a case-specific iterative search method that must restart the optimization process whenever the design conditions change. In contrast, the proposed context-conditioned DRL framework requires a relatively long training stage (approximately 12 h), but the trained agent can be repeatedly used for new design cases with only a small number of trial evaluations. As a result, during the inference stage, the GA requires approximately 30–40 times the design time of the SFO-Agent, and its computational cost increases more markedly with model scale.

In terms of design quality, the two methods are comparable overall, and both achieve code compliance with relatively high rewards. For the short-span Case 1 and the medium-span Case 2, the rewards are nearly identical. For the large-span Case 3, where the structural complexity increases with more members, SFO-Agent outperforms GA by a clear margin; the SFO-Agent design uses 12.24 t less material, corresponding to a reduction of approximately 16.58%.

Overall, both SFO-Agent and the conventional GA demonstrate strong performance in optimizing space frame structures, achieving material reduction while satisfying code requirements. By incorporating action entropy, SFO-Agent promotes broader exploration and can deliver superior solutions in more challenging cases. Moreover, the pretraining-and-reuse paradigm is substantially more efficient than GA for large batches of similar design tasks, leading to significant savings in design time.

4.3. Extension to Other Structural Types

To further validate the applicability of SFO-Agent to space frame structural optimization, the proposed approach is extended to three additional common structural forms: single-layer lamella cylindrical reticulated shell (SL-LCRS), single-layer hyperbolic paraboloid reticulated shell (SL-HPRS), and single-layer sunflower-type spherical reticulated shell (SL-SSRS). The corresponding structural layouts are shown in Figure 14.

Following the proposed methodology, a corresponding parametric environment and SFO-Agent are constructed and trained separately for each new structural type, because different structural forms may have different geometric generation rules, state–action mappings, and design-variable meanings. After tuning hyperparameters such as learning rates, each agent is trained via off-policy training and evaluated on 100 randomly generated design cases. The training average reward curves and test results are presented in Figure 15. It can be observed that: Across the three structural forms, the SFO-Agents show rapid learning progress, with a marked improvement emerging around 150–200 episodes. After approximately 250 episodes, the mean reward remains consistently above the code threshold and gradually stabilizes. In the 100-case randomized evaluation, the agents achieve mean rewards of 11.11, 18.47, and 10.92, respectively, indicating that the learned policies can reliably generate code-compliant and material-efficient designs. The results also show that the learning difficulty varies among different structural types. This difference is mainly related to the geometric generation rules, force-transfer mechanisms, and feasible design-space sizes of the corresponding structures. When the structural response changes smoothly with respect to mesh and section variables, the agent can more easily learn stable optimization patterns. By contrast, structures with stronger sensitivity to geometric or section changes may lead to a less smooth reward landscape and therefore require more exploration during training. Notably, a small number of low-reward cases are observed, and the SL-HPRS tests include a few cases below the code threshold, suggesting occasional failures under specific design conditions. However, because RL operates through stepwise interaction rather than one-shot prediction, such low-quality outcomes are immediately revealed by environment feedback and can be handled through targeted post-processing or additional refinement.

5. Conclusions

In this study, we developed a context-conditioned DRL framework for the optimization of space frame structures, termed SFO-Agent. An interactive SFO-Environment was constructed by integrating parametric modeling with FEM. Through iterative training, SFO-Agent autonomously interacts with the FEM solver and learns the underlying relationships between design variables and structural performance, thereby progressively improving its design capability. Experimental results demonstrate the feasibility of SFO-Agent for optimizing space frame structures and suggest strong potential for practical engineering applications. The detailed conclusions are as follows.

• The SFO-Agent was established by coupling an actor network with critics and target critics. Parametric modeling, FEM, and a code-compliant reward function were then developed and embedded into the SFO-Environment. Off-policy training within a context-conditioned framework enabled continuous interaction between SFO-Agent and SFO-Environment, allowing the agent to learn reusable policies across varying design cases for automated optimization of space frame structures.
• A single-layer Kiewitt dome was selected as the benchmark, and the 600-episode off-policy training results demonstrated the progressive improvement of the agent. The analysis of a representative design case verified the rationality of the learned strategy. In addition, tests on 100 randomly generated cases and full-domain sweeps across context parameters showed robust performance over varying design cases.
• Compared with the GA, the context-conditioned DRL agent learned reusable design patterns during training and therefore required substantially less time for optimization during the inference stage. The maximum-entropy formulation enhanced action exploration, leading to improved performance on more challenging models. Moreover, the proposed method was extended to several additional forms of space frame structures, further supporting its effectiveness.
• In this study, the SFO-Agent was evaluated on common forms of space frame structures to demonstrate its effectiveness, while its applicability to a broader range of structural types still requires further extension, particularly for non-standard configurations. In addition, the current generalization capability mainly refers to adaptation to different design contexts within the same parametric structural type, rather than direct topological generalization across different structural systems. The test results also indicate that, under certain design conditions, the agent may produce low-reward solutions, suggesting that the robustness and overall optimization performance of the proposed framework should be further improved.