Optimizing Retaining Walls through Reinforcement Learning Approaches and Metaheuristic Techniques

: The structural design of civil works is closely tied to empirical knowledge and the design professional’s experience. Based on this, adequate designs are generated in terms of strength, operability, and durability. However, such designs can be optimized to reduce conditions associated with the structure’s design and execution, such as costs, CO 2 emissions, and related earthworks. In this study, a new discretization technique based on reinforcement learning and transfer functions is developed. The application of metaheuristic techniques to the retaining wall problem is examined, deﬁning two objective functions: cost and CO 2 emissions. An extensive comparison is made with various metaheuristics and brute force methods, where the results show that the S-shaped transfer functions consistently yield more robust outcomes.


Introduction
Today's society is experiencing a period of rapid growth and technological advancement that has accelerated exponentially over the years.This progression compels various industries to modernize, adapting to the evolving needs of consumers and the available resources and technologies.In this context, the construction industry has made significant strides in its design methods, gradually incorporating innovative techniques that pave the way for interdisciplinary approaches.These advancements ensure that civil work designs are optimized in terms of cost, material usage, associated carbon emissions, and other critical parameters [1].
In the realm of sustainable design, it is important to recognize that the construction industry is responsible for 33% of the energy produced globally and 30% of the total greenhouse gas emissions worldwide [2].This significant environmental impact stems from the considerable carbon dioxide emissions associated with various elements in civil works [3].These include the materials and machinery utilized throughout a project's life cycle from construction and maintenance to eventual demolition [4].As a result, the need for more sustainable practices and a reduced ecological footprint in the construction industry has become increasingly evident.
Given the pressing climate emergency, numerous processes and industries have been compelled to adopt sustainable measures or update existing practices.Developing sustainable building approaches can help mitigate the environmental impacts caused by human activity [5].This global environmental situation has urged the construction sector to adopt and implement strategies to reduce or mitigate its ecological impact.Regarding carbon emissions, several studies have sought to minimize this parameter in various structures.For instance, ref. [6] employs a Finite Element Model (FEM) and a multi-objective genetic algorithm in the BIM modeling process to reduce a building's carbon emissions.Similarly, ref. [3] presents a comprehensive calculation of emissions involved in the life cycle of road construction, from material extraction to the end of its useful life, emphasizing the importance of earthworks and machinery efficiency.The study also highlights the relevance of material selection in optimizing emissions.
In [7], the researchers economically optimized footings using the MINLP (Mixed-Integer Non-Linear Programming) method.This study achieved a significant reduction of up to 63% in the total cost of the analyzed foundations.Additionally, various examples in the literature demonstrate the application of artificial intelligence algorithms within the construction industry.These include the use of computer vision for detecting cracks and defects in buildings [8], pavements [9], and bridges [10], as well as object detection on construction sites [11] and masonry segmentation [12].
In the domain of retaining walls, similar research efforts have been undertaken.For instance, ref. [13] presents a hybrid metaheuristic optimization method called h-BOASOS that minimizes the weight and cost of cantilever retaining walls.This approach combines the butterfly optimization algorithm (BOA) and symbiosis organism search (SOS) algorithm, outperforming other algorithms in benchmark tests, real-world engineering design problems, and cantilever retaining wall problems of various heights.In [14], the authors propose a hybrid k-means cuckoo search algorithm, merging the cuckoo search metaheuristic for continuous space optimization with the unsupervised k-means learning technique for discretizing solutions.The algorithm employs a random operator to assess the k-means operator's contribution and is benchmarked against a harmony search variant.The results reveal that incorporating the k-means operator significantly improves the solution quality, and the hybrid algorithm surpasses the harmony search approach.Finally, ref. [15] applies the shuffled shepherd optimization algorithm (SSOA) to optimize reinforced concrete cantilever retaining wall structures under static and seismic loading conditions.The optimization seeks to minimize the cost while adhering to stability and strength constraints based on ACI 318-05 requirements.Comparing SSOA results with other meta-heuristics highlights the algorithm's accuracy and convergence rate efficiency.
The numerous studies mentioned above highlight the ongoing efforts to optimize the design of various civil engineering elements and emphasize the importance of interdisciplinary collaboration in updating the industry.This article outlines the development process for optimizing retaining walls.Firstly, it explains the structural calculations and considerations required for designing the retaining wall.The second section discusses optimization techniques, their functionality, and their application to the problem.
Following this, the associated calculation model is parameterized to comply with the design standards, identifying crucial design aspects that can serve as variables in the optimization process.Subsequently, objective functions for each aspect are defined: one equation determines the economic cost, while another calculates carbon dioxide emissions, both applied to the retaining wall model.
Finally, a new algorithm that solves discrete problems using metaheuristics that naturally operate in continuous search spaces is proposed.Specifically, metaheuristic techniques such as the Sine Cosine Algorithm (SCA), Whale Optimization Algorithm (WOA), and Gray Wolf Optimization (GWO) are employed and integrated with SARSA and QL reinforcement learning techniques to address the discrete retaining wall problem and assess their performance and computational times.Statistical tests are used to compare the significance of the results obtained.
Extensive experiments were conducted to compare the implemented techniques, yielding notably robust and favorable results for static methods using S-shaped transfer functions.These results were obtained from 31 independent runs and their significance was validated through the Wilcoxon-Mann-Whitney [16] statistical test.The remainder of the paper is organized as follows: Section 2 outlines the retaining wall problem's modeling, while Section 3 introduces the optimization techniques.Section 4 presents the experimental findings, and Section 5 concludes the paper with discussions and final observations.

Concrete Retaining Walls
The present section lists the most critical design parameters considered.However, before delving into them, it is necessary to know the initial condition of the problem under consideration.At this point, it should be noted that the model considers the design of a cantilever-type wall with a structural backfill that has known resistance parameters and no overload.Furthermore, in geometrical terms, the model analyzes one linear wall meter as the problem's initial condition.Finally, the selection of the number and diameter of bars that meet the design requirements will be determined using the same.Having said that, the main calculation bases are detailed below.

Calculation of applicants:
The design proposes that the structural fill exerts two types of thrust on the wall.The first type corresponds to the thrust of the soil under static conditions, while the second type corresponds to the thrust generated by the soil under pseudostatic (seismic) conditions [17].These concepts are represented by Equations (1) and (2).

Calculation of the requesting moment at point A:
The design outlines the calculation of forces at point A, defined as the most unfavorable point on the wall in terms of stress caused by the soil thrust.This condition is represented through Equation (3) and Figure 1.
where: Quantification of the design requesting forces: Once the applied moment has been determined, the moment generated by the wall's self-weight at point A must be calculated.Finally, the applied moment is defined as the sum of the previously computed moments, which are then added to obtain the design moment and axial load, as illustrated in Equations ( 4)- (7). where: where: where: where: Steel amount for stirrups: For the calculation of the reinforcement required in the wall stirrups, the dimensionless method is applied, defined through Equations ( 8)- (11).
where: ν = Dimensionless shear factor in the structure.
where: w = Calculation ratio for the steel area.µ = Calculation dimensionless factor.ν = Dimensionless shear factor in the structure.
where: Verification of overturning resistance: For the overturning resistance verification, two central moments must be determined.The first one corresponds to the resistant moment exerted by the wall's self-weight, the foundation, and the soil on the bottom (Figure 2 and Equation ( 12)) up to the outermost point of the wall (the lower end of the foundation), defined as point B. Similarly, the moment generated by the active and seismic thrusts up to that point must be determined (Equation ( 13)).With this information, it is possible to calculate the overturning safety factor using Equations ( 14) and (15). where:

Slip resistance verification:
The procedure for the slip resistance verification follows the same principle as the rollover verification; that is, the applicant and resistant forces must be determined through Equations ( 16)- (19).
where: Verification of allowable stress: Within the verification section, it is necessary to check that the stresses transmitted to the soil (maximum and minimum) do not exceed the allowable stress given by the previous geotechnical study.The verification is carried out by means of Equation (20).
where: Verification of percentage of support: Since the stress transmitted to the soil may adopt negative values, it must be checked that the foundation does not tend to lift out of the soil.Once the percentage of support has been defined, it is checked that the stress generated by the weight of the system does not exceed the admissible soil stress, with Equations ( 21)- (24).

Reinforcing reinforcement for foundation under structural backfill:
The uplift effect generated by the negative tension in the soffit area can generate additional tensile stresses in the foundation, so a reinforcement must be sized in the footing located in the soffit area of the wall.For this purpose, the design moment is determined from the forces involved (Figure 3) and then the dimensionless method is applied (Equation ( 25)).
where: Foundation armor: In addition to the reinforcement calculated above, a reinforcement for the foundation, both longitudinal and transverse, must be dimensioned.This is carried out by means of Equations ( 26) and (27). where: where:

Calculation of the steel area for the foundation:
In the case of the reinforcement for the foundation, both longitudinal and transverse, the amount of steel required is determined with Equations ( 28) and (29). where: where: Minimum armor: The amount of steel calculated above must be verified with respect to the corresponding minimum amount of steel, calculated through Equation (30).
where: emin = Minimum amount of steel required [cm 2 ].Ag = Longitudinal or cross-sectional area of the footing, as appropriate [cm 2 ].
Verification of concrete shear strength: Check if the designed concrete section has the required minimum shear strength through the Equations ( 31) and (32).

Restrictions
The design problem must respond to specific constraints to be considered correct.These limitations, known as constraints, will be fundamental in determining whether the solution proposed in the optimization process is (or is not) valid.The following are the constraints identified.
µlim: According to the initial conditions for the application of the dimensionless sizing method used to determine the amount of steel required, the relationship of Equation (33).µ = µlim (33) where: µ = Dimensionless factor, calculated in Equation (8).µlim = Calculation limit dimensionless factor, equal to 0.3047 for the case of analysis.Slip safety factor: As with the overturning safety factor, a minimum value of slip safety factor must be met to avoid the occurrence of this type of failure.That said, the restriction is set forth in Equations ( 34) and (35).
where: FSSD = Seismic slip safety factor, calculated by Equation (18).Rollover safety factor: The design must meet minimum safety criteria [18], including the overturning safety factor, which avoids abrupt failures due to overturning when loads not foreseen in the initial design are applied.Under this criterion, the conditions are defined in Equations ( 36) and (37).
Allowable stress: As a minimum requirement, the stresses produced by the interaction between the foundation and the supporting soil must be lower than the allowable bearing capacity provided by the geotechnical study accompanying any civil works design.The condition defining this restriction was previously established (Equations ( 20) and ( 23)).
Percentage of foundation support: Due to the eccentricity of the load system, there is a possibility that the stress distribution at the base of the foundation will generate a negative stress zone.In simple terms, this implies that the foundation would tend to uplift.To ensure that the stresses are properly distributed over the supporting soil, the percentage of the supported section (Equation ( 24)) must comply with the constraint outlined in Equation (38).

Target Function
The objective functions representative of the parameters under study turn out to be linear, and are presented below.
where: CEMt = Total between the sum of cost and emissions.

Techniques to Be Used
In general, design problems involve numerous variables to consider, which may also be of different natures.The combination of multiple options results in a vast number of potential solutions that are impossible to cover manually or iteratively.The need arising from this type of analysis can be addressed through various currently available optimization techniques [19].
Before delving into the optimization techniques applied to the analysis problems relevant to this paper, it is essential to understand what the optimization methods entail.Generally, optimization methods consist of a set of rules and techniques applied to a problem to find the solution that best fits the objective pursued by the process.Optimization methods are divided into two main groups: the first group comprises techniques classified as exact, according to their characteristics [19], which can analyze the entire existing search space and find the best solution for the problem.On the other hand, there are cases where the search space is too large to be explored within a reasonable time frame [19].In these situations, it is necessary to apply incomplete techniques that focus on analyzing local maxima identified within the function under study.
Another classification method applied to optimization problems focuses on the nature of the decision variables present.Thus, there are continuous optimization problems, meaning their variables represent continuous real spaces.On the other hand, combinatorial optimization problems have variables with integer values or sets of integers.Naturally, a third type of problem involves a mix of both variable types.In these cases, the problem's character is called mixed optimization.
Metaheuristics are iterative processes designed to be applied to any problem type (unlike heuristics, whose configuration is associated with a specific problem type) and, therefore, guide a subordinate heuristic to find an efficient solution in terms of approximation to the global optimum of the problem and the time required to find the proposed solution [20].Likewise, metaheuristics are subdivided into two groups, depending on the type of solution they provide, and can be based on a single solution or a population of solutions.The above description is graphically represented in Figure 4.

Exact methods
Approximate methods  Regardless of the operation adopted for a metaheuristic, there are general parameters they must follow.First, the problem set must include an objective function, which represents the aspect of the problem to be optimized.For the purposes of this research, the objective functions to be designed will aim to quantify the costs and carbon dioxide emissions associated with the adopted design.

Branch and X Constraint programming
Once the objective function is defined, two concepts common to any function must be defined: the domain and its constraints.The domain of the objective function directly responds to the values that can be adopted by the variables considered in the problem, which are defined through the constraints.These constraints are conditions (formulated as functions) that limit the problem.All possible solutions that result from the intersection between the constraints of the problem make up the search space that will be considered in the optimization process.
Population-based metaheuristic techniques utilize a structured approach consisting of three "for" loops of iterations, solutions or individuals, and decision dimensions.Under this structure, perturbations to each dimension of the solutions during each iteration are performed through the method indicated in line 5 of Algorithm 1, where ∆ refers to the characteristic perturbation or movement operators of each technique.As previously mentioned, in this paper, the metaheuristic techniques that have been implemented and discretized to adapt to the presented problem are: the Whale Optimization Algorithm (WOA) [22], the Sine Cosine Algorithm (SCA) [23], and the Grey Wolf Optimizer (GWO) [24].The general scheme of each population-based metaheuristic is represented in Algorithm 1.
Algorithm 1 General scheme of metaheuristics.
1: Initialize a random swarm 2: for iteration (t) do end for 8: end for

Proposal: Discretization Schemes Selector
The resolution of complex and frequent combinatorial problems in industry is a priority for both academia and industries.A smart scheme selector for discretization is proposed, which integrates existing methods to balance exploration and exploitation, avoiding local optimizations.The balance between exploration and exploitation is a key factor in the performance of a metaheuristic.The method uses an intelligent operator to determine the appropriate discretization scheme at each iteration, to achieve the best quality results.
This proposal is based on the Binarization Schemes Selector (BSS), which has been proposed and utilized in [25][26][27][28], where a smart selector at a higher level selects from a set of actions, in this case transfer functions, to better choose how to discretize our continuous variables obtained from the metaheuristic, in order to use them in the discrete domain of our problem.
In this work, Q-Learning (QL) [29] and SARSA [30] are implemented as the intelligent operator of the proposal, which selects the discretization technique to be used based on a reward system, with which it learns in a deterministic manner.
The reward in the RL algorithm is crucial for the good performance of these algorithms; thus, in the literature, there are several methods to calculate rewards.We have implemented the same rewards used in BSS [25,26], which have been proposed in [31,32], which are presented in Equations ( 43 and The reward or punishment is judged by the outcome obtained by the performance of the action.Thus, it is important to define which comparison measures will be used to discriminate the outcome.In this work, the comparison measure is the fitness obtained in each iteration of the optimization process, and it is compared with the best fitness obtained.If the fitness improves, the action is rewarded, whereas if the fitness worsens, the action is punished.In this work, two states are defined, which refer to the phases of a metaheuristic: exploration and exploitation.These states were not chosen randomly because, as previously mentioned, the objective of this work is to improve the balance between exploration and exploitation of the metaheuristics to obtain better results.This process is represented in Figure 5.In the literature, different authors [33,34] propose metrics that allow us to quantify the diversity of individuals in population algorithms, where the Dimensional Diversity by Hussain stands out [35].Let Div be the diversity of the population at a particular time, and to calculate Div, the following equation is used: where xd denotes the average of individuals in dimension d, x d i is the value of the i-th individual in dimension d, n is the number of individuals in the population, and l is the size of the dimension of the individuals.One of the methods to estimate exploration and exploitation is proposed by Morales-Castañeda et al. [36], who, based on the quantification of the diversity of a population, proposed a method to estimate exploration and exploitation in terms of percentages.The percentages of exploration (XPL%) and exploitation (XPT%) are calculated as follows: where Div is the determination of the diversity state given by Equation ( 49) and Div max denotes the maximum value of the diversity state found throughout the optimization problem.Equations ( 50) and (51) are generic, so it is possible to use any other metric that calculates the diversity of a population.Therefore, the transition of states will be determined through the following method (Figure 6).

Discretization
There are various ways to transfer continuous values to binaries [37,38], but there is little documentation on discretization.In this proposal, discretization is generated through transfer functions commonly used in two-step binarization.The transfer functions achieve transferring a continuous value to a value in the range of [0, 1].The discretization proposal is integrated under the general scheme of MH, as presented in Algorithm 2. This discretization function is broken down in Algorithm 3, where the input of the discretization function is the set of solutions, the user-defined parameter beta, and the type of transfer function.In the algorithm, for each dimension (line 5), we check if the value of our position X i,d is greater than r 1 , which corresponds to a random number between [0, 1] (line 6); if this is met and also the value of beta > r 2 another random number between [0, 1] (line 7), then we update our X i,d to the value of the best solution for that dimension (line 8), but if beta is not greater, then it is updated with a random value (line 10).Otherwise, if the value of X i,d is not greater than r 1 , then the element is not updated.Finally (line 17), the discretized value of the solutions X is returned.Algorithm 2 Discrete general scheme of metaheuristics.end for 16: end for 17: return X

Experimental Results
Three metaheuristics of different classes and complexities have been run for the experimental analysis: SCA, GWO, and WOA.These MH have been tested in 11 instances and run 31 times independently on an i9-10900k, with 32 Gb of RAM and in a Python 3.7 implementation.The extensive experimental comparison has also been carried out on different discretization schemes: on the one hand, on a static scheme using only one of the eight classical transfer functions (S-shaped and V-shaped); and together with the proposal explained in Section 4, using Q-Learning and SARSA as smart selector, each of them evaluating different rewards.
The representative design problem of a retaining wall involves several design variables.Given this situation, and depending on the time required for processing, a list of parameters considered as decision variables will be detailed.Subsequently, these points will take a variable value according to the evaluation to be performed, as determined in the iterative process of the optimization techniques.The following variables used are defined in Table 1.
In order to simplify the representation of results, the main results obtained will be presented, with access to all the results in the open repository https://github.com/joselemusr/DSS-Retaining-walls.Table 2 presents the nomenclature used to identify each variation of the algorithms, while Tables 3 and 4 present the results obtained by GWO using static schemes based on V-shaped and S-shaped, where this technique has presented the best performance in conjunction with static schemes, where in the first row we can see the version of the algorithm used, in the first column the name of the instance, where the terminations "-C" correspond to those whose objective function calculation is cost-based (Equation ( 40)), "-E" to those that correspond to CO 2 emissions (Equation ( 41)) and "-C+E" to the objective function that considers the sum of both (Equation ( 42)).The second column individualizes the best known value calculated by grid search, and then the three columns are repeated indicating the best value found by the version indicated in the column, the average value of the 31 independent runs and the Relative Percentile Destivation (RPD), which is calculated according to Equation (52).While in intermediate rows the average values for the set of instances using the same objective function are presented.Tables 5-10 present the results obtained using the proposed dynamic techniques based on Q-Learning and SARSA.In order to analyze the GWO technique, which was the best-performing technique, it is necessary to quantify the RPD obtained in ranges.These categorizations were established according to a minimum and maximum range, average and standard deviation (Table 11).With the ranges already established, the distribution in GWO of the different techniques is represented in Table 12.Table 13 presents the parameters utilized in this research, providing information about the set of instances used.The table comprises several data columns, organized as follows: 1.
Inst. lists each of the studied walls in sequential order.

2.
Opt. displays the optimal value for each respective instance.

3.
Best indicates the best value achieved during the execution.

4.
Shape identifies the top-performing algorithm that reached the best value.

5.
Fitness denotes the fitness or efficiency value of the best algorithm.6.
The remaining columns represent the design parameters.
This tabular representation effectively communicates the key parameters and findings from our research, allowing readers to quickly grasp the results and assess the performance of various algorithms in the study.

Distribution Analysis
The violin plots (Figures 7-9) presented in this section offer a clear and concise visualization of the results obtained from our investigation of the algorithms employed.A representative instance was selected to showcase the results using the best-performing metaheuristic: GWO.Upon examining the plots, it is evident that the violins corresponding to Algorithms S1-S4 exhibit lower data dispersion compared to the others, signifying a greater efficiency in minimizing the problem.Overall, the violin plots provide an effective and lucid representation of the findings from our study, which will assist researchers and subject matter experts in making informed decisions when selecting techniques for future research and practical applications.The plots are structured as follows: the Y-axis displays the fitness range, which refers to the data density at that level, while the X-axis represents all the evaluated algorithms.

Conclusions
This paper introduces a novel approach to optimizing retaining structural wall designs by implementing an innovative discretization method that utilizes reinforcement learning and transfer functions.Conventional structural design practices rely on empirical knowledge and experience, resulting in designs that prioritize strength, operability, and durability.However, these designs have room for improvement by minimizing their associated costs and CO 2 emissions.
Designs created using experience and empirical knowledge can produce feasible results that are often acceptable in cost and operability.However, these designs do not necessarily represent the optimal solution for a given scenario.Retaining wall design problems are considered combinatorial optimization challenges, which can quickly result in a combinatorial explosion due to the large number of variables and extensive discrete domains involved.Consequently, it is crucial to use incomplete techniques to solve these problems within a reasonable computational time frame.Metaheuristics have emerged as a prominent solution for addressing such challenges.
This work outlines the calculation procedure for determining the dimensions of a retaining wall, framing it as an optimization problem.Three metaheuristic techniques are implemented to evaluate their performance and use various transfer functions for the discretization process.Additionally, a novel discretization method, based on Q-Learning and SARSA, is employed to select the appropriate transfer function for each iteration during the discretization process.
The proposed model aims to minimize costs, CO 2 emissions, or a weighted combination of both, considering the design parameters detailed in Table 1 as problem variables, with the constraints being those inherent to the design of a retaining wall, as detailed in Section 2.1.Experimental results have been obtained using the Sine Cosine Algorithm (SCA), Whale Optimization Algorithm (WOA), and Gray Wolf Optimization (GWO) metaheuristics, with each run independently for 31 runs across 11 problem instances.After evaluating all the techniques across the three objective functions, the p-values obtained by the Wilcoxon-Mann-Whitney test for the GWO are presented, revealing the best results for the static versions using S-shaped transfer functions.GWO demonstrated significant differences in its results, indicating that it outperformed the other techniques for this specific problem.This does not imply that the other techniques are inadequate; rather, it suggests that the GWO S-shaped technique performs better for this particular problem.
The development of new optimization techniques, such as the one in this article, is crucial.It not only benefits the engineering profession, but also has considerable environmental and economic implications.By optimizing design processes and reducing costs and CO 2 emissions, the industry can contribute to a more sustainable future.One discussion point in this work is selecting transfer functions used in the discretization process.S-shaped functions provided the best results in terms of minimizing costs and CO 2 emissions, as demonstrated.However, other functions might offer better results for different problems or under various conditions, which the non-free lunch theorem also supports.Investigating how different transfer functions could affect results and determining an optimal function for a specific problem would be interesting.
Another discussion is the comparison of the results obtained with the different metaheuristic algorithms used in this work.While GWO with S-shaped transfer gave the best results, the other two algorithms also produced acceptable results.However, it could be argued that more metaheuristic algorithms should be compared to fully evaluate the effectiveness of this technique in retaining wall design optimization.In addition, it would be interesting to compare the results obtained with the approach proposed in this work with other retaining wall design optimization approaches, such as those based on other machine learning techniques.
Finally, the application of this approach to real-world design problems could be discussed.While the results are promising, it is possible that the implementation of this approach in real construction projects may be more complicated than the experimental results suggest.For example, there may be time and resource constraints that were not accounted for in this work that could affect the ability to use this approach in a real construction environment.Therefore, it could be argued that further research is needed to determine how this approach can be effectively implemented in real-world construction projects and how the practical challenges associated with its implementation can be addressed.
Future research should consider a broader range of assessment techniques, parameter variations, and other considerations.These distinctions could lead to results that are closer to reality.There also remains the challenge of extending these techniques to other construction problems, allowing specific processes to be designed more efficiently and accurately.Overall, the paper presents a promising approach to improving civil work designs, and the results suggest that further research in this field could lead to significant advances in the field.

Figure 1 .
Figure 1.Stress distribution and location of point A in the retaining wall.
generated by the self-weight at point A [T • m].d1 = Distance from the centroid of the prismatic section of the wall to point A [m]. N1 = Eigenweight of the prismatic section of the wall [T].d2 = Distance from the centroid of the triangular section of the wall to point A [m]. N2 = Eigenweight of the triangular section of the wall [T].

φ=
Reduction factor for flexocompression equal to 0.83.β = Reduction of the characteristic strength of concrete equal to 0.85.f c = Characteristic resistance of concrete to compression [T/m 2 ]. b = Width of the wall, corresponding to 1 [m].d = Width of the base of the wall without covering [m].
N u = Design axial load [T].φ = Reduction factor for flexocompression equal to 0.83.β = Reduction of the characteristic strength of concrete equal to 0.85.f c = Characteristic resistance of concrete to compression [T/m 2 ]. b = Width of the wall, corresponding to 1 [m].d = Width of the base of the wall without covering [m].
area [cm 2 ]. w = Calculation ratio for steel area .β = Reduction of the characteristic strength of the concrete equal to 0.85.f c = Characteristic resistance of concrete to compression [T * m 2 ]. b = Width of the wall, corresponding to 1 [m].d = Width of the base of the wall without covering [m].f y = Steel creep [T/cm 2 ].

Figure 2 .
Figure 2. Resisting forces acting on the retaining wall.
requesting force [T].qe = Static thrust exerted by the filler [T/m].qs = Seismic thrust exerted by the backfill [T/m].b = Width of the wall, corresponding to 1 [m].Fres = (Ns + Nm + N1 + N f ) • µ (17) where: Fres = Slip-resistant forces [T].Ns = Self-weight of the soil on the bottom [T].Nm = Self-weight of the wall wedge and the soil above it [T].N1 = Self-weight of the prismatic section of the wall [T].N f = Self-weight of the wall foundation [T].µ = Coefficient of friction between the foundation and the soil.Seismic slip safety factor.Fres = Slip-resistant forces [T].Fsol = Slip-requesting force [T].Seismic slip safety factor.Fres = Slip-resistant forces [T].qe = Static thrust exerted by the filler [T].
σ = Stress transmitted to the foundation soil [T/m 2 ].Ns = Self-weight of the soil on the bottom [T].x1 = Distance from the centroid of Ns to point B [m]. Nm = Self-weight of the wall wedge and the soil above it [T].x2 = Distance from the centroid of Nm to point B [m]. N1 = Self-weight of the prismatic section of the wall [T].x3 = Distance from the centroid of N1 to point B [m]. N f = Self-weight of the wall foundation [T].L = Total length of the foundation [m].b = Width of the wall, corresponding to 1 [m].MsB = Moment calculated at point B [T • m].
: e = Eccentricity of forces [m].Ns = Self-weight of the soil on the bottom [T].x1 = Distance from the centroid of Ns to point B [m]. Nm = Self-weight of the wall wedge and the soil above it [T].x2 = Distance from the centroid of Nm to point B [m]. N1 = Self-weight of the prismatic section of the wall [T].x3 = Distance from the centroid of N1 to point B [m] of the foundation supported on the ground [m].L = Total length of the foundation [m].e = Eccentricity of forces [m].
e f f ective = Effective stress applied to the foundation soil.Ns = Self-weight of the soil on the bottom [T].Nm = Self-weight of the wall wedge and the soil above it [T].N1 = Self-weight of the prismatic section of the wall [T].N f = Self-weight of the wall foundation [T].b = Width of the wall, which corresponds to 1 [m].g = Section of the foundation supported on the ground [m].Section of the foundation supported on the ground [m].L = Total length of the foundation [m].
generated by backfill soil measured at design point [T • m].M f = Moment generated by the foundation measured at the design point [T • m].Ms f = Moment generated by the foundation soil measured at design point [T • m].

Figure 3 .
Figure 3. Stresses involved in structural reinforcement design.

Figure 5 .
Figure 5. Q-Learning and SARSA scheme for different rewards.

Figure 6 .
Figure 6.Discretization scheme selector with Q-Learning or SARSA as Smart Operator.

gammaν 2 σ
Existing soil density [T/m 3 ].hz Height of wall [m].cs Seismic thrust coefficient.MsA Moment calculated at point A [T • m].M active Static moment generated by the ground at point A [T • m].M seismic Seismic moment generated by the ground at point A [T • m].M pp Moment generated by the self-weight at point A [T • m].d1 Distance from the centroid of the prismatic section of the wall to point A [m]. N1 Eigenweight of the prismatic section of the wall [T].d2 Distance from the centroid of the triangular section of the wall to point A [m]. N2 Eigenweight of the triangular section of the wall [T].M A Total moment at point A [T • m].MsA Moment calculated at point A [T • m].M pp Moment generated by self-weight at point A [T • m].Meu Design moment [T • m].γ f Moment majorization factor.Nu Design axial load [T].Nt Own weight of the wall [T].µ Dimensionless calculation factor.φ Reduction factor for flexocompression equal to 0.83.β Reduction of the characteristic strength of concrete equal to 0.85.f c Characteristic resistance of concrete to compression [T/m 2 ]. b Width of the wall, corresponding to 1 [m].d Width of the base of the wall without covering [m].Dimensionless shear factor in the structure.w Calculation ratio for the steel area.A Dimensionless calculation factor.f y Width of the base of the wall without covering [m].M r Overturning resisting moment [T • m].Ns Self-weight of soil on bottom [T].x1 Distance from the centroid of Ns to point B [m]. Nm Dead weight of the wall wedge and the soil above it [T].x2 Distance from the centroid of Nm to point B [m]. N1 Self weight of the prismatic section of the wall [T].x3 Distance from the centroid of N1 to point B [m]. N f Self weight of the wall foundation [T].x4 Distance from the centroid of Nf to point B [m]. MSB Moment resisting overturning [T].M active B Self-weight of the soil on the bottom [T].M seismic B Distance from the centroid of Ns to point B [m]. FSSV Overturning seismic safety factor.FSEV Rollover static safety factor.Fsol Slip requesting force [T].Fres Slip resistant forces [T].FSSD Seismic slip safety factor.L Total length of foundation [m].g Section of the foundation supported on the ground [m].e Eccentricity of forces [m].Acronyms Part e f f ective Effective stress applied to the foundation soil.Mdesign Design moment of reinforcement reinforcement [T • m].Mdl Design moment of longitudinal reinforcement [T • m].Bdt Maximum foundation flight [m].Bm Width of wall base [m].Pp Self-weight of the wall [T].Mdt Design moment of transverse reinforcement [T • m].T Design stress [T].Md Corresponding design moment (Mdl or Mdt) [T • m]. )-(47).

Table 3 .
Comparison of the metaheuristics GWO S-shaped.

Table 8 .
Comparison of the metaheuristics GWO-Q-Learning.

Table 9 .
Comparison of the metaheuristics WOA-Q-Learning.

Table 10 .
Comparison of the metaheuristics SCA-Q-Learning.

Table 13 .
Design parameters of the best results for each version.

d
Effective shoe height (height without cover) [m].As Required steel area [cm 2 ].Fy Yield stress of steel [T/cm 2 ]. emin Minimum amount of steel required [cm 2 ].Ag Longitudinal or cross-sectional area of the footing, as appropriate [cm 2 ].Concrete modification factor.For normal concrete, λ = 1.µlim Calculation limit dimensionless factor, equal to 0.3047 for the case of analysis.CMt Total cost of retaining wall [CLP].CM1 Cost of one cubic meter of concrete [CLP/m 3 ].PM1 Total volume of concrete used [m 3 ].CM2 Cost of one kilogram of steel [CLP/kg].PM2 Total kilograms of steel used [kg].EMt Total carbon dioxide emissions [T].EM1 Tons of carbon emitted per cubic meter of concrete [T/m 3 ].PM1 Total volume of concrete used [m 3 ].EM2 Tons of carbon emitted per kilogram of steel [T/kg].PM2 Total kilograms of steel used [kg].