Multi-Agent Transfer Learning Based on Evolutionary Algorithms and Dynamic Grid Structures for Industrial Applications

Abstract

Distributed production systems have to increasingly balance economic goals such as energy efficiency and productivity with critical technical requirements such as flexibility, real-time capability, and reliability. This paper presents a novel approach for distributed optimization by means of Evolutionary State-based Potential Games with dynamic grid structures. More in detail, we leverage the combination of Potential Games which provide rigorous convergence guarantees with population-based optimization to improve the efficiency of the learning process. Specifically, we address challenges of previous approaches including inefficient best response strategies, insufficient coverage of the state–action space and the lack of knowledge transfer among agents. The developed strategies are evaluated on a industrial system of laboratory scale. The results highlight advances in evolutionary state-based knowledge transfer and an improved coverage resulting in efficient control policies. By leveraging dynamic grid structures, Evolutionary State-based Potential Games enable the maximization of weighted production targets while simultaneously eliminating process losses resulting in improvements in the considered metrics compared to state-of-the-art methods.

1. Introduction

Over recent decades, the integration of Artificial Intelligence (AI) and the Internet of Things (IoT) has significantly transformed process automation [1,2] to Cyber–Physical Systems (CPS) [3] initiated by applications such as self-learning [4], anomaly detection [5] and predictive maintenance [6].

In order to meet modern industrial demands, workflows have to become more adaptive and flexible as a key to enhanced efficiency, reduced errors and ensured scalability. Consequently, there is a growing interest in distributed optimization strategies that rely on flexible, autonomous control approaches [7,8]. Addressing these requirements calls for a distributed Multi-Agent System (MAS) approach [9], capable of aligning local objectives with global optimization goals.

Typical approaches for machine learning-based distributed optimization algorithms are found in both game theory (GT) and reinforcement learning (RL). While RL is commonly applied in areas like robot navigation [10], job shop scheduling [11], and autonomous driving [12], GT is increasingly used in domains such as urban traffic control [13] and cloud security [14]. The use of multi-agent GT architectures for production systems with simple best response learning for optimization in large-scale production environments based on Potential Games (PG) has been demonstrated and further extended by state informations in [15]. Ref. [16] built on this using multi-step model predictors in model-based learning to train State-based Potential Game (SbPG) players in simulated environments for self-optimizing manufacturing systems. Furthermore, Transfer Learning (TL) approaches have been introduced to infer similarities between players during training [17]. In parallel, other works combined SbPGs with Stackelberg strategies in modular systems, assigning roles to players to facilitate multi-objective optimization via simplified utility functions [18], and incorporating gradient-based learning methods [19].

Based on these principles, we are expanding these strategies and focus on the following challenges: First, RL reaches its limits in complex multi-agent systems, particularly due to high computing costs, slow convergence, and problems with generalization [20,21]. In GT, the assumption of rational players and the search for stable equilibria such as the Nash equilibrium complicate practical applicability in dynamic, real-world scenarios [22]. Second, the best response strategies employed before rely on static grids which appear to be highly ineffective for dynamic systems [20]. This results in increased computational load and incorrect decisions, as well as limited generalizability [21]. Third, in dynamic and complex production environments with multiple products and machines, traditional optimization methods reach their limits [23]. High variance, machine dependencies, and constant rescheduling make it difficult to make efficient decisions and adapt quickly to new production conditions [24].

To tackle the first challenge, we propose a novel agent structure that combines GT-based learning with Evolutionary Algorithms (EAs). This enables more diverse strategies for optimizing manufacturing [25]. Combining EAs with best response strategies [15,26] can be expected to offer more efficient learning. To address the inherent limitations of static grid structures for best response learning, we propose a novel dynamic grid structure and integrate it into an EA-based learning strategy. Furthermore, we tackle the complexity of dynamic and complex environments by introducing a knowledge transfer framework which allows us to exchange gained knowledge between individual learning agents, further increasing the efficiency of the self-learning process.

The main contributions of this work can be summarized as follows:

• We introduce a novel Evolutionary State-based Potential Game, called Evo-SbPG, to enable adaptive and scalable distributed optimization. We formally provide convergence guarantees for the novel game structure.
• We propose a state–action representation, called dynamic grid structures (DGS), which allows for a more flexible representation of agent policies.
• We propose a novel EA-embedded knowledge transfer scheme between agents to share existing knowledge and explore new strategies on the DGS level.
• We evaluate the approaches developed in a production environment and compare the results of different alternative learning strategies, highlighting the effectiveness of the approach.

The structure of this work is divided into an overview and evaluation of the current literature in Section 2. In Section 3, the problem statement is presented and Section 4 presents the framework and the defintion of the Evo-SbPG. Section 5 explains the fundamental approach of the DGS, which is extended by a multi-agent knowledge transfer. The convergence analysis is provided in Section 6. An analysis of the results and an extensive discussion are provided in Section 7, with a final conclusion in Section 8.

2. Related Work

In this section, we discuss research on process optimization in distributed manufacturing, use of EAs for distributed optimization, and knowledge transfer in multi-agent systems.

2.1. Optimization in Distributed Manufacturing Processes

Distributed manufacturing describes the production of goods in a spatially separated arrangement of production facilities and locations. In contrast to traditional manufacturing processes, which are characterized by a central manufacturing and production process, this method of manufacturing has been enabled by IoT [2], adaptive manufacturing [27] and advanced automation technology. The benefits of distributed manufacturing processes are characterized by shorter lead times, flexibility to meet local market requirements and resilience to total downtime.

Distributed manufacturing processes play an increasing role in industrial applications [8,15,28]. Based on the prevailing interaction scheme between agent and environment, approaches from the fields of RL [15,29] and GT [16,19] have been employed. Particularly, specific game types like PG and SbPGs [15] have been used for distributed optimization in production environments. This has been extended to model based SbPG by means of multi-step model predictors [16] to train SbPG players in simulated environments. We extend the current state of the art in several aspects. First, we propose an evolutionary SbPG approach allowing us to integrate EA techniques with convergence guarantees. Second, we employ dynamic grid structures which prove useful for the more efficient use of best response strategies.

2.2. Optimization Using EAs

EAs are stochastic, meta-heuristic optimization methods inspired by natural processes [30,31]. Initially, individual solutions are generated and grouped into a population, whose evolution is driven by parameters such as selection pressure, mutation rate, and recombination rate [32,33,34]. Each solution is evaluated by a fitness function that determines its performance with respect to the optimization objective. A deeper understanding of the fitness landscape enables strategies to overcome local optima [35], while population diversity can be maintained by controlling convergence and introducing exit criteria. Combining EA with local optimization methods leads to hybrid approaches that balance global exploration and local exploitation, offering robust solutions to complex problems [36]. EA-based approaches, which are widely used in the optimization of industrial processes, have been recently applied to self-optimizing PLC control systems [37]. These studies demonstrate the potential of EA-driven MAS to optimize complex production environments through adaptive algorithms and dynamic action state mapping.

In the field of EA, various approaches and techniques have been developed to solve complex problems. Key methods include Genetic Algorithms (GAs) [38], Evolutionary Strategies (ES) [39] and other forms of stochastic optimization [40,41]. These methods provide robust and flexible solutions, but face challenges in scalability and parameter tuning. The inherent stochastic nature of these algorithms introduces variability in reproducibility, potentially leading to overfitting and reducing the generalizability of solutions [42,43]. In addition, convergence is typically not assured by EA-based methods. We are using the challenge posed by limited resources and use EA in combination with SbPG to provide solutions that can be used for complex systems with resource-saving state–action representation.

2.3. Knowledge Transfer in Multi-Agent Systems

TL is a machine learning (ML) technique that use existing knowledge structures from one task or domain to improve the performance of a related but different task [44]. Instead of training a model from scratch, it uses pre-trained models or knowledge structures, adapting them to the specific requirements of the new task [45]. This strategy reduces the need for large datasets and extensive computation, making it particularly effective for tasks with limited data or overlapping features with prior knowledge [45,46]. TL is used for approaches in applications such as multitask learning [47], transfer reinforcement learning [44,48], auxiliary learning [49] or unsupervized domain expansion [50]. There are also various strategies for transferring parameters, rewards or policies, each designed to improve learning efficiency and adaptability in novel task environments [44,48]. Collectively, these methods aim to improve the generalization and applicability of learned knowledge across tasks.

Within the field of agent-based TL, a variety of methodologies have been developed to address different challenges [44,48]. Another subfield is domain adaptation [51], where existing knowledge structures from the source domain are adapted to differences in data distribution within target tasks. A key challenge in distributed knowledge sharing is the difference in how source and target tasks are handled [49,50]. This can lead to negative transfer of irrelevant or harmful information [52]. In the context of this study, we propose an approach to knowledge transfer between multiple EA agents that incorporates the entire DGS of each individual. This approach involves the exchange of domain-specific strategies and knowledge structures to improve convergence.

3. Problem Description

We focus on distributed production systems as illustrated in Figure 1.

The system, divided into functional sub-areas, operates on two interconnected levels, the process level and the communication level, see Figure 2. At the process level, production transfer involves the movement of goods between sub-processes. Simultaneously, knowledge transfer enables continuous communication via interfaces such as Ethernet, field bus systems, or wireless links. Depending on system complexity, interactions can occur sequentially or in parallel while forming discrete, continuous, or hybrid process structures.

We model the structure of distributed manufacturing systems using graph theory. Particularly, we formulate a directed graph $\mathcal{G}(\mathcal{V},\mathcal{E})$ with vertices set $\mathcal{V}$ and edges $\mathcal{E}$. Further, we consider a group of actuators $\mathcal{N}=1;2;\dots ;N$ with corresponding action space ${\mathcal{A}}_i\in {\mathbb{R}}^c\times {\mathbb{N}}^d$ and the state space of the production system $\mathcal{S}\in {\mathbb{R}}^m$. For each actuator, we define its upstream state space ${\mathcal{S}}_{prior}^{A_i}=\left\{s_i\in \mathcal{S}|\exists e=(s_i,A_i)\right\}$ and a downstream state space ${\mathcal{S}}_{next}^{A_i}=\left\{s_j\in \mathcal{S}|\exists e=(A_i,s_j)\right\}$. To model the production objectives, we define fitness functions ${\Psi }_i(S^{A_i},a)$ with $S^{A_i}\in {\mathcal{S}}^{A_i}={\mathcal{S}}_{prior}^{A_i}\cup {\mathcal{S}}_{next}^{A_i}\cup {\mathcal{S}}^g$ and $a_i\in {\mathcal{A}}_i$ where ${\mathcal{S}}^g$ denotes global states known to all agents.

In our study, we allow agent–agent knowledge transfer via the communication level. This leads to a transfer set ${\mathcal{T}}_{i,j}$ that promotes knowledge transfer between EA agents i and j. Here, transfer is limited to agent pairs, defined as $e=(A_i,A_j)$.

Consequently, the overall objective of distributed optimization can be stated as the maximization of a globally defined fitness

$$\underset{\forall a\in \mathcal{A}}{max}{\Psi }_{global}(S^{A_i},a).$$

(1)

by solely maximizing the local fitness functions ${\Psi }_i$ of each module. The latter is achieved by combining ideas from SbPG to evolutionary processes. Such distributed production scenarios can be found in various areas of industrial production, such as food production, automotive industry, or pharmaceutical production.

4. Framework and Evo-SbPG Definition

In this section, we introduce an EA-based multi-agent structure, as shown in Figure 3. To this end, we employ an EA-based agent i for each actuator with corresponding action set $A_i$ including discrete actions $a_i\in \mathbb{N}$ or continuous actions $a_i\in \mathbb{R}$.

To allow for distributed optimization according to Equation (1), we propose to leverage multi-agent structures from game theory, namely SbPG, and define them in the context of EAs. This results in an EA agent structure defined as follows.

Definition 1.

A game $Evo\text{-}SbPG(\mathcal{N},\mathcal{A},\mathcal{S},\left\{{\Psi }_i\right\},{\Psi }_{global})$ defines an Evo-SbPG if a global objective ${\Psi }_{global}:a\times S\to {\mathcal{R}}^{d_i}$ can be found that for every state–action pair $[s,a]\in S\times \mathcal{A}$ conforms to the conditions

$${\Psi }_i(s,a_i,a_{-i})-{\Psi }_i(s,a_i^{\prime },a_{-i})={\Psi }_{global}(s,a_i,a_{-i})-{\Psi }_{global}(s,a_i^{\prime },a_{-i})$$

(2)

and

$$\begin{array}{c}\hfill {\Psi }_{global}(s^{\prime },a_i)\ge {\Psi }_{global}(s,a_i).\end{array}$$

(3)

Thereby, the optimization uses an evolutionary population-based learning process.

Equation (3) is standard in SbPGs and assures the contractivity of the potential function with respect to the state variable. The most crucial characteristic of standard SbPGs [26] is their convergence properties to equilibrium points given a best response strategy. Furthermore, various conditions have been derived to prove the existence of an SbPG for a given game setting [53]. These properties directly translate to Evo-SbPG. In what follows, we will present a novel best response learning strategy using evolutionary operations, bringing together convergent best response learning and evolutionary operations.

5. EA-Based Learning with Dynamic Grids

A central aspect is the interaction of SbPGs within the EA framework. In what follows, building on the research of [15,19] regarding best response learners, we start with extending the best response learners to using DGS. Then, we demonstrate the adaptation of evolutionary principles through population modelling and detail the integration of genetic operators by recombination, mutation, and selection, which is categorized into local and global scales. Lastly, we show the implementation of a convergent knowledge transfer procedure.

5.1. Characteristics of Best Response Learning

Existing approaches for optimizing SbPG are based on best response learning [15], later expanded to gradient-based learning in [19]. These works use static, fixed grids to map states into resulting actions. The grids are defined by support vectors $s_t^{A_i}$, $l=1,2,\dots ,L$ whose assigned state value is fixed while the action values are continuously adjusted according to a best response stategy, see Figure 4a). The computation of an action $a_{i,t+1}$ given an actual state value $s_{i,t}$ is then computed as a distance-weighted sum of all support vectors

$$a_{i,t+1}=\sum _l{\displaystyle \frac{w_i^{l,k}}{{\sum }_m{w}_i^{l,m}}}·a_{i,l}^l.$$

(4)

with

$$w_i^{l,t}={\displaystyle \frac{1}{{\left(D_i^{l,t}\right)}^2+y}}$$

(5)

and

$$D_i^{l,t}={\Vert s_t^{A_i}-S_t^{A_i}\Vert }_2^2.$$

(6)

However, the static grids are inefficient for nonlinear optimization while best response strategies often exhibit slow convergence. In the following, we extend the learning strategies using a population-based learning within dynamic, adjustable grid structures as shown in Figure 4b). The process schematic of dynamic adjustment and coverage maintenance is illustrated in Figure 4c).

To this end, we first consider the support vectors as individuals of a population consisting of its state values, action value and an associated fitness values. Second, we use recombination and mutation to encourage diversity in the action update. Third, we update the state values within the population based on the system dynamics. Fourth, we use a combination of local and global selection to ensure a constant population size in every step. We will now present these steps in detail.

5.2. Population of Support Vectors

We define, for each support vector, an individual $\overrightarrow{I}$ within a population $\overrightarrow{P}$ defined as

$$\overrightarrow{I}(s^{A_i},a_i,\psi )={\left[\begin{array}{c}{s}^{A_i}\\ a_i\\ \psi \end{array}\right]}^T.$$

(7)

with state support vector $s^{A_i}\in {\mathcal{S}}^{A_i}$, corresponding action $a_i\in A_i$ and associated fitness $\psi$. We employ a finite number $\nu$ of individuals in a population ${\overrightarrow{P}}_i^t$ of agent i resulting in

$${\overrightarrow{P}}_i^t={\left[\begin{array}{c}{\overrightarrow{I}}_1^t,{\overrightarrow{I}}_2^t,\dots ,{\overrightarrow{I}}_{n_{\nu }}^t\end{array}\right]}^T.$$

(8)

Note that the above population covers state–action space and fitness value, while the action has to be optimized. Hence, we propose to employ the regular evolutionary operations, i.e., recombination and mutation, solely to the actions, while updating the state vector via the regular system dynamics.

5.2.1. Recombination

To maintain genetic diversity and enhance robustness, individuals are recombined. This mechanism is addressed in [37] by

$$a_{l+k}(t+1)={\alpha }_k{a}_i\left(t\right)+(1-{\alpha }_k)a_j\left(t\right),k=1\dots K,$$

(9)

where the different individual’s actions $a_i\left(t\right)$ and $a_j\left(t\right)$ are sampled from the population and recombined to form the action $a_{l+k}(t+1)$, ${\alpha }_k$ is a random parameter, and K is the number of recombined individuals.

5.2.2. Mutation

To explore novel solutions, random mutations are introduced to expand the search space. This process supports the exploration of new strategies and is described by

$$a_{l+K+m}(t+1)=a_i\left(t\right)+ϵ\left(t\right),m=1\dots M$$

(10)

with random value $ϵ$ and number of mutated individuals M. This genetic adaptation enables individuals to expand the search space.

5.2.3. Update of Support Vectors

In addition to the action update, we have to consider the update of the support vector. In theory, we can also use recombination and mutation. However, this would ignore the system dynamics. Hence, we opt to simply set the new support vector to be equal to $s_{t+1}^{A_i}$, i.e., the state obtained after applying $a_i\left(t\right)$ to the system. Hence, we obtain the new individual ${\overrightarrow{I}}_{new}$ as

$${\overrightarrow{I}}_{new}(s_{t+1}^{A_i},a_{i_{t+1}},{\psi }_{t+1}).$$

(11)

where ${\psi }_{t+1}$ is the achieved fitness.

5.3. Local and Global Selection

The above updates result in an increased population which has to be reduced to its original size. To this end, we have to adress two goals, namely to select individuals with the best fitness but at the same time preserving the grid coverage. To this end, we propose a local selection for coverage preservation followed by a standard global selection.

5.3.1. Local Selection

The task of local selection is to preserve the grid’s coverage of the state space. Let

$$conv{\overrightarrow{P}}_i^t:=\bigcap _{{\overrightarrow{P}}_i^t}{s}^{A_i}.$$

(12)

denote the convex hull of the grid, i.e., the smallest convex shape that encloses all points. Then, we define two distinct cases for the population update:

$${\overrightarrow{P}}_i^t\stackrel{{\overrightarrow{I}}_{new}}{\to }{\overrightarrow{P}}_i^{t+1}=\left\{\begin{array}{cc}\mathrm{update}(s_{t+1}^{A_i}),\hfill & \mathrm{if}{s}_{t+1}^{A_i}\in conv{\overrightarrow{P}}_i^t\mathrm{or}\hfill \\ \mathrm{add}(s_{t+1}^{A_i}),\hfill & \mathrm{if}{s}_{t+1}^{A_i}\notin conv{\overrightarrow{P}}_i^t\hfill \end{array}\right.$$

(13)

Note that the property $s_{t+1}^{A_i}\in conv{\overrightarrow{P}}_i^t$ can be easily determined by applying the QuickHill algorithm [54].

In case of add($s_{t+1}^{A_i}$), the individual ${\overrightarrow{I}}_{new}$ is simply added to the population. The case of update($s_{t+1}^{A_i}$) is more involved as the corresponding support vector is already present in the coverage of the state–action representation. To avoid an accumulation of support vectors within a small area, but make use of the obtained fitness update, we propose a local update step of the grid.

To this end, we evaluate whether adding the new individual ${\overrightarrow{I}}_{new}$ improves the local fitness of the state–action representation ${\Psi }_i$ at position $s_{t+1}^{A_i}$ by the fitness value ${\psi }_{t+1}$. Since the comparable fitness ${\psi }_t$ at the point of interest may not correspond directly to an individual in the population ${\overrightarrow{P}}_i^t$, we estimate the fitness value using Delaunay triangulation $DT\left({\overrightarrow{P}}_i^t\right)$ of the three nearest support vectors. We use Delaunay triangulation to divide a set of points into triangles so that no point lies within the perimeter of a triangle [55]. This maximizes the smallest angles of the triangles and thus avoids very pointed, numerically unfavorable shapes. For a comprehensive derivation of the calculations, see Appendix A.

The calculated fitness of the state–action representation is used to decide on exploitation or a rejection of the information with

$${\overrightarrow{P}}_i^t\stackrel{{\overrightarrow{I}}_{new}}{\to }{\overrightarrow{P}}_i^{t+1}=\left\{\begin{array}{cc}\mathrm{add}\left(\right),\hfill & \mathrm{if}{\psi }_t\le {\psi }_{t+1}\hfill \\ \mathrm{reject}\left(\right).\hfill & \mathrm{else}\hfill \end{array}\right.$$

(14)

Consequently, a new individual is only added to the population, if its fitness is better compared to a virtual fitness calculated from neighboring support vectors. We denote this as local selection.

5.3.2. Global Selection

As the size of the population after local selection typically exceeds the maximum number of individuals, we use a standard selection approach in that we select individuals based on their fitness:

$$\mathrm{Top}\left(n_{\nu }\right)({\overrightarrow{I}}_{A_i}^t,{\Psi }_i).$$

(15)

where $\mathrm{Top}\left(n_{\nu }\right)$ denotes the selection of the $n_{\nu }$ individuals with the best fitness.

5.4. Agent-Based Knowledge Transfer

As stated before, the training process can be enhanced using knowledge transfer between agents, i.e., agents exhibiting similar behavior can share information. The aim is to enable faster learning with minimal training requirements by utilising existing knowledge to improve performance, see Figure 5.

To this end, we have to solve two basic issues. First, we have to derive metrics, which determine the degree of similarity between the agent’s state–action representations. Second, we leverage our population-based learning approach to transfer knowledge between the agents.

In order to transfer knowledge, we use the individual’s representation consisting of support vector, action and fitness as a normalized feature vector on which we define a similarity metric as follows:

$$d(I_i,I_j)=\sqrt{{\displaystyle \sum _{i=1}^d}{(I_i-I_j)}^2}.$$

(16)

Hence, the knowledge transfer takes place between agents i and j with the most consistent match according to ${min}_{I_i\in {\overrightarrow{P}}_i}d(I_i,I_j)$.

A reasonable knowledge transfer can only be achieved if integrating the individual $I_i$ improves the fitness value ${\psi }_j$. This transfer mechanism ${\mathcal{T}}_{i,j}:I_i,I_j\to \mathbb{R}$ is described by

$${\mathcal{T}}_{i,j}=\{I_i\in P_i|{\psi }_j\left(I_j\right)<{\psi }_j(I_j\cup \left\{I\right\}\}.$$

(17)

Hence, only the individual I is selected for transfer if integrating it into the target state–action representation $F_j$ improves the fitness value ${\psi }_j$. By requiring a strictly positive effect on target fitness, we ensure that source knowledge acts as a catalyst rather than a source of interference in the sensitive early stages of adaptation.

Technical implementations of the update state–action representation by DGS, see Algorithm A1, the agent-based evolutionary process, see Algorithm A2, and an excerpt of the multi-agent knowledge transfer, see Algorithm A3, are provided as pseudocode in Appendix B.

6. Convergence Analysis

In this section, we discuss the convergence properties of the proposed Evo-SbPG based learning algorithms. To this end, we first recall results on the convergence of conventional SbPG. In fact, it has been shown in [26] that SbPG converge to a local equilibrium if the learning algorithm follows a best response or gradient-based learning scheme. This is due to the symmetric Hessian of the underlying game structure. Hence, if we can prove a best response behavior of our proposed learning scheme, this yields convergence proofs of the EVO-SbPG. To this end, we have to analyze the analysis of the Evo-SbPG learning using the global selection, for which we can state the following:

Theorem 1.

Let ${\psi }_{max}$ and ${\psi }_{min}$ denote the maximum and minimum fitness value across the population. Then, under the global selection scheme, we have

$$\begin{array}{cc}\hfill {\psi }_{max}(t+1)& \ge {\psi }_{max}\left(t\right),\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill {\psi }_{min}(t+1)& \ge {\psi }_{min}\left(t\right).\hfill \end{array}$$

(19)

Proof. 

Recall that we select the new population $P_{t+1}$ of size $n_{\nu }$ out of the old population $P_t$ of size $n_{\nu }$ plus additional individuals collected in population $P_t^{*}$ of size $n_{\nu }^{*}$ obtained by recombination and mutation, local selection and knowledge transfer. To this end, we employ the TopK sampling, i.e., we select the best $n_{\nu }$ individuals.

Suppose we have $max\left\{{\psi }_I\left(t\right)|I\in P_t^{*}\right\}\le {\psi }_{min}\left(t\right)$, then none of the additional individuals are selected yielding ${\psi }_{min}(t+1)={\psi }_{min}\left(t\right)$. Otherwise, if $max\left\{{\psi }_I\left(t\right)|I\in P_t^{*}\right\}>{\psi }_{min}\left(t\right)$, at least one of the additional individuals will replace the worse individual, yielding ${\psi }_{min}(t+1)>{\psi }_{min}\left(t\right)$. The same arguments hold also for ${\psi }_{max}(t+1)$, concluding the proof.    □

The above theorem basically proves that the fitness only increases or remains constant during an update. This is essentially a best response learning scheme. This remains valid under the TL scheme as well as the local selection as both just add additional individuals. Consequently, we can state the following convergence behavior.

Theorem 2.

Given an Evo-SbPG with correspondingly chosen fitness functions and the above EA-based learning scheme, the algorithm converges to a local Nash equilibrium.

Proof. 

This follows directly from the underlying Evo-SbPG structure and the fact [26] that the learning scheme represents a best response strategy.    □

7. Results and Discussion

In this section, the proposed approaches and developed strategies undergo thorough evaluation and critical analysis conducted on a distributed production environment. To this end, after introducing the testbed, we first compare the novel approach in different settings with other state-of-the-art methods. We then present the actor policies followed by a discussion of the coverage of the action space.

7.1. Implementation in a Laboratory Test Field

The Bulk Good Laboratory Process (BGLP) is a flexible, intelligent production system that specializes in the continuous transport of bulk materials shown in Figure 6.

The process takes place across three independent operating modules. Station 1 is a preparation unit where material is distributed via a conveyor belt. Station 2 uses a vibrating conveyor for processing. Station 3 uses a rotary valve for continuous dosing of the material. In addition, level sensors are used to monitor the process.

For the considered scenario, we assign an agent to each actuator, i.e., the conveyor, vibration conveyor, two vacuum pumps and rotary feeder. All actuators have a normalized action space of $[0,1]$ except for the on–off vibration conveyor. The state space of each agent consist of the fill levels of the upstream and downstream buffer respectively. We further define the local fitness function ${\Psi }_i$ for each agent i using the overflow and emptiness of the upstream and downstream buffers $L_p^i$ and $L_s^i$, the energy consumption $P^i$, and the production targets $V_D$ as

$${\Psi }_i={\displaystyle \frac{1}{1+{\alpha }_L{L}_p^i}}+{\mathbb{1}}_{i\ne N}{\displaystyle \frac{1}{1+{\alpha }_L{L}_s^i}}+{\mathbb{1}}_{i=N}{\displaystyle \frac{1}{1-{\alpha }_D{V}_D}}+{\displaystyle \frac{1}{1+{\alpha }_P{P}^i}}.$$

(20)

with weighing factors ${\alpha }_L$, ${\alpha }_D$ and ${\alpha }_P$. The distributed production process iterates with a sequence T, where the fill level of the individual silos and hoppers are covered by $h^N$. The overflow and emptiness of the buffers are calculated by

$$\begin{array}{cc}\hfill L_s^i={\int }_0^T{\mathbb{1}}_{h_s^i>H_s^i}\left(h_s^i\right)dt,& L_p^i={\int }_0^T{\mathbb{1}}_{h_p^i>H_p^i}\left(h_p^i\right)dt,\hfill \end{array}$$

(21)

with upper limit $H_s^i$ and lower limit $H_p^i$. The production demand is calculated with

$$V_D={\int }_0^T{\dot{D}}_{t}dt,\mathrm{with}{\dot{D}}_t\left\{\begin{array}{cc}{\dot{V}}_{N,out}-{\dot{V}}_{N,in},\hfill & \mathrm{if}{h}^N=0\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(22)

where the inflow ${\dot{V}}_{N,in}$ and the outflow ${\dot{V}}_{N,out}$ is revered to the station.

7.2. Results at the BGLP

We evaluate the strategies of the Evo-SbPG using the BGLP process. To this end, we compare the results of DGS, EA DGS, and EA DGS Trans with those of the Vanilla SbPG [15] and GB SbPG M1 and GB SbPG M2 approaches [19]. Both the Vanilla SbPG and GB SbPG approach are based on fixed grid structures. However, the Vanilla SbPG approach uses random action generation, whereas the GB SbPG approach uses gradient optimization methods. GB SbPG M1 is based on a single-leader–follower dynamic, and GB SbPG M2 with multiple leader–follower relationships. The described approaches were carried out with the proposed parameters from the literature. While the DGS approach only uses dynamic grid structures, the EA DGS approach combines dynamic representation with evolutionary strategies. The EA DGS Trans approach additionally combines both with knowledge transfer. An excerpt from the fitness values of the local agents and the global resulting objective are shown in

Table 1. Average fitness rating of each agent.

Agent	Vanilla SbPG	GB SbPG M1	GB SbPG M2	DGS	EA DGS	EA DGS Trans
Agent 1	2.152	2.610	2.703	2.394	2.228	2.404
Agent 2	1.892	2.635	2.544	2.285	2.012	2.317
Agent 3	1.944	2.207	2.109	2.304	2.199	2.324
Agent 4	1.561	1.467	1.711	2.052	2.390	2.108
Agent 5	1.127	1.171	1.319	1.422	1.746	1.476
Total	8.675	10.089	10.387	10.456	10.574	10.629

.

The contributions of each agent to global fitness are presented in Figure 7.

Figure 7 shows that the different variants of our approach, i.e., DGS, EA DGS, and EA DGS Trans outperform Vanilla SbPG and gradient based approaches with EA DGS Trans performing best. Further, they yield strategies where the individual agents contribute more evenly to the overall objective. This indicates improved coordination between individual agent policies.

We further provide results for the individual objectives for transport, overflow, demand and power consumption as shown in

Table 2. Average process parameters per strategy.

Total	Vanilla SbPG	GB SbPG M1	GB SbPG M2	DGS	EA DGS	EA DGS Trans
transport	0.984	0.451	0.475	0.854	0.957	0.884
overflow	0.034	0.000	0.000	0.018	0.009	0.017
demand	127.130	129.850	130.793	118.448	115.027	116.623
power	0.726	0.440	0.446	0.652	0.700	0.669

.

Although all approaches enable control of the BGLP process, significant differences in their individual process characteristics become apparent. The best response strategies Vanilla SbPG, DGS, EA DGS, and EA DGS Trans exhibit higher transport rates, whereas GB SbPG M1 and GB SbPG M2 show lower performance in this regard while requiring lower energy consumption.

A key feature of the DGS, EA DGS, and EA DGS Trans approaches is highlighted in the time plots shown in Figure 8. All the established strategies Vanilla SbPG, GB SbPG M1, and GB SbPG M2 are primarily focused on identifying and maintaining a single optimal control point while the effort required to establish or maintain control points varies depending on the strategy.

However, these results do not reflect typical control behavior in such event-driven distributed control systems which are typically characterized by on–off control behavior. As shown, the novel approaches are more aligned with such realistic plant control behavior, as they support dynamic adaptation to fluctuating system conditions. This adaptive behavior is particularly evident in the transport, overflow, and power profiles of the DGS, EA DGS, and EA DGS Trans strategies, as illustrated in Figure 8. We advacate this to the better exploration of the state–action space and improved coverage due to the DGS.

This interpretation is confirmed by an extended breakdown of the individual process objectives with the individual agents at the BGLP, see

Table 3. Average process parameters for each agent and production target at the BGLP.

		Vanilla SbPG	GB SbPG M1	GB SbPG M2	DGS	EA DGS	EA DGS Trans
Agent 1	transport	0.191	0.080	0.089	0.172	0.186	0.177
	overflow	0.000	0.000	0.000	0.000	0.000	0.000
	demand	21.008	21.019	21.195	21.095	21.024	21.040
	power	0.045	0.040	0.041	0.044	0.045	0.045
Agent 2	transport	0.228	0.080	0.090	0.184	0.203	0.192
	overflow	0.001	0.000	0.000	0.001	0.000	0.001
	demand	30.094	22.540	24.081	27.213	30.303	27.994
	power	0.162	0.077	0.082	0.137	0.148	0.141
Agent 3	transport	0.200	0.080	0.089	0.171	0.212	0.179
	overflow	0.007	0.000	0.000	0.004	0.000	0.003
	demand	19.686	23.030	24.953	17.282	25.087	17.039
	power	0.027	0.027	0.027	0.027	0.027	0.027
Agent 4	transport	0.168	0.115	0.109	0.159	0.178	0.164
	overflow	0.025	0.000	0.000	0.012	0.009	0.012
	demand	24.444	32.021	31.285	24.895	23.655	24.864
	power	0.251	0.179	0.174	0.240	0.263	0.247
Agent 5	transport	0.197	0.095	0.099	0.167	0.177	0.172
	overflow	0.000	0.000	0.000	0.000	0.000	0.000
	demand	31.898	31.240	29.279	27.963	14.960	25.686
	power	0.241	0.116	0.122	0.205	0.218	0.210

.

While Vanilla SbPG and DGS, EA DGS and EA DGS Trans strategies focus on self-optimizing state–action representations, GB SbPG M1 and GB SbPG M2 pursue goal-oriented optimization. This leads to higher transport values by the Vanilla SbPG and DGS methods, whereas GB-SbPGM1 and GB SbPG M2 approaches result in lower transport performance. Similar behavior is reflected in the material overflow characteristics. Vanilla SbPG and DGS tolerate minor overflow to prioritize overall optimization, whereas GB SbPG M1 and GB SbPG M2 aim to minimize excess. The Vanilla SbPG and DGS strategies show positive lower demand than GB-SbPGM1 and GB SbPG M2. Power consumption is inversely proportional to demand, while Vanilla SbPG and DGS use a higher power supply with their strategies.

7.3. Analysis and Interpretation

Based on the parameters recorded at the BGLP, we perform a Kernel Density Estimation (KDE) as a violin plot. For statistical classification, we also include the Interquartile Range (IQR) by the quartiles of $25\%$ and $75\%$, the mean value $\mu$, and the single and double standard deviations $1\sigma$ and $2\sigma$ in the representation, see Figure 9.

The recorded values from the BGLP process are treated as independent and identically distributed samples. To estimate the univariate probability density function $\widehat{f}$ at a given point x from a recorded dataset $\{x_1,x_2,\dots ,x_n\}$, we employ a KDE defined by

$$\widehat{f}\left(x\right)={\displaystyle \frac{1}{nh}}\sum _{i=1}^{n}K\left({\displaystyle \frac{x-x_i}{h}}\right),$$

(23)

where n denotes the number of data points and h represents the bandwidth. As kernel $K(·)$, we used a Gaussian Kernel with

$$K\left(u\right)={\displaystyle \frac{1}{\sqrt{2\pi }}}{e}^{-{\displaystyle \frac{1}{2}}{u}^2}.$$

(24)

Other potential kernel functions include uniform, triangular, biweight, or triweight kernels. For our evaluation, we determine the bandwidth h by scaling Scott’s Rule with a factor $bw$ to minimize the mean squared error, defined as

$$h=bw·h_{Scott}=bw·\sigma n^{-1/5}.$$

(25)

While the bandwidth could alternatively be determined via Silverman’s Rule or manual adjustment, we selected a scaling factor of $bw=0.3$. Mathematical smoothing of the KDE can attenuate or exaggerate outliers in the data points, creating a continuous distribution shape that goes beyond the individual data values. In addition to the $25\%$ and $75\%$ IQR quartiles shown in red, the upper and lower whisker limits are estimated and plotted using $1.5$ times the IQR.

7.4. Visualization of the State–Action Representation

In addition to the performance improvements, we are also interested in the coverage of the state–action representation of the DGS. To this end, we visualize the state–action representation by the DGS in Figure 10. Specifically, we show the barchart of action values, a bird view of the state space DGS, the resulting action grid, and the fitness surface of each agent. Together, these representations support both qualitative and quantitative analysis.

First, the DGS covers the state space entirely for agents 2–4, indicating the strength of the EA-based optimization which allows us to explore the state space with maximum coverage. The reduced coverage of agent 1 and 5 are due to the process conditions. Specifically, the rotary feeder is directly responsible for demand fulfilment, restricting the state space while the conveyor belt is typically only operated for smaller input buffer values. Further, we observe comparably smooth fitness surfaces indicating a certain optimal region within the state space which align with the action surface as well. Overall, our approach succeed in coverageging the full state–action space according to the optimization objectives resulting in the better usage of actuation as discussed before.

7.5. Tuning and Configuration Parameters

The reproducibility of the investigations is secured through strictly controlled study conditions, enabling direct comparability with reference approaches. The BGLP operates with an iteration cycle of $T_I=10$ s, allowing for the comprehensive capture of system dynamics via a discretized parameter sampling rate of $T_S=0.5$ s.

We set up the same episodic training procedure for each experiment. This procedure contains nine training episodes and one testing episode. During training, the DGS were continuously adapted to the observed behavior and adaptive learning strategies of the agents.

The fitness function, which defines the objective of agent training, was specified by weighting factors of the upstream and downstream buffer ${\alpha }_L=1$, the production demand target ${\alpha }_D=4$, and energy consumption ${\alpha }_P=0.001$.

The parameters and system settings presented and used are methodologically and numerically directly comparable with the configurations of the reference approaches Vanilla SbPG [15] and GB SbPG M1 and GB SbPG M2 approaches [19].

7.6. Scalability, Transferability and Limitations

The results demonstrate that agents formed autonomously interact both cooperatively and competitively, learning from one another. The Evo-SbPG agent architecture, developed for this study, enables parallel operation and linear scaling by proportionally expanding comparable modules. Although tested in a specific setting, the approach’s underlying mechanisms are transferable to other distributed production systems. We refer to [17] for a more detailed discussion. A limitation of the approach is its inherent reliance on the availability of the full state vector, which might be restrictive in real-world applications. Nevertheless, incorporating corresponding state observations or soft sensors might mitigate this problem. We leave an extension to partly observable process to future research. Furthermore, the results presented are based on simulation models in lab environments which might suffer from the sim2real gap. Approaches to account for the sim2real gap will be part of future research as well. Thanks to its stable core logic, this approach can also be applied to similar decentralized processes including dynamic supply chain management, traffic control, coordination of autonomous vehicles and voltage and frequency stabilization in smart grids.

8. Conclusions

In this paper, we proposed a novel approach for distributed optimization using population-based learning and DGS. To this end, we present Evo-SbPG, a new game theory-based framework that integrates EA strategies in SbPG. Further, we address the complexity of dynamic environments and extend the approach by DGS, EA DGS and EA DGS Trans strategies for present dynamic state–action representations. The strategies are evaluated in order to solve multi-objective optimization problems in distributed production systems. The results demonstrate improved performance in state–action representation and can be adapted for various applications. By leveraging, the approach ensures high computational efficiency and scalability, allowing it to handle increasingly complex strategy spaces without the exponential cost typically associated with fine-grained discretization.

Future research will exploit the insights gained from DGS transitions to enhance state–action representation in more intricate coupled systems. Additionally, we will further explore and extend the foundations of knowledge transfer in MAS systems and its representation through graph theory, developing more comprehensive approaches in the process. Specifically, we aim to investigate knowledge transfer mechanisms in multi-objective Evo-games, such as knowledge distillation as well as the adaptability of DGS in non-stationary environments.