Chaos Engineering for Resilient Manufacturing: A Digital Twin Perspective

Abstract

Chaos engineering is currently being used by some of the largest software companies in the world to develop IT systems that can withstand turbulent production conditions. This study presents an approach for applying chaos engineering across manufacturing systems to improve system resilience. For this purpose, the proposed chaos twin framework includes a four-step process for the manufacturing and interlinked digital twin system. A case study is discussed based on a material flow simulation. The case study shows the example implementation of the chaos twin framework for improving the resilience of a manufacturing system. The research intends to enhance the field of designing and operating manufacturing systems by demonstrating novel perspectives on applying chaos engineering within the cyberspace. Additionally, the research intends to provide a more industry-oriented guide to facilitate the practical application of chaos engineering within the manufacturing industry.

1. Introduction

1.1. Rationale

Industry 5.0 ecosystems can be characterised by human-centric, resilient, and sustainable value creation [1,2] and are, in addition to the so-called twin, green, and digital transition [3], sometimes described as the future industrial paradigm for supporting the realisation of the United Nations Agenda 2030 [4]. Within the Industry 5.0 concept, system resilience is defined as a key characteristic of manufacturing systems [5,6]. As a method to test how systems respond to stress, chaos engineering is used by some of the largest software companies in the world, such as Netflix, Amazon, Google, Facebook, LinkedIn, and Microsoft [7]. Chaos engineering comprises approaches that aim to experiment with an operational system “[…] to build confidence in the system’s capability to withstand turbulent conditions in production” [8]. Netflix seems to be one of the first movers in chaos engineering when it started developing an initiative named Chaos Monkey [9], which is evidenced to be a quite successful initiative across Netflix [10].

This approach seemed counterintuitive at first, as failures are purposefully injected into a live production network system [11]. An analogy used to describe chaos engineering is to compare it with the purpose of a vaccine: “[…] we inject harm into our system to build an immunity” [12], implying that there will never be immunity to all failures, simply because it is impossible to be aware of all possible failures in a complex system. Even though the application of chaos engineering has a demonstrated track record in the software industry for building resilient production systems, it has hardly been applied to the manufacturing domain, although some authors have started to explore its potential for manufacturing, as described in [13,14].

Against this backdrop, this study aims to investigate how the methodology and principles of chaos engineering can be used to create value in the context of manufacturing systems primarily by improving the resilience of the system and using digital twin technology.

1.2. Theoretical Background

Chaos engineering is “[…] the discipline of experimenting on a system to build confidence in the system’s capability to withstand turbulent conditions in production” [8]. It can be utilised to identify a steady state, which can be characterised as the normal behaviour of the system in operation [8,15]. Subsequently, faults are injected to stress the system and to evaluate the system’s resilience under real operational conditions [15,16]. Chaos engineering can typically be performed in four steps (

Table 1. Steps for experimentation in chaos engineering.

Step	Following the Idea from [17]	Following the Idea from [8]
1	Hypothesis	Define the steady state
2	Testing	Define the hypothesis
3	Blast Radius	Introduces variables
4	Insights	Disprove hypothesis

).

Poltronieri et al. try to mitigate that chaos engineering can be an expensive practice, with high setup and operations costs, by suggesting using a digital twin of an IT service for chaos engineering [15]. This so-called ChaosTwin allows the conduction of experiments within a safer environment, which can minimise related costs and provide useful feedback at the system design stage [18].

1.3. Research Hypothesis and Research Questions

The underlying main research hypothesis (RH) for the study is:

RH: Chaos engineering in manufacturing systems will only be pursued by manufacturing companies if the consequences of the experimentation have no negative impact on, but adequately reflect, the actual behaviour of the real-world manufacturing system.

This means resilience is the capacity of a manufacturing system to return to a competitive state “[…] by preparing for, responding to, and recovering from internal or external and known or unknown disturbances” [19]. Currently, emerging industry standards for utilising digital twin technology can support the implementation of chaos engineering completely in cyberspace. The cyberspace allows for safe experimentation without harming physical assets through the intentional induction of failures, disruptions, or other disaster events. Hence, chaos engineering and the learning generated from its application can help manufacturing companies to improve their readiness for Industry 4.0 and Industry 5.0. The context of the research’s main ideas and concepts is summarised in Figure 1.

The following research question (RQ) including sub-research questions (SRQs) are formulated for this study:

RQ: How can chaos engineering principles be applied to create value for the design and operation of manufacturing systems?

• SRQ1: What are the important process phases for applying chaos engineering in a manufacturing cyberspace?
• SRQ2: How can the digital twin technology in combination with simulation models support the application of chaos engineering in manufacturing systems?

1.4. Research Methodology

For pursuing RQ and SRQs, the research methodology follows a mostly qualitative research approach, which can be structured into three main phases (Figure 2).

A narrative literature review is used to describe the state-of-the-art for the field of chaos engineering. The literature review (Section 2) serves to present an overview of the current state of academic and industrial research and the application for the scope of this study. It further aims to place the research within literature by highlighting the research gaps.

For framework development (Section 3), the underlying hypothesis is that through the increasing dissemination of digital twins across the manufacturing industry, chaos engineering can be safely executed in a digital twin environment without causing harm to the physical system, i.e., the hardware, such as machine tools. The chaos twin framework provides a process model with four defined steps for the application of chaos engineering. The case study (Section 4) serves as an example implementation of the chaos twin framework for improving the resilience of a simple manufacturing system, which helps to test and verify the chaos twin framework.

2. Literature Review

2.1. Method

The literature review follows a narrative approach covering recent publications within the scope of this study. This scope covers:

• Digital twins in manufacturing, since chaos engineering for manufacturing will be incorporated into the digital twin which allows the execution of chaos experiments in cyberspace without interfering with the actual hardware of the manufacturing system.
• Resilience in manufacturing, since resilience is an important target of Industry 4.0/5.0 where chaos engineering is expected to have a large impact.
• Chaos engineering in manufacturing, since this will be the specific application case for chaos engineering in this study.

2.2. Digital Twins in Manufacturing

For many years, digital twins have been discussed by academics and industry actors as one of the key enabling technologies to support the implementation of the Industry 4.0 paradigm across the manufacturing industry. This leads to the application of more digital twins throughout manufacturing systems [20]. Digital twins are characterised by offering digital artifacts for managing the lifecycle data of manufacturing assets. Manufacturing assets in this sense can be products, machine tools and other manufacturing equipment, software services, production cells and lines, or even whole factories and value networks [21]. The trend of increasingly available digital twins allows for utilising the lifecycle data of connected assets for conducting experiments across the manufacturing system in cyberspace.

Table 2 highlights current studies on the application of digital twin technologies and standardisation trends in a broader manufacturing context. Standardisation trends for an interoperable digital twin are promoted, for example, by the Industrial Digital Twin Association (IDTA) as well as by international standards, i.e., ISO 23247-1:2021 [22], ISO/IEC 20924:2024 [23], and ISO/IEC 30173:2023 [24].

The IDTA seems to be one of the relevant industry-driven organisations currently developing technologies and industry standards for the application of the digital twin across the manufacturing industry, with more than 100 member companies [25]. The key technology promoted by the IDTA is the so-called Asset Administration Shell (AAS), the digital twin in Industry 4.0 and manufacturing [26]. Also, in academic research, the AAS technology is widely known for describing digital twins or aspects of digital twins in manufacturing systems, e.g., as demonstrated by [27,28,29,30].

Table 2. Some relevant studies for digital twins in manufacturing.

Study	Focus on Application or Standardisation
Liu et al. present a more general survey on the current status of implementing digital twins [31].	Application: Classification of digital twin implementations covering technologies and features.
He and Bai provide a review of the digital-twin technology supporting sustainable and intelligent manufacturing [32].	Application: Intelligent Manufacturing with digital twins allowing intelligent sensing and simulation.
Nee and Ong present a compilation of works on digital twin applications across the industry [33].	Application: Compilation of studies which discuss different applications, spanning from systems levels to the level of specific processes and products.
Arm et al. describe the automated design of the AAS standard for implementing the digital twin [34].	Standardisation: Introduction of a configuration wizard for AAS creation in Industry 4.0.
Lu et al. discuss the model-based definition of the AAS for manufacturing assets [35].	Standardisation: Introduces the Model-Based Definition (MBD)-assisted digital twin for designing Industry 4.0 production lines based on the AAS standard.
Gregory et al. discuss a model-based systems engineering approach to support the engineering of standardised digital twins [36].	Standardisation: Introduction of requirements and functions for developing digital twins while considering relevant ISO, IEC, IEEE standards.
Belfadel et al. propose an open platform design for digital twin standards [37].	Standardisation: Introduction of an architecture and digital twin platform aligned with IEC and AAS standards.
The Industrial Internet Consortium Standards Task Group presents the best practice paper on Global Industry Standards for Industrial IoT [38].	Standardisation: Provides an overview of global standardisation activities in the context of digital twins in an industry environment.
The Plattform Industry 4.0 and the IDTA present a digital twin reference model based on the AAS standard [39].	Standardisation: Highlights the industry standard for digital twins, namely the AAS, in the German Industry 4.0 context.

Table 2. Some relevant studies for digital twins in manufacturing.

Study	Focus on Application or Standardisation
Liu et al. present a more general survey on the current status of implementing digital twins [31].	Application: Classification of digital twin implementations covering technologies and features.
He and Bai provide a review of the digital-twin technology supporting sustainable and intelligent manufacturing [32].	Application: Intelligent Manufacturing with digital twins allowing intelligent sensing and simulation.
Nee and Ong present a compilation of works on digital twin applications across the industry [33].	Application: Compilation of studies which discuss different applications, spanning from systems levels to the level of specific processes and products.
Arm et al. describe the automated design of the AAS standard for implementing the digital twin [34].	Standardisation: Introduction of a configuration wizard for AAS creation in Industry 4.0.
Lu et al. discuss the model-based definition of the AAS for manufacturing assets [35].	Standardisation: Introduces the Model-Based Definition (MBD)-assisted digital twin for designing Industry 4.0 production lines based on the AAS standard.
Gregory et al. discuss a model-based systems engineering approach to support the engineering of standardised digital twins [36].	Standardisation: Introduction of requirements and functions for developing digital twins while considering relevant ISO, IEC, IEEE standards.
Belfadel et al. propose an open platform design for digital twin standards [37].	Standardisation: Introduction of an architecture and digital twin platform aligned with IEC and AAS standards.
The Industrial Internet Consortium Standards Task Group presents the best practice paper on Global Industry Standards for Industrial IoT [38].	Standardisation: Provides an overview of global standardisation activities in the context of digital twins in an industry environment.
The Plattform Industry 4.0 and the IDTA present a digital twin reference model based on the AAS standard [39].	Standardisation: Highlights the industry standard for digital twins, namely the AAS, in the German Industry 4.0 context.

2.3. Resilience in Manufacturing

Resilience is the capacity of a manufacturing system to return to a competitive state “[…] by preparing for, responding to, and recovering from internal or external and known or unknown disturbances” [19]. Improving and maintaining resilience is an important target for modern manufacturing systems in Industry 4.0/5.0. Academics are discussing different frameworks for this purpose, focusing, for example, on strategies, optimisation, the human operator, and systems engineering (

Table 3. Some relevant studies for resilience in manufacturing.

Study	Focus of Frameworks
Neumann et al. discuss a framework for supporting resilience in the electrical manufacturing industry [19].	Strategies: 110 resilience patterns for supporting value creation
El-Halwagi et al. propose a framework for the disaster-resilient design of manufacturing facilities by using process integration [40].	Strategies: 12 principal strategies for disaster-resilient design
Feng et al. present a time-based resilience metric for smart manufacturing systems [41].	Optimisation: Time-based resilience metric and solution framework in the context of job shop scheduling.
Romero and Stahre propose a framework for the resilient operator in Industry 5.0 [42].	Human: Operator 5.0 concept with humans to support system resilience.
Zhang et al. present uncertainty management for the shopfloor based on digital twin technology [43].	Strategy and optimisation: Reconfiguration design method for shopfloors and based on digital twins with 16 steps and four phases.
Mousavi et al. discuss a model-based systems engineering (MBSE) framework for the disruption analysis of supply chains [44].	Systems Engineering: Application of MBSE methodology to disruption management of a supply chain and application of MBSE tools for problem definition.
Hossain et al. discuss the role of systems engineering attributes in enhancing supply chain resilience [45].	Systems Engineering: Conceptual model for supply chain resilience for four phases of resilience.

).

Current frameworks in the field of resilience might support the future application of chaos engineering in the manufacturing domain, as these frameworks enable the modelling and analysis of complex and dynamic manufacturing systems. Especially, analysing and understanding the interrelation of the technical system with the product and product lifecycle, fabrication and assembly processes, organisation, human, value network, business model (following the structure from [46]) and its impact on the system resilience might be essential for manufacturing systems in an increasingly uncertain environment.

2.4. Chaos Engineering in Manufacturing

Chaos Engineering in manufacturing focuses on reducing the impacts of unknown and unimagined failures, disruptions, and disasters, i.e., disturbances, during all lifecycle phases of manufacturing systems. In general, chaos engineering seems rather underexplored in academic literature and industrial practice.

Table 4. Some relevant studies for chaos engineering in manufacturing.

Study	Focus of Application
Doan et al. derive requirements and show an approach for applying chaos engineering to improve the resilience of manufacturing systems [14].	Manufacturing Systems
Dave discusses a chaos engineering platform as a part of Site Reliability Engineering to improve efficiency and downtime in manufacturing operations [47].	Manufacturing System
Konstantinou et al. discuss the use of chaos engineering for supporting resilience in cyber-physical systems [13].	Cyber-physical system
Kalka and Szydlo present a chaos engineering tool for IoT devices to test types of failures and failure scenarios [48].	Cyber-physical system
Fogli et al. describe chaos engineering for resilience assessment of digital twins [49].	Digital twin system
Poltronieri et al. published chaos engineering and chaos twin approaches for improving resilience in IT services [15,18].	Service system
Akuthota outlines the application perspective of chaos engineering for microservices [50].	Service system
Dedousis et al. discuss the support of operational resilience of critical infrastructure using chaos engineering [51].	Critical infrastructure system
Naqvi at el. highlight the evaluation of self-adaptive and self-healing capabilities of systems using chaos engineering [52].	Variety of different systems
Baily et al. present a chaos engineering approach for measuring resilience across system-of-systems [53].	System-of-Systems

highlights some relevant studies that discuss the application of chaos engineering for different types of systems. Manufacturing approaches for using chaos engineering as a part of site reliability engineering [47] as well as for supporting resilience [14] are discussed. Other authors describe the application of chaos engineering for cyber-physical, digital twin, service, and critical infrastructure systems. Further studies discuss the implementation of chaos engineering for a variety of systems and system-of-systems. Some of the currently published approaches seem to focus on using chaos engineering for assessment purposes rather than for supporting the synthesis and design phases of systems.

2.5. Research Gap

Specific frameworks for utilising chaos engineering to design and continuously improve manufacturing systems seem to be insufficiently discussed. Doan et al. just published requirements and a process model for the application of chaos engineering in a learning factory environment to reduce the risk of experimentation, while also recommending the use of digital twins as an alternative approach to risk reduction [14]. Hence, the application of chaos engineering within manufacturing systems using the concept of digital twins is yet to be explored in sufficient detail. In the context of chaos engineering and digital twins, the development of a theoretical framework seems to be relevant to provide some initial guidance on implementing chaos engineering across manufacturing systems such as complex Industry 4.0/5.0 factories.

Running chaos experiments by purposely inducing failures in manufacturing systems is time-consuming and costly, and can directly impact the safety, security, and quality of fabrication and assembly processes. As a result, the disadvantages might often outweigh the potential advantages of chaos experimentation. The increasing utilisation of cyber-physical systems with standardised digital twins might become a game changer in the future since digital twin technology allows for experimentation completely in cyberspace and essentially reduces the barriers and potential negative impacts of chaos engineering in manufacturing.

3. Chaos Twin Framework

3.1. Framework Introduction

The chaos twin framework is developed by transferring the chaos engineering idea with its defined principles and process phases from the software to the manufacturing and digital twin domains. Subsequently, the chaos twin framework with its four synthesised process steps is discussed to reflect the specific characteristics and needs of manufacturing systems. Essential theoretical concepts that have been used to develop the framework are:

• The ChaosTwin [15,18] as inspiration for using a digital twin for chaos engineering and for the name of the framework.
• The process steps [8,17] (Table 1) are used to derive the chaos engineering process steps for a manufacturing context.
• The industrial digital twin technology, as developed by [26], with its specific standardised sub-models and as listed under [54]. This technology allows chaos engineering within the cyberspace of manufacturing systems.
• The resilience engineering framework according to [19] helps to classify the resilience impact of chaos engineering in manufacturing.

3.2. Process Steps

The chaos twin framework’s process is based on the process steps from [8,17]. By utilising a chaos twin for experimentation, it is possible to avoid unnecessary damage to the physical manufacturing system. Figure 3 highlights the chaos twin framework for manufacturing systems.

The implementation of the chaos twin framework across the manufacturing domain requires the design and operation of a digital twin system, which in turn allows chaos experiments in cyberspace to generate learning that can be sufficiently transferred into the real world.

Step 1 “Hypothesis”: The first step of the framework is to define chaos experiments for a given baseline of the manufacturing system. Chaos experiments must be able to reflect failures, disruptions, and disaster events, i.e., disturbances, across the manufacturing system holistically, e.g., by applying the logic in

Table 5. Logical structure for defining events for chaos experiments.

Value Creation Domains	Events
	Failures (Fl)	Disruptions (Dp)	Disasters (Ds)
	Failures are events with small-scale negative impacts limited to a sub-system or domain level.	Disruptions are events with medium-scale negative impacts limited to the system level including multiple sub-systems or domains.	Disasters are events with large-scale negative impacts beyond the system level including impacts across multiple other systems.
1. Product and Product Lifecycle	Fl11, Fl12, …, Fl1n	Dp11, Dp12, …, Dp1n	Ds11, Ds12, …, Ds1n
2. Fabrication and assembly processes	Fl21, Fl22, …, Fl2n	Dp21, Dp22, …, Dp2n	Ds21, Ds22, …, Ds2n
3. Organisation	Fl31, Fl32, …, Fl3n	Dp31, Dp32, …, Dp3n	Ds31, Ds32, …, Ds3n
4. Human	Fl41, Fl42, …, Fl4n	Dp41, Dp42, …, Dp4n	Ds41, Ds42, …, Ds4n
5. Value network	Fl51, Fl52, …, F5n	Dp51, Dp52, …, Dp5n	Ds51, Ds52, …, Ds5n
6. Business model	Fl61, Fl62, …, F6n	Dp61, Dp62, …, Dp6n	Ds61, Ds62, …, Dp6n

, which is inspired by the work of [46], for structuring value creation in manufacturing.

A chaos experiment is characterised by a change of a set of variables that intend to cause chaos in the manufacturing system, i.e., by having the potential to lead to failure, disruption, or disaster events. Variable changes might be selected randomly within a realistic limit to reflect the uncertainty of designing and operating a manufacturing system. Further, the definition of chaos experiments is dependent on the resilience phase, i.e., whether the experiment takes place during the anticipation, coping, or adaptation phase as described in

Table 6. Potential impacts of chaos engineering during the three resilience phases.

Resilience Phase	The Potential Impact of Chaos Engineering
Anticipation phase (before the disturbance event occurs)	Chaos engineering can support the anticipation of potential failure, disruption, and disaster events throughout the manufacturing system by randomly defining and running chaos experiments to test the system’s robustness for these unforeseen events. Chaos engineering can support this phase by defining and running chaos experiments in a more targeted manner on the riskiest part of the business model. The overarching goal is to develop an understanding of the robustness of the manufacturing system with its important drivers and barriers as well as their interdependencies and improve the system’s flexibility.
Coping phase (during an active disturbance event)	During the occurrence of a failure, disruption, or disaster event, chaos engineering can help to mimic the disaster within cyberspace, i.e., the digital twin, through accordingly defined and executed chaos experiments. These chaos experiments help to understand the implications of the failure, disruption, or disaster event across the manufacturing system much more quickly if the digital twin reflects the real-world behaviour sufficiently. Understanding the implications subsequently fosters the development and implementation of suitable coping strategies for the manufacturing system to counter the failure, disruption, or disaster event.
Adaptation phase (after the occurrence of the disturbance event)	After the failure, disruption, or disaster event occurred, chaos engineering can be used to define and run chaos experiments similar in scope and impact to the original failure, disruption, or disaster event. The experiments can be slightly varied, and potentially, machine learning algorithms can learn what an ideal future response of the manufacturing system to this type of failure, disruption, or disaster event would be. Eventually, the design of the manufacturing system can be adapted to enable this identified ideal future response. Moreover, the learnings from the chaos experiments during the coping phase can be used to adapt the business model of the manufacturer to actively benefit from or exploit the impacts of the failure, disruption, and disaster events in such a manner that competitive advantages for the manufacturer can be realised.

. Variable changes might comprise changing takt times, energy supply to the facilities, number of workers and team composition, equipment failure, or a cyber-attack via different attack vectors.

Step 2 “Testing”: The second step is the design and implementation of the chaos experiment in the digital twin, i.e., the digital environment. By running the chaos experiment, the digital twin functions as the chaos twin of the manufacturing system [15]. The digital twin should follow the Asset Administration Shell standard and related sub-model standards of the Industrial Digital Twin Association [54]. These standards allow for modelling and adequately reflecting the real-world behaviour of the manufacturing system by offering industry-relevant sub-models for composing digital twin systems(-of-systems), as also elaborated by [21]. For example, the loss of a key component supplier can be run as a chaos experiment with a material flow simulation of the manufacturing system. In this case, the material flow simulation represents a specific sub-model standard of the digital twin system. The resulting failed and delayed operations from the chaos experiment, e.g., resulting material defects or inbound logistics delays, are analysed within the simulation and their impact is compared to the baseline of the manufacturing system. For example, the increase in the throughput time of products due to the loss of the key supplier is compared to the baseline throughput time of the system.

Step 3 “Blast radius”: Within the third step, the resulting impacts from the chaos experiments compared to the baseline of the manufacturing system are analysed in more depth. Subsequently, specific design changes are derived and tested within the digital twin. The design changes aim at mitigating the negative impacts from the chaos experiments to the baseline of the manufacturing system. For example, a design change might be to add a new machine tool or to change the layout of the manufacturing system. The design changes are iteratively developed based on a cycle of analysis and synthesis between Step 2 and Step 3. This ensures a continuous refinement of the design changes based on the chaos experiments.

Step 4 “Insights”: The fourth step is to generate learnings from the design changes that have been tested within the chaos twin system. Efficacious improvement measures for the real manufacturing systems are selected, designed, and implemented. The implemented measures should eventually improve the resilience of the real manufacturing system by increasing its capability to cope with chaos while reducing the impact of failure, disruption, or disaster events such as failed and delayed operations.

3.3. Resilience Aspects of Chaos Engineering in Manufacturing

Resilience is a targeted value of a manufacturing system that can be improved through the utilisation of chaos engineering. Resilience can be structured into an anticipation phase, which aims at anticipating and preparing for potential failures, disruptions, and disasters; a coping phase, which foremost aims at reducing the negative impact of an occurring failure, disruption, or disaster; and an adaptation phase, which translates learnings from the failure, disruption, or disaster event into permanent improvements of the manufacturing system [19]. Table 6 highlights the potential impacts of chaos engineering during each of these three phases.

4. Case Study

4.1. Introduction to the Case Study

This case study shows how the digital twin technology presented by the IDTA can potentially be used to facilitate chaos engineering. The sub-model standard IDTA 02005 specifies the provision of simulation models for application within the AAS [55]. Thus, the use of a material flow simulation model for this case study seems suitable to demonstrate the efficacy of the chaos engineering framework for testing and verification purposes. Figure 4 shows the idea of the case study and illustrates how the material flow simulation could potentially be embedded as a sub-model, according to the IDTA 02005 standard, into the standardised digital twin technology, namely the AAS. However, as part of this study, the material flow simulation was not implemented into the AAS sub-model standard 02005.

To test and verify the chaos twin framework, the four process steps for working with chaos engineering are exemplarily conducted using a material flow simulation model. The creation of the material flow simulation model itself has been inspired by a real manufacturing case from the Danish industry with a material flow presented in Figure 5. However, the model eventually reflects a fictive case with a focus on managing a fleet of Automated Intelligent Vehicles (AIVs) for the assembly of three product variants S, M, and L. The material flow simulation model is created in Siemens Tecnomatix Plant Simulation v15.1. A description of relevant simulation elements is listed in

Table 7. Simulation elements.

Element No.	Name	Description of the Area
1	Source	Distributes product demand at a given rate and quantity, i.e. creates orders for product variants S, M, L.
2	Kitting area	Two stations (Kitting 1 and 2) kit the orders received from the source.
3	Worker pool and broker	In this simulation, the “workers” element is used to represent Automated Intelligent Vehicles (AIVs), as they can move freely. The worker pool represents the charging area for the AIVs. The broker distributes the number of AIVs and their speed.
4	Assembly area of product variants S and M	Four stations (Delivery 1 to 4) can be used to unload the kitted pallets. A conveyor belt moves the kitted pallets from each of these stations to a subsequent, dedicated assembly station(Assembly 1 to 4). A conveyor belt moves the assembled product from the assembly stations to two possible pick-up places (Pick-up 1 and 2).
5	Assembly area of product variant L	Two stations (Delivery 5 and 6) are used to unload the kitted pallets. A conveyor belt moves the kitted pallets from each of these stations to a subsequent, dedicated assembly station (Assembly 5 and 6). A conveyor belt moves the assembled product from the assembly stations to a pick-up place (Pick-up 3)
6	Drain for product variants S and product M	The drain is the delivery station (Delivery 7) for the assembled products (variants S and M).
7	Drain for product variant L	The drain is the delivery station (Delivery 8) for the assembled products (variant L).

.

4.2. Application of the Chaos Twin Framework

4.2.1. Step 1 “Hypothesis”

The baseline of the manufacturing system in the use case is measured based on the key performance indicators (KPIs) listed in

Table 8. Data collected on baseline performance.

Key Performance Indicators	Baseline
Lead time mean for all products (hh:mm:ss)	03:58:46
Throughput per day mean for all products (pcs.)	17
Cycle time mean for all products (hh:mm:ss)	01:46:32
Kitting resources mean working	1%
Kitting resources mean waiting	32%
Kitting resources mean failed	0%
Kitting resources mean blocked	67%
Transport units mean transporting	37%
Transport units mean en route to job	11%
Transport units mean waiting	53%
Transport units mean failed	0%
Assembly stations mean working	92%
Assembly stations mean waiting	8%
Assembly stations mean failed	0%
Assembly stations mean blocked	0%
Total time for product mix	10:09:42

within the material flow simulation model, i.e., the potential sub-model of the AAS. The baseline serves as the foundation for defining the chaos experiments. Five chaos experiments are defined (

Table 9. Chaos experiments.

Chaos ID	What?	When?	How?	Reasoning/ Comments
1	The kitting area kits the wrong subassembly	5% of total processing time	Introducing a constant failure for both kitting stations.	The current kitting process is manual; thus, the system may be vulnerable to varying kitting errors
2	The product mix is changed	Through the whole material flow simulation	Exchange of the production volume of the two products with the highest volume with the two products with the lowest volume	Market changes may change the product mix of the manufacturing system
3	One transport unit fails	Throughout the whole simulation	Take one transport unit out of the simulation
4	Failures in assembly stations in the assembly area of product variants S and M	10% of total processing time	Introducing a constant failure to the assembly stations for product variants S and M.	Machinery may wear down, or the employment of new employees requires training period(s)
5	All assembly times increase	Throughout the whole simulation	All assembly times are increased by 10 min	Packaging changes for all products, new features are added to the assembly, or more difficult assembly process steps are introduced.

).

Within this case study, the chaos experiments are defined during the anticipation phase of resilience with a focus on the outlined assembly process. Initial chaos experiments can be defined based on an initial risk assessment, e.g., by trying to identify the riskiest parts of the system [14]. However, the majority of chaos experiments should be defined randomly if possible. Consequently, a random selection of parameters for the chaos experiments allows us to identify and test for previously unknown and unimagined failures, disruptions, and disasters and to evaluate their impacts such as for critical and interlinked chains of events. For example, the combined failure of an assembly station and transportation unit might lead to overloading the safety and maintenance department, which then cannot respond on time anymore to other more safety-critical events.

4.2.2. Step 2 “Testing”

The chaos experiments are run in the material flow simulation model of the manufacturing system. The new KPIs are measured for the different chaos experiments as highlighted in

Table 10. Comparison of the baseline KPIs with the KPIs resulting from the chaos experiments.

Key Performance Indicators	Baseline	Chaos 1	Chaos 2	Chaos 3	Chaos 4	Chaos 5
Lead time mean for all products (hh:mm:ss)	03:58:46	03:58:46	04:20:53	02:20:36	04:26:15	05:02:56
Throughput per day mean for all products (pcs.)	17	17	16.8	16.7	16.8	14.9
Cycle time mean for all products (hh:mm:ss)	01:46:32	01:46:25	01:48:39	01:48:16	01:48:59	02:02:14
Kitting resources mean working	1%	1%	1%	1%	1%	1%
Kitting resources mean waiting	32%	32%	33%	32%	33%	33%
Kitting resources mean failed	0%	0%	0%	0%	0%	0%
Kitting resources mean blocked	67%	67%	66%	67%	66%	67%
Transport units mean transporting	37%	37%	37%	35%	37%	32%
Transport units mean en route to job	11%	11%	10%	10%	10%	9%
Transport units mean waiting	53%	53%	53%	5%	53%	58%
Transport units mean failed	0%	0%	0%	50%	0%	0%
Assembly stations mean working	89%	89%	99%	98%	99%	100%
Assembly stations mean waiting	11%	11%	61%	56%	56%	61%
Assembly stations mean failed	0%	0%	6%	12%	5%	6%
Assembly stations mean blocked	0%	0%	0%	0%	6%	0%
Total time for product mix (dd:hh:mm)	10:09:42	10:09:42	10:12:32	10:14:13	10:12:54	11:21:26

.

The total production time and cycle time increase, while the throughput per day decreases, for the chaos experiments Chaos 2, Chaos 3, Chaos 4, and Chaos 5. For example, the total time for producing the product mix increases from ca. 10 d 9 h to ca. 10 d 13 h for Chaos 2 and Chaos 4 and goes up to ca. 11 d 21 h for Chaos 5 due to the induced failures and disruptions defined in Table 9. Thus, Chaos 2, Chaos 3, Chaos 4, and Chaos 5 lead to negative impacts on the manufacturing system, whereas Chaos 5 seems to have the qualitatively most serious impact on the system in terms of delays. Chaos 1 does not seem to have any negative impacts, as there are no changes to the KPIs compared to the baseline.

4.2.3. Step 3 “Blast Radius”

The negative impact of the chaos experiments is prioritised in terms of its negative impact on the baseline of the manufacturing system. Thus, Chaos 5 is chosen for driving digital design changes, as it results in the biggest negative impact on the lead, throughput and cycle times, and total time for product mix. The digital design change comprises the addition of new assembly stations for either the S and M products (design change 1) or the L products (design change 2). The motivation behind both design changes is to create redundancies in the production capacity for the assembly of the product mix to eventually absorb the increase in assembly times for the products by allowing parallelisation of assembly processes. However, the constraint of only having the possibility to implement one of the design changes reflects the cost constraints often faced by companies and the need to select one out of many investment opportunities.

In

Table 11. Comparison of KPIs for the digital design changes.

Key Performance Indicators	Baseline	Chaos 5	Chaos 5 After Design Change 1	Chaos 5 After Design Change 2
Lead time mean for all products (hh:mm:ss)	03:58:46	05:02:56	04:04:27	04:42:30
Throughput per day mean for all products (pcs.)	17.0	14.9	14.9	16.3
Cycle time mean for all products (hh:mm:ss)	01:46:32	02:02:14	02:01:10	01:52:41
Kitting resources mean working	1%	1%	1%	1%
Kitting resources mean waiting	32%	33%	32%	32%
Kitting resources mean failed	0%	0%	0%	0%
Kitting resources mean blocked	67%	67%	68%	67%
Transport units mean transporting	37%	32%	33%	37%
Transport units mean en route to job	11%	9%	10%	11%
Transport units mean waiting	53%	58%	57%	52%
Transport units mean failed	0%	0%	0%	0%
Assembly stations mean working	89%	100%	81%	88%
Assembly stations mean waiting	11%	61%	19%	12%
Assembly stations mean failed	0%	6%	0%	0%
Assembly stations mean blocked	0%	0%	0%	0%
Total time for product mix (dd:hh:mm)	10:09:42	11:21:26	11:20:17	10:19:30

, the impacts compared to the baseline are shown for Chaos 5 after each of these two design changes has been implemented in the simulation. Based on the analysis, design change 2 causes a higher lead time but enables the production of more products and results in the production of the entire product mix approx. one day faster.

4.2.4. Step 4 “Insights”

Learnings from the digital design changes and chaos experiments in the previous step eventually lead to the implementation of an additional assembly station for the L products in the real manufacturing system, i.e., the realisation of design change 2 in the real world. The design change intends to improve the resilience of the system by increasing its robustness to unexpectedly increasing assembly times. Consequently, it helps to mitigate the impact of Chaos 5. Since the impact of the other chaos experiments (Chaos 1 to Chaos 4) is rather limited compared to the impact of Chaos 5, no further design changes are being proposed for the real manufacturing system.

5. Discussion

5.1. Critical Reflections and Limitations

Chaos Engineering in manufacturing focuses on reducing the impacts of unknown and unimagined failures, disruptions, and disasters during all lifecycle phases of manufacturing systems. Uncovering these unimagined, and therefore unknown, disturbances during the design phase, for example, by identifying a production layout design that would not allow the evacuation of personnel in the case of a specific equipment failure, might be today neglected due to the complexity of manufacturing systems and the difficulty of identifying these unlikely but high-impact events, or events that depend on the specific dynamic interplay of complex manufacturing sub-systems such as machine tools, humans, and supply chains.

The case study demonstrates a possible efficacy of the proposed chaos twin framework for improving the system resilience in the specific fictive case. The system’s robustness to cope with potential unforeseen events was improved in the material flow simulation. Eventually, a suitable design solution was found and tested to maintain productivity by reducing the impact of a specific chaos from a 14% increase in total production time for the product mix to only a 4% increase in total production time.

However, the case study does not reflect the application of chaos engineering within a real production or digital twin environment. It showcases a fictive case, which has been implemented in one material flow simulation only and not in a real standardised digital twin setup such as the AAS. Hence, the implementation of the material flow simulation into the AAS sub-model standard 02005 and the realisation of data exchange between different AAS sub-models was not tested. Also, disturbances and connected impacts that would occur during real operations are not fully covered by the simple material flow simulation model, as every model has its limitations and does not reflect real-world behaviour in all its details. In addition to only relying on one material flow simulation case study, the small number of chaos experiments is another limitation of this study. Thus, the study only demonstrates the example implementation of chaos engineering in a well-defined simulation environment and it does not allow to draw conclusions about the general usefulness and validity of the approach for manufacturing systems in combination with digital twin technology.

However, the chaos engineering approach could potentially be useful for application across manufacturing systems if the chaos experiments are limited to the industrial digital twin of the system. The induction of random or unknown disturbances to the actual hardware of a running manufacturing system seems implausible as an option for manufacturers, as it conflicts with manufacturing realities where disturbances often lead to high costs, which would typically outweigh the benefits of such an experiment. In the future, we believe that the material flow simulation model will become embedded in the AAS sub-model 02005 and will be based on time series data, thus reflecting the real operational state of the manufacturing system.

The presented adoption of chaos engineering principles, i.e., the four process steps (Step 1: Defining the steady state, Step 2: Set up a hypothesis, Step 3: Introduce variables that reflect real-world events, Step 4: Verify the outcome), seem to be potentially suitable for realising chaos experiments and learnings within the cyberspace of manufacturing systems. However, the question remains whether future experimentation in industrial digital twins is suitable for deriving conclusions about the actual behaviour of the manufacturing system, since digital twins provide only a model of the real world. Even though industrial digital twin models are based on relevant lifecycle data of real manufacturing assets, some uncertainty regarding the fidelity, calibration, or validation of digital twin models might remain and could negatively impact the usefulness of chaos experiments. Also, the adoption of chaos engineering might raise the question of whether it can adequately cope with the complexity of manufacturing value networks. For example, some value networks might be substantially dependent on conditions that are difficult to measure, such as domain knowledge and experience of key personnel or trust between different stakeholders, and hence cannot be easily modelled via digital twin-based chaos engineering.

A barrier to applying the framework seems to be the current maturity of manufacturing systems in terms of using standardised industrial digital twins. Another implementation barrier may be the setup cost by increasing the level of digitalisation and evolving the digital twin maturity level. However, a relatively high maturity of the digital twin technology might be required, since chaos engineering might be most effective if it has access to the live data of a manufacturing system for defining the experiments.

It is expected that not all manufacturing companies are likely to benefit equally from adapting chaos engineering. Differentiating between the types of manufacturing systems is relevant as the number of potential failures and the associated costs can vary greatly depending on the specific system. Because of relatively high cost of potential disturbances, companies using less flexible manufacturing systems, with, for example, rigid production lines and just-in-time inventory management, might likely benefit from using the chaos engineering method.

5.2. Differentiation of Chaos Engineering in Simulation Models from Scenario Testing with Simulation Models

Testing different manufacturing scenarios within complex simulation models is a commonly used practice in manufacturing engineering. For example, creating different layout and manufacturing process chain variants within a material flow simulation model for testing key performance indicators (KPIs), such as machine utilisation, cycle or throughput times, is often used to verify a manufacturing system design before starting the actual system integration process. Differentiating this commonly used practice from chaos engineering with digital twins therefore seems relevant for providing context about the novelty and relevance of chaos engineering in manufacturing. The main difference between traditional scenario testing using, for example, material flow simulation models and chaos engineering is that chaos engineering induces random disturbances such as failures, disruptions, and disaster events into the model. Consequently, a random selection of parameters for the chaos experiments allows the identification and testing of previously unknown and unimagined disturbances as well as to evaluate their impacts. Traditional scenario testing in simulation models is often based on known parameters and system relations for testing different predefined manufacturing scenarios. However, the differences and commonalities between chaos engineering and scenario testing with simulation models may need to be further investigated to develop a clear and unique value proposition for chaos engineering in manufacturing.

5.3. Research Contributions

The research outcomes create knowledge gains within two main dimensions: (A) the academic community, and (B) manufacturing practitioners of the wider industry community.

(A)

For the academic manufacturing community, the research outcomes intend to enhance the field of designing and operating manufacturing systems by demonstrating new perspectives of applying chaos engineering to improve the resilience of manufacturing systems. More specifically, the outcomes might contribute to the research of improving resilience in manufacturing throughout the anticipation, coping, and adaptation phases of resilience. Consequently, the outcomes can contribute to the theory of engineering design methodologies for resilient manufacturing systems by proposing a new theoretical design process with specific phases.

(B)

For manufacturing practitioners, the study intends to provide a novel industry-oriented guide on how to practically apply chaos engineering to real-life manufacturing systems by using the AAS. This guideline aims to facilitate the practical application of chaos engineering outside of the software industry. Additionally, the research might contribute to creating novel application cases, services, and business models for the implementation of digital twin technology by showing the potential benefits of chaos engineering. The search for and definition of new application cases, services, and business models for digital twins (such as, for example, highlighted in [56]) is an ongoing and relevant task for the industry.

Additionally,

Table 12. Examples of how chaos engineering can address future Industry 5.0 challenges.

Industry 5.0 challenges across its three pillars	Potential focus areas for coping with the challenges
	Chaos engineering might…
Increasing human-centric value creation	(a) … help to identify potential safety and health-related risks as well as potential design and operational flaws in manual workspaces and propose relevant design improvements. (b) …facilitate the testing of changes to relevant system parameters, such as takt speed, tools, manufacturing tasks, team compositions and their impact on the human. (c) …be used to continuously test the limitations of changes in Fluid Manufacturing Systems, as described by [57], for finding ideal anthropocentric solutions.
2. Improving sustainability across the value network	(a) … support the identification of potential sources, risks, and impacts for emitting hazardous outputs. (b) …help to identify peak consumptions in terms of energy, water, and materials and thus can facilitate the creation of more robust supply chains. (c) …contribute to testing parameters in industrial symbiosis networks, as described by [58], or circular economy networks, and the impact of disturbances on the network, for example, the sudden failure of network actors.
3. Improving the resilience of organisations	(a) …support the identification of potential black swan events and scenarios as well as of other less severe but potentially risky failures, disruptions, and disaster events for the manufacturing system and help to derive efficacious countermeasures. (b) …facilitate the testing for ICT network vulnerabilities, especially within the digital network, for cyber-attacks and other disruptions such as equipment failure. (c) …help to identify and prepare for potential failure, disruption, and disaster events as well as vulnerabilities across the supplier network.

highlights how chaos engineering might address future Industry 5.0 challenges which require further empirical validation.

5.4. Future Research

Future research should further examine how chaos engineering can be implemented beyond material flow simulations in a complete and mature industrial digital twin system such as the AAS and its sub-models. The efficacious generation of random chaos experiments also seems to be an important future research area. Finding the differences and commonalities between chaos engineering and scenario testing with simulation models is of relevance for future research to develop a clear and unique value proposition for chaos engineering in manufacturing. The utilisation of real-time or near real-time digital twin data for carrying out chaos experiments further seems to be an interesting future research opportunity. This would enable chaos experiments for a live manufacturing system and would help to uncover manufacturing system vulnerabilities continuously. Additionally, future research might explore the potential of applying the chaos twin framework to the Industrial Metaverse as a highly mature implementation of the digital twin. The idea of the Industrial Metaverse is to reflect the real manufacturing system with a high level of detail by interlinking the data streams between the physical manufacturing assets and the digital twin system [59]. Consequently, there might be an opportunity for using the chaos twin framework in combination with the Industrial Metaverse to improve resilience of a manufacturing system. The Industrial Metaverse allows the realisation of a highly detailed digital environment, and thus of a digital model that is quite close to the real world. Running chaos experiments in this model is expected to create results similar to running the same experiments in the real world while being highly cost-effective. Also, potential design changes for improving the resilience can be cost-effectively implemented and tested in the Industrial Metaverse.

6. Conclusions

The research presented an approach, supported by a case study, for applying chaos engineering in manufacturing systems to improve resilience. For this purpose, the chaos twin framework includes a four-step process. The first step of the framework aims to define chaos experiments for a given baseline of the manufacturing system. The second step focuses on the design and implementation of the chaos experiment in the digital twin system. Within the third step, resulting impacts from the chaos experiments are compared to the baseline of the manufacturing system and potential design changes for mitigating the impacts are tested in the digital twin system. Eventually, the fourth step addresses the creation of learnings from the design changes that have been tested within the chaos twin system and their eventual implementation in the real manufacturing system. To verify the chaos twin framework, a case study was discussed based on a material flow simulation model as a potential part of an industrial digital twin sub-model. The case study showed the example implementation of the chaos twin framework for improving the system’s resilience. The research results can potentially serve as a framework for applying chaos engineering in manufacturing systems and as a stepping stone for future academic research and industrial implementation.