Safety-Aware Multi-Agent Deep Reinforcement Learning for Adaptive Fault-Tolerant Control in Sensor-Lean Industrial Systems: Validation in Beverage CIP

Abstract

Fault-tolerant control in safety-critical industrial systems demands adaptive responses to equipment degradation, parameter drift, and sensor failures while maintaining strict operational constraints. Traditional model-based controllers struggle under these conditions, requiring extensive retuning and dense instrumentation. Recent safe multi-agent reinforcement learning (MARL) frameworks with control barrier functions (CBFs) achieve real-time constraint satisfaction in robotics and power systems, yet assume comprehensive state observability—incompatible with sensor-hostile industrial environments where instrumentation degradation and contamination risks dominate design constraints. This work presents a safety-aware multi-agent deep reinforcement learning framework for adaptive fault-tolerant control in sensor-lean industrial environments, achieving formal safety through learned implicit barriers under partial observability. The framework integrates four synergistic mechanisms: (1) multi-layer safety architecture combining constrained action projection, prioritized experience replay, conservative training margins, and curriculum-embedded verification achieving zero constraint violations; (2) multi-agent coordination via decentralized execution with learned complementary policies. Additional components include (3) curriculum-driven sim-to-real transfer through progressive four-stage learning achieving 85–92% performance retention without fine-tuning; (4) offline extended Kalman filter validation enabling 70% instrumentation reduction (91–96% reconstruction accuracy) for regulatory auditing without real-time estimation dependencies. Validated through sustained deployment in commercial beverage manufacturing clean-in-place (CIP) systems—a representative safety-critical testbed with hard flow constraints (≥1.5 L/s), harsh chemical environments, and zero-tolerance contamination requirements—the framework demonstrates superior control precision (coefficient of variation: 2.9–5.3% versus 10% industrial standard) across three hydraulic configurations spanning complexity range 2.1–8.2/10. Comprehensive validation comprising 37+ controlled stress-test campaigns and hundreds of production cycles (accumulated over 6 months) confirms zero safety violations, high reproducibility (CV variation < 0.3% across replicates), predictable complexity–performance scaling ($R^2=0.89$), and zero-retuning cross-topology transferability. The system has operated autonomously in active production for over 6 months, establishing reproducible methodology for safe MARL deployment in partially-observable, sensor-hostile manufacturing environments where analytical CBF approaches are structurally infeasible.

1. Introduction

Fault-tolerant control in safety-critical industrial systems confronts a fundamental challenge: maintaining stable operation and strict constraint satisfaction despite equipment degradation, sensor failures, and time-varying process dynamics. Many industrial processes—including fluid networks, chemical dosing, cleaning systems, and heat-exchange operations—operate under highly dynamic conditions combining partial observability, nonlinear responses, and evolving system parameters. In safety-critical applications such as food, beverage, pharmaceutical, and chemical processing, controllers must ensure reproducibility, traceability, and stable operation while adapting to equipment aging and operational changes, all while minimizing sensor exposure and calibration demand. These factors collectively define a context requiring adaptive and intelligent control architectures capable of functioning with minimal, uncertain feedback information—what we term sensor-lean operation with fault-tolerant capabilities.

Traditional control approaches face three fundamental limitations that hinder adaptive fault-tolerant operation. First, they struggle to maintain simultaneous hard safety constraints without extensive instrumentation: maintaining critical process variables within strict bounds while optimizing secondary objectives requires dense sensor networks and manual tuning [1,2]. Second, model-based strategies such as model predictive control (MPC) depend on accurate system models that degrade under parameter drift, equipment aging, and operational changes—requiring continuous recalibration and expert intervention [1,2]. While hardware redundancy approaches address sensor failures through ancillary backup systems [3], such strategies increase system complexity, maintenance burden, and instrumentation costs—ultimately conflicting with sensor-lean operational imperatives in cost-sensitive, contamination-critical manufacturing environments. Third, conventional controllers demand extensive commissioning: each configuration change necessitates manual retuning, consuming hours to days of engineering time depending on system complexity, creating operational bottlenecks and production delays [1,4]. These limitations motivate the exploration of data-driven, adaptive control methodologies capable of autonomous fault-tolerant operation under uncertainty.

Reinforcement learning (RL) has emerged as a promising data-driven alternative for adaptive fault-tolerant control, enabling controllers to learn optimal policies through experience without explicit system models [5,6]. Recent safe multi-agent reinforcement learning (MARL) frameworks incorporating control barrier functions (CBFs) have demonstrated real-time constraint satisfaction in robotics [7], autonomous vehicles [8], and power systems [9], providing formal mathematical safety guarantees through analytical barrier derivation. However, these approaches assume comprehensive state observability (full sensor coverage) and accurate system models for barrier function design—prerequisites often violated in harsh industrial environments where sensor deployment is economically prohibitive, technically unreliable due to chemical/thermal exposure, or incompatible with hygienic design requirements. Beyond safe MARL, hardware redundancy approaches address sensor failures through ancillary backup systems [3]. While effective for mitigating single-point failures, such strategies increase system complexity, maintenance burden, and instrumentation costs—ultimately conflicting with sensor-lean operational imperatives in cost-sensitive, contamination-critical manufacturing environments. Despite these advances, industrial RL deployment faces critical barriers preventing production adoption in sensor-lean, fault-tolerant contexts. Gap 1: Safety guarantees under partial observability. While CBF-based safe MARL provides formal guarantees, these guarantees hold only when barrier functions h(s) can be accurately evaluated—requiring either direct measurement or reliable estimation of all state variables entering h(s). In sensor-lean industrial systems with regime-dependent dynamics (parameter drift, equipment aging, unmodeled disturbances), neither condition is satisfied. Existing RL approaches without CBF lack mathematical guarantees of constraint satisfaction, providing only probabilistic risk reduction insufficient for zero-tolerance safety requirements. Safe RL frameworks [10,11] remain predominantly theoretical, with recent surveys [11] highlighting persistent gaps between theoretical safety guarantees and validated industrial implementations under authentic disturbances in partially-observable regimes. Gap 2: Simulation-to-reality transfer failure. The sim-to-real gap causes trained policies to degrade when deployed on physical systems due to modeling errors and unmodeled dynamics [12]. Existing approaches either require extensive online fine-tuning (unacceptable for safety-critical systems) or assume access to abundant real system data (impractical for commissioning). Recent work demonstrates curriculum-based sim-to-real transfer in robotics [13], yet validated protocols for industrial process control with regime-dependent dynamics and partial observability remain absent. Gap 3: Sensor dependency and fault detection. Current RL-based process control assumes comprehensive state measurements including flows, pressures, temperatures, and chemical concentrations [14], conflicting with sensor-lean imperatives where minimizing wetted instrumentation reduces contamination risk and maintenance burden. Recent work on minimal sensing [15] demonstrates potential but relies on real-time state estimation, creating failure vulnerabilities. Furthermore, existing approaches lack integrated mechanisms for fault detection and diagnosis using reduced sensor sets—critical for fault-tolerant operation. Gap 4: Limited sustained production validation. While recent work demonstrates initial RL deployments in industrial settings [4,14], comprehensive validation establishing long-term operational stability, systematic safety verification across diverse configurations, and quantified economic benefits remains scarce in the literature. Most studies report simulation results, laboratory demonstrations, or single-configuration pilot tests rather than sustained multi-configuration production operation with complete safety documentation and economic quantification necessary for widespread industrial adoption [12]. Specifically, comprehensive validation demonstrating fault-tolerant operation under equipment degradation, sensor failures, and forced disturbances across diverse system configurations remains absent from the literature.

To address these gaps, this work proposes a safety-aware multi-agent deep reinforcement learning framework for adaptive fault-tolerant control in sensor-lean industrial systems. The framework integrates four synergistic design principles: (1) multi-layer safety mechanisms, providing formal constraint satisfaction guarantees through constrained action spaces, prioritized safety-focused learning, and layered verification achieving zero violations during training and deployment; (2) multi-agent coordination, enabling robust distributed control with learned complementary policies and decentralized execution suitable for fault-tolerant operation; (3) progressive curriculum learning, bridging simulation-to-reality gaps through domain randomization and staged complexity escalation achieving high-fidelity transfer without manual fine-tuning; and (4) offline sensor fusion and validation framework, using extended Kalman filter post-control reconstruction enabling sensor-lean operation while maintaining comprehensive audit trails for regulatory compliance and fault diagnosis. The modular architecture abstracts control logic, state representation, and safety mechanisms into reusable components, enabling adaptation across pharmaceutical, chemical, and food processing applications with systematic reconfiguration rather than redesign.

The proposed architecture is validated through comprehensive industrial implementation in clean-in-place (CIP) systems for preservative-free beverage manufacturing—a representative testbed exhibiting all target challenges: hard safety constraints (flow rate $\ge 1.5$ L/s, volume bounds $[100,200]$ L), sensor-lean requirements (aggressive chemical/thermal environments degrading wetted instrumentation), multi-circuit operational complexity (diverse hydraulic architectures), and zero-tolerance safety requirements (contamination prevention).

A multi-agent deep Q-network (MADQN) instantiation demonstrates architecture viability through deployment and sustained production operation at VivaWild Beverages (Colima, Mexico). Controlled stress-test validation across three hydraulic configurations (complexity 2.1–8.2/10) achieves the following: control precision 3–5× better than industrial standards (coefficient of variation: 2.9–5.3% versus <10% threshold), zero safety violations with perfect constraint satisfaction, predictable complexity–performance scaling ($R^2=0.89$), and 70% instrumentation reduction with quantified economic benefits. The framework currently operates autonomously in active production, managing daily CIP operations while generating continuous operational data for future comprehensive long-term studies.

This work distinguishes itself from CBF-based safe MARL by addressing a complementary operational regime: rather than deriving analytical barrier functions h(s) from accurate models and full state measurements, our framework learns implicit safety boundaries directly from observable state [V, xs, xe] through extensive domain randomization spanning regime-dependent parameter variations ($Pe\in [3.0, 7.0]$), achieving comparable constraint satisfaction (zero violations) without requiring comprehensive instrumentation or real-time state estimation. This paradigm shift from analytical to learned barriers is particularly suited for harsh industrial environments where sensor density and model fidelity assumptions underlying CBF methods are structurally incompatible with operational constraints.

Contributions and Novelty

This study extends reinforcement learning applications to field-level deployment within industrial processes characterized by partial observability, stringent safety requirements, and limited instrumentation. The proposed framework addresses the persistent gap between simulated RL studies and deployable industrial control systems by introducing safety-aware multi-agent learning, multi-layer safety validation, and reproducible deployment methodology for fault-tolerant control. While multi-agent deep Q-networks [16,17,18,19] and extended Kalman filter validation [20] are established algorithms with proven efficacy in their respective domains, our novelty lies not in the algorithms themselves (MADQN and EKF are well established) but in their architectural integration for a complementary operational regime—sensor-lean, partially-observable, harsh industrial environments where control barrier function prerequisites are structurally incompatible—achieving safety-aware adaptation and sustained validated deployment absent from the prior literature. Our contributions are architectural and operational, not algorithmic:

• Safety-aware multi-agent deep reinforcement learning for fault-tolerant control: Integrated safety architecture ensuring constraint satisfaction in distributed multi-agent systems through four complementary mechanisms: (1) constrained action projection onto feasible sets preventing unsafe exploration, (2) prioritized safety-focused experience replay oversampling critical events 5–10×, (3) conservative safety margins with training thresholds 20% tighter than deployment requirements, and (4) curriculum-embedded safety verification. Stage advancement requires demonstrated compliance before progression, ensuring safety-aware capability development throughout training. The framework supports scalable N-agent configurations—validated through dual-agent implementation coordinating inlet/outlet pump control in CIP systems—achieving zero violations across all validation tests and sustained production operation. Agents learn complementary safety-aware policies through shared reward signals and coordinated constraint satisfaction, demonstrating emergent cooperative behavior (Pearson correlation r = −0.36 to −0.63) without explicit coordination rules while maintaining independent decentralized execution.
• Sensor-lean operation via offline sensor fusion and learning: A modular, component-based architecture ensuring reliable operation with minimal sensory inputs, embedding safety-layer constraints as structural elements of the decision-making pipeline. Extended Kalman filter (EKF) validation operates as an offline auditing component, reconstructing system trajectories and validating controller performance without interfering with real-time control. This architecture enables sensor-lean operation (eliminating 70% wetted instrumentation) while maintaining comprehensive audit trails for regulatory compliance (FDA 21 CFR Part 11, ISO 9001) and fault diagnosis capabilities, achieving 91–96% reconstruction accuracy with 30–45 s convergence times. The architecture abstracts domain-specific details into configurable components (state representation, reward engineering, constraint formulation), enabling adaptation across pharmaceutical, chemical, and food processing applications with systematic reconfiguration procedures rather than complete redesign.
• Curriculum-driven sim-to-real transfer protocol: Comprehensive curriculum learning protocol enabling high-fidelity simulation-to-reality transfer (85–92% performance retention) without manual fine-tuning through progressive complexity escalation and domain randomization. The structured four-stage curriculum systematically exposes agents to increasing operational variability, disturbances, and multi-agent coordination challenges, bridging the sim-to-real gap while maintaining formal safety guarantees throughout training.
• Industrial deployment validation and architectural transferability: Sustained production deployment demonstrates operational readiness across three diverse hydraulic architectures (complexity 2.1–8.2/10) without manual retuning—validating architectural generalization through zero-reconfiguration transfer. Comprehensive stress-test validation campaigns provide controlled performance assessment with complete instrumentation coverage (storage: 8.1 min/484 samples, mixing: 11.2 min/671 samples, UHT: 10.2 min/614 samples), confirming zero safety violations, predictable complexity–performance scaling ($R^2=0.89$), and superior precision (CV: 2.9–5.3% vs. 10% industrial standard). The framework currently operates autonomously in active production, generating continuous operational data and enabling future comprehensive long-term stability studies and economic impact quantification. Preliminary economic analysis indicates substantial sensor reduction benefits (USD 12,000–18,000 per circuit) and maintenance savings (USD 6000–10,000 annually), with ongoing deployment enabling rigorous multi-year ROI validation.

To the best of the authors’ knowledge, no prior work has demonstrated a safety-aware multi-agent reinforcement learning architecture for adaptive fault-tolerant control with (1) formal mathematical safety guarantees validated under authentic industrial conditions achieving zero violations across validation tests and sustained production operation, (2) comprehensive curriculum learning protocol enabling high-fidelity sim-to-real transfer (85–92% performance retention) without manual fine-tuning, (3) sensor-lean operation eliminating 70% instrumentation through comprehensive offline learning and sensor fusion rather than real-time estimation, and (4) sustained production deployment with validated architectural transferability across diverse hydraulic configurations (complexity 2.1–8.2/10) achieving zero-retuning generalization.

The remainder of this paper is structured as follows. Section 2 reviews related work on safe reinforcement learning, multi-agent systems, and industrial automation. Section 3 presents the proposed component-based architecture and design principles. Section 4 characterizes the CIP control problem as an industrial validation testbed. Section 5 details MADQN implementation, training methodology, and curriculum design. Section 6 describes experimental protocols, deployment procedures, and validation metrics. Section 7 presents comprehensive performance results from stress-test validation campaigns. Section 8 discusses implications, transferability to other domains, and limitations. Section 9 concludes with a summary of the contributions and future research directions.

2. Related Work

2.1. AI for Fault Diagnosis and Fault-Tolerant Control

Fault-tolerant control (FTC) addresses system operation under component failures, sensor faults, and actuator degradation through reconfiguration, redundancy, or adaptive compensation strategies [21,22]. Traditional FTC approaches employ analytical redundancy and model-based fault detection and isolation (FDI), requiring accurate system models and extensive fault characterization [23]. Recent work explores data-driven FDI using machine learning for fault detection in industrial systems [24,25], demonstrating improved robustness to modeling uncertainty compared to analytical methods.

Deep-learning-based fault diagnosis has achieved significant success in rotating machinery [26], chemical processes [27], and manufacturing systems [28] through convolutional neural networks (CNNs) and recurrent architectures processing sensor time-series data. However, these approaches focus on fault detection rather than fault-tolerant control—identifying anomalies without autonomous adaptation to maintain operation under degraded conditions.

Reinforcement learning for fault-tolerant control has gained momentum in recent years, with promising applications across diverse domains. Liu and Liang [29] proposed deep deterministic policy gradient (DDPG) for spacecraft attitude control under actuator failures, achieving finite-time convergence through optimized sliding mode control parameters. Jiang et al. [30] combined model predictive dontrol with RL for quadcopter fault tolerance, demonstrating real-time applicability with data-based fault detection achieving satisfactory trajectory tracking under multiple faults. Kim et al. [31] introduced transformer-based adaptation for a quadrotor FTC, showing robust performance under actuator failures through online adaptation mechanisms. Treesatayapun [32] presented event-triggered RL-based FTC for discrete-time systems, achieving 25% reduction in data transmission while maintaining closed-loop performance. However, these works focus on aerospace/robotics applications without addressing harsh chemical environments, sensor-lean requirements, and zero-tolerance safety constraints characteristic of industrial process control.

The intersection of multi-agent RL and fault-tolerant control presents additional challenges. Multi-agent systems must maintain coordination despite individual agent failures or communication degradation. Existing MARL frameworks assume reliable inter-agent communication and full observability [16,17], conflicting with sensor-lean FTC requirements where agents operate under partial information and potential sensor failures. Our framework uniquely integrates safety-aware MARL with offline sensor fusion and formal constraint satisfaction for industrial FTC deployment in harsh manufacturing environments. Beyond control-oriented MARL, recent work explores neuro-symbolic knowledge transfer to coordinate heterogeneous agents through distributed knowledge graphs and learned graph representations [33]. These approaches highlight the importance of structured cooperation mechanisms in multi-agent systems, but they target communication and knowledge sharing rather than safety-critical physical process control with hard operational constraints.

Recent industrial RL deployments demonstrate practical viability with enhanced safety mechanisms. Su et al. [34] provided a comprehensive review of safe RL methods for modern power systems, emphasizing emerging techniques including Lagrangian relaxation, control barrier functions, and constrained policy optimization achieving real-time constraint satisfaction in large-scale systems. Zheng et al. [35] applied safe RL to gold cyanide leaching processes using chance control barrier functions and augmented Lagrangian optimization, achieving superior performance compared to baseline algorithms while satisfying joint chance constraints. Ye et al. [36] demonstrated real-time price-based demand response for manufacturing processes using safe deep RL with hybrid action spaces, addressing electricity cost optimization under production constraints. Jendoubi and Bouffard [37] proposed hierarchical multi-agent reinforcement learning for energy management in microgrids, demonstrating effective coordination of multiple controllable devices without communication overhead through distributed policy learning. These works validate safe RL and MARL potential for industrial optimal control but predominantly require dense state measurements and assume benign operating environments—contrasting with harsh chemical/thermal conditions and sensor-lean requirements in food and beverage manufacturing where wetted instrumentation degradation and contamination risks dominate design considerations.

2.2. Reinforcement Learning for Industrial Process Control

Deep reinforcement learning has emerged as a promising approach for adaptive industrial control, with deep Q-networks (DQNs) demonstrating human-level performance in complex decision-making tasks [5,6]. The extension to industrial process control addresses the limitations of traditional model-based methods by learning optimal policies from experience without requiring explicit system models [4,14].

Liu et al. [38] provided a comprehensive survey of RL applications in chemical process control, highlighting successful simulation demonstrations but noting significant barriers to industrial deployment: extensive simulation tuning requirements, lack of formal safety guarantees, and sim-to-real transfer challenges. Spielberg et al. [14] applied a deep RL to process control tasks, demonstrating potential for adaptive optimization, yet identified safety validation and real-time execution constraints as critical deployment barriers. Nian et al. [4] emphasized that while RL shows promise for handling nonlinear dynamics and parameter uncertainty, transition from simulation to physical systems remains the primary obstacle preventing widespread adoption.

Multi-agent reinforcement learning (MARL) extends single-agent approaches to distributed control scenarios common in industrial systems [16,17]. Foerster et al. [16] introduced counterfactual multi-agent policy gradients, enabling credit assignment in cooperative settings, while Zhang et al. [17] provided theoretical foundations for multi-agent learning convergence. Recent surveys [18] and industrial applications [19] demonstrate MARL scalability for factory-wide dynamic scheduling in semiconductor manufacturing, achieving robust performance through leader–follower architectures. However, existing MARL frameworks assume reliable inter-agent communication and dense state feedback—impractical in harsh industrial environments with chemical exposure, thermal extremes, and limited instrumentation access. Furthermore, coordination mechanisms in published work rely on explicit communication protocols requiring dedicated network infrastructure, conflicting with sensor-lean operational imperatives.

Recent advances in intelligent manufacturing further illustrate the potential of deep RL for large-scale scheduling and resource allocation. Zhang et al. [39] proposed a dual resource scheduling method for production equipment and rail-guided vehicles based on a proximal policy optimization (PPO) algorithm, demonstrating effective handling of spatiotemporal coupling and dynamic constraints in flexible manufacturing systems. However, such applications typically rely on dense instrumentation and operate under benign safety conditions, contrasting with the sensor-lean, zero-tolerance CIP environment addressed in this work.

2.3. Safe Reinforcement Learning

Safety guarantees constitute critical requirement for industrial RL deployment, addressed theoretically through constrained Markov decision process (CMDP) frameworks [10,40,41]. Altman [40] established mathematical foundations for constrained optimization in sequential decision problems, while Garcia and Fernandez [10] provided a comprehensive survey of safe RL methods with recent advances [11], highlighting persistent gaps between theoretical frameworks and validated industrial implementations. Safe RL approaches are categorized into modification of the exploration process, modification of the optimality criterion, and external knowledge incorporation.

Thomas and Brunskill [42] developed high-confidence off-policy evaluation enabling statistical safety guarantees during policy assessment—critical for scenarios where online experimentation carries unacceptable risks. Ray et al. [43] benchmarked safe exploration algorithms, demonstrating trade-offs between performance and conservatism: aggressive exploration achieves higher rewards but increased constraint violations, while conservative approaches sacrifice performance for safety assurance.

Despite theoretical advances, industrial validation of safe RL frameworks remains limited. Existing approaches either sacrifice performance through excessive conservatism [43], require known system models conflicting with adaptive control objectives, or provide only probabilistic safety guarantees insufficient for zero-tolerance industrial requirements. Recent reviews [11] emphasize the need for practical implementations combining multiple complementary mechanisms, achieving deterministic constraint satisfaction. The gap between theoretical safety frameworks and deployable industrial systems with mathematical constraint satisfaction under authentic disturbances motivates integrated multi-layer safety mechanisms combining multiple complementary strategies.

2.4. Curriculum Learning and Sim-to-Real Transfer

Curriculum learning progressively increases task complexity during training, enabling more efficient learning and improved generalization [44,45]. Bengio et al. [44] demonstrated that presenting training examples in meaningful order—from simple to complex—accelerates convergence and improves final performance compared to random sampling. Soviany et al. [45] provided a recent survey highlighting curriculum design strategies: task-level progression, data-level selection, and hybrid approaches combining multiple curriculum dimensions.

The simulation-to-reality gap presents a fundamental challenge for RL deployment: policies trained in simulation degrade when transferred to physical systems due to modeling errors, unmodeled dynamics, and sensor/actuator imperfections. Existing sim-to-real approaches employ domain randomization (varying simulation parameters to expose policy to diverse conditions), system identification (refining models through real data), or online adaptation (fine-tuning policies on physical system). Recent work [46] introduces online correction mechanisms for policy transfer in robotics, demonstrating improved robustness through learned adaptation strategies. However, online fine-tuning conflicts with safety requirements in industrial settings where exploration during physical deployment carries contamination, equipment damage, or production disruption risks.

Systematic curriculum protocols specifically designed for industrial safety-critical systems—integrating progressive complexity escalation with formal safety verification at each stage—remain unexplored in the literature. Furthermore, quantitative assessment of sim-to-real transfer quality (performance retention without fine-tuning) lacks standardized metrics and comprehensive industrial validation across diverse configurations.

2.5. Sensor Reduction and State Estimation

Recent work explores minimal sensing approaches for RL-based control, demonstrating that comprehensive offline training can compensate for reduced online instrumentation [15,47]. These approaches use state estimation techniques including extended and unscented Kalman filters [48]—to reconstruct unmeasured variables, enabling control with partial observations.

However, existing sensor reduction frameworks employ state estimation within real-time control loops, creating single points of failure: estimator divergence, sensor dropout, or model mismatch directly impact control decisions, potentially causing constraint violations or instability. Industrial deployment requires distinguishing between control-critical measurements (essential for real-time decisions) and validation-critical measurements (necessary for regulatory compliance but not real-time control). No prior work validates sensor-lean RL control, eliminating wetted instrumentation entirely during operation while maintaining comprehensive audit trails through offline reconstruction.

The distinction between real-time state estimation (used for control) versus offline validation (used for documentation) fundamentally alters failure modes and deployment risk profiles. Our framework relegates EKF validation exclusively to offline post-control analysis, eliminating real-time estimation dependencies while maintaining regulatory compliance documentation capabilities.

2.6. Industrial Deployment Challenges

Industrial RL deployment faces practical barriers beyond algorithmic performance [49]. Lee et al. [49] identified key obstacles: interpretability requirements for operator acceptance, robustness to equipment aging and parameter drift, commissioning time and retuning effort for new configurations, and integration with existing plant-wide control systems. Conventional industrial controllers require 24–48 h of manual tuning per configuration change, creating operational bottlenecks during equipment modifications, recipe changes, or topology reconfigurations.

Furthermore, industrial environments impose constraints absent in research settings: zero-tolerance safety requirements with regulatory consequences for violations, limited instrumentation access due to hygienic design imperatives, aggressive chemical/thermal exposure degrading sensor reliability, and requirement for comprehensive documentation and audit trails supporting FDA, EHEDG [50], and ISO compliance. These practical constraints motivate architectural approaches prioritizing component-based transferability, enabling deployment across diverse configurations without circuit-specific retuning.

2.7. Safe Multi-Agent Reinforcement Learning and Control Barrier Functions

Recent advances in safe multi-agent reinforcement learning (MARL) have demonstrated formal constraint satisfaction through the integration of control barrier functions (CBFs) [7,51]. CBFs provide mathematical certificates of forward invariance: given a safe set $\mathcal{C}=\left\{s\right|h\left(s\right)\ge 0\}$ defined by barrier function $h:\mathcal{S}\to \mathbb{R}$, the condition $\dot{h}\left(s\right)\ge -\alpha h\left(s\right)$ guarantees $s\left(t\right)\in \mathcal{C}$ for all $t\ge 0$ under admissible control [51]. This elegant formalism has enabled safe learning in robotics [7], autonomous vehicles [8], and power systems [9].

Qin et al. [7] introduced decentralized neural barrier certificates for multi-agent systems, learning barrier functions alongside policies to enable scalable safe coordination. Taylor et al. [52] demonstrated that learned CBFs can outperform analytical designs under model uncertainty. Recent work extends CBFs to stochastic systems [53] and high-dimensional state spaces [54], addressing scalability challenges. The integration of CBF with curriculum learning [13] has shown promise in robotics, progressively tightening safety constraints as policies mature. Recent advances in Lagrangian-based safe reinforcement learning provide complementary approaches to CBF methods [41]. Bandyopadhyay and Bhasin proposed the Actor–Critic–Identifier–Lagrangian (ACIL) architecture combining barrier Lyapunov functions with online Lagrange multiplier learning for state-constrained optimal control in continuous-time uncertain nonlinear systems. Their approach demonstrates formal safety guarantees through online adaptation without compromising constraint satisfaction during learning.

However, CBF applicability hinges on two critical prerequisites: (1) comprehensive state observability—all variables entering $h\left(s\right)$ must be measured or reliably estimated at runtime; and (2) model fidelity—dynamics $\dot{s}=f(s,u)$ must be known sufficiently accurately to derive valid $\dot{h}\left(s\right)$ [51]. These conditions are frequently violated in harsh industrial environments where sensor deployment faces economic constraints (USD 3k–5k per flow/pressure transducer), technical limitations (fouling, chemical attack degrading sensors within months), and hygienic design requirements (minimizing wetted instrumentation to reduce contamination risk) [1]. Furthermore, industrial processes exhibit regime-dependent dynamics—parameters drift due to equipment aging, fouling accumulation, and operational changes—making fixed analytical barrier derivation infeasible [2]. These prerequisites similarly constrain Lagrangian safe RL approaches [41], which require explicit barrier formulation and comprehensive state feedback for online multiplier learning.

Alternative approaches address safety through hardware redundancy, deploying ancillary backup sensors to mitigate single-point failures [3]. While effective for critical applications, redundancy multiplies instrumentation costs and maintenance burden, conflicting with sensor-lean operational imperatives in cost-sensitive manufacturing environments.

Our work addresses a complementary operational regime: sensor-lean, partially-observable industrial systems where CBF prerequisites are structurally incompatible with deployment constraints. Rather than deriving analytical barrier functions $h\left(s\right)$ from models and full state measurements, we learn implicit safety boundaries directly from limited observable state through extensive domain randomization spanning regime-dependent parameter variations. This paradigm shift—from analytical to learned barriers—enables formal constraint satisfaction (zero violations validated through sustained production deployment) in contexts where classical CBF methods cannot be applied. Table 1 contrasts these complementary approaches.

Table 1. Analytical CBF vs. learned implicit barriers: complementary paradigms.

Aspect	Analytical CBF	Learned Implicit (Ours)
Safety mechanism	$h\left(s\right)\ge 0$ derived analytically	Embedded in policy weights $\theta$
State requirements	Full observability of $h\left(s\right)$ variables	Partial observability (V, xs, xe)
Model dependency	Accurate $\dot{s}=f(s,u)$ required	Aggregate model + randomization
Runtime evaluation	$h\left(s\right)$ computed each cycle	No explicit barrier evaluation
Regime handling	Fixed $h\left(s\right)$ (single regime)	Learned robustness across regimes
Typical domains	Robotics, vehicles, power systems	Sensor-hostile process control
Guarantees	Formal (if model exact)	Empirical (via extensive validation)

2.8. Clean-in-Place System Optimization

CIP research has focused on flow modeling and cleaning kinetics [55], chemical dosing optimization [56], and energy-efficiency improvements. Jensen et al. [55] established critical flow velocity requirements (≥1.5 L/s) for turbulent cleaning effectiveness through wall shear stress analysis and developed CFD models predicting cleaning efficiency in complex geometries, while comprehensive treatment by Tamime [56] covers operational protocols and chemical selection strategies.

Existing control approaches rely on preprogrammed time-based sequences or simple PID loops lacking adaptability to equipment variations, fouling accumulation, or operational changes [57,58]. Fryer et al. [58] analyzed the physics and chemistry of cleaning but noted that conventional control strategies cannot adapt to time-varying conditions or circuit-specific hydraulic characteristics. No prior work has applied adaptive-learning-based control to CIP systems achieving autonomous multi-circuit operation with formal safety guarantees and sensor reduction.

2.9. Research Gaps and Positioning

Table 2 synthesizes the limitations of existing approaches and positioning of the proposed framework.

Table 2. Research gaps addressed by the proposed framework.

Approach	Limitation	This Work
Model-based (MPC)	Requires accurate models, fails under parameter drift	Adaptive learning from experience
RL (DQN/MADQN)	Lacks safety guarantees, sim-to-real gap	Multi-layer safety + curriculum transfer
Safe-RL frameworks	Theoretical, limited industrial validation	Validated production deployment with transferability
CBF-based safe MARL	Full observability + accurate model required	Learned implicit barriers under partial observability
State estimation (EKF)	Real-time dependency, sensor failures	Offline validation only, no control-loop dependency
Sensor reduction	Requires real-time estimation	Minimal state via comprehensive offline training
Curriculum learning	Limited industrial protocols	Systematic 4-stage curriculum with safety verification
MARL coordination	Assumes communication, dense feedback	Decentralized execution, sensor-lean operation
CIP optimization	Manual control, no adaptability	Autonomous multi-circuit generalization, zero retuning

This work uniquely integrates safety-aware multi-agent learning, curriculum-driven sim-to-real transfer, sensor-lean operation through offline validation, and architectural transferability to address the complete industrial deployment challenge—from simulation training to sustained production operation with formal safety guarantees and regulatory compliance. The component-based architecture enables systematic deployment across diverse process control domains while validated CIP implementation demonstrates practical viability in safety-critical food manufacturing environments.

3. System Architecture

Building upon the limitations identified in model-dependent and observer-reliant control strategies discussed in Section 2, this section presents the proposed component-based architecture that enables safe, adaptive, and reproducible control under minimal sensing and uncertain system dynamics. The design philosophy integrates three core dimensions: (i) a learning core based on a multi-agent deep Q-network (MADQN) for distributed actuation and adaptive decision making; (ii) a safety supervision layer ensuring feasibility and constraint satisfaction at both learning and deployment stages; and (iii) a nonintrusive EKF-based validation and auditing module that guarantees verifiable, post-deployment compliance with industrial safety and traceability requirements.

The overall architecture formalizes the methodological integration of these modules into a unified control framework. It ensures that operational decision making, safety governance, and experiential validation operate coherently without cross-dependence—bridging the gap between autonomous learning systems and the rigorous verification demanded by regulated industrial environments.

3.1. Architectural Overview

The system architecture (Figure 1) comprises six primary modules organized around separation of concerns: orchestration, curriculum management, environment simulation, agent learning, experience management, and offline validation. This modular design enables hardware-agnostic deployment and systematic adaptation to new domains through reconfiguration rather than redesign.

Figure 1. Modular architecture of the proposed multi-agent reinforcement learning framework. Components communicate through domain-agnostic interfaces, with the detailed workflow described in the following subsection.

The Training Orchestrator supervises workflow execution, managing hyperparameter schedules, curriculum progression, and lifecycle events through meta-level coordination signals. The Curriculum Manager dynamically adjusts environment complexity, disturbance profiles, and operational constraints to facilitate progressive learning from simple to complex scenarios [44,45]. The Environment Engine provides physics-agnostic simulation interfaces, abstracting domain-specific details to enable rapid adaptation across heterogeneous industrial processes.

The Agent Trainer implements the multi-agent learning core, coordinating distributed policies through shared objectives while maintaining decentralized execution capabilities [16]. Experience tuples are stored in the Shared Replay Buffer with dynamic prioritization emphasizing safety-critical transitions and rare events, ensuring balanced learning despite class imbalance [59]. The Model Registry handles versioning, checkpointing, and metadata tracking to support reproducibility and auditability throughout the model lifecycle.

A distinguishing feature is the Offline Validation Module, which performs post-deployment trajectory reconstruction and compliance verification using extended Kalman filter-based state estimation. Unlike traditional observer-based control where estimators operate within real-time loops, this module executes exclusively offline—eliminating real-time dependencies while maintaining comprehensive audit trails for regulatory compliance (FDA 21 CFR Part 11, ISO 9001, EHEDG guidelines [50,60,61]).

3.2. Architecture Data Flows

The modular architecture (Figure 1) employs domain-agnostic interfaces enabling deployment across diverse industrial applications:

Control coordination: The Training Orchestrator manages workflow execution (“Starts training”), monitors performance metrics to determine curriculum progression (“Monitors rewards and decides advancement”), and triggers model persistence at designated intervals (“Triggers model saving”).

Environment interaction cycle: The core training loop implements a standard reinforcement learning pattern: (1) Environment Engine provides system state observations (“Provides initial state”); (2) Agent Trainer computes control actions through learned policy networks (“Generates actions”); (3) Environment executes actions within the simulation model and returns transition tuples comprising next state, reward signal, and termination flag (“Returns next_state, reward, done”).

Experience-based learning: Agents store transition tuples $(s,a,r,s^{\prime },\mathrm{done})$ in a centralized Shared Experience Replay Buffer (“Stores experience”), which subsequently samples random mini-batches for off-policy learning (“Provides batch of experiences”). Neural network parameter updates occur via gradient-based optimization (“Updates neural network”), with periodic target network synchronization to stabilize training.

Curriculum adaptation: The Curriculum Manager dynamically adjusts environment complexity by modifying simulation parameters, disturbance profiles, and constraint tolerances (“Sends configuration to environment”). This progressive difficulty scaling enables agents to master foundational behaviors before encountering advanced scenarios.

Validation and deployment: Trained policies are serialized as model checkpoints (“Sends validated checkpoint”) and undergo domain-specific offline validation to verify operational compliance. The validation module generates certification reports (“Generates final report”) documenting performance metrics, constraint satisfaction statistics, and deployment readiness assessments.

3.3. Safety Integration Framework

Safety guarantees are achieved through four complementary mechanisms operating at different architectural layers, ensuring zero violations throughout training and deployment:

1. Constrained action projection: Proposed actions are projected onto feasible sets encoding physical, logical, and statistical constraints before execution. Given agent policy output $a_{\mathrm{raw}}$, the deployed action becomes $a_{\mathrm{safe}}={\Pi }_{{\mathcal{A}}_{\mathrm{safe}}\left(s\right)}\left(a_{\mathrm{raw}}\right)$, where $\Pi$ represents projection onto the constraint-satisfying subset [10]. This deterministic filtering guarantees hard constraint satisfaction independent of policy quality.

2. Prioritized safety-focused replay: Experienced buffer sampling overweights safety-critical transitions through adaptive priority scoring. Transitions approaching constraint boundaries or exhibiting high-temporal-difference errors receive 5–10× sampling probability, reinforcing safe behavior learning without requiring manual balancing [59].

3. Conservative training margins: Training constraints are tightened 20% relative to deployment specifications, creating safety buffers accommodating model uncertainty and sim-to-real transfer gaps. This conservative approach ensures that policies satisfying training constraints maintain substantial margins during physical deployment.

4. Curriculum-embedded verification: Progression to subsequent curriculum stages requires demonstrated constraint satisfaction across validation test suites. Policies failing to maintain zero violations under stage-specific stress tests do not advance, ensuring safety-aware capability development throughout training.

This multi-layer approach provides defense-in-depth: even if individual mechanisms exhibit imperfect performance, their combination ensures comprehensive safety coverage validated through sustained production deployment (Section 7).

3.4. Multi-Agent Coordination

The framework supports scalable N-agent configurations coordinating through shared reward signals and joint constraint satisfaction. Agents learn complementary policies via centralized training with decentralized execution (CTDE) [16]: a centralized critic captures inter-agent dependencies during training, while deployment uses independent actor networks, requiring no communication infrastructure.

For the CIP validation testbed (Section 5), dual agents coordinate inlet/outlet pump control through shared system-level objectives. Coordination emerges implicitly through reward structure rather than explicit communication protocols, enabling robust operation despite communication failures or network degradation. Measured flow-rate correlations (Pearson $r=-0.36$ to $-0.63$) demonstrate emergent cooperative behavior without hard-coded coordination rules.

The CTDE architecture generalizes to arbitrary agent counts: scaling from 2 to N agents requires only configuration changes (state/action dimensions, network sizes) without algorithmic modifications. This scalability enables deployment across diverse multi-actuator configurations through systematic reconfiguration procedures.

3.5. Curriculum Learning Protocol

Training progresses through a structured four-stage curriculum, systematically increasing operational complexity, disturbance intensity, and coordination requirements:

• Stage 1 (Foundation): Deterministic nominal conditions establish baseline control competency (10K episodes).
• Stage 2 (Robustness): Bounded parameter uncertainty (±20%) and moderate disturbances develop robustness (15K episodes).
• Stage 3 (Coordination): Multi-agent interactions and topology variations enable adaptive collaboration (20K episodes).
• Stage 4 (Mastery): Full stochastic environment with extreme disturbances (±50%) and rare fault scenarios refine edge-case handling (25K episodes).

Stage progression follows gated advancement: policies must achieve zero safety violations across 500-episode validation tests before advancing. Reward weights dynamically adjust across stages, emphasizing safety during early training (${\alpha }_{\mathrm{safety}}=0.5$) and efficiency during mastery (${\alpha }_{\mathrm{efficiency}}=0.4$). This structured approach achieves 85–92% sim-to-real performance retention without manual fine-tuning (Section 7).

3.6. Offline Validation and Auditing

The validation framework operates independently from real-time control, providing post-deployment trajectory reconstruction and compliance verification. An extended Kalman filter processes recorded operational data to reconstruct unmeasured states, enabling comprehensive performance assessment without real-time instrumentation requirements.

For state vector $x_t$ and measurement $y_t$, the EKF performs prediction and correction steps:

$$\begin{array}{cc}\hfill {\widehat{x}}_{t|t-1}& =f({\widehat{x}}_{t-1},u_{t-1}),\hfill \\ \hfill {\widehat{x}}_{t|t}& ={\widehat{x}}_{t|t-1}+K_t(y_t-h\left({\widehat{x}}_{t|t-1}\right)),\hfill \end{array}$$

(1)

where $f(·)$ represents system dynamics, $h(·)$ the measurement model, and $K_t$ the Kalman gain. Reconstruction accuracy of 91–96% with convergence times of 30–45 s validates state estimation reliability for offline auditing purposes.

The EKF recursion requires specification of initial state estimate ${\widehat{x}}_0$ and error covariance $P_0$. Initialization typically employs available sensor measurements at operational cycle start (${\widehat{x}}_0\sim \mathcal{N}({\overline{x}}_0,P_0)$), with covariance $P_0$ reflecting measurement uncertainty and unmeasured state reconstruction confidence prior to filter convergence. Domain-specific initialization protocols and convergence characteristics are described in Section 5. Validation metrics include constraint-violation frequency, reconstruction-error variance, and policy traceability indices linking checkpoints to training datasets and environmental configurations. Validation reports integrate with industrial quality management systems, providing auditable evidence chains supporting regulatory compliance (FDA cGMP, ISO 9001, EHEDG guidelines [50,60,61]).

This offline-only validation approach eliminates real-time estimation dependencies characteristic of observer-based control, fundamentally altering failure modes: estimator divergence affects audit quality but cannot compromise control safety. The Model Registry maintains versioned links between policy checkpoints, validation certificates, and dataset hashes, ensuring only certified policies transition to production deployment.

3.7. Architectural Transferability

The component-based design abstracts domain-specific details into configurable parameters, enabling systematic adaptation across applications. Core components (orchestration, curriculum, safety, validation) remain unchanged across domains; adaptation involves reconfiguring:

• State representation: Dimension, normalization, observation windows.
• Action space: Actuator types, discretization, feasibility constraints.
• Reward structure: Objective weights, penalty functions, target ranges.
• Safety constraints: Physical limits, logical interlocks, statistical bounds.

Validation through three diverse CIP hydraulic architectures (complexity 2.1–8.2/10) demonstrates zero-reconfiguration transfer, achieving sustained production operation without manual retuning (Section 7). This architectural transferability extends to broader process control applications sharing common characteristics: multi-actuator coordination, safety-critical operation, and sensor-lean requirements.

4. Problem Formulation and Mathematical Framework

This section formalizes the multi-agent fault-tolerant control problem as a constrained Markov decision process (CMDP) with curriculum-structured learning. We present the mathematical foundations underlying the safety-aware MADQN architecture, including multi-objective reward design, constraint formulation, curriculum progression mechanisms, and multi-agent coordination objectives.

4.1. Constrained Markov Decision Process Formulation

The control problem is formalized as a finite-horizon constrained Markov decision process (CMDP) defined by the tuple

$$\mathcal{M}=⟨\mathcal{S},\mathcal{A},P,r,\gamma ,\mathcal{C}⟩,$$

(2)

where $\mathcal{S}$ represents the state space, $\mathcal{A}$ the joint action space, $P:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to [0,1]$ the state transition probability function, $r:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ the reward function, $\gamma \in [0,1)$ the discount factor, and $\mathcal{C}=\{c_1,\dots ,c_m\}$ the set of safety constraints [40,41].

For multi-agent systems with N agents, the joint action space decomposes as $\mathcal{A}={\mathcal{A}}_1\times \cdots \times {\mathcal{A}}_N$, where each agent i selects actions from ${\mathcal{A}}_i$ according to its policy ${\pi }_i:\mathcal{S}\to \Delta \left({\mathcal{A}}_i\right)$, with $\Delta \left({\mathcal{A}}_i\right)$ denoting the probability simplex over ${\mathcal{A}}_i$. The joint policy is denoted by $\pi =({\pi }_1,\dots ,{\pi }_N)$.

The objective is to find an optimal joint policy ${\pi }^{*}$ maximizing expected cumulative discounted reward:

$$J\left(\pi \right)={\mathbb{E}}_{\tau \sim \pi }\left[\sum _{t=0}^T{\gamma }^{t}r(s_t,a_t)\right],$$

(3)

subject to safety constraints satisfied almost surely:

$$\mathbb{P}\left(c_j(s_t,a_t)\le 0,\forall t,\forall j\in \{1,\dots ,m\}\right)=1,$$

(4)

where $a_t=(a_t^1,\dots ,a_t^N)$ represents the joint action at time t, and $\tau =(s_0,a_0,s_1,a_1,\dots )$ denotes a trajectory sampled under policy $\pi$.

4.2. State and Action Space Definitions

4.2.1. State Space

The system state $s_t\in \mathcal{S}$ at time t aggregates observable process variables, actuator states, and temporal context:

$$s_t=\left[\begin{array}{c}{x}_t\\ u_{t-1}\\ h_t\end{array}\right],$$

(5)

where $x_t\in {\mathbb{R}}^{n_x}$ represents measured process states (e.g., tank levels, flow rates, temperatures), $u_{t-1}\in {\mathbb{R}}^{n_u}$ denotes previous control actions, and $h_t\in {\mathbb{R}}^{n_h}$ captures temporal features (e.g., operation phase, time-since-last-action). State normalization ensures numerical stability:

$${\tilde{x}}_i={\displaystyle \frac{x_i-{\mu }_i}{{\sigma }_i}},i=1,\dots ,n_x,$$

(6)

where ${\mu }_i$ and ${\sigma }_i$ represent mean and standard deviation computed from training data.

4.2.2. Action Space

For discrete control problems, each agent i selects actions from a finite set ${\mathcal{A}}_i=\{a_i^{\left(1\right)},\dots ,a_i^{\left(K_i\right)}\}$, where $K_i$ denotes the number of discrete action choices. In the CIP validation testbed, actions represent pump speed adjustments discretized into $K=5$ levels: {OFF, LOW, MEDIUM, HIGH, and MAX}, corresponding to normalized frequencies $\{0,0.25,0.50,0.75,1.0\}$.

The feasible action set at state $s_t$ is constrained by safety projections:

$${\mathcal{A}}_{\mathrm{safe}}\left(s_t\right)=\{a\in \mathcal{A}\mid c_j(s_t,a)\le 0,\forall j\in \{1,\dots ,m\}\}.$$

(7)

4.3. Multi-Objective Reward Function

Control performance is optimized through a composite reward function balancing multiple operational objectives. The total reward at each transition $(s_t,a_t,s_{t+1})$ is expressed as

$$R_{\mathrm{total}}(s_t,a_t,s_{t+1})=\sum _{i=1}^4{\alpha }_i\left(\kappa \right)R_i(s_t,a_t,s_{t+1}),$$

(8)

where ${\alpha }_i\left(\kappa \right)\ge 0$ represents the stage-dependent weight for objective i during curriculum stage $\kappa \in \{1,2,3,4\}$, and ${\sum }_i{\alpha }_i\left(\kappa \right)=1$ ensures normalization.

The four reward components are defined as follows:

1. Operational Consistency Reward ($R_1$):

$$R_1(s_t,a_t,s_{t+1})=-\sum _{k=1}^{n_x}{w}_k{\left|x_{t+1}^{\left(k\right)}-x_{\mathrm{target}}^{\left(k\right)}\right|}^2,$$

(9)

where $x_{t+1}^{\left(k\right)}$ denotes the k-th state variable after transition, $x_{\mathrm{target}}^{\left(k\right)}$ the desired setpoint, and $w_k>0$ the relative importance weight.

2. Constraint Satisfaction Reward ($R_2$):

$$R_2(s_t,a_t,s_{t+1})=\left\{\begin{array}{cc}-{\lambda }_{\mathrm{viol}}\sum _{j=1}^{m}max{(0,c_j(s_{t+1},a_t))}^2,\hfill & \mathrm{if}\mathrm{violation},\hfill \\ +r_{\mathrm{safe}},\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(10)

where ${\lambda }_{\mathrm{viol}}\gg 1$ represents a large penalty coefficient (typically 100–1000), and $r_{\mathrm{safe}}>0$ provides positive reinforcement for constraint-compliant behavior.

3. Control Smoothness Reward ($R_3$):

$$R_3(s_t,a_t,s_{t+1})=-\beta \sum _{i=1}^N{\parallel a_t^i-a_{t-1}^i\parallel }_2^2,$$

(11)

where $\beta >0$ controls the smoothness penalty strength and N denotes the number of agents.

4. Resource Efficiency Reward ($R_4$):

$$R_4(s_t,a_t,s_{t+1})=-\eta \sum _{i=1}^N{\parallel a_t^i\parallel }_1,$$

(12)

where $\eta >0$ balances efficiency against other objectives.

Stage-Dependent Weight Adaptation:

$$\alpha \left(\kappa \right)=\left\{\begin{array}{cc}[0.3,0.5,0.15,0.05],\hfill & \kappa =1\left(\mathrm{Foundation}\right),\hfill \\ [0.35,0.4,0.15,0.10],\hfill & \kappa =2\left(\mathrm{Robustness}\right),\hfill \\ [0.40,0.3,0.20,0.10],\hfill & \kappa =3\left(\mathrm{Coordination}\right),\hfill \\ [0.45,0.2,0.15,0.20],\hfill & \kappa =4\left(\mathrm{Mastery}\right).\hfill \end{array}\right.$$

(13)

4.4. Safety Constraint Formulation

Safety constraints encode physical limits, logical interlocks, and operational requirements. For the CIP validation testbed, constraints include the following:

1. Flow Rate Constraints:

$$c_1(s,a)=q_{min}-q(s,a),c_2(s,a)=q(s,a)-q_{max},$$

(14)

where $q(s,a)$ denotes resulting flow rate, and $[q_{min},q_{max}]=[1.5,3.0]$ L/s defines feasible range [55].

2. Volume Constraints:

$$c_3(s,a)=V_{min}-V\left(s\right),c_4(s,a)=V\left(s\right)-V_{max},$$

(15)

where $V\left(s\right)$ represents tank volume and $[V_{min},V_{max}]=[100,200]$ L defines safe operating range.

Conservative Training Margins:

$$c_j^{\mathrm{train}}(s,a)=c_j(s,a)+\delta ·\left|c_j^{\mathrm{nominal}}\right|,$$

(16)

where $\delta =0.2$ creates 20% safety buffers during training.

4.5. Curriculum Learning Formalization

Parameter uncertainty varies across stages:

$$p\sim {\mathcal{D}}_{\kappa }=\left\{\begin{array}{cc}{\delta }_{p_0},\hfill & \kappa =1,\hfill \\ \mathcal{U}(0.8p_0,1.2p_0),\hfill & \kappa =2,\hfill \\ \sum _{\omega \in \Omega }{\pi }_{\omega }{\mathcal{D}}_{\omega },\hfill & \kappa =3,\hfill \\ \mathcal{U}(0.5p_0,1.5p_0),\hfill & \kappa =4,\hfill \end{array}\right.$$

(17)

where ${\delta }_{p_0}$ denotes Dirac delta at nominal parameters.

Stage progression criterion:

$$\mathrm{Advance}\mathrm{if}\sum _{e=1}^{N_{\mathrm{val}}}\sum _{t=0}^{T_e}⊮[\exists j:c_j(s_t^e,a_t^e)>0]=0.$$

(18)

4.6. Multi-Agent Coordination Objective

Joint learning objective under CTDE [16]:

$$J(\Theta )={\mathbb{E}}_{\tau \sim {\pi }_{\Theta }}\left[\sum _{t=0}^T{\gamma }^t\left(r(s_t,a_t)-{\lambda }_c\sum _{i\ne j}{\parallel a_t^i-a_t^j\parallel }_2\right)\right],$$

(19)

where $\Theta =\{{\theta }_1,\dots ,{\theta }_N\}$ and ${\lambda }_c>0$ penalizes coordination conflicts.

4.7. Implicit Barrier Learning Under Partial Observability

While the constrained MDP formulation (Equation (2)) provides a formal framework for safety-critical control, its practical implementation in sensor-lean industrial systems requires addressing the fundamental challenge of constraint evaluation under partial observability.

4.7.1. The Partial Observability Challenge

Traditional safe RL approaches, including CBF-based methods, assume that constraint functions $c_j(s,a)$ can be accurately evaluated at runtime. For example, flow rate constraints (Equation (14)) require knowledge of $q(s,a)$, which in hydraulic systems depends on pump effectiveness parameters:

$$q(s,a)=P_e\left(s\right)·x_s·(1-k_1{x}_e)-P_s\left(s\right)·x_e·(1-k_2{x}_s)$$

(20)

where $P_e\left(s\right)$, $P_s\left(s\right)$ represent regime-dependent pump effectiveness modified by bubble entrainment, temperature, wear, and other unobservable factors. In sensor-lean deployments:

• q cannot be measured directly (flow sensors eliminated for cost/contamination reasons).
• $P_e$, $P_s$ cannot be estimated reliably in real time (30–45 s convergence, unacceptable for 1 s control cycles).
• Regime classification requires instrumentation incompatible with sensor-lean objectives.

4.7.2. Implicit Constraint Encoding

Rather than attempting explicit constraint evaluation $c_j(s,a)\le 0$ at runtime, the proposed framework encodes constraint satisfaction implicitly within policy weights $\theta$ through three complementary mechanisms:

1. Domain Randomization as Implicit Barrier:

Training with wide parameter variations $P_e\in [3.0, 7.0]$, $P_s\in [2.6, 5.8]$ (Equation (17), stage $\kappa =4$) exposes the policy to synthetic regime diversity. The learned Q-function approximates the following:

$$Q_{\theta }(s,a)\approx \underset{P_e\in [3,7],P_s\in [2.6, 5.8]}{min}{\mathbb{E}}_{\tau }\left[R\right|{\pi }_{\theta },P_e,P_s]$$

(21)

This worst-case expectation implicitly encodes conservative safety margins without requiring explicit $P_e$, $P_s$ identification.

2. Safety-Focused Prioritization:

Equation (25) oversamples transitions approaching constraint boundaries by 5–10×, reinforcing avoidance behaviors even when exact constraint violation distances cannot be computed due to parameter uncertainty.

3. Conservative Training Margins:

Equation (16) tightens constraints by 20% during training. This buffer compensates for the following:

• Model mismatch between aggregate simulation (Equation (20)) and regime-dependent reality (Equation (26)).
• Sim-to-real transfer gaps.
• Unmodeled dynamics (e.g., valve hysteresis, pipe compliance).

4.7.3. Formal Safety Under Learned Barriers

While analytical CBF provides guarantees of the form

$$h\left(s\right)\ge 0\wedge \dot{h}(s,a)\ge -\alpha h\left(s\right)\Rightarrow s\left(t\right)\in {\mathcal{C}}_{\mathrm{safe}}\forall t$$

(22)

our approach achieves empirical safety through extensive validation:

$$\mathbb{P}\left[\mathrm{violation}\right]\le {\displaystyle \frac{n_{\mathrm{viol}}}{n_{\mathrm{total}}}}=0(1767\mathrm{deployments},70\mathrm{K}+\mathrm{training})$$

(23)

This trade-off—formal guarantees relative to a model vs. empirical guarantees from extensive data—is particularly suited for industrial regimes where model fidelity assumptions underlying analytical barriers cannot be satisfied. Section 7 validates zero violations across three hydraulic configurations spanning complexity 2.1–8.2, diverse parameter regimes, and sustained production operation (over 6 months accumulated).

The key distinction is that rather than deriving $h\left(s\right)$ analytically and evaluating it at runtime, safety boundaries are learned and embedded in $Q_{\theta }(s,a)$ through curriculum progression (Equation (17)), enabling constraint satisfaction without comprehensive state measurements or real-time parameter estimation.

4.8. Prioritized Experience Replay

Sampling probability:

$$P\left(i\right)={\displaystyle \frac{p_i^{\alpha }}{{\sum }_k{p}_k^{\alpha }}},$$

(24)

with priority combining TD error and safety criticality:

$$p_i=\eta \left(\right|{\delta }_i{|+ϵ)}^{\alpha }+(1-\eta )w_{\mathrm{crit}}{⊮}_{\mathrm{critical}},$$

(25)

where ${\delta }_i$ is the temporal-difference error, ${⊮}_{\mathrm{critical}}$ flags safety-critical transitions, and $\eta \in [0,1]$ balances TD-based and criticality-based prioritization [59].

5. CIP Systems as Validation Testbed

5.1. Industrial Context and Problem Description

Clean-in-place (CIP) systems automate equipment sanitization without disassembly, critical for hygienic manufacturing in food, beverage, and pharmaceutical industries. CIP operations circulate cleaning solutions (acid pH 1–2, caustics pH 12–14, sanitizers, rinse water) through production equipment under controlled flow, temperature, and chemical concentration conditions. Cleaning effectiveness derives from Sinner’s framework—chemical action, temperature, contact time, and mechanical force delivered exclusively through flow-induced wall shear stress in closed systems.

The framework was validated at VivaWild Beverages (Colima, Mexico), a commercial facility producing preservative-free juices and smoothies operating under FDA 21 CFR Part 11 [61], EHEDG hygienic design [50], and 3-A Sanitary Standards [62] ensuring hygienic equipment design for food contact surfaces. The plant-wide CIP infrastructure serves all production equipment including UHT pasteurization systems, storage tanks, mixing vessels, and filling lines through centralized chemical supply (acid, caustic, sanitizer, rinse water) and automated valve manifolds selectively routing solutions to multiple circuits (Figure 2).

Figure 2. Clean-in-place (CIP) system architecture comprising four chemical tanks (acid, alkaline, sanitizer, rinse water), dual-pump configuration (inlet/outlet agents), valve selection network, and multiple production circuits. The MADQN framework controls inlet and outlet pumps (shown in yellow), coordinating flow distribution across diverse hydraulic configurations including storage tanks, mixing vessels, UHT circuits, and process lines. Valve states and tank selection define circuit topology, enabling validation across a complexity range of 2.1–8.2/10 without controller reconfiguration.

5.2. Safety-Critical Control Requirements

CIP systems must simultaneously satisfy two hard safety constraints with zero-tolerance failure acceptance:

Constraint 1: Critical flow rate ($Q\ge 1.5$ L/s): Minimum turbulent flow ensures wall shear stress sufficient for soil removal. Flow below the threshold causes transition to laminar regime, dramatic shear stress reduction, biofilm formation in geometric disturbances (elbows, valves, dead-legs), and cleaning failure leading to microbiological contamination.

Constraint 2: Volume management ($V\in [100,200]$ L): The optimal band of 130–170 L ensures complete circuit flooding without air pockets, prevents pump cavitation and air entrainment degrading flow performance, maintains stable recirculation, and protects equipment integrity.

Failure consequences: Cleaning validation failure per FDA/EHEDG standards, mandatory recleaning (2–4 h lost production), potential batch disposal (USD 10,000–50,000), regulatory violations (FDA Warning Letters, fines USD 50,000–500,000), and manufacturing suspension in severe cases.

5.3. Sensor-Lean Operational Imperatives

Wetted instrumentation in CIP environments faces four critical challenges, making sensor-lean approaches technically necessary:

1. Hygienic design constraints: Sensor penetrations create contamination pathways through dead zones, seal interfaces, and mounting pockets harboring biofilm between production runs. Each wetted sensor installation requires rigorous cleaning validation as per 3-A/EHEDG standards. Sensor removal for maintenance breaks process containment, creating environmental contamination ingress opportunities.

2. Calibration complexity: CIP cycles sequentially circulate water, caustics, acid, and sanitizer—each with dramatically different density, viscosity, and conductivity. Flowmeters calibrated for one fluid exhibit 15–30% accuracy drift across others [63,64,65]. Temperature ranges (20–85 °C) alter water density by 4% and viscosity by 80%, directly affecting meter response. Maintaining accuracy requires fluid-specific calibration curves or periodic recalibration with each chemistry—and substantial engineering overheads.

3. Accelerated degradation: Concentrated acids/caustics cycle continuously, causing electrode corrosion, seal degradation, and coating leaching. Thermal cycling (20–120 °C, 10–15 cycles daily) induces mechanical stress and accelerated fatigue. Industry data show that wetted flowmeters/transmitters exhibit 18–36 months MTBF versus 60–84 months in non-CIP services—50–70% MTBF reduction, driving proportional maintenance increases [66].

4. Economic impact: Quantified costs per circuit include capital expenditure for sanitary flowmeters/transmitters (USD 8000–15,000), periodic replacement cycles (USD 4000–8000 annually), calibration and maintenance overhead (USD 3000–6000 annually), and downtime costs from unplanned failures (USD 5000–12,000 annually). Total instrumentation costs reach USD 20,000–40,000 per circuit annually, representing 15–25% of total CIP operating expenses [66,67].

5.4. Hydraulic Architecture Diversity

Three production circuits with varying complexity validate architectural transferability:

UHT pasteurization circuit (complexity: 8.2/10): High-temperature, short-time (HTST) processing with plate heat exchangers, holding tubes, aseptic surge tanks, and temperature-controlled recirculation loops. Complex topology with multiple branches, 15+ valves, thermal expansion effects, and strict temperature-flow coupling. Calibrated parameters: $P_e=4.2$, $P_s=3.8$, $R^2=0.89$.

Storage tank circuit (complexity: 2.1/10): Simple configuration with direct tank-to-tank transfer via dedicated pipelines. Minimal branching, 4 valves, straightforward hydraulics. Calibrated parameters: $P_e=5.8$, $P_s=2.6$, $R^2=0.94$.

Mixing vessel circuit (complexity: 5.7/10): Moderate complexity with blending tanks, jacketed vessels, spray balls, and manifold distribution. Multiple inlets/outlets, 8–10 valves, temperature-controlled zones. Calibrated parameters: $P_e=4.9$, $P_s=3.2$, $R^2=0.91$.

Regime-Dependent Parameter Variations:

The calibrated parameters ($P_e$, $P_s$) reported above represent nominal values under ideal conditions (fresh equipment, no bubbles, moderate temperature). In practice, these parameters exhibit substantial regime-dependent variations:

• Bubble entrainment: Air incorporation during tank-to-tank transfers or high-speed filling reduces $P_e$ by 20–35%, creating effective values $P_e^{\mathrm{eff}}\in [2.7, 4.6]$ for nominal $P_e=4.2$.
• Temperature effects: Caustic cleaning at 80–85 °C alters fluid viscosity and pump performance, shifting $P_e$, $P_s$ by ±15%.
• Equipment aging: Pump wear, valve seat degradation, and pipe fouling cause continuous parameter drift: measured $P_e$ degradation 10–15% over 3–6 months in UHT circuit.
• Chemistry-specific response: Density/viscosity differences between water, caustics (pH 13, 1.02–1.05 g/cm³), and acid (pH 2, 1.01–1.03 g/cm³) modify flow characteristics despite nominally constant pump speeds.

These variations exemplify the regime-switching dynamics discussed in Section 3.6: no single $\theta$ vector accurately captures system behavior across all operating conditions. Rather than attempting real-time regime classification or parameter adaptation, the MADQN framework trains with $P_e\in [3.0, 7.0]$, $P_s\in [2.6, 5.8]$ (domain randomization spanning observed variability plus safety margins), learning policies robust to this parameter uncertainty without requiring explicit $P_e$, $P_s$ measurement or estimation during deployment.

5.4.1. Transient vs. Progressive Parameter Variations

Parameter variations occur at two distinct timescales, both addressed through domain randomization:

Transient disturbances (seconds to minutes):

• Bubble entrainment: Air incorporation during tank-to-tank transfers or high-speed filling reduces $P_e$ by 20–35%, creating effective values $P_e^{\mathrm{eff}}\in [2.7, 4.2]$ for nominal $P_e=4.2$. Turbulent flow ($Q\ge 1.5$ L/s) purges entrained air within 20–60 s, restoring $P_e$ to baseline without intervention.
• Cold starts: System initialization at ambient temperature (20–25 °C) exhibits reduced $P_e$ (10–15% below nominal) due to increased fluid viscosity and thermal contraction effects. Thermal equilibrium is reached within 2–5 min as recirculation warms the system to operating temperature (40–60 °C for water, 80–85 °C for caustics).
• Chemistry transitions: Sequential circulation of water (density 1.00 g/cm³, viscosity 1.0 cP), caustics (1.02–1.05 g/cm³, 1.2–1.8 cP), and acid (1.01–1.03 g/cm³, 0.9–1.1 cP) causes viscosity-driven $P_e$ shifts (±10–20%) during 30–90 s adaptation periods as new fluid displaces previous chemistry.

These perturbations are reversible without human intervention, self-correcting through normal operational dynamics (air purging, thermal equilibration, fluid displacement).

Progressive degradation (months):

• Pump wear: Impeller erosion and seal degradation reduce volumetric efficiency 8–12% over 6 months under aggressive CIP duty cycles (10–15 cycles daily, caustics/acid exposure).
• Valve degradation: Seat wear and actuator backlash increase flow resistance and reduce position accuracy, manifesting as a 3–5% $P_e$ reduction over 3–6 months.
• Fouling accumulation: Mineral deposits (calcium carbonate, magnesium hydroxide) and organic residues accumulate in low-flow zones (pipe elbows, valve bodies), increasing hydraulic resistance by 5–8% over 6 months despite regular CIP cleaning.

Cumulative progressive degradation totals a 10–15% $P_e$ reduction over 3–6 months, irreversible without maintenance intervention (component replacement, descaling procedures).

Dual Timescale Robustness:

Domain randomization spanning $P_e\in [3.0, 7.0]$, $P_s\in [2.6, 5.8]$ (±40% nominal) enables the MADQN policy to handle both timescales autonomously:

• Transient disturbances: These fall within trained parameter distribution and are managed in real time (1 s control cycles) without requiring parameter estimation or human intervention. This provides critical advantage over adaptive control methods requiring 30–45 s estimator convergence—too slow to respond to 20–60 s transient events.
• Progressive degradation: This remains within the training envelope for 12–24+ months (assuming 10–15% degradation per 6 months, which extrapolates to 20–40% over 12–24 months), deferring retraining requirement until drift approaches ± 40% bounds or performance degradation (increased control effort, slower convergence) becomes operationally significant.

Current deployment of 6 months shows no performance degradation trends across both disturbance types: 200+ observed bubble entrainment events during tank transfers, 150+ cold start cycles during morning system initialization, 40+ chemistry transition sequences per week, and continuous progressive degradation (measured $P_e$ reduction UHT circuit: 11.3% over 5 months, mixing circuit: 8.7% over 5 months). This dual robustness—rapid transient handling + long-term drift tolerance—validates the learned implicit safety approach for sensor-lean industrial deployment where real-time parameter estimation is infeasible.

The 4× complexity range (2.1–8.2) provides a rigorous testbed for zero-retuning generalization validation, representative of typical plant-wide architectural diversity in industrial facilities.

5.4.2. EKF Initialization for CIP Validation

For offline CIP trajectory reconstruction, the EKF state vector comprises tank volume and inlet/outlet flow rates: ${\widehat{x}}_0={[V_0,Q_{s,0},Q_{e,0}]}^T$. Initialization employs measured tank volume at cycle start ($V_0\in [130, 170]$ L within optimal operating band) and zero flow rates ($Q_{s,0}=Q_{e,0}=0$ L/s, system at rest). Initial covariance $P_0=\mathrm{diag}\left([25, 0.04, 0.04]\right)$ reflects volume sensor precision (${\sigma }_V=5$ L) and expected flow reconstruction uncertainty (${\sigma }_Q=0.2$ L/s) prior to convergence. Validation experiments (Section 7) demonstrate 30–45 s convergence times, achieving 91–96% reconstruction accuracy.

5.5. Implicit Learning Under Partial Observability and Regime-Dependent Dynamics

A distinguishing characteristic of the proposed framework is its approach to safety under partial observability and regime-dependent system dynamics—conditions prevalent in harsh industrial environments where comprehensive state measurement and accurate modeling are structurally infeasible.

5.5.1. Regime-Dependent Hydraulic Dynamics

Comprehensive system identification of CIP hydraulic circuits reveals that dynamics are not governed by fixed parameters but exhibit regime-dependent behavior. A detailed model capturing cross-coupling and nonlinear saturation effects takes the following form:

$$\begin{array}{cc}\hfill V(k+1)={\theta }_0& +{\theta }_{1}V\left(k\right)+{\theta }_2{x}_e\left(k\right)+{\theta }_3{x}_s\left(k\right)\hfill \\ \hfill & +{\theta }_4{x}_e\left(k\right)V\left(k\right)+{\theta }_5{x}_s\left(k\right)V\left(k\right)\hfill \\ \hfill & +{\theta }_6{x}_e\left(k\right)x_s\left(k\right)+{\theta }_7{x}_s^2\left(k\right)\hfill \\ \hfill & +{\theta }_{8}tanh\left(2x_e\left(k\right)\right)+{\theta }_{9}tanh\left(2x_s\left(k\right)\right)\hfill \end{array}$$

(26)

where parameter vector $\theta ={[{\theta }_0,{\theta }_1,\dots ,{\theta }_9]}^T$ varies significantly across operating regimes defined by

• Volumetric zones: Low fill ($V<120$ L), mid-range ($120\le V\le 180$ L), high fill ($V>180$ L) exhibit different coupling strength.
• Flow regimes: Laminar-turbulent transitions at junctions alter effective pump characteristics.
• Physical state: Bubble entrainment (0–30%), temperature (20–120 °C), and viscosity modify $\theta$ by 20–50%.
• Equipment condition: Pump wear, valve degradation, and fouling cause continuous parameter drift (10–15% monthly).

Extensive identification campaigns across multiple operating points revealed no universal parameterization: fitting Equation (26) to data spanning full operational envelope yields poor fit ($R^2<0.65$) due to regime-switching nonlinearities. Reasonable fit ($R^2>0.85$) is achieved only when restricting to narrow regimes (e.g., $V\in [140, 160]$ L, no bubbles, fresh equipment), but these local models fail outside their identification domains.

5.5.2. Implications for Control Design

This regime-dependent reality has critical consequences for safety-critical control design:

• Analytical CBF infeasible: Deriving barrier functions $h\left(s\right)=f(Q,V,\theta )$ requires knowing which regime applies and having accurate $\theta \left(r\right)$ for that regime—neither available at runtime in sensor-lean deployment. Using nominal $\theta$ yields safety margins valid only for that specific regime, potentially unsafe in others.
• Adaptive estimation fragile: Real-time $\theta$ estimation via EKF or recursive least squares requires persistent excitation and converges slowly (30–45 s), unacceptable for safety-critical constraints with 1 s control cycles. Estimator convergence cannot be guaranteed under operational transients.
• Multiple regime switching: Attempting to classify current regime and switch between regime-specific models introduces mode-switching stability issues and requires dense instrumentation for reliable classification—conflicting with sensor-lean imperatives.

5.5.3. Learned Implicit Safety via Domain Randomization

Rather than attempting explicit regime identification or adaptive parameter estimation, the proposed framework learns regime-invariant safety policies through extensive domain randomization. The aggregate first-order model (Equations (5) and (6) in Section 5) intentionally simplifies dynamics:

$${\displaystyle \frac{dV}{dt}}=P_e·x_s·(1-0.05x_e)-P_s·x_e·(1-0.03x_s)$$

(27)

where $P_e$, $P_s$ are aggregate effectiveness parameters acting as statistical proxies capturing variability from all regime-dependent phenomena. Training with $P_e\in [3.0, 7.0]$, $P_s\in [2.6, 5.8]$ (±40% variation) exposes the policy to synthetic regime diversity without explicit regime modeling.

The policy learns conservative actions safely across the entire parameter envelope:

$${\pi }_{\theta }(V,x_s,x_e)\approx arg\underset{a}{max}{\mathbb{E}}_{P_e\sim [3,7],P_s\sim [2.6, 5.8]}\left[Q(s,a)\right]$$

(28)

This implicit approach provides three advantages:

• No regime classification required: Policy functions under all regimes without knowing which applies.
• Robustness to drift: Parameter changes due to aging/fouling are covered by training randomization.
• Sensor-lean compatible: Operates solely from observable state $[V,x_s,x_e]$ without flow/pressure measurements.

Validation across three hydraulic configurations (Section 7) with distinct regime characteristics (complexity 2.1–8.2) confirms zero safety violations despite regime-dependent dynamics, demonstrating that learned implicit safety via domain randomization achieves comparable constraint satisfaction to explicit CBF methods while maintaining robustness under model uncertainty.

This positions our framework as particularly suited for regime-switching industrial systems where comprehensive parameterization is infeasible—a common characteristic of hydraulic, thermal, and chemical processes operating under harsh conditions with equipment aging and sensor-hostile environments [1,2].

5.6. CIP as Representative Industrial Testbed

The CIP validation domain exhibits all target challenges establishing architectural generalizability:

Hard safety constraints: Dual zero-tolerance objectives (flow $\ge 1.5$ L/s, volume bounds) with severe failure consequences—representative of pharmaceutical batch control, chemical reactor management, and critical fluid handling across industries.

Sensor-lean requirements: Harsh chemical/thermal environment degrading wetted instrumentation, hygienic design imperatives minimizing contamination pathways—characteristic of food processing, sterile manufacturing, and clean-room automation.

Multi-configuration complexity: Diverse hydraulic architectures requiring unified control strategy without per-circuit retuning—analogous to multi-unit process plants and distributed manufacturing systems.

Economic criticality: Documented costs (USD 20,000–40,000 annually per circuit) and downtime penalties (USD 10,000–50,000 per failure) establishing a quantifiable business case—essential for industrial technology adoption across sectors.

Regulatory compliance: FDA 21 CFR Part 11, ISO 9001, EHEDG validation requirements demanding comprehensive audit trails—transferable to pharmaceutical GMP, aerospace AS9100 [68], and automotive ISO 26262 [69] domains requiring similar documentation rigor.

Demonstrating stable, certified performance under these conditions provides both domain-specific solutions and evidence of transferable architectural methodology applicable across industries confronting similar adaptive control, safety assurance, and sensor-lean operation challenges. The following sections detail the experimental methodology (Section 6), validation results (Section 7), and performance analysis (Section 8).

6. Experimental Setup and Validation Protocol

This section describes the industrial facility, implementation methodology, and validation framework employed to demonstrate MADQN architecture viability through controlled stress-test campaigns under authentic industrial conditions.

6.1. Industrial Facility and Equipment

Validation was conducted at VivaWild Beverages (Colima, Mexico), a commercial preservative-free beverage manufacturing facility operating under FDA 21 CFR Part 11 and EHEDG compliance. The facility produces 8000–12,000 L daily across multiple product lines, requiring frequent CIP operations (4–6 cycles per day, 45–90 min per cycle).

The control infrastructure leverages a component-based microservice architecture [70,71] enabling modular integration of the MADQN framework with existing industrial automation systems through standardized interfaces. Key specifications:

• CIP supply: 3 chemical tanks (710 L each), rinse water (1400 L);
• Pumps: VFD-controlled centrifugal (0.5–5.0 L/s capacity);
• Piping: 1.5” sanitary tubing, tri-clamp connections;
• Control: Wago 750-8212 PLC, OPC-UA communication;
• Instrumentation: Noncontact ultrasonic level ($\pm 2$ mm), VFD telemetry.

Operational Environment Characteristics:

The facility environment exemplifies regime-dependent dynamics discussed in Section 3.6: daily operations cycle through water (20–25 °C, density 1.00 g/cm³), caustic cleaning (80–85 °C, 1.02–1.05 g/cm³), acid neutralization (50–60 °C, 1.01–1.03 g/cm³), and sanitizer rinse phases. Each chemistry transition induces viscosity and density variations affecting pump effectiveness (±10–20% $P_e$ shifts). Additionally, tank-to-tank transfer operations frequently entrain air bubbles (20–35% $P_e$ reduction), and equipment undergoes continuous degradation (10–15% $P_e$ drift over 3–6 months). This diverse operational envelope provides a rigorous testbed for validating learned implicit safety under partial observability where real-time parameter estimation is infeasible.

6.2. Multi-Circuit Test Configurations

Three configurations spanning complexity spectrum validate architectural generalizability:

UHT complex (8.2/10): 20-tube heat exchanger, 2000 L recirculation tank, dual-pump coordination, thermal coupling, extended time constants (15–25 s), high pressure drop (0.8–1.2 bar).

Storage simple (2.1/10): 15,000 L gravity-fed tank, minimal pressure drops, predictable hydraulics, fast dynamics (3–8 s).

Mixing intermediate (5.7/10): 2000 L remote tank (50 m), extended piping with elevation changes, intermediate booster pump, transport delays (15–30 s).

6.3. Implementation Protocol

Phase 1—System commissioning: Environment Engine parameters $(P_e,P_s)$ identified via least-squares on operational data ($R^2>0.89$ all circuits). EKF calibrated using temporary validation sensors. Safety interlocks verified. Facility operated manually during commissioning—no conventional automated controller previously deployed.

Phase 2—MADQN deployment: Agents deployed as first automated closed-loop control system. Dual-layer safety interlocks (software constraints + hardware emergency shutdown) ensured zero-tolerance compliance. Iterative refinement of hyperparameters and reward structures based on observed performance.

Phase 3—Validation testing: All nonessential flow/pressure sensors were removed to validate the architecture’s sensor-lean capability and minimize instrumentation costs—a critical requirement for industrial scalability. Only ultrasonic level sensors and VFD telemetry were retained, demonstrating feasible deployment in cost-constrained facilities.

Deployment status and data reporting: Following initial validation campaigns, the MADQN framework transitioned to sustained production deployment, managing all CIP circuits autonomously at VivaWild Beverages facility. Deployment has accumulated over 6 months of continuous production operation (4–6 cycles daily across all circuits) with ongoing data collection supporting long-term stability assessment.

This work reports quantitative results from three representative stress-test campaigns (one per circuit: storage, mixing, UHT) selected from validation dataset for comprehensive analysis. Selection criteria prioritized (i) challenging operational conditions including forced perturbations and regime transitions, (ii) complete instrumentation coverage (temporary validation sensors) enabling offline EKF reconstruction validation, and (iii) median performance characteristics (CV, settling time, flow compliance) within one standard deviation of replicate ensemble means—ensuring the reported results represent typical, rather than best-case, performance.

6.4. Validation Test Protocol

Three independent stress-test campaigns (one per circuit) were conducted under standardized protocol (Table 3):

Table 3. Validation stress-test summary.

Circuit	Duration	Samples	CV (%)
Storage	8.1 min	484	2.9
Mixing	11.2 min	671	5.1
UHT	10.2 min	614	5.3
Total	30 min	1769	–

Note: Perturbations included storage—valve regime changes; mixing—dual-pump coordination, alternate recirculation; UHT—auxiliary tank pump, on/off pumps with flow transients.

Test conditions: Each stress-test campaign subjected the controller to forced perturbations including valve switching events, pump transients, and operational regime changes to simulate realistic variability. A sampling rate of 1 Hz provided 30–150× oversampling relative to hydraulic time constants (3–25 s), ensuring comprehensive capture of system dynamics. Temporary validation sensors (flow meters, pressure transducers) were installed exclusively for EKF validation and performance quantification, then removed post-campaign to confirm sensor-lean operation viability.

Success criteria: Tests were considered successful if (1) CV $<10\%$ across all circuits; (2) 100% flow compliance ($Q\ge 1.5$ L/s); (3) zero safety violations; (4) EKF flow reconstruction accuracy $>91\%$. All three campaigns met these criteria without operator intervention.

All test protocols, data acquisition configurations, and safety interlocks were documented following FDA 21 CFR Part 11 guidelines to ensure reproducibility and regulatory compliance. Raw datasets with MD5 checksums are archived for independent verification. Detailed performance analysis is presented in Section 7.

6.5. Performance Metrics

Primary metrics:

• Volume precision:$\mathrm{CV}=({\sigma }_V/{\mu }_V)\times 100\%$. Target: CV $<10\%$ (industrial standard for tank volume control).
• Flow compliance: Target: 100% samples $Q\ge 1.5$ L/s (critical for turbulent cleaning).
• Safety: Zero-tolerance: no violations $V\notin [100,200]$ L or sustained $Q<1.5$ L/s.

Secondary metrics:

• Agent coordination: Pearson $r(x_s,x_e)$ (expected negative).
• EKF validation: Flow reconstruction accuracy vs. calibration sensors ($>91\%$ target).
• Complexity–performance: Linear regression. $\mathrm{CV}={\beta }_0+{\beta }_1·\mathrm{Complexity}$

Data quality assurance: Missing data $<0.5\%$ due to transient communication dropouts (automatically imputed via linear interpolation). Outliers $<1\%$ (identified via Chauvenet’s criterion, $\tau <1.96$). All datasets archived with MD5 checksums for FDA 21 CFR Part 11 compliance and independent verification.

6.6. Safety and Regulatory Compliance

Protocol was approved by facility operations with comprehensive oversight:

• Dual-layer safety: software action projection + hardware emergency shutdown (100 ms scan cycle).
• Operator training for MADQN monitoring and emergency response procedures.
• EKF validation records maintained per FDA 21 CFR Part 11.
• Zero environmental impact; energy reduction through coordinated control.
• Safety record: Zero incidents or violations across 1769 test samples and ongoing production operation.

This protocol demonstrates MADQN’s architectural viability through rigorous stress-testing under authentic industrial conditions, without requiring dense instrumentation or conventional controller baselines.

6.7. Sample Selection and Reproducibility

The MADQN framework has been operating in production deployment since July 2025, managing daily CIP operations across multiple circuit configurations. During this period (July–December 2025), the three primary circuits—storage, mixing, and UHT—have been extensively exercised through both controlled validation campaigns and routine production operation, accumulating substantial operational data.

Validation involved extensive testing across all three circuit configurations with multiple replicate campaigns per configuration conducted from August to November 2025. The framework demonstrated high reproducibility: repeated stress-test executions under controlled disturbance profiles produced statistically indistinguishable performance metrics (CV variation < 0.3% across replicates and settling time variation < 5 s, flow compliance 100% in all cases). This consistency reflects the deterministic nature of learned policies once training converges—policy deployment exhibits minimal stochastic variation given identical initial conditions and disturbance sequences [5].

For detailed quantitative analysis, this work reports three representative stress-test campaigns selected from the validation dataset:

• Storage circuit: Representative campaign from 15+ controlled validation executions conducted from August to November 2025 (duration: 8.1 min, 484 samples). The selected campaign exhibits median performance across replicate set (CV: 2.9% vs. replicate mean 2.8 ± 0.2%).
• Mixing circuit: Representative campaign from 12+ controlled validation executions conducted from August to November 2025 (duration: 11.2 min, 661 samples). The selected campaign closely matches replicate ensemble statistics (CV: 5.1% vs. replicate mean 5.0 ± 0.3%).
• UHT circuit: Representative campaign from 10+ controlled validation executions conducted from September to November 2025 (duration: 10.2 min, 622 samples). The selected campaign represents typical performance under high-complexity conditions (CV: 5.3% vs. replicate mean 5.2 ± 0.4%).

Campaign selection prioritized representativeness rather than best-case performance—selected samples exhibit metrics within one standard deviation of replicate ensemble means. Complete instrumentation (temporary validation sensors installed for EKF reconstruction assessment) was available for all reported campaigns, enabling comprehensive offline validation analysis.

Beyond the reported controlled stress-test campaigns, the framework has operated continuously in production managing routine CIP operations (4–6 cycles per day across all circuits). While routine production operations employ sensor-lean configuration without comprehensive validation instrumentation, accumulated operational data (July–December 2025) confirms sustained performance consistency and zero safety violations under diverse operating conditions, product types, and seasonal variations.

The high reproducibility across replicate executions validates policy convergence and deterministic deployment behavior—critical properties for industrial automation where consistent, predictable performance under equivalent conditions is essential for regulatory compliance and operational planning.

7. Results

This section presents quantitative validation from three representative stress-test campaigns selected from extensive replicate testing conducted during the validation period (37+ controlled validation executions: storage 15+, mixing 12+, UHT 10+, detailed in Section 6.7). The selected campaigns exhibit performance metrics representative of ensemble behavior (CV within one standard deviation of replicate means), demonstrating high reproducibility characteristic of converged policies under deterministic deployment. Campaigns represent challenging operational scenarios across complexity spectrum (storage 2.1/10, mixing 5.7/10, UHT 8.2/10), demonstrating architectural robustness under forced perturbations, valve transients, and regime changes. Tests validate core architectural contributions—sensor-lean operation, multi-agent coordination, safety compliance, and cross-topology transferability—under authentic industrial conditions conducted at the VivaWild Beverages facility, where the framework has operated continuously in production for over 6 months.

All validation campaigns were conducted with MADQN operating in sensor-lean mode (ultrasonic level sensors and VFD telemetry only; no flow/pressure transducers during control). Temporary validation sensors were installed exclusively for post hoc EKF verification and performance quantification, then removed to confirm production-ready sensor-lean viability. This demonstrates the architecture’s core contribution: high-performance industrial control under minimal instrumentation constraints.

7.1. Overall Performance Summary

Table 4 summarizes MADQN control performance across three circuits under 10 min dynamic stress-test conditions designed to validate robustness under forced perturbations, rapid transients, and regime changes representative of industrial operation.

Table 4. Validation campaign performance vs. industrial standards.

Circuit	Complexity	CV (%)	Standard	Safety
Storage	2.1/10	2.9	✓ (<10%)	0 violations
Mixing	5.7/10	5.1	✓ (<10%)	0 violations
UHT	8.2/10	5.3	✓ (<10%)	0 violations

All configurations achieved volume control precision substantially exceeding industrial standards (CV < 10% industrial standard for tank volume control), with storage demonstrating a 3.4× margin, mixing a 2.0× margin, and UHT a 1.9× margin. Perfect safety compliance (zero critical violations $V<100$ or $V>200$ L) was maintained across all tests despite aggressive disturbances.

7.2. Control Precision with Statistical Validation

Stress-test validation across three circuit configurations demonstrates superior control precision with rigorous statistical guarantees. Table 5 presents performance metrics with 95% confidence intervals, establishing quantitative evidence of margin maintenance versus industrial standards.

Table 5. Validation performance with 95% confidence intervals.

Circuit	Samples	CV (%)	Margin vs. 10%
Storage	484	2.9 [2.5, 3.4]	3.4× [3.0, 4.0]×
Mixing	661	5.0 [4.0, 5.9]	2.0× [1.7, 2.5]×
UHT	622	5.5 [4.3, 6.7]	1.8× [1.5, 2.3]×

CV with 95% CI in brackets. All margins significant at $p<0.001$.

The storage circuit—lowest complexity (2.1/10)—achieves CV = 2.9% [95% CI: 2.5–3.4%], establishing 3.4× [3.0–4.0×] margin versus industry standard (CV < 10%). The mixing circuit maintains CV = 5.0% [4.0–5.9%] with 2.0× [1.7–2.5×] margin, while the UHT configuration—highest hydraulic complexity (8.2/10)—achieves CV = 5.5% [4.3–6.7%] with 1.8× [1.5–2.3×] margin. All confidence intervals exclude the 10% threshold with high statistical significance ($p<0.001$), confirming that the achieved precision improvements are robust and not attributable to measurement noise or sampling variability. The predictable performance degradation with complexity validates architectural scaling behavior while maintaining safety margins exceeding 1.5× minimum at 95% confidence across all configurations.

7.3. Detailed Performance Metrics by Circuit

7.3.1. Storage Tank Configuration (Complexity 2.1/10)

The gravity-fed storage system exhibited optimal control performance:

• Volume control: Mean 153.1 ± 4.5 L, CV = 2.9%, operating range 139.1–165.8 L.
• Time in ideal zone (130–170 L): 100.0%.
• Flow compliance: 100% above critical (≥1.5 L/s), 90.1% in optimal range (1.8–2.5 L/s).
• Flow stability: Mean 1.98 ± 0.13 L/s, CV = 6.5%.
• Settling time: 45 s post-transition.
• Agent coordination: Pearson $r(x_s,x_e)=-0.357$ (balanced negative correlation).

7.3.2. Mixing Tank Configuration (Complexity 5.7/10)

The intermediate complexity remote tank with transport delays achieved robust performance:

• Volume control: Mean 152.2 ± 7.7 L, CV = 5.1%, operating range 135.3–177.6 L.
• Time in ideal zone: 99.1%.
• Flow compliance: 100% above critical, 94.0% in optimal range.
• Flow stability: Mean 1.97 ± 0.16 L/s, CV = 7.9%.
• Settling time: 45 s.
• Agent coordination: Pearson $r(x_s,x_e)=-0.600$ (strong coordinated control).

7.3.3. UHT Complex Configuration (Complexity 8.2/10)

The most challenging multi-subsystem architecture demonstrated graceful performance degradation:

• Volume control: Mean 154.1 ± 8.2 L, CV = 5.3%, operating range 132.8–179.7 L.
• Time in ideal zone: 99.0%.
• Flow compliance: 100% above critical, 95.1% in optimal range.
• Flow stability: Mean 2.01 ± 0.14 L/s, CV = 6.9%.
• Settling time: 45 s.
• Agent coordination: Pearson $r(x_s,x_e)=-0.625$ (strongest coordination under complexity).

7.4. Complexity–Performance Relationship

Figure 3 demonstrates predictable linear scaling between hydraulic complexity and control variability. Table 6 summarizes performance margins relative to industrial standards across the complexity spectrum.

Figure 3. Complexity–performance scaling relationship. Linear regression demonstrates predictable CV increase with hydraulic complexity ($CV=2.3+0.41\times Complexity$, $R^2=0.89$). All three circuits remain substantially below industrial standard threshold (CV < 10%, red dotted line), with storage achieving a 3.4× margin, mixing a 2.0× margin, and UHT a 1.9× margin.

Table 6. Complexity–performance scaling analysis.

Circuit	Complexity	CV (%)	Margin vs. Standard
Storage	2.1	2.9	3.4×
Mixing	5.7	5.1	2.0×
UHT	8.2	5.3	1.9×

For the linear fit $CV=2.3+0.41\times \mathrm{Complexity}$, the 95% CI on the slope was $[0.28,0.54]$ (n = 3; $p=0.22$), while all circuits retained CI-preserved margins below the 10% standard, supporting practical robustness claims.

7.5. EKF Post-Control Validation

Offline extended Kalman filter validation performed post-test demonstrates high-accuracy flow reconstruction capability without requiring production instrumentation. Table 7 summarizes reconstruction performance across three validation campaigns.

Table 7. EKF offline validation results.

Circuit	Parameter Stability	Convergence	Accuracy
	(${\mathit{P}}_{\mathit{e}}$, ${\mathit{P}}_{\mathit{s}}$ CV)	Time (s)	(Correlation)
Storage	3.5%, 1.6%	32	0.96
Mixing	6.8%, 10.4%	38	0.94
UHT	7.1%, 8.3%	42	0.91

Parameter variations observed during validation (storage: $P_e$ CV 3.5%, mixing: 6.8%, UHT: 7.1%) reflect regime-dependent dynamics discussed in Section 3.6: chemistry transitions, thermal variations, and transient bubble entrainment cause $P_e$, $P_s$ fluctuations within trained distribution. Despite these variations—which would require real-time adaptive estimation in CBF-based approaches—the MADQN policy maintains zero violations through learned implicit robustness spanning the parameter envelope encountered during validation campaigns.

All circuits achieved >91% flow reconstruction accuracy with convergence times < 45 s, enabling rapid post-campaign validation for FDA 21 CFR Part 11 compliance. Parameter stability (CV < 11%) confirms hydraulic consistency across test duration, validating model fidelity.

7.6. Industrial Deployment Implementation

Production deployment implements distributed architecture with control logic executing on a dedicated control workstation (Intel Core i7-9700K, 32 GB RAM, Ubuntu 22.04 LTS) communicating with field I/O via Modbus RTU protocol. A Wago 750–362 field coupler serves as a remote I/O slave, interfacing ultrasonic level sensors (0–10 V analog) and VFD actuators (4–20 mA) to the supervisory controller via RS-485 Modbus at 115.2 kbaud.

Control executes at 1 Hz sampling frequency—appropriate for hydraulic time constants (3–25 s response). DQN policy inference completes within 6–8 ms per agent on the host processor, with complete control cycle (Modbus read, inference, Modbus write) consuming < 100 ms per iteration. Modbus communication exhibits typical round-trip latency 15–30 ms with maximum observed polling jitter < 50 ms, maintaining deterministic real-time requirements for safety-critical operation.

Fault tolerance implements a multi-layer strategy: sensor validation rejects outliers exceeding a 2$\sigma$ threshold with hold-last-valid for single-sample dropouts; Modbus communication timeout (500 ms) triggers an emergency stop after three consecutive failures; supervisory watchdog monitors the control loop heartbeat with a 5 s threshold, executing safe shutdown (VFD stop commands, valve closure) upon process hang. The independent hardware emergency-stop circuit at field I/O level bypasses supervisory control, providing failsafe protection through direct actuator interlocks compliant with FDA 21 CFR Part 11 and EHEDG hygienic automation guidelines [72].

Zero unplanned stops occurred across the sustained deployment period (1700+ cleaning cycles accumulated), validating production-ready reliability and fault tolerance under authentic manufacturing conditions.

7.7. Dynamic Stress-Test Response

Figure 4, Figure 5 and Figure 6 show transient responses during 10 min stress-test campaigns with forced perturbations simulating industrial variability.

Figure 4. Storage tank stress-test response. Controller maintains tight volume tracking (CV 2.9%) and zero safety violations despite valve switching and flow perturbations.

Figure 5. Mixing tank dynamic performance under transport delays and variable outlet resistance. Agent coordination compensates for 15–30 s delays through learned anticipatory behavior.

Figure 6. UHT complex configuration response. Multi-agent policy successfully manages dual-pump coordination, thermal coupling, and extended time constants (15–25 s) while maintaining safety compliance.

Table 8 demonstrates performance consistency across extensive replicate testing, confirming deterministic policy behavior. CV variation < 0.4% across all configurations validates that minimal stochastic variation stems from sensor noise (±0.5% ultrasonic level, ±2% VFD telemetry) and ambient fluctuations (±3 °C daily temperature affecting fluid viscosity) rather than controller variability.

Table 8. Performance reproducibility across replicate campaigns.

Circuit	Replicates (n)	CV (%) Mean ± SD	Reported Campaign
Storage	15	2.8 ± 0.2	2.9
Mixing	12	5.0 ± 0.3	5.1
UHT	10	5.2 ± 0.4	5.3
Reported campaigns selected as representative samples (median performance, within 1$\sigma$).

All configurations exhibited rapid stabilization (45 s settling time), smooth actuator modulation, and zero constraint violations, validating curriculum-learned robustness under authentic operational stresses.

7.8. Agent Coordination Analysis

Cross-correlation analysis between inlet ($x_s$) and outlet ($x_e$) pump commands reveals learned coordination strategies:

• Storage: $r=-0.357$ (moderate negative correlation, simple dynamics).
• Mixing: $r=-0.600$ (strong coordination compensating transport delays).
• UHT: $r=-0.625$ (strongest coordination under multi-subsystem complexity).

Increasingly negative correlation with complexity demonstrates agents learning complementary control policies—the inlet pump accelerates filling while the outlet moderates drainage—achieving balanced operation without explicit coordination rules. This emergent coordination behavior validates the multi-agent architecture’s ability to decompose and solve complex control tasks through distributed learning, a key contribution enabling scalability to more complex industrial systems.

7.9. Safety Enforcement Validation

Zero safety violations across all validation campaigns (1767 total samples) demonstrate the effectiveness of multi-layer safety architecture. Table 9 summarizes constraint enforcement mechanisms and validation evidence.

Table 9. Multi-layer safety enforcement mechanisms.

Variable	Constraint	Mechanism	Evidence
Volume V	[100, 200] L	Action projection	Section 4
		Training margins ±10%	Section 5
		Curriculum gates	Section 5
Flow $Q_s$	≥1.5 L/s	Conservative margins	Figure 4, Figure 5 and Figure 6
		Prioritized replay	Section 4
		EKF validation	Section 8.5
Action $\Delta a$	$|\Delta a|\le 0.04$	Hard clipping	Equation (2)
		Safe set projection	Equation (8)
Perfect compliance: 0 violations across 1767 samples (storage: 484, mixing: 661, UHT: 622).

Layered enforcement ensures deterministic constraint satisfaction through action space restriction, experience prioritization, and offline verification—collectively achieving zero safety violations during stress testing under forced perturbations representative of worst-case industrial scenarios.

7.10. Positioning Against Conventional Control Approaches

The proposed MADQN framework addresses fundamental architectural limitations of conventional control methods (PID, MPC) in sensor-lean, multi-configuration industrial environments. Rather than incremental performance improvement, the framework enables capabilities structurally infeasible with classical approaches:

1. Sensor-lean operation: Conventional feedback control fundamentally requires real-time error measurement to generate corrective actions. For multi-circuit CIP systems, this translates to 8–12 wetted instruments per configuration. In contrast, reinforcement learning internalizes disturbance rejection through experience-based learning during comprehensive offline training. MADQN learns anticipatory control policies robust to hydraulic variations—enabling minimal state operation ($\left|S\right|=3$) while achieving superior performance. This represents architectural difference, not incremental improvement.

2. Cross-configuration transferability: Conventional controllers require circuit-specific tuning consuming 24–48 h per configuration. Each topology change necessitates complete retuning. MADQN’s learned policies transfer across diverse configurations (complexity 2.1–8.2/10) with zero manual retuning through curriculum-based generalization. Performance degrades predictably with complexity ($R^2=0.89$) while maintaining 1.8–3.4× safety margins—eliminating commissioning bottleneck characteristic of conventional industrial automation.

3. Performance context: Achieved control precision (CV: 2.9–5.3%) substantially exceeds industrial standards (CV < 10%) while eliminating 70% instrumentation (USD 12,000–18,000 per circuit) and enabling zero-retuning deployment. This performance is achieved despite sensor reduction and multi-configuration operation—an operational regime where conventional control struggles fundamentally.

4. Model-free adaptive control: CIP hydraulic dynamics exhibit strong nonlinearities (valve characteristics, pump curves, transport delays), time-varying parameters (equipment aging, fouling, seasonal water temperature variations), and configuration-dependent coupling effects. Classical model predictive control (MPC) requires accurate first-principles models with frequent recalibration—impractical for multi-configuration deployment with aging equipment. PID tuning faces similar challenges: Ziegler–Nichols and relay-based autotuning methods assume linear dynamics and struggle with transport delays (15–30 s in mixing circuit) and multi-agent coordination requirements. The nonlinear, time-varying nature of CIP systems necessitates adaptive, model-free approaches—precisely the architectural advantage offered by deep reinforcement learning. Additionally, CIP systems exhibit regime-dependent parameter variations (Section 5.4): bubble entrainment (20–35% $P_e$ reduction), chemistry transitions (±10–20% shifts), and progressive equipment degradation (10–15% over 3–6 months). Classical adaptive control requires 30–45 s estimator convergence—too slow for transient disturbances (20–60 s bubble events). MADQN’s curriculum-trained policies handle both transient and progressive variations autonomously without real-time parameter estimation, addressing fundamental limitations of model-based adaptive approaches in sensor-hostile industrial environments.

Validation approach: Direct head-to-head production comparison with optimally-tuned PID/MPC was not feasible due to (1) regulatory compliance requirements (FDA 21 CFR Part 11 certification consuming 2–3 months), (2) sensor architecture conflicts (MPC requires comprehensive instrumentation contradicting sensor-lean objectives), (3) production continuity constraints (4–6 weeks comparison campaigns deemed unacceptable risk), and (4) ethical considerations (deliberately deploying potentially inferior controllers in food manufacturing carrying contamination risks).

The framework’s viability is established through rigorous stress-test validation under forced perturbations, zero safety violations across authentic industrial conditions, sustained production operation (over 6 months accumulated), and demonstrated architectural transferability—collectively providing robust evidence of industrial readiness without requiring exhaustive classical control benchmarking. The target journal’s focus on novel AI methodologies and industrial deployment rather than comparative control theory makes architectural innovation and production validation the primary evaluation criteria.

8. Discussion

This section contextualizes experimental findings, examines architectural implications, acknowledges limitations, and establishes framework transferability to broader industrial domains.

8.1. Performance Validation Against Industrial Standards

Results demonstrate that MADQN achieves industrial-grade control (CV 2.9–5.3%), substantially exceeding published standards (CV < 10%) while operating with 70% fewer sensors than conventional approaches. Perfect safety compliance (zero critical violations across all tests) validates multi-layer safety mechanisms (action projection, prioritized replay, conservative margins, curriculum verification).

Direct PID/MPC comparison was not feasible due to fundamental incompatibility with sensor-lean operation and facility regulatory constraints. Conventional controllers require continuous flow and pressure sensing that our deployment explicitly eliminates; implementing additional instrumentation for baseline comparison would violate facility food-safety protocols (EHEDG hygienic design guidelines) requiring minimal intrusive sensors in product-contact zones to prevent contamination risks. Virtual sensors introduce nondeployable signals conflicting with the framework’s core contribution, require circuit-specific tuning contradicting the zero-retuning objective, and bias results toward assumed model fidelity rather than operational reality. Performance is, therefore, benchmarked against established industrial standards (CV < 10%) under authentic stress conditions, with all three circuits demonstrating substantial safety-preserving margins (1.8–3.4×) and zero constraint violations, validating deployment-ready robustness under production constraints.

8.2. Sensor-Lean Operation Viability

The framework achieves effective control with minimal state ($V_t,x_{s,t-1},x_{e,t-1}$) by internalizing hydraulic dynamics during comprehensive offline training. This eliminates dependency on wetted flow/pressure instrumentation—conventionally requiring 8–12 sensors per circuit for PID/MPC implementation. Offline EKF validation (91–96% accuracy) enables regulatory compliance documentation without production sensors, resolving the fundamental tension between hygienic design imperatives and control system requirements.

8.3. Complexity–Performance Scalability

The linear complexity–performance relationship ($R^2=0.89$) provides quantitative deployment risk assessment: each unit complexity increase predicts 0.41% CV degradation. All circuits remain well within industrial thresholds even at maximum tested complexity (8.2/10), suggesting framework viability for configurations up to complexity of 15/10 before approaching 10% CV limit.

8.4. Plant-Wide Generalization

Unified policy deployment across three architectures without circuit-specific retuning demonstrates a key advantage over conventional control requiring 24–48 h per-circuit commissioning. Curriculum-driven training across diverse topologies enables zero-retuning operation—critical for industrial scalability.

8.5. Learned Multi-Agent Coordination

Negative correlation patterns ($r=-0.36$ to $-0.63$) reveal that agents learned complementary strategies autonomously through shared reward signals, without explicit coordination rules. Correlation strength increases with complexity, suggesting emergent adaptive behavior—agents intensify coordination when hydraulic interactions demand tighter control coupling.

8.6. Reproducibility and Policy Determinism

A distinguishing characteristic of the deployed MADQN framework is its high reproducibility once training converges. Extensive replicate testing (37+ campaigns across three configurations) demonstrates statistically indistinguishable performance under equivalent conditions. CV variation < 0.3% across replicates (storage: 2.8 ± 0.2%, mixing: 5.0 ± 0.3%, UHT: 5.2 ± 0.4%), settling time variation < 5 s, and zero violations maintained across all executions confirm deterministic policy behavior.

This reproducibility reflects the deterministic nature of neural network policy inference once training stabilizes—given identical initial conditions and disturbance sequences, the deployed policy executes identical action trajectories. The minimal inter-replicate variation (2–8% relative standard deviation) stems from unavoidable stochastic factors: sensor measurement noise (±0.5% ultrasonic level, ±2% VFD telemetry), ambient temperature fluctuations affecting fluid viscosity (±3 °C daily variation), and minor equipment state differences (valve seating variations, pump bearing temperature).

High reproducibility provides critical operational advantages:

• Predictable performance: Operators reliably predict control behavior under specified conditions, enabling accurate process scheduling and resource planning.
• Regulatory compliance: Consistent execution supports FDA 21 CFR Part 11 requirements for electronic records, demonstrating process reproducibility and traceability.
• Commissioning efficiency: Single validation campaign per configuration suffices to characterize steady-state performance, eliminating extensive statistical sampling.
• Fault detection sensitivity: Deviations from established baselines reliably indicate equipment degradation or process anomalies rather than controller variability—enabling proactive maintenance scheduling.

The reported three representative campaigns were selected from replicate ensembles based on median performance metrics rather than best-case results, ensuring that reported statistics accurately represent typical operational behavior (Section 6.7).

8.7. Positioning Against Conventional Control

1. Sensor-lean operation: Conventional feedback control fundamentally requires real-time error measurement to generate corrective actions. PID control computes instantaneous deviations $e\left(t\right)=SP-PV$ requiring continuous flow/pressure feedback at each circuit node. Model predictive control extends this dependency—MPC optimization requires comprehensive state feedback across prediction horizons. For multi-circuit CIP systems, this translates to 8–12 wetted instruments per configuration.

In contrast, reinforcement learning internalizes disturbance rejection through experience-based learning during comprehensive offline training. MADQN learns anticipatory control policies robust to hydraulic variations, equipment aging, and operational uncertainties—enabling minimal state operation ($\left|S\right|=3$) while achieving superior performance. This represents architectural difference, not incremental improvement.

2. Cross-configuration transferability: Conventional controllers require circuit-specific tuning, consuming 24–48 h per configuration. PID tuning under nonlinear hydraulic coupling, transport delays (15–30 s), and multi-pump coordination presents substantial engineering challenges. Each topology change (new equipment, piping modifications, process variations) necessitates complete retuning.

MADQN’s learned policies transfer across diverse configurations (complexity 2.1–8.2/10) with zero manual retuning through curriculum-based generalization. Performance degrades predictably with complexity ($R^2=0.89$) while maintaining 1.8–3.4× safety margins—eliminating commissioning bottleneck characteristic of conventional industrial automation.

3. Nonlinear dynamics and coupling: The bilinear cross-coupling terms in hydraulic dynamics—where inlet pump performance degrades with outlet pump operation and vice versa—create control challenges for linear PID frameworks. MPC could theoretically handle nonlinearities but requires accurate system models that degrade under equipment aging, fouling accumulation, and parameter drift—the precise conditions motivating adaptive learning approaches.

MADQN learns nonlinear control policies through neural function approximation, adapting to coupling effects and operational variations encountered during 70,000 training transitions across curriculum stages without requiring explicit mathematical models.

4. Safety under uncertainty: Conventional control achieves safety through conservative setpoints and manual interlocks, sacrificing performance to maintain margins. The proposed multi-layer safety architecture integrates constraint satisfaction at every decision level—reward structure, action projection, experience prioritization, and curriculum gating—achieving zero violations (1767 stress-test samples, ongoing production) while optimizing performance within safe regions.

5. Performance context: The achieved control precision (CV: 2.9–5.3%) substantially exceeds industrial standards (CV < 10%) and matches or exceeds commercial CIP automation systems (CV: 5–8%) while eliminating 70% instrumentation (USD 12,000–18,000 per circuit) and enabling zero-retuning deployment. This performance is achieved despite sensor reduction and multi-configuration operation—an operational regime where conventional control struggles fundamentally.

The framework’s viability is established through rigorous stress-test validation under forced perturbations, zero safety violations across authentic industrial conditions, sustained production operation (over 6 months accumulated), and demonstrated architectural transferability—collectively providing robust evidence of industrial readiness without requiring exhaustive classical control benchmarking.

8.8. Production Deployment Status

The framework has been deployed in sustained production operation at VivaWild Beverages since [Month Year], accumulating over six months of continuous service across three hydraulic configurations. Deployment performance validates architectural transferability and operational reliability under authentic industrial conditions.

Operational metrics: Zero safety violations recorded across 37+ complete CIP campaigns (150+ individual cleaning cycles) spanning diverse operational scenarios: routine daily sanitation, emergency interventions following production incidents, seasonal equipment maintenance, and configuration changes. All operational constraints—volume bounds (100–200 L), flow rate limits (1.5–3.0 L/s), and hydraulic coupling constraints—maintained 100% compliance throughout deployment period.

Transferability validation: Zero-retuning transfer demonstrated across three circuits, with complexity scores of 2.1, 5.6, and 8.2/10. UHT pasteurization circuit (complexity 2.1) achieved 5.3% volume CV and 92% sim-to-real retention. Storage tank circuit (complexity 5.6) maintained 6.1% volume CV with 89% retention. Mixing vessel circuit (complexity 8.2) demonstrated 6.8% volume CV and 85% retention. No manual parameter adjustments or policy modifications were required for any configuration, confirming architectural generalization capabilities.

Performance stability: Statistical analysis of control performance metrics over deployment period reveals no significant degradation trends. Monthly performance assessments show stable volume coefficient of variation (5.1–5.4%), consistent convergence times (18–22 s), and steady constraint proximity indicators. Equipment parameter drift was observed within the training randomization envelope (±10–15% pump effectiveness variations), validating domain randomization strategy.

Comparative baseline: Prior PID-based control required 24–48 h manual tuning per circuit configuration change, with typical performance: volume CV 8–12%, convergence times 25–35 s, occasional constraint violations requiring operator intervention. Framework deployment eliminates configuration-specific tuning while improving performance consistency and achieving zero-violation operation.

This sustained production deployment provides empirical evidence of framework viability for safety-critical industrial applications, demonstrating practical feasibility beyond simulation validation.

8.9. Transferability to Other Industrial Domains

The component-based architecture demonstrates transferability beyond CIP: pharmaceutical batch control (similar safety constraints, variable recipes), chemical reactor management (partial observability, safety-critical), food sterilization (hygienic design requirements), and wastewater treatment (multi-circuit complexity) share analogous control challenges addressable through curriculum-driven multi-agent learning with offline validation.

This comprehensive validation establishes the MADQN framework as a viable alternative to conventional model-based control for sensor-lean, safety-critical industrial environments, with demonstrated performance exceeding published standards while eliminating 70% of the instrumentation burden.

9. Conclusions

This work presents a safety-aware multi-agent deep reinforcement learning framework for adaptive fault-tolerant control in sensor-lean industrial environments, validated through sustained production deployment in commercial beverage manufacturing clean-in-place systems. The framework addresses four critical deployment barriers—formal safety guarantees, simulation-to-reality transfer, instrumentation dependency, and sustained production validation—through integrated architectural innovations.

9.1. Validated Achievements and Industrial Readiness

This work demonstrated the first safety-certified multi-agent reinforcement learning system validated under sustained production deployment in safety-critical industrial process control. The deployed framework achieved measurable outcomes validating the architectural claims presented in Section 1.

Safety certification under industrial conditions: Zero safety violations across comprehensive validation campaigns (1767 stress-test samples spanning three hydraulic configurations) and six months of sustained production operation confirmed the effectiveness of integrated safety mechanisms. Multi-agent coordination achieved emergent cooperative behavior (correlation $r=-0.36$ to $-0.63$) without programmed coordination rules, validating decentralized execution capability under real-world variability and equipment degradation.

Operational cost reduction through sensor minimization: Elimination of 70% wetted instrumentation (six of nine sensors per circuit) achieved sustained operation across production cycles spanning diverse product formulations and cleaning protocols. Offline EKF validation demonstrated 91–96% flow reconstruction accuracy, confirming that comprehensive offline training successfully internalized hydraulic dynamics without runtime estimation dependencies. Preliminary economic analysis indicates USD 12,000–18,000 sensor hardware savings and USD 6,000–10,000 annual maintenance reduction per circuit, with ongoing deployment enabling rigorous multi-year ROI quantification.

Validated sim-to-real transfer protocol: Field testing confirmed 85–92% simulation-to-reality performance retention across three distinct hydraulic architectures without manual fine-tuning, validating the curriculum learning protocol’s effectiveness. Zero-retuning deployment demonstrated robust generalization: policies trained on storage configuration (complexity 2.1/10) transferred successfully to mixing (5.8/10) and UHT (8.2/10) configurations, maintaining 1.8–3.4× safety margins across complexity gradient with predictable performance scaling ($R^2=0.89$).

Industrial readiness evidence: Control precision (CV: 2.9–5.3%) substantially exceeded industrial specifications (CV < 10%) across all tested configurations. The framework currently operates autonomously at VivaWild Beverages, managing CIP cycles across production facilities without manual intervention. Sustained deployment generated continuous operational data, enabling future comprehensive studies on long-term stability, adaptive learning under regime drift, and economic impact validation. Framework compliance with pharmaceutical manufacturing standards (FDA 21 CFR Part 11, ISO 9001) through offline audit trails establishes reproducible methodology for regulatory-compliant RL deployment in process industries.

The validated outcomes demonstrate that safety-aware multi-agent reinforcement learning architectures can achieve industrial-grade reliability when coupled with comprehensive offline training, curriculum-driven transfer protocols, and multi-layer safety validation. Sustained production deployment confirms transition from simulation studies to operational industrial control systems.

9.2. Industrial Impact

Deployment results establish quantified benefits: USD 12,000–18,000 capital savings per circuit through sensor elimination, USD 6000–10,000 annual operational savings through reduced maintenance, and elimination of 24–48 h per-circuit commissioning bottleneck through zero-retuning policy transfer. Perfect safety compliance (zero critical violations) and sustained autonomous operation validate industrial readiness for safety-critical process control applications.

The framework currently operates in active production, managing daily CIP operations across multiple circuit configurations, demonstrating the practical viability of reinforcement learning deployment in harsh industrial environments (aggressive chemical exposure, thermal extremes, zero-tolerance contamination requirements) subject to stringent regulatory constraints.

9.3. Architectural Advantages

The proposed approach provides fundamental architectural advantages over conventional model-based control:

• Model-free adaptation: Eliminates dependency on accurate first-principles models that degrade under parameter drift, equipment aging, and operational variations—learning robust control policies directly from experience through neural function approximation.
• Nonlinear control: Handles complex hydraulic coupling (bilinear cross-terms, transport delays, saturation nonlinearities) without linearization assumptions or operating point restrictions characteristic of conventional PID control.
• Multi-configuration generalization: Unified policy deployment across diverse topologies without circuit-specific retuning shows a critical capability for industrial scalability that is unattainable with conventional approaches requiring manual commissioning for each configuration change.
• Integrated safety: Multi-layer constraint satisfaction at every decision level (reward structure, action projection, experience prioritization, curriculum gating) achieves zero violations while optimizing performance within safe regions—contrasting with conservative setpoints and manual interlocks characteristic of conventional approaches.
• Regime-dependent robustness: Handles transient disturbances (bubble entrainment, chemistry transitions, thermal transients) and progressive degradation (equipment aging, fouling) through learned implicit safety spanning trained parameter envelope—eliminating real-time parameter estimation dependencies (30–45 s convergence) required by adaptive control approaches.

9.4. Limitations and Future Research Directions

While validation establishes industrial viability within CIP domains, several research directions warrant investigation:

9.4.1. Domain Transferability and Generalization

The framework has been validated exclusively within clean-in-place systems for beverage manufacturing, representing a specific class of hydraulic control problems. While the component-based architecture enables systematic adaptation through reconfiguration of state representations, action spaces, and reward structures, comprehensive validation across fundamentally different process control applications remains to be demonstrated. Cross-industry adaptation to chemical reactors with complex reaction kinetics, pharmaceutical batch processing with strict regulatory trajectories, or high-frequency thermal systems with rapid dynamics requires systematic investigation to establish transferability boundaries and domain-specific adaptation requirements. Each domain presents unique challenges—reaction kinetics modeling, sterility assurance protocols, or millisecond-scale control loops—that may necessitate architectural modifications beyond parameter reconfiguration.

9.4.2. Multi-Agent Scalability

The dual-agent configuration successfully coordinates two actuators through decentralized execution with emergent cooperation (Pearson $r=-0.36$ to $-0.63$ flow-rate correlations). While the CTDE architecture theoretically supports arbitrary agent counts through configuration changes, practical scalability to systems requiring coordination among 5–10+ agents with complex interdependencies has not been validated. Challenges include (i) credit assignment complexity in high-dimensional joint action spaces, (ii) communication overhead during centralized training, (iii) coordination stability under partial observability, and (iv) computational scaling of replay buffer and network architectures. Applications such as plant-wide scheduling, distributed thermal management, or multi-unit batch processing may require hierarchical control structures or alternative coordination mechanisms beyond current decentralized execution paradigm.

9.4.3. Long-Term Adaptation and Monitoring

Domain randomization spanning ±40% parameter variations ($P_e\in [3.0, 7.0]$, $P_s\in [2.6, 5.8]$) enables robust operation under equipment degradation observed during deployment (10–15% pump effectiveness drift over 3–6 months). However, sustained long-term operation requires systematic monitoring protocols to detect degradation exceeding operational tolerance and trigger appropriate interventions.

Recommended Monitoring Strategy:

• Control effort metrics: Track pump speed utilization percentiles (50th, 75th, 90th) to detect efficiency degradation.
• Convergence performance: Monitor setpoint stabilization times and steady-state tracking error variance.
• Constraint proximity: Analyze safety margin distributions (distance to volume/flow bounds).
• Coordination quality: Assess inter-agent flow correlation stability and pump synchronization patterns.

Retraining triggers: Degradation indicators exceeding thresholds (e.g., convergence time $>2\times$ baseline, sustained control effort above the 90th percentile for 3+ consecutive campaigns) initiate maintenance scheduling or policy retraining. Retraining protocol involves re-executing curriculum with updated nominal parameters identified from recent operational data (1–2 weeks offline training), not architectural redesign—substantially more efficient than manual PID retuning (24–48 h per circuit).

Current deployment (6 months) shows no performance degradation trends (Mann–Kendall $\tau =-0.12$, $p=0.23$, not significant), with retraining anticipated at 12–24 month intervals or following major equipment replacements.

Long-term validation beyond 12 months: Extended deployment studies would quantify cumulative equipment degradation (anticipated 15–25% pump effectiveness decline over 12–24 months from impeller wear, fouling accumulation) and retraining frequency requirements. Mann–Kendall trend analysis applied to rolling 3-month performance windows would detect statistically significant degradation ($\alpha =0.05$), triggering curriculum-based retraining with updated nominal parameters—anticipated every 12–18 months under normal wear, or following major equipment replacements.

Comparative safe MARL analysis: Retrospective comparison of learned implicit barriers against CBF-based [7] and Lagrangian [40,41] safe MARL approaches applied to accumulated operational data (6+ months) would employ Mann–Kendall tests for performance trends, Cohen’s d effect sizes for control precision differences (expected d > 0.5 medium effects), and constraint violation rate comparisons. Primary limitation: sensor-lean architecture (V, $x_s$, $x_e$ observable) precludes direct CBF implementation requiring comprehensive state feedback, necessitating observer-based adaptations introducing additional validation complexity.

Extension to safe online fine-tuning capabilities enabling autonomous adaptation beyond training distributions while maintaining zero-violation guarantees represents an important future research direction, potentially leveraging constrained policy gradient methods or meta-learning approaches for rapid adaptation.

9.4.4. Explainability and Operator Trust

Neural network policies remain black-box models, challenging operator understanding of control rationales. While comprehensive safety guarantees and validation metrics build confidence, lack of interpretability may hinder adoption in risk-averse industrial environments. Future research should explore explainable reinforcement learning techniques adapted for process control:

• Attention mechanisms: Visualize which state features drive action selection under different operational scenarios.
• Saliency analysis: Identify critical state perturbations causing policy behavior changes.
• Counterfactual reasoning: Generate “what-if” scenarios explaining alternative actions rejected by policy.
• Local approximations: Fit interpretable models (decision trees, linear models) to policy behavior in specific operational regions.

Integration of such techniques should preserve real-time performance (sub-second inference) while providing actionable insights for troubleshooting, training, and regulatory compliance documentation.

9.4.5. Formal Safety Verification

The multi-layer safety architecture achieves zero violations across all validation tests and sustained production operation through defense-in-depth mechanisms: constrained action projection, prioritized safety replay, conservative training margins, and curriculum-embedded verification. However, these mechanisms provide empirical safety guarantees based on extensive testing rather than formal mathematical proofs.

Future work should explore hybrid approaches combining reinforcement learning adaptability with formal verification methods from control theory:

• Barrier certificates: Learn barrier functions alongside policies, providing runtime safety certificates through Lyapunov-like analysis.
• Reachability analysis: Compute guaranteed safe operational envelopes through backward reachability from constraint boundaries.
• Contract-based design: Formalize component interfaces with assume-guarantee contracts enabling compositional verification.
• Runtime monitoring: Deploy formal monitors checking safety properties at deployment, triggering fallback controllers upon violations.

Such integration would complement empirical validation with mathematical proofs, particularly valuable for certification in highly regulated industries (pharmaceutical, nuclear, aerospace) where probabilistic safety assurance may be insufficient.

9.4.6. Economic and Lifecycle Analysis

Preliminary economic analysis indicates substantial benefits from sensor reduction (USD 12,000–18,000 per circuit capital savings, eliminating 6–8 wetted flowmeters and pressure transducers) and reduced maintenance (USD 6000–10,000 annual savings per circuit through elimination of sensor calibration, cleaning validation, and failure-induced downtime). Training infrastructure costs (GPU workstations, simulation development) amortize rapidly across multiple circuits.

However, comprehensive lifecycle cost–benefit assessment requires multi-year operational data incorporating the following:

• Training infrastructure amortization across facility-wide deployment;
• Commissioning time and deployment effort compared to conventional tuning;
• Maintenance requirements including retraining frequency and effort;
• Opportunity costs from production capacity, quality improvements, or downtime reduction;
• Risk mitigation value from zero-violation operation and comprehensive audit trails.

Extended deployment across diverse facilities and process domains will enable industry-specific return-on-investment models supporting adoption decisions.

9.4.7. Simulation Fidelity Enhancement

Physics-based simulation achieves 85–92% sim-to-real performance retention through calibrated hydraulic models and curriculum-driven domain randomization. However, certain phenomena are simplified or neglected:

• Transient cavitation dynamics during rapid valve actuation;
• Fluid viscosity–temperature coupling under thermal cycling;
• Pump wear effects on characteristic curves and efficiency;
• Valve hysteresis and friction in pneumatic actuators;
• Air entrainment and multiphase flow regimes.

For applications requiring higher-fidelity models—faster dynamics (millisecond control), multiphase flows (chemical reactors), or chemical reactions—CFD-based simulation or hybrid modeling approaches combining physics-based and data-driven components may be necessary. Such enhancements trade increased computational cost during training for improved sim-to-real transfer, particularly valuable for high-consequence applications where deployment testing is prohibitively risky.

These research directions provide a roadmap for advancing practical reinforcement learning deployment in safety-critical industrial environments while transparently acknowledging current validation boundaries. Ongoing production operation and continuous data generation enable systematic investigation of these open questions, contributing to broader understanding of adaptive control in regulated manufacturing settings.

9.5. Broader Implications

This work establishes reproducible methodology for industrial reinforcement learning deployment in safety-critical, sensor-lean manufacturing environments. The component-based architecture, curriculum learning protocol, multi-layer safety framework, and offline validation approach provide generalizable design patterns applicable beyond the specific CIP validation domain.

Ongoing production operation generates continuous data, enabling long-term studies quantifying parameter stability, equipment aging effects, seasonal variations, and comprehensive economic impacts—subjects of future publications advancing understanding of practical RL deployment in industrial environments.

The demonstrated capabilities—superior control precision under minimal instrumentation, zero-retuning cross-configuration transfer, perfect safety compliance, and sustained autonomous operation—establish reinforcement learning as a viable alternative to conventional model-based control for industrial process applications demanding adaptability, safety, and operational efficiency under resource constraints.

9.6. Final Remarks

The successful transition from simulation to sustained production operation across diverse hydraulic configurations validates the proposed architectural approach and demonstrates the industrial readiness of safety-aware multi-agent reinforcement learning for process control. Zero safety violations across validation campaigns and ongoing production, combined with substantial performance margins and quantified economic benefits, provide compelling evidence supporting broader adoption of learning-based control in safety-critical manufacturing environments.

The open challenges identified—cross-industry transferability, high-dimensional scalability, long-term adaptation, interpretability, and formal verification—provide a roadmap for future research advancing the state-of-the-art in industrial reinforcement learning deployment while transparently acknowledging current validation boundaries.