Development of a Multi-Robot System for Autonomous Inspection of Nuclear Waste Tank Pits

Featured Application

The Autonomous Pit Exploration System (APES) project is designed to revolutionize the inspection of nuclear waste-storage tank pits using an advanced multi-robot system. This system aims to enhance safety and efficiency in nuclear waste management by providing a robust solution for the detailed inspection of storage tanks. The project is structured in three phases, focusing on data collection, mechanical design, and system integration, culminating in rigorous testing at the Idaho National Laboratory. Utilizing a telescopic arm for superior reach and an electric vehicle (EV) pickup truck for safe, long-duration operations, APES ensures coordinated robot operations via a master control package and digital twin models. Preliminary tests have successfully demonstrated the system’s capabilities in tasks such as Simultaneous Localization and Mapping (SLAM) and generating detailed point cloud maps, validating both hardware and software designs. This innovative approach promises significant improvements in the inspection process, reducing risks and enhancing precision in monitoring nuclear waste storage.

Abstract

This paper introduces the overall design plan, development timeline, and preliminary progress of the Autonomous Pit Exploration System project. This project aims to develop an advanced multi-robot system for the efficient inspection of nuclear waste-storage tank pits. The project is structured into three phases: Phase 1 involves data collection and interface definition in collaboration with Hanford Site experts and university partners, focusing on tank riser geometry and hardware solutions. Phase 2 includes the selection of sensors and robot components, detailed mechanical design, and prototyping. Phase 3 integrates all components into a cohesive system managed by a master control package which also incorporates digital twin and surrogate models, and culminates in comprehensive testing and validation at a simulated tank pit at the Idaho National Laboratory. Additionally, the system’s communication design ensures coordinated operation through shared data, power, and control signals. For transportation and deployment, an electric vehicle (EV) is chosen to support the system for a full 10 h shift with better regulatory compliance for field deployment. A telescopic arm design is selected for its simple configuration and superior reach capability and controllability. Preliminary testing utilizes an educational robot to demonstrate the feasibility of splitting computational tasks between edge and cloud computers. Successful simultaneous localization and mapping (SLAM) tasks validate our distributed computing approach. More design considerations are also discussed, including radiation hardness assurance, SLAM performance, software transferability, and digital twinning strategies.

1. Introduction

Nuclear waste management is a critical aspect of the United States’ energy policy, given the country’s extensive use of nuclear technology for power generation, medical applications, and defense purposes. The safe handling, storage, and disposal of nuclear waste are paramount to protecting public health, the environment, and national security. The US Department of Energy (DOE) oversees numerous facilities dedicated to managing nuclear waste, including the Hanford Site in Washington State, which is one of the most significant and complex environmental remediation projects in the country. The Hanford Site is home to 177 underground storage tanks containing approximately 56 million gallons (0.23 million tons) of radioactive and chemical waste [1]. These tanks are grouped into what are known as tank farms, and the management of this waste is a high priority for the DOE due to its potential environmental and health hazards.

Regular inspections of the pits of these tanks is crucial for several reasons. On one hand, the structural integrity of the concrete tank pits must be continually monitored to prevent leaks and potential contamination of the surrounding soil and groundwater. On the other hand, accurate mapping of as-is conditions and radiation level assessment of the pit interiors are necessary to plan and execute waste-retrieval and -treatment operations. However, given the hazardous nature of the environment proximal to the tanks, traditional manual inspection methods are challenging and pose significant risks to human workers. Figure 1 [2] illustrates ongoing waste leak tests in a pump pit prior to waste-transfer operations. In order to perform such inspections, the deployment of heavy machinery is needed to lift the pit cover blocks, normally consisting of reinforced concrete and steel. These cover blocks can easily weigh in the range of 10 to 30 tons or more for a valve pit as shown in Figure 2. Both the costs and risks for performing such replacement operations can be concerning. Additionally, any manual pit inspections or other operations with the pit cover opened will require multiple personnel wearing heavy personal protective equipment (PPE) to prevent radioactive hazards. Despite these precautions, these operations still pose significant health risks to the personnel and potential contamination to the surrounding environments.

Figure 2 illustrates the isometric view of a typical concrete nuclear waste-storage tank pit. Combining the information in Figure 2 and Figure 3, an alternative approach for pit inspection would be delivering sensors directly into the tank through the view ports with the assistance of autonomous robotic platforms. This will eliminate the need of replacing the heavy cover blocks, and even by using non-autonomous, teleoperated robotic platforms, the radioactive hazard towards on-site personnel can be greatly mitigated.

Despite the numerous advancements and designs in robotic systems, we have identified a significant gap in the current body of research and development. Specifically, there is a notable absence of a multi-robot system that is effectively designed and developed for the unmanned inspection of nuclear waste-storage facilities. These environments pose unique challenges due to confined spaces, high radiation levels, and complex structural layouts [3,4], which cannot be adequately addressed by the aforementioned systems. While there have been various proposals and implementations of robotic systems for other aspects of nuclear site inspection and decommissioning [5,6], none have demonstrated the capability to comprehensively and efficiently perform unmanned inspections in these highly hazardous areas. This gap underscores the need for innovative solutions that integrate multiple robots with complementary capabilities, thereby enhancing coverage, improving safety, and ensuring thorough inspections.

In order to close this research gap, we propose the design of the Autonomous Pit Exploration System (APES). APES is a multi-robot system with six primary groups of components, namely EV pickup truck, central control panel, onboard electrical cabinet, outside pit equipment, and two robots—FIU robotic arm and INL-Nexxis crawler robot, each carrying inspection sensors. The overall communication plan of the system is shown in Figure 4.

In this work, we focus on providing one integrated solution in advancing the field of unmanned inspection technologies for nuclear waste-storage facilities. The primary contributions of this paper are as follows:

• Integration of six primary groups of components into a multi-robot system: This lays the foundation of the Autonomous Pit Exploration System to operate two types of robots to perform autonomous inspection for nuclear waste-storage tank pits.
• Propose the “System on a Truck” concept: Utilize an EV pickup truck to support the system for a full 10 h shift, eliminating risks associated with combustible fuels and enhancing the safety and efficiency of transportation and deployment.
• Digital twinning for autonomous operations: Create a digital twin framework using intermittent signatures captured by robotics system. The development of autonomy for the robotic systems will increase the likelihood of system adoption by the site and will ensure adequate data collection needed for digital twin generation and artificial intelligence (AI) models training.
• Novel actuator designs: The combination of the pipe clamper and the telescopic arm designs is selected for the superior reach, load distribution, stability, making it an ideal solution for accessing and inspecting the tank pits.

The rest of this paper is organized as follows: Section 2 discusses the research and design works relevant to the proposed APES designs; Section 3 provides the overview of the proposed project timeline and discusses the overall communication plan and transportation and actuator components of the system; Section 4 discuss the three preliminary experiments to validate the early design of APES and presents their results; Section 5 is a discussion of design considerations including software transferability and digital twinning strategies; Section 6 concludes this paper and discusses the future endeavors.

2. Related Works

The development of autonomous systems for inspection and maintenance tasks in hazardous environments has been an area of significant research and development. Regarding our design and development of APES, the related work and studies include robot designs to traverse the narrow passages and hazardous environments, those robots developed to operate in nuclear sites, and multi-robot systems for autonomous inspections.

First, multiple robot designs can perform operations inside narrow and potentially hazardous environments. Daniyana et al. [7] developed a wheeled robot with monocular camera to inspect the structural health of natural gas pipelines. Schoor et al. [8] designed an explosive device disposal robot able to traverse various indoor terrains including stairs and vertical walls and also has operator-training capabilities. Cao et al. [9] designed a tunnel-inspection robot which is able to travel in GPS-denied dam conduit tunnels to perform structural health monitoring (SHM) tasks while eliminating the dynamic objects including the inspection personnel.

Second, in terms of nuclear site cleanup and decommissioning, robots are valued for performing the “dirty, dull, and dangerous” jobs that would pose risks to human workers [6]. Recent advances in robust sensors, autonomy, and human–robot interfaces have renewed interest in deploying teams of robots to tackle nuclear-inspection tasks more safely and efficiently [5,6]. Moreover, inspecting underground waste tanks is challenging due to confined spaces, high radiation, and complex structures. A single robot cannot easily achieve full coverage or handle all required sensing modalities. This concept aligns with recent research demonstrating that teams of complementary robots can perform radiation mapping and facility inspection more effectively than isolated units [6]. Such multi-robot ecosystems are key to covering large, complex nuclear environments. Other related works include Florida International University (FIU) team’s earlier proposed robot design for tank bottom inspection at Hanford Site [10], and Hirose’s “Serpentine Robots” [11] which were deployed to perform inspections of multiple locations including the Fukushima Daiichi Nuclear Power Plant [12,13].

And finally, a number of multi-robot system have been designed and deployed for tasks in hazardous environments. Notable projects include the European Union’s Robotics for Inspection and Maintenance (RIMA) initiative [14,15,16] and the DARPA Subterranean Challenge [17,18,19]. These projects emphasize the importance of coordination and communication between multiple robots to achieve efficient and reliable operation in complex settings. Moreover, the development of radiation-hardened robotic systems is essential for operating in radioactive environments. On one hand, the Box Encapsulation Plant (BEP) was designed and developed to enhance the efficiency and safety of nuclear waste postprocessing for the United Kingdom, leveraging advanced robotics and autonomous systems to handle and treat hazardous materials effectively [20]. On the other hand, researchers from Mitsubishi Heavy Industries and Kobe University utilized dual-arm manipulators to carry out the decommissioning operation of the primary containment vessel (PCV) at Fukushima Daiichi Nuclear Power Plant Unit 3 [21].

3. Conceptual Design

3.1. Design Overview

The proposed design of APES focuses on developing an advanced robotic system to efficiently inspect and interact with nuclear waste-storage tank pits the at the tank farms of Hanford Site, WA. The project is structured into several interconnected tasks, each addressing critical aspects of the system’s design, integration, and deployment. The following overview outlines the conceptual design and methodology that will guide the development process over a three-year period.

3.1.1. Phase 1: Data Collection and Interface Definition

Phase 1 of the design initiates with essential collaborations with Hanford Site experts to gather comprehensive data and metrics. This will include detailed configuration requirements and procedures, with special attention to critical elements such as tank riser geometry. Continuous dialogue and feedback mechanisms will be established to ensure alignment with site requirements and facilitate prompt adjustments during the design phase.

In addition, defining the APES interfaces accurately is also a priority. Leveraging the expertise of university partners including collborators at Florida International University (FIU), the team will plan and design hardware solutions to connect robotic components with tank risers. This phase will set the direction for subsequent design efforts and identify any necessary non-standard parts.

3.1.2. Phase 2: Component Selection and Hardware Design

In Phase 2 of design, the selection of sensors and robot components is crucial. The focus will be on using standard off-the-shelf components to enable rapid replacement and maintenance. This task will ensure that the chosen sensors meet the specific form factor and durability requirements for successful tank system operations.

And immediately upon closing up the selection tasks, the mechanical design phase will commence. Part or component drawings will be compiled using SolidWorks 2024, followed by a detailed assembly of all components and eventually a digital twin for simulation purposes. In the meant time, FIU team will assist in developing the first autonomous robotic arm configuration, and team at the Idaho National Laboratory (INL) will assemble and maintain a crawler robot with comprehensive sensor stack for textured point cloud collection and 3D reconstruction. Prototyping will be conducted to validate and refine designs.

3.1.3. Phase 3: Integration, Testing, and Demonstration

Phase 3 of the design will commence with the integration of all parts into a cohesive system. This task will involve bringing together the various vehicles, robots, and sensors into a complete whole unit. A master control package will be developed to manage the operations of the robots and sensors, enabling both automated and manual control. Concurrently, we will procure all system components, prioritizing critical and long lead time items to ensure timely assembly and testing.

Once the components are procured, the system software design will be undertaken by the INL Digital Engineering group. This will include the development of digital twin and surrogate models, incorporating radiological and 3D imaging/mapping capabilities. These models will synthesize data collected from the various sensors and devices, providing a comprehensive control and analysis framework. Auxiliary control systems will be added as necessary to ensure the system’s functionality and efficiency.

The final phase of the project will involve building a fully simulated 1:1 nuclear waste-storage tank pit at INL, where all technological components will be assembled, configured, and tested. This mockup will serve as a testing ground to validate the entire system’s performance. Following successful testing, a comprehensive final project report will be prepared, summarizing the design documentation, testing results, and operational procedures. Finally, INL will host an on-site demonstration for project collaborators and custodians of the Hanford Site, serving as an acceptance test and introductory training session to familiarize them with the new technology.

3.2. System Communication Design

To ensure the APES system operates with minimal jitter and latency, a time-sensitive networking (TSN) approach is implemented. TSN provides deterministic network communication, which is critical for systems that require precise timing and synchronization. Below is a conceptual design of the TSN approach tailored for minimizing jitter and latency across mixed signal paths like USB, Ethernet, and CAN.

• Gateway Devices: Use gateway devices that can bridge different communication protocols (e.g., USB-to-Ethernet, CAN-to-Ethernet) while supporting TSN features. These gateways will ensure that all data, regardless of its original protocol, can be managed under the TSN framework.
• Synchronization of Heterogeneous Networks: Implement synchronization mechanisms to align the timing of USB, Ethernet, and CAN networks. This involves using precise time synchronization protocols (IEEE 802.1AS [22]) to ensure that all devices, regardless of the communication protocol, operate on the same time base.
• Priority Queuing and Scheduling: Assign priority levels to different types of traffic based on their criticality. For example, control signals for the robotic arm and crawler robot should be assigned the highest priority, while less critical data like monitoring and logging information can be assigned lower priorities. Time-Aware Shaper (IEEE 802.1Qbv [23]) can be used to schedule-transmission windows for high-priority traffic, reducing latency and jitter.

Also proper network configuration and management are crucial for the effective implementation of TSN in APES:

• Centralized Network Management: Use a centralized network-management system to configure and monitor TSN parameters across all devices. This system will handle tasks such as setting up time slots, managing synchronization, and monitoring network performance.
• Traffic Shaping: Implement traffic shaping techniques to control the flow of data and prevent network congestion. This involves configuring the Time-Aware Shaper to allocate specific time slots for high-priority traffic, ensuring that it is transmitted without delay.
• Redundancy and Failover Mechanisms: Incorporate redundancy and failover mechanisms to enhance network reliability. This includes implementing redundant paths and devices that can take over in case of a failure, ensuring continuous operation without interruptions.

And next, we determined the overall system plan consisting of every component and how these components and sections are connected as shown in Figure 4.

First of all, the FIU robotic arm communicates directly with the On-board Electrical Cabinet, where the motor drivers and data-acquisition (DAQ) systems are located. The motor drivers in the electrical cabinet receive control signals from the Jetson computer to actuate the motors of the robotic arm. Additionally, the sensors, such as the 2D RGB camera, depth camera, and LiDAR, send data back to the Syslogic^@ RML A4AGX rugged computer (referred as Jetson computer in the later text) via the logic board for processing, which allows the robotic arm to make decisions based on its situational awareness built upon the sensor data.

Second, the Nexxis crawler robot also connects to the onboard electrical cabinet, with its motors and encoders controlled by the same motor drivers housed in the electrical cabinet. Like the robotic arm, the crawler robot’s sensors, including the camera and LiDAR, provide feedback to the system for navigation and task execution. The data and control signals are relayed through the central Jetson computer, which manages the crawler’s movements and other functions.

Next, the control panel serves as the central interface for the operator to interact with the system. The control panel is connected to the onboard electrical cabinet via Ethernet cables, allowing real-time control over the robotic arm, crawler robot, and external equipment. The control panel receives feedback from the Jetson computer and provides a user interface with a keyboard, mouse, and augmented reality/virtual reality (AR/VR) headset. The human-machine interface (HMI) panel and additional screens offer visual feedback, while both the game pad controller and keyboard-mouse set are connected via USB to allow for manual operation if necessary.

The outside pit equipment is another subsystem that interacts with the onboard electrical cabinet. It is connected to the motor drivers, encoders, and other components in the electrical cabinet. This equipment includes various mechanical actuators and sensors (such as presence sensors, pneumatic valves, and cylinders) that allow the robotic system to interact with the environment, particularly in hazardous or confined spaces. The equipment is controlled via signals from the Jetson computer and the electrical cabinet.

Last but not least, the EV Truck provides power to the entire system. The power systems and transformers in the truck are connected to the on-board electrical cabinet, ensuring that all components receive the required power for operation. This section enables the system to be mobile and independent of external power sources during field operations.

Overall, the connections between the sections allow for a fully integrated system where data, power, and control signals are shared across all subsystems, ensuring effective and coordinated operation. The interactions between these sections make it possible to perform complex inspection and task execution in potentially hazardous environments.

To enhance failure recovery and fault tolerance in inter-system communication, the system design incorporates redundancy and periodic health checks within the communication pathways. By monitoring the health and status of connections between the Jetson computer and other components, the system can automatically switch to backup pathways or alternative modes of operation when faults are detected, ensuring robust communication and minimizing operational disruptions.

Additionally, the hybrid control and perception hub based on the Jetson node is designed to implement graceful degradation strategies, allowing it to continue functioning even when certain subsystems experience failures. This includes a prioritization of critical tasks, enabling the system to maintain essential operations like navigation and sensor data collection while deferring non-essential functions until normal operation can be restored, thus ensuring continued performance in challenging environments.

3.3. Transportation and Tooling Design

This section will discuss the strategies and mechanisms developed or proposed to address these challenges, focusing on both the transportation of the system to the site and the reliable deployment of robots within the tank pits. By addressing these key aspects, the design aims to enhance operational efficiency, safety, and compliance with regulatory standards.

3.3.1. Transportation Approach

In Phase 2, the first major design question to be answered is how the entire system can be transported to the site and then deliver robots inside the tank pits. And the second question is how the robots can be powered during the field work for a full day shift of 10 h, as the site is know to have limited access to regular AC power. Usually, portable power solutions such as mobile generators or fuel cells can be utilized to provide this. However, safety concerns are paramount, as these power sources involve combustible fuels that pose significant fire and explosion hazards in environments containing radioactive materials. Additionally, regulatory compliance is also a challenge, given the stringent oversight and approval processes required for new power sources.

To address both design questions, we decide to transport and power the system simultaneously in the form of a light-duty electric truck. In our case studies we have employed a Ford F-150 Lighting as shown in Figure 5. A circuit schematic of the system is shown as in Figure 6. The entire multi-robot system is to be housed and mounted on the EV truck. This approach offers several benefits, including reduced emissions, which is crucial for maintaining a safer and cleaner environment around sensitive nuclear waste sites. This “System on a Truck” approach also ensures that all components are securely transported to the site. As a mobile power source with high-capacity batteries, the EV can support the system for a full 10 h shift upon one charge, which was validated in previous studies [24], and can be supplemented with renewable energy sources like solar panels. This approach aims to reduce operational costs through lower electricity and maintenance expenses, while also minimizing logistical challenges related to fuel supply and storage. Additionally, eliminating combustible fuels enhances safety by reducing fire hazards and risks, creating a cleaner and safer environment for sensitive activities. Moreover, EVs produce minimal emissions and operate with less vibration when parked compare to other power-supplying utility vehicles, ensuring better safety and environmental compliance, and minimal disruption to monitoring systems at sensitive sites like nuclear waste tank farms.

Furthermore, to ensure the accuracy of LiDAR and radiation sensors during the continuous deployment cycle, we implement a strategy for on-site calibration and drift correction. This involves periodic recalibration using reference targets and known radiation sources to compensate for potential drift caused by thermal fluctuations and electromagnetic interference. Real-time monitoring of sensor outputs allows for dynamic adjustments, ensuring data integrity throughout the 10 h operational period.

3.3.2. Power Management Design

In an EV truck with limited power delivery paths, such as a peak capacity of 9.6 kW across multiple outlets, effective real-time power budgeting is essential to regulate energy distribution among high-load subsystems like robot motors, computers, and communication devices. A power-management system is implemented to monitor and allocate power dynamically based on the current demand of each subsystem. This system continuously assesses the power consumption of each component and prioritizes power allocation according to predefined criteria, ensuring that critical systems receive the necessary power while maintaining overall system stability. For example, during peak operational demands, the system can allocate more power to essential tasks, such as driving robot motors, while temporarily reducing power to non-critical components like auxiliary communication devices.

To prevent brownouts when multiple high-demand peaks occur simultaneously, the power-management system employs several mechanisms. One effective strategy is the implementation of load shedding, which automatically reduces or disconnects power to non-essential subsystems when the total power demand approaches the truck’s maximum capacity. Additionally, the system utilizes power-limiting settings for specific components, allowing them to operate within safe consumption thresholds. Advanced algorithms can predict demand trends based on historical usage patterns, enabling proactive adjustments to power distribution before a brownout situation arises. By integrating these mechanisms, the EV truck ensures reliable operation of its high-load subsystems while minimizing the risk of power disruptions during simultaneous peak demands. This ensures a stable and efficient power supply, crucial for maintaining the functionality of robotic systems and other critical operations.

3.3.3. Pipe Tooling Design

In addition, the tooling mechanism is designed as shown in Figure 7. To ensure stability and reliable data recording in tank pits, especially given the small opening and long reach of the robot, an internal pipe clamping mechanism is being explored. This mechanism fits inside the pipe, providing necessary stability without taking up additional external space. Its adjustability allows secure fastening at various points within the pipe, accommodating different diameters and conditions. The enhanced stability provided by this mechanism improves data quality by reducing movement and potential measurement errors, which is crucial for accurate monitoring and assessment in sensitive environments like nuclear waste tank farms.

After the robotic arm has been delivered inside the tank pits through the view port pipe, sensor stacks, e.g., radiation sensors are delivered to the various locations inside the pits by actuating the robotic arms. A conceptual figure of the digital twin simulating this process is shown in Figure 7a. The robotic arm design concept will be further discussed in Section 3.3.4.

3.3.4. Robotic Arm Design

Although the end-effectors or sensors to be delivered can be varied, the central idea is to have a robotic arm able to adjust the sensors to their desired reference position and orientation for data acquisition. Between the original proposed design plans of a telescopic arm and a scissor arm, we decided to select and further iterate on the telescopic arm design due to its greater reach, precision, and compactness when retracted. Telescopic arms can extend and retract linearly, providing greater reach into deep and confined spaces [25], which is particularly useful in narrow and deep access points typical of nuclear waste tank pits. They also offer greater precision and control in positioning, crucial for accurately targeting specific areas within the tank for radiation delivery. Additionally, when retracted, telescopic arms occupy less space compared to scissor arms, making them easier to maneuver and store in tight environments.

Furthermore, telescopic arms generally offer better load distribution and stability when extended, important for handling the weight and equipment associated with radiation delivery systems. They also tend to have fewer moving parts exposed to the environment, reducing the risk of contamination and wear and tear, leading to improved longevity and less frequent maintenance. The fully retracted and extended telescopic arm design concept are shown in Figure 8a and Figure 8b, respectively. In contrast, scissor robotic arms may have limited reach, be bulkier and more complex in design, and have multiple pivot points and linkages that introduce additional points of failure and require more maintenance. Overall, the telescopic robotic arm design provides enhanced reach, precision, and ease of use, making it a superior choice for delivering and positioning sensor stacks inside nuclear waste tank pits.

Given the spatial constraints and potential for misalignment within tank risers, solving the telescopic arm’s inverse kinematics (IK) under dynamic constraints such as minor misalignments and radial offsets is essential. The IK problem for the telescopic arm is addressed using a combination of geometric and iterative numerical methods. An initial estimate is obtained through geometric analysis, which is then refined using an iterative Jacobian-based approach. The Jacobian matrix, connecting joint velocities to end-effector velocities, is computed at each iteration. Using the Jacobian transpose or pseudo-inverse, joint angles are iteratively adjusted to minimize errors between the desired and actual end-effector positions. Real-time sensor feedback is utilized to continuously monitor the arm’s position and orientation, dynamically adjusting the IK solution to maintain alignment with the target despite minor misalignments or radial offsets. Additionally, a proportional-integral-derivative (PID) control algorithm fine-tunes the arm’s movements for precise positioning.

We developed a comprehensive simulation to monitor and control the operation of a telescopic arm, focusing on both motor reaction moments and motor forces. Figure 8 shows both the fully retracted and extended telescopic arm poses in the simulation. This simulation takes both the reaction moment and motor forces as feedback to improve the arm’s trajectory tracking performance. By continuously monitoring the motor reaction moments and forces, we can ensure that the motors operate within their safe limits, preventing potential damage and improving the system’s reliability.

By integrating motor feedback, precise end effector positioning, and Rapidly-exploring Random Tree (RRT) motion planning for the end effector pose with object avoidance, our simulation provides a robust and effective control strategy for operating the telescopic arm. This combination ensures smooth, accurate, and safe movements, making the system highly reliable for various applications.

3.4. Radiation Hardness Assurance Design

Ensuring the radiation hardness and electromagnetic compatibility (EMC) of onboard electronics is crucial for the reliability and safety of systems operating within the tank pit environment, which is often subject to significant neutron and gamma radiation fluxes. The following strategies are essential for achieving these objectives:

3.4.1. Radiation Hardness Shielding

Utilizing radiation-hardened components is a proactive approach to mitigate the effects of radiation on electronic systems. These components are specifically designed and tested to withstand high levels of radiation without significant degradation in performance [26]. Manufacturers of radiation-hardened electronics often provide detailed specifications on the radiation tolerance levels, which should be matched to the environmental conditions of the tank pit.

The Onboard Electrical Cabinet houses critical components such as motor drivers, DAQ systems, and the central Jetson computer. To protect these components from radiation:

• Material Selection: Use high-density materials like lead, tungsten, and specialized radiation-hardened alloys to construct the cabinet and enclosures.
• Layered Shielding: Implement a layered shielding approach, combining different materials to attenuate both neutron and gamma radiation effectively.
• Component Encapsulation: Encapsulate sensitive electronic components within the cabinet using radiation-hardened materials to provide an additional layer of protection.

In addition, the FIU robotic arm and the Nexxis crawler robot are equipped with sensors, cameras, and LiDAR systems that require protection from radiation:

• Sensor Housing: Design sensor housings using radiation-hardened materials to shield the sensors from direct exposure to radiation.
• Cable Shielding: Use radiation-resistant cables and conduit to protect the communication lines between the sensors, motors, and the onboard electrical cabinet.
• Modular Shielding: Implement modular shielding solutions that can be easily replaced or upgraded as needed to maintain protection levels over time.

By integrating these radiation hardness shielding strategies, the system communication design can effectively protect the onboard electronics from the adverse effects of neutron and gamma radiation. This ensures that the system remains functional, reliable, and safe throughout its operational life in the tank pit environment. The combination of robust shielding materials, modular designs, and continuous monitoring will provide a high level of protection, enabling the system to perform complex inspection and task execution in potentially hazardous environments.

3.4.2. Electromagnetic Compatibility Compliance

Ensuring EMC involves rigorous testing and validation against relevant standards. Conducting tests such as radiated and conducted emissions, susceptibility to electromagnetic interference, and electromagnetic pulse (EMP) resistance are critical [27]. Compliance with standards like MIL-STD-461 [28] for military applications or IEC 61000 series [29] for commercial applications can help verify the EMC performance of the systems.

3.5. Control Logic and Safety Assurance Dseign

In teleoperation scenarios, especially those involving restricted or slow views due to poor-quality signals, collision-avoidance logic and safety envelopes are critical for ensuring the safe and efficient operation of robotic systems. The use of an AR/VR interface can enhance the operator’s situational awareness, but the underlying logic and safety mechanisms must be robust to compensate for signal limitations. Below are the key components and strategies for implementing collision-avoidance logic and safety envelopes in such scenarios.

3.5.1. Collision-Free Control Logic

Collision-free control logic is designed to prevent the robotic system from colliding with obstacles, other robots, or the environment. It involves several layers of detection, decision-making, and action.

Firstly, sensor integration is essential. A combination of sensors, such as LiDAR, depth cameras, ultrasonic sensors, and proximity sensors, creates a comprehensive map of the environment, providing real-time data on the position and movement of obstacles. These data are processed in real-time using algorithms that detect potential collisions. Techniques such as path planning, obstacle detection, and predictive modeling are employed to anticipate and avoid collisions.

Secondly, advanced motion planning algorithms like RRT and Dynamic Window Approach (DWA) generate safe trajectories for the robotic system to follow. These algorithms consider the robot’s current position, velocity, and the positions of detected obstacles. In scenarios where immediate action is required, reactive control mechanisms can override planned motions to avoid collisions by executing pre-defined evasive maneuvers based on sensor input.

3.5.2. Safety Envelope

The safety envelope defines the permissible operating space for the robotic system, ensuring it remains within safe boundaries while performing tasks. It is a dynamic, adaptive boundary that adjusts based on the robot’s environment and operational context. Dynamic safety zones are defined around the robotic system, adjusting based on the robot’s speed, proximity to obstacles, and the operational context. These zones can be visualized in the AR/VR interface to provide the operator with a clear understanding of safe operating limits.

Proximity alert systems are implemented to notify the operator when the robotic system approaches the safety envelope boundaries. These alerts can be visual, auditory, or haptic feedback provided through the AR/VR interface. To ensure immediate cessation of motion and prevent collisions, automatic braking and emergency stop mechanisms are configured to halt the robot’s movement if it breaches the safety envelope.

3.5.3. Augmented Reality/Virtual Reality Interface Enhancements

Haptic feedback is integrated into the Augmented Reality/Virtual Reality (AR/VR) controllers to provide tactile alerts for proximity warnings and collisions, enhancing the operator’s ability to respond quickly to potential hazards. Multi-modal feedback, combining visual, auditory, and haptic cues, ensures the operator receives critical information through multiple channels, which is particularly useful in scenarios with poor signal quality where one mode of communication might be compromised.

3.6. Compensations for Poor-Quality Signals

In scenarios where signal quality is poor, additional strategies can be employed to maintain safe teleoperation:

• Local Autonomy: Equip the robotic system with local autonomy capabilities, allowing it to make real-time collision-avoidance decisions independently when communication with the operator is delayed or interrupted.
• Buffering and Smoothing: Implement buffering and data-smoothing techniques to reduce the impact of latency and jitter on the operator’s view. This can involve predictive modeling to fill in gaps in the data stream.
• Redundant Communication Channels: Use redundant communication channels to ensure continuous data flow. If one channel experiences degradation, the system can switch to an alternative channel to maintain connectivity.
• Pre-Defined Safe States: Define pre-programmed safe states or behaviors for the robotic system to assume if signal quality drops below a certain threshold. This ensures the system defaults to a safe and predictable state during communication disruptions.

4. Experimental Results

Although the APES development is still in its early stages, significant progress has been made in several key experimental areas. First of all, we successfully planned and completed experiments focused on EV power quality and load capacity, which are crucial for ensuring that the system can handle the electrical demands and maintain stable performance under various conditions.

In addition to power-related experiments, we have also tested edge and cloud computing resources allocation via a Robot Operating System 2 (ROS 2) architecture. These are essential for optimizing the computational capabilities of the system, ensuring that processing tasks are efficiently distributed between local (edge) and remote (cloud) resources. Furthermore, in-lab simultaneous localization and mapping (SLAM) experiments have been completed, which are fundamental for the system’s navigation and environmental mapping capabilities.

4.1. EV Power Tests

For EV power testing, our case studies focused on connecting an electrical load equivalent to the expected power consumption to the EV truck circuit as shown in Figure 6, simulating the real-world APES applications. During the EV power tests aimed at evaluating power quality, several key parameters were measured quantitatively to ensure compliance with standards such as IEEE 519 [30,31], particularly in environments sensitive to nuclear instrumentation. The primary metrics assessed included harmonic distortion, voltage sag and swell, and transient response.

Harmonic distortion was measured using specialized power quality analyzers that capture the voltage and current waveforms across various frequencies. These analyzers compute the Total Harmonic Distortion (THD) values for both voltage and current, providing insights into the degree of distortion present in the electrical system. This analysis is critical in nuclear environments where sensitive instrumentation can be adversely affected by harmonics, potentially leading to erroneous readings or system malfunctions.

Voltage sag and swell were evaluated by monitoring the voltage levels over time using the same power quality analyzers. These devices recorded voltage fluctuations, allowing for the identification and quantification of any sags (temporary reductions in voltage) or swells (temporary increases in voltage) that occurred during the tests. Additionally, transient response was assessed by observing how quickly the system could recover from sudden changes in load or input conditions, which is crucial for maintaining stable operation in sensitive environments. Compliance with IEEE 519 or equivalent standards was verified by correlating the measured data against the allowable limits defined in the standard. This involved analyzing the harmonic content and voltage variations to ensure they remained within specified thresholds, thereby confirming that the power quality met the stringent requirements necessary for the reliable operation of nuclear instrumentation.

Through a series of controlled tests, the system’s power delivery consistency, load-handling capability, and battery performance were evaluated under various conditions. The initial load test revealed that the vehicle provided stable power with minimal distortion and no unexpected interruptions, ensuring reliable performance. In the operational limit test, the 240 V system maintained loads up to 7.4 kW before tripping, offering valuable insights into the system’s maximum operational capacity. During a simulated full-scale APES operation, the vehicle’s battery depleted by only 6% over an hour, indicating that the system has sufficient capacity for a full 10 h day shift while retaining reserve power for transportation needs. A truncated 60 s result of one of the baseline tests is shown in Figure 9. For more details, readers are welcome to consult our earlier publications [24].

These findings underscore the potential of electric vehicles as reliable power sources for field operations. The load test results demonstrated the system’s ability to deliver consistent and stable power, critical for the precise and safe inspection of nuclear waste tank pits. The operational limit test provided essential data on the system’s capacity, helping to define its operational boundaries and optimize its performance. Additionally, the battery performance test highlighted the efficiency and endurance of the power system, suggesting that the APES can operate for extended periods without requiring frequent recharging, which is crucial for missions in remote and challenging environments.

4.2. ROS 2 Communication Tests

4.2.1. Distributed Cloud Computing

To design the software to navigate the crawler robot inside the tank pits to collect sensor data, we need to take the limitation of the onboard computational resources into consideration. Fortunately, the overall design plan utilizes tethered communication which includes Ethernet cable connections. In addition, it is possible to use ROS 2 in-domain communication to split computations between multiple computers. Therefore, we propose the idea of distributing computational tasks between an edge robot computer and a more powerful cloud-based computer using ROS 2 in-domain communication over an Ethernet connection.

The cloud computer, in this case study the Syslogic^@ RML A4AGX rugged computer, is equipped with a 12-core NVIDIA Arm^@ Cortex A78AE CPU and a 2048-core Ampere GPU with 64 Tensor Cores. It has significantly more processing capacity which is more suitable to handle computationally intensive processes such as odometry computation, map generation, 3D visualization in RViz, and potentially data processing or logging. This separation of responsibilities allows the edge system to remain focused on time-sensitive tasks like sensor publishing and motion control, ensuring that latency-sensitive operations are not delayed. Both systems communicate seamlessly using ROS 2’s domain ID mechanism over Ethernet, which provides reliable and high-throughput communication for large data topics like /camera/rgb/image_raw and /scan. The software system used in this case study is adapted from the Real-Time Appearance-Based Mapping (RTAB-Map) package [32] provided by the official website of Yahboom Technology Inc. (Shenzhen, China) [33].

Due to the delayed access to the planned Nexxis crawler robot, we utilized a Yahboom Rosmaster R2 robot with a Jetson Orin Nano onboard computer to test our software package. The limited computational resources of the Jetson Orin Nano made it impractical to run all the ROS 2 nodes and topics on its own, necessitating a plan for testing a distribution of computational tasks between the edge and cloud computers.

The resulting ROS 2 rqt graph is shown in Figure 10. This diagram illustrates how computational responsibilities and topic communications are distributed between the edge and cloud computers via Ethernet connection, both operating under ROS_Domain ID = 32. The edge-side (highlighted with the magenta dashed box) is responsible for low-level hardware interfacing and real-time sensor data publishing, while the cloud-side (blue dashed box) manages higher-level perception, localization, and visualization tasks.

On the edge-side, the robot is still interfacing with the RGB-D camera and LiDAR sensors. These nodes publish essential data-streaming topics including /camera/rgb/image_raw and /scan. The robot processes these streams locally and publishes them into the shared ROS 2 domain, making them available to subscribers running on the cloud-based system.

On the cloud-side, the Syslogic computer subscribes to these raw sensor topics for higher-level processing, such as mapping, localization, and visualization. Key nodes on the cloud computer include /rtabmap/rtabmap and /ekf_localization, which are responsible for SLAM, sensor fusion, and broadcasting transforms for RViz. RTAB-Map modules like /map, /mapGraph, and /mapData are also hosted on the cloud, significantly reducing the computational load on the edge device. This separation of responsibilities ensures real-time responsiveness at the edge while leveraging the cloud’s computational power for intensive processing through efficient ROS 2 in-domain communication.

During the test, we also run the system performance monitor software on both cloud and edge computers 5 min after we start the ROS 2 software on both sides to prevent observing significant fluctuations or surges in CPU usage. Both computers are running on Ubuntu 22.04.5 LTS Linux-based operating systems, and their truncated 60 s CPU usages are shown in Figure 11a,b. As can be observed from these plots, both computers are keeping their CPU usages mostly under 50% when running all the tasks in Figure 10. This validates our approach and showcases great potential for distributing more available resources for other robotic tasks without exhausting the robot onboard computer’s resources or compromising performances.

4.2.2. Time Synchronization on Distributed Computers

In multi-ROS 2 node systems operating on distributed computers, effective time synchronization is crucial for accurately correlating data from various sources, especially in applications involving sensor fusion in Extended Kalman Filter (EKF) or RTAB-Map pipelines. Commonly used strategies for time synchronization include the Network Time Protocol (NTP), which synchronizes clocks across multiple devices with millisecond accuracy over the internet and finer precision in local networks. An alternative is Chrony, which excels in environments with variable network latency or intermittent connectivity, making it suitable for mobile or dynamic systems. Additionally, the built-in time synchronization mechanisms of the Data-Distribution Service (DDS) protocol allow for precise synchronization across nodes within the same DDS domain, facilitating seamless integration in ROS 2 applications.

To verify the effectiveness of the chosen time synchronization strategy against sensor fusion accuracy requirements, extensive testing was conducted by collecting sensor data from multiple sources such as LiDAR and cameras under varying synchronization conditions. The accuracy of the EKF or RTAB-Map outputs was assessed by comparing estimated positions and maps against ground truth data, utilizing metrics like Root Mean Square Error (RMSE) and covariance analysis. Results indicated that effective time synchronization significantly enhanced the reliability and accuracy of the fused data, confirming that the selected synchronization strategy met the performance requirements. By ensuring tight synchronization between nodes, the system maintained high fidelity in localization and mapping, which is essential for successful operation in complex environments.

4.2.3. Failover Mechanisms upon Communication Loss

In operations such as SLAM-based navigation and radiation mapping, maintaining system functionality during communication loss between edge devices and the cloud is essential. Several failover mechanisms can be implemented to ensure continued operation in these scenarios, enhancing the robustness and reliability of the system:

• Local autonomy: By enabling local processing capabilities, edge devices can continue executing critical operations independently when connectivity to the cloud is lost. For instance, in SLAM-based navigation, the edge device can utilize onboard sensors to perform real-time mapping and localization without relying on cloud-based data. This local autonomy allows the system to maintain functionality and complete tasks, even in the absence of a communication link.
• Redundant communication paths: ensure continued operation when communication between the edge and cloud is lost. These mechanisms enable mission-critical operations like SLAM-based navigation and radiation mapping to proceed with minimal disruption.
• Automatic switching to backup nodes: On the edge-side of the system, the corresponding backup nodes could be named e.g., /rgbd_camera_node_backup and /lidar_sensor_node_backup, providing redundancy for the primary sensor nodes. On the cloud-side, backup EKF localization node could be designated as /ekf_localization_node_backup, ensuring that higher-level processing tasks can continue seamlessly in the event of a failure. The mechanism for triggering automatic switching involves continuous health monitoring of the primary nodes, where each node periodically checks the status of its counterpart. If a primary node fails to respond within a predefined timeout period or if it detects a critical error, the system will automatically switch to the backup node. This can be implemented through a combination of watchdog timers and heartbeat signals, which ensure that the backup nodes are activated without significant delay. Additionally, the edge and cloud nodes can communicate their operational status through ROS 2 topics, allowing the system to maintain awareness of which nodes are active. This proactive approach to node management not only minimizes downtime but also enhances the overall reliability of the system, ensuring that critical operations like SLAM-based navigation and radiation mapping can continue without interruption.
• Health monitoring and diagnostics: This enables the system to detect communication issues proactively. The edge devices can continuously monitor the status of the communication link and the performance of critical subsystems. If a fault is detected, the system can trigger predefined failover procedures, such as switching to local autonomy or engaging backup systems. This proactive approach ensures that the system can respond to emerging issues before they escalate into significant failures.

4.2.4. Cybersecurity and Quality of Service Enforcement

The design of the ROS 2 DDS middleware stack utilizes several strategies to manage dynamic bandwidth allocation and enforce Quality of Service (QoS) parameters, particularly to prevent sensor data loss or delays during network congestion. This is especially crucial for throughput-intensive topics such as RGBD streams and LiDAR data, which require reliable and timely transmission to ensure accurate environmental perception and system performance.

One of the primary methods employed is the configuration of QoS policies, which allow developers to tailor the behavior of data transmission based on the specific requirements of each topic. The middleware supports various QoS settings, including reliability, durability, deadline, and latency budget. For instance, the Reliability QoS policy can be set to “Reliable”, ensuring that all messages are delivered even in the event of network congestion or packet loss. Additionally, the Latency Budget policy can be adjusted to specify acceptable latency thresholds for different data streams. This flexibility enables the system to prioritize critical sensor data, such as LiDAR and RGBD streams, while managing less critical data flows more conservatively.

To protect ROS 2 traffic from potential interception, spoofing, or command injection, we also implement a cybersecurity framework that includes the use of Transport Layer Security (TLS) for encrypted communication between nodes, ensuring data integrity and confidentiality over the network. Additionally, we employ a hardened communication protocol stack that incorporates security measures such as access control lists (ACLs), robust authentication mechanisms, and regular security audits to mitigate risks associated with physical attack vectors, such as Ethernet cabling, in field-deployed systems.

4.2.5. Dynamic Bandwidth Allocation

For our ROS 2 implementation, dynamic bandwidth allocation is further managed through the use of DataWriter and DataReader configurations that adapt to network conditions. This includes implementing mechanisms such as flow control and adaptive bitrate streaming, where the middleware can adjust the rate of data transmission based on current network capacity. When congestion is detected, the system can throttle back the transmission rate for less critical topics or implement strategies such as message aggregation to reduce the bandwidth footprint. By monitoring network performance metrics, the DDS stack can dynamically adjust its behavior, ensuring that high-priority data are transmitted promptly while maintaining overall system stability. These combined strategies effectively minimize the risk of sensor data loss or delay during periods of high throughput, thereby enhancing the robustness and reliability of the ROS 2 framework in demanding operational environments.

4.3. Mapping Tests

After performing distributed computing tests, we conduct mapping and navigation simulations in the tank pit environment. The reason for running this simulation and the subsequent real-world deployment is that we will need to deploy the crawler robot to traverse inside the pits to build 3D point cloud maps as well as perform structural health and radiation level monitoring. This simulation is fully ran on the cloud computer. However, the subsequent in-lab SLAM testing is completed using the distributed computing in the same ROS 2 domain as discussed in the section above.

Using open-source packages like RTAB-Map for this application can be advantageous. RTAB-Map is designed specifically for real-time operation and is well-suited for environments with dynamic changes, making it a good fit for office spaces or tank-like environments where conditions may vary. Its capabilities in visual-inertial odometry and robust loop closure detection allow it to effectively handle the complexities introduced by the robot’s extended arm and the surrounding environment. Additionally, being open-source enables flexibility in customization and integration with other components of the robotic system, allowing developers to tailor the SLAM implementation to meet specific requirements. Overall, RTAB-Map’s strengths in adaptability, efficiency, and community support make it a suitable choice for the present application, particularly in environments that demand both real-time performance and robustness in mapping and localization.

To extrapolate the office-based SLAM results to the complex geometries and harsh environmental conditions expected in the Hanford tank pits, we utilized simulation-in-the-loop techniques that model the specific characteristics of the target environment. This approach enables us to simulate various scenarios, including the presence of obstacles, varying light conditions, and thermal fluctuations, allowing the algorithms to be tested rigorously before deployment. The simulations provide a controlled environment where different SLAM parameters can be fine-tuned, ensuring that the algorithms are robust and capable of handling the complexities of the tank pits.

Additionally, by integrating real-world data and physical models into the simulation, we can assess how the SLAM algorithms will perform under the expected conditions, such as electromagnetic interference and the unique structural features of the tank pits. This iterative process not only refines the algorithms but also enhances their adaptability and reliability, ultimately leading to improved mapping accuracy and operational efficiency in the field. By validating our SLAM configurations through simulation-in-the-loop, we can confidently transition to real-world applications, knowing that the system is well-prepared for the challenges it will encounter.

Due to the regulatory restrictions by INL Classification Office, we are not permitted to disclose the results of the SLAM simulation inside the virtual tank pit environment to the public. However, the real-world SLAM testing inside an office space, as shown in Figure 12b, provides valuable insights and confirms the effectiveness of our software configuration for the future crawler robot, demonstrating the potential for efficient mapping using open-source packages like RTAB-Map [32]. In addtion, the tethered configuration as shown in Figure 12a is also mimicking the planned robot powering and communication approach.

The use of the Yahboom R2 robot for software validation introduces significant hardware abstraction; therefore, we have developed a strategic plan to carefully map the control and sensing stack to the Nexxis crawler’s architecture. This involves aligning software interfaces and communication protocols to ensure compatibility for SLAM, navigation, and sensor data processing. However, discrepancies are still anticipated in latency, actuator response, and sensor fidelity. Variations in processing power and communication bandwidth may lead to increased latency in command execution on the Nexxis crawler. Additionally, differences in mechanical design and control algorithms could affect actuator response, influencing maneuverability and precision. Finally, sensor fidelity may vary due to different calibration settings, specifications, or noise characteristics of the sensors on the Nexxis crawler, which could impact the accuracy and reliability of the data used for mapping and navigation.

5. Discussion

5.1. Overall Design Considerations

In this paper, we introduce the overall design plan, development timeline, and preliminary design and experimental progress of the APES project we have achieved so far. The proposed project timeline of the APES focuses on developing an advanced multi-robot system for efficient operation to inspect the nuclear waste tank pits at Hanford Site, WA, which is further structured into three phases. Phase 1 involves data collection and interface definition in collaboration with Hanford Site experts and university partners, focusing on tank riser geometry and hardware solutions. In Phase 2, the selection of sensors and robot components ensures the use of standard off-the-shelf parts, followed by detailed mechanical design and prototyping. Phase 3 integrates all components into a cohesive system, managed by a master control package developed by the INL Digital Engineering group, incorporating digital twin and surrogate models. The final phase involves building a simulated tank pit at INL for comprehensive testing and validation, culminating in an on-site demonstration.

Also in Phase 1 and the early stages of Phase 2, the system’s communication design includes direct interfaces between the FIU robotic arm, Nexxis crawler robot, onboard electrical cabinet, control panel, and EV pickup truck power source, ensuring coordinated operation through shared data, power, and control signals for effective task execution in hazardous environments.

Also in Phase 2, the transportation and deployment of the entire system were critical design considerations. Operating multiple robots and sensors in a remote location presents logistic challenges, particularly power supply and recharging. The F-150 Lightning, for instance, can deliver up to 9.6 kW of AC power through its onboard outlets, effectively acting as a mobile generator for tools or electronics in the field. In fact, Ford advertises that the extended-range Lightning’s battery (131 kWh) can power an average home for about three days during an outage [34]. APES leverages this capability by using the EV as a base station: the robots can recharge from the truck’s battery via high-power outlets, and the vehicle can also carry computing equipment, communication relays, or even a robot manipulator arm. The EV’s role is not just a passive charging dock; it can autonomously follow the robot team to provide on-demand power and networking. Other works include installing robotic payload on Segways and iRobots to form a convoy [35], and using a number of Clearpath Huskies as chassis to carry robotic arms to perform construction tasks [36]. Both works, however, are limited to indoor, lab space uses only and may not sustain long-time deployment to meet the requirement to be deployed continuously for a 10 h shift at nuclear waste storage sites.

To address the power requirements, we decided to use an EV pickup truck, in this case study a Ford F-150 Lighting, which can support the system for a full 10 h shift with one charge. In addition, the tooling mechanism for the robot was designed to ensure stability and reliable data recording in the tank pits. An internal pipe clamping mechanism was developed to fit inside the pipe, providing necessary stability without taking up additional external space. Last but not least, after evaluating different arm designs, we tentatively selected the telescopic arm design over the scissor arm. The telescopic arm can extend and retract linearly, providing greater reach into deep and confined spaces, which is particularly useful in narrow and deep access points typical of nuclear waste tank pits.

5.2. SLAM Performance Evaluations

Utilizing a distributed computing approach, we have demonstrated the feasibility of splitting computational tasks between edge and cloud computer inside the same ROS 2 domain. The successful execution of SLAM tasks, as well as the efficient generation of colored point cloud maps and odometry estimates from the cloud side, validates our approach.

In fact, ROS 2 has already been field-tested in large-scale multi-robot scenarios. ROS 2 was tested to have lower data-loss rates and message latency between nodes therefore outperforms ROS 1 in real-time performance [37]. Also in DARPA’s Subterranean Challenge of 2020, a hybrid ROS 1/ROS 2 networking system allowed a fleet of heterogeneous robots to maintain communications and share observations deep underground [38]. In light of many peers’ works, APES uses ROS 2 for its built-in support of distributed computing and reliability features, ensuring that each robot (and any remote operators) stay interconnected even in the complex geometry and radio frequency (RF)-unfriendly conditions around thick concrete waste tanks.

In addition, simulating feature-scarce environments, low-light conditions, and occluded geometry is crucial for testing the performance of SLAM algorithms prior to having detailed digital twins of tank pits. One effective approach is to utilize 3D modeling software such as Blender, Gazebo, or Unity to create synthetic environments that replicate the characteristics of tank pits. These environments can be designed with minimal distinct features or landmarks to assess the algorithm’s ability to identify and track sparse features. Additionally, introducing objects that create occlusions, such as walls and barriers, allows for the evaluation of the algorithm’s performance in maintaining localization and mapping accuracy under limited visibility conditions. Adjusting lighting settings to simulate low-light environments further enhances the realism of the simulations.

Incorporating variability within the synthetic environments is essential for thorough testing. This can be achieved by adjusting parameters such as sensor noise, camera characteristics, and illumination levels. By introducing noise to the sensor data from RGB-D cameras or LiDAR, the robustness of the SLAM algorithm can be evaluated under more challenging conditions. Furthermore, varying camera parameters, such as exposure settings and frame rates, allows for an assessment of the algorithm’s adaptability to different sensor configurations typical of low-light scenarios. Controlled real-world testing in environments like laboratories or warehouses also provides valuable insights, enabling the simulation of feature-scarce and occluded conditions while ensuring control over lighting and object placement.

To enhance the simulation process, robotics simulation frameworks including Nvidia Isaac Sim and ROS 2 Gazebo are integrated to create digital twin environments. These platforms allow for the creation of dynamic simulations that incorporate realistic physics and sensor data. Data-augmentation techniques can also be applied to existing datasets to simulate low-light and feature-scarce conditions by applying image-processing filters or randomly obscuring features. By employing these comprehensive simulation strategies, researchers can effectively validate and optimize SLAM algorithms, ensuring they are well-prepared for deployment in real-world tank pit environments where such challenges are prevalent.

5.3. Issues with Mapping Software Deployment

Due to the delayed availability of Nexxis crawler robot, we employed a Yahboom Rosmaster R2 robot with a Jetson Orin Nano onboard computer for preliminary testing. Despite the limited computational resources of the Jetson Orin Nano, we managed to selectively run critical ROS 2 nodes and topics, refining our code and resource-management strategies. The robot interfaced with RGB-D cameras and 2D LiDAR, publishing essential data streams to the shared ROS 2 domain for higher-level processing on the Syslogic computer. This setup ensured real-time responsiveness at the edge while leveraging the cloud’s computational power for intensive processing tasks such as SLAM, sensor fusion, and visualization.

However, this mapping process introduces discrepancies in latency, actuator response, and sensor fidelity. The Yahboom R2 may exhibit lower latency due to its optimized hardware, while the Nexxis crawler may experience delays in data transmission and processing. Differences in motor specifications between the two systems can lead to variations in actuator response, affecting the execution of motion commands. Furthermore, discrepancies in sensor fidelity may arise from variations in sensor resolution and accuracy, impacting the quality of environmental perception and navigation capabilities.

To address these discrepancies, several mitigation strategies are implemented. Calibration and tuning of the control algorithms are essential to harmonize actuator performance and sensor data across both platforms. Latency compensation techniques, such as predictive modeling, help minimize the impact of delays on control command execution. Robust error handling mechanisms are also integrated to enhance system reliability, allowing the APES system to achieve accurate software validation and reliable operation in real-world scenarios. These efforts ultimately contribute to the robustness and effectiveness of the system, enabling it to perform complex tasks in challenging environments.

5.4. Challenges and Strategies for Developing Digital Twins

Digital twins are currently being targeted for systems and facilities that have permanently installed instrumentation and control systems. The high radiation environment of the Hanford tank pits doesn’t allow for installation of sensors that would survive between the three-year inspection periods. This creates a challenge for creating digital twins for the pit environments. The team proposes that a digital twin of an environment can instead be created using signals captured by robotics during inspection.

Whereas a digital twin usually implies real-time sensing of the system being observed, the processes that occur to degrade these pits occur over long periods of time. Moving the inspections from a three-year basis to a shorter basis would not only allow for a higher likelihood of detecting an issue sooner, but would also create the datasets needed for AI to perform anomaly detection. The digital twin will include multiple signals of the pit including: radiation, temperature, pressure, and humidity. Utilizing these signatures over the period of the year, it could be determined what signals indicate an anomaly rather than just a change of environment in or around the pit environment.

One challenge that will be difficult to overcome is the lack of any existing datasets that are sufficient to train an AI algorithm. Part of this challenge can be tacked using virtual environment mocking the conditions expected to be in a pit with NVIDIA Isaac. It is currently assumed the signals of a ‘healthy’ pit will largely be shared across many of the pits to be inspected. Thus, initial training of the AI models can be done with signals gathered from multiple pit inspections rather than repeat inspections of the same pit, which will allow for quicker production of anomaly detection. Once the model is trained and pits are reinspected, the model can be fine-tuned into digital twins for each individual pit.

6. Conclusions

In conclusion, this paper has outlined the comprehensive design plan, development timeline, and preliminary progress of the APES project, which aims to develop an advanced multi-robot system for efficient inspection of nuclear waste tank pits at the Hanford Site, WA. The project is structured into three critical phases. Phase 1 emphasizes data collection and interface definition through collaboration with experts and partners, focusing on tank riser geometry and hardware solutions. Phase 2 concentrates on the selection and integration of sensors and robot components, ensuring the use of standard off-the-shelf parts, followed by detailed mechanical design and prototyping. Phase 3 involves integrating all components into a cohesive system managed by a master control package, incorporating digital twin and surrogate models, culminating in comprehensive testing and validation in a simulated tank pit at INL, and finally, an on-site demonstration.

The proposed multi-robot system for autonomous inspection offers significant benefits, including increased operational efficiency, enhanced safety for human operators, and the ability to cover large and hazardous areas systematically. In addition, the protection strategies discussed in Section 3.4 may guard the system from radiation hazards and electromagnetic interferences (EMI), and therefore assure smoother data collection when monitoring pits structural health and radiation levels, thereby facilitating on-site decision-making and risk mitigation. Additionally, the conceptual clamping and tooling designs discussed in Section 3.3 highlight the adaptability of the robotic system to various inspection tasks, which can be crucial in the dynamic and complex environments of nuclear waste-storage facilities.

Future work will focus on the deployment and testing of the actual crawler robot in the nuclear waste tank pits. We aim to enhance the computational capabilities of the edge devices to handle more ROS 2 nodes and topics, reducing reliance on cloud-based systems. Additionally, we plan to integrate advanced sensor fusion techniques and machine learning algorithms to improve the accuracy and efficiency of structural health and radiation level monitoring. By addressing these challenges, we seek to develop a robust and reliable multi-robot system that can operate autonomously in hazardous environments, contributing to the safe and efficient inspection of nuclear waste storage facilities.