Search Results (6)

This paper presents a novel synthetic learning-based approach for solving the component-to-slot assignment problem in robotics using a SCARA robot. The method uses a fully simulated environment that generates and annotates scenes based on rules and visual features. Within this environment, we train a permutation-invariant neural model to predict correct assignments between detected components and predefined target slots. Set Transformer-based encoders are combined with a self-attention MLP scoring head. Assignment prediction is optimized using an improved soft Hungarian loss function. To increase data realism and generalizability, we implement a synthetic dataset generation module on the NVIDIA Omniverse platform. This setup enables precise control over scene composition and component placement. The resulting model achieves high matching accuracy on complex layouts with variable numbers of components and demonstrates strong generalization across multiple configurations. Our results validate the feasibility of learning bijective mappings in simulated assembly scenarios, providing a foundation for scalable real-world robotic pick-and-place tasks. Tests were also conducted on actual robot units. Full article

High-quality 3D objects play a crucial role in digital twins, while synthetic data generated from these objects have become essential in deep learning-based computer vision applications. The task of collecting and labeling real defects on industrial object surfaces has many challenges and efforts, while synthetic data generation feasibly replicates huge amounts of labeled data. However, synthetic datasets lack realism in their rendered images. To overcome this issue, this paper introduces a single framework for 3D industrial object reconstruction and synthetic defect generation for digital twin smart factory applications. In detail, NeRF is applied to reconstruct our custom industrial 3D objects through videos collected by a smartphone camera. Several NeRF-based models (i.e., Instant-NGP, Nerfacto, Volinga, and Tensorf) are compared to choose the best outcome for the next step of defect generation on the 3D object surface. To be fairly evaluated, we train four models using the Nerfstudio framework with our three custom datasets of two objects. From the experiment’s results, Instant-NGP and Nerfacto achieve the best outcomes, outperforming all other methods significantly. The exported meshes of 3D objects are refined using Blender before loading into NVIDIA Omniverse Code to generate defects on the surface with the Replicator. To evaluate the object detection performance and to verify the benefits of synthetic defect data, we conducted experiments with YOLO-based models on our synthetic and real-plus-synthetic defects. From the experiment’s results, the synthetic defect data contribute to improving YOLO models’ generalization capability with the highest and lowest accuracy mAP@0.5 enhancement of 18.8 and 1.5 percent on YOLOv6n and YOLOv8s, respectively. Full article

Digital twins are increasingly utilized across all lifecycle stages of physical entities. Augmented reality (AR) offers real-time immersion into three-dimensional (3D) data, which provides an immersive experience with dynamic, high-quality, and multi-dimensional digital twins. A robust and customizable data platform is essential to create scalable 3D digital twins; Universal Scene Description (USD) provides these necessary qualities. Given the potential for integrating immersive AR and 3D digital twins, we developed a software application to bridge the gap between multi-modal AR immersion and USD-based digital twins. Our application provides real-time, multi-user AR immersion into USD-based digital twins, making it suitable for time-critical tasks and workflows. AR digital twin software is currently being tested and evaluated in an application we are developing to train astronauts. Our work demonstrates the feasibility of integrating immersive AR with dynamic 3D digital twins. AR-enabled digital twins have the potential to be adopted in various real-time, time-critical, multi-user, and multi-modal workflows. Full article

Virtual reality software might be challenging to utilize for beginners and unskilled professionals who do not have a programming or 3D modeling background. Concurrently, there is a knowledge gap in software project design for intuitive virtual reality authoring tools, which were supposed to be easier to use. These tools are frequently insufficient due to a lack of support and standard operating procedures. This study evaluates the validity of fourteen design guidelines for the development of intuitive virtual reality authoring tools. Adopting the Design Science Research approach, a previous study completed the first steps of the protocol by identifying problems, defining solution objectives, and developing and demonstrating the design guidelines. This work evaluates their application in a experiment. A group of engineering students with no prior experience in creating virtual worlds were tasked with examining the design guidelines while using the NVIDIA Omniverse Enterprise as an exemplary use case by following the tutorials in the platform. Previously, the students answered a Likert-scale questionnaire and a focus group interview, with eighteen questions about how they perceived these guidelines. An average score value for the tool was estimated through the questionnaire answers and a correlation analysis and, using the Pearson Correlation Coefficient (PCC), it was confirmed that most guidelines scores behaved as expected and were ranked according to the use-case functionality. The participants understood the guidelines’ definition and could decide if they agreed or disagreed with their presence during the experiment. We evaluated that, in accordance with the Design Science Research, the proposed artifact is useful, i.e., the design guidelines for virtual reality authoring tools perform what they are designed to do and are operationally reliable in accomplishing their goals. Full article

Various software environments have been developed in the past to create digital twins of single cells or a digital twin of a factory. Each environment has its own strengths and weaknesses and has been designed with a specific focus. The environments that are able to holistically simulate complete factories are limited in terms of the modelling details required for the analysis of single manufacturing cells (e.g., manufacturer-independence of the individual digital twins) and their ability for virtual commissioning. This paper presents three options for realising a virtual commissioning of linked cells using a 3D integration platform with NVIDIA Omniverse, consisting of two different digital models fused into a combined model, also representing material flow. First, with a source/sink solution and unidirectional connector controlled by OPC UA; secondly, with a bidirectional connector, developed in the course of this elaboration, and an extension of the 3D integration platform controlled by Apache Kafka; thirdly, with a bidirectional connector and using only an extension of the 3D integration platform. The research demonstrates that virtually commissioning multiple linked digital twins from different manufacturers in a 3D platform with material flow makes a significant contribution to the industrial metaverse. Full article

This paper proposed a multi-layered 3D NDT (normal distribution transform) scan-matching approach for robust localization even in the highly dynamic environment of warehouse logistics. Our approach partitioned a given 3D point-cloud map and the scan measurements into several layers regarding the degree of environmental changes in the height direction and computed the covariance estimates for each layer using 3D NDT scan-matching. Because the covariance determinant is the estimate’s uncertainty, we can determine which layers are better to use in the localization in the warehouse. If the layer gets close to the warehouse’s floor, the degree of environmental changes, such as the cluttered warehouse layout and position of boxes, would be significantly large, while it has many good features for scan-matching. If the observation at a specific layer is not explained well enough, then the layer for localization can be switched to other layers with lower uncertainties. Thus, the main novelty of this approach is that localization robustness can be improved even in very cluttered and dynamic environments. This study also provides the simulation-based validation using Nvidia’s Omniverse Isaac sim and detailed mathematical descriptions for the proposed method. Moreover, the evaluated results of this study can be a good starting point for further mitigating the effects of occlusion in warehouse navigation of mobile robots. Full article