Next Article in Journal
Modeling Mortality Based on Pollution and Temperature Using a New Birnbaum–Saunders Autoregressive Moving Average Structure with Regressors and Related-Sensors Data
Next Article in Special Issue
Microwave Tomography Using Neural Networks for Its Application in an Industrial Microwave Drying System
Previous Article in Journal
Meander Thin-Film Biosensor Fabrication to Investigate the Influence of Structural Parameters on the Magneto-Impedance Effect
 
 
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Supporting Visualization Analysis in Industrial Process Tomography by Using Augmented Reality—A Case Study of an Industrial Microwave Drying System †

1
Division of Interaction Design and Software Engineering, Department of Computer Science and Engineering, Chalmers University of Technology, SE-41296 Gothenburg, Sweden
2
Institute for Pulsed Power and Microwave Technology (IHM), Karlsruhe Institute of Technology (KIT), 76133 Karlsruhe, Germany
3
Department of Applied Physics, University of Eastern Finland, FI-70210 Kuopio, Finland
*
Author to whom correspondence should be addressed.
This paper is an extension version of the conference paper: Zhang, Y.; Yadav, R.; Omrani, A.; Fjeld, M. A Novel Augmented Reality System To Support Volumetric Visualization in Industrial Process Tomography. In Proceedings of the 2021 Conference on Interfaces and Human Computer Interaction, Online, 21–23 July 2021.
Sensors 2021, 21(19), 6515; https://doi.org/10.3390/s21196515
Received: 2 September 2021 / Revised: 24 September 2021 / Accepted: 26 September 2021 / Published: 29 September 2021
(This article belongs to the Special Issue Tomographic Sensors for Industrial Process Control)

Abstract

:
Industrial process tomography (IPT) based process control is an advisable approach in industrial heating processes for improving system efficiency and quality. When using it, appropriate dataflow pipelines and visualizations are key for domain users to implement precise data acquisition and analysis. In this article, we propose a complete data processing and visualizing workflow regarding a specific case—microwave tomography (MWT) controlled industrial microwave drying system. Furthermore, we present the up-to-date augmented reality (AR) technique to support the corresponding data visualization and on-site analysis. As a pioneering study of using AR to benefit IPT systems, the proposed AR module provides straightforward and comprehensible visualizations pertaining to the process data to the related users. Inside the dataflow of the case, a time reversal imaging algorithm, a post-imaging segmentation, and a volumetric visualization module are included. For the time reversal algorithm, we exhaustively introduce each step for MWT image reconstruction and then present the simulated results. For the post-imaging segmentation, an automatic tomographic segmentation algorithm is utilized to reveal the significant information contained in the reconstructed images. For volumetric visualization, the 3D generated information is displayed. Finally, the proposed AR system is integrated with the on-going process data, including reconstructed, segmented, and volumetric images, which are used for facilitating interactive on-site data analysis for domain users. The central part of the AR system is implemented by a mobile app that is currently supported on iOS/Android platforms.

1. Introduction

1.1. Industrial Process Tomography

Industrial process tomography (IPT), as a non-intrusive and commonly-used imaging technique, has been effectively harnessed in many manufacturing components for inspections, monitoring, product quality control, and safety issues [1,2,3,4,5,6]. IPT underpins and facilitates the extraction of qualitative and quantitative data regarding the related industrial processes, which is usually visualized in various ways for people to understand its nature, measure the critical process characteristics, and implement process control in a complete feedback network [7]. Some typical representatives of IPT, such as microwave tomography (MWT) [8,9,10,11,12,13,14,15], electrical resistance tomography (ERT) [16], and electrical capacitance tomography (ECT) [17] are widely used for industrial purposes such as moisture detection [8,9], crack detection and powder flow in pipes, and flow pattern detection of granules. In our study, we concentrate on a unique industrial microwave drying process [10] which uses precise drying and heating equipment for polymer foams with the aid of MWT, as displayed in Figure 1. The targets of this drying process are firstly heating the object foam placed on the conveyor belt and then detecting the post-drying moisture distribution, by adopting the MWT technique. In this context, it is therefore critical for domain users to comprehend and interpret the process data for intelligent control purposes. Hence, how to develop effective tools or systems which provide people with straightforward and comprehensible visualizations pertaining to the process data becomes a central topic. In this work, we propose a novel and comprehensive solution to support process-critical visualization analyses through the specific case–the MWT-controlled industrial drying processes–by exploiting the interactive augmented reality (AR) technique.

1.2. HEPHAISTOS Heating Technology and Microwave Tomography

Drying by microwaves has been widely used, especially in industry, for different applications and purposes. Microwaves can penetrate into the material and provide volumetric heating, in contrast to conventional methods. In microwave drying applications, providing sufficient uniformity of heating distribution is imperative, especially in industrial-scale production [18]. Intelligent control [19] of distributed microwave sources (magnetrons) is a novel idea to increase the efficiency of the drying system, resulting in high-quality processing and reducing the drying time, both of which are highly desirable for the industry. During the drying process, it is only possible to adjust the amplitude and pulse duration of the magnetrons. The decision for controlling these parameters is chosen based on the input information of the moisture location and its level inside the sample.
The microwave drying system which we are presently working with is named HEPHAISTOS (shown in Figure 1). The oven has length and height of 3 m and 1 m, respectively, and is equipped with 18 magnetron sources operating at a frequency of 2.45 GHz. This industrial microwave applicator has a patented hexagonal [20,21] cross-section design delivering high electromagnetic (EM) field homogeneity during the drying process. The system is equipped with a conveyor belt that enables the continuous drying process. Its principal areas of applications are in material processing, for example, thermal curing of fiber composites and drying of porous foams. An MWT system [8] is designed and integrated with the HEPHAISTOS to recover the volumetric information of the moisture location and its level during the drying process of the polymer foam. The power level and pulse duration of the magnetrons will be adjusted based on the input information from the MWT.

1.3. Visualization and Augmented Reality

Visualization plays a dominant role in tomography-controlled industrial processes in that it can display the relevant data in an observation-friendly manner, which is then provided to domain users. The information related to the designated process, for example, a 2D graph of the tomographic spectrum or a 3D reconstruction of the voxel data, if collected and visualized instantly in a reasonable way, can help in better understanding of the process among the users [7]. Sophisticated data analysis always demands efficient reproduction of the measurement regarding the imaging process by IPT to uncover the material distribution inside the closed containers where the industrial process is executed [22]. It is therefore imperative to observe and analyze the reconstructed tomographic images which imply the state and status of the process flow, supported with tomograms and graphs, by desirable visualization without interfering the ongoing industrial processes directly [17,23]. Thus, how to get domain users instant and accurate access to various process data by effective visualizations so as to implement real-time data observation and analysis throughout the whole process becomes the main argument of this article.
One of the cutting-edge techniques—augmented reality (AR) (a variation of virtual reality (VR)), in which virtual objects are superimposed in the real world [24]—has been demonstrated and applied in numerous fields due to its capability of providing interactive interfaces of visualized digital content [6,25]. Moreover, AR can provide functional tools that support users undertaking domain-related tasks, especially facilitating them in data visualization and interaction because of its ability to jointly augment the physical space and the user’s perception [7,26]. The adoption of AR in benefiting IPT and its related fields is currently still scarce, resulting in a gap between AR developers and industrial applications [27]. However, some recent research has already pioneered the exploitation of AR to diversify the data visualization and analysis for industrial settings, especially within the context of IPT [6,17,23]. Based on the interactivity and accessibility which AR can provide, we aim to explore the deeper usage of this technique in the IPT domain to support the process data visualization and analysis, by exemplifying a complete case study of the industrial microwave drying process with MWT measurement. The main contributions of this work are as follows:
  • Propose an entire data processing and visualizing workflow of the IPT controlled industrial process.
  • Pioneer the study of integrating the up-to-date AR technique to support IPT data visualization and on-site analysis for domain users.
This article is organized as follows: In Section 2, the state-of-the-art related work of MWT applications and the exploration of AR in related contexts are introduced. In Section 3, An comprehensive overview of the data processing and visualizing workflow of the case—MWT for microwave drying is presented, including the engaged equipment and materials, of which the time reversal imaging algorithm, the post-imaging segmentation, and the volumetric visualization. The conceptualization and the specification of how the AR approach can support effective data visualization and analysis are given in Section 4. In Section 5, the results of the case study are presented and discussed. The concluding remarks and future work are given in Section 6.

2. Related Work

2.1. Microwave Tomography for Industrial Process Applications

A comprehensive review on applications of the MWT in the process industry is listed in [28]. Microwave sensing and imaging methods for application in multi-phase flow monitoring and metering in the oil and gas industry are given in [29,30]. In the food industry, MWT has been widely applied for inspection and quality control [31,32,33]. Sensing of low permittivity materials, such as gaseous substances or foam with high air content, with MWT is reported in [34]. Moreover, the authors have reported their contribution in [9,35,36,37] for the present application of MWT in estimating the moisture content in polymer foam.

2.2. Visualization and Augmented Reality

Under the circumstances that research into using AR to benefit IPT related fields was insufficient, over the last two decades researchers have been investigating how to apply AR to provide novel and effective visualization approaches for tomographic process data. In 2001, Mann et al. [38] developed an AR application to capture and visualize the dynamics of the mixture in stirred chemical reactors operated by ERT. Recently, Nowak et al. [17] benchmarked a new AR solution to visualize industrial tomography data and further enable collaborative in-situ analysis, by making use of the up-to-date Microsoft HoloLens 2 headset (https://www.microsoft.com/en-us/hololens, 15 August 2021). More interestingly, they formulated deeper research to investigate a more advanced prototype for underpinning complex data visualization and analysis in entire 3D surroundings within the same IPT context [23]. On the premises that some research has opened the horizon of adopting cutting-edge AR to generate effective visualizations for IPT data, we have proposed a novel AR framework for better volumetric visualization and onsite analysis for MWT aided processes [6]. Based on this previous work, this study explores more efficient AR methodologies for domain users by replenishing a complete data processing and visualizing workflow as well as improving the central AR application.

3. Data Processing and Visualizing Workflow

The case we deal with in this article is a unique microwave drying process for polymer materials with the aid of MWT measurement. To observe and monitor the whole process, it is critical to unveil the intact dataflow with regard to every phase, and displaying it in a visualization-friendly manner. In this part, we introduce the proposed data processing and visualizing pipeline of our case, consisting of three modules: Time reversal imaging, post-imaging segmentation, and volumetric visualization.

3.1. Practical Challenges

To design and integrate an MWT system into an industrial drying system the following practical guidelines should be taken into account. First, the unwanted EM leakage power coming from the entrance aperture can take down the MWT system, and therefore has to be blocked. In this regard, many types of antennas which are already employed for microwave imaging purposes cannot be used in this application. Here, the X-band frequency range for the MWT is chosen, and the WR-90 open-waveguide is employed to act simultaneously as an antenna and microwave high pass filter. Second, fast data acquisition is essential to enable real-time monitoring and providing rapid input information for the control unit. Hence, a high number of antennas cannot be chosen to develop the MWT sensor and more so applying iterative-based optimization techniques are challenging. Third, the imaging modality should be compatible facing different conditions like low or high-contrast media. Consequently, the developed microwave imaging should be compatible with the low number of antennas and be able to perform under the mentioned conditions.

3.2. Microwave Tomography: Time-Reversal Imaging Algorithm

Here, we use the qualitative time-reversal(TR) imaging for detecting the moisture location inside the polymer foam. The super-resolution characteristic of TR imaging facilitates the detection of the targets even in rich clutter media. The imaging set-up for a fixed cross-section (i.e., y × z ), is shown in Figure 2 where layer 0 is the free-space, layer 1 represents the polymer foam with thickness t 1 , and layer 2 is either free-space or a perfect electric conductor (PEC). The multistatic antenna array with N element is fixed and the distance of the antenna to the top of the polymer foam is t 0 , and the center to center distance between two adjacent antennas is d. In free space, the relative dielectric constant is denoted as ϵ r , 0 whereas the relative dielectric constant of the layer 1 and layer 2 are set to ϵ r , 1 and ϵ r , 2 , respectively. In the next section, the multistatic data matrix (MDM) for the electric field will be built, and an asymptotic expression for the Green’s function of the multilayer media will be obtained and used in TR imaging. Here, the main interest is in imaging a subsurface target and extracting a representative 3D model for AR visualization and not determining the target material.

3.2.1. Scattering Model and Time Reversal Imaging

Consider an active array of N tranceivers from which an N × N MDM is constructed. Each element of the MDM matrix represents the received scattered field by ith ( i = 1 , 2 , , N ) antenna when jth ( j = 1 , 2 , N ) antenna is in the transmitting mode due to the inhomogeneities in the region of the interest (ROI) can be expressed [39,40,41]
E sct ( 1 ) ( ρ r i , ρ t j ) = i ω μ 0 Ω 1 G ¯ ¯ eb ( 01 ) ( ρ r i , ρ ) · O ( ρ ) E tot ( 1 ) ( ρ ) d ρ .
Here, E tot ( 1 ) denotes the total electric field in layer 1 and E sct ( 1 ) is the scattered field received by the ith antennas. O ( ρ ) = i ω ϵ 0 ( ϵ Ω 1 ϵ r , 1 ) is the object function where ϵ Ω denotes the target relative dielectric constants in the domain Ω 1 , and μ 0 and ϵ 0 denote free-space permeability and permittivity, respectively. Here, time convention of e i ω t is assumed and suppressed where ω is the angular frequency. In (1), the vector ρ r and ρ t represent source and observation points while ρ = ( y , z ) is the location of the pixel point in the ROI. G ¯ ¯ eb ( 01 ) is the background (multilayer media without any inhomogeneities) dyadic Green’s function (DGF). The superscript ( 01 ) denotes that the source point is located in layer 1 and the observation point is in layer 0. It should be noted that the analytical equations are written for the electric fields, however, a de-embedding procedure will be applied later to convert the measured S-parameters to the electric field.
A multistatic model for the received scattered field that describes each element of the MDM matrix can be obtained and written as
E sct ( 1 ) ( ρ r i , ρ t j ) = i ω μ 0 Ω 1 G eb x x ( ρ r i , ρ ) O ( ρ ) G eb x x ( ρ , ρ t j ) d ρ .
In deriving (2) from (1) the symmetry property of the DGF is used, and furthermore, it is assumed that both transmitting and receiving antennas are x-polarized, so the x x term of the DGF is employed. The MDM can be expressed in the compact form in the angular frequency domain ω as
K ( ω ) = i ω μ 0 Ω 1 O ( ρ ) g b ρ i ( ρ , ω ) g b ρ j ( ρ , ω ) d ρ ,
where g b ρ = [ G e b x x ( ρ , ρ 1 ) , G e b x x ( ρ , ρ 2 ) , . . . , G e b x x ( ρ , ρ N ) ] N × 1 is the frequency-domain steering vectors of the x x components of the background DGF and · is the transpose operator. Decomposition of the time-reversal operator (DORT) can be applied to selectively focus on the inhomogeneities in the media using singular value decomposition (or SVD) of the MDM matrix [42]. The SVD’s of the matrix K ( ω ) is expressed in terms of the eigenvalues via the equation K ( ω ) = U ( ω ) Λ ( ω ) V ( ω ) , where Λ d i a g ( Λ ) = [ ν 1 , ν 2 , , ν N ] is a real diagonal matrix consisting of eigenvalues, while U u i ( ω ) , i = 1 , 2 , , N and V v i ( ω ) , i = 1 , 2 , , N are the left and right matrices consisting of normalized eigenvectors [42,43] of time reversal operator (TRO).
For focusing on the pth scatterer, we employ e p ( ω ) = λ p ( ω ) u p ( ω ) as a new excitation for the transmitting antennas and calculate the propagated fields using the x x component of the background DGF. Here, λ p is the pth eigenvalue and u p is the pth left singular vector. As a result the location of pth scatterer is synthetically obtained by the following imaging function
D p ( ρ ) = ω e p T ( ω ) g b ρ ( ρ , ω ) d ω ,
where ρ is any arbitrary point in the ROI.

3.2.2. TR-DORT Simulation Results

In this section, TR-DORT imaging simulation results are demonstrated using the derived asymptotic Green’s function for different moisture scenarios. To generate the synthetic electric-field data, the 3D scenario of the MWT schematic shown in Figure 2 is developed in the commercial software CST Studio Suite with time-domain solver. In the 3D setup, the foam with dimension Ω foam = [ 25 , 25 ] × [ 15 , 15 ] × [ 4 , 4 ] is chosen and the open-ended waveguide antennas is placed in free space at a distance of 16 cm from top of the foam. The 7 × 7 MDM matrix is extracted for the bandwidth of 4 GHz with the center frequency of 10 GHz (X-band range) and the number of frequency is N f = 1001 . It should be noted that the response of the antennas is removed from the measured S-parameter using a calibration proposed in [44,45]. In this study, the dielectric constant of the dry polymer foam is assumed to be known and is ϵ r , 2 = 1.16 0.01 j . The wet-spots are modelled by a spherical shape with the defined dielectric constant corresponding to the different moisture levels. The correlation between the dielectric constant and the wet-basis moisture levels is measured by cavity-perturbation method as well as transmission line technique at 2.45 GHz [46]. For this characterization, the polymer foam density is 28 kg m 3 .
In the first scenario, we have considered one wet-spot in the polymer foam with a radius of 1.5 cm located at the center of the polymer foam, i.e., ( 0 cm , 0 cm in y z plane with 30 % moisture level and rest of the foam is considered dry). Figure 3 shows the reconstruction image associated with the first strong eigenvalue, and employing the derived Green’s function. As can be seen, using TR-DORT, the image domain is decomposed into the wet-spot location and dry parts. Further, we have considered multiple wet-spots with spherical shape (and same moisture levels) located at different positions in certain length of the foam. For this case, the reconstruction using the TR algorithm are performed at different time-stamps and are shown in Figure 4.

3.3. Post-Imaging Segmentation

After the designated drying process, under most circumstances there will be some regions of the polymer foam not being completely dry. The difference between dry and non-dry areas is represented by allocating different colors in the reconstructed MWT images, which entail different moisture levels retained in the polymer foams. As shown in Figure 4, the bright yellow parts stand for high moisture while the dark blue parts stand for low moisture. The colors become more yellow when there is more moisture. There exists significant implication for domain users to precisely assess the moisture distribution of the material so as to gauge the success of the conducted drying process.
To comprehensively help people to understand the outputted results, as well as to increase the accountability regarding the visualization, we deployed an entirely automatic segmentation method to visualize the high moisture areas on the reconstructed MWT images. Following our previously published work [12], we harness the MWT Segmentation based on K-means (MWTS-KM) segmentation algorithm to highlight the desired high moisture areas of the post-dried polymer foams on the reconstructed MWT images engaged in our study. As shown in Figure 5, we implement the segmentation on the corresponding nine MWT images originating from different timestamps of the drying process. The high moisture areas are displayed with light grey colors. Hence, the domain users are able to observe the segmented results to either conduct further visualization analysis or evaluate the success of the process.

3.4. Volumetric Visualization

For volumetric visualization of the moisture in the foam, the reconstructed moisture in the cross-section of the foam from t = 1 , 2 , 9 is used. Following the post-imaging segmentation process, the processed TR reconstructed images at different time slots are stacked to represent the three-dimensional information of the moisture, as shown in Figure 6. The dominant wet-spots are represented by the color yellow whereas the dry part is represented by the blue color. The yz plane is discretized into 80 × 20 pixels for simplified rendering and real-time display.

4. Augmented Reality for Visualization

To better support the onsite analysis related to the informative data processing and visualizing workflow, we present a novel AR framework which enhances the interactivity and immediacy of data analysis. The proposed system has three main components, as displayed in Figure 7.
  • MWT-controlled industrial microwave drying process: A unique heating and drying process operated in a confined chamber with sophisticated industrial settings. In our study, the target is a microwave drying process for polymer foams undergone by the precise HEPHAISTOS microwave oven system shown in Figure 1.
  • Users: Operators who control and run the drying equipment, or researchers who take onsite observations and collect data for further analysis.
  • AR App: The core part of our proposed system. As a preliminary stage, the application is manifested as a mobile App run in iOS/Android mobile devices, and used for interactive and collaborative volumetric visualization and analysis. In comparison with our previous work [6], a multiple floating interface provided and overlaid in a real environment and containing information from our proposed data workflow is the main component of the application.
The principal part of the initiated AR system is a mobile App developed to visualize the necessary information revealed from the aforementioned data processing and visualizing pipeline, while it supports interactive and collaborative onsite data analysis. This App serves as an intermediary tool between the users and the precise experimental equipment used for displacing the visualized information regarding the ongoing process. The users, for example, operators, can interact with the visualizations projected on the AR interface, with the mobility of merely holding a mobile device alongside the bulky equipment. As Figure 8 illustrates, people from different communities, such as practical operators or academic researchers, have the capacity to interactively and instantly observe the information related to the process by simply activating the App without initiating other facilities. The App was developed via Unity and Vuforia engines, aiming at the present stage for iOS/Android portable devices such as smartphones and tablets. By pointing the camera of the devices towards the precise equipment (the HEPHAISTOS system) used for performing drying and heating, the App will then be activated, displaying a floating interface with the necessary information (figures, images, etc) virtually superimposed. Compared to our previous work [6], the application itself has been ameliorated to not only a single interface but on-site multiple interfaces containing various visualizations obtained from the data pipeline. Users are able to switch to the visualizations they require for the desired analyses. As shown in Figure 8, some different floating interfaces incorporating different modalities of visualizations are presented and supported by switching after activating the App. The mobility of the AR interface provides users with ongoing visualizations as they move around the equipment that runs the designated process.

5. Discussion

In the article, we propose a comprehensive and informative data processing and visualizing workflow regarding IPT related fields by concentrating on a specific case–an MWT controlled industrial microwave drying system. We elicit the satisfactory time reversal imaging algorithm, the automated post-imaging segmentation, and the volumetric visualization which characterize the whole process of our case. Furthermore, we present a novel AR framework to support the visualization analysis by incorporating the necessary information used for practical needs. In this part, we narrate the insights gained as well as the existing limitations of our methodology.

5.1. Insights

The first concern of our study is how to present a complete data processing pipeline with informative visualizations, while these modalities are incorporated by the second focus–a preliminary AR framework to facilitate interactive and collaborative onsite data analysis regarding volumetric visualization in IPT systems. As a pioneer study in this related context, our proposed AR system opens the new horizon of deploying this interactive technique into IPT, boosting the complicated data processing and visualizing with an MWT-controlled microwave drying process. Based on our study, we present the following findings:
Interactivity. The most prominent advantage of this system is that the information visualization by the AR interface is completely interactive. By using the AR App after activation, a user can easily get instant access to the visualized data by observing the interface which incorporates the information needed for decision-making. In addition, users are offered the option of focusing on different visualizations from among the different modalities of data we have projected into the virtual interface. Our AR system provides a new perspective of interactive data analysis to benefit IPT related users.
Mobility. Another strength of this system is its mobility. Generally, controlling and supervising an IPT related process requires implementation in specific and precise equipment and scientific laboratories, such as the drying apparatus and the HEPHAISTOS laboratory as mentioned before. Researchers who observe the data and analyze the ongoing process invariably conduct this work on dedicated computers located at different places. However, with this system they are empowered to hold a portable mobile device (a smartphone or tablet) with a running App to closely implement the data analysis beside the precise equipment. The mobility of our system greatly reduces the time cost while improving the working efficiency.
Information richness. The AR interface is used for displaying the significant information needed for boosting the decision-making of the industrial process. In our example, we showcase the visualized data characterizing the specifications of the process. Specifically, we report the first AR application of bringing a complete data processing and visualizing workflow related to complex IPT process which contains the key information used for further analysis. For instance, in our MWT–controlled microwave drying process, a generated MWT image by time-reversal technique is able to show the precise moisture distribution, while the up-to-date volumetric visualization has the capability to reveal the parameters of the drying material and the operating situation of the equipment.
Mutual collaboration. Our system enables in-situ collaboration by providing a number of virtual AR interfaces comprising different information, including all the sophisticated visualizations emerging from the workflow (Figure 8). Under this setting, users from different communities, such as hands-on operators, academic researchers, and engineers, can independently conduct visualization analysis by observing the corresponding visualizations, and interact with each other on common problems as well. This shared mode breaks the gap of different communities working separately but then integrates them into a complete human-in-the-loop working pattern. Through our system, users concentrating on distinct targets originally can jointly interact and discuss at the same location, simultaneously having the access to the same interface containing visualizations.

5.2. Limitations

While we have listed some superiorities of the our proposed solutions, we must also acknowledge there still exist some deficiencies in this preliminary AR framework supporting the visualization of data workflow. Foremost, the dataflow itself does not include all the necessary information related to the MWT-controlled process. For instance, only one modality of volumetric visualization is discussed in our study, whereas involving more formats of data presentations would be more advantageous. Moreover, although we have set a precedent for bringing the AR technique to benefit IPT especially in complex visualization analysis, we have not yet evaluated our system through a systematic user study to gain constructive and critical feedback. Obtaining early-stage findings has not been sufficient for formulating a reliable and robust AR system which should be deemed as a supportive tool in industrial settings. Only those prototypes which are established and improved based on proper usability testing can be the real-time applications at this stage. That is, a comprehensive experiment of user aspects should be mandatory for a constructive human-in-the-loop system, and is not included in this study at current stage.
In addition, our preliminary AR system is embodied as a mobile App on portable smartphones or tablets. This setting can provide the fundamental AR experience for targeted users, but it limits the AR interface to small, bounded screens. Generally, a functional AR application elicits high interactivity and immersivity to users due to its property to put the virtual elements in front of people. The formulation of AR in handheld mobile devices to some extent hinders the essential advantages of AR, which should be tackled by introducing more advanced headsets that can make AR scenes more interactive and immersive.

6. Conclusions and Future Work

In this article, we first propose a complete data processing and visualizing workflow in IPT system by using the MWT-controlled microwave drying case, then we present a preliminary AR system in the same context to support complex visualization analysis. The data pipeline contains an up-to-date time reversal imaging algorithm for reconstructing MWT images, a post-segmentation part which automatically visualizes the desired areas on MWT images, and a straightforward volumetric visualization part. The overall realization of the reported AR framework is conducted by a mobile AR App run on smartphones/tablets, aiding our specific MWT controlled microwave drying process. We successfully prove that this novel system brings new conceptualization in this domain, by offering interactivity and mobility on users’ in-situ data analysis and visualizations. Moreover, we emphasize that our AR system has been improved compared to the previous version by eliciting multiple switchable floating interfaces which incorporate multimodal visualizations of the designated process. Thus, the significant increase of the information richness in this interdisciplinary area is generated. The emergence of this system facilitates mutual collaboration by integrating different communities of users into joint discussions and interactions at the same location. This working mode greatly promotes the synergy pertaining to data analysis and visualization in IPT systems.
In future work, the first priority will be bringing more informative visualizations into the data pipeline, for instance, getting more categories and more formats of volumetric visualizations that would give users a better comprehension of the process output. Furthermore, involving constructive usability and usage testing is another critical concern, for example, designing a systematic user study by engaging adequate people to perform specific tasks in our AR system, collecting feedback, and revising the system correspondingly. In addition, prototyping on advanced AR equipment such as Microsoft HoloLens 2 or Magic Leap headsets will be another concentration, which should break the limit in screen displaying and yield higher interactivity and immersivity for targeted users.

Author Contributions

Conceptualization, Y.Z., A.O., R.Y.; methodology, Y.Z., A.O., R.Y.; design, Y.Z., A.O., R.Y.; formal analysis, Y.Z., A.O., R.Y.; investigation, Y.Z., A.O., R.Y.; writing—original draft preparation, Y.Z., A.O., R.Y.; writing—review and editing, Y.Z., A.O., R.Y., M.F.; visualization, Y.Z., A.O., R.Y.; supervision, M.F.; project administration, M.F.; funding acquisition, M.F. All authors have read and agreed to the published version of the manuscript.

Funding

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 764902, and and in part by the Academy of Finland, the Finnish Center of Excellence of Inverse Modeling and Imaging (project 312344) and Academy of Finland (project 321761).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data presented in this study are available on request from the corresponding authors.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Ismail, I.; Gamio, J.; Bukhari, S.A.; Yang, W. Tomography for multi-phase flow measurement in the oil industry. Flow Meas. Instrum. 2005, 16, 145–155. [Google Scholar] [CrossRef]
  2. Tapp, H.; Peyton, A.; Kemsley, E.; Wilson, R. Chemical engineering applications of electrical process tomography. Sensors Actuators B Chem. 2003, 92, 17–24. [Google Scholar] [CrossRef]
  3. Nolet, G. Seismic Tomography: With Applications in Global Seismology and Exploration Geophysics; Springer Science & Business Media: Berlin, Germany, 2012; Volume 5. [Google Scholar]
  4. Primrose, K. Application of process tomography in nuclear waste processing. In Industrial Tomography; Elsevier: Amsterdam, The Netherlands, 2015; pp. 713–725. [Google Scholar]
  5. Plaskowski, A.; Beck, M.; Thorn, R.; Dyakowski, T. Imaging Industrial Flows: Applications of Electrical Process Tomography; Institute of Physics: Philadelphia, PA, USA, 1995. [Google Scholar]
  6. Zhang, Y.; Yadav, R.; Omrani, A.; Fjeld, M. A Novel Augmented Reality System To Support Volumetric Visualization in Industrial Process Tomography. In Proceedings of the 2021 Conference on Interfaces and Human Computer Interaction, Online, 21–23 July, 2021; pp. 3–9. [Google Scholar]
  7. Zhang, Y.; Nowak, A.; Rao, G.; Romanowski, A.; Fjeld, M. Is Industrial Tomography Ready for Augmented Reality? A Need-finding Study of How Augmented Reality Can Be Adopted by Industrial Tomography Experts. In Proceedings of the 2021 11th IEEE International Conference on Virtual Reality and Visualization (ICVRV), Nanchang, China, 17–20 October, 2021. in press. [Google Scholar]
  8. Omrani, A.; Link, G.; Jelonnek, J. A Multistatic Uniform Diffraction Tomographic Algorithm for Real-Time Moisture Detection. In Proceedings of the 2020 IEEE Asia-Pacific Microwave Conference (APMC), Hong Kong, China, 8–11 December 2020; pp. 437–439. [Google Scholar] [CrossRef]
  9. Yadav, R.; Omrani, A.; Vauhkonen, M.; Link, G.; Lähivaara, T. Microwave Tomography for Moisture Level Estimation Using Bayesian Framework. In Proceedings of the 2021 15th European Conference on Antennas and Propagation (EuCAP), Dusseldorf, Germany, 22–26 March 2021; pp. 1–5. [Google Scholar] [CrossRef]
  10. Zhang, Y.; Fjeld, M. Condition Monitoring for Confined Industrial Process Based on Infrared Images by Using Deep Neural Network and Variants. In Proceedings of the 2020 2nd International Conference on Image, Video and Signal Processing, Singapore, 20–22 March 2020; pp. 99–106. [Google Scholar]
  11. Zhang, Y.; Ma, Y.; Omrani, A.; Yadav, R.; Fjeld, M.; Fratarcangeli, M. Automatic image segmentation for microwave tomography (mwt) from implementation to comparative evaluation. In Proceedings of the 12th International Symposium on Visual Information Communication and Interaction, Shanghai, China, 20–22 September 2019; pp. 1–2. [Google Scholar]
  12. Zhang, Y.; Ma, Y.; Omrani, A.; Yadav, R.; Fjeld, M.; Fratarcangeli, M. Automated microwave tomography (Mwt) image segmentation: State-of-the-art implementation and evaluation. J. WSCG 2020, 2020, 126–136. [Google Scholar]
  13. Zhang, Y.; Fjeld, M.; Said, A.; Fratarcangeli, M. Task-Based Colormap Design Supporting Visual Comprehension in Process Tomography; EuroVis 2020—Short Papers; Kerren, A., Garth, C., Marai, G.E., Eds.; The Eurographics Association: Norrköping, Sweden, 2020. [Google Scholar] [CrossRef]
  14. Zhang, Y.; Fjeld, M. “I Am Told to Be Happy”: An Exploration of Deep Learning in Affective Colormaps in Industrial Tomography. In Proceedings of the 2021 2nd International Conference on Artificial Intelligence and Information Systems (ICAIIS 2021), Chongqing, China, 28–30 May 2021; Association for Computing Machinery: New York, NY, USA, 2021. [Google Scholar] [CrossRef]
  15. Zhang, Y.; Fjeld, M.; Fratarcangeli, M.; Said, A.; Zhao, S. Affective Colormap Design for Accurate Visual Comprehension in Industrial Tomography. Sensors 2021, 21, 4766. [Google Scholar] [CrossRef]
  16. Daily, W.; Ramirez, A.; Binley, A.; LeBrecque, D. Electrical resistance tomography. Lead. Edge 2004, 23, 438–442. [Google Scholar] [CrossRef]
  17. Nowak, A.; Woźniak, M.; Rowińska, Z.; Grudzień, K.; Romanowski, A. Towards in-situ process tomography data processing using augmented reality technology. In Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers, London, UK, 9–13 September 2019; pp. 168–171. [Google Scholar]
  18. Roussy, G.; Bennani, A.; Thiebaut, J. Temperature runaway of microwave irradiated materials. J. Appl. Phys. 1987, 62, 1167–1170. [Google Scholar] [CrossRef]
  19. Sun, Y. Adaptive and Intelligent Temperature Control of Microwave Heating Systems with Multiple Sources. Ph.D. Thesis, KIT Scientific Publishing, Karlsruhe, Germany, 2016. [Google Scholar]
  20. Link, G.; Kayser, T.; Koester, F.; Weiss, R.; Betz, S.; Wiesehoefer, R.; Sames, T.; Boulkertous, N.; Teufl, D.; Zaremba, S.; et al. Faserverbund-Leichtbau mit Automatisierter Mikrowellenprozesstechnik Hoher Energieeffizienz (FLAME): Schlussbericht des BMBF-Verbundprojektes; KIT Scientific Reports; Technical Report; Karlsruher Institut für Technologie: Karlsruhe, Germany, 2015; Volume 7701. [Google Scholar] [CrossRef]
  21. Bykov, Y.V.; Rybakov, K.I.; Semenov, V.E. High-temperature microwave processing of materials. J. Phys. D Appl. Phys. 2001, 34, R55–R75. [Google Scholar] [CrossRef]
  22. Grudzien, K. Visualization system for large-scale silo flow monitoring based on ECT technique. IEEE Sens. J. 2017, 17, 8242–8250. [Google Scholar] [CrossRef]
  23. Nowak, A.; Zhang, Y.; Romanowski, A.; Fjeld, M. Augmented Reality with Industrial Process Tomography: To Support Complex Data Analysis in 3D Space. In Proceedings of the Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers (UbiComp’21), Online, 21–27 September 2021. [Google Scholar]
  24. Azuma, R.T. A survey of augmented reality. Presence Teleoper. Virtual Environ. 1997, 6, 355–385. [Google Scholar] [CrossRef]
  25. Ferrario, A.; Weibel, R.; Feuerriegel, S. ALEEDSA: Augmented Reality for Interactive Machine Learning. In Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 25–30 April 2020; pp. 1–8. [Google Scholar]
  26. Fraga-Lamas, P.; Fernández-Caramés, T.M.; Blanco-Novoa, Ó.; Vilar-Montesinos, M.A. A review on industrial augmented reality systems for the industry 4.0 shipyard. IEEE Access 2018, 6, 13358–13375. [Google Scholar] [CrossRef]
  27. de Souza Cardoso, L.F.; Mariano, F.C.M.Q.; Zorzal, E.R. A survey of industrial augmented reality. Comput. Ind. Eng. 2020, 139, 106159. [Google Scholar] [CrossRef]
  28. Wu, Z.; Wang, H. Microwave Tomography for Industrial Process Imaging: Example Applications and Experimental Results. IEEE Antennas Propag. Mag. 2017, 59, 61–71. [Google Scholar] [CrossRef]
  29. Wu, Z.; McCann, H.; Davis, L.E.; Hu, J.; Fontes, A.; Xie, C.G. Microwave-tomographic system for oil- and gas-multiphase-flow imaging. Meas. Sci. Technol. 2009, 20, 104026. [Google Scholar] [CrossRef]
  30. Sabzevari, F.M.; Winter, R.S.C.; Oloumi, D.; Rambabu, K. A Microwave Sensing and Imaging Method for Multiphase Flow Metering of Crude Oil Pipes. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2020, 13, 1286–1297. [Google Scholar] [CrossRef]
  31. Tobon Vasquez, J.A.; Scapaticci, R.; Turvani, G.; Ricci, M.; Farina, L.; Litman, A.; Casu, M.R.; Crocco, L.; Vipiana, F. Noninvasive Inline Food Inspection via Microwave Imaging Technology: An Application Example in the Food Industry. IEEE Antennas Propag. Mag. 2020, 62, 18–32. [Google Scholar] [CrossRef]
  32. Becker, F.; Schwabig, C.; Krause, J.; Leuchs, S.; Krebs, C.; Gruna, R.; Kuter, A.; Langle, T.; Nuessler, D.; Beyerer, J. From Visual Spectrum to Millimeter Wave: A Broad Spectrum of Solutions for Food Inspection. IEEE Antennas Propag. Mag. 2020, 62, 55–63. [Google Scholar] [CrossRef]
  33. Ricci, M.; Crocco, L.; Vipiana, F. Microwave Imaging Device for In-Line Food Inspection. In Proceedings of the 2020 14th European Conference on Antennas and Propagation (EuCAP), Copenhagen, Denmark, 15–20 March 2020; pp. 1–4. [Google Scholar] [CrossRef]
  34. Och, A.; Hölzl, P.A.; Schuster, S.; Scheiblhofer, S.; Zankl, D.; Pathuri-Bhuvana, V.; Weigel, R. High-Resolution Millimeter-Wave Tomography System for Nondestructive Testing of Low-Permittivity Materials. IEEE Trans. Microw. Theory Tech. 2021, 69, 1105–1113. [Google Scholar] [CrossRef]
  35. Omrani, A.; Yadav, R.; Vauhkonen, M.; Link, G.; Lähivaara, T.; Jelonnek, J. A combined microwave imaging algorithm for localization and moisture level estimation in multilayered media. In Proceedings of the 2021 15th European Conference on Antennas and Propagation (EuCAP), Dusseldorf, Germany, 22–26 March 2021; pp. 1–5. [Google Scholar] [CrossRef]
  36. Yadav, R.; Vauhkonen, M.; Link, G.; Betz, S.; Lähivaara, T. Microwave tomography for estimating moisture content distribution in porous foam using neural networks. In Proceedings of the 2020 14th European Conference on Antennas and Propagation (EuCAP), Copenhagen, Denmark, 15–20 March 2020; pp. 1–5. [Google Scholar] [CrossRef]
  37. Lähivaara, T.; Yadav, R.; Link, G.; Vauhkonen, M. Estimation of Moisture Content Distribution in Porous Foam Using Microwave Tomography with Neural Networks. IEEE Trans. Comput. Imaging 2020, 6, 1351–1361. [Google Scholar] [CrossRef]
  38. Mann, R.; Stanley, S.; Vlaev, D.; Wabo, E.; Primrose, K. Augmented-reality visualization of fluid mixing in stirred chemical reactors using electrical resistance tomography. J. Electron. Imaging 2001, 10, 620–630. [Google Scholar] [CrossRef]
  39. Chew, W.C. Waves and Fields in Inhomogenous Media; IEEE Press: Piscataway, NJ, USA, 1999. [Google Scholar]
  40. Chew, W.; Wang, Y. Reconstruction of two-dimensional permittivity distribution using the distorted Born iterative method. IEEE Trans. Med. Imaging 1990, 9, 218–225. [Google Scholar] [CrossRef] [PubMed]
  41. Tai, C.T. Dyadic Green’s Functions in Electromagnetic Theory; IEEE Press: Piscataway, NJ, USA, 1994. [Google Scholar]
  42. Yavuz, M.E.; Teixeira, F.L. Space–Frequency Ultrawideband Time-Reversal Imaging. IEEE Trans. Geosci. Remote Sens. 2008, 46, 1115–1124. [Google Scholar] [CrossRef]
  43. Janalizadeh, R.C.; Zakeri, B. A Source-Type Best Approximation Method for Imaging Applications. IEEE Antennas Wirel. Propag. Lett. 2016, 15, 1707–1710. [Google Scholar] [CrossRef]
  44. Sadeghi, S.; Mohammadpour-Aghdam, K.; Ren, K.; Faraji-Dana, R.; Burkholder, R.J. A Pole-Extraction Algorithm for Wall Characterization in Through-the-Wall Imaging Systems. IEEE Trans. Antennas Propag. 2019, 67, 7106–7113. [Google Scholar] [CrossRef]
  45. Sadeghi, S.; Mohammadpour-Aghdam, K.; Faraji-Dana, R.; Burkholder, R.J. A Novel Algorithm for Wall Characterization in through the wall Imaging based on Spectral Analysis. In Proceedings of the 2018 18th International Symposium on Antenna Technology and Applied Electromagnetics (ANTEM), Waterloo, ON, Canada, 19–22 August 2018; pp. 1–2. [Google Scholar] [CrossRef]
  46. Soldatov, S.; Kayser, T.; Link, G.; Seitz, T.; Layer, S.; Jelonnek, J. Microwave cavity perturbation technique for high-temperature dielectric measurements. In Proceedings of the 2013 IEEE MTT-S International Microwave Symposium Digest (MTT), Seattle, WA, USA, 2–7 June 2013; pp. 1–4. [Google Scholar] [CrossRef]
Figure 1. A side view of the HEPHAISTOS microwave oven system. Main modules of the oven are represented by number tags 1, 2, 3, 4, 5. Tag 1 and Tag 5 represent the entrance of the wet foam and exit doors for the dry foam, respectively on the conveyor belt. Tags 2, 3, 4 indicate the three modular heating system with inbuilt hexagonal cavity with high power microwave heating sources and control system block. The MWT system will be integrated near the entrance door. The system is located at KIT, Germany.
Figure 1. A side view of the HEPHAISTOS microwave oven system. Main modules of the oven are represented by number tags 1, 2, 3, 4, 5. Tag 1 and Tag 5 represent the entrance of the wet foam and exit doors for the dry foam, respectively on the conveyor belt. Tags 2, 3, 4 indicate the three modular heating system with inbuilt hexagonal cavity with high power microwave heating sources and control system block. The MWT system will be integrated near the entrance door. The system is located at KIT, Germany.
Sensors 21 06515 g001
Figure 2. 2D schematic of the MWT setup with open-ended waveguide antennas. In the present study, layer 0 and layer 2 represent free-space (air), and layer 1 represents the polymer foam. The region represented by the green color indicates the moisture wet-spot inside the foam.
Figure 2. 2D schematic of the MWT setup with open-ended waveguide antennas. In the present study, layer 0 and layer 2 represent free-space (air), and layer 1 represents the polymer foam. The region represented by the green color indicates the moisture wet-spot inside the foam.
Sensors 21 06515 g002
Figure 3. Reconstruction of one wet-spot moisture case with TR-DORT where the true location is marked by black dash lines. The moisture area is represented by the yellow color and the dark blue color represents the dry part.
Figure 3. Reconstruction of one wet-spot moisture case with TR-DORT where the true location is marked by black dash lines. The moisture area is represented by the yellow color and the dark blue color represents the dry part.
Sensors 21 06515 g003
Figure 4. The order of the timestamp-marked TR images: top left: t 1 ; top middle: t 2 ; top right: t 3 ; middle left: t 4 ; middle middle: t 5 ; middle right: t 6 ; bottom left: t 7 ; bottom left: t 8 ; bottom right: t 9 .
Figure 4. The order of the timestamp-marked TR images: top left: t 1 ; top middle: t 2 ; top right: t 3 ; middle left: t 4 ; middle middle: t 5 ; middle right: t 6 ; bottom left: t 7 ; bottom left: t 8 ; bottom right: t 9 .
Sensors 21 06515 g004
Figure 5. The nine timestamp-marked TR MWT images after reconstruction from the process with their corresponding segmented results. The order of the timestamp-marked images are same as given in Figure 4. In the segmented results, the light grey areas are the visualized parts, which represent high moisture ares of the polymer foam used for drying.
Figure 5. The nine timestamp-marked TR MWT images after reconstruction from the process with their corresponding segmented results. The order of the timestamp-marked images are same as given in Figure 4. In the segmented results, the light grey areas are the visualized parts, which represent high moisture ares of the polymer foam used for drying.
Sensors 21 06515 g005
Figure 6. Volumetric visualization of the moisture distribution in the yz plane as estimated by the MWT at different time slots. The term y s and z s denotes the pixels in the y and z directions, respectively. The dominant wet-spots are represented in yellow and the dry part is represented as blue.
Figure 6. Volumetric visualization of the moisture distribution in the yz plane as estimated by the MWT at different time slots. The term y s and z s denotes the pixels in the y and z directions, respectively. The dominant wet-spots are represented in yellow and the dry part is represented as blue.
Sensors 21 06515 g006
Figure 7. The conceptualization framework of the proposed AR system. MWT-controlled industrial microwave drying process block: This covers the whole running procedure of the process, providing data and elements used for analysis. Users block: This represents the people involved in this context. For instance, the operators who control and run the process and the researchers who observe and analyze the process are the main composition of this part. AR App block: This entails the AR implementation, which was a mobile AR App developed on iOS/Android platform at the initial stage. The App allows different communities of people to get access to the key information from the proposed data processing, visualize workflow interactively, and conduct the onsite analysis collaboratively [6].
Figure 7. The conceptualization framework of the proposed AR system. MWT-controlled industrial microwave drying process block: This covers the whole running procedure of the process, providing data and elements used for analysis. Users block: This represents the people involved in this context. For instance, the operators who control and run the process and the researchers who observe and analyze the process are the main composition of this part. AR App block: This entails the AR implementation, which was a mobile AR App developed on iOS/Android platform at the initial stage. The App allows different communities of people to get access to the key information from the proposed data processing, visualize workflow interactively, and conduct the onsite analysis collaboratively [6].
Sensors 21 06515 g007
Figure 8. The realization of the proposed AR system. Users hold the handheld smartphones/tablets to run the AR App alongside the industrial setting. After the App activation, users can then observe the visualizations displayed on the AR interfaces. At present, the App is supported via iOS/Android mobile devices. Users are empowered to switch to different visualizations regarding different phases of the process. The visualizations shown on the interfaces are multimodal, including the infrared image showing the condition of the process and the 3D/2D reconstruction of the MWT images (the upper left small figure on the floating interface in the upper right subfigure); the reconstructed MWT images with their segmented results to show the high moisture areas (the middle right subfigure); the volumetric visualization (the bottom right subfigure) [6].
Figure 8. The realization of the proposed AR system. Users hold the handheld smartphones/tablets to run the AR App alongside the industrial setting. After the App activation, users can then observe the visualizations displayed on the AR interfaces. At present, the App is supported via iOS/Android mobile devices. Users are empowered to switch to different visualizations regarding different phases of the process. The visualizations shown on the interfaces are multimodal, including the infrared image showing the condition of the process and the 3D/2D reconstruction of the MWT images (the upper left small figure on the floating interface in the upper right subfigure); the reconstructed MWT images with their segmented results to show the high moisture areas (the middle right subfigure); the volumetric visualization (the bottom right subfigure) [6].
Sensors 21 06515 g008
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Zhang, Y.; Omrani, A.; Yadav, R.; Fjeld, M. Supporting Visualization Analysis in Industrial Process Tomography by Using Augmented Reality—A Case Study of an Industrial Microwave Drying System. Sensors 2021, 21, 6515. https://doi.org/10.3390/s21196515

AMA Style

Zhang Y, Omrani A, Yadav R, Fjeld M. Supporting Visualization Analysis in Industrial Process Tomography by Using Augmented Reality—A Case Study of an Industrial Microwave Drying System. Sensors. 2021; 21(19):6515. https://doi.org/10.3390/s21196515

Chicago/Turabian Style

Zhang, Yuchong, Adel Omrani, Rahul Yadav, and Morten Fjeld. 2021. "Supporting Visualization Analysis in Industrial Process Tomography by Using Augmented Reality—A Case Study of an Industrial Microwave Drying System" Sensors 21, no. 19: 6515. https://doi.org/10.3390/s21196515

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop