Next Article in Journal
Conceptual Process Design, Energy and Economic Analysis of Solid Waste to Hydrocarbon Fuels via Thermochemical Processes
Next Article in Special Issue
Moving Multiscale Modelling to the Edge: Benchmarking and Load Optimization for Cellular Automata on Low Power Microcomputers
Previous Article in Journal
Dynamic Model for Biomass and Proteins Production by Three Bacillus Thuringiensis ssp Kurstaki Strains
Previous Article in Special Issue
A Framework in Calibration Process for Line Structured Light System Using Image Analysis
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Approaches for Motion Control Interface and Tele-Operated Overhead Crane Handling Tasks

1
Graduate School of Comprehensive Human Sciences, University of Tsukuba, 1-1-1 Tennodai, Tsukuba-shi 305-3577, Japan
2
School of Design, Hunan University, Changsha 410082, China
3
Faculty of Art & Design, University of Tsukuba, 1-1-1 Tennodai, Tsukuba-shi 305-3577, Japan
*
Author to whom correspondence should be addressed.
Processes 2021, 9(12), 2148; https://doi.org/10.3390/pr9122148
Submission received: 27 October 2021 / Revised: 24 November 2021 / Accepted: 24 November 2021 / Published: 28 November 2021
(This article belongs to the Special Issue Advances in Digital Design and Manufacturing)

Abstract

:
This study aimed to identify the effects of different approaches to a motion control interface (MCI) in tele-operated crane handling tasks. In this study, due to the difficulty of applying the actual equipment to the experiment, we presented a prototype system of a tele-operated overhead handling (TOH) crane. Specifically, we investigated participants’ task performance including the accuracy of task completion during unloading, heart rate variation, workload, and the relationships between these factors when four motion control approaches were used: pointing (P), keyboard (K), orientation (O), hand-free gesture (HG). Experiments were conducted with two groups of participants: 21 university students and 11 crane operators used each of the four control methods. A task condition for handling iron blocks was tested. The efficacy of each motion control approach for task performance was evaluated by a within-subject experiment with a novice group. The expert group was used for comparing the task performance and satisfaction in the prototype system with the novices, evaluating whether the prototype system was reproducible for a real setting in the construction site. The results showed that the task completion time, the weight of physical demand, and the overall scores for workload were significantly impacted by the type of motion control: when HG was used, the task completion time increased. Particularly, using HG had the potential to increase the overall workload score, while physical laboriousness was also potentially increased by HG. Conversely, unloading accuracy, heart rate, and mental demand were not affected by motion control approaches. Generally, the expert group spent more time completing the tasks, but they performed better unloading accuracy than the novices in all methods. Ninety-one percent of the experts gave positive feedback on the reproducibility of the prototype system.

1. Introduction

To reduce the hazardous operation of having human operators in the factory or on the construction site, tele-operated equipment was frequently studied in recent decades. This equipment assists human operators in remotely finishing heavy engineering tasks in hostile environments [1], such as construction-use tower cranes [2], mining machines [3], excavators [4,5], and stacking cranes [6,7,8]. The electric overhead travelling (EOT) cranes are the most widely used mechanical aids for handling large and heavy objects over a limited distance on the shopfloor [9]. Each EOT crane has three-axial movements, following the X, Y, and Z-axis [10]. The EOT crane requires human operators to manually control each movement in the mid-air cabin. Each EOT crane has three main controls: for the Y-axial longitudinal travelling (LT) of the crane itself, the X-axial cross-traveling (CT) of the trolly, and the Z-axial hoisting control (HC) of the spreader [9,10]. A trolly-mounted electromagnet spreader (ES) is often used to attract ferromagnetic materials. Although improvements in the mechanical performance and productivity of cranes were frequently developed in previous studies, for example, cable swing prevention [11], hoist stabilization [12], body stabilization [13], the safety issue of human operators involved in hazardous operations in dusty, toxic, and hostile environments remains. Thus, the development of a teleoperation system of EOT cranes, namely the tele-operated overhead handling (TOH) crane system, is necessary.
The improvements in the experience and task performance regarding human factors, were addressed in previous research [14,15,16]. The concept of a human-centered design has become an essential aspect that must be addressed in teleoperation [17,18,19]. It is valuable for researchers and designers to consider these aspects in the manufacturing and service industries [20,21]. The issue of visualization was widely addressed in previous studies. Human operators have a limited perception of the operation process and environment through the visual display interface (VDI) due to the visual limitations [2]. In this regard, previous research suggested that increasing the vision of human operators assisted them and prevented blind lifting [22]. Additionally, some studies focused on providing available visual information such as telepresence [23], or the augmentation of human perception [24,25]. As for tele-operated tower crane operations, Chi et al. proposed an effective VDI to improve task performance and relieve workload by using an augmented path planning process [26,27]. As for the approaches to the motion control interface (MCI) specific to tele-operated crane operation, Abdulla et al. revealed the physical laboriousness of human operators when using a conventional joysticks interface [28]. Since the approaches to MCI in teleoperation are not restricted by the cabin setting, the consistency of the conventional joystick is controversial. However, the evaluation and comparison of different approaches regarding the specific TOH crane operation, were rarely discussed in previous studies. To offer valuable references to MCI development in the future, the investigation of the effects of task performance and experience on the broader possibilities of applied approaches, is the main motivation of this study.

Related Works

Four widely applied approaches, namely input modalities [29] for MCI in teleoperations, were studied in previous research. Commonly used approaches are pointing (P), keyboard (K), orientation (O), and hand-free gesture (HG). The first approach, P, described in [30,31], was used for completing teleoperation by a point-and-click on the interface menu of displays. A second approach using K, developed in [32], remotely activated the equipment’s movements through the comprehensive keys. P and K are considered classical control methods for human-to-equipment interaction, which were originally developed for teleoperation. A third approach, O, described in [33,34], normally compared to the joystick, is the most widely used approach for controlling the general crane movements by orientation [35]. The fourth approach of HG, studied in [36], presented the new possibility to control the end-effectors of mechanical equipment through body movements. Instead of using complex hand actions to remotely operate equipment, HG provides a natural and intuitive method of manipulation through hand gestures. Based on the previous research, we developed new control methods for P, K, O, and HG, specified with the TOH crane execution in this study.
In related studies, Chi et al. [2] developed a new VDI assisting human operators in precisely controlling crane movements in teleoperation. A 6 DoF robotic arm was modified for simulation [27]. The effectiveness of the developed VDI on handling tasks was evaluated by task performance and workload. Task performances indicated by the completion time [28] and workload indicated by the six weights of the dimension of NASA-TLX between a novice group and an expert group, were compared. The reproducibility of the prototype was subjectively assessed by the experts through a standard 7-point Likert questionnaire. However, other factors related to the task performance such as unloading accuracy were not discussed; human-related indicators such as heart rate were not considered. Doisy et al. [37] investigated the effects of different approaches for camera and motion control on tele-operated robot maze travelling. The iRobot Create program was used for the experiments. The comparison of three techniques for robot control was conducted with a novice group. The efficacy of each approach was evaluated by the travelling performance, defined as the parameters of time spent and number of collisions. The workload of participants was assessed by the weight of mental demand and physical demand through a raw NASA-TLX questionnaire. The heart rate variation was used as a measure. Mower et al. compared joystick control modes and investigated the most intuitive control spaces for tele-operated constrained tasks with a 7DoF KUKA LWR, by conducting a 21-participant experiment [38]. However, the subjective responses of participants were not collected and evaluated; the comparison of proficiency was not conducted; the weighted score of workload indicating overall workload, which was multiplied by the scale score for each dimension and then divided by 15 [39], was not applied. Both of these studies used a teleoperation room.
Compared with the related studies, this study mainly aimed to investigate the effects of approaches to MCI on the TOH crane handling task. Specifically, the use of P, K, O, and HG, were evaluated and compared by the task performance and workload.
Based on the methodologies introduced by the previous studies, we modified a classical Cartesian 3D printer named MK3S+ [40,41] as a prototype to reproduce the crane mechanism. We conducted the experiments with a novice group and an expert group in a separated teleoperation room. The task performances of the experts were compared with the novices’. In this study, the TOH crane handling tasks were evaluated through the following factors: task performance, which comprised completion time and unloading accuracy; heart rate variation, which could be used as an indication of the phycological state and workload level of the participants [42,43,44]; and the weights of physical demand, mental demand (0–5), and the weighted score of workload (0–100), which were assessed and collected through a computerized NASA-TLX test [45]. Additionally, the reproducibility of the prototype was evaluated by the experts through a 7-point Likert questionnaire.

2. Materials and Methods

2.1. Prototype TOH Crane System

It is difficult to directly operate a real crane for experimental use due to the production requirements and safety protocol in factories. An MK3S+ [40,41], which has a highly matched mechanism to the original EOT crane, was modified [41,46], reproducing the three-axial mechanical movements and the ES attraction, which is shown in Figure 1. The axial movements were translated accordingly. The TOH crane prototype was essentially driven by a programmed Raspberry Pi 4B [47]: The operating system on the PC was Microsoft Windows 10 professional 64 bit. OctoPrint [48], an open-source API, was used to send the G-code to control the equipment, retrieving data such as the coordinate of the trolly, and two levels of speed of the movements. The original SuperPINDA distance sensor sensed the spatial position during the handling process. A PC-based controller software was designed. Intuitive menus using iconic indications were presented in software UI. The screen capture of the controller software is shown in Figure 2. There were two main functions of the software: control and record. As for the function of control, operators were able to control the crane movements directly. Movements, speed, and ES activation menus were shown on the right side of the interface. Researchers on another monitoring PC were in charge of the recording function, which proposedly enabled researchers to record performance-related parameters. In addition, when an error occurred, such as a severe collision, researchers could remotely stop the handling task on the monitoring PC.
There were four views used for aiding operators in precepting environment, which are shown in Figure 2 and Figure 3: (a) global view, which provided general environment information to the operators, including the equipment, shopfloor, and objects; (b) cabin view, reproducing the observing position in the conventional EOT crane’s cabin; (c) bird view, which provided general positioning information at the top; and (d) top view, which moved along with trolly. Detailed positioning information between objects and the shopfloor, was provided by the top view. Four views were displayed in the main area. Real-time trolly positions were shown next to the video streams.
The finished prototype is shown in Figure 3. The system set-up diagram is shown in Figure 4. The human operators controlled the EOT crane prototype via MCI, while the environment information was displayed through VDI in a teleoperation room (TR). The lifting and handling field (LHF) was where the actual crane operation and environment sensing was carried out.

2.2. Approaches for Motion Control Interface (MCI)

In this study, four commercially available input devices were developed and used as approaches to MCI following P, K, O, and HG, which are shown in Figure 5. They were connected to the PC via USB 2.0.
Each modality indicated a certain trial of task completion. In each trial, one of the most widely acclaimed devices was selected from Microsoft and Logitech.
Table 1 shows the information and main control methods of input devices. The ergonomic-friendly design of the Microsoft Ergonomic series and Logitech G X56 was suggested [49]. Leap motion was the most widely used input device for hands-free interaction in previous research [50]. Devices of O and HG were programmed to adapt to the software: O was programmed by Logitech G X56 H.O.T.A.S. [51], HG was programmed by using Leap motion developer Kit [52]. In trial P, participants could simply point-and-click interactive buttons on the interface menu to control every crane movement. In trial K, the participants used “W, A, S, D” keys to control the CT and the LT, using the upward and downward key to control the HC, while the ES was activated by pressing the “space” key. The participants controlled the crane movements at full speed by simultaneously pressing the “Shift” key. The CT and the LT were intuitively translated into the joystick’s orientation in the trial O. The HC was controlled by the “up and down” orientation of the POV, while the ES was activated by the C stick. The participants could control the speed of crane movements by fully orienting. Additionally, the cursor could be hidden by “pinky”. In the trial HG, the CT, LT, and HC were controlled by the “right fist movement” while the activation of ES was controlled by the “left fist releasing” gesture. The speed of the crane movement followed the speed of the “right fist movement”. Specifically, there were no fixed positions to place input devices. The placement was based on the participants’ preferences and comfort.

2.3. Tele-Operated Handling Tasks

The TR was performed in a semi-anechoic room at the University of Tsukuba, which was located 15 m from the LHF. Participants used four motion control methods, remotely operating the handling tasks through the MCI in the TR. The LHF was located in a standard lab environment where mechanical interference and another distractions were not present. The participants and the LHF were in two separate rooms. In this case, the participants had no chance to directly see the operations other than through the video feeds of VDI during the tasks. Figure 6 shows the experiment set-up.
The tele-operated handling task is shown in Figure 6. The goal of the task was to lift five iron blocks from the shopfloor using the ES and remotely handle them to the loading area. Since the dimension of the loading area was limited, participants were supposed to stack blocks in at least two layers, similar to the real unloading process. The boundary of the loading area was highlighted in white, which was used for judging whether the block was released outside the boundary. The operation was judged as a failed unloading when the iron block was released out of or on the boundary. The participants could slightly adjust the position of the iron blocks in the loading area. However, the participants could not make adjustments if their operation resulted in the collapse of the stacked blocks. The interspace of the holder was covered by blinds, in this case, the participants were not distracted by the lab environment.
A side-to-side handling task condition where iron blocks were handled from the fixed position to the loading area, which is one of the most typical tasks in the real factory, was tested with four trials.
The experiment set-up was performed in a LAN environment with the peak download speed of 432.47 Mbps, while the peak network latency was 3 ms. This meant that the participants could barely sense the unsynchronized movements.

2.4. Experiment

A within-subject experiment was conducted with a novice group comprised of 21 university students (Mage = 24.3 ± 2.7 years old), who were familiar with the four types of motion control: P, K, O, and HG. An expert (compared) group comprised 11 professional EOT crane operators (Mage = 39.5 ± 4.5 years old), who had over five years of experience in crane handling operation, and was used for comparing the task performance and satisfaction with the prototype system with those of the novices. In addition, subjective responses from the expert group were available as evidence, and could be used for evaluating whether the prototype TOH crane system was reproducible compared with real applications, judging whether functions of the prototype system met the needs in the field. Participants received compensation of JPY 1000 (about USD 10), and they were told prior to the experiment that they could potentially win a bonus of JPY 2000 (about USD 20) depending on their performance.
The procedure of the experiment was divided into the following steps (Figure 7). First, each participant filled in a questionnaire of personal information, and was orally introduced to the prototype TOH crane system by researchers. The procedure of the experiment was assigned to each participant through a document-based introduction. Then, each participant watched a 5 min peaceful video unrelated to the experiment to relax. These five minutes of relaxation were used for collecting the resting heartrate of participants (baseline). After that, in order to help participants get familiar with the teleoperation, researchers gave a more specific oral introduction regarding the control methods of each approach, and 15 min of practice time was allocated to each participant. Next, each participant individually started tele-operated handling tasks by using each control method. Following this, the participants completed the NASA-TLX test [45]. The experts were invited to complete a 7-point Likert questionnaire for subjective feedback on the reproducibility of the prototype when the tasks were finished. Those participants who missed some of the tasks were required to finish the rest. The experiment was finished when each step had been checked for completeness. Considering the COVID-19 protocol at the university, the health condition of all participants was checked, including body temperature measurement and symptom assessment. Additionally, all participants were required to wear a mask during the entire process. Moreover, all apparatus were strictly disinfected before they were used.

2.5. Performance and Workload Measures

The performance and workload of the participants during the handling tasks were assessed by four measures. First, the completion time was defined as the time spent on completing a trial for each participant, in seconds. Second, unloading accuracy, which was the number of times when iron blocks were accurately unloaded in the loading area, was measured. Those blocks, which were released in the loading area, were not judged as accurate unloading. Third, the heart rate of the participants in the rest state (baseline), and during each trial, was calculated and applied as a measure by using the XOSS Heart rate armband. Measures were based on the heart rate variation in percentage between baseline and each trial, similar to the previous study [37]. Last, the weight (0–5) and the weighted score (0–100) of the workload of the participants, were assessed and collected after each trial through a computerized NASA-TLX test [45]. Particularly, two dimensions of NASA-TLX were extracted as measures: the weight of the physical and mental demand of the participants, which were directly related to their physical laboriousness, and physiological circumstances.

2.6. After-Tasks Survey

After the tasks, to evaluate whether the prototype system was reproducible to the real setting in the field, a 7-point Likert questionnaire with the question of “Generally, I think this prototype system is reproducible to the reality”, was assigned to the experts.

2.7. Data Analysis and Data Visualization

Licensed SPSS 28.0 was used for data input and data analysis. The level of significance was set at p < 0.05. Box plots were used for visualizing the distribution of data, similar to the visualization in previous research [2].

3. Results

3.1. Task Completion Time

Figure 8 shows the time that the novice group and the expert group spent on completing each trial, in seconds. We found that the average completion time using HG was (Mnovice = 1006.67 ± 365.15); where (Mexpert = 1374.91 ± 377.18) spent the most time using HG, while, on average, the least time for both groups was spent on trial O (Mnovice = 553.76 ± 238.94); (Mexpert = 798.00 ± 135.06). Notably, the average task completion time of the expert group was significantly higher than the novice group in each trial. A one-way repeated measures ANOVA was conducted with the novice group to compare the effects of approaches to MCI on task completion time in P (M = 635.48 ± 169.64), K (M = 634.24 ± 198.29), O, and HG trials. There was a significant effect of the approaches, F(3,18) = 9.588, p < 0.001. Six paired-samples t-tests were used to make post hoc comparisons between trials. There were significant differences between P and HG; t(20) = −4.456, p < 0.001, K and HG; t(20) = −5.246, p < 0.001, and O and HG; t(20) = −5.482, p < 0.001. Contrarily, there were no statistical differences in the scores for the P and K, P and O, and K and O. The descriptive statistics of completion time are shown in Table 2. The results of post hoc comparisons are shown in Table 3.

3.2. Accuracy

The accuracy that the novice group and the expert group performed in each trial is shown in Figure 9. The best accuracy was performed in trial O (Mnovice = 3.81 ± 1.21); (Mexpert = 5.00 ± 0.00), while the worst was found in trial HG (Mnovice = 3.14 ± 1.32); (Mexpert = 3.64 ± 1.43). Moreover, we found that, on average, the expert group had a better accuracy than the novice group in each trial. Specifically, all participants from the expert group achieved five out of five accurate unloadings using O. The descriptive statistics are shown in Table 2. However, no significant effect of the approaches on unloading accuracy was found in a Friedman’s test, χ2 (3) = 4.170, p = 0.244.

3.3. Heart Rate Variation

Figure 10 shows the heart rate variation of the novice group, in percentage. The average heart rate variation using P (M = 6.79 ± 8.21) was the smallest, while the largest variation was found in trial HG (M = 9.17 ± 5.97). The descriptive statistics are presented in Table 2. The effect of approaches to MCI on heart rate variation in P, K (M = 8.16 ± 7.13), O (M = 6.92 ± 5.41) and HG trials, was compared by a one-way repeated measures ANOVA. However, the results showed that there was no significant effect of the approaches on heart rate variation, F(3,18) = 1.590, p = 0.227.

3.4. Weight of Workload

Figure 11 shows the weight of physical demand and mental demand of the novice group in each trial. The average physical demand in trial HG (M = 3.71 ± 1.42) required the most, while trial K (M = 1.86 ± 1.65) required the least physical labor. Table 2 shows the descriptive statistics. A Friedman’s test was conducted to compare the effect of approaches to MCI on the weight of physical demand in P (M = 2.10 ± 1.51), K, O (M = 2.10 ± 1.67) and HG trials. There was a significant effect of the approaches, χ2 (3) = 24.225, p = 0.827, p < 0.001. Post hoc tests were carried out between trials using the Wilcoxon test. Significant differences were found between P and HG (z = −3.590, p < 0.001), K and HG (z = −3.405, p < 0.001), and O and HG (z = −3.375, p < 0.001) after Bonferroni adjustment. However, no significant statistical differences were found in the scores for the P and K, P and O, and K and O. Table 3 shows the results of the post hoc tests.
The average metal demand in P (M = 2.24 ± 1.45), K(M = 2.29 ± 1.27), O (M = 2.33 ± 1.43), and HG (M = 2.05 ± 1.53) trials showed similar weights. The descriptive statistics are presented in Table 2. A comparison of the effect of approaches on the weight of mental demand in P, K, O, and HG trials, was also made using a Friedman’s test. However, no significant effect of the approaches was found, χ2 (3) = 0.892, p = 0.827.

3.5. Weighted Score of Workload

The weighted score of the workload of the novice group in each trial is shown in Figure 12. The lowest average weighted score was in trial P (M = 47.09 ± 10.78), while the average weighted score of trial HG (M = 72.73 ± 11.32) was the highest. Notably, the average weighted scores of P, K (M = 51.86 ± 9.34), and O (M = 49.81 ± 13.22) trials were close. The descriptive statistics are presented in Table 2. The effects of the approaches to the weighted score of workload in P, K, O, and HG trials, were compared by a one-way repeated measures ANOVA. A significant effect of the approaches was found, F(3,18) = 33.064, p < 0.001. Post hoc comparisons between trials using six paired-sample t-tests, were conducted. Significant differences were found between P and HG; t(20) = −8.526, p < 0.001, K and HG; t(20) = −9.565, p < 0.001, and O and HG; t(20) = −6.803, p < 0.001. Conversely, there were no statistical differences in the scores for the P and K, P and O, and K and O. Table 3 shows the results of post hoc comparisons.

3.6. Reproducibility

According to the response to the question: “Overall, I think this system is reproducible to the reality”, 91% of the experts gave positive feedback on the reproducibility of the prototype system (46% strongly agree; 36% agree; 9% slightly agree; 9% slightly disagree). The responses above “neither agree nor degree” were defined as positive, while the responses below “neither agree nor degree” were defined as negative.

4. Discussion

In this study, the evaluation and comparison of approaches to MCI based on a designed prototype TOH crane system, were carried out. In addition, the reproducibility of the prototype system, was surveyed. The details of the findings were discussed in the following sections.

4.1. Approaches for Motion Control Interface (MCI)

As for approaches to MCI in TOH crane operations, this study suggested that the task completion time, physical demand, and the overall score of workload were significantly impacted by the approaches. Specifically, the time spent on completing the tasks increased when HG was used, while the physical laboriousness and the overall score of the task workload rose in trial HG. Notably, no significances were found in mechanical and conventional approaches: P, K, and O. Following this, many researchers tried to offer the possibility of achieving intuitive and cableless human–machine interactions through new approaches that were able to extract human motion in controlling machine movements, such as hand gestures [36]. However, in specific TOH crane handling tasks, although the HG had advantages in the timesaving teaches pendant and light panel, the HG directly resulted in the increase in the task completion time and workload, especially the physical fatigue is potentially affected by the mid-air hand interaction. The requirements of crane handling operation are minimum operational errors. In this case, human operators are normally attentive to elaborate operations, which is suggested by the lack of effects of the approaches on the unloading accuracy, heart rate variation, and the weight of mental demand. In a TOH crane handling task, the human operators had limited perception of the environment. In this regard, the sense of control and scale cannot be appropriately translated between comprehensive human motion and accurate machine execution using the HG, which might lead to hesitation and repetitive adjustments in teleoperations.

4.2. Comparison of Task Performance between Groups

The task completion time and the unloading accuracy between the novice group and expert group were compared in this study. According to the results in Figure 8 and Figure 9, the expert group spent more time completing each trial than the novice group. The average discrepancies in the completion times between the experts and the novices were 497.25 s in trial P; 404.31 s in trial K; 244.24 s in trial O; and 368.24 s in trial HG. Additionally, even though the experts spent, on average, 798 s on trial O, O was still the only trial where the expert group spent less than 1000 s to complete. Contrarily, the average accuracy performed by the expert group was better than the novice group in each trial. There was a significant discrepancy; on average, 1.19 times of the accurate unloading in trial O. Notably, all the participants from the expert group performed five out of five successful unloadings in trial O.
Considering all participants had enough instructions and practice for each trial, the results of task performance suggested that the experts tended to perform with the best accuracy, rather than minimizing the completion time. In addition, since O has become the most widely used approach to control the conventional EOT crane movements, the experts normally had more experience in control crane execution when using O. Regardless of the phenomenal accuracy performed by the experts in trial O, the subjective pre-judgement from the expertise in using O, might discourage the experts from interacting with the other approaches. Conversely, the performance of the novices might provide not only objective feedback on each approach for MCI, but also valuable evidence in evaluating whether the MCI is novice-friendly.

4.3. Recommendation on MCI Development

The results of this study might provide the following recommendations for researchers and designers in developing the MCI for the TOH crane system in the future. First, the application of HG should be avoided. Compared with other approaches, the use of HG might worsen the task performance, resulting in the increase in physical laboriousness and overall workload, which might lead to not only the reduction in productivity, but also the negative experience of both the experts and the beginners during the operation. Second, to develop an MCI appropriate for both the experts and the beginners, the application of O might be prioritized, while P and K could be integrated accordingly. In this case, the demands of operation preferences, task performance, and experience could be met. Lastly, compared to evaluating the MCI with the real crane in the factory, it is possible to conduct the usability test with this type of prototype system in a more flexible and economic technique.

5. Conclusions

This study evaluated and compared four approaches to MCI in TOH crane handling tasks: P, K, O, and HG. Participants’ task performance, heart rate variation, workload, and the relationships between these factors were statistically investigated in a novice group. An expert group was used to compare task performance with the novices. The reproducibility of the prototype system was evaluated by the expert group.
This study investigated task performance by calculating the completion time and the unloading accuracy in each approach. The task completion time was significantly impacted by the approaches. Specifically, the completion time increased when HG was used. Conversely, the unloading accuracy was not affected by the approaches. In this study, heart rate variation as a percentage was applied as a measure. The workload after each trial was recorded by using the NASA-TLX test. The results showed that the weight of physical demand and the overall score of workload were significantly affected by the approaches. When HG was used, the physical laboriousness and the overall workload increased. The comparison of the completion time and the accuracy between the novice group and the expert group was carried out. The experts had a greater accuracy, although it took more time to complete the tasks. In this study, 91% of the experts gave positive feedback on the reproducibility of the prototype system.
There are four main limitations to this study. First, there is the network latency. The unsynchronized movements had a significant effect on teleoperation [53]. However, this study focused on the effects of approaches to MCI on the TOH crane handling tasks. In order to control the condition of network latency, this study was performed in a LAN environment using the wired connection. In the future, we need to increase the conditions of latency regarding constant latency [54] and time-varying latency [55]. Second, the prototype was not complex enough. Safety concerns such as the prevention of collisions between existing facilities and workers on the shopfloor should be integrated into future tests. The study should also be incorporated with practical situations for evaluation in the future. Third, it was found that the weight of physical demand increased when HG was used. However, the physical loading was not clearly identified and distinguished in this study. Other potential indicators such as EMGs [56] can be used for assessing physical laboriousness. Lastly, the number of trials and participants was still insufficient. We should increase the number of trials and participants in the future.

Author Contributions

All authors contributed to the work of this article. Z.Y. designed the experiment and tested the methods with the experiment. Z.Y. analyzed the experiment results. Z.Y. wrote the original draft of the Introduction, Materials and Methods, Results, Discussion, and Conclusion. J.L. provided technical support in the section of Materials and Methods and assisted the experiment that was conducted with the expert group. H.Z. reviewed and edited the original draft of this article. E.O. provided technical supports in the section of Materials and Methods and gave advice on the design of the experiment. E.O. reviewed and edited the original draft of this article. S.-H.L. supervised the whole process of this study and the writing of this article. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by JST SPRING, Grant Number JPMJSP2124.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data are available from the corresponding author upon reasonable request.

Acknowledgments

This work was supported by JST SPRING, Grant Number JPMJSP2124. We would like to thank Shengxian Hu, Ye Tian, Ruotian Zhang for data collection and all participants for helping complete this study. We would like to thank TQCC Co., Ltd. of China for providing technical supports.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Alonso, D.; Sánchez, P.; Ortiz, F.; Pastor, J.; Álvarez, B.; Iborra, A. Experiences Developing Safe and Fault-Tolerant Tele-Operated Service Robots. A Case Study in Shipyards. Serv. Robot. Appl. 2008, 2008, 159–182. [Google Scholar] [CrossRef]
  2. Chi, H.; Kim, J.; Chen, Y.; Kang, S.; Song, J.; Hsieh, S. Development of user interface for tele-operated cranes. Adv. Eng. Inform. 2012, 26, 641–652. [Google Scholar] [CrossRef]
  3. Tiwari, R.; Knowles, J.; Danko, G. Bucket trajectory classification of mining excavators. Autom. Constr. 2013, 31, 128–139. [Google Scholar] [CrossRef]
  4. Kim, D.; Oh, K.W.; Hong, D.; Kim, Y.K.; Hong, S.-H. Motion control of excavator with tele-operated system. In Proceedings of the 26th International Symposium on Automation and Robotics in Construction (ISARC), Austin, TX, USA, 24–27 June 2009; pp. 341–347. [Google Scholar] [CrossRef] [Green Version]
  5. Kim, D.; Kim, J.; Lee, K.; Park, C.; Song, J.; Kang, D. Excavator tele-operation system using a human arm. Autom. Constr. 2009, 18, 173–182. [Google Scholar] [CrossRef]
  6. Hanna, K.; Hannu, K.; Helena, T. User experience targets as design drivers: A case study on the development of a remote crane operator station. In Proceedings of the 31st European Conference on Cognitive Ergonomics, New York, NY, USA, 26–28 August 2013; Volume 25, pp. 1–9. [Google Scholar] [CrossRef]
  7. Hannu, K.; Hanna, K.; Jaakko, H. Enhancing the user experience of the crane operator: Comparing work demands in two operational settings. In Proceedings of the 30th European Conference on Cognitive Ergonomics, New York, NY, USA, 28–31 August 2012; pp. 37–44. [Google Scholar] [CrossRef]
  8. Karvonen, H.; Koskinen, H.; Tokkonen, H.; Hakulinen, J. Evaluation of User Experience Goal Fulfillment: Case Remote Operator Station. Lecture Notes in Computer Science. In Proceedings of the Virtual, Augmented and Mixed Reality, Heraklion, Greece, 22–27 June 2014; Volume 8526. [Google Scholar] [CrossRef]
  9. Sen, R.N.; Das, S. An ergonomics study on compatibility of controls of overhead cranes in a heavy engineering factory in West Bengal. Appl. Ergon. 2000, 31, 179–184. [Google Scholar] [CrossRef]
  10. Lee, H.H. Modeling and control of a three-dimensional overhead crane. J. Dyn. Syst. Meas. Control 1998, 120, 471–476. [Google Scholar] [CrossRef]
  11. Chen, S.J.; Hein, B.; Wörn, H. Swing attenuation of suspended objects transported by robot manipulator using acceleration compensation. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA, USA, 29 October–2 November 2007; pp. 2919–2924. [Google Scholar]
  12. Masoud, Z.N. Effect of hoisting cable elasticity on anti-sway controllers of quay-side container cranes. Nonlinear Dyn. 2009, 58, 129–140. [Google Scholar] [CrossRef]
  13. Ju, F.; Choo, Y.S.; Cui, F.S. Dynamic response of tower crane induced by the pendulum motion of the payload. Int. J. Solids Struct. 2006, 43, 376–389. [Google Scholar] [CrossRef] [Green Version]
  14. Chen, J.Y.C.; Haas, E.C.; Barnes, M.J. Human performance issues and user interface design for teleoperated robots. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 2007, 37, 1231–1245. [Google Scholar] [CrossRef]
  15. Murphy, R.R. Human-Robot interaction in rescue robotics. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 2004, 34, 138–153. [Google Scholar] [CrossRef]
  16. Voshell, M.; Woods, D.D.; Phillips, F. Overcoming the keyhole in human-robot coordination: Simulation and evaluation. Proc. Hum. Factors Ergon. Soc. Annu. Meet 2005, 49, 442–446. [Google Scholar] [CrossRef]
  17. Son, H.I.; Franchi, A.; Chuang, L.L.; Kim, J.; Bulthoff, H.H.; Giordano, P.R. Human-Centered Design and Evaluation of Haptic Cueing for Teleoperation of Multiple Mobile Robots. IEEE Trans. Cybern. 2013, 43, 597–609. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  18. Sano, A.; Fujimoto, H.; Takai, T. Human-centered scaling in micro-teleoperation. In Proceedings of the 2001 ICRA, IEEE International Conference on Robotics and Automation, Seoul, Korea, 21–26 May 2001; Volume 1, pp. 380–385. [Google Scholar] [CrossRef]
  19. Penizzotto, F.; Slawiñski, E.; Salinas, L.R.; Mut, V.A. Human-centered control scheme for delayed bilateral teleoperation of mobile robots. Adv. Robot. 2015, 29, 1253–1268. [Google Scholar] [CrossRef]
  20. Schneider, M.L.; Arble, F.B.; Olson, N.; Wolff, D. An overview of ergonomic considerations in computer systems. In Proceedings of the 7th Annual Computer Personnel Research Conference, Miami, FL, USA, 26–27 June 1980; pp. 116–124. [Google Scholar] [CrossRef]
  21. Kling, R. The organizational context of user-centered software designs. MIS Q. 1977, 1, 41–52. [Google Scholar] [CrossRef] [Green Version]
  22. Zekavat, P.R.; Bernold, L. Embedded Wireless communication platform addresses crane safety and efficiency. In Proceedings of the Construction Research Congress: Construction in a Global Network, Atlanta, GA, USA, 19–21 May 2014; pp. 309–318. [Google Scholar] [CrossRef] [Green Version]
  23. Jouppi, N.P. First steps towards mutually-immersive mobile telepresence. In Proceedings of the 2002 ACM Conference on Computer Supported Cooperative Work, New York, NY, USA, 16–20 November 2002; p. 354. [Google Scholar] [CrossRef]
  24. Burke, J.L.; Murphy, R.R.; Coovert, M.D.; Riddle, D.L. Moonlight in miami: Field study of human-robot interaction in the context of an urban search and rescue disaster response training exercise. Hum.-Comput. Interact. 2004, 19, 85–116. [Google Scholar] [CrossRef]
  25. Turro, N.; Khatib, O.; CosteManiere, E. Haptically Augmented Teleoperation. In Proceedings of the 2001 ICRA, IEEE International Conference on Robotics and Automation, Seoul, Korea, 21–26 May 2001; Volume 1, pp. 386–392. [Google Scholar] [CrossRef]
  26. Chi, H.L.; Kang, S.C.; Hsieh, S.H.; Wang, X. Optimization and evaluation of automatic rigging path guidance for tele-operated construction crane. In Proceedings of the ISARC, the International Symposium on Automation and Robotics in Construction, Sydney, Australia, 9–11 July 2014; Volume 31, p. 1. [Google Scholar] [CrossRef] [Green Version]
  27. Chi, H.L.; Kang, S.C.; Hsieh, S.H. Experiences on designing user interfaces for a tele-operated crane. In Proceedings of the 11th International Conference on Construction Applications of Virtual Reality, Weimar, Germany, 3–4 November 2001; pp. 413–424. [Google Scholar]
  28. Abdullah, U.N.N.; Handroos, H. Investigation on sense of control parameters for joystick interface in remote operated container crane application. Proc. AIP Conf. 2017, 1885, 020077. [Google Scholar] [CrossRef]
  29. Karray, F.; Alemzadeh, M.; Abou Saleh, J.; Arab, M.N. Human-computer interaction: Overview on state of the art. Int. J. Smart Sens. Intell. Syst. 2017, 1, 137–159. [Google Scholar] [CrossRef] [Green Version]
  30. Fiorini, P.; Bejczy, A.K.; Schenker, P.S. Integrated interface for advanced teleoperation. IEEE Control Syst. Mag. 1993, 13, 15–20. [Google Scholar] [CrossRef]
  31. Olivares, R.; Zhou, C.; Bodenheimer, B.; Adams, J.A. Interface evaluation for mobile robot teleoperation. In Proceedings of the ACM Southeast Conference, Savannah, GA, USA, 7–8 March 2003; Volume 112, p. 118. [Google Scholar]
  32. Alves, R.L.; Vassallo, R.F.; Freire, E.O.; Bastos-Filho, T.F. Teleoperation of a mobile robot through the Internet. In Proceedings of the 43rd IEEE Midwest Symposium on Circuits and Systems, Lansing, MI, USA, 8–11 August 2000; Volume 2, pp. 930–933. [Google Scholar] [CrossRef]
  33. Sian, N.E.; Yokoi, K.; Kajita, S.; Kanehiro, F.; Tanie, K. Whole body teleoperation of a humanoid robot development of a simple master device using joysticks. J. Robot. Soc. Jpn. 2004, 22, 519–527. [Google Scholar] [CrossRef]
  34. Zhu, D.; Gedeon, T.; Taylor, K. “Moving to the centre”: A gaze-driven remote camera control for teleoperation. Interact. Comput. 2011, 23, 85–95. [Google Scholar] [CrossRef]
  35. Huysmans, M.A.; De Looze, M.P.; Hoozemans, M.J.; Van Der Beek, A.J.; Van Dieen, J.H. The effect of joystick handle size and gain at two levels of required precision on performance and physical load on crane operators. Ergonomics 2006, 49, 1021–1035. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  36. Grasshoff, J.; Hansen, L.; Kuhlemann, I.; Ehlers, K. 7DoF Hand and Arm Tracking for Teleoperation of Anthropomorphic Robots. In Proceedings of the ISR, 47st International Symposium on Robotics, Munich, Germany, 21–22 June 2016; pp. 1–8. [Google Scholar]
  37. Doisy, G.; Ronen, A.; Edan, Y. Comparison of three different techniques for camera and motion control of a teleoperated robot. Appl. Ergon. 2017, 58, 527–534. [Google Scholar] [CrossRef]
  38. Mower, C.E.; Merkt, W.; Davies, A.; Vijayakumar, S. Comparing Alternate Modes of Teleoperation for Constrained Tasks. In Proceedings of the IEEE 15th International Conference on Automation Science and Engineering, Vancouver, BC, Canada, 22–26 August 2019; pp. 1497–1504. [Google Scholar] [CrossRef] [Green Version]
  39. Hart, S.G.; Staveland, L.E. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Adv. Phycol. 1988, 52, 139–183. [Google Scholar]
  40. Prusa Research: Prusa i3 MK3S+. Available online: https://www.prusa3d.com/ (accessed on 13 March 2021).
  41. Bessler, N.; Ogiermann, D.; Buchholz, M.-B.; Santel, A.; Heidenreich, J.; Ahmmed, R.; Zaehres, H.; Brand-Saberi, B. Nydus One Syringe Extruder (NOSE): A Prusa i3 3D printer conversion for bioprinting applications utilizing the FRESH-method. HardwareX 2019, 6, e00069. [Google Scholar] [CrossRef]
  42. Roscoe, A.H. Assessing pilot workload. Why measure heart rate, HRV and respiration. Biol. Psychol. 1992, 34, 259–287. [Google Scholar] [CrossRef]
  43. Roscoe, A.H. Heart rate as a psychophysiological measure for in-flight workload assessment. Ergonomics 1993, 36, 1055–1062. [Google Scholar] [CrossRef] [PubMed]
  44. Turner, J.R.; Carroll, D. Heart rate and oxygen consumption during mental arithmetic, a video game, and graded exercise: Further evidence of metabolically-exaggerated cardiac adjustments. Psychophysiology 1985, 22, 261–267. [Google Scholar] [CrossRef] [PubMed]
  45. Biferno, M.A. Mental Workload Measurement: Event-Related Potentials and Ratings of Workload and Fatigue (NASA CR 177354), Moffett Field; NASA Ames Research Center: Mountain View, CA, USA, 1985.
  46. Amza, C.G.; Zapciu, A.; Popescu, D. Paste Extruder—Hardware Add-On for Desktop 3D Printers. Technologies 2017, 5, 50. [Google Scholar] [CrossRef] [Green Version]
  47. Raspberry Pi Foundation: Raspberry Pi 4B. Available online: https://www.raspberrypi.org/products/raspberry-pi-4-model-b/ (accessed on 27 March 2021).
  48. OctoPrint. Available online: https://octoprint.org/ (accessed on 27 March 2021).
  49. McLoone, H.E.; Jacobson, M.; Hegg, C.; Johnson, P.W. User-centered design and evaluation of a next generation fixed-split ergonomic keyboard. Work 2010, 37, 445–456. [Google Scholar] [CrossRef]
  50. Tao, D.; Diao, X.; Wang, T.; Guo, J.; Qu, X. Freehand interaction with large displays: Effects of body posture, interaction distance and target size on task performance, perceived usability and workload. Appl. Ergon. 2021, 93, 103370. [Google Scholar] [CrossRef]
  51. Logitech G X56 H.O.T.A.S. Available online: https://support.logi.com/hc/zh-cn/articles/360024844133 (accessed on 26 April 2021).
  52. Ultraleap for Developers. Available online: https://developer.leapmotion.com/ (accessed on 16 May 2021).
  53. Ando, N.; Lee, J.H.; Hashimoto, H. A study on influence of time delay in teleoperation. In Proceedings of the IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Atlanta, GA, USA, 19–23 September 1999; pp. 317–322. [Google Scholar] [CrossRef]
  54. Shahdi, A.; Sirouspour, S. Adaptive/Robust Control for Time-Delay Teleoperation. IEEE Trans. Robot. 2009, 25, 196–205. [Google Scholar] [CrossRef]
  55. Smith, C.; Jensfelt, P.A. Predictor for operator input for time-delayed teleoperation. Mechatronics 2010, 20, 778–786. [Google Scholar] [CrossRef]
  56. Jackson, J.A.; Srinivasan, D.; Mathiassen, S.E. Consistent individual motor variability traits demonstrated by females performing a long-cycle assembly task under conditions differing in temporal organization. Appl. Ergon. 2020, 85, 103046. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Comparison of mechanism between a Cartesian 3D printer and an EOT crane. (a) The mechanism of a cartesian 3D printer. (b) The mechanism of an EOT crane.
Figure 1. Comparison of mechanism between a Cartesian 3D printer and an EOT crane. (a) The mechanism of a cartesian 3D printer. (b) The mechanism of an EOT crane.
Processes 09 02148 g001
Figure 2. Screen capture of the controller software. (a) Global view. (b) Conventional cabin view. (c) Bird view. (d) Top view.
Figure 2. Screen capture of the controller software. (a) Global view. (b) Conventional cabin view. (c) Bird view. (d) Top view.
Processes 09 02148 g002
Figure 3. The mechanical prototype of the tele-operated EOT crane.
Figure 3. The mechanical prototype of the tele-operated EOT crane.
Processes 09 02148 g003
Figure 4. The system set-up diagram.
Figure 4. The system set-up diagram.
Processes 09 02148 g004
Figure 5. Input devices. (P) Pointing device. (K) Keyboard. (O) Orientation. (HG) Hand-free gesture.
Figure 5. Input devices. (P) Pointing device. (K) Keyboard. (O) Orientation. (HG) Hand-free gesture.
Processes 09 02148 g005
Figure 6. The experiment set-up. (a) Teleoperation room (TR). (b) The lifting and handling field (LHF).
Figure 6. The experiment set-up. (a) Teleoperation room (TR). (b) The lifting and handling field (LHF).
Processes 09 02148 g006
Figure 7. The procedure of the experiment.
Figure 7. The procedure of the experiment.
Processes 09 02148 g007
Figure 8. Distribution of task completion time (s) finishing with each approach for MCI.
Figure 8. Distribution of task completion time (s) finishing with each approach for MCI.
Processes 09 02148 g008
Figure 9. Distribution of unloading accuracy (number of times) by each approach to MCI.
Figure 9. Distribution of unloading accuracy (number of times) by each approach to MCI.
Processes 09 02148 g009
Figure 10. The distribution of heart rate relative to the baseline in % during tasks using each approach for MCI by novice group.
Figure 10. The distribution of heart rate relative to the baseline in % during tasks using each approach for MCI by novice group.
Processes 09 02148 g010
Figure 11. Distribution of the weight of task workload using each approach to MCI by novice group.
Figure 11. Distribution of the weight of task workload using each approach to MCI by novice group.
Processes 09 02148 g011
Figure 12. The distribution of the weighted score of task workload using each control method by novice group.
Figure 12. The distribution of the weighted score of task workload using each control method by novice group.
Processes 09 02148 g012
Table 1. Main control methods of input devices.
Table 1. Main control methods of input devices.
PKOHG
Device nameMicrosoft ergonomic mouseMicrosoft ergonomic keyboardLogitech G X56Leap motion
PlatformPC-drivenPC-drivenPC-drivenPC-driven
Hand actionPointing and clickingPressingOrientingGestures
CT control←, → button in X axisA, D keysStick ←, →Right fist moves in X axis
LT control↑, ↓ button in Y axisW, S keysStick ↑, ↓Right fist moves in Y axis
HC control↑, ↓ button in Z axis↑, ↓ keysPOV ↑, ↓Right fist moves in Z axis
ES controlOn/off buttonSpace keyC stickLeft fist releasing
Full speedFull speed buttonShift keyFully orientingFaster moving
Hide cursorN/AN/APinkyN/A
Table 2. Descriptive statistics of task completion time, accuracy, heart rate variation, the weight of workload, and the weighted score of the workload for each trial.
Table 2. Descriptive statistics of task completion time, accuracy, heart rate variation, the weight of workload, and the weighted score of the workload for each trial.
TrialCompletion Time 1AccuracyHeart Rate Variation 2Weight of Physical
Demand
Weight of Mental
Demand
Weighted Score of Workload
NoviceExpertNoviceExpertNoviceNoviceNoviceNovice
P635.48 ± 169.641132.73 ± 295.483.76 ± 1.244.00 ± 1.106.79 ± 8.212.10 ± 1.512.24 ± 1.4547.09 ± 10.78
K634.24 ± 198.291038.55 ± 233.533.76 ± 0.894.45 ± 0.948.16 ± 7.131.86 ± 1.652.29 ± 1.2751.86 ± 9.34
O553.76 ± 238.94798.00 ± 135.063.81 ± 1.215.00 ± 0.006.92 ± 5.412.10 ± 1.672.33 ± 1.4349.81 ± 13.22
HG1006.67 ± 365.151374.91 ± 377.183.14 ± 1.323.64 ± 1.439.17 ± 5.973.71 ± 1.422.05 ± 1.5372.73 ± 11.32
1 Unit: Second. 2 Unit: %.
Table 3. The results of post hoc comparisons of completion time, the weight of physical demand, and the weighted score of workload between trials.
Table 3. The results of post hoc comparisons of completion time, the weight of physical demand, and the weighted score of workload between trials.
TrialsCompletion TimeWeight of Physical DemandWeighted Score of Workload
t-Valuep-Valuez-Valuep-Valuet-Valuep-Value
P-K0.0360.972−0.4760.634−1.9830.061
P-O1.7090.103−0.1030.918−0.8990.379
P-HG−4.4560.0001−3.5900.0001−8.5260.0001
K-O2.0680.052−0.6820.4951.0650.299
K-HG−5.2460.0001−3.4050.0001−9.5650.0001
O-HG−5.4820.0001−3.3750.0001−6.8030.0001
1p < 0.001.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Yu, Z.; Luo, J.; Zhang, H.; Onchi, E.; Lee, S.-H. Approaches for Motion Control Interface and Tele-Operated Overhead Crane Handling Tasks. Processes 2021, 9, 2148. https://doi.org/10.3390/pr9122148

AMA Style

Yu Z, Luo J, Zhang H, Onchi E, Lee S-H. Approaches for Motion Control Interface and Tele-Operated Overhead Crane Handling Tasks. Processes. 2021; 9(12):2148. https://doi.org/10.3390/pr9122148

Chicago/Turabian Style

Yu, Zhefan, Jianping Luo, Han Zhang, Eiji Onchi, and Seung-Hee Lee. 2021. "Approaches for Motion Control Interface and Tele-Operated Overhead Crane Handling Tasks" Processes 9, no. 12: 2148. https://doi.org/10.3390/pr9122148

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop