Improving Teleoperator Efficiency Using Position–Rate Hybrid Controllers and Task Decomposition

Round 1

Reviewer 1 Report (Previous Reviewer 1)

In the revised manuscript, the reviewer doesn't see much improvement regarding the comments given in first round review. In fact, there's no even "reply to comments" and the revisions made are not highlighted in the new submission.

The reviewer is disappointed to find out that no efforts have been made by the authors to address the most fundamental issues like the original contributions, technical and experimental key details. In this case, the reviewer deems this submission no difference than the first submission with major flaws remaining.

Author Response

1. The contribution of this work appears to be limited considering that both position/rate and hybrid teleoperation control strategies have been well investigated in literature. The specific task under study is too structured to address and poses no challenges in robotic teleoperation. In fact, as the task can be clearly decomposed into certain simple sub-tasks, with the sensors used in this study an autonomous robotic system without intervention of human being can do the same job with even better accuracy. So the reviewer is not sure why a teleoperated control strategy is necessary for such task.

Response: Thank you for pointing this out. Initially, the goal was to develope a process recognizer that could be applied even in an unstructured environment. To this end, a process recognizer based on vision images was designed, but the recognition rate was very low, so it could not be applied. Although all the systems used in this paper have been dismantled and we can no longer conduct experiments, we are trying to improve the recognition of subtasks even in an unstructured environment by enabling vision-based recognition of subtasks in the environment we are currently researching.

In addition, in an environment such as a nuclear power plant, automatic control cannot be used because it is necessary to prepare for an unexpected situation or a high risk factor. Since MADMOFEM remote work is a task that needs to be carried out quickly in a set order, it was confirmed as a result of the experiment that it helps a lot through the restraint of the used hybrid controller or master device.

 

2. For the 2D monitoring image problem in teleoperation, actually many solutions have been proposed and used in practice to tackle this problem, such as 3D screen with 3D glasses, VR HMD (head mounted devices), etc. As such 3D display devices are not expensive and have become a norm in teleoperation practice, why the authors insist on the 2D monitoring method?

Response: Thank you for the observation. The global camera is equipped with a zed stereo camera, so we experimented while viewing the 3D screen with 3D glasses. However, each person has a different time to adapt to the 3D screen, and since wearing it for a long time can cause dizziness, we decided to experiment with only the 2D camera. We totally agree with the opinion that 3D displays help with teleoperation.

 

3. To the reviewer's opinion, the reduction in cognitive burden of operator is due to the haptic constraint when only one-DOF motion is necessary instead of the hybrid control method, which enables the operator to focus on the simpler manipulation with the haptic assistance.

Response: Thank you for this suggestion. Yes, you're right. It was intended to reduce the cognitive load of the operator by reducing the degree of freedom of operation, and to reduce the workload by reducing the working time through the hybrid controller. We modified the sentence (line 68) “Prolonged telemanipulation or interpretation of the 2D camera images of the remote site can increase the cognitive load experienced by the operator.” to “Prolonged telemanipulation can cause heavy workload and interpretation of the 2D camera images of the remote site can increase the cognitive load experienced by the operator.”

 

 

4. The process recognizer depends on the distance measurement between slave robot and target to identify current state, however it is not mentioned/discussed in this work the possible situations where operation error occurs and the distance varies around the state switching threshold. It should provide some margin for human manipulation inaccuracy and/or mistakes.

Response: Thank you for pointing this out. We also set an appropriate threshold value through several trial end errors in consideration of the operation error. Additionally, since the operation cannot be performed due to an error caused by an operation error (for example, if an x, y error occurs in c2 state, it cannot be inserted by moving only in the z-axis direction in c3 state), the operation is performed through fine manipulation (c4 or s3 state).

Furthermore, if the condition is satisfied and the state is transferred, it is designed not to go to the previous state.

 

5. The authors are suggested to proof read the manuscript carefully before its submission. The first several paragraphs are apparently from the journal template and should be removed in the official draft. Very strange why this problem has been overlooked by the authors.

Response: Thank you for pointing this out. It seems that the file was uploaded incorrectly due to a file error during the submission process.

Author Response File: Author Response.pdf

Reviewer 2 Report (New Reviewer)

The study focuses on the improving a teleoperator efficiency using position-rate hybrid controllers and task decomposition. The paper shows good theoretical and practical background. My main attention to the paper is that:

1. The paper needs additional formatting in some places.

2. In all equations must be dot or comma in the end of equations.

3. Please explaine more deeply, pg. 10/11, lines: 253:254 <In our current teleoperation system, the process recognizer outputs the result based on the position of the slave and the target.>.

Author Response

Response to Reviewer 2 Comments

The study focuses on the improving a teleoperator efficiency using position-rate hybrid controllers and task decomposition. The paper shows good theoretical and practical background. My main attention to the paper is that:

1. The paper needs additional formatting in some places.

Response: Thank you for pointing this out. Fixed additional formatting and some typos.

2. In all equations must be dot or comma in the end of equations.

Response: Thank you for pointing this out. Added points or commas at the end of all equations.

3. Please explain more deeply, pg. 10/11, lines: 253:254 <In our current teleoperation system, the process recognizer outputs the result based on the position of the slave and the target.>.

Response: A process recognizer is used to allocate an advantageous teleoperation control method for each subtask. It inputs the position of the EEFs and the target objects based on the slave robot bases, and outputs the index of the sub-task being performed by the robot.

We modified the sentence “In our current teleoperation system, the process recognizer outputs the result based on the position of the slave and the target.” to “In our current teleoperation system, the process recognizer outputs the result of the current subtask index based on the position of the slave and the target.”

Author Response File: Author Response.pdf

Reviewer 3 Report (New Reviewer)

COMMENTS RECOMMENDATIONS:

11. The topic can be considered as interesting and suitable for the journal. The article aims with improving of teleoperator efficiency using position-rate hybrid controllers and task decomposition. According to current stat-of-the-art (chapter Introduction) such system can bring significant contribution for applications, where the remotely controlled arms are needed due to hazardous environment conditions (e.g. space industry, operation in nuclear power plants, etc.). The contribution and efficiency of designed approach was proved by evaluation of NASA task load index. The authors presented quite high efficiency improvement (there was stated time reduction between 17 – 34% compared with conventional position controllers). On the other hand the description, approach, experiment and discussion can be improved.

22. Some minor gamar improvements can be recommended, e.g. lines 26-27, where to many words „task“ are used. And others. Recommended to check the whole article for typos once again.

33.   I recommend to add any introduction (at least few lines) between the main-chapter heading and sub-chapter heading, which briely describe / introduce the whole main chapter, and therefore bring some basic overview for the reader, which can help him oriented better in the text.

44.   If I understand correctly, the experiment was carried out on the MADMOFEM device designed by Shin. I am missing any brief description of it. I wasnt able to recognise, where are located cameras – e.g. in Figure 4 there are mentioned „Lef hand-eye camera“ / „R H-E camera“ / „Global camera“. Their positions can influence the process of navigation, isnt it?

55.   I supose, that the both haptic devices with parallel kinematics control the dual-arm robotic system. The vertical lift (telescopic mast with range 9m) is controled by another device? What about redundant axis?

66.      Fig. 1: there are shown only the labels c1, c2, c4, c5, c6. Is it possible to shown there the c3, c7, s1-s5 as well?

77.      Line 194: „All communication between programs was based on the TCP/IP protocol (Fig. 6).“ Which programs? Can you specified them more detailed – their task, in which programming language they were prepared, they were designed by you, etc.

88.      Line 203: „The sequence of the tasks is shown in Figs. 1 and 2.1.“ What do you mean under Fig. 2.1 ? Or i tis sub-chapter 2.1 ?

99.      Which role plays the Visualization in ROS in the middle of View 1, Figure 4 ? It is only simulation, or it helps to the operator to navigate as well?

110.   There is not obvious, how was measured the completion time. Only stated, that with hybrid control 212s and 257 without hybrid control. There is any influence of training as well?

111.   Line 234: „Subjects scored between 0 and 100 for six questionnaire measurements“. How many subjects were tested. All of them did the same task by both control approaches? There was changed their order as well (conventional – hybrid control / hybrid - conventional)?

112.   From my point of view, the main issue with the article in present form is its leght and not enough detailed description of the workplace, experiment and mainly section discussion (there is ). So I recommend to improve and extend these parts and describe it more detailed. There are mentioned main features of technical equipment, thats ok, however, at least company, country and any link to producer homepage (where any technical specifications, or datasheets can be downloaded) can be added.

113.   Reference list seems to be adequate. It contains 31 items the text. They are properly cited. However it seams that not all of the references contains all relevant data, mainly the articles published in conference proceedings (data about the location, dates, etc.).

114.   The results, or contribution can be described a bit more.

 

115.   Any video as supplementary material can be helpful.

 

Author Response

 

11. The topic can be considered as interesting and suitable for the journal. The article aims with improving of teleoperator efficiency using position-rate hybrid controllers and task decomposition. According to current stat-of-the-art (chapter Introduction) such system can bring significant contribution for applications, where the remotely controlled arms are needed due to hazardous environment conditions (e.g. space industry, operation in nuclear power plants, etc.). The contribution and efficiency of designed approach was proved by evaluation of NASA task load index. The authors presented quite high efficiency improvement (there was stated time reduction between 17 – 34% compared with conventional position controllers). On the other hand the description, approach, experiment and discussion can be improved.

 

22. Some minor gamar improvements can be recommended, e.g. lines 26-27, where to many words „task“ are used. And others. Recommended to check the whole article for typos once again.

Response: Thank you for pointing this out. We modified the sentence “In [10], method for formulating the desired motion from any number of task frames associated with the task to a set of task objectives was developed.” to “In [2], a method has been developed to formulate the desired behavior in any number of task frames associated with the task to optimize motion control in a limited workspace.”, and fixed some typos.

 

33. I recommend to add any introduction (at least few lines) between the main-chapter heading and sub-chapter heading, which briely describe / introduce the whole main chapter, and therefore bring some basic overview for the reader, which can help him oriented better in the text.

Response: Thank you for the observation. We added content between the main chapter title and sub-chapter title.

 

44. If I understand correctly, the experiment was carried out on the MADMOFEM device designed by Shin. I am missing any brief description of it. I wasnt able to recognise, where are located cameras – e.g. in Figure 4 there are mentioned „Lef hand-eye camera“ / „R H-E camera“ / „Global camera“. Their positions can influence the process of navigation, isnt it?

Response: MADMOFEM is described in 2.1. Figure 4 shows a camera view image. In the left-hand picture of Figure 3, a pan-tilt camera is attached to the 2-link structure between the two slave robots. In Figure 4, it can be seen that each hand-eye camera (realsense) is attached in View 2. The camera does not affect the recognition result because the process recognizer uses eef and target point positions as input based on the base of the slave robot.

 

55. I supose, that the both haptic devices with parallel kinematics control the dual-arm robotic system. The vertical lift (telescopic mast with range 9m) is controled by another device? What about redundant axis?

Response: The vertical lift is operated by a separate program. It includes a GUI that can control the mobile robot at the bottom of the robot system and vertical lift with some buttons.

 

66. Fig. 1: there are shown only the labels c1, c2, c4, c5, c6. Is it possible to shown there the c3, c7, s1-s5 as well?

Response: Thank you for this suggestion. C3 and C7 states are in standby state, so there is nothing to display. Since there is a spatial restriction to display all of the pictures, the picture was marked as an example only for the clutch task. Like the C2 state, the degree of freedom of S1 state is restricted to move in the x-y plane, and the states S2 and S3 are restricted in the z-axis direction.

 

77. Line 194: „All communication between programs was based on the TCP/IP protocol (Fig. 6).“ Which programs? Can you specified them more detailed – their task, in which programming language they were prepared, they were designed by you, etc.

Response: A program written in C++ on the master PC is used as a TCP/IP server and connected to two master devices, and the master devices’ position and velocity required for input of the slave robot and force values for generating haptic rendering are calculated. The slave PC is programmed in C++ and includes a controller to operate the slave robot, forward/inverse kinematics, and a process recognizer.

 

88. Line 203: „The sequence of the tasks is shown in Figs. 1 and 2.1.“ What do you mean under Fig. 2.1 ? Or i tis sub-chapter 2.1 ?

Response: Thank you for pointing this out. 2.1 was incorrectly stated and deleted.

 

99. Which role plays the Visualization in ROS in the middle of View 1, Figure 4 ? It is only simulation, or it helps to the operator to navigate as well?

Response: Since it is impossible to know what posture the robot is in only from the camera image, it is an added image to compensate for this. It can be checked that the robot is not in a position close to singularity and that there is no collision between the two robots.

 

110. There is not obvious, how was measured the completion time. Only stated, that with hybrid control 212s and 257 without hybrid control. There is any influence of training as well?

Response: Thank you for the observation. Since it was not possible to attach additional sensors to measure the time accurately, we started and ended data logging with a mouse click to check the time measured. In teleoperation experiment, as the experiment progressed, the experiment time tended to decrease as the user became accustomed to the teleoperation.

111. Line 234: „Subjects scored between 0 and 100 for six questionnaire measurements“. How many subjects were tested. All of them did the same task by both control approaches? There was changed their order as well (conventional – hybrid control / hybrid - conventional)?

Response: Seven subjects participated in the experiment, and the order was randomly determined for each set.

 

112. From my point of view, the main issue with the article in present form is its leght and not enough detailed description of the workplace, experiment and mainly section discussion (there is ). So I recommend to improve and extend these parts and describe it more detailed. There are mentioned main features of technical equipment, thats ok, however, at least company, country and any link to producer homepage (where any technical specifications, or datasheets can be downloaded) can be added.

Response: Thank you for this suggestion. We added details about the robot system and scenario.

 

113. Reference list seems to be adequate. It contains 31 items the text. They are properly cited. However it seams that not all of the references contains all relevant data, mainly the articles published in conference proceedings (data about the location, dates, etc.).

Response: Thank you for pointing this out. There was one duplicate citation, so it has been removed and some citations have been corrected.

 

114. The results, or contribution can be described a bit more.

Response: Thank you for pointing this out. Related content has been added.

 

115. Any video as supplementary material can be helpful.

Response: Thank you for this suggestion. We attached the video.

Author Response File: Author Response.pdf

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

This paper presents a position-rate hybrid control strategy for a specific teleoperation task so called MADMOFEM (operating a manual drive mechanism of a fuel exchange machine). Efficiency improvement was demonstrated through NASA-TLX comparing with pure position teleoperation. The reviewer has following major comments on this work:

1. The contribution of this work appears to be limited considering that both position/rate and hybrid teleoperation control strategies have been well investigated in literature. The specific task under study is too structured to address and poses no challenges in robotic teleoperation. In fact, as the task can be clearly decomposed into certain simple sub-tasks, with the sensors used in this study an autonomous robotic system without intervention of human being can do the same job with even better accuracy. So the reviewer is not sure why a teleoperated control strategy is necessary for such task.
2. For the 2D monitoring image problem in teleoperation, actually many solutions have been proposed and used in practice to tackle this problem, such as 3D screen with 3D glasses, VR HMD (head mounted devices), etc. As such 3D display devices are not expensive and have become a norm in teleoperation practice, why the authors insist on the 2D monitoring method?
3. To the reviewer's opinion, the reduction in cognitive burden of operator is due to the haptic constraint when only one-DOF motion is necessary instead of the hybrid control method, which enables the operator to focus on the simpler manipulation with the haptic assistance.
4. The process recognizer depends on the distance measurement between slave robot and target to identify current state, however it is not mentioned/discussed in this work the possible situations where operation error occurs and the distance varies around the state switching threshold. It should provide some margin for human manipulation inaccuracy and/or mistakes.
5. The authors are suggested to proof read the manuscript carefully before its submission. The first several paragraphs are apparently from the journal template and should be removed in the official draft. Very strange why this problem has been overlooked by the authors.

Reviewer 2 Report

This paper presents the design and experimental verification of a hybrid position/velocity control scheme for a particular task related to nuclear robotics. The results show the new control scheme performs favourably compared to a standard position control mode. 

I like this paper! The work is interesting and you have found a simple, intuitive solution for a complex problem. 

I believe the main contribution here is the automated switching from position to rate-control; you should make what you believe to be your main contribution absolutely clear at the end of the introduction.

I think a better test in the experiment would be to compare between your automatic switching algorithm to a manual switching mode (i.e. using a button to switch), as you have noted examples of in your introduction. The subtasks could remain, and the operator is alerted to which control mode they should be in. You should make a note of this in the conclusion.

I am aware of how hard it has been recently to carry out experiments with participants due to Covid, but 7 participants is very low. I would expect to see at least 15 for statistical analysis. You can address this as a limitation in your conclusion. 

Section 2 appears to contain text from the paper template and should be removed. 

All algebraic representations should be explained directly after declaration, for example x_(m,t) in eq. (1) is not clear. Additionally, it appears the notation should be x_m(t) instead of x_(m,t). Please check all notation to make sure they are clear and understandable.

Figure 7 does not show any useful information. Consider breaking it down into a graph for each axis and labelling points of interest.

You should have some sort of reference for NASA-TLX, maybe some examples of use in the introduction. 

You should note what tests were used to obtain your p-values. 

It is not made clear exactly what the hybrid mode is being tested against in the experiment - is it purely the position-control of the same type used in the hybrid mode? 

Reviewer 3 Report

Dear authors,

Although the paper is written appropriately, I found some major concerns with the study presented as following:

- Participants: it is not clear what kind of participants joined the evaluation.

- Consent: There is no description on ethical consideration nor on written consent from the participants

- Overall design: the method proposed in the paper is to pre-program switching of two different teleoperation methods. However, if such pre-programming is necessary, I would say that autonomous, not teleoperated, operation of the robot shall be mostly possible, especially for the navigation part. Besides, it is not clear what kind of instructions were provided to the participants - If not trained appropriately, I imagine that participants would have been confused with the switching of the teleoperation method change. Some human-interface settings for realizing natural switching of teleoperation strategies, as well as practical implementation of switching shall be the main difficulty in such teleoperation system. I think the results shown in the paper are too obvious. I would like to suggest the authors to consider at least one of the above issues. 

Reviewer 4 Report

The authors of this paper presented a multi-stage control strategy for a teleoperation system.  By switching between two control modes that emphasize on accuracy and movement efficiency, respectively, the authors claimed that the proposed teleoperation system can allow different users to finish the so-called MADMOFEM test more easily and efficiently.  Although seven subjects were enrolled in the test, very little information about the details of the experiments was provided so it’s hard to evaluate if the experiments were properly designed to avoid introducing unwanted biases. Also, the NASA-TLX test was purely based on individual opinions.

Overall, the writing and organization of this paper are good. However, experimental part of the paper needs to be significantly improved in order to validate the design efficacy.  In order to further improve the paper, the following issues need to be addressed:

1.       In section of Control method, the first three paragraphs seem to be copied from manuscript instructions and are unrelated to the paper.

2.       In Fig. 1, it would be great if snapshots of the images can be added to better visualize each sub-task. Also, the authors should explain the meaning of different color boxes.

3.       Eq. (3) is missing a parenthesis.

4.       On page 5, the following phrases unnecessarily repeat twice: “Our teleoperation system assigns position and rate control modes based on the current subtask. We selected an advantageous control method from the current subtask among the two control methods to complete the MADMOFEM tasks. For example, because of the distance from the target position (the position of the shaft or driver) to the initial position, we used rate control to approach the target quickly and switched to position control when the target was close, allowing the task to be completed accurately and safely.”

5.       In Fig. 5, the two sub-figures can be combined into one with appropriate labeling.

6.       Fig. 6 was not properly cited in the paper.

7.       In Fig. 7, are the plotted results extracted from one specific subject? If so, please add corresponding information. Also, a table showing the performance of all the seven subjects should be included for this comparison experiment.

8.       In Fig. 8 and 9, are the separate dots denoting the numbers of special cases? Were these separately labeled data points even counted in the statistical analysis? The expression is very confusing.

9.       In Fig. 11, the comparison results plotted in Fig. 11 can be quite subjective if the evaluation process is only based on a questionnaire.