Implementation of a Flexible and Lightweight Depth-Based Visual Servoing Solution for Feature Detection and Tracing of Large, Spatially-Varying Manufacturing Workpieces
and Ashutosh Tiwari 1Round 1
Reviewer 1 Report
This paper proposes a solution for detecting the edges of a workpiece and tracing them with a robot end-effector without prior knowledge of their location or shape. The problem is in line with the thematic of the journal, is useful for this field of research and is clearly stated. The literature review is pertinent to the subject addressed.
However, the methods and procedures proposed by the authors in the research are not explained in detail. Thus, figure 3 presents the workflow used for detecting the dependence values of different environmental variables. However, in section 2.3.1 the procedure is not explained in detail. Similarly, figure 5 presents the workflow used for detecting and moving towards edges with the Mover6. However, in section 2.3.2 they do not explain in detail the function of each block in the diagram. For example, how they calculate the distance to the object at the pixel coordinates (equation 1 is used??), how they do deprojection, where they place the reference systems, etc.
Similarly, figure 6 shows the workflow for detecting the meeting point between a flat plane and a curved edge. Section 2.3.3 does not detail the fundamental steps of this workflow. In these sections (2.3.1, 2.3.2 and 2.3.3) the authors only explain how the experiments were carried out, which should perhaps be explained in the results section.
In my opinion, the authors have greatly simplified the initial problem stated. Throughout the paper the authors state "large, spatially-varying manufacturing workpieces" (in the title), "a highly detailed and complex workpiece" (lines 234-235), Figure 1 shows “an example aerostructure needing to be traced” and so on. However, they have worked with simple, small, 2D workpieces. The complex workpiece in figure 4b was only used for detecting edges and comparing the 3 cameras, but not for edge tracing. Figure 8 shows a simpler, smaller piece ("1 x 0.5 rectangular workpiece " in line 335) which was used for acute feature identification and extraction. The workpiece in Figure 10 has a similar profile to the workpiece in Figure 1, but only the point where the curve began is found and this point is traced along the whole shape (lines 325-326), not the curved profile (lines 397-398 “only the non-curved part of the workpiece is analysed, and not the whole curve”). In experimental test (section 3.2.3) where 3D workpiece is being traced, only 4 different features were detected and traced, but not the contour of the 3D workpiece, as lines 397-398 appears “only the non-curved part of the workpiece is analysed, and not the whole curve”.
Moreover, in this sense, two (sections 3.2.1 and 3.2.2) of the three experimental setups proposed for the identification and tracing of workpieces are carried out in simulation using Gazebo, and only one (section 3.2.3) works with a real robot but is very simplistic. According to the authors, in this setup only two of the four tests had satisfactory results, (line 411 “Tests one and two were unsuccessful”).
Therefore, on the basis of the results presented, the following claims of the authors cannot be concluded:
- In lines 426-427 “the example aerostructure’s edges, both obtuse and acute, would be detected and traceable”
- In lines 431-432 “the method … is generic enough to be used for multiple workpieces; while remaining accurate enough to still detect and trace these large scale workpieces effectively”
- In lines 439-440 “for identifying and tracing workpiece edges regardless of size, spatial profile…”
I fully agree with the authors' conclusion in lines 451-452 “While the analysis showed that the solution provided is robust within the simulation, the real world would have provided a higher level of noise and uncertainty”, although this conclusion invalidates and rejects the method proposed for the problem posed.
In section 2.1. "Selection of Physical Cameras ..." three different cameras are proposed, but only one of them, Intel RealSense, had depth-sensing capability. For the methods proposed by the authors this capability is mandatory. Although the experiments presented in section 3.1.2 "Comparison Experiments" are properly planned and conducted, the conclusion drawn by the authors was highly expected, as appears in the lines 309-310 “The Intel RealSense d435i was chosen to detect and trace the edges of the workpiece due to the depth-sensing capabilities”.
The title of the article does not reflect that a robotic arm will work automatically using a depth camera and neither does the technique of visual servoing. In my opinion it would be highly desirable to include these core concepts in the title.
Heatmaps presented in figures 9a and 9b are unusual. In figure 9a, the highest disparity is at the bottom left, while in figure 9b it is at the bottom right. Why is this?
Section 2.2 "Selection of Software" does not include OpenCV, although this library is mentioned throughout the paper.
Several typos appear:
- Diagrams in figures 2, 3, 5 and 6 contain “fidn”, “origonal”, “tolarenace”, “gloabal”, “inciment”, “increment”.
- Line 426 contains “Looking back to Figure ?? as an example, the example aerostructure’s edges”
- Line 466 contains ”Funding: This research was funded by NAME OF FUNDER grant number XXX.”
In the text, quantitative values should appear with numbers, such as in line 268 "within one meter", in line 339 "ten times", and so on.
Author Response
Thank you for reviewing our manuscript. After carefully reading your comments, we have made the necessary changes, as detailed in the attached document.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The reviewed paper deals with the issue of detection and tracing of large, spatially-varying manufacturing workpieces. This is an issue that professionally fits into the concept of the journal. The paper has a good professional level and I have not major comments on its content.
But, I recommend to rework the abstract of the paper which us very general.
Fig. 4 is not clear and I recommend to rework it.
Fig 7 is very difficult to read and it has a lie infirmative value.
Fig 11,12,13 are difficult to rea.
At the same time, I recommend to expand the chapter Conclusion about a part “ future work” about specific results, and emphasize the benefitd of the research and the presented published papers.
After these comments I recommend to publish this paper
Author Response
Thank you for reviewing our manuscript. After carefully reading your comments, we have made the necessary changes, as detailed in the attached document.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
In the new paper, the authors have corrected some of the revisions, but not the most important ones, which I indicate below.
1.- About my previous comments "the authors have greatly simplified the initial problem posed". In the new version of the paper, they have simply added a new experiment in Gazebo simulation with the piece of Figure 1 with additional noise for verification. That is only the new improvement they have included. Therefore, in my opinion, the document presents the problems discussed in my previous review in that it does not include experiments in a real environment, with workpieces of different geometry and size, materials, etc. The claims of the authors, noted in my previous review, cannot be concluded.
2.- About my previous comments “methods and procedures proposed by the authors in the research are not explained in detail”. The authors have included lines 190-197, where they only rewrite the text that appears in the workflow blocks in Figure 5. There is even repetition of the text in lines 200-203. In addition, the authors have decided only to divide section “2. Materials and Methods” into two sections “2. Selection of Hardware and Software” and “3. Development of Experimental Setup”. However, they have kept the same subsections and the same order, so the final effect is the same. The change is not an improvement on the previous version. The proposed algorithms are mixed with experimental setup, and this is confusing and, in some cases, repetitive. In my opinion, the authors should explain the algorithms in the "Materials and Methods" section and in another section explain the "experimental setup" used to validate the algorithms.
3.- About my previous comments “Selection of Physical Cameras ...". The authors do not describe in detail the experiments for the comparison of the 3 cameras, so it would be difficult to repeat them. As an example, in Lines 192-194 “A depth sensor was then used to provide the camera with the Z coordinates, the distance between the camera and the workpiece”. What was the depth sensor used, was it a scanning sensor, where was it placed? The authors do not mention that the Intel RealSense camera was the one used. In my opinion, the sections dedicated to compare the 3 cameras do not make a relevant contribution to the presented work, because only one of them, Intel RealSense, had depth-sensing capability. For the methods proposed by the authors this capability is mandatory. In the new version of the paper the authors include a paragraph (Lines 197-200) for measuring depth by other methods. Before the results of the experiments, they already comment that the chosen camera measures the distance (“While the camera used was able to easily get the distance …”). Therefore, with this paragraph they try to justify the comparison of the 3 cameras, but without much success.
The conclusion drawn by the authors was highly expected, as appears in the lines 345-350 “The Intel RealSense d435i was chosen to detect and trace the edges of the workpiece (Figure 7) due to the depth-sensing capabilities, allowing the distance of the workpiece to be obtained without any intrinsic calibrations needing to be completed. Additionally, the edges of the workpiece were detected clearly, and the solution was lightweight and required minimal expert setup. Using this camera, the various edges were identified, extracted and then finally traced.”
In my opinion the authors should have focused exclusively on modelling the behaviour of the Intel RealSense d435i camera in detail. In this way the behaviour explained in lines 395-408 and the problems with the distance in the heatmap could be better understood.
Author Response
Thank you for your second round of revisions.
After looking through your comments, we feel we have made the appropriate changes, as detailed in the attached file.
Author Response File: Author Response.pdf
Round 3
Reviewer 1 Report
The authors have revised the document following my suggestions and comments 1.1, 1.2 and 1.3, as it appears in the cover letter. Thank you very much.
Below, I point out some typographical errors and other suggestions in text editing, which can be considered minor revisions.
Typos
- Line 12: “regardless of of the cardinal direction”
- Figure 3: “Deproject the pixel coordinates ad the distance into 3D coordinate”
- Line 114: “ten meters” – “10 meters”
- Line 239: “(similar to the one seen in Figure 1,”
- Line 252: “the ground(Figure 7)”
- Line 240: “The areospace structure”
- Line 310: “canny edge detection method”
- Figure 17, 18 and 19: “Y coordinate distnace”
- Line 420: “workpiece(10mm ±5mm)”
- Figure 21, 22 and 23: “a aerospace inspired workpiece”
Other suggestions in text editing
- Lines 156 – 158: “A depth sensor was then used to provide the camera with the Z coordinates, the distance between the camera and the workpiece.”
Which sensor was used? The Intel Realsense D435i?
- Lines 211-214 should be deleted
“With this optimal setup, the three cameras were taken and compared against each other as well as against a ground truth image. After grading each output image, it was then established which camera would be most suitable for edge tracing.”
- Figure 6a is not referenced in the text.
- In Figure 14 and Figure 15 appears: “Absolute euclidean error (m)” – Error should be measured in mm?
Author Response
Thank you for the latest list of recommendations, we have attached a response document detailing our changes.
Author Response File: Author Response.pdf
