Round 1

Reviewer 1 Report

n section 4.2, the implementation should be explained in more detail.

Although the MOD IVI is proposed to enhance the IVI for a larger augmentable space, results obtained are very similar.

While in the introduction two software libraries are mentioned (see [10] and [11]) that eliminate the visual marker requirement, the test suit is implemented on the Lenovo Phab 2 Pro. It should be justified because you have not used these two software libraries to make the test suit implementation.

Justify why two of the five object manipulation methods were randomly selected.

Table 1 should highlight for each test the minimum values of min and mean.

Author Response

Reviewer 1: Changes made in response to comments from Reviewer 1 are highlighted in yellow in the revised paper.

·         In section 4.2, the implementation should be explained in more detail.

o   Section 4.2 completely re-wrote with extended details and a Figure depicting the supporting libraries. Beginning at Line 177 of the revised submission.

 

·         while in the introduction two software libraries are mentioned (see [10] and [11]) that eliminate the visual marker requirement, the test suit is implemented on the Lenovo Phab 2 Pro. It should be justified because you have not used these two software libraries to make the test suit implementation.

o   Added in explanation in Section 4.2: Lenovo Phab 2 supports depth sensing. Beginning at Line 177 of the revised submission.

 

·         Justify why two of the five object manipulation methods were randomly selected.

o   Added in explanation in the first paragraph of Section 7 (each user only test two methods to avoid exhausting the users). Beginning at Line 275 of the revised submission.

 

·         Table 1 should highlight for each test the minimum values of min and mean.

o   We assume what is requested is the highlight of shortest completion time for each method. This is an excellent suggestion. Updated Table-1 with highlights. Beginning at Line 279 of the revised submission.

Author Response File: Author Response.pdf

Reviewer 2 Report

Authors have considered conditions of virtual objects manipulation methods for Augmented Reality handheld devices. Especially marker-based and marker-less approaches were confronted from the perspective of objects’ size as a technique’s/method’s parameter.

Provided comparison and analysis of marker systems and marker-less systems (section 3) is interesting and constructive however superiority of marker-less systems should not be stated so firmly. Flexibility of marker-less systems is not effortless and automatic. It is evident for single marker systems, but multi-markers systems might be comparable with marker-less systems.

In the main research authors concentrated on examining selected objects manipulation methods (derived from marker-based systems) in the context of marker-less AR systems. Though characteristics of selected methods are available (provided references), their brief description (comparable to that in 4.1.1. section) would emphasised their main features and differences.

Test cases were designed appropriately, however mainly their quantitative results were provided. Even though post experiment surveys were conducted theirs detailed motivation and results analysis were not adequately explained.

Post-tests questionnaires detailed analysis should be reported as presented conclusions were not evident and tasks completion time is just a partial experiments parameter.

Qualitative part of the experiments analysis, if provided, would supplement quite interesting, though still not unequivocal conclusions.

Some doubts concern also literature review. Authors should refer to more recent literature positions  - the latest are from 2015. AR human-machine interaction research area is an emerging research field, very well documented in contemporary publications.

Author Response

Reviewer 2: Changes made in response to comments from Reviewer 2 are highlighted in blue in the revised paper.

·         Authors have considered conditions of virtual objects manipulation methods for Augmented Reality handheld devices. Especially marker-based and marker-less approaches were confronted from the perspective of objects’ size as a technique’s/method’s parameter.

o   No response needed

 

·         Provided comparison and analysis of marker systems and marker-less systems (section 3) is interesting and constructive however superiority of marker-less systems should not be stated so firmly. Flexibility of marker-less systems is not effortless and automatic. It is evident for single marker systems, but multi-markers systems might be comparable with marker-less systems.

o   This is an excellent point. We have included a new paragraph in the Section-3 intro section clarifying that our discussion assumes single Marker. Beginning at Line 118 of the revised submission.

 

·         In the main research authors concentrated on examining selected objects manipulation methods (derived from marker-based systems) in the context of marker-less AR systems. Though characteristics of selected methods are available (provided references), their brief description (comparable to that in 4.1.1. section) would emphasised their main features and differences.

o   These statements describe our paper rather well. These do not seem to be recommendation nor criticism. We do not know how to respond to these statements.

 

·         Test cases were designed appropriately, however mainly their quantitative results were provided. Even though post experiment surveys were conducted theirs detailed motivation and results analysis were not adequately explained.

·         Post-tests questionnaires detailed analysis should be reported as presented conclusions were not evident and tasks completion time is just a partial experiments parameter.

·         Qualitative part of the experiments analysis, if provided, would supplement quite interesting, though still not unequivocal conclusions.

o   These three points all point to the need to include more analysis of qualitative user feedback. In response, we have extended Section 9 with a table of average questionnaire results, and a discussion of how these numbers can be interpreted. Beginning at Line 342 of the revised submission.

 

 

·         Some doubts concern also literature review. Authors should refer to more recent literature positions  - the latest are from 2015. AR human-machine interaction research area is an emerging research field, very well documented in contemporary publications.

o   It seems that the reviewer feels more recent citation(s) should be cited. However, without more specifics, we are unsure how to respond to this comment. We note that both of the other two reviewers responded Yes to the question:

“Does the introduction provide sufficient background and include all relevant references”

Author Response File: Author Response.pdf

Reviewer 3 Report

- The introduction is clear and well written. As minor feedback, I would list the findings/contributions of the paper in a bullet point list. It is easier to spot them for the reader.

 

- Line 176. The test suite is 176 implemented on the Lenovo Phab 2 Pro [9] because this was the only available commercial-grade mobile AR device at the time of the study”.

That’s not a solid justification. I would argue that you probably chose that device because is a Google Tango capable device which support depth sensors. Would be interesting to compare markerless depth enabled experiences on Tango vs ARKit, as Tango was discontinued.

 

- Experiments are well explained. Nonetheless, Figure 6, 7 and 8 could be combined into a single one with three subfigures.

 

- Paper is overall interesting. Markerless AR is a key factor to make AR happen to a broad audience. Future work should be extended and more detailed.

Author Response

Reviewer 3: Changes made in response to comments from Reviewer 3 are highlighted in green in the revised paper.

·         The introduction is clear and well written. As minor feedback, I would list the findings/contributions of the paper in a bullet point list. It is easier to spot them for the reader.

o   Concerning about formatting requirements of the journal, we responded to this by Bold-font in the abstract. We hope this is acceptable to the Journal’s format requirement. Beginning at Line 20 of the revised submission

 

·         Line 176. The test suite is 176 implemented on the Lenovo Phab 2 Pro [9] because this was the only available commercial-grade mobile AR device at the time of the study”. That’s not a solid justification. I would argue that you probably chose that device because is a Google Tango capable device which support depth sensors. Would be interesting to compare markerless depth enabled experiences on Tango vs ARKit, as Tango was discontinued.

o   Thank you for raising this excellent point, we have significantly extended Section 4.2, and included an extended discussion Section 10 to address this issue.   Beginning at Line 177 of the revised submission

o   We agree that it is interesting to experiment with the latest depth sensing technology. However, as detailed in our new discussion (lines 392-409, in the conclusion section of the revised paper), our study and results are targeted at direct object manipulation in Markerless AR applications and are independent from the depth sensing technology that enabled Markerless AR capability.

o   It would be indeed interesting to study and compare interact-ability and usability of applications based on hardware vs software depth sensing. However, once again, our study and conclusion are focused on direct object manipulation methods of Markerless AR applications.

o   Lastly, at the time of this study (mid 2017), it is indeed the case that Tango was the only AR-capable consumer-grade mobile device.

 

·         Experiments are well explained. Nonetheless, Figure 6, 7 and 8 could be combined into a single one with three subfigures.

o   Excellent suggestion. Thank you.

§  Small scale tests: combined Figures 2,3, and 4

§  Large scale tests: combined Figures 5, and 6,

§  Working with occlusion and movement limitation: combine Figures 7,8

 

·         Paper is overall interesting. Markerless AR is a key factor to make AR happen to a broad audience. Future work should be extended and more detailed.

o   Extended the last section, and the last paragraph with more detailed articulation of potential future work.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The comment "Although the MOD IVI is proposed to enhance the IVI for a larger augmentable space, results obtained are very similar." are not considered

Author Response

This is our mistake. The cut/paste process we went through missed posting our response to this excellent point raised. Our most sincere apologies.

This issue is new addressed (Please refer to Lines 382 -387). We believe it is important to share both successful and credible unsuccessful trials. This results help point to the proper direction for investigation for the future researchers.

Author Response File: Author Response.pdf

Reviewer 2 Report

Authorsexperimentson virtual objects manipulation methods for Augmented Reality handheld devicestackle a valid research problem, howeverthe discussion considering differencesbetween marker-based and marker-less approaches seems to betendentiousandtoocategorical.It distortsperception of evaluated methods.

Provided comparison and analysis of marker systems and marker-less systems (section 3), even besides recentrevision, neglects the methods’functionalchallenges. I disagree with a statement that “inter-object occlusion in Markerless AR is of no special concern” (line 133). In Markerless AR systems, real objects and their characteristics features become reference points constituting physicalenvironmentmarkers. Their occlusion is of high importance, influencing quality of handheld device tracking efficient(similarly like in Marker AR). Moreover,in reference to Marker systems,“Markerless AR users are free to move around and examine virtual objects from any viewing angle” (line 139) onlyif the tracking system is appropriately designed(thephysicalspace is registered or tracked), otherwise user movements are also restricted. In other words virtual objects are embedded into real environment structure and changeability of real environment may influence stability of virtual components imposition process.

I have also some doubts concerning elaborated method’s contributions. Incorporated modifications (lines 168-172) do not refer directly to object manipulation strategy, but viewing camera context settings (moving object in reference to the user/camera).

Due to system’sfunctionalambiguities(Marker vs Markerless) main goal of the paper “to understand the implications of adopting Marker methods for Markerless AR” (line 196) is not clear and seems to be dependent on quality of tracking systems.

Providedresearchmethodology and testing protocols are well presented and convincing,however taking into account above concerns they should be probably revised again.

Author Response

·         Provided comparison and analysis of marker systems and marker-less systems (section 3), even besides re-centre vision, neglects the methods’ functional challenges. I disagree with a statement that “inter-object occlusion in Markerless AR is of no special concern” (line 133). In Markerless AR systems, real objects and their characteristics features become reference points constituting physical environment markers. Their occlusion is of high importance, influencing quality of handheld device tracking efficient (similarly like in Marker AR). Moreover, in reference to Marker systems, “Markerless AR users are free to move around and examine virtual objects from any viewing angle” (line 139) only if the tracking system is appropriately designed (the physical space is registered or tracked), otherwise user movements are also restricted. In other words virtual objects are embedded into real environment structure and changeability of real environment may influence stability of virtual components imposition process.

o   Thank you sincerely for pointing this out. The sub-section title and the last sentence (as pointed out, line 133) is indeed miss-leading. This is our over-sight. Please allow us to explain.

§  What we meant to point out is: with Marker AR, the maintaining the visibility and tracking of the Marker is key throughout the entire object manipulation process. While with Markerless AR, there is not requirement of maintaining the visibility of any particular object/marker and thus the user has much more freedom of movement.

§  The sub-section title and the statement on line-133 is miss-leading because it claims inter-object occlusion is of no concerns [which, as very clearly described here, is of course not true!].

o   We are in total agreement with the reviewer’s view [that, inter-object occlusion is of key-importance], and we are indeed very aware and concerned with this issue.

§  This concern is brought to the forefront and assessed in our test cases:

·         Large scale Test-4 (translation across physical barrier which can cause partial/temporarily object occlusion). Please refer to Figure 4a.

·         Large scale Test-6 (translation with partial visual occlusion). Please refer to Figure 5a.

o   We have updated lines 129 to 133, to clarify the point we want to make.

 

·         I have also some doubts concerning elaborated method’s contributions. Incorporated modifications (lines 168-172) do not refer directly to object manipulation strategy, but viewing camera context settings (moving object in reference to the user/camera).

o   As discussed in the lines prior to Lines 168-172, the motivation for the Mod IVI method is based on our concern that when users have increased freedom of movement, and when users want to manipulate objects with increased sizes, camera viewing context may also become increasingly important and may be an integral part of object manipulation design.

o   Our study result agrees with the reviewer’s observation that, the proposed method does not contribute significantly.

o   Though unsuccessful, there are two motivations to include this method:

§  First, we continue to believe camera context setting should be considered and should be an integral part of object manipulation design. This is especially true when it comes to the placement of virtual objects where the position and orientation of the virtual object in relation to physical objects can be of key importance.

§  Second, we believe it is important to share both successful and unsuccessful attempts in scientific studies. Our hope is future researchers can build their design based on our attempt, or at the very least, avoid what we have tried.

 

·         Due to system’s functional ambiguities (Marker vs Markerless) main goal of the paper “to understand the implications of adopting Marker methods for Markerless AR” (line 196) is not clear and seems to be dependent on quality of tracking systems.

o   By “functional ambiguities”, we assume the reviewer is referring to the concern that (Line 133) it is very possible to miss-interpret our claim that “inter-object occlusion is of no concerns in Markerless AR” [which, as explained above, is a miss-communication and we attempted to fix this issue in the update].

o   Once again, the Marker and Markerless AR differ in three ways: (Lines 123 to 142)

§  Augmented Space Dimension

§  Visibility to Specific Physical Objects (the marker AR object)

§  User Movement

o   The goal is to examine the existing Marker AR methods in the context of the above differences.

o   In terms of “quality of tracking systems”

§  In Marker AR: after the virtual object is selected and during the manipulation process, the user/application must maintain the tracking of the fixed Marker-object [the pre-defined physical object]

§  In Markerless AR: after the virtual object is selected and during the manipulation process, the user/application does not need to track the fixed Marker-object.

Reviewer 3 Report

I endorse the manuscript to be accepted in the present form

Author Response

Thank you. No response from us.

Round 3

Reviewer 1 Report

No comments

Reviewer 2 Report

Current version of the revised manuscript has already eliminated controversial Marker and Marker-less methods comparison issues. Though overall contribution is very subtle Authors conducted interesting experiments and documented them acceptably. Summing up I can recommend manuscript for publication in the Journal as main misstatements were corrected.

Author Response

Thank you sincerely. We have no response to the latest comment.