SatelliteSkill5—An Augmented Reality Educational Experience Teaching Remote Sensing through the UN Sustainable Development Goals

Round 1

Reviewer 1 Report

Dear author(s),

I have read with much interest your paper titled “SatelliteSkill5 - an Augmented Reality Educational Experience Teaching Remote Sensing through the UN Sustainable Development Goals”. The study takes a new perspective on the subject, providing data and interpretations that contribute to enriching the existing literature.

However, there are some issues that I noticed could be improved.

Please find below minor points in the communication which needs clarification / reanalysis / rewrites and/or additional information and suggestions for what could be done to improve

1.     For the abstract section I suggest a clearer short description of the applied research methodology.

2.     I suggest a clearer outline of the overall purpose of the research and associated research questions at the end of the Introduction section.

3.     In the Materials and methods section, I suggest to add separate sections regarding: research method and instrument, sample, data collection, data analysis. Considering the complexity of the study and the manuscript, this would be very useful to provide a clearer picture of the research design.

4.     For tables and figures I suggest keeping a short caption and transferring the explanations to the text.

5.     I suggest highlighting the limitations of the study.

Author Response

Thank you very much for the review and useful comments. Our edits are as follows:

1. For the abstract section I suggest a clearer short description of the applied research methodology.

We have shortened and reworded the abstract to hopefully better highlight what the methodology incorporates. These revisions also relate to a query from two other reviewers on the instrument/application and we have hopefully clarified this in the revised version.

2. I suggest a clearer outline of the overall purpose of the research and associated research questions at the end of the Introduction section.

We have revised the abstract and introduction closing – and hope this better defines what we hope to do with the submission.

3. In the Materials and methods section, I suggest to add separate sections regarding: research method and instrument, sample, data collection, data analysis. Considering the complexity of the study and the manuscript, this would be very useful to provide a clearer picture of the research design.

Thank you for highlighting this lack of clarity, we have received similar comments on the method and instrument from a second reviewer, and have used this opportunity to revise how we introduce the existing sections that we had hoped covered these, in Section 4, particularly those concerning data collection. This has now also been revised in Section 4 and in subsequent discussion taking the suggestion from the reviewers to highlight that it is an informal evaluation using teacher opinion rather than formal quantitative evaluation. This can be seen in Section 4, and also all reference to quantitative or qualitative analysis has been removed from the abstract and the discussion in Section 5.

4. For tables and figures I suggest keeping a short caption and transferring the explanations to the text.

We have improved the resolution of Figures 1-3, as per a comment from Reviewer 2 – these figures have had panes removed, been resized and the captions have been reduced accordingly and much of the text moved to the main body text. We hope the remaining captions strike the right balance post-edits between the standard of captions that can be read ‘without reference to the text’ and being overly lengthy, as per the reviewer comment.

5. I suggest highlighting the limitations of the study.

This also corresponds with comments from Reviewer 2, and to answer this we have provided additional information on limitations both in terms of the design of the evaluation stage in Section 5.2. and also the technical limitations of the augmented reality experience in a new Sub-Section 5.3.

Reviewer 2 Report

Based on advances in augmented reality (AR) apps, this paper presents the development of the SatelliteSkill5 App, a free, mobile digital AR app for Android and iPhone in Unity AR. The SatelliteSkill5 aids users in conceptualizing sensing (RS) technology by showcasing the potential of datasets such as multispectral images, SAR backscatter, drone orthophotography, or bathymetric LIDAR.

The method was evaluated via quantitative and qualitative approaches in 12 schools with  

861 students. The results showed that students and teachers perceived learning in an easy-to-use AR environment positively and that AR and the SDGs help users better conceptualize parts of the curriculum.

The paper describes the augmented reality (AR) apps well enough, and the overall presentation makes the issue comprehensible to the reader. The system was developed considering pedagogic principles, which add value to the educational character of the app. The technical development of the parts of the system is clearly described, and its novelty and usage become apparent.

However, the weak points of the paper are in the evaluation part. There is no account of the validity and reliability of the measurements and the instrument. This is because the questionnaire is not a formal, valid evaluation instrument but includes a number of simple questions/items posited for the needs of the present inquiry, which, however, convey just participants' opinions about the usability of the system under study. To this end, this is no evaluation per se.

Moreover, 12 cases is a small sample to apply statistics.

The 'evaluation' part is very small compared to the other parts of the work, so even my comments are negative regarding this part, and given that it is hard to ask for a major revision for this, I would be satisfied if the authors just present this part as teachers' opinions about SatelliteSkill5 App, and not as evaluation of it.

Comments on systems limitations would also be nice to mention and elaborate more in the discussion section.

Based on advances in augmented reality (AR) apps, this paper presents the development of the SatelliteSkill5 App, a free, mobile digital AR app for Android and iPhone in Unity AR. The SatelliteSkill5 aids users in conceptualizing sensing (RS) technology by showcasing the potential of datasets such as multispectral images, SAR backscatter, drone orthophotography, or bathymetric LIDAR.

The method was evaluated via quantitative and qualitative approaches in 12 schools with  

861 students. The results showed that students and teachers perceived learning in an easy-to-use AR environment positively and that AR and the SDGs help users better conceptualize parts of the curriculum.

The paper describes the augmented reality (AR) apps well enough, and the overall presentation makes the issue comprehensible to the reader. The system was developed considering pedagogic principles, which add value to the educational character of the app. The technical development of the parts of the system is clearly described, and its novelty and usage become apparent.

However, the weak points of the paper are in the evaluation part. There is no account of the validity and reliability of the measurements and the instrument. This is because the questionnaire is not a formal, valid evaluation instrument but includes a number of simple questions/items posited for the needs of the present inquiry, which, however, convey just participants' opinions about the usability of the system under study. To this end, this is no evaluation per se.

Moreover, 12 cases is a small sample to apply statistics.

The 'evaluation' part is very small compared to the other parts of the work, so even my comments are negative regarding this part, and given that it is hard to ask for a major revision for this, I would be satisfied if the authors just present this part as teachers' opinions about SatelliteSkill5 App, and not as evaluation of it.

Comments on systems limitations would also be nice to mention and elaborate more in the discussion section.

Author Response

Thank you very much for the review and useful comments. Our edits are as follows:

"There is no account of the validity and reliability of the measurements and the instrument.  I would be satisfied if the authors just present this part as teachers' opinions about SatelliteSkill5 App, and not as evaluation of it."

This also corresponds with comments from Reviewer 1, and to answer each of these we have provided additional information on this limitation at the very beginning of the results section, highlighting that we are using an informal rather than instrumental evaluation in Section 4 and renaming Section 4.1 and 4.2 .We have also removed reference to Quant/Qual in the abstract to demonstrate that this is teacher opinion used as a proxy.

"Comments on systems limitations would also be nice to mention and elaborate more in the discussion section"

We have also expanded on the technical limitations of the augmented reality experience by adding a new section 5.3.

Reviewer 3 Report

The article responds to the "Scope" of the journal.

It is suggested to incorporate references on augmented reality; the theoretical approach to this concept is insufficient. 

Author Response

Thank you very much for the review and useful comment. Our edits are as follows:

"It is suggested to incorporate references on augmented reality; the theoretical approach to this concept is insufficient. "

We have added a number of new Augmented Reality references to answer this comment  from special issues in IJGI on “Public Participation in 2021: New Forms, New Modes, New Questions?”, from “Geovisualization and Map Design”, from an IEEE  journal and an additional paper relating to student comprehension and learning.

Reviewer 4 Report

This manuscript introduces SatelliteSkill5 which is a creative and game-based E-learning approach. Authors have claimed that it is an Augmented Reality (AR) based Remote Sensing teaching tool. This has not been justified in the manuscript accordingly. It is just a useful mobile application to be used by students. A special sub-chapter in Chapter "3. Technical Development of the SatelliteSkill5 App" is required in order to explain the AR features of the tool. How the artificial tags and labels are linked to the real world scenes?

Apart from the technical issues, quality of the English language is excellent. 

1. Introduction

Although the problem definition and objectives are clearly defined in the Introduction section, a large amount of past effort was unfortunately omitted. ISPRS (International Society for Photogrammetry and Remote Sensing) was one of the pioneer organization to address the benefits of E-learning tools in the area of Geomatics and relevant sciences.
ISPRS Workshop on "Tools and Techniques for E-Learning" held in Potsdam in 2005 is just an important example how early ISPRS took actions in this field.
https://www.isprs.org/proceedings/XXXVI/6-W30/

I recommend to mention such events of ISPRS and to cite some relevant articles.

Figures 1, 2 and 3

Resolution of Figure 1 and Figure 2 should be improved.

Table 2

There is not a reference to Table 2 in the text.

3. Technical Development of the SatelliteSkill5 App

Augmented Reality (AR) is an environment where real world scenes are attached by artificial models and labels. It is not clear to me how the location and orientation is achieved. Can students have satellite data special to their location or not?

Author Response

Thank you very much for the review and useful comments. Our edits are as follows:

Previous work of ISPRS - namely Potsdam and other events in Education/Capacity Building.

Thanks for drawing this early work to our attention, we have added a number of relevant (RS/Photogrammetry) citations from early ISPRS initiatives such as those you have suggested in 2005, and also identified three similar ISPRS publications from recent years to demonstrate the breadth of ISPRS education and capacity building initiatives already under way. 

Resolution of Figure 1 and Figure 2 should be improved.

We have taken comments on board, and have also used this opportunity to increase the size and improve the resolution of Figures 1-3. We have removed panes in these figures and the captions have been reduced accordingly to also address a comment from Reviewer 1. We hope this has improved the readability of these figures. We have also reduced the caption content and moved elements of this into the text.

Table 2 - There is not a reference to Table 2 in the text.

Thank you for spotting this – we have added this now in Section 2.4, and has lead us to find and correct Figure caption errors for a duplicate ‘Fig 5’ in Section 4 and have updated its number and subsequent images accordingly.

Augmented Reality (AR) is an environment where real world scenes are attached by artificial models and labels. It is not clear to me how the location and orientation is achieved. Can students have satellite data special to their location or not?

We have included additional information ion how the challenges relate to the users real-world environment in a new sub-section, 3.4. We have also clarified in Section 3.1 that these are curated datasets, and therefore not dynamic and updated depending on the users country. Interestingly we looked at some real-time data/location information in earlier iterations, but data privacy and other child-protection issues discouraged us and moved us towards the ‘curated’ model. We have also referred to this in our revisions for the new Section 5.4 on ‘future work’