Digital Routes in Greek History’s Paths

Round 1

Reviewer 1 Report

The proposal is very innovative and of great interest in the use of VR for Cultural Heritage dissemination. However, as an initial presentation of a project under development, and not a final evaluation of the functionality and operability of certain virtual environment, I consider that it would be important to present in more detail the data related to the methodology and organizational structure of the project itself. In the following lines I make certain suggestions:

Introduction:

At what point of development is the RoTH Project? What execution deadlines have you proposed and when is the platform expected to be operational?

Which agents have been involved in the project and what is the general task of each of them?

Materials and Methods:

What criteria has been followed to choose the sites, monuments and places included in the platform?

What games and/or heritage documentation works have served as a reference or source of inspiration for the style of 3D modelling and photorealistic renderings?

Where the platform will be hosted and how it is expected to be accessed.

Discussion:

How have the target groups of potential players been studied?

How do you plan to test and evaluate your player experience and its educational impact?

Formal suggestions:

Sentences from 100 to 108: These two paragraphs present certain redundancy of terms and adjectives that make reading difficult. Synthesizing and summarizing the description would help a better reading comprehension.

Table 1: Since there are cells that start the text in lowercase while most do it in uppercase, unifying styles is recommended.

Sentences 132 and 136: avoiding the repetition of “content” by resorting to a synonym is suggested.

Phrases 144-147 and 212-216: Both paragraphs refer to the player. In the first one, it is referred to him as masculine and the second has been wrote in gender-inclusive terms. It is convenient to unify styles and opt for a single formula for all the text.

242: platform instead of “platform”

252: Following the structure of the previous figures, it would be convenient to complete figure 4. It could be done with the representation of another of the elements listed, for example the navigation map.

257: It would be more enriching if a visualization of some of the exhibited pieces were shown in detail instead of two points of view of the exhibition hall.

326: “as well” instead “9as well”

Comments for author File: Comments.pdf

Author Response

Authors would like to thank reviewers for their comments which significantly help to further improve the quality of the proposed paper. Comments have been clustered according to the section where changes have been made, while authors are more than happy to provide any further information or expand / change what is needed according to reviewers’ proposals.

Abstract. Abstract has been rewritten in order to include all reviewers’ comments. Most significantly there have been excluded from abstract mentions on (a) augmented reality app and (b) collection of users’ comments. These two features will most probably been developed in a later stage, when the whole app will be available to end users. The following comment have been taken into account:

• In abstract we can find reference to an augmented reality app, but in the text as in the keywords, there are not references to an "augmented" fruition, that is the addition of virtual and digital content to a real and physical objects, but we find only refers to a virtual app, such as the web platform and the virtual reality app, we suggest a change in the term, or a better specification regarding the use of the word “augmented”. (Reviewer 3)
• The abstract discusses the collection and use of user comments, but this element does not seem to have made it into the paper. This should either be expanded on in the paper or removed from the abstract. (Reviewer 2)

Introduction. In the introduction section the following changes have been made. Most important of them can be viewed in the paper’s new edition with trach changes.

• At what point of development is the RoGH Project? What execution deadlines have you proposed and when is the platform expected to be operational? (Reviewer 1) It is not clear from the paper whether the paper describes a design concept or a completed piece of work, although the overall impression is that this is a design concept and has not yet been implemented or tested. (Reviewer 2)
• Which agents have been involved in the project and what is the general task of each of them? (Reviewer 1)

Proposed addition in the paper: RoGH is an under – development research project aiming to deliver an application of Technology Readiness Level 6 (TRL-6), following the classification used by HORIZON 2020, by the mid of 2022. The proposed level of readiness implies that the application consists of technology capable to be demonstrated in relevant environment and not a sys-tem complete and qualified or an actual system proven in operational environment. Even though the application is not tested and fully operational significant advances have been made in terms of software development, historical content and 3D models making it func-tional for demonstration.

The whole project is developed by three (3) agents including: (a) Cultural Technology Lab of the Department of Informatics and Telecommunications of National and Kapodis-trian University of Athens (CulTech / EKPA), (b) National Hellenic Research Foundation (NHRF) and (c) Content Management in Culture P.C. (Comic) a private, software devel-opment company specialized in the field of culture. EKPA and Comic contribute in the field of the application’s development, while NHRF provides rich historical content from its sources (e.g. from its own researches, maps, documents, bibliography e.t.c.) as well as historical advice and architectural drawings for 3-D modelling.

• The prediction in l.53-4 is interesting, but would equally benefit from a little unpacking - why does Uricchio suggest this? Do you agree? This may become easier to clarify with reference to the use case in this article. (Reviewer 2)

Proposed addition in the paper: … Furthermore, Uricchio (2005) predicted that the exploration of the relationship between history and games will become the future research questions that will preoccupy scholars in the near future (Chapman, Foka & Westin, 2017). Since then, the field of historical game studies become a distinct research area (part of the game studies area) gathering its own research interest, bibliography directly related with how history should be interpreted – represented – played with (Chapman, Foka & Westin, 2017). The proposed use – case in current article comes to contribute towards this research area by implementing the aspect of Virtual Reality when teaching / presenting history in a gamified and more interactive way.

• In the introduction there is a proper excursus of the relationship between history and videogames or serious games, but, since the paper is about the creation of an open database, even for educational purposes, there is no mention of any international regulatory reference to the importance of the scientific veracity of the source, such as the London Charter and the Seville Charter. It is advisable to emphasize these aspects better, both in the introduction and in the text, only in line 301 there is a reference to a data control system but without specifying anything else. (Reviewer 3)

Proposed addition in the paper: … Moreover, significant interest and development have been posed on the importance of the scientific veracity of the source, where there is an ongoing international regulation process (Carrillo et. al., 2013). In 2006, the London Charter (http://www.londoncharter.org) was conceived aiming to set a series of recommendations and specific guidance related with the Computer-based Visualisation of Cultural Heritage. The whole process was based on extensive consultation alongside with significant efforts for establishing best practices – wide accepted and under international consensus - in heritage visualization processes. The Principles of Seville followed as a mean to increase applicability to the London Charter specifically to archaeological heritage. Proposed principles include aspects nowadays implemented to all projects related with visualization of archaeological heritage, such as:

1. Interdisciplinarity: professionals from different branches of knowledge.
2. Purpose: clear purpose of the project’s goals.
3. Complementarity: digital archaeological heritsage as a complementary and not as an alternative tool to more traditional forms of reaching cultural her-itage.
4. Authenticity: provide a clear distinguish what is real, genuine or authentic from what is not.
5. Historical rigour: historical and archaeological documentation should be provided alongside with digital visualizations.
6. Efficiency: achieving appropriate economic and technological sustainability.
7. Scientific transparency: provide results ready to be confirmed or refuted by other experts in the field.
8. Training and evaluation: Virtual archaeology has its own specific language and techniques, requiring training among community members and constant evaluation.

• The focus on digital games is not thoroughly explained in the introduction. I suggest that the authors clearly establish the relevance of this topic early in the introduction. (Reviewer 2)
  • Authors believe that a more expanded analysis on digital games may mislead readers from the platform’s educational and cultural characteristics. We would be glad to follow reviewer’s comment in case his opinion differs.
• The assertion in line 40 would benefit from either a reference or further unpacking (particularly developed in what way?) (Reviewer 2)
  • Reference has been added.

Materials and Methods:

• What criteria has been followed to choose the sites, monuments and places included in the platform? (Reviewer 1)

Proposed addition in the paper: The experienced members of National Hellenic Research Foundation (NHRF) involved in choosing the sites, monuments and places included in the platform. The criteria used for the proposed selection included availability of rich and historically accurate content / in-formation, according to their archives. It should be taken into account that NHRF provided a variety of content, from text and photos, up to architectural drawings and depictions of engravings. Most significantly, the team members took active part in choosing the historical buildings – monuments that were recreated in VR technology and offered their expertise in order to implement historical accurate 3-D modeling.

• What games and/or heritage documentation works have served as a reference or source of inspiration for the style of 3D modelling and photorealistic renderings? (Reviewer 1)
  • National Hellenic Research Foundation provides rich historical content from its sources (e.g. from its own researches, maps, documents, bibliography e.t.c.) as well as historical advice and architectural draw-ings for 3-D modelling.
• Where the platform will be hosted and how it is expected to be accessed. (Reviewer 1)

Proposed addition in the paper: The proposed application is currently (as long as the development is running) hosted in a Virtual Machine owned by National and Kapodistrian University of Athens (CulTech / EKPA) while it is planned that end – users will have access to the application (a) either by downloading from OCULUS store and STEAM store, (b) either by directly installed to VR headsets of the client when the product will have been implemented by a school or a museum.

• The open nature of the infrastructure, alongside the intention that submission will be open to all following a process of evaluation, is also very welcome. That said, it is not immediately clear whether the submitted resources (e.g. the historical works and other resources) will be published for others to use in their own games, guides, etc, so licensing and access should be clarified. (Reviewer 2)
  • The submitted resources (e.g. the historical works and other resources) will not be published for others to use in their own games, guides, etc.
• What device types does the user require to make use of this application, and how will the choice of device influence the user experience? (Reviewer 2)
• What technical platform is it written in (Unity?) and why? (Reviewer 2)

Proposed addition in the paper for the above 2 comments: The platform is written by using Unity (Figure 1) as a result of its multiplatform ca-pability, the exceptional rendering skills that Unity has and the well-structured commu-nity willing to support developers and projects. Moreover, Unity has been used in a series of previous projects leading to the existence of significant knowledge skills among the members of the developers’ team. The whole development put emphasis on providing us-ers the capability of using almost any VR device types to make use of this application in-cluding: HTC VIVE, HTC COSMOS, OCULUS QUEST 2, OCULUS QUEST, OCULUS RIFT, OCULUS RIFT S, VALVE INDEX, VARJO VR-3.

• Do users perceive it as a game? (Reviewer 2)
  • RoGH is an application that has many characterizes resembling, “serious games” even if it does not fall exactly into this category. Moreover, the platform includes a gamified digital application / experience leading users close to what it is described as a “game feeling”.
• What is(are) the primary gameplay loop(s)? From the paper as it stands I understand that players in effect visit locations and learn about them in order to collect trophies, which they can then virtually '3d print' and display. (Reviewer 2)

Proposed addition in the paper: Overall, the gameplay loop includes (a) players visiting locations and learning  about them in order to collect trophies, which they can then virtually 3-D print' and dis-play and in addition (b) players collecting points by gaining / finishing achievements / campaigns, exchanged with 3-D souvenirs for their collection.

• The effective system of gamification is interesting, would be useful to have a scheme of gameplay, clarifying better the system of questions, the generation of new ones, and the system of satisfaction (line 154). (Reviewer 3)
  • A new Figure has been included to provide an instant view. Please take into consideration that all figures provided have been directly taken via the VR mask.
• It is advisable to better clarify, through a scheme or the integration of fig.1 the different apps made, it is not clear the distinction of the app nomenclatures "VR RoGH Chronos", "Digital Routes", "Content Access lab". As an alternative, if similar aspects and features were already explained in other papers, please clearly refer to them. (reviewer 3)

Proposed addition in the paper: The Digital Routes" project’s overall design is portrayed in Figure 1. In the upper lay-er end-users and content creators are presented, while in the middle level the two main applications are presented alongside with the technologies used for their development, namely: (a) RoGH platform and (b) the 3-D Content Repository, Their interaction makes feasible to provide to end-users the VR chronology (named VR RoGH Chronos by the Greek word “chronos” meaning time) where they can where they can tour in greek history by choosing place or time / chronological period.

As far as "Content Access lab" it is not described in the current paragraph because it has been already mentioned in a previous one that it is referring to the name of an internal application in the platform.

Discussion.

• How have the target groups of potential players been studied? (Reviewer 1)
• How do you plan to test and evaluate your player experience and its educational impact? (Reviewer 1) How have you evaluated (or how will you evaluate) the product? It is mentioned that it will be entertaining and educational, both of which can be evaluated to some extent. (Reviewer 2)
• It would be helpful to give a very clear idea of plans for future work, including implementation, requirements and evaluation, and to specify why you have decided to publish at this stage. (Reviewer 2)

Proposed addition in the paper: The above – mentioned target groups of ed users have been recognized and accepted by the Greek agency called General Secretariat for Research and Innovation (GSRI), which merely funded the project as a Smart Specialization Strategy Priority in the area of “Cultural and Creative industries”. As a result of the on – going nature of the project an evaluation of the proposed platform and its operation is not evaluated yet. The evaluation will be conducted by external experts coming from the field of the VR in education, while the agents taking part in the project have decided to use the Greek educational system (e,g, upper-secondary education and university’s students) as the testing tube for the applications as a whole. After testing and evaluation, as well as after forthcoming adds-on, the platform will get available to mass audience via Oculus store and STEAM store. The features that most probably will be added in first level (even before testing and evaluation) include areas of interest such as:

• Requirements:
  • Multiuser functionality,
  • Virtual avatar assistance and
  • Analytics for users’ behavior when using the platform.
• Implementation:
  • Implement new requirements,
  • Design and integrate extra historical sites and
  • Implementing new content (including 3-D models).

Formal Suggestions. The following comments have been followed and significant changes have been conducted:

• Sentences from 100 to 108: These two paragraphs present certain redundancy of terms and adjectives that make reading difficult. Synthesizing and summarizing the description would help a better reading comprehension. (Reviewer 1)
• There are a number of typos and minor grammatical issues, for example l. 47 is probably missing a 'which', as in 'Kurt Squire's dissertation, which presents' (Reviewer 2)
• Table 1: Since there are cells that start the text in lowercase while most do it in uppercase, unifying styles is recommended. (Reviewer 1)
• Line 131 refers to 'vary', which should probably read 'variety' (Reviewer 2)
• Sentences 132 and 136: avoiding the repetition of “content” by resorting to a synonym is suggested. (Reviewer 1)
• Phrases 144-147 and 212-216: Both paragraphs refer to the player. In the first one, it is referred to him as masculine and the second has been wrote in gender-inclusive terms. It is convenient to unify styles and opt for a single formula for all the text. (Reviewer 1). The user is referred to as 'he' or 'him' in some cases, 'he/she' in some cases, but 'they' or 'them' in others; it may be worth considering standardising on a particular approach throughout, perhaps the neutral 'they'. I suggest a thorough proofread throughout. (Reviewer 2)
• In Lines 169-170 there is a close repetition of the term "experiential experience.... experience". (Reviewer 3)
• 242: platform instead of “platform”. (Reviewer 1)
• 252: Following the structure of the previous figures, it would be convenient to complete figure 4. It could be done with the representation of another of the elements listed, for example the navigation map. (Reviewer 1)
  • Figure 4 has been enriched.
• 257: It would be more enriching if a visualization of some of the exhibited pieces were shown in detail instead of two points of view of the exhibition hall. (Reviewer 1)
  • A case of exhibited piece is provided. If enrichment with more pieces is needed we are happy to provide.
• 326: “as well” instead “9as well”. (Reviewer 1 and 2)

Author Response File: Author Response.pdf

Reviewer 2 Report

I thank the authors for the opportunity to review this fascinating paper, which describes work completed in a very worthwhile area with significant potential. The open nature of the infrastructure, alongside the intention that submission will be open to all following a process of evaluation, is also very welcome. That said, it is not immediately clear whether the submitted resources (e.g. the historical works and other resources) will be published for others to use in their own games, guides, etc, so licensing and access should be clarified.

The focus on digital games is not thoroughly explained in the introduction. I suggest that the authors clearly establish the relevance of this topic early in the introduction. The assertion in line 40 would benefit from either a reference or further unpacking (particularly developed in what way?)

The prediction in l.53-4 is interesting, but would equally benefit from a little unpacking - why does Uricchio suggest this? Do you agree? This may become easier to clarify with reference to the use case in this article.

There are a number of typos and minor grammatical issues, for example l. 47 is probably missing a 'which', as in 'Kurt Squire's dissertation, which presents'. Line 131 refers to 'vary', which should probably read 'variety'.  Line 326 has a stray 9 and a missing 'as'. The user is referred to as 'he' or 'him' in some cases, 'he/she' in some cases, but 'they' or 'them' in others; it may be worth considering standardising on a particular approach throughout, perhaps the neutral 'they'. I suggest a thorough proofread throughout.

In l.89 VR is discussed. It may be useful to discuss both VR and augmented reality at this stage, clearly define both and identify which is more suitable for your use case. It may not initially be obvious to the reader whether you are introducing an AR application that can be used by real-world visitors to tourist sites or a VR application accessible from any location. This is particularly the case since there have been a number of AR approaches to this problem, and the abstract explicitly describes an AR application, whilst the screenshots suggest virtual reality. I recommend further developing the literature review in this area. If it is the case that you intend to provide both AR and VR modes, as section 3.3 indicates, then it would be helpful to explain this and discuss the effect on the gameplay and user experience. This also sounds like a difficult and time-consuming problem, so it would be helpful to explain how you have kept (or intend to keep) this task manageable.

The concept looks excellent, but the user experience could be expressed more clearly. For example, questions that came to my mind while reading this article included:

• What device types does the user require to make use of this application, and how will the choice of device influence the user experience?
• What technical platform is it written in (Unity?) and why?
• Do users perceive it as a game?
• If/when this is used in AR, what is the gameplay? For example, is it presented as a 'scavenger hunt'?
• What is(are) the primary gameplay loop(s)? From the paper as it stands I understand that players in effect visit locations and learn about them in order to collect trophies, which they can then virtually '3d print' and display.
• How have you evaluated (or how will you evaluate) the product? It is mentioned that it will be entertaining and educational, both of which can be evaluated to some extent.

The abstract discusses the collection and use of user comments, but this element does not seem to have made it into the paper. This should either be expanded on in the paper or removed from the abstract.

Finally, it is not clear from the paper whether the paper describes a design concept or a completed piece of work, although the overall impression is that this is a design concept and has not yet been implemented or tested. If so it would be helpful to give a very clear idea of plans for future work, including implementation, requirements and evaluation, and to specify why you have decided to publish at this stage. If the work is completed, or nearly so, it would be helpful to present some outcomes and evaluation, even if these are preliminary.

Author Response

Authors would like to thank reviewers for their comments which significantly help to further improve the quality of the proposed paper. Comments have been clustered according to the section where changes have been made, while authors are more than happy to provide any further information or expand / change what is needed according to reviewers’ proposals.

Abstract. Abstract has been rewritten in order to include all reviewers’ comments. Most significantly there have been excluded from abstract mentions on (a) augmented reality app and (b) collection of users’ comments. These two features will most probably been developed in a later stage, when the whole app will be available to end users. The following comment have been taken into account:

• In abstract we can find reference to an augmented reality app, but in the text as in the keywords, there are not references to an "augmented" fruition, that is the addition of virtual and digital content to a real and physical objects, but we find only refers to a virtual app, such as the web platform and the virtual reality app, we suggest a change in the term, or a better specification regarding the use of the word “augmented”. (Reviewer 3)
• The abstract discusses the collection and use of user comments, but this element does not seem to have made it into the paper. This should either be expanded on in the paper or removed from the abstract. (Reviewer 2)

Introduction. In the introduction section the following changes have been made. Most important of them can be viewed in the paper’s new edition with trach changes.

• At what point of development is the RoGH Project? What execution deadlines have you proposed and when is the platform expected to be operational? (Reviewer 1) It is not clear from the paper whether the paper describes a design concept or a completed piece of work, although the overall impression is that this is a design concept and has not yet been implemented or tested. (Reviewer 2)
• Which agents have been involved in the project and what is the general task of each of them? (Reviewer 1)

Proposed addition in the paper: RoGH is an under – development research project aiming to deliver an application of Technology Readiness Level 6 (TRL-6), following the classification used by HORIZON 2020, by the mid of 2022. The proposed level of readiness implies that the application consists of technology capable to be demonstrated in relevant environment and not a sys-tem complete and qualified or an actual system proven in operational environment. Even though the application is not tested and fully operational significant advances have been made in terms of software development, historical content and 3D models making it func-tional for demonstration.

The whole project is developed by three (3) agents including: (a) Cultural Technology Lab of the Department of Informatics and Telecommunications of National and Kapodis-trian University of Athens (CulTech / EKPA), (b) National Hellenic Research Foundation (NHRF) and (c) Content Management in Culture P.C. (Comic) a private, software devel-opment company specialized in the field of culture. EKPA and Comic contribute in the field of the application’s development, while NHRF provides rich historical content from its sources (e.g. from its own researches, maps, documents, bibliography e.t.c.) as well as historical advice and architectural drawings for 3-D modelling.

• The prediction in l.53-4 is interesting, but would equally benefit from a little unpacking - why does Uricchio suggest this? Do you agree? This may become easier to clarify with reference to the use case in this article. (Reviewer 2)

Proposed addition in the paper: … Furthermore, Uricchio (2005) predicted that the exploration of the relationship between history and games will become the future research questions that will preoccupy scholars in the near future (Chapman, Foka & Westin, 2017). Since then, the field of historical game studies become a distinct research area (part of the game studies area) gathering its own research interest, bibliography directly related with how history should be interpreted – represented – played with (Chapman, Foka & Westin, 2017). The proposed use – case in current article comes to contribute towards this research area by implementing the aspect of Virtual Reality when teaching / presenting history in a gamified and more interactive way.

• In the introduction there is a proper excursus of the relationship between history and videogames or serious games, but, since the paper is about the creation of an open database, even for educational purposes, there is no mention of any international regulatory reference to the importance of the scientific veracity of the source, such as the London Charter and the Seville Charter. It is advisable to emphasize these aspects better, both in the introduction and in the text, only in line 301 there is a reference to a data control system but without specifying anything else. (Reviewer 3)

Proposed addition in the paper: … Moreover, significant interest and development have been posed on the importance of the scientific veracity of the source, where there is an ongoing international regulation process (Carrillo et. al., 2013). In 2006, the London Charter (http://www.londoncharter.org) was conceived aiming to set a series of recommendations and specific guidance related with the Computer-based Visualisation of Cultural Heritage. The whole process was based on extensive consultation alongside with significant efforts for establishing best practices – wide accepted and under international consensus - in heritage visualization processes. The Principles of Seville followed as a mean to increase applicability to the London Charter specifically to archaeological heritage. Proposed principles include aspects nowadays implemented to all projects related with visualization of archaeological heritage, such as:

1. Interdisciplinarity: professionals from different branches of knowledge.
2. Purpose: clear purpose of the project’s goals.
3. Complementarity: digital archaeological heritsage as a complementary and not as an alternative tool to more traditional forms of reaching cultural her-itage.
4. Authenticity: provide a clear distinguish what is real, genuine or authentic from what is not.
5. Historical rigour: historical and archaeological documentation should be provided alongside with digital visualizations.
6. Efficiency: achieving appropriate economic and technological sustainability.
7. Scientific transparency: provide results ready to be confirmed or refuted by other experts in the field.
8. Training and evaluation: Virtual archaeology has its own specific language and techniques, requiring training among community members and constant evaluation.

• The focus on digital games is not thoroughly explained in the introduction. I suggest that the authors clearly establish the relevance of this topic early in the introduction. (Reviewer 2)
  • Authors believe that a more expanded analysis on digital games may mislead readers from the platform’s educational and cultural characteristics. We would be glad to follow reviewer’s comment in case his opinion differs.
• The assertion in line 40 would benefit from either a reference or further unpacking (particularly developed in what way?) (Reviewer 2)
  • Reference has been added.

Materials and Methods:

• What criteria has been followed to choose the sites, monuments and places included in the platform? (Reviewer 1)

Proposed addition in the paper: The experienced members of National Hellenic Research Foundation (NHRF) involved in choosing the sites, monuments and places included in the platform. The criteria used for the proposed selection included availability of rich and historically accurate content / in-formation, according to their archives. It should be taken into account that NHRF provided a variety of content, from text and photos, up to architectural drawings and depictions of engravings. Most significantly, the team members took active part in choosing the historical buildings – monuments that were recreated in VR technology and offered their expertise in order to implement historical accurate 3-D modeling.

• What games and/or heritage documentation works have served as a reference or source of inspiration for the style of 3D modelling and photorealistic renderings? (Reviewer 1)
  • National Hellenic Research Foundation provides rich historical content from its sources (e.g. from its own researches, maps, documents, bibliography e.t.c.) as well as historical advice and architectural draw-ings for 3-D modelling.
• Where the platform will be hosted and how it is expected to be accessed. (Reviewer 1)

Proposed addition in the paper: The proposed application is currently (as long as the development is running) hosted in a Virtual Machine owned by National and Kapodistrian University of Athens (CulTech / EKPA) while it is planned that end – users will have access to the application (a) either by downloading from OCULUS store and STEAM store, (b) either by directly installed to VR headsets of the client when the product will have been implemented by a school or a museum.

• The open nature of the infrastructure, alongside the intention that submission will be open to all following a process of evaluation, is also very welcome. That said, it is not immediately clear whether the submitted resources (e.g. the historical works and other resources) will be published for others to use in their own games, guides, etc, so licensing and access should be clarified. (Reviewer 2)
  • The submitted resources (e.g. the historical works and other resources) will not be published for others to use in their own games, guides, etc.
• What device types does the user require to make use of this application, and how will the choice of device influence the user experience? (Reviewer 2)
• What technical platform is it written in (Unity?) and why? (Reviewer 2)

Proposed addition in the paper for the above 2 comments: The platform is written by using Unity (Figure 1) as a result of its multiplatform ca-pability, the exceptional rendering skills that Unity has and the well-structured commu-nity willing to support developers and projects. Moreover, Unity has been used in a series of previous projects leading to the existence of significant knowledge skills among the members of the developers’ team. The whole development put emphasis on providing us-ers the capability of using almost any VR device types to make use of this application in-cluding: HTC VIVE, HTC COSMOS, OCULUS QUEST 2, OCULUS QUEST, OCULUS RIFT, OCULUS RIFT S, VALVE INDEX, VARJO VR-3.

• Do users perceive it as a game? (Reviewer 2)
  • RoGH is an application that has many characterizes resembling, “serious games” even if it does not fall exactly into this category. Moreover, the platform includes a gamified digital application / experience leading users close to what it is described as a “game feeling”.
• What is(are) the primary gameplay loop(s)? From the paper as it stands I understand that players in effect visit locations and learn about them in order to collect trophies, which they can then virtually '3d print' and display. (Reviewer 2)

Proposed addition in the paper: Overall, the gameplay loop includes (a) players visiting locations and learning  about them in order to collect trophies, which they can then virtually 3-D print' and dis-play and in addition (b) players collecting points by gaining / finishing achievements / campaigns, exchanged with 3-D souvenirs for their collection.

• The effective system of gamification is interesting, would be useful to have a scheme of gameplay, clarifying better the system of questions, the generation of new ones, and the system of satisfaction (line 154). (Reviewer 3)
  • A new Figure has been included to provide an instant view. Please take into consideration that all figures provided have been directly taken via the VR mask.
• It is advisable to better clarify, through a scheme or the integration of fig.1 the different apps made, it is not clear the distinction of the app nomenclatures "VR RoGH Chronos", "Digital Routes", "Content Access lab". As an alternative, if similar aspects and features were already explained in other papers, please clearly refer to them. (reviewer 3)

Proposed addition in the paper: The Digital Routes" project’s overall design is portrayed in Figure 1. In the upper lay-er end-users and content creators are presented, while in the middle level the two main applications are presented alongside with the technologies used for their development, namely: (a) RoGH platform and (b) the 3-D Content Repository, Their interaction makes feasible to provide to end-users the VR chronology (named VR RoGH Chronos by the Greek word “chronos” meaning time) where they can where they can tour in greek history by choosing place or time / chronological period.

As far as "Content Access lab" it is not described in the current paragraph because it has been already mentioned in a previous one that it is referring to the name of an internal application in the platform.

Discussion.

• How have the target groups of potential players been studied? (Reviewer 1)
• How do you plan to test and evaluate your player experience and its educational impact? (Reviewer 1) How have you evaluated (or how will you evaluate) the product? It is mentioned that it will be entertaining and educational, both of which can be evaluated to some extent. (Reviewer 2)
• It would be helpful to give a very clear idea of plans for future work, including implementation, requirements and evaluation, and to specify why you have decided to publish at this stage. (Reviewer 2)

Proposed addition in the paper: The above – mentioned target groups of ed users have been recognized and accepted by the Greek agency called General Secretariat for Research and Innovation (GSRI), which merely funded the project as a Smart Specialization Strategy Priority in the area of “Cultural and Creative industries”. As a result of the on – going nature of the project an evaluation of the proposed platform and its operation is not evaluated yet. The evaluation will be conducted by external experts coming from the field of the VR in education, while the agents taking part in the project have decided to use the Greek educational system (e,g, upper-secondary education and university’s students) as the testing tube for the applications as a whole. After testing and evaluation, as well as after forthcoming adds-on, the platform will get available to mass audience via Oculus store and STEAM store. The features that most probably will be added in first level (even before testing and evaluation) include areas of interest such as:

• Requirements:
  • Multiuser functionality,
  • Virtual avatar assistance and
  • Analytics for users’ behavior when using the platform.
• Implementation:
  • Implement new requirements,
  • Design and integrate extra historical sites and
  • Implementing new content (including 3-D models).

Formal Suggestions. The following comments have been followed and significant changes have been conducted:

• Sentences from 100 to 108: These two paragraphs present certain redundancy of terms and adjectives that make reading difficult. Synthesizing and summarizing the description would help a better reading comprehension. (Reviewer 1)
• There are a number of typos and minor grammatical issues, for example l. 47 is probably missing a 'which', as in 'Kurt Squire's dissertation, which presents' (Reviewer 2)
• Table 1: Since there are cells that start the text in lowercase while most do it in uppercase, unifying styles is recommended. (Reviewer 1)
• Line 131 refers to 'vary', which should probably read 'variety' (Reviewer 2)
• Sentences 132 and 136: avoiding the repetition of “content” by resorting to a synonym is suggested. (Reviewer 1)
• Phrases 144-147 and 212-216: Both paragraphs refer to the player. In the first one, it is referred to him as masculine and the second has been wrote in gender-inclusive terms. It is convenient to unify styles and opt for a single formula for all the text. (Reviewer 1). The user is referred to as 'he' or 'him' in some cases, 'he/she' in some cases, but 'they' or 'them' in others; it may be worth considering standardising on a particular approach throughout, perhaps the neutral 'they'. I suggest a thorough proofread throughout. (Reviewer 2)
• In Lines 169-170 there is a close repetition of the term "experiential experience.... experience". (Reviewer 3)
• 242: platform instead of “platform”. (Reviewer 1)
• 252: Following the structure of the previous figures, it would be convenient to complete figure 4. It could be done with the representation of another of the elements listed, for example the navigation map. (Reviewer 1)
  • Figure 4 has been enriched.
• 257: It would be more enriching if a visualization of some of the exhibited pieces were shown in detail instead of two points of view of the exhibition hall. (Reviewer 1)
  • A case of exhibited piece is provided. If enrichment with more pieces is needed we are happy to provide.
• 326: “as well” instead “9as well”. (Reviewer 1 and 2)

Author Response File: Author Response.pdf

Reviewer 3 Report

This paper is about an interesting work concerning the dissemination and fruition of the Greek archaeological and historical heritage from antiquity to the modern era, through the creation of a complex open web platform and the build of a virtual reality app, usable through virtual reality headset (cardboard).  

The platform is a real repository of all kinds of multimedia content such as video, audio, text, 3d models, and in addition it is open, so anyone can access to view and to upload new contents. 

About the VR app, the user is free to create their own way of use according to time and space criteria, in order to solve the unsolved quests proposed. 

1. In abstract we can find reference to an augmented reality app, but in the text as in the keywords, there are not references to an "augmented" fruition, that is the addition of virtual and digital content to a real and physical objects, but we find only refers to a virtual app, such as the web platform and the virtual reality app, we suggest a change in the term, or a better specification regarding the use of the word “augmented”.
2. In the introduction there is a proper excursus of the relationship between history and videogames or seriousgames, but, since the paper is about the creation of an open database, even for educational purposes, there is no mention of any international regulatory reference to the importance of the scientific veracity of the source, such as the London Charter and the Seville Charter. It is advisable to emphasize these aspects better, both in the introduction and in the text, only in line 301 there is a reference to a data control system but without specifying anything else.
3. The effective system of gamification is interesting, would be useful to have a scheme of gameplay, clarifying better the system of questions, the generation of new ones, and the system of satisfaction (line 154).
4. In Lines 169-170 there is a close repetition of the term "experiential experience.... experience".
5. The conclusion paragraph simply proposes a summary of the goals of the general project and vague future implementations, placed on 5 general points, but does not refer to studies of user interface, or communicative effectiveness through user analysis, which are essential for this type of application.
6. It is advisable to better clarify, through a scheme or the integration of fig.1 the different apps made, it is not clear the distinction of the app nomenclatures "VR RoGH Chronos", "Digital Routes", "Content Access lab". As an alternative, if similar aspects and features were already explained in other papers, please clearly refer to them.

Comments for author File: Comments.pdf

Author Response

Authors would like to thank reviewers for their comments which significantly help to further improve the quality of the proposed paper. Comments have been clustered according to the section where changes have been made, while authors are more than happy to provide any further information or expand / change what is needed according to reviewers’ proposals.

Abstract. Abstract has been rewritten in order to include all reviewers’ comments. Most significantly there have been excluded from abstract mentions on (a) augmented reality app and (b) collection of users’ comments. These two features will most probably been developed in a later stage, when the whole app will be available to end users. The following comment have been taken into account:

• In abstract we can find reference to an augmented reality app, but in the text as in the keywords, there are not references to an "augmented" fruition, that is the addition of virtual and digital content to a real and physical objects, but we find only refers to a virtual app, such as the web platform and the virtual reality app, we suggest a change in the term, or a better specification regarding the use of the word “augmented”. (Reviewer 3)
• The abstract discusses the collection and use of user comments, but this element does not seem to have made it into the paper. This should either be expanded on in the paper or removed from the abstract. (Reviewer 2)

Introduction. In the introduction section the following changes have been made. Most important of them can be viewed in the paper’s new edition with trach changes.

• At what point of development is the RoGH Project? What execution deadlines have you proposed and when is the platform expected to be operational? (Reviewer 1) It is not clear from the paper whether the paper describes a design concept or a completed piece of work, although the overall impression is that this is a design concept and has not yet been implemented or tested. (Reviewer 2)
• Which agents have been involved in the project and what is the general task of each of them? (Reviewer 1)

Proposed addition in the paper: RoGH is an under – development research project aiming to deliver an application of Technology Readiness Level 6 (TRL-6), following the classification used by HORIZON 2020, by the mid of 2022. The proposed level of readiness implies that the application consists of technology capable to be demonstrated in relevant environment and not a sys-tem complete and qualified or an actual system proven in operational environment. Even though the application is not tested and fully operational significant advances have been made in terms of software development, historical content and 3D models making it func-tional for demonstration.

The whole project is developed by three (3) agents including: (a) Cultural Technology Lab of the Department of Informatics and Telecommunications of National and Kapodis-trian University of Athens (CulTech / EKPA), (b) National Hellenic Research Foundation (NHRF) and (c) Content Management in Culture P.C. (Comic) a private, software devel-opment company specialized in the field of culture. EKPA and Comic contribute in the field of the application’s development, while NHRF provides rich historical content from its sources (e.g. from its own researches, maps, documents, bibliography e.t.c.) as well as historical advice and architectural drawings for 3-D modelling.

• The prediction in l.53-4 is interesting, but would equally benefit from a little unpacking - why does Uricchio suggest this? Do you agree? This may become easier to clarify with reference to the use case in this article. (Reviewer 2)

Proposed addition in the paper: … Furthermore, Uricchio (2005) predicted that the exploration of the relationship between history and games will become the future research questions that will preoccupy scholars in the near future (Chapman, Foka & Westin, 2017). Since then, the field of historical game studies become a distinct research area (part of the game studies area) gathering its own research interest, bibliography directly related with how history should be interpreted – represented – played with (Chapman, Foka & Westin, 2017). The proposed use – case in current article comes to contribute towards this research area by implementing the aspect of Virtual Reality when teaching / presenting history in a gamified and more interactive way.

• In the introduction there is a proper excursus of the relationship between history and videogames or serious games, but, since the paper is about the creation of an open database, even for educational purposes, there is no mention of any international regulatory reference to the importance of the scientific veracity of the source, such as the London Charter and the Seville Charter. It is advisable to emphasize these aspects better, both in the introduction and in the text, only in line 301 there is a reference to a data control system but without specifying anything else. (Reviewer 3)

Proposed addition in the paper: … Moreover, significant interest and development have been posed on the importance of the scientific veracity of the source, where there is an ongoing international regulation process (Carrillo et. al., 2013). In 2006, the London Charter (http://www.londoncharter.org) was conceived aiming to set a series of recommendations and specific guidance related with the Computer-based Visualisation of Cultural Heritage. The whole process was based on extensive consultation alongside with significant efforts for establishing best practices – wide accepted and under international consensus - in heritage visualization processes. The Principles of Seville followed as a mean to increase applicability to the London Charter specifically to archaeological heritage. Proposed principles include aspects nowadays implemented to all projects related with visualization of archaeological heritage, such as:

1. Interdisciplinarity: professionals from different branches of knowledge.
2. Purpose: clear purpose of the project’s goals.
3. Complementarity: digital archaeological heritsage as a complementary and not as an alternative tool to more traditional forms of reaching cultural her-itage.
4. Authenticity: provide a clear distinguish what is real, genuine or authentic from what is not.
5. Historical rigour: historical and archaeological documentation should be provided alongside with digital visualizations.
6. Efficiency: achieving appropriate economic and technological sustainability.
7. Scientific transparency: provide results ready to be confirmed or refuted by other experts in the field.
8. Training and evaluation: Virtual archaeology has its own specific language and techniques, requiring training among community members and constant evaluation.

• The focus on digital games is not thoroughly explained in the introduction. I suggest that the authors clearly establish the relevance of this topic early in the introduction. (Reviewer 2)
  • Authors believe that a more expanded analysis on digital games may mislead readers from the platform’s educational and cultural characteristics. We would be glad to follow reviewer’s comment in case his opinion differs.
• The assertion in line 40 would benefit from either a reference or further unpacking (particularly developed in what way?) (Reviewer 2)
  • Reference has been added.

Materials and Methods:

• What criteria has been followed to choose the sites, monuments and places included in the platform? (Reviewer 1)

Proposed addition in the paper: The experienced members of National Hellenic Research Foundation (NHRF) involved in choosing the sites, monuments and places included in the platform. The criteria used for the proposed selection included availability of rich and historically accurate content / in-formation, according to their archives. It should be taken into account that NHRF provided a variety of content, from text and photos, up to architectural drawings and depictions of engravings. Most significantly, the team members took active part in choosing the historical buildings – monuments that were recreated in VR technology and offered their expertise in order to implement historical accurate 3-D modeling.

• What games and/or heritage documentation works have served as a reference or source of inspiration for the style of 3D modelling and photorealistic renderings? (Reviewer 1)
  • National Hellenic Research Foundation provides rich historical content from its sources (e.g. from its own researches, maps, documents, bibliography e.t.c.) as well as historical advice and architectural draw-ings for 3-D modelling.
• Where the platform will be hosted and how it is expected to be accessed. (Reviewer 1)

Proposed addition in the paper: The proposed application is currently (as long as the development is running) hosted in a Virtual Machine owned by National and Kapodistrian University of Athens (CulTech / EKPA) while it is planned that end – users will have access to the application (a) either by downloading from OCULUS store and STEAM store, (b) either by directly installed to VR headsets of the client when the product will have been implemented by a school or a museum.

• The open nature of the infrastructure, alongside the intention that submission will be open to all following a process of evaluation, is also very welcome. That said, it is not immediately clear whether the submitted resources (e.g. the historical works and other resources) will be published for others to use in their own games, guides, etc, so licensing and access should be clarified. (Reviewer 2)
  • The submitted resources (e.g. the historical works and other resources) will not be published for others to use in their own games, guides, etc.
• What device types does the user require to make use of this application, and how will the choice of device influence the user experience? (Reviewer 2)
• What technical platform is it written in (Unity?) and why? (Reviewer 2)

Proposed addition in the paper for the above 2 comments: The platform is written by using Unity (Figure 1) as a result of its multiplatform ca-pability, the exceptional rendering skills that Unity has and the well-structured commu-nity willing to support developers and projects. Moreover, Unity has been used in a series of previous projects leading to the existence of significant knowledge skills among the members of the developers’ team. The whole development put emphasis on providing us-ers the capability of using almost any VR device types to make use of this application in-cluding: HTC VIVE, HTC COSMOS, OCULUS QUEST 2, OCULUS QUEST, OCULUS RIFT, OCULUS RIFT S, VALVE INDEX, VARJO VR-3.

• Do users perceive it as a game? (Reviewer 2)
  • RoGH is an application that has many characterizes resembling, “serious games” even if it does not fall exactly into this category. Moreover, the platform includes a gamified digital application / experience leading users close to what it is described as a “game feeling”.
• What is(are) the primary gameplay loop(s)? From the paper as it stands I understand that players in effect visit locations and learn about them in order to collect trophies, which they can then virtually '3d print' and display. (Reviewer 2)

Proposed addition in the paper: Overall, the gameplay loop includes (a) players visiting locations and learning  about them in order to collect trophies, which they can then virtually 3-D print' and dis-play and in addition (b) players collecting points by gaining / finishing achievements / campaigns, exchanged with 3-D souvenirs for their collection.

• The effective system of gamification is interesting, would be useful to have a scheme of gameplay, clarifying better the system of questions, the generation of new ones, and the system of satisfaction (line 154). (Reviewer 3)
  • A new Figure has been included to provide an instant view. Please take into consideration that all figures provided have been directly taken via the VR mask.
• It is advisable to better clarify, through a scheme or the integration of fig.1 the different apps made, it is not clear the distinction of the app nomenclatures "VR RoGH Chronos", "Digital Routes", "Content Access lab". As an alternative, if similar aspects and features were already explained in other papers, please clearly refer to them. (reviewer 3)

Proposed addition in the paper: The Digital Routes" project’s overall design is portrayed in Figure 1. In the upper lay-er end-users and content creators are presented, while in the middle level the two main applications are presented alongside with the technologies used for their development, namely: (a) RoGH platform and (b) the 3-D Content Repository, Their interaction makes feasible to provide to end-users the VR chronology (named VR RoGH Chronos by the Greek word “chronos” meaning time) where they can where they can tour in greek history by choosing place or time / chronological period.

As far as "Content Access lab" it is not described in the current paragraph because it has been already mentioned in a previous one that it is referring to the name of an internal application in the platform.

Discussion.

• How have the target groups of potential players been studied? (Reviewer 1)
• How do you plan to test and evaluate your player experience and its educational impact? (Reviewer 1) How have you evaluated (or how will you evaluate) the product? It is mentioned that it will be entertaining and educational, both of which can be evaluated to some extent. (Reviewer 2)
• It would be helpful to give a very clear idea of plans for future work, including implementation, requirements and evaluation, and to specify why you have decided to publish at this stage. (Reviewer 2)

Proposed addition in the paper: The above – mentioned target groups of ed users have been recognized and accepted by the Greek agency called General Secretariat for Research and Innovation (GSRI), which merely funded the project as a Smart Specialization Strategy Priority in the area of “Cultural and Creative industries”. As a result of the on – going nature of the project an evaluation of the proposed platform and its operation is not evaluated yet. The evaluation will be conducted by external experts coming from the field of the VR in education, while the agents taking part in the project have decided to use the Greek educational system (e,g, upper-secondary education and university’s students) as the testing tube for the applications as a whole. After testing and evaluation, as well as after forthcoming adds-on, the platform will get available to mass audience via Oculus store and STEAM store. The features that most probably will be added in first level (even before testing and evaluation) include areas of interest such as:

• Requirements:
  • Multiuser functionality,
  • Virtual avatar assistance and
  • Analytics for users’ behavior when using the platform.
• Implementation:
  • Implement new requirements,
  • Design and integrate extra historical sites and
  • Implementing new content (including 3-D models).

Formal Suggestions. The following comments have been followed and significant changes have been conducted:

• Sentences from 100 to 108: These two paragraphs present certain redundancy of terms and adjectives that make reading difficult. Synthesizing and summarizing the description would help a better reading comprehension. (Reviewer 1)
• There are a number of typos and minor grammatical issues, for example l. 47 is probably missing a 'which', as in 'Kurt Squire's dissertation, which presents' (Reviewer 2)
• Table 1: Since there are cells that start the text in lowercase while most do it in uppercase, unifying styles is recommended. (Reviewer 1)
• Line 131 refers to 'vary', which should probably read 'variety' (Reviewer 2)
• Sentences 132 and 136: avoiding the repetition of “content” by resorting to a synonym is suggested. (Reviewer 1)
• Phrases 144-147 and 212-216: Both paragraphs refer to the player. In the first one, it is referred to him as masculine and the second has been wrote in gender-inclusive terms. It is convenient to unify styles and opt for a single formula for all the text. (Reviewer 1). The user is referred to as 'he' or 'him' in some cases, 'he/she' in some cases, but 'they' or 'them' in others; it may be worth considering standardising on a particular approach throughout, perhaps the neutral 'they'. I suggest a thorough proofread throughout. (Reviewer 2)
• In Lines 169-170 there is a close repetition of the term "experiential experience.... experience". (Reviewer 3)
• 242: platform instead of “platform”. (Reviewer 1)
• 252: Following the structure of the previous figures, it would be convenient to complete figure 4. It could be done with the representation of another of the elements listed, for example the navigation map. (Reviewer 1)
  • Figure 4 has been enriched.
• 257: It would be more enriching if a visualization of some of the exhibited pieces were shown in detail instead of two points of view of the exhibition hall. (Reviewer 1)
  • A case of exhibited piece is provided. If enrichment with more pieces is needed we are happy to provide.
• 326: “as well” instead “9as well”. (Reviewer 1 and 2)

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report