Effects of Driver Expertise and Device Type on Digital Traffic Safety Education: An Experimental Study

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Please pay more attention to the quality of Figures and typos in future works.

Comments for author File: Comments.pdf

Comments on the Quality of English Language

Please find attached the comments regarding the article.

Author Response

Manuscript ID: applsci‐3786383 Hyunjin Jang and Hyun K. Kim How Driver Expertise and Device Type Influence the Effective‐ness of Digital Traffic Safety Education The primary goal of the research presented in this article is to develop a VR (Virtual Reality)‐based application for driver safety training and to compare its effectiveness with mobile‐based learning. The article concerns a significant social problem regarding the effectiveness of education in the area of road traffic regulations, which means that its results may have an impact on public safety. In general, the article is developed correctly and meets the requirements of research work. However, it needs improvements as described below. Detailed comments:
-> Thank you for your valuable valuable feedback.

[Comment1] The study employed a small research group, which significantly limits the statistical evaluation and generalizability of the obtained results. What is the reason for such a small group and how might this affect the obtained results?
-> We completely agree with the reviewer's valuable comments. It would have been better if the study had included a larger number of participants and we acknowledge that's one of the limitations of this study. However, as many HCI studies are conducted with 20 participants, this study also followed the same approach. Therefore, we added the following to Section 5.3: “Moreover, this study was conducted with a small number of participants, with 10 individuals in each group. This indicates that it is difficult to ensure the representativeness of the results. Thus, future studies should recruit more participants to verify the statistical evaluation and generalizability.”

[Comment2] Is it possible to investigate the influence of gender and age on the obtained results?
->  Thank you for your valuable and insightful comments. We completely agree that examining the effects of gender and age could provide additional insights. However, the distribution across age groups was uneven because age was not considered as a sampling criterion when recruiting participants. Regarding gender, statistical analyses were conducted but revealed no significant differences between groups. Therefore, we decided not to include gender-related comparison results in the revised manuscript based on these findings.

[Comment3]  Please specify to what extent, in the Authorsʹ opinion, the developed application may influence the actual behavior of drivers on the road (e.g. a decrease in the number of offenses).
-> Thank you very much for your valuable suggestion. In our opinion, the developed application could contribute to enhancing drivers’ knowledge of traffic laws, reducing the driver confusion, and traffic offenses. Therefore, we have added the following to Section 6 to clarify the impact of the developed program on drivers’ actual on-road behavior: “In this study, the developed VR-based driver safety training application is expected to provide drivers with accurate information on traffic regulations, reduce driver confusion, and consequently decrease the occurrence of accidents in actual driving situations.”
Additionally, while the current study did not measure long-term behavioral outcomes, future research will assess such effects using objective indicators. Thus, we have added the following to Section 5.3: “In future studies, delayed assessments should be conducted and long-term learning effectiveness should be evaluated using objective indicators, such as the decrease in the number of traffic offenses.”

[Comment4] Line 293 – is the value for the alpha coefficient written correctly?
-> Thank you for your valuable feedback. The alpha coefficient has been revised to 0.05.

[Comment5] Please improve the readability of Fig. 1 and the quality of Fig. 6.
-> Based on the reviewer's valuable comments, we have improved the readability of Figure 1 and changed Figure 6.

[Comment6] Please add the doi for each reference in the References section.
-> Thank you for your valuable comment and we have added the doi to all references accordingly.

[Comment7] Please check the whole article for typos, e.g. line 91.
-> Thank you for your valuable feedback. We have thoroughly checked the whole manuscript and revised the spelling accordingly.

Reviewer 2 Report

Comments and Suggestions for Authors

Detailed feedback:

• Lines 25-25: Keywords can be sorted alphabetically.
• Lines 28-30: In the Introduction, the authors mention the following: “With recent societal changes and technological advancements, traffic laws are frequently being updated, making it increasingly challenging for drivers to clearly understand the regulations and stay informed through effective traffic safety education.”. They fail to provide the background as to which country this statement applies, as this is not true for all countries.
• Lines 30-32: More detail is needed to put the research in context. Which countries did Faus et al. investigate when they reviewed the 22 studies?
• Lines 41-42: The sentence needs to be rewritten for better clarity.
• Lines 70-75: Add a sentence indicating to the reader that these are the hypotheses that will be investigated and answered with this research study.
• Line 76: It is beneficial to have a paragraph following Section 2 to inform the reader what will be discussed next, rather than having two headings following one another.
• Lines 86, 91, 108: Note the spacing between sentences. In some cases, there are no spaces between sentences, and in other cases, there are too many.
• Line 147: Note the spelling of "vehicle".
• Line 159: At this stage, it is unclear why the prototype was developed. In the previous section, numerous other VR simulators are mentioned. The authors need to clarify why they decided to develop their own VR system. Link this motivation to what was mentioned in Section 1 – the goals of the research.
• Line 175: Early in Section 3.2, the authors refer to different parts of Figure 2 without properly introducing the figure. The figure should also be moved closer to this section, as the reader needs to scroll too far to look at it.
• Line 190: The inline citation for "Lee et al." is incomplete.
• Line 219: This section can benefit from reminding the reader of what the study is about, why it was done, etc. This is only done in Section 4.4 – “This study compared the learning effectiveness of a VR-based driver safety education system with that of mobile-based educational content.”
• Lines 238-240: The sentence is difficult to read.
• Line 248: In both cases and in the text, the authors refer to Tables 2 and 3 as the “…questionnaire…”. The details in the tables do not look like how a typical questionnaire would be set up. The authors should either replace the tables with the actual questionnaire or change the captions and references to the tables and not refer to them as “questionnaires”.
• Line 258: The authors did not mention which existing usability questionnaire they used. For example, the SUS is the most well-known questionnaire used in UX research.
• Line 258: What is meant by “I felt fullness of head while using this prototype.” In Table 2?
• Line 363: Not the spelling of “opinions”.
• Line 365: Note the spelling of “interest”.
• Line 440: 50% of the references are older than 5 years.
• Line 443: Why is the name of the journal abbreviated? This occurs with multiple references throughout the list of References.
• Lines 480-481: Should the URL be underlined?

 

The main concerns:

1. What makes this study unique? There are existing simulators (VR) that the authors could have used. Why did they create their own system?
2. It is unclear why a VR system was needed. According to the authors, it seems that the entire system was a quiz. Nowhere was it mentioned that the users could look around, and according to what they see, confusion was less, for example. Is the VR headset necessary, or could the same/similar results be obtained by using a bigger monitor? In lines 355-358, the authors mention “…reduced head rotation…”. This makes the reader wonder if the VR headset was necessary. The authors need to elaborate in detail to answer these concerns.
3. The authors did not mention which UX questionnaire they used. There are numerous proven questionnaires available, but nothing is mentioned.

Author Response

[Comment1] Lines 25-25: Keywords can be sorted alphabetically.
-> Thank you for your valuable comments. We have sorted keywords alphabetically.

[Comment2] Lines 28-30: In the Introduction, the authors mention the following: “With recent societal changes and technological advancements, traffic laws are frequently being updated, making it increasingly challenging for drivers to clearly understand the regulations and stay informed through effective traffic safety education.”. They fail to provide the background as to which country this statement applies, as this is not true for all countries.
-> We agree that the background is not true for all countries. Thus, the sentence has been revised as follows: “With recent societal changes and technological advancements in South Korea, traffic laws have been frequently revised, making it increasingly challenging for drivers to clearly understand the regulations and stay informed through effective traffic safety education.”

[Comment3] Lines 30-32: More detail is needed to put the research in context. Which countries did Faus et al. investigate when they reviewed the 22 studies?
-> Thank you for your valuable suggestion. We completely agree that the explanation for this sentence was inadequate in the original manuscript, and we have revised the sentence as follows in accordance with your suggestion: “Faus et al. [1] reviewed 22 studies conducted in four continents, including Thailand, the United Kingdom, the United States, and Australia, on road safety education for adults, and reported that it is difficult to determine whether such education has a direct impact on reducing accident rates.”

[Comment4] Lines 41-42: The sentence needs to be rewritten for better clarity.
-> Based on your valuable comments, the sentence has been rewritten as follows and has been revised to clearly express the research findings on the benefits of VR-based education: “They found that the VR-based educational program provided greater immersion and improved road safety awareness compared to traditional video- and presentation-based road safety training.”

[Comment5] Lines 70-75: Add a sentence indicating to the reader that these are the hypotheses that will be investigated and answered with this research study.
-> Accordingly, we added the following sentence to clearly indicate the hypotheses: “The following hypotheses were developed to be examined in this study.”

[Comment6] Line 76: It is beneficial to have a paragraph following Section 2 to inform the reader what will be discussed next, rather than having two headings following one another.
-> Section 2 has added the following sentence: “We conducted a literature review on learning effectiveness, usability, and universal design for developing a more effective VR-based driver safety training prototype. Additionally, we analyzed existing VR services to identify their limitations and enhance the distinctiveness of this study.”

[Comment7] Lines 86, 91, 108: Note the spacing between sentences. In some cases, there are no spaces between sentences, and in other cases, there are too many.
-> We carefully checked the manuscript and revised the spacing.

[Comment8] Line 147: Note the spelling of "vehicle".
-> Based on your valuable comments, the spelling has been revised and we carefully checked the manuscript.

[Comment9] Line 159: At this stage, it is unclear why the prototype was developed. In the previous section, numerous other VR simulators are mentioned. The authors need to clarify why they decided to develop their own VR system. Link this motivation to what was mentioned in Section 1 – the goals of the research.
-> To clearly express and strengthen the unique differences of our own VR program, we added the following contents: “Existing VR-based programs primarily focus on enhancing drivers’ awareness of accident risks by simulating hazardous driving situations. However, existing programs are often ineffective in helping drivers acquire knowledge of traffic regulations and learn how to respond effectively to real-world driving situations. Thus, this study developed a VR-based driver safety training prototype that simulates confusing real-world scenarios while also enabling learners to acquire accurate knowledge of traffic regulations.”

[Comment10] Line 175: Early in Section 3.2, the authors refer to different parts of Figure 2 without properly introducing the figure. The figure should also be moved closer to this section, as the reader needs to scroll too far to look at it.
-> Accordingly, Figure 1 and 2 have been moved to under the first paragraph of Section 3.2 and the contents have also been changed to better readability.

[Comment11] Line 190: The inline citation for "Lee et al." is incomplete.
-> We have added inline citation.

[Comment12] Line 219: This section can benefit from reminding the reader of what the study is about, why it was done, etc. This is only done in Section 4.4 – “This study compared the learning effectiveness of a VR-based driver safety education system with that of mobile-based educational content.”
-> Thank you for your valuable suggestion. Accordingly, Section 4 has been added the summary of the research: “This study evaluated and compared the learning effectiveness of a VR-based driver safety education system and a mobile-based educational program. In addition, we analyzed the developed prototype in terms of usability, presence, aesthetics, and sickness.”

[Comment13] Lines 238-240: The sentence is difficult to read.
-> Thank you for your valuable comment. We have rewritten the sentence to improve the readability: “It was selected due to its suitability for comparing the usability and learning effectiveness of driver safety education between mobile and VR devices.”

[Comment14] Line 248: In both cases and in the text, the authors refer to Tables 2 and 3 as the “…questionnaire…”. The details in the tables do not look like how a typical questionnaire would be set up. The authors should either replace the tables with the actual questionnaire or change the captions and references to the tables and not refer to them as “questionnaires”.
-> Thank you for your valuable comments. We have revised the captions of Table 2 and 3: “Table 2. Factors and items of usability questionnaire for …”, “Table 3. Factors and items of learning effectiveness questionnaire for …”

[Comment15] Line 258: The authors did not mention which existing usability questionnaire they used. For example, the SUS is the most well-known questionnaire used in UX research.
-> Thank you for your valuable comments. We agree that we did not mention which UX questionnaire. To specify the references that are used to design the questionnaire, we have added the following explanation to Section 4.3.: “The questionnaire items for usability, presence, aesthetics, and learning effectiveness were designed with reference to prior studies [32, 33, 34, 35]. The items of sickness were developed based on the Virtual Reality Sickness Questionnaire (VRSQ) [36].”
The added references are as follows:
Witmer, B.G.; Singer, M.J. Measuring presence in virtual environments: A presence questionnaire. Presence. 1998, 7(3), 225–240. https://doi.org/10.1162/105474698565686
Witmer, B.G.; Singer, M.J. The factor structure of the presence questionnaire. Presence: Teleoperators and Virtual Environments. 2005, 14(3), 298–312. https://doi.org/10.1162/105474605323384654
Sacks, R.; Perlman, A.; Barak, R. Construction safety training using immersive virtual reality. Construction Management and Economics. 2013, 31(9), 1005–1017. https://doi.org/10.1080/01446193.2013.828844
 Le, Q.T.; Pedro, A.; Park, C.S. A social virtual reality based construction safety education system for experiential learning. Journal of Intelligent and Robotic Systems. 2015, 79(3), 487–506. https://doi.org/10.1007/s10846-014-0112-z

[Comment16] Line 258: What is meant by “I felt fullness of head while using this prototype.” In Table 2?
-> This content is also utilized in the VRSQ [36] and SSQ [37], which are widely used to measure sickness, and this study adopted the item.

[Comment17] Line 363: Not the spelling of “opinions”. / Line 365: Note the spelling of “interest”.
-> We carefully checked the manuscript and revised the spelling.

[Comment18] Line 440: 50% of the references are older than 5 years.
-> We completely agree with the reviewer's valuable comments. Thus, we have added the following references that were published after 2020, including those suggested by the reviewer:
Saghafian, M.; Laumann, K.; Akhtar, R.S.; Skogstad, M.R. The evaluation of virtual reality fire extinguisher training. Psychol. 2020, 11, 593466. https://doi.org/10.3389/fpsyg.2020.593466
Renganayagalu, S.K.; Mallam, S.C.; Nazir, S. Effectiveness of VR head mounted displays in professional training: A systematic review. Knowl. Learn. 2021, 26(4), 999–1041. https://doi.org/10.1007/s10758-020-09489-9
Scorgie, D.; Feng, Z.; Paes, D.; Parisi, F.; Yiu, T.W.; Lovreglio, R. Virtual reality for safety training: A systematic literature review and meta-analysis. Safety Science. 2024, 171, 106372. https://doi.org/10.1016/j.ssci.2023.106372
Alonso, F.; Faus, M.; Riera, J.V.; Fernandez-Marin, M.; Useche, S.A. Effectiveness of driving simulators for drivers’ training: a systematic review. Applied Sciences 2023, 13(9), 5266. https://doi.org/10.1016/j.jsr.2024.08.002
Yang, F.; Goh, Y.M. VR and MR technology for safety management education: An authentic learning approach. Safety Science. 2022, 148, 105645. https://doi.org/10.1016/j.ssci.2021.105645

[Comment19] Line 443: Why is the name of the journal abbreviated? This occurs with multiple references throughout the list of References.
-> We have carefully checked the name of journal and revised the references.

[Comment20] Lines 480-481: Should the URL be underlined?
-> We have unified the format by removing underlines from all URLs in the References section, including the corresponding URL.

The main concerns:

1. What makes this study unique? There are existing simulators (VR) that the authors could have used. Why did they create their own system?
   -> Thank you very much for your valuable feedback. We have specified unique differences of our own VR program and strengthened the contribution of this study. In Section 3, the following sentences were added: “Existing VR-based programs primarily focus on enhancing drivers’ awareness of accident risks by simulating hazardous driving situations. However, existing programs are often ineffective in helping drivers acquire knowledge of traffic regulations and learn how to respond effectively to real-world driving situations. Thus, this study developed a VR-based driver safety training prototype that simulates confusing real-world scenarios while also enabling learners to acquire accurate knowledge of traffic regulations. The prototype was developed based on findings from a literature review identifying factors that enhance learning effectiveness, usability, and universal design principles.”
2. It is unclear why a VR system was needed. According to the authors, it seems that the entire system was a quiz. Nowhere was it mentioned that the users could look around, and according to what they see, confusion was less, for example. Is the VR headset necessary, or could the same/similar results be obtained by using a bigger monitor? In lines 355-358, the authors mention “…reduced head rotation…”. This makes the reader wonder if the VR headset was necessary. The authors need to elaborate in detail to answer these concerns.
   -> Thank you for your valuable and insightful comments. Based on the suggestion, we have thoroughly revised Section 1 and 3.1.
   First, we restructured the introduction. To clarify the necessity of VR in this research, we have outlined the advantages of VR and VR simulation related to research objectives: “Faus et al. [1] reviewed 22 studies conducted in four continents, including Thailand, the United Kingdom, the United States, and Australia, on road safety education for adults, and reported that it is difficult to determine whether such education has a direct impact on reducing accident rates. They emphasized the need for further evaluations to identify more effective educational strategies. Virtual reality (VR) offers new possibilities for addressing these limitations. Virtual reality (VR) provides realistic visual (visual realism) and auditory elements (auditory realism), allowing users to feel as if they are part of the environment [2]. These immersive characteristics make VR particularly effective for training cognitive skills and the acquisition of procedural knowledge [3]. Furthermore, VR simulation could be effectively applied to driver education and training. Simulation-based training allows users to safely experience complex or hazardous real-world scenarios without actual risk, thereby enhancing understanding and decision-making capabilities [4]. Therefore, it is essential to explore the development and effectiveness of traffic safety education programs that incorporate advanced technologies such as VR simulation.”
   Furthermore, we have added the following explanation to Section 3.1 to explain that users could look around from a driver perspective. It specifically explained that users can enhance their decision-making skills and learning of traffic laws by immersively experiencing confusing situations during real-world driving: “The prototype enables users to explore a 360-degree virtual driving environment from a driver perspective, offering an immersive simulation of potentially confusing situations. It also supports effective learning of traffic regulations and improves decision making through quizzes and explanatory feedback.”
3. The authors did not mention which UX questionnaire they used. There are numerous proven questionnaires available, but nothing is mentioned.
   -> Thank you for your valuable comments. We agree that we did not mention which UX questionnaire. To specify the references that are used to design the questionnaire, we have added the following explanation to Section 4.3.: “The questionnaire items for usability, presence, aesthetics, and learning effectiveness were designed with reference to prior studies [32, 33, 34, 35]. The items of sickness were developed based on the Virtual Reality Sickness Questionnaire (VRSQ) [36].”

Reviewer 3 Report

Comments and Suggestions for Authors

This paper investigates the differential effectiveness of VR-based and mobile-based training for driver traffic‑safety education. The overall structure is coherent and the topic aligns well with current research frontiers. There are some minor issues:

1. Participants completed the questionnaire immediately after the main experiment. This study did not include delayed or follow-up assessments, so it was not possible to evaluate long-term learning effects.
2. Learning effectiveness are assessed solely with subjective rating scales, with no objective knowledge test or behavioural performance indicators.
3. Although the entire experiment duration is reported, the authors do not specify how long the VR and mobile experimental sessions lasted respectively, or whether the durations were identical. Unequal exposure time could affect memory and thus bias the results.
4. Other small points:

1. Section 4.5.1 reports the usability and aesthetics results twice, leading to redundancy.
2. The ethics approval number is inconsistent between line 231 and line 437.
3. Figure 6 is low-resolution. Please replace it with a higher quality version.

Please double check and correct any remaining minor errors.

Overall, I recommend the manuscript be accepted after minor revisions.

Author Response

This paper investigates the differential effectiveness of VR-based and mobile-based training for driver traffic‑safety education. The overall structure is coherent and the topic aligns well with current research frontiers. There are some minor issues:
-> Thank you for your valuable comments.

1. Participants completed the questionnaire immediately after the main experiment. This study did not include delayed or follow-up assessments, so it was not possible to evaluate long-term learning effects.
-> We agree that this study did not include delayed or follow-up assessments. Thus, we have added the following to Section 5.3: “The learning effectiveness was assessed solely through subjective rating scales administered immediately after the experiment, without objective knowledge tests or behavioral performance indicators. Moreover, follow-up assessments were not conducted, making it impossible to evaluate the long-term learning effects. In future studies, delayed assessments should be conducted and long-term learning effectiveness should be evaluated using objective indicators, such as the decrease in the number of traffic offenses.”

2. Learning effectiveness are assessed solely with subjective rating scales, with no objective knowledge test or behavioural performance indicators.
-> We completely agree with the reviewer's valuable comments. In Section 5.3, the following sentences were added “The learning effectiveness was assessed solely through subjective rating scales administered immediately after the experiment, without objective knowledge tests or behavioral performance indicators. Moreover, follow-up assessments were not conducted, making it impossible to evaluate the long-term learning effects. In future studies, delayed assessments should be conducted and long-term learning effectiveness should be evaluated using objective indicators, such as the decrease in the number of traffic offenses.”

3. Although the entire experiment duration is reported, the authors do not specify how long the VR and mobile experimental sessions lasted respectively, or whether the durations were identical. Unequal exposure time could affect memory and thus bias the results.
-> We agree with the reviewer's opinion. Therefore, we added the following to Section 4.4. to specify the VR and mobile experimental sessions lasted identical: “The entire experimental session lasted approximately 60 minutes. The VR and mobile experimental sessions were conducted for the same duration of 15 minutes each. Participants were free to take breaks or withdraw at any time.”

4. Other small points:

1. Section 4.5.1 reports the usability and aesthetics results twice, leading to redundancy.
   -> Duplicate content from Section 4.5.1 was deleted.
2. The ethics approval number is inconsistent between line 231 and line 437.
   -> We have changed the ethics approval number correctly.
3. Figure 6 is low-resolution. Please replace it with a higher quality version.
   -> Based on the reviewer's comments, we have changed Figure 6.

Please double check and correct any remaining minor errors.
-> We have carefully proofread the manuscript and corrected minor typographical, grammatical, and formatting errors throughout.

Overall, I recommend the manuscript be accepted after minor revisions.

 

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

The authors improved the research paper significantly and addressed all of my previous concerns. A job well done. I spotted the following minor issues that must be addressed:

Lines 31-34: The authors mentions the following: “Faus et al. [1] reviewed 22 studies conducted in four continents, including Thailand, the United Kingdom, the United States, and Australia, on road safety education for adults, and reported that it is difficult to determine whether such education has a direct impact on reducing accident rates.”. Note the following: Research is conducted “on” four continents. After mentioning the words “four continents” the authors lists “Thailand, the United Kingdom, the United States, and Australia”, which are all countries on different continents. The sentence, thus, needs to be corrected.

Line 199: The authors refer to Figure 1C. I assume it should refer to Figure 2C or only Figure 1? Note that this is the only reference to Figure 1 in the text, so maybe the reference should be to Figure 1 then and not 2C. I spotted that Figure 1 is referenced in Line 224. If this is the first reference to it, then I suggest that Figures 1 and 2 be swapped around.

Figure 2: Note the top left corner of the figure, there are numbers printed on top of one another.

 

Author Response

[Comment1] Lines 31-34: The authors mentions the following: “Faus et al. [1] reviewed 22 studies conducted in four continents, including Thailand, the United Kingdom, the United States, and Australia, on road safety education for adults, and reported that it is difficult to determine whether such education has a direct impact on reducing accident rates.”. Note the following: Research is conducted “on” four continents. After mentioning the words “four continents” the authors lists “Thailand, the United Kingdom, the United States, and Australia”, which are all countries on different continents. The sentence, thus, needs to be corrected.
-> Thank you for your valuable feedback. We completely agree that the sentence needs to be corrected. We have revised the sentence as follows in accordance with your suggestion: “Faus et al. [1] reviewed 22 studies on road safety education for adults conducted in Thailand, the United Kingdom, the United States, and Australia, and reported that it is difficult to determine whether such education has a direct impact on reducing accident rates.”

[Comment2] Line 199: The authors refer to Figure 1C. I assume it should refer to Figure 2C or only Figure 1? Note that this is the only reference to Figure 1 in the text, so maybe the reference should be to Figure 1 then and not 2C. I spotted that Figure 1 is referenced in Line 224. If this is the first reference to it, then I suggest that Figures 1 and 2 be swapped around.
-> Thank you for your suggestion. We swapped the positions of Figures 1 and 2. Additionally, Figure 2 has been placed under the fourth paragraph of Section 3.2, and the first paragraph has been revised as follows: “To design a quiz-type gamified learning system, we developed a quiz UI to increase learner motivation and learning effectiveness. Additionally, to enhance intrinsic motivation through achievement, a strategy known to improve learning effectiveness in educational virtual games—an interface showing that no fines are imposed for correct quiz answers—was designed (Figure 1 (c)).”

[Comment3] Figure 2: Note the top left corner of the figure, there are numbers printed on top of one another.
-> We carefully checked the figures and revised the errors.