Exploring the Usability and Interaction Experience of the Artsteps Virtual Exhibition Platform by Preschool Children

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

1. Statistical Analysis Needs More Rigor: paper presents percentages and mean scores for questionnaire items, the analysis lacks statistical rigor. For example, the authors mention differences by age and gender (e.g., older girls navigated better), but no statistical tests (e.g., t-tests, ANOVAs) are reported to verify whether these differences are statistically significant. additionally 20 participates are limited number and its hard to draw strong conclusions.A more detailed discussion of the stats and how the small sample size might affect the results would really help strengthen the paper’s credibility.
   
   
2. 20 participates are limited number and its hard to draw strong conclusions.A more detailed discussion of the stats and how the small sample size might affect the results would really help strengthen the paper’s credibility.
3. The stacked bar chart on page 7 (Figure 3) is visually effective but lacks precise numerical annotations. Consider adding percent labels on each segment to improve interpretability. Also, briefly interpret the figure in-text rather than solely referencing it.
4. The paper discussing about using a modified version of the SUS, but it doesn’t explain exactly what questions were asked, how the smiley ratings were turned into scores, or whether the results were reliable like reporting something like Cronbach’s alpha. Adding more detail about how the method was applied would really help make things clearer and easier to replicate.

5. he related work section would benefit from the addition of more recent references that specifically address HCI design for non-readers, touchscreen usability for young children, and digital scaffolding strategies to support early learners.

Author Response

Comment1: Statistical Analysis Needs More Rigor: paper presents percentages and mean scores for questionnaire items; the analysis lacks statistical rigor. For example, the authors mention differences by age and gender (e.g., older girls navigated better), but no statistical tests (e.g., t-tests, ANOVAs) are reported to verify whether these differences are statistically significant. Additionally, 20 participants is a limited number, and it's hard to draw strong conclusions. A more detailed discussion of the stats and how the small sample size might affect the results would really help strengthen the paper’s credibility.

Response1: We thank the reviewer for this important observation. We would like to inform you that since the initial submission of the manuscript, we continued data collection in the same educational setting and expanded the participant pool from 20 to 35 preschool-aged children before receiving the reviewer feedback. The revised version of the manuscript now reflects this updated sample size and reports results accordingly. While we acknowledge that even 35 participants may still limit the applicability of inferential statistics, the increased sample enhances the descriptive robustness and supports more stable trends in the data. Furthermore, the revised manuscript includes an explicit note in the methodology section acknowledging the exploratory nature of the study and the limitations of generalizability. We also added a dedicated paragraph in the discussion to contextualize the findings as indicative rather than conclusive, and to outline directions for future work involving larger and more diverse populations.

 

Comment2: 20 participates are limited number and its hard to draw strong conclusions. A more detailed discussion of the stats and how the small sample size might affect the results would really help strengthen the paper’s credibility.

Response2: We fully agree with the reviewer’s concern regarding the limitations imposed by a small sample size. As mentioned in our response to Comment 1, the participant number has since increased from 20 to 35 children prior to receiving the reviewer feedback. This updated figure is now reflected in the revised manuscript, and all statistical summaries have been updated accordingly.

 

Comment3: The stacked bar chart on page 7 (Figure 3) is visually effective but lacks precise numerical annotations. Consider adding percent labels on each segment to improve interpretability. Also, briefly interpret the figure in-text rather than solely referencing it.

Response3: We thank the reviewer for this valuable suggestion. In the revised manuscript, we have enhanced Figure 3 by adding percentage labels to each segment of the stacked bars, making the distribution of positive, neutral, and negative responses immediately visible. This improves the interpretability and clarity of the visual data. 

Furthermore, we have added a short in-text interpretation in the results section, summarizing the key takeaways from the figure (e.g., high enjoyment levels, challenges in comprehension for some users, and differences by age group). These additions aim to support the reader in quickly grasping the significance of the visualized results and to better integrate the figure with the surrounding narrative.

 

Comment4: The paper discussing about using a modified version of the SUS, but it doesn’t explain exactly what questions were asked, how the smiley ratings were turned into scores, or whether the results were reliable like reporting something like Cronbach’s alpha. Adding more detail about how the method was applied would really help make things clearer and easier to replicate.

Response4: We appreciate this important observation. In response, we have revised the Methodology section to include a more detailed explanation of how the modified System Usability Scale (SUS) was adapted for preschool-aged children. Specifically, we now provide the exact list of items used in the child-friendly version, which was inspired by Vlachogianni & Tselios (2022). We have also added the calculated Cronbach’s alpha for the adapted scale, which yielded a reliability coefficient of α = 0.82, indicating good internal consistency. These additions aim to improve transparency, replicability, and methodological rigor, in line with the reviewer’s suggestion.

 

Comment5: The related work section would benefit from the addition of more recent references that specifically address HCI design for non-readers, touchscreen usability for young children, and digital scaffolding strategies to support early learners.

Response5: We appreciate this helpful recommendation. In the revised manuscript, we have enriched the Related Work section by incorporating additional recent and foundational literature that specifically addresses key areas of early-childhood human-computer interaction.

Reviewer 2 Report

Comments and Suggestions for Authors

1. How does the study account for possible variability in children’s prior exposure to digital devices? This could affect their comfort and performance when interacting with Artsteps. Please clarify and control for digital familiarity in participant profiles.
2. The emotional responses and quotes from children are very engaging. But how were these responses systematically coded or interpreted beyond anecdotal reporting? Please add more detail on how qualitative data were processed.
3. Were all themes equally engaging for the children? Can you assess if specific content types (e.g., animated AI vs. reading room) had more impact? or add a comparative analysis across thematic zones.
4. How was adapted SUS's reliability assessed in this new format with icons? A basic psychometric validation is necessary. Or maybe add supporting data or references for validation.
5. While the study mentions interaction fluency and autonomy, it does not deeply explore task-level behaviors (e.g., how long children took to complete an action, where they hesitated). Can you add a finer-grained interaction analysis?
6. The platform’s visual complexity caused confusion in younger users. Could design iterations based on UI simplification or audio support be tested in a follow-up study to measure improvements? Please plan a controlled usability redesign study.
7. To expand the technical foundation of the current research, authors should reference related works focusing on real-time interaction, visual tracking, and child-centered system usability that utilize machine learning and sensor modalities. This work provides complementary methods and interaction insights that could strengthen the discussion:
   https://doi.org/10.1016/j.engappai.2025.110076
8. The findings suggest strong emotional engagement, especially among older girls. But is there any long-term follow-up on whether this engagement translates to measurable digital literacy gains? Please add or plan a longitudinal study.

Author Response

Comment1: How does the study account for possible variability in children’s prior exposure to digital devices? This could affect their comfort and performance when interacting with Artsteps. Please clarify and control for digital familiarity in participant profiles.

Response1: We thank the reviewer for raising this important point. In response, we have revised the Methodology section to clarify how prior digital exposure was considered. Although we did not apply a formal screening instrument or digital proficiency test, we collected background information from educators and parents about each child’s typical exposure to digital devices (e.g., tablet use at home, touchscreen familiarity in class activities).

Based on these reports, we observed that the children had moderate and relatively homogeneous levels of digital familiarity, primarily through the use of educational apps on tablets or smartphones. None of the participants had prior experience with the Artsteps platform. While this variability could still influence individual performance, we partially controlled for it through consistent scaffolding and adult assistance during the sessions. We have now added this clarification in the revised text and acknowledge it as a limitation in the Discussion section.

Comment2: The emotional responses and quotes from children are very engaging. But how were these responses systematically coded or interpreted beyond anecdotal reporting? Please add more detail on how qualitative data were processed.

Response2: We appreciate the reviewer’s interest in the qualitative aspect of the study. In the revised manuscript, we have now included a description of the qualitative analysis process used to handle children’s verbal comments and observable reactions during the usability sessions. Although the study is primarily exploratory and descriptive, qualitative data were systematically coded using inductive thematic analysis, with two researchers independently reviewing observation notes and transcribed comments. Discrepancies were discussed until consensus was reached. Codes were grouped into higher-order themes related to emotional engagement (e.g., excitement, confusion, enjoyment, frustration). We have added this clarification in the Methodology section and expanded the Results section to indicate how these thematic groupings informed our interpretation.

Comment3: Were all themes equally engaging for the children? Can you assess if specific content types (e.g., animated AI vs. reading room) had more impact? or add a comparative analysis across thematic zones.

Response3: We thank the reviewer for this thoughtful question. In response, we have revised the Results section to include a comparative qualitative analysis across the three thematic zones presented in the Artsteps environment: (1) Animated AI, (2) Reading Room, and (3) Children’s Creations.

Based on our observations and verbal data, we found that the Animated AI zone elicited the highest levels of spontaneous excitement and verbal interaction, likely due to its dynamic visual and auditory stimuli. The Children’s Creations zone also prompted high engagement, particularly because it allowed children to see their own or their peers’ work, fostering personal connection. In contrast, the Reading Room generated relatively lower levels of observable enthusiasm, especially among younger participants, likely due to its static nature and higher reliance on narrative comprehension.

We have now included a paragraph in the Results section to highlight these comparative patterns, and added a note in the Discussion reflecting on how content modality may influence engagement among pre-readers.

Comment4: How was adapted SUS's reliability assessed in this new format with icons? A basic psychometric validation is necessary. Or maybe add supporting data or references for validation.

Response4: We thank the reviewer for this critical point. In the revised manuscript, we have clarified that the adapted SUS version for preschool children used in this study was based on existing validated child-friendly modifications. To further support the psychometric soundness of our adaptation, we report the internal consistency of the 10-item scale as measured by Cronbach’s alpha, which yielded a value of α = 0.82. This indicates good reliability and aligns with accepted thresholds for usability testing instruments. We have now added this information explicitly in the Methodology section, and we also cite the relevant validation literature to enhance transparency and replicability.

Comment5: While the study mentions interaction fluency and autonomy, it does not deeply explore task-level behaviors (e.g., how long children took to complete an action, where they hesitated). Can you add a finer-grained interaction analysis?

Response5: We thank the reviewer for this insightful suggestion. In the revised manuscript, we have included a more detailed interaction-level analysis based on structured observation logs maintained during the study. Specifically, we now report key behavioral markers such as average time to complete core actions, frequency of help requests, and common hesitation points (e.g., navigating menus, placing objects).

Comment6: The platform’s visual complexity caused confusion in younger users. Could design iterations based on UI simplification or audio support be tested in a follow-up study to measure improvements? Please plan a controlled usability redesign study.

Response6: We thank the reviewer for this suggestion. We acknowledge that some younger participants (particularly aged 4–4.5) experienced occasional confusion due to the platform’s visual density and multi-layered interface. This issue is now more clearly described in the Results section. Also, in the revised manuscript, we have expanded the Discussion and Conclusion sections to outline a plan for a controlled usability redesign study. 

Comment7: To expand the technical foundation of the current research, authors should reference related works focusing on real-time interaction, visual tracking, and child-centered system usability that utilize machine learning and sensor modalities. This work provides complementary methods and interaction insights that could strengthen the discussion:
https://doi.org/10.1016/j.engappai.2025.110076

Response7: We thank the reviewer for highlighting this relevant and timely study. We have now incorporated the suggested reference into the Discussion section, as it offers valuable insights into how machine learning and sensor-based systems can enhance real-time interaction and adaptive user modeling in child–computer interaction contexts.

Comment8: The findings suggest strong emotional engagement, especially among older girls. But is there any long-term follow-up on whether this engagement translates to measurable digital literacy gains? Please add or plan a longitudinal study.

Response8: ​​We thank the reviewer for this insightful observation. As the current study was designed as an exploratory short-term intervention, no formal longitudinal follow-up was conducted at the time of submission. However, we agree that examining the persistence of engagement and its potential impact on digital literacy development—particularly in relation to spatial navigation, multimodal interaction, and self-directed exploration—would constitute a valuable next step.

In the revised manuscript, we have added a discussion of this limitation and have proposed a longitudinal research plan to investigate whether early engagement with platforms like Artsteps can lead to measurable improvements in digital fluency over time.

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

accept