Next Article in Journal
RGB-Depth Camera-Based Assessment of Motor Capacity: Normative Data for Six Standardized Motor Tasks
Next Article in Special Issue
Virtual Training on Stress Management for the Mothers of Children with Disabilities in the United Arab Emirates
Previous Article in Journal
Do Homicide Perpetrators Have Higher Rates of Delayed-Suicide Than the Other Offenders? Data from a Sample of the Inmate Population in Italy
Previous Article in Special Issue
Preventing Postpartum Depression in the Early Postpartum Period Using an App-Based Cognitive Behavioral Therapy Program: A Pilot Randomized Controlled Study
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 19
Issue 24
10.3390/ijerph192416990
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Does Online Implementation Make a Difference in the Effects of a Mental Health Curriculum at Schools?

by
Margarida Frade dos Santos
1,*,
Celeste Simões
1,2,
Anabela Caetano Santos
1,2,
Paula Lebre
1,3 and
Ilaria Grazzani
4
1
Faculdade de Motricidade Humana, Universidade de Lisboa, 1495-751 Lisbon, Portugal
2
Instituto de Saúde Ambiental, Faculdade de Medicina, Universidade de Lisboa, 1649-028 Lisbon, Portugal
3
Instituto de Etnomusicologia INET-MD, Faculdade de Motricidade Humana, Universidade de Lisboa, 1495-751 Lisbon, Portugal
4
Department of Human Science for Education “R. Massa”, University of Milano-Bicocca, 20126 Milan, Italy
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2022, 19(24), 16990; https://doi.org/10.3390/ijerph192416990
Submission received: 28 November 2022 / Revised: 14 December 2022 / Accepted: 15 December 2022 / Published: 17 December 2022
(This article belongs to the Special Issue Digital Mental Health: Innovative Technology and Service)
Versions Notes

Abstract

:
COVID-19 changed and challenged education, with schools obliged to adapt to online settings. This study aims to evaluate the impact of a mental health curriculum implemented at schools, considering the implementation settings: online, onsite, and mixed (online and onsite). From kindergarten to high school, 933 students were evaluated by teachers regarding their social and emotional learning, strengths and difficulties, and academic outcomesin two measuring times: pre- and post-test. A qualitative analysis of teachers’ adaptations to the online implementation was also conducted. Results revealed a positive impact with both mixed and onsite implementation. However, the mixed format demonstrated significant positive changes between the pre—and post-test, namely in relationship skills, responsible decision-making, internalized problems, and academic achievement. The mixed format with few online activities appears to have a more positive impact on students. Nevertheless, implementing social and emotional skills (SES) activities exclusively online seems to positively affect some SES domains more than onsite and mixed formats. Teachers used synchronous (e.g., digital platforms) and asynchronous (e.g., extra resources) adaptations for the implementation. This study shows that implementing mental health programs at schools, in this case, PROMEHS, is beneficial for students, even amidst the pandemic, and regardless of the implementation settings.

1. Introduction

With the COVID-19 pandemic, the world was challenged in many ways that implied multiple changes in all contexts, with consequences that affected the population’s mental health [1,2,3]. Specifically, in education systems, the pandemic originated challenges that had to be addressed without preparation or planning due to the lockdowns, including the total adaptation to an online learning approach [4,5,6]. Subjects, activities, and programs designed to be implemented onsite had to be adapted. Teachers struggled due to a lack of experience in online teaching, little preparation time, and proper skills [7,8].
Research about the COVID-19 impact on mental health has been widely active, with cross-sectional studies depicting the periods of the lockdown being the most published/present, lacking studies with longitudinal designs covering the periods before, during, and after lockdown periods [9]. Regarding children and adolescents, meta-analyses and systematic reviews have reported an increased manifestation of internalized (e.g., anxiety and depression) and externalized (e.g., aggressiveness) problems, relationship problems, sleep problems, reduction of life satisfaction, and wellbeing [1,10,11,12,13,14,15]. Not only has the pandemic threatened children and adolescents’ mental health, but also the development of their social and emotional skills [9,14,16]. In a study by Martinsone and colleagues [9], students from Latvia, Italy, and Portugal were evaluated in two time periods during COVID-19 and lockdown measures. Results revealed an increase in internalized and externalized problems and a reduction in social and emotional skills [9]. Their study also found a significant negative correlation between adolescents’ social and emotional skills and internalized and externalized problems. Therefore, social and emotional skills can act as a protective factor and resilience, reducing the impact of COVID-19 and the manifestation of internalized and externalized problems [14,17].
Before the pandemic, the World Health Organization reported that, worldwide, 1 in 8 children and adolescents between 6- and 18-years-old experienced a mental health problem and that fifty percent of these problems began at 14 [14]. Preventive measures and intervention programs focused on reducing the risk of future problems were developed, implemented, and evaluated to address various areas, such as mental health, wellbeing, social and emotional skills, and resilience, with schools considered the ideal context for the implementation [14,18,19]. The benefits associated with implementing these programs at school are [19,20,21,22,23]: an improvement in self-efficacy, wellbeing, academic achievement, and the relationship with oneself and others; protection of at-risk students; and decreased behavioral problems, conflicts, and emotional problems.
In pre-pandemic conditions, mental health programs implemented in the school context had a considerate positive impact with a recognition of the importance of integrating social and emotional skills into the school curriculum [19]. With COVID-19 lockdowns and its already identified implications on children and adolescents’ mental health, it is even more critical to implement these programs, for example, during lockdown periods with online school. The implementations can occur by developing programs and interventions or adapting existing ones to the online context, with further evaluation of their impact [2,24,25,26,27].
Programs and interventions implemented online, using digital technologies, platforms, or apps, have been used in and out of the school context to promote mental health [28,29,30,31,32,33]. However, most of these programs in the school context are: self-paced, students do it during breaks; are implemented by other professionals (e.g., psychologists, school counselors, health professionals) that are not the teacher; or the teacher intervention is only to present the program and to give support regarding the platform use, but they are not involved through the process [34,35,36,37]. The results of a systematic review indicated that online digital programs with high levels of human interaction were the most effective [37]. Taking this into account, the intervention by a teacher might be a determinant of students’ motivation and engagement during the program, also having a higher impact on the expected outcomes [38,39]. However, research on online learning entails mostly the subjects of the traditional school curriculums, disregarding program implementation at schools by teachers around the themes of mental health, resilience, and social and emotional skills [40,41].
To our knowledge, research around the online adaptations of social and emotional skills and mental health programs in the school context is scarce, especially for the pandemic period, with only a few studies being published. For example, Currie and colleagues [42] examined the online learning adaptations teachers applied when delivering the program Second Step. Although accessed via an online platform, this program was developed to be implemented onsite (i.e., in class) [42]. A toolbox was created for online adaptations in collaborative work between elementary school teachers and researchers [42]. Adaptations around the categories defined included: the use of breakout rooms; using the chat window to share with the whole class; using online discussion boards or other platforms; using paper to write or draw; using non-verbal signs or sticky notes; performing roleplays for the camera followed by group discussion; recording the scene and then show in class; and maintaining the original activity if students had room to move when activities included movement [42]. Additionally, instead of using handouts for each student, teachers asked them to answer verbally or use collaborative tools to share and submit answers, pictures, work, or other documents [42].
A social and emotional learning program entitled Adventures Aboard the S.S. GRIN was adapted to online school learning amidst COVID-19 and implemented in third-grade students [43]. The program revealed an increment of social and emotional skills after the implementation [43].
Chen and Adams [44] interviewed preschool teachers to identify strategies implemented in online schooling to promote children’s social and emotional development. The strategies more frequently mentioned were reading a book followed by a discussion, the use of visuals, and engagement in targeted conversations [44]. Teachers also noted that with remote learning, they expanded their teaching tools [44].
The support from schools and teachers was even more critical during the lockdown to help children and adolescents develop skills and strategies to cope with those stressful events [42]. That can be provided with programs that develop those skills [24], with digital technologies crucial for maintaining the interaction between peers and teachers and allowing the continuity of learning through online school [14]. Nevertheless, implementing these programs at the school’s reopening is crucial to support students and answer their social and emotional needs, to minimize the consequences of the lockdown and the overall pandemic [24,45].
The present study aims to contribute to evidence on the impact of mental health programs at schools while addressing gaps regarding the knowledge on the impact of a program implementation in different school learning settings since they are mostly implemented onsite, with little research about online implementation. Although programs’ adaptations to online settings have been published, little is known about the adaptations’ effects on the program’s impact. The study also addresses a literature gap on the program’s impact when implemented during COVID-19 and in the school context. It also answers directions by the scientific community to evaluate programs implemented in this period. Therefore, the present study aims to investigate the impact of a mental health curriculum at schools implemented by teachers during COVID-19, considering different implementation settings. More specifically, this study aims to contribute to the understanding of the impact of the implementation of the PROMEHS curriculum on mental health, social and emotional skills, and academic outcomes, considering the settings of implementation, namely online, onsite, and mixed (online and onsite), which occurred due to the restrictions imposed by the pandemic. It also aims to evaluate if the results of the mixed implementation differ considering the number of online activities implemented and the school levels. Lastly, we aim to identify the online digital adaptations teachers made to the curriculum developed for onsite implementation.

2. Materials and Methods

2.1. Study Design

This study is part of an Erasmus + Project, PROMEHS—Promoting Mental Health at Schools, that developed, implemented, and evaluated an evidence-based universal curriculum to promote students’ mental health from ages 3 to 18. PROMEHS was the first mental health curriculum developed in collaboration between researchers, policymakers, and scientific institutions from seven European countries (Italy, Croatia, Greece, Latvia, Malta, Portugal, and Romania). It was a quasi-experimental study with experimental and waiting groups evaluated in two measuring times (pre- and post-test). Between measuring times, the experimental group had teachers’ training and curriculum implementation at school and online. In contrast, the waiting group had the teachers’ training and implementation after the post-test.
Students were evaluated by three sources: teachers, parents, and themselves. Students’ self-report started at age 8. This study only includes data collected in Portugal, specifically the teachers’ evaluations of students from the experimental group.
At the beginning of the school year of 2020/2021, the research team scheduled meetings with policymakers and wellness organizations, who then contacted school principals who could be interested in PROMEHS. Then, each school principal contacted teachers to assess their interest in participating. After the selection of schools, meetings were scheduled with teachers and school psychologists to present the project and the evaluation process. Informed consents were sent to parents through the teachers. The collected consents were kept in the schools, and the research team created an identification code for each student that teachers further completed to guarantee confidentiality and anonymity. This code was used in both moments of evaluations and by the three sources of students’ evaluations. The links for the online surveys were sent via e-mail to teachers, who then shared them with the parents and students. Students completed the survey in class and gave their assent online before starting the survey. However, some had to be completed at home due to the lockdown imposed in January 2021. Due to limitations at schools or home, some surveys were collected using paper versions, which schools then sent to the research team, who inserted the data into the database. The pre-test was between December 2020 and mid-February 2021, and the post-test was between the end of May 2021 and mid-July 2021. Between evaluations, teachers undertook a total of 25 h of training (16 h were undertaken before the implementation and started in January 2021, and 9 h were supervision during the implementation) given by the research team. After the first part of the training, teachers had to implement in class a minimum of twelve activities of the curriculum (four of each theme that comprises PROMEHS: Promoting Social and Emotional Learning; Promoting Resilience; and Preventing Social, Emotional and Behavioral Problems). For more information about the PROMEHS project and methodology procedures, see Cefai and colleagues’ study [46].
The PROMEHS project was developed between February 2019 and August 2022. In Portugal, the implementation phase began in February of 2021, corresponding to a lockdown phase. Facing these situations, the research team allowed teachers two options: starting to implement during the lockdown and creating adaptations for the activities to enable online implementation or waiting until the lockdown was over and beginning the implementation onsite.

2.2. Participants

For the quantitative analysis, a total of 1134 students from kindergarten to high school were evaluated by teachers. A total of 201 were eliminated due to having more than fifty percent of missing values [47], namely students that were not assessed on pre- or post-test. The final sample included 933 students evaluated in both pre- and post-test.
Data from a total of 55 teachers were used in the qualitative analysis. From these, 35 used a mixed method (online and onsite implementation), and 20 used an onsite method.

2.3. Measures and Procedures

Quantitative analysis:
Teachers completed the Strengths and Difficulties Questionnaire (SDQ) [48], the Social Skills Improvement System Social-Emotional Learning Brief Scales (SSIS-SEL) [49,50,51], and an Academic Engagement scale.
Mental Health: The SDQ [48] assesses children and adolescents’ mental health by measuring behavioral and emotional difficulties. The SDQ comprises 25 items organized in five scales (five items each). Four scales are related to difficulties (emotional symptoms, behavioral problems, hyperactivity, and peer problems), and one to strengths (prosocial behavior). A 3-point Likert scale ranging from 0 (not true) to 2 (certainly true) is used. Higher scores on the difficulties scales mean more significant difficulties, and on the strengths scale, higher prosocial behavior. In this study, the model that considers internalized (i.e., emotional symptoms and peer problems scales), externalized problems (i.e., conduct problems and hyperactivity scales), and prosocial behavior was used [52]. Cronbach’s alpha reported acceptable to good internal consistency (internalized problems: αpre = 0.78, αpost = 0.80; externalized problems: αpre = 0.87, αpost = 0.87; prosocial behavior: αpre = 0.87, αpost = 0.87).
Social and Emotional Skills (SES): The SSIS-SEL [49,50,51] evaluates children and adolescents’ SES (self-awareness, self-regulation, social awareness, relationship skills, and responsible decision-making) using a 20 items scale (four items per domain). A 4-point Likert scale is used, ranging from 0 (never) to 3 (almost always), with a higher score meaning higher perceived social and emotional skills. Cronbach’s alpha reported acceptable to good internal consistency (self-awareness: αpre = 0.77, αpost = 0.79; self-regulation: αpre = 0.84, αpost = 0.86; social awareness: αpre = 0.87, αpost = 0.87; relationship skills: αpre = 0.79, αpost = 0.81; responsible decision making: αpre = 0.89, αpost = 0.88).
Academic outcomes: It is composed of 3 single items that evaluate students’ academic motivation, engagement in learning, and academic performance. A 5-point Likert scale ranging from 0 (very poor) to 4 (excellent) is used, with a higher score meaning higher academic engagement in the three items.
School level: Participants were grouped by the different age grades of the two activities’ manuals developed in PROMEHS (k-4th: kindergarten to 4th grade; 5th–12th: 5th to 12th grade).
Qualitative analysis: During the implementation, teachers evaluated each activity they conducted in their class. In each, they had to indicate the theme of the activity selected [Social and Emotional Learning (SEL), Resilience or Behavioral, Social and Emotional Problems], the method (online or onsite), as well as the adaptations made to the activities. Based on the first two indications, three variables were created: type of implementation (onsite, online, or mixed); the number of activities implemented online (sum of activities online); and number of SEL activities implemented online (sum of SEL activities online). The last indication teachers mentioned—adaptations for each activity—will be the subject of a qualitative analysis.

2.4. Data Analysis

Quantitative data analysis was performed using IBM SPSS Statistics (version 28.0). Missing data for the SDQ was between 0 and 0.4%; for SSIS-SEL, the percentage was between 0.2 and 1.1%; and for Academic Outcomes, between 0 and 0.1%. Thus, listwise deletion was used since missing data below 5% is inconsequential [47]. Cronbach’s alpha (α) was used to evaluate internal consistency, which was acceptable to good.
A mixed between-within subjects analysis of variance (ANOVA) was conducted to assess the impact of the type of implementation used on the results across two time periods on students’ SDQ, SSIS-SEL, and Academic Outcomes scores. To assess the interaction effect, Wilks’s Lambda was reported. Only the onsite and mixed (online and onsite) implementation methodologies were used. Then, a one-way ANOVA was conducted to assess the impact of the number of online activities implemented on the domains evaluated. The dependent variables were computed for the difference between the post- and pre-test. The independent variable (number of activities) was grouped into two categories according to the number of activities implemented online (1: 1–3 activities; 2: 4–7 activities). Next, a mixed between-within subjects ANOVA was conducted on the same dependent variables to explore the differences between the number of activities performed online and by school level.
The SEL theme of PROMEHS was the only one where implementation occurred totally online, totally onsite, or mixed. Therefore, we conducted a mixed between-within subjects ANOVA to assess the impact of the methodology used on the results across two time periods on students’ scores of SSIS-SEL. The variable number of SEL activities implemented varied from 3 to 8.
For these analyses, it was assured the assumptions were met. When Levene’s test for homogeneity of variants was significant, a more stringent significance value was defined (p < 0.001) for the mixed between-within subjects [53]. For the one-way ANOVA, when the assumption was violated, the Welch test was performed [53]. Since the sample size is reasonably similar, the ANOVA is reasonably robust to violations of the homogeneity of variance assumption [54]. Post-hoc comparisons were made using the Tukey test. Cohen’s d guidelines were used to interpret the effect sizes obtained [53], with small, medium, and large effects being 0.01, 0.06, and 0.14, respectively [55].
In the qualitative analysis, the MAXQDA 2022 software was used. Thematic analysis [56] included five steps according to LeCompte [57]: (1) tidying up with the organization of all data files in the software; (2) finding units of analysis by reading all files and identifying possible categories for the units; (3) creating stable sets of units of analysis; (4) creating patterns; and (5) assembling structures.

3. Results

3.1. Comparison between Implementation Methodology (Onsite and Mixed)

A mixed between-within subjects ANOVA was conducted to assess the impact of onsite and type of implementation (type effect) on mental health (internalized problems, externalized problems, total of difficulties, and prosocial behavior), social and emotional competencies (SES: self-awareness, self-regulation, social awareness, relationship skills, and responsible decision-making), and academic outcomes (academic motivation, engagement in learning, and academic performance), across the two time periods.
The analysis did not reveal a significant interaction for type × time of implementation on any domains of mental health. There was a significant effect of time in all (see Table 1). There was a significant effect for the type of implementation, only for internalized problems [F(1, 931) = 3.95, p = 0.047, ηp2 = 0.004)], although with a small effect. The onsite group had a smaller reduction of internalized problems than the mixed group, as presented in Table 1 (for more detailed information, see Table S1 in the Supplemental Material).
For SES, there was a significant type × time of implementation interaction on social awareness [F(1, 928) = 7.75, p = 0.005, ηp2 = 0.008], relationship skills [F(1, 928) = 6.43, p = 0.011, η2 = 0.007], and responsible decision making [F(1, 928) = 5.09, p = 0.024, ηp2 = 0.005], although will small effect sizes. There was an effect of time in all domains of social and emotional skills, as seen in Table 2. Additionally, there is an effect of type of implementation for relationship skills [F(1, 928) = 7.65, p = 0.006, ηp2 = 0.008] and responsible decision-making [F(1, 928) = 4.94, p = 0.026, ηp2 = 0.005], with the mixed group reporting higher levels on these domains compared to the onsite group. However, the effect sizes were small. For more detailed information, see Table S2 in the Supplemental Material.
For academic outcomes, there was a significant type × time of implementation interaction on academic performance [F(1, 930) = 7.19, p = 0.007, ηp2 = 0.01] and a significant effect of time in all domains (see Table 3 or, for more detailed information, Table S3 in the Supplemental Material). In addition, there was a significant effect for the type of implementation in academic motivation [F(1, 930) = 6.36, p = 0.012, ηp2 = 0.01] and academic achievement [F(1, 930) = 5.43, p = 0.020, ηp2 = 0.006], with the onsite group having higher levels when compared to the mixed group.

3.2. Impact of the Number of Activities Implemented Online

As previously mentioned, a one-way ANOVA was conducted to evaluate the impact of the number of activities implemented online on mental health, social and emotional skills, and academic outcomes. For this analysis, we used the difference between both evaluation periods for each domain. Table 4, Table 5 and Table 6 present descriptive data, the F test for all the domains evaluated.
For all domains of mental health, there was a significant difference between groups, with group 1 (fewer online activities) having higher differences between evaluation periods, which means a higher reduction of problems and a higher increase in prosocial behavior (Table 4).
There was a significant difference between groups for self-awareness, social awareness, and relationship skills for SES domains, except for self-regulation and responsible decision-making. As shown in Table 5, group 1 shows higher differences between moments, indicating higher gains in social and emotional skills.
Regarding academic outcomes, no significant differences between groups were observed in all domains, which means that the differences in academic outcomes did not differ between the number of activities implemented.

3.3. Differences in Impact Considering the Number of Activities Online and School Level

A mixed between-within subjects ANOVA was conducted to explore the relation between the number of online activities and school level on the domains evaluated. The grouped variable, number of activities online (1: 1–3 activities; 2: 4–7 activities), was used, as well as the school level (k-4th: kindergarten to 4th grade; 5th–12th: 5th to 12th grade). Tables S4–S12 in the Supplemental Material present the means, standard deviations, F tests, and effect sizes.
On mental health domains, there was a significant interaction time × group × school level on externalized problems [F(1, 581) = 9.83, p = 0.002, ηp2 = 0.02], and prosocial behavior [F(1, 581) = 8.86, p = 0.003, ηp2 = 0.02], although with small effect sizes. In all domains, a significant interaction was observed on time x group of online activities and on time × school level. A significant effect of time, for p < 0.05, was also observed. In both the k-4th and 5th–12th groups, there is a higher reduction in group 1 of activities, i.e., the group with one to three online activities.
Between subjects, there was a significant interaction group × school level for internalized problems [F(1, 581) = 11.69, p < 0.001, ηp2 = 0.02]. Regarding group effect, there was a significant difference in all domains. The effect of the school level was only significant for internalized problems, and K-4th had a higher reduction than 5th–12th.
On SES domains, there was a significant interaction for time × group × school level on self-regulation [F(1, 578) = 9.30, p = 0.002, ηp2 = 0.02], social awareness [F(1, 578) = 17.20, p < 0.001, ηp2 = 0.03], relationship skills [F(1, 578) = 10.77, p = 0.001, ηp2 = 0.02], and responsible decision making [F(1, 578) = 11.11, p < 0.001, ηp2 = 0.02]. There was a significant interaction for time × group for all domains except for self-regulation. Interaction time × school level and effect of time were significant for all SES domains. Except for the group with fewer online activities from 5th–12th, where a reduction of self-regulation was shown, all other domains increased at post-test. A higher increase in the group with fewer online activities in both school levels was observed for the other domains.
Between subjects, there was a significant interaction, group × school level, only for relationship skills [F(1, 578) = 4.09, p = 0.044, ηp2 = 0.007]. Between groups (less online activities vs. more online activities), it was significant only for self-regulation; for school levels, it was significant for all, except for self-regulation and responsible decision-making. K-4th had a higher reduction in the SES domains compared to 5th–12th.
On academic outcomes, there was a significant interaction for time × group × school level for academic achievement [F(1, 580) = 5.55, p = 0.019, ηp2 = 0.01]. No significant interaction time × group was observed, whereas a significant interaction time × school level and a main time effect was found for all domains.
The interaction group × school level and the group effect were not significant between subjects. The school level effect was significant for all outcomes, with better results for k-4th.

3.4. SES Results Regarding Different Implementation Methods: Online, Onsite, and Mixed

A mixed between-within subjects ANOVA was conducted to assess the impact of the method used to implement SES activities (online, onsite, or mixed) on the students’ SSIS-SEL scores. For this analysis, only the cases with four SES activities were selected since it was the number requested for this theme in the project.
The analysis did report a significant type × time interaction for self-regulation [F(2, 667) = 3.73, p = 0.025, ηp2 = 0.011]. Onsite implementation showed a lower mean at the pre-test and a higher mean at the post-test than mixed implementation. There was also a significant effect of time for all domains, as presented in Table 7. Type of implementation was significant in self-awareness [F(2, 667) = 3.02, p = 0.049, ηp2 = 0.009], relationship skills [F(2, 667) = 4.34, p = 0.013, ηp2 = 0.013], and responsible decision making [F(2, 667) = 3.44, p = 0.033, ηp2 = 0.010]. Post-hoc comparisons using the Tukey test were used. For the domains where it was significant between groups, the online method had significantly higher means than the onsite group (see Table 7 or Tables S13 and S14 in the Supplemental Material).

3.5. Qualitative Analyses

Of the 35 teachers included in the dataset implemented in a mixed format, four were excluded for not mentioning digital adaptations, despite having implemented activities online. This analysis aimed to identify which adaptations teachers made when implementing activities online that included digital resources (e.g., platforms, game-apps, presentations, etc.). Two main categories were defined: synchronous (i.e., implemented during the online class schedule); and asynchronous activities (i.e., implemented outside the online class schedule). For each category, different subcategories were identified and will be presented below. For the present summary, the statements that are more representative of each category were selected (for more information regarding the number of subjects that mention each theme and subtheme, consult Table S15 in the Supplemental Material).

3.5.1. Synchronous

Synchronous adaptations were the most used by teachers during online implementation. These adaptations for synchronous classes included: digital platforms, presentations, and virtual classrooms.
Digital platforms were the most indicated by the teachers (twenty teachers). These include platforms such as Wordwall, Word Art, Mentimeter, Padlet, Poll Everywhere, Nearpod, VoKI, and Random Picker, which teachers used to adapt the activity to online classes. Three teachers did not specify which platform was used.
“Instead of writing on the board, they wrote on the chat, and afterward, it created a word cloud with the Word Art App.” Teacher BF (3rd grade)
“Presented the solutions wheel in the Worldwall app.” Teacher LPR (7th grade).
“The worksheet was transformed in Microsoft Forms with 33 questions and two hypotheses to choose from”, and also, “I created in the Padlet App a mural for students to share the diverse experiences they have lived.” Teacher CS (10th grade).
Presentations. This subcategory includes presentations of videos (on YouTube), music, images, PowerPoint slides of the activity, or sharing the screen to present worksheets, and was mentioned by sixteen teachers.
“I presented the story in PowerPoint, and the worksheet was also in the presentation, and I filled it out (as each child would identify their skills, talents, and passions).” Teacher AS (5th grade).
“Lastly, we talked about the changing hat—meaning I shared a picture of different umbrellas.” Teacher CC (3rd grade).
“The session began with the presentation of part of a music.” Teacher RA (5th grade).
Six teachers mentioned virtual classrooms, mainly used as adaptations for group activities, where teachers defined groups and put the students in different virtual rooms.
“The class was divided into two groups, and they were in two different virtual rooms to debate the consequences and strategies.” Teacher MPB (4th grade).
“Rooms in the TEAM Platform for a discussion in a more restricted group.” Teacher CR (9th grade).

3.5.2. Asynchronous

Asynchronous adaptations included the execution of part of the activity and extra resources where teachers sent more activities for students to do/see after.
Execution of part of the activity before, i.e., teachers sent part of the activity before the online class, so students prepared it in advance, was mentioned by seven teachers.
“I used the asynchronous class. Students saw the video alone, at home, and answered the questions of the video exploration also at home, sending me their conclusions.” Teacher PC (10th grade).
“Parents were asked to share the video of the story.” Teacher TF (Kindergarten).
“Students wrote two individual qualities/characteristics and enumerated one quality/characteristic positive of each one of their classmates in a worksheet. Afterwards, they sent me the worksheet via e-mail.” Teacher CC (3rd grade).
Lastly, three teachers sent extra resources for students.
“In the Google Classroom Platform shared a cards game about emotions and breathing exercises.” Teacher AA (4th grade).
“Students watched three more YouTube videos in the asynchronous class.” Teacher PC (10th grade).

4. Discussion

This study intended to investigate the impact of different implementation settings (online, mixed, and onsite) of a mental health curriculum at schools on students’ mental health indicators, social and emotional skills, and academic outcomes due to the restrictions imposed by the pandemic. It also aimed to evaluate the differences in impact considering the number of online activities implemented by itself and with school levels. Lastly, our qualitative analysis aimed to identify the online digital adaptations that teachers made to the curriculum, which were developed for onsite implementation. The findings from this study revealed that PROMEHS implementation during COVID-19 positively impacted students in both mixed and onsite implementation, despite more expressive results for the mixed format. Students who had fewer activities online showed greater improvements than those who had more online activities. When considering the school level, younger students (from kindergarten to 4th grade) had more positive results than older students (from 5th to 12th grade). When comparing the implementation format for SES activities, a totally online format reported better results, with an increase of SES at the post-test. Lastly, teachers reported different adaptations to implement the PROMEHS curriculum for the online format in synchronous (e.g., using digital platforms) and asynchronous (e.g., executing part of the activity before synchronous class) classes. The findings will be discussed in turn.
As mentioned, results showed that implementing the PROMEHS curriculum during COVID-19 positively impacted students, with the reduction of behavior problems and an increase in social and emotional skills and academic achievement, despite onsite or mixed implementation. Nevertheless, more expressive results were found in the mixed implementation on several domains, such as internalized problems, social awareness, relationship skills, and responsible decision-making. These results show that an online implementation, amid a lockdown period, was beneficial for children and adolescents. We can hypothesize that it acted as a protective factor since the literature on the impact of the lockdown reported an increase in mental health problems and academic achievement [11,14,58] and identified social and emotional skills, emotional regulation, and psychological resilience as protective factors [17,59]. Additionally, as reported in the study of Martinsone and colleagues [9], between evaluation moments that occurred during and after lockdown periods, students showed an increase of internalized and externalized problems and a reduction of social and emotional skills during COVID-19. Their study also found a significant negative correlation between adolescents’ social and emotional skills and internalizing and externalizing problems.
Feelings of social disconnectedness and loneliness [13], lack of social support, absence of interactions with peers and teachers, and further interpersonal relationship problems [59,60,61,62] were concerns manifested by children and adolescents caused by school closures and the absence of their structural activities [59]. In this study, students that were part of the mixed group had the implementation of PROMEHS activities by their teachers during online classes. Even though the PROMEHS curriculum was developed to be implemented onsite, teachers adapted it to implement the activities during this difficult time. The activities part of the PROMEHS curriculum includes methodologies involving interactions between students and teachers, using teamwork, games, debates, and discussions. By fomenting these types of activities when many problems emerged, they were working towards reducing the impact of lockdown and also to lessen its effects on children and adolescents’ development [59,63].
Besides the impact on students’ mental health during lockdown periods, studies have also reported a reduction in students’ academic adjustment, achievement, performance, and motivation during the pandemic [15,64,65,66]. In the current study, after PROMEHS implementation, an increase in academic motivation and achievement was observed in both groups (mixed and onsite), with students from the mixed implementation group reporting higher academic achievement than those from the onsite group. Thus, it seems that PROMEHS positively impacted students’ mental health and social and emotional skills, which might have helped them cope better with the situation but also helped them with academic demands. Nevertheless, although PROMEHS also positively impacted academic motivation, it was less expressive for the mixed implementation. According to Klootwijk and colleagues [66], students had shown a decrease in academic motivation during online school compared to when school was onsite. Thus, students from the onsite group might have shown higher academic motivation when the implementation started, which could have been heightened by the PROMEHS curriculum, resulting in a more expressive increase than the mixed group.
It is important to consider that despite the adverse outcomes of lockdown on children and adolescents, some have thrived with it [14,61]. For example, a UK study that evaluated school students during the COVID-19 lockdown showed that those who reported improved mental wellbeing had lower levels of anxiety and depression, improved their relationship with others, felt less left out and lonely, and were less bullied [61]. When analyzing the results obtained in the current study, it is also important to consider that PROMEHS might positively impact those suffering from the lockdown and those who coped well. In this case, the PROMEHS program might be an extra support, adding more strategies to their repertoire.
When analyzing the mixed group, interesting results were found concerning the number of activities implemented online. The group with fewer online activities (one to three) showed better results than those with more online activities (four to seven). These results show that the PROMEHS curriculum helped students cope with the lockdown since it revealed an increase in SESs and a reduction of mental health problems, as previously indicated. Though, despite the adaptations performed by teachers for online implementation, the program had a higher impact when activities were mostly performed onsite.
In our sample, children and adolescents reduced their internalized and externalized problems and increased their prosocial behavior with PROMEHS implementation. Nevertheless, it was higher in the groups with mixed implementation, specifically for those with fewer online activities. In accordance with the results from the study of Liu and colleagues [67], that the school reopening might reduce by itself the prevalence of internalized problems, we can hypothesize that by having online implementation which is maintained when transitioning by having the implementation of the activities in onsite settings, it can contribute to a higher reduction of these problems. On the contrary, if children increase the expression of emotional and behavioral problems and reduce prosocial behavior, as found by Wang and colleagues [68], the continuity of the implementation can serve as relevant support regarding the needs that children and adolescents might show [24,45].
Regarding the school level, we found that students from kindergarten to 4th grade had better results than those from 5th to 12th grade, with a higher reduction of mental health problems and increased social and emotional skills and academic outcomes. These results (except for self-regulation in older students) might be related to a higher impact of lockdown on adolescents, who suffered more compared to children, and therefore had higher impacts on their mental health [11,14,58], or to the program itself since, according to Yeager [69], these programs tend to show more effectiveness in children
For SEL activities, when comparing the three types of implementations (online, onsite, and mixed), all revealed positive results. There was a significant difference in self-awareness, relationship skills, and responsible decision-making between online and onsite settings. The first reported better results but no differences between groups for self-regulation and social awareness. Again, these findings show the importance of PROMEHS implementation during and after the lockdown.
Considering previous studies on the impact of the pandemic [10,11,12,13], our study shows the importance of implementing programs that promote the mental health of children and adolescents to counteract the impacts of the pandemic.
The results of this study are consistent with previous studies on the impact of mental health programs implemented at schools, which report an increase in social and emotional skills, academic achievement, and a reduction of behavioral problems [19,20,70]. It is also consistent with the results from the study by Li and colleagues [43], which reported increases in social and emotional skills with the adaptation of an SEL program for online learning during COVID-19. In addition, our results align with the study of Cefai and colleagues, which compiles data from the six European countries where PROMEHS was implemented, including Portugal [46]. Nevertheless, it was not possible to evaluate the impact of a total online implementation for the whole implementation period or consider the various PROMEHS themes.
The results from the qualitative analysis reported that teachers could adapt the program to an online setting using their digital skills. Considering that the implementation did not occur during the first lockdown, teachers were already familiarized and had more knowledge about online learning, which could have facilitated their ability to adapt the activities. Adaptations occurred mainly in synchronous classes using platforms such as Wordwall, Nearpod, Padlet, and others. They also used video, pictures, and PowerPoint presentations and used virtual classes to enable teamwork between groups. Asynchronous adaptations included the execution of part of the activity and extra resources. These adaptations from teachers align with those suggested by Currie and colleagues [42] for the online delivery of social and emotional skills programs, and Chen and Adams [44].
The present study has limitations and strengths. Regarding limitations, it was not possible to evaluate a total online implementation. Further, the pre-test was conducted before and during the lockdown, which can produce bias regarding the results. Teachers might perceive students differently when evaluating them before or during the lockdown. Lastly, there was no information about students’ age, and for that reason, school-level groups were created.
This study adds knowledge regarding the impact of interventions during the pandemic, answering the United Nations’ solicitation on evaluating interventions conducted during this critical period [2]. Second, this study evaluates social and emotional skills, a domain that, to our knowledge, is lacking in evaluation during the pandemic. Third, this study had participants from urban and rural areas from different regions in Portugal, which allowed us to have a more global evaluation of the program’s impact in the country. Fourth, the sample comprises students from different school levels, being evaluated in groups using the school level variable. Lastly, this study identified adaptations of mental health programs to an online context made by teachers, which can help professionals develop or adapt programs.
Further investigation is needed regarding the impact of mental health and wellbeing programs at schools, which can be helpful for other periods of crisis, namely other pandemics or for situations that demand online settings (e.g., lockdown due to health risks). Additionally, the literature on the impact of the lockdown on children and adolescents’ social and emotional skills, as well as the mental health of this population after lockdown, is scarce, limiting the discussion. It is important to consider more rigorous adaptations when adapting onsite programs to online and designing programs specifically for online settings. These adaptations should consider digital competencies and the reality/needs of the school and family contexts.

5. Conclusions

Answering the question presented in the title Does online implementation make a difference in the effects of a mental health curriculum at schools?, we can say that it does make a difference in students’ SES, mental health, and academic outcomes. Compared to those who had an exclusive implementation onsite, results show that those who started the implementation online benefited more, with a more expressive and significant reduction of internalized problems and an increase in relationship skills, responsible decision making and academic achievement. Students who had a total online implementation of SEL activities reported higher self-awareness, relationship skills, and responsible decision-making.
However, despite these differences between implementation settings, all students benefited from the implementation of PROMEHS, regardless of the type of implementation, which reveals that with proper adaptations, it is possible to implement these programs during online school, not limited to the onsite setting. The results of this study also reinforce the already extensive research on mental health programs implemented at schools.
When focusing on the mixed implementation, those with fewer activities online reported decreased mental health problems and increased SES and academic outcomes compared to those with more activities online. In addition, younger students from kindergarten to 4th grade had more expressive results in all domains than older students from 5th to 12th grade.
Teachers that conducted part of the implementation online were able to adapt the PROMEHS curriculum to this setting, using both synchronous and asynchronous adaptions. Synchronous adaptations focused on the use of digital platforms, the presentation of videos, music and pictures, and virtual rooms to transform the activity initially developed for onsite implementation. The limitations of online classes were related to the time per class, which made teachers proceed with the execution of part of the activity in asynchronous classes.
Finally, implementing this program amid a lockdown was positive and important in reducing its impact on children and adolescents, with social skills being promoted and mental health problems reduced. Therefore, we can also conclude that implementing mental health programs at school during the pandemic can be an important protective factor and can be used as a way to counteract its impact on children and adolescents.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/ijerph192416990/s1, Table S1: Mental Health for mixed and onsite implementation groups across the two time periods. P-values for interaction (type × time), time and type effects; Table S2: SES for mixed and onsite groups across the two time periods. F tests and effect sizes for interaction (type × time), time and type effects; Table S3: Academic Outcomes for mixed and onsite groups across the two time periods. Mean values, standard deviations, F tests, and effect sizes for interaction (type × time), time and type effects; Table S4: Mental Health for School Level and the number of online activities across the two time periods. Mean values, standard deviations; Table S5: Mental Health for School Level and the number of online activities across the two time periods. F tests and effect sizes for interaction within subjects; Table S6: Mental Health for School Level and the number of online activities across the two time periods. F tests and effect sizes for interaction between subjects; Table S7: SES for School Level and the number of online activities across the two time periods. Mean values, standard deviations; Table S8: SES for School Level and the number of online activities across the two time periods. F tests and effect sizes for interaction within subjects; Table S9: SES for School Level and the number of online activities across the two time periods. F tests and effect sizes for interaction between subjects; Table S10: Academic Outcomes for School Level and the number of online activities across the two time periods. Mean values, standard deviations; Table S11: Academic Outcomes for School Level and the number of online activities across the two time periods. F tests and effect sizes for interaction within subjects; Table S12: Academic Outcomes for School Level and the number of online activities across the two time periods. F tests and effect sizes for interaction between subjects; Table S13: SES across the two time periods considering the methodology. Means and standard deviations; Table S14: SES domains across the two time periods considering the methodology. F tests and effect sizes, for interaction (type × time), time and type effects; Table S15: ‘Teachers’ adaptations for online activities using digital resources (Number of Subjects that mention each theme and subtheme).

Author Contributions

Conceptualization, M.F.d.S. and C.S.; methodology, M.F.d.S.; validation, M.F.d.S. and C.S.; formal analysis, M.F.d.S. and C.S.; investigation, M.F.d.S., C.S., A.C.S., P.L. and I.G.; resources, M.F.d.S., C.S., A.C.S., P.L. and I.G.; data curation, M.F.d.S.; writing—original draft preparation, M.F.d.S.; writing—review and editing, C.S., A.C.S., M.F.d.S., and I.G.; visualization, M.F.d.S.; supervision, C.S., A.C.S. and P.L.; project administration, C.S. and I.G.; funding acquisition, I.G. All authors have read and agreed to the published version of the manuscript.

Funding

This work was conducted within a funded Erasmus + KA3 research project by the EU, named PROMEHS—Promoting Mental Health at Schools. No. 606689-EPP-1-2018-2-IT-EPPKA3-PI-POLICY. It was supported by the PROMEHS Project—Promoting Mental Health at Schools (grant number BID/PROMEHS/2021 attributed to the first author). It was also supported by the Portuguese National Foundation for Science and Technology (FCT-MCTES) [grant number COVID/BD/151772/2021 attributed to the third author].

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by the Ethics Committee of the Environmental Health Institute, Faculty of Medicine, University of Lisbon, Lisbon, Portugal (protocol code: 74/20; date: 20 March 2020).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available upon reasonable request from the corresponding author.

Acknowledgments

The authors would like to acknowledge Lúcia Canha, Ana Marta Fonseca, Dória Santos, and Margarida Gaspar de Matos for their contribution to the project development in Portugal. The authors are particularly grateful to the schools and participants who supported and allowed the data collection.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Bussières, E.-L.; Malboeuf-Hurtubise, C.; Meilleur, A.; Mastine, T.; Hérault, E.; Chadi, N.; Montreuil, M.; Généreux, M.; Camden, C. Consequences of the COVID-19 Pandemic on Children’s Mental Health: A Meta-Analysis. Front. Psychiatry 2021, 12, 2125. [Google Scholar] [CrossRef] [PubMed]
  2. United Nations Policy Brief: COVID-19 and the Need for Action on Mental Health. 2020. Available online: https://reliefweb.int/report/world/policy-brief-covid-19-and-need-action-mental-health-13-may-2020?gclid=EAIaIQobChMImfrqjMqJ_AIVB7qWCh1AqgQBEAAYAiAAEgJebPD_BwE (accessed on 14 December 2022).
  3. Lee, J. Mental Health Effects of School Closures during COVID-19. Lancet Child Adolesc. Health 2020, 4, 421. [Google Scholar] [CrossRef] [PubMed]
  4. Chogyel, N.; Wangdi, N.; Dema, Y. Evaluating the Challenges in Online Learning during the COVID-19 Pandemic in a Middle Secondary School. Int. J. Didact. Stud. 2021, 2, 101459. [Google Scholar] [CrossRef]
  5. Martin, F.; Sun, T.; Westine, C.D. A Systematic Review of Research on Online Teaching and Learning from 2009 to 2018. Comput. Educ. 2020, 159, 104009. [Google Scholar] [CrossRef]
  6. DeCoito, I.; Estaiteyeh, M. Transitioning to Online Teaching During the COVID-19 Pandemic: An Exploration of STEM Teachers’ Views, Successes, and Challenges. J. Sci. Educ. Technol. 2022, 31, 340. [Google Scholar] [CrossRef] [PubMed]
  7. Assunção Flores, M.; Gago, M. Teacher Education in Times of COVID-19 Pandemic in Portugal: National, Institutional and Pedagogical Responses. J. Educ. Teach. 2020, 46, 507–516. [Google Scholar] [CrossRef]
  8. Ma, K.; Chutiyami, M.; Zhang, Y.; Nicoll, S. Online Teaching Self-Efficacy during COVID-19: Changes, Its Associated Factors and Moderators. Educ. Inf. Technol. 2021, 26, 6675–6697. [Google Scholar] [CrossRef]
  9. Martinsone, B.; Stokenberga, I.; Damberga, I.; Supe, I.; Simões, C.; Lebre, P.; Canha, L.; Santos, M.; Santos, A.C.; Fonseca, A.M.; et al. Adolescent Social Emotional Skills, Resilience and Behavioral Problems during the COVID-19 Pandemic: A Longitudinal Study in Three European Countries. Front. Psychiatry 2022, 13, 1738. [Google Scholar] [CrossRef] [PubMed]
  10. Jones, E.A.K.; Mitra, A.K.; Bhuiyan, A.R.; Dankulincova, Z. Impact of COVID-19 on Mental Health in Adolescents: A Systematic Review. Int. J. Environ. Res. Public Health 2021, 18, 2470. [Google Scholar] [CrossRef]
  11. Panchal, U.; Salazar de Pablo, G.; Franco, M.; Moreno, C.; Parellada, M.; Arango, C.; Fusar-Poli, P. The Impact of COVID-19 Lockdown on Child and Adolescent Mental Health: Systematic Review. Eur. Child Adolesc. Psychiatry 2021, 1, 1–27. [Google Scholar] [CrossRef] [PubMed]
  12. Meherali, S.; Punjani, N.; Louie-Poon, S.; Rahim, K.A.; Das, J.K.; Salam, R.A.; Lassi, Z.S.; Matos, D. Mental Health of Children and Adolescents Amidst COVID-19 and Past Pandemics: A Rapid Systematic Review. Int. J. Environ. Res. Public Health 2021, 18, 3432. [Google Scholar] [CrossRef] [PubMed]
  13. Magson, N.R.; Freeman, J.Y.A.; Rapee, R.M.; Richardson, C.E.; Oar, E.L.; Fardouly, J. Risk and Protective Factors for Prospective Changes in Adolescent Mental Health during the COVID-19 Pandemic. J. Youth Adolesc. 2021, 50, 44–57. [Google Scholar] [CrossRef] [PubMed]
  14. Sharma, M.; Idele, P.; Manzini, A.; Aladro, C.; Ipince, A.; Olsson, G.; Banati, P.; Anthony, D. Life in Lockdown: Child and Adolescent Mental Health and Well-Being in the Time of COVID-19; UNICEF Office of Research—Innocenti: Florence, Italy, 2021.
  15. Branje, S.; Morris, A.S. The Impact of the COVID-19 Pandemic on Adolescent Emotional, Social, and Academic Adjustment. J. Res. Adolesc. 2021, 31, 486–499. [Google Scholar] [CrossRef] [PubMed]
  16. Green, K.H.; van de Groep, S.; Sweijen, S.W.; Becht, A.I.; Buijzen, M.; de Leeuw, R.N.H.; Remmerswaal, D.; van der Zanden, R.; Engels, R.C.M.E.; Crone, E.A. Mood and Emotional Reactivity of Adolescents during the COVID-19 Pandemic: Short-Term and Long-Term Effects and the Impact of Social and Socioeconomic Stressors. Sci. Rep. 2021, 11, 11563. [Google Scholar] [CrossRef] [PubMed]
  17. Grazzani, I.; Agliati, A.; Cavioni, V.; Conte, E.; Gandellini, S.; Spagnolo, M.L.; Ornaghi, V.; Rossi, F.M.; Cefai, C.; Bartolo, P.; et al. Adolescents’ Resilience During COVID-19 Pandemic and Its Mediating Role in the Association Between SEL Skills and Mental Health. Front. Psychol. 2022, 13, 801761. [Google Scholar] [CrossRef]
  18. Lai, K.Y.C.; Hung, S.F.; Lee, H.W.S.; Leung, P.W.L. School-Based Mental Health Initiative: Potentials and Challenges for Child and Adolescent Mental Health. Front. Psychiatry 2022, 13, 1196. [Google Scholar] [CrossRef] [PubMed]
  19. Cefai, C.; Simões, C.; Caravita, S. A Systemic, Whole-School Approach to Mental Health and Well-Being in Schools in the EU; Publications Office of the European Union: Luxembourg, 2021. [Google Scholar]
  20. Durlak, J.A.; Weissberg, R.P.; Dymnicki, A.B.; Taylor, R.D.; Schellinger, K.B. The Impact of Enhancing Students’ Social and Emotional Learning: A Meta-Analysis of School-Based Universal Interventions. Child Dev. 2011, 82, 405–432. [Google Scholar] [CrossRef] [PubMed]
  21. Green, A.L.; Ferrante, S.; Boaz, T.L.; Kutash, K.; Wheeldon-Reece, B. Social and Emotional Learning during Early Adolescence: Effectiveness of a Classroom-Based SEL Program for Middle School Students. Psychol. Sch. 2021, 58, 1056–1069. [Google Scholar] [CrossRef]
  22. Goldberg, J.M.; Sklad, M.; Elfrink, T.R.; Schreurs, K.M.G.; Bohlmeijer, E.T.; Clarke, A.M. Effectiveness of Interventions Adopting a Whole School Approach to Enhancing Social and Emotional Development: A Meta-Analysis. Eur. J. Psychol. Educ. 2019, 34, 755–782. [Google Scholar] [CrossRef] [Green Version]
  23. Weare, K.; Nind, M. Promoting Mental Health and Wellbeing in Schools. In Interventions and Policies to Enhance Wellbeing; Wiley Blackwell: Hoboken, NJ, USA, 2014; pp. 1–48. [Google Scholar] [CrossRef]
  24. Hamoda, H.M.; Chiumento, A.; Alonge, O.; Hamdani, S.U.; Saeed, K.; Wissow, L.; Rahman, A. Addressing the Consequences of the COVID-19 Lockdown for Children’s Mental Health: Investing in School Mental Health Programs. Psychiatr. Serv. 2021, 72, 729–731. [Google Scholar] [CrossRef]
  25. United Nations Children’s Fund The Role of Remote Modalities in Implementing Mental Health and Psychosocial Support Programs and Services in the Education Sector; UNICEF: New York, NY, USA, 2022.
  26. Barker, R.; Hartwell, G.; Bonell, C.; Egan, M.; Lock, K.; Viner, R.M. Research Priorities for Mental Health in Schools in the Wake of COVID-19. J. Epidemiol. Community Health (1978) 2022, 76, 448–450. [Google Scholar] [CrossRef]
  27. Holmes, E.A.; O’Connor, R.C.; Perry, V.H.; Tracey, I.; Wessely, S.; Arseneault, L.; Ballard, C.; Christensen, H.; Cohen Silver, R.; Everall, I.; et al. Multidisciplinary Research Priorities for the COVID-19 Pandemic: A Call for Action for Mental Health Science. Lancet Psychiatry 2020, 7, 547–560. [Google Scholar] [CrossRef] [PubMed]
  28. Corrieri, S.; Heider, D.; Conrad, I.; Blume, A.; König, H.H.; Riedel-Heller, S.G. School-Based Prevention Programs for Depression and Anxiety in Adolescence: A Systematic Review. Health Promot. Int. 2014, 29, 427–441. [Google Scholar] [CrossRef] [Green Version]
  29. Liverpool, S.; Mota, C.P.; Sales, C.M.D.; Čuš, A.; Carletto, S.; Hancheva, C.; Sousa, S.; Cerón, S.C.; Moreno-Peral, P.; Pietrabissa, G.; et al. Engaging Children and Young People in Digital Mental Health Interventions: Systematic Review of Modes of Delivery, Facilitators, and Barriers. J. Med. Internet. Res. 2020, 22, e16317. [Google Scholar] [CrossRef] [PubMed]
  30. Champion, K.E.; Parmenter, B.; McGowan, C.; Spring, B.; Wafford, Q.E.; Gardner, L.A.; Thornton, L.; McBride, N.; Barrett, E.L.; Teesson, M.; et al. Effectiveness of School-Based EHealth Interventions to Prevent Multiple Lifestyle Risk Behaviours among Adolescents: A Systematic Review and Meta-Analysis. Lancet Digit. Health 2019, 1, e206–e221. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  31. Grist, R.; Porter, J.; Stallard, P. Mental Health Mobile Apps for Preadolescents and Adolescents: A Systematic Review. J. Med. Internet. Res. 2017, 19, e176. [Google Scholar] [CrossRef] [Green Version]
  32. Hollis, C.; Falconer, C.J.; Martin, J.L.; Whittington, C.; Stockton, S.; Glazebrook, C.; Davies, E.B. Annual Research Review: Digital Health Interventions for Children and Young People with Mental Health Problems—A Systematic and Meta-Review. J. Child Psychol. Psychiatry 2017, 58, 474–503. [Google Scholar] [CrossRef] [PubMed]
  33. Zhou, X.; Edirippulige, S.; Bai, X.; Bambling, M. Are Online Mental Health Interventions for Youth Effective? A Systematic Review. J. Telemed. Telecare 2021, 27, 638–666. [Google Scholar] [CrossRef]
  34. Hugh-Jones, S.; Pert, K.; Kendal, S.; Eltringham, S.; Skelton, C.; Yaziji, N.; West, R. Adolescents Accept Digital Mental Health Support in Schools: A Co-Design and Feasibility Study of a School-Based App for UK Adolescents. Ment. Health Prev. 2022, 27, 200241. [Google Scholar] [CrossRef]
  35. Calear, A.L.; Christensen, H.; Mackinnon, A.; Griffiths, K.M.; O’Kearney, R. The YouthMood Project: A Cluster Randomized Controlled Trial of an Online Cognitive Behavioral Program With Adolescents. J. Consult. Clin. Psychol. 2009, 77, 1021–1032. [Google Scholar] [CrossRef]
  36. Osborn, T.L.; Rodriguez, M.; Wasil, A.R.; Venturo-Conerly, K.E.; Gan, J.; Alemu, R.G.; Roe, E.; Arango, G.S.; Otieno, B.H.; Wasanga, C.M.; et al. Single-Session Digital Intervention for Adolescent Depression, Anxiety, and Well-Being: Outcomes of a Randomized Controlled Trial with Kenyan Adolescents. J. Consult. Clin. Psychol. 2020, 88, 657–668. [Google Scholar] [CrossRef] [PubMed]
  37. Garrido, S.; Millington, C.; Cheers, D.; Boydell, K.; Schubert, E.; Meade, T.; Nguyen, Q.V. What Works and What Doesn’t Work? A Systematic Review of Digital Mental Health Interventions for Depression and Anxiety in Young People. Front. Psychiatry 2019, 10, 759. [Google Scholar] [CrossRef] [PubMed]
  38. Shum, A.K.; Lai, E.S.; Leung, W.G.; Cheng, M.N.; Wong, H.K.; So, S.W.; Law, Y.W.; Yip, P.S. A Digital Game and School-Based Intervention for Students in Hong Kong: Quasi-Experimental Design. J. Med. Internet. Res. 2019, 21, e12003. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  39. Lai, E.S.Y.; Kwok, C.L.; Wong, P.W.C.; Fu, K.W.; Law, Y.W.; Yip, P.S.F. The Effectiveness and Sustainability of a Universal School-Based Programme for Preventing Depression in Chinese Adolescents: A Follow-Up Study Using Quasi-Experimental Design. PLoS ONE 2016, 11, e0149854. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  40. Martin, F.; Bacak, J.; Polly, D.; Dymes, L. A Systematic Review of Research on K12 Online Teaching and Learning: Comparison of Research from Two Decades 2000 to 2019. J. Res. Technol. Educ. 2021, 1–20. [Google Scholar] [CrossRef]
  41. Geesa, R.L.; Geesa, R.L.; Robbins, K.; Shively, K. The Collaborative Model for Teaching O-SEL: Preparing Educators to Design. J. Online Learn. Res. 2022, 8, 67–100. [Google Scholar]
  42. Currie, C.; Currie, C.; Rajanbabu, P.; King, D. A Toolbox of Adaptations for Online Delivery of SEL Programming. J. Online Learn. Res. 2022, 8, 7–35. [Google Scholar]
  43. Li, L.; Flynn, K.S.; DeRosier, M.E.; Weiser, G.; Austin-King, K. Social-Emotional Learning Amidst COVID-19 School Closures: Positive Findings from an Efficacy Study of Adventures Aboard the S.S. GRIN Program. Front. Educ. 2021, 6, 213. [Google Scholar] [CrossRef]
  44. Chen, J.J.; Adams, C.B. Drawing from and Expanding Their Toolboxes: Preschool Teachers’ Traditional Strategies, Unconventional Opportunities, and Novel Challenges in Scaffolding Young Children’s Social and Emotional Learning During Remote Instruction Amidst COVID-19. Early Child Educ. J. 2022, 1–13. [Google Scholar] [CrossRef] [PubMed]
  45. Minkos, M.L.; Gelbar, N.W. Considerations for Educators in Supporting Student Learning in the Midst of COVID-19. Psychol. Sch. 2021, 58, 416–426. [Google Scholar] [CrossRef]
  46. Cefai, C.; Camilleri, L.; Bartolo, P.; Grazzani, I.; Cavioni, V.; Conte, E.; Ornaghi, V.; Agliati, A.; Gandellini, S.; Tatalovic Vorkapic, S.; et al. The Effectiveness of a School-Based, Universal Mental Health Programme in Six European Countries. Front. Psychol. 2022, 13, 925614. [Google Scholar] [CrossRef]
  47. Hair, J.F.; Black, W.C.; Babin, B.J.; Anderson, R.E. Multivariate Data Analysis; Pearson Education Limited: Harlow, UK, 2014; ISBN 129202190X. [Google Scholar]
  48. Goodman, R. The Strengths and Difficulties Questionnaire: A Research Note. J. Child Psychol. Psychiatry 1997, 38, 581–586. [Google Scholar] [CrossRef]
  49. Anthony, C.J.; Brann, K.L.; Elliott, S.N.; Garis, E.J. Examining the Structural Validity of the SSIS SEL Brief Scales—Teacher and Student Forms. Psychol. Sch. 2022, 59, 260–280. [Google Scholar] [CrossRef]
  50. Elliot, S.N.; DiPerna, J.C.; Anthony, C.J.; Lei, P.; Gresham, F.M. Social Skills Improvement System, Social Emotional Learning (SSIS-SEL) Brief Scales; SAIL CoLab: Scottsdale, AZ, USA, 2020. [Google Scholar]
  51. Anthony, C.J.; Elliott, S.N.; Yost, M.; Lei, P.W.; DiPerna, J.C.; Cefai, C.; Camilleri, L.; Bartolo, P.A.; Grazzani, I.; Ornaghi, V.; et al. Multi-Informant Validity Evidence for the SSIS SEL Brief Scales across Six European Countries. Front. Psychol. 2022, 13, 4325. [Google Scholar] [CrossRef]
  52. Goodman, A.; Lamping, D.L.; Ploubidis, G.B. When to Use Broader Internalising and Externalising Subscales Instead of the Hypothesised Five Subscales on the Strengths and Difficulties Questionnaire (SDQ): Data from British Parents, Teachers and Children. J. Abnorm. Child Psychol. 2010, 38, 1179–1191. [Google Scholar] [CrossRef]
  53. Pallant, J. SPSS Survival Manual, 7th ed.; Routledge: London, UK, 2020; ISBN 9781003117452. [Google Scholar]
  54. Stevens, J.P. Applied Multivariate Statistics for the Social Sciences; Routledge: London, UK, 2012; ISBN 9781136910708. [Google Scholar]
  55. Cohen, J. Statistical Power Analysis for the Behavioral Sciences, 2nd ed.; Lawrence Erlbaum Associates: New York, NY, USA, 1988; ISBN 9780203771587. [Google Scholar]
  56. Ritchie, J.; Lewis, J.; National Centre for Social Research (Great Britain). Qualitative Research Practice: A Guide for Social Science Students and Researchers, 1st ed.; Ritchie, J., Lewis, J., Eds.; SAGE Publications: London, UK, 2003; ISBN 076197092. [Google Scholar]
  57. LeCompte, M.D. Analyzing Qualitative Data. Theory Pract. 2000, 39, 146–154. [Google Scholar] [CrossRef]
  58. Kauhanen, L.; Wan Mohd Yunus, W.M.A.; Lempinen, L.; Peltonen, K.; Gyllenberg, D.; Mishina, K.; Gilbert, S.; Bastola, K.; Brown, J.S.L.; Sourander, A. A Systematic Review of the Mental Health Changes of Children and Young People before and during the COVID-19 Pandemic. Eur. Child Adolesc. Psychiatry 2022, 1, 1–19. [Google Scholar] [CrossRef] [PubMed]
  59. Viner, R.; Russell, S.; Saulle, R.; Croker, H.; Stansfield, C.; Packer, J.; Nicholls, D.; Goddings, A.-L.; Bonell, C.; Hudson, L.; et al. School Closures During Social Lockdown and Mental Health, Health Behaviors, and Well-Being Among Children and Adolescents During the First COVID-19 Wave. JAMA Pediatr. 2022, 176, 400. [Google Scholar] [CrossRef] [PubMed]
  60. Branquinho, C.; Santos, A.C.; de Matos, M.G. Our voice and reflections for the new wave of COVID-19: Youth in social confinement. Psicol. Saúde Doença 2020, 21, 624–632. [Google Scholar] [CrossRef]
  61. Soneson, E.; Puntis, S.; Chapman, N.; Mansfield, K.L.; Jones, P.B.; Fazel, M. Happier during Lockdown: A Descriptive Analysis of Self-Reported Wellbeing in 17,000 UK School Students during Covid-19 Lockdown. Eur. Child Adolesc. Psychiatry 2022, 1, 1–16. [Google Scholar] [CrossRef]
  62. Hossain, M.M.; Nesa, F.; Das, J.; Aggad, R.; Tasnim, S.; Bairwa, M.; Ma, P.; Ramirez, G. Global Burden of Mental Health Problems among Children and Adolescents during COVID-19 Pandemic: An Umbrella Review. Psychiatry Res. 2022, 317, 114814. [Google Scholar] [CrossRef] [PubMed]
  63. Orben, A.; Tomova, L.; Blakemore, S.J. The Effects of Social Deprivation on Adolescent Development and Mental Health. Lancet Child Adolesc. Health 2020, 4, 634–640. [Google Scholar] [CrossRef] [PubMed]
  64. Senft, B.; Liebhauser, A.; Tremschnig, I.; Ferijanz, E.; Wladika, W. Effects of the COVID-19 Pandemic on Children and Adolescents from the Perspective of Teachers. Front. Educ. 2022, 7, 7. [Google Scholar] [CrossRef]
  65. Hammerstein, S.; König, C.; Dreisörner, T.; Frey, A. Effects of COVID-19-Related School Closures on Student Achievement-A Systematic Review. Front. Psychol. 2021, 12, 4020. [Google Scholar] [CrossRef] [PubMed]
  66. Klootwijk, C.L.T.; Koele, I.J.; van Hoorn, J.; Güroğlu, B.; van Duijvenvoorde, A.C.K. Parental Support and Positive Mood Buffer Adolescents’ Academic Motivation During the COVID-19 Pandemic. J. Res. Adolesc. 2021, 31, 780–795. [Google Scholar] [CrossRef] [PubMed]
  67. Liu, Y.; Yue, S.; Hu, X.; Zhu, J.; Wu, Z.; Wang, J.; Wu, Y. Associations between Feelings/Behaviors during COVID-19 Pandemic Lockdown and Depression/Anxiety after Lockdown in a Sample of Chinese Children and Adolescents. J. Affect. Disord. 2021, 284, 98–103. [Google Scholar] [CrossRef] [PubMed]
  68. Wang, J.; Wang, Y.; Lin, H.; Chen, X.; Wang, H.; Liang, H.; Guo, X.; Fu, C. Mental Health Problems Among School-Aged Children After School Reopening: A Cross-Sectional Study During the COVID-19 Post-Pandemic in East China. Front. Psychol. 2021, 12, 5168. [Google Scholar] [CrossRef]
  69. Yeager, D.S. Social-Emotional Learning Programs for Adolescents. Future Child. 2017, 27, 73–94. [Google Scholar] [CrossRef]
  70. Domitrovich, C.E.; Durlak, J.A.; Staley, K.C.; Weissberg, R.P. Social-Emotional Competence: An Essential Factor for Promoting Positive Adjustment and Reducing Risk in School Children. Child Dev. 2017, 88, 408–416. [Google Scholar] [CrossRef]
Table
Table 1. Mental health for mixed and onsite implementation groups across the two time periods. p-values for interaction (type × time), time, and type effects.
Table 1. Mental health for mixed and onsite implementation groups across the two time periods. p-values for interaction (type × time), time, and type effects.
OnsiteMixedp
PrePost PrePostType
Effect		Time
Effect		Type × Time
Interaction
nMSDMSDnMSDMSD
Internalized
Problems		3480.420.350.350.345850.390.340.300.330.047<0.0010.243
Externalized
Problems		0.530.430.470.420.490.440.460.430.402<0.0010.155
Prosocial
Behavior		1.580.471.600.451.520.481.590.460.195<0.0010.079
Table
Table 2. SES for mixed and onsite groups across the two time periods. p-values for interaction (Type × Time), time, and type effects.
Table 2. SES for mixed and onsite groups across the two time periods. p-values for interaction (Type × Time), time, and type effects.
OnsiteMixedp
PrePost PrePostType
Effect		Time
Effect		Type × Time
Interaction
nMSDMSDnMSDMSD
Self-awareness3482.930.573.070.515822.980.643.170.620.051<0.0010.112
Self-regulation2.960.633.040.662.990.723.080.700.373<0.0010.806
Social
Awareness		3.220.613.260.553.190.633.340.590.459<0.0010.005
Interpersonal relationships3.190.543.270.513.250.583.410.580.006<0.0010.011
Responsible Decision
Making		3.230.613.290.583.280.653.420.630.026<0.0010.024
Table
Table 3. Academic Outcomes for mixed and onsite groups across the two time periods. Mean values, standard deviations, and p-values for interaction (type × time), time, and type effects.
Table 3. Academic Outcomes for mixed and onsite groups across the two time periods. Mean values, standard deviations, and p-values for interaction (type × time), time, and type effects.
OnsiteMixedp
PrePost PrePostType
Effect		Time
Effect		Type × Time
Interaction
nMSDMSDnMSDMSD
Academic
Motivation		3483.831.013.921.085843.680.973.751.010.0120.0010.676
Engagement in Learning 3.791.043.851.023.660.963.761.010.0700.0010.465
Academic Achievement3.751.013.821.013.550.893.740.960.020<0.0010.007
Table
Table 4. Mean differences between pre- and post-test, standard deviations, and F tests for the mental health domains within each group.
Table 4. Mean differences between pre- and post-test, standard deviations, and F tests for the mental health domains within each group.
DomainsGroupMSDF Test
Internalized Problems1−0.130.35F(1, 450.51) = 11.27, p < 0.001 a
2−0.050.26
Externalized Problems1−0.080.33F(1, 462.26) = 11.74, p < 0.001 a
20.010.25
Prosocial Behavior10.120.45F(1, 472.62) = 6.79, p = 0.009 a
20.030.36
Note: 1: 1 to 3 activities implemented online; 2: 4 to 7 activities implemented online; a Welch’s F test conducted.
Table
Table 5. Mean values, standard deviations, and F tests for SES domains within each group.
Table 5. Mean values, standard deviations, and F tests for SES domains within each group.
DomainsGroupMSDF Test
Self-awareness10.270.64F(1, 448.97) = 7.37, p = 0.007 a
20.140.47
Self-regulation10.100.56F(1, 467.95) = 0.49, p = 0.483 a
20.070.44
Social Awareness10.220.62F(1, 436.72) = 8.89, p = 0.003 a
20.080.44
Relationship Skills10.230.55F(1, 580) = 9.14, p = 0.003
20.100.45
Responsible
Decision Making		10.180.61F(1, 580) = 2.89, p = 0.090
20.110.51
Note: 1: 1 to 3 activities implemented online; 2: 4 to 7 activities implemented online; a Welch’s F test conducted.
Table
Table 6. Mean values, standard deviations, and F tests for academic outcomes within each group.
Table 6. Mean values, standard deviations, and F tests for academic outcomes within each group.
DomainsGroupMSDF Test
Academic
Motivation		10.070.77F(1, 582) = 0.00, p = 0.970
20.070.72
Engagement in Learning 10.110.75F(1, 582) = 0.05, p = 0.831
20.090.72
Academic
Achievement		10.220.75F(1, 582) = 0.51, p = 0.477
20.180.70
Note: 1: 1 to 3 activities implemented online; 2: 4 to 7 activities implemented online.
Table
Table 7. SES domains across the two time periods, considering the method. p-values for interaction (type × time), time, and type effects.
Table 7. SES domains across the two time periods, considering the method. p-values for interaction (type × time), time, and type effects.
OnlineOnsiteMixedp
PrePost PrePost PrePostType
Effect		Time
Effect		Type × Time
Interaction
nMSDMSDnMSDMSDnMSDMSD
Self-awareness2993.040.623.170.611442.890.643.030.592272.990.633.110.630.049<0.0010.895
Self-regulation3.050.663.140.672.910.703.060.712.970.792.990.760.089<0.0010.025
Social Awareness3.250.563.320.603.190.673.250.613.250.633.310.580.518<0.0010.946
Relationship Skills3.320.553.420.583.170.613.330.583.250.573.330.580.013<0.0010.843
Responsible
Decision Making		3.330.593.440.633.190.683.270.673.310.683.360.640.033<0.0010.431
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

dos Santos, M.F.; Simões, C.; Santos, A.C.; Lebre, P.; Grazzani, I. Does Online Implementation Make a Difference in the Effects of a Mental Health Curriculum at Schools? Int. J. Environ. Res. Public Health 2022, 19, 16990. https://doi.org/10.3390/ijerph192416990

AMA Style

dos Santos MF, Simões C, Santos AC, Lebre P, Grazzani I. Does Online Implementation Make a Difference in the Effects of a Mental Health Curriculum at Schools? International Journal of Environmental Research and Public Health. 2022; 19(24):16990. https://doi.org/10.3390/ijerph192416990

Chicago/Turabian Style

dos Santos, Margarida Frade, Celeste Simões, Anabela Caetano Santos, Paula Lebre, and Ilaria Grazzani. 2022. "Does Online Implementation Make a Difference in the Effects of a Mental Health Curriculum at Schools?" International Journal of Environmental Research and Public Health 19, no. 24: 16990. https://doi.org/10.3390/ijerph192416990

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop