Reproducibility and Validity of a Self-Administered Food Safety Assessment Tool on Children and Adolescent’s Risk Perception, Knowledge, and Practices

The present study aimed to verify the instrument’s reliability and validity for assessing children and adolescents’ risk perception, knowledge, and food safety practices in the school context. Moreover, it aimed to test the hypothesis that both application methods (paper and pencil (PAPI) and online) are valid. The instrument comprised three questionnaires and followed a strict protocol to combine online and PAPI models, resulting in five application forms. The sample consisted of 439 Brazilian students from 10 to 14 years old (y/o). The results related to reliability and validity indicated that the knowledge questionnaire presented adequate stability and discriminant validity coefficients. The self-reported practices questionnaire obtained acceptable coefficients of stability and internal consistency. Regarding risk perception data, it was observed that students attribute a low probability of Foodborne Diseases (FBD) outbreaks occurrence and low severity of possible symptoms. In addition, students demonstrated insufficient knowledge and inadequate practices on issues related to failures associated with the time and temperature of preparation, storage, and exposure of food, contamination of food, and consumption of unsafe food. In this context, the reproducibility and validity indices need to be interpreted and discussed correctly, and young people in food safety actions are a priority in facing FBD.


Introduction
World organizations are debating food safety as it plays a primary role in the global population's health. For example, in the document that defines the global strategies for food security (2022-2030), the World Health Organization (WHO) states that achieving food and nutrition security without food safety is not feasible. Another aspect highlighted in the document was the vulnerability of specific population groups such as children. In these cases, rates of Food Insecurity (FI) and Foodborne Diseases (FBD) outbreaks always stand out compared to other groups [1,2].
Prioritizing actions that protect children and adolescents can be crucial to minimizing the effects of FI exacerbated by the COVID-19 pandemic. In Brazil, with the COVID-19 pandemic, extreme nutritional status emerged, which must be discussed from different perspectives [3]. In this context, during 2021 and 2022, the VIGISAN survey showed that increasing degrees of moderate and severe FI in households were proportional to the presence of residents under 18 y/o. In addition, water insecurity, strongly associated with

Materials and Methods
A cross-sectional and quantitative study was conducted between August/2021 and August/2022 in the Federal District-Brazil and approved by the Ethics Committee of the Faculty of Health Sciences of the University of Brasília-CEP/FS UnB (CAAE No 02033218.0.0000.0030). The sample consisted of 439 students aged 10 to 14 y/o from public (n = 4) and private (n = 1) schools, with the participation of a public school located in a rural area. The schools were conveniently selected, and the students' participation was conditioned to the presentation of the consent terms signed by parents/legal guardians and free and informed consent signed by the students.
This research continued the study by Batista et al. [34], who developed the theoretical steps proposed by Pasquali [35] to create a research instrument. The instrument is a tool composed of three questionnaires that assess risk perception, knowledge, and food safety practices (Supplementary Material Figure S1). It presented the guarantees of content validity, understanding, and apparent validity (credibility). The "risk perception" questionnaire considers the school environment, given the possibility of simultaneously evaluating the student and the reality observed in the school [34].
Batista et al. could not complete the empirical and statistical steps due to the advent of the SARS-CoV-2 pandemic. However, these steps are necessary to obtain accurate, valid, and interpretable data. Furthermore, it is essential to conclude them due to the scarcity of literature that addresses the perception of food safety risk in the target audience, as well as for being the only instrument in Portuguese that proposes to evaluate factors and constructs related to the theme with children and adolescents [34].
Therefore, two steps were necessary for this study: (i) empirical procedures, with the application of the instrument considering the online and physical environments; and (ii) statistical procedures, assessing instrument reliability and validity.
The first step included the sample definition, test instructions, and instrument administration. Then, five applications were performed according to the order of the instrument presentation: (i) online then PAPI (n = 24), (ii) PAPI then online (n = 21), (iii) Online then online (n = 22), (iv) PAPI then PAPI (n = 31), and (v) a single application-online or PAPI (n = 341). All steps were developed in the school and the virtual environment, with the support of the Google Form platform.
The second step was necessary to conduct a more in-depth assessment of the instrument's measurement properties and guarantee the quality of the results [35]. For this, stability and internal consistency criteria were adopted to determine the instrument's reliability. Validity was assessed using discriminant validity.
The first four application methods were performed in the first step of the study to evaluate the stability coefficient using the test-retest method. Thus, it was possible to verify the degree to which similar results were obtained at two different times [36]. Furthermore, the fifth application was performed to verify the internal consistency of the knowledge questionnaires and "self-reported practices"-another reliability measure. Therefore, the sample to determine this measure consisted of the first application carried out in four phases and the final fifth stage (n = 439).
To obtain the discriminant validity of the knowledge questionnaire, we invited dietitians and undergraduate nutrition students from the last semester to compose the group of nutrition specialists (n = 41). The target audience comprised the lay group, using the consistency analysis sample (n = 439). The procedure must demonstrate that the instrument items measure the desired content domain [36]. To this end, a comparison was made between the two groups.
The developed steps are summarized in Figure 1. The "risk perception" construct coefficients were not determined, as it is considered that the person assesses the risk in the face of the current situation. The perception is so subjective that it can change quickly, impacting the reproducibility of the results. The risks in the present are assessed immediately, so we must consider the impact bias/affect heuristic [37].
Judging risk is directly related to people's feelings about risks, and their acceptance [28,[38][39][40]. It considers social constructions, cognitive factors, beliefs, and motivations [41][42][43]. Other characteristics, such as individual differences and demographic factors, can also influence comparative perceptions of the magnitude of risks [44]. Therefore, assessing the reproducibility of a subjective, multidimensional construct anchored in several factors becomes challenging.
Descriptive analyses of mean, standard deviation, percentage distribution of socioeconomic data, risk perception, knowledge, and food safety practices were performed. The "risk perception" construct coefficients were not determined, as it is considered that the person assesses the risk in the face of the current situation. The perception is so subjective that it can change quickly, impacting the reproducibility of the results. The risks in the present are assessed immediately, so we must consider the impact bias/affect heuristic [37].
Judging risk is directly related to people's feelings about risks, and their acceptance [28,[38][39][40]. It considers social constructions, cognitive factors, beliefs, and motivations [41][42][43]. Other characteristics, such as individual differences and demographic factors, can also influence comparative perceptions of the magnitude of risks [44]. Therefore, assessing the reproducibility of a subjective, multidimensional construct anchored in several factors becomes challenging.
Descriptive analyses of mean, standard deviation, percentage distribution of socioeconomic data, risk perception, knowledge, and food safety practices were performed.

Results
The study included 439 students aged 10 to 14 y/o (female = 54.7%; male = 45.3%) with a mean age of 12.1 years (±1.36), enrolled in public (49.2%) and private (50.8%) schools. The entire sample (n = 439) was used to analyze internal consistency and discriminant validity. A subsample (n = 98) was used to verify the stability coefficient. For this reason, the instrument was applied at an interval of 11 days. The second sample comprised 64.3% female and 35.7% male, with a mean age of 12.1 years (±1.8). The samples and their respective statistical analyses are described in Figure 2.

Results
The study included 439 students aged 10 to 14 y/o (female = 54.7%; male = 45.3%) with a mean age of 12.1 years (±1.36), enrolled in public (49.2%) and private (50.8%) schools. The entire sample (n = 439) was used to analyze internal consistency and discriminant validity. A subsample (n = 98) was used to verify the stability coefficient. For this reason, the instrument was applied at an interval of 11 days. The second sample comprised 64.3% female and 35.7% male, with a mean age of 12.1 years (±1.8). The samples and their respective statistical analyses are described in Figure 2. The results of the stability and internal consistency coefficients are presented in Table  1. In the test of discriminant validity for the knowledge questionnaire, on average, the group of experts (9.76, ±0.99) presented higher knowledge than the lay group (6.55, ±1.73). This difference (−3.21) was significant with p = 0.000.
The application of the instrument (n = 439) resulted in data on risk perception (Tables  2 and 3), knowledge (Table 4), and self-reported practices (Table 5) of food safety.
In the first part of the instrument, the risk perception questions consider its two dimensions-probability and severity. As for the first, it can be seen in Table 2 that most students identified as "very low" or low the chances of themselves (72.7%) or a colleague (61%) becoming sick due to having eaten a snack served at the school where they study. As for severity, most students believe that they would present symptoms with no or low severity if they became ill after eating food at school. The results of the stability and internal consistency coefficients are presented in Table 1. In the test of discriminant validity for the knowledge questionnaire, on average, the group of experts (9.76, ±0.99) presented higher knowledge than the lay group (6.55, ±1.73). This difference (−3.21) was significant with p = 0.000.
The application of the instrument (n = 439) resulted in data on risk perception (Tables 2 and 3), knowledge (Table 4), and self-reported practices (Table 5) of food safety.
In the first part of the instrument, the risk perception questions consider its two dimensions-probability and severity. As for the first, it can be seen in Table 2 that most students identified as "very low" or low the chances of themselves (72.7%) or a colleague (61%) becoming sick due to having eaten a snack served at the school where they study. As for severity, most students believe that they would present symptoms with no or low severity if they became ill after eating food at school. Table 2. Relative and absolute frequencies of items refer to the "probability" dimension (A1 and A2) and the perception of lethality (A3) of the risk perception questionnaire.

Reliability and Validity Coefficients
Studies that investigate the invariance of measures in different ways of applying an instrument use online and face-to-face methods [50][51][52][53]. The multiple applications make it possible to observe if there is a consistent performance in different situations, according to values established in the literature. However, it becomes crucial to consider the value's meaning within the research context [54].
Singh and Sagar argued that it is necessary to compare results obtained traditionally and validation studies to establish psychometric properties to reach conclusions from online methods. For this, using the same study sample or sampling frame is recommended. In this way, it will be possible to show whether or not there are significant differences between the methods [55].
The "knowledge" and "practices" constructs obtained good and excellent (KR-20 and ICC > 0.7) levels of stability [46], except for the "PAPI then online" application of the "knowledge" construct, which showed a low level of stability (KR-20 = 0.452). Since this application was used as a complementary form of analysis, it is considered that adequate stability coefficients were obtained for the construct's "knowledge" and "practices". Thus, it is observed that in different environments the results are similar.
Other studies that did not necessarily use the same coefficients to assess the stability of instruments but tested the same forms of application also found similar results between different models. Different authors have supported the comparability of the reliability of online and PAPI questionnaires [56][57][58].
In their research, Van de Looij-Jansen and de Wilde evaluated the differences in the responses of mental health and behavior indicators by an ANCOVA test. No significant differences were found between the responses online and PAPI methods. However, it is essential to emphasize that, in the mentioned study, each method was applied to a different group of subjects, differently from our research [59]. Likewise, Gorrasi et al., applying the two methods to different groups, evaluated whether the two approaches could influence the results of the three questionnaires. The results revealed no significant differences in the Orthorexia Nervosa and Muscle Dysmorphia scales [60].
In contrast, in agreement with this study, Ward et al. examined differences between two data collection techniques with the same group of individuals. Among the evaluated scales, three showed no significant differences in the scores of the two approaches. Therefore, the literature presents data that corroborates the current study's findings [61].
Regarding the "knowledge" construct, according to Cronbach's alpha coefficient, a value of 0.370 was obtained, indicating low internal consistency. When considering construct validity, which assesses the extent to which the set of items represents the construct to be measured [36], it is noted that the knowledge questionnaire encompasses different types of knowledge in food safety, such as (i) hand hygiene, (ii) cross-contamination, (iii) adequate food storage, (iv) consumption of raw or undercooked foods, v) exposure time of food at room temperature, (vi) unsafe foods, (vii) the presence of vectors and urban pests, (viii) cleaning food consumed raw, and (ix) pesticides, thus affecting the index. Therefore, the heterogeneity of knowledge regarding food safety can lead to low Cronbach's alpha coefficients.
Corroborating the previous finding, other studies that obtained internal consistency indexes below the recommended (0.45 and 0.55) consider the findings adequate given the limited number of items, a wide range of knowledge, and different addressed concepts [62,63]. For example, in a study of cultural adaptation of an instrument to assess the quality of life of Brazilian adolescents (12 to 16 y/o), Teixeira et al. obtained internal consistency values (Cronbach's alpha) that ranged from 0.12 to 0.73. The authors pointed out that possible reasons for the low values include the presence of subjective constructs and heterogeneous content, which is in line with the present study [64].
As for the coefficient of the practice's questionnaire (Cobrach's alpha = 0.612), the value is considered acceptable [48,49]. Nunnally suggests that in the initial stages of the research, values from 0.7 to 0.5 are sufficient [65]. Van Griethuijsen et al. consider an acceptable reliability level when values range from 0.6 to 0.7 [66]. Thus, this study's value is adequate. The exclusion of items with low internal consistency in the knowledge and practice questionnaires was tested to verify the increase in reliability. However, no acceptable values were found with the exclusion of items.
Finally, concerning the validity of the knowledge questionnaire, the results suggest that the items presented high discriminant validity. Thus, they are adequate in assessing knowledge about food safety. Guadagnin et al. use this assessment to validate a questionnaire assessing knowledge in nutrition and its usefulness in assessing changes in knowledge among participants in a Nutrition Education Program applied in the workplace. In this study, the expert group scores were significantly higher than those of the lay group in all domains [67].

Application of the Instrument
In the present study, it was not possible to assess the risk perception since the sanitary risk of the schools was not verified. However, it is inferred that low probability and severity results may be due to the students' feelings about the school environment [28,38]. For example, the study by de Andrade et al. evaluated consumers' risk perception and optimistic bias in commercial restaurants. The results indicated their optimism about the restaurant where they were interviewed, attributing similar risk to food prepared at home and in the studied restaurants. The authors pointed out that consumers may have incorporated a feeling of affection and identity into the place, corresponding to having their meals at home [6].
Regarding the perception of lethality, it is observed that 34.7% of the students consider a person's chance of dying when consuming contaminated food as very low or low. A study developed by Ovca et al. carried out with individuals between 10 and 12 years old pointed out that most participants agreed that food poisoning could be fatal, indicating a high perception of lethality, in contrast to our results. The study by Ovca et al. was developed in Slovenia, and its result can be attributed to the country's degree of development [68]. According to the Organisation for Economic Co-operation and Development (OECD) and the Programme for International Student Assessment (PISA), this country has one of the highest socioeconomic, social, and cultural statuses [69].
In the knowledge questionnaire, the pattern of the results showed a mean correctness of 60%, a minimum average of 9.1%, and a maximum of 100%. Other studies also found median results ranging from 65.8% to 72% [21,31,68]. The study carried out in Australia with students aged 11 and 18 presented an average of 42% [23].
The results presented above are in line with what is discussed by Syeda et al. [26]. The authors showed that previous studies with children, young people, and university students in Europe, Australia, Canada, and the United States suggest a lack of knowledge, interest, and perception of susceptibility to FBD, pointing up the importance of working with these individuals.
It is observed from the knowledge questionnaire that 43.7% of students are unaware of the consequences of violating the time versus temperature binomial and that 29.4% carry out inappropriate practices related to the theme. In the study by Ovca, Jevšnik and Raspor, only 48.8% of respondents reported storing leftover food in the refrigerator [68]. Similar results were found with food handlers, with inadequate practices related to this binomial during food exposure and insufficient knowledge about temperature control [6,70]. It should be noted that the violation resulting from the lack of control of the time versus temperature binomial in food production is considered the leading global risk factor for FBD [71]. Therefore, understanding the subject is essential.
It should be noted that the issue of unsafe food needs more attention. A significant portion of the students missed questions about the consumption of raw eggs or eggs with soft yolk (57.4%), consumption of fruits without prior cleaning (40.8%), and consumption of fruits and vegetables with pesticides (43.1%). As for practices, 29.9% of students consume raw eggs or eggs with soft yolks with some frequency. Students demonstrated knowledge about the implications of consuming spoiled foods (95.9%). However, 59.2% believe that removing the moldy part of the bread reduces or eliminates the risk of getting sick, and 19.8% reported consuming this food after removing the spoiled part.
It is also noted that 72.9% believe that unsafe foods always have undesirable characteristics, and 48.1% consume foods with expiry dates as long as they look good. In the studies by Batista et al. and Ovca et al., the students considered the appearance and smell of food as reliability indicators for its consumption [34,72]. Cheng et al. pointed out that some students thought that expired foods could be consumed after boiling or heating or while they looked good [73]. Efforts should be focused on this theme, as contamination by pathogenic microorganisms does not change the characteristics of food.
It is expected that the studied age group has cognitive prerequisites that allow them to consider that appearance differs from reality and that there are invisible particles (microorganisms) since such cognitive skills are developed in the infants [74][75][76][77]. This result may be related to the possible difficulty of human beings attributing risk to what cannot be seen, as in the case of microorganisms. However, it is necessary to assess more broadly how these skills are being worked with students.
Another relevant result (67% of the students) indicates the lack of knowledge of the correct practice of how to sanitize raw vegetables. Other studies report insufficient knowledge about vegetable hygiene [34,78]. The results of Burke and Dworkin indicate that 86% of the students believed that it was sufficient to rinse with water a salad contaminated with raw chicken juice [78].
Notably, these foods are related to an increasing number of FBD outbreaks worldwide, with a high occurrence of Salmonella [79,80]. A study developed in China suggested that consuming raw vegetables represents a potential source of human infection by salmonellosis [81]. Outbreaks were related to the consumption of melon [82], watermelon [83], apple, tomato, celery, lettuce, radish, and orange [84,85]. The correct hygiene procedure effectively contributes to mitigating outbreaks related to these foods.
The third part of the instrument evaluated students' self-reported food safety practices. As for hand hygiene, only 42.4% of the students reported consistently washing them before a meal. Therefore, it is necessary to reflect and investigate whether the public knows the importance of carrying out this practice. Simply recognizing that the process must include soap/detergent may not be enough to support educational actions. Contrary to these results, Osei-Tutu et al. found that many students (70%) indicated hand hygiene before meals. In this study, handwashing education is part of the School Health Education Program in conjunction with the WASH Program at the school in the studied district [10].
The data referring to question "nine" point to students' expressive participation in preparing food or meals in the home environment (96.4%). Other studies presented comparable data, such as by Ovca [21,28,68]. In Ovca et al., 21.5% of children and adolescents dealt with potentially dangerous items such as meat, fish, and green leaves [68]. Therefore, investigating which types of food are often involved in their daily handling practices is critical.
The study by Thakadu et al. evaluated the hygiene education of students aged between eight and 16, with the participation of teachers. These professionals highlighted challenges for hygiene education, such as time constraints to adequately address the topic, inadequate prior knowledge about health education, and lack of health material to guide them [86]. Given the observed results and reflecting on the provision of knowledge in schools, the methodological aspects in this environment must be considered since the pedagogical body is responsible for ensuring food safety, as it is a shared responsibility [87]. Therefore, the importance of training the teaching staff is reinforced.
When considering adult handlers and consumers, unsafe handling practices are often perceived, despite their taking place with an acceptable level of knowledge [6,9,16,88]. It is suggested that if the theme is worked in the appropriate cognitive phase, considering psychological aspects such as risk perception, the behaviors would be well established, with sufficient knowledge and good manipulation practices.

Conclusions
This study completed the validation steps of the instrument developed by Batista et al., thus presenting guarantees of reliability (stability and internal consistency) and validity (discriminant validity) [34]. The internal consistency result of the knowledge questionnaire does not invalidate the quality of the instrument, given the nature of the studied variable. It is necessary to reflect on the nature of the investigated constructs and the meaning of the reliability parameters in the real context of the research. The general guidelines established in the literature do not consider the specificities and subjectivities included in the studies.
The study confirms the hypothesis that the different application methods (online and PAPI) are valid. The potential use of the online version stands out, as it allows for faster data return, cost reduction, and convenience bias. This practical value is being evidenced in the ongoing COVID-19 pandemic.
Considering the data presented and studies aimed at the adult audience, it is inferred that if the theme is worked in the appropriate cognitive phase, considering psychological aspects such as risk perception, the behaviors would be well established, with sufficient knowledge and good manipulation practices.
Using a valid and reliable instrument, well-formulated strategies can be carried out considering the young audience, a fundamental link in the food chain to mitigate Foodborne Diseases. In this way, fighting food insecurity arising from the lack of food safety is essential. Finally, the limits of this study are represented by the nature of the investigated constructs, the lack of critical literature that allows the results to be reflected broadly, and the social desirability when responding.