Next Article in Journal
Stress and the City: Mental Health in Urbanized vs. Rural Areas in Salzburg, Austria
Previous Article in Journal
Integrated Climate Change Mitigation and Public Health Protection Strategies: The Case of the City of Bologna, Italy
Previous Article in Special Issue
The Brief Health Literacy Scale for Adults: Adaptation and Validation of the Health Literacy for School-Aged Children Questionnaire
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Evaluation of the Psychometric Properties of the Digital Health Literacy Instrument (DHLI-BrA) for Use in Brazilian Adolescents

by
Mariane Carolina Faria Barbosa
1,*,
Ana Luiza Peres Baldiotti
1,
Ramon Targino Firmino
2,
Saul Martins Paiva
1,
Ana Flávia Granville-Garcia
3 and
Fernanda de Morais Ferreira
1
1
Department of Pediatric Dentistry, Federal University of Minas Gerais (UFMG), Belo Horizonte 31270-901, MG, Brazil
2
Department of Dentistry, Federal University of Campina Grande (UFCG), Patos 58708-110, PB, Brazil
3
Postgraduate Program in Dentistry, State University of Paraíba (UEPB), Campina Grande 58429-500, PB, Brazil
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2024, 21(11), 1458; https://doi.org/10.3390/ijerph21111458
Submission received: 30 August 2024 / Revised: 9 October 2024 / Accepted: 17 October 2024 / Published: 31 October 2024
(This article belongs to the Special Issue Health Literacy: Measurements, Interventions and Evaluation)

Abstract

:
This study evaluated the psychometric properties of the Portuguese version of Digital Health Literacy Instrument for Brazilian Adolescents (DHLI-BrA). Two hundred and sixty Brazilian adolescents answered the DHLI-BrA and the Brazilian version of quality-of-life and health literacy instruments: WHOQOL-Bref, eHEALS-BrA, NVS-BR, and REALMD-20. Then, they answered a questionnaire on sociodemographics, health, Internet access, and digital health aspects. The data collection was conducted between September and December of 2022.The statistical test assessed internal consistency, stability, discriminant and convergent validities, exploratory factor analysis (EFA), and confirmatory factor analysis (CFA). Item response theory (IRT) analysis using 2PL was applied to performance-based DHLI-BrA. The DHLI-BrA self-reported questions demonstrated almost perfect internal consistency (α and ω = 0.83) and good stability (ICC = 0.906; 95% CI: 0.75–0.95). In EFA and CFA, the best-adjusted model was composed of six factors (χ2 = 229.173 (df = 174, p = 0.003), CFI = 0.944, TLI = 0.933, RMSEA = 0.035, and RMSR = 0.047). The performance-based DHLI-BrA demonstrated moderate internal consistency (α = 0.57 and KR20 = 0.56) and good stability (ICC = 0.86, 95% CI: 0.76–0.92). In EFA and CFA, the best-adjusted model was composed of a single factor (χ2 = 17.901 (df = 14, p = 0.2113), CFI = 0.952, TLI = 0.927, RMSEA = 0.033, and RMSR = 0.038). IRT analyses revealed item discrimination ranging from −0.71 to 1.83 and difficulty from −1.53 to 1.02. Convergent validity of the self-reported DHLI-BrA was obtained by its correlation with the eHEALS-BrA (r = 0.45) and REALMD-20 (r = 0.19), besides the performance-based DHLI-BrA with its correlation with the NVS (r = 0.47) and REALMD-20 (r = 0.44). The DHLI-BrA demonstrated adequate psychometric properties to measure the self-reported, performance-based digital health literacy of Brazilian typically developing adolescents.

1. Introduction

Technological advances and increased access to the Internet have brought about significant changes to society and transformed people’s lives [1,2]. Digital devices, such as smartphones, notebooks, and tablets, have become healthcare tools with the potential to improve quality of life, disseminate information, and provide access to health services [3,4].
It is estimated that 82.7% of Brazilian households have Internet access, and adolescents are the population group that most access the Internet (90.2%), mainly via smartphones [5]. Therefore, the Internet is part of their lives, and they have a high level of familiarity and ability to use information and communication technologies [4,6,7].
Adolescents have unique characteristics as they undergo significant physical, emotional, and mental maturation, which can lead to questions about their body and health [2,7,8]. Thus, the Internet is an attractive and accessible resource for teenagers to search for information and self-manage their health [4,7,9]. However, the expansion of health information in the media, especially the dissemination of false or misleading content, jeopardizes the decision-making and self-management processes, leading to the development of harmful health beliefs [10,11,12].
It is known that there is an association between high levels of health literacy (HL) and assertive health behaviors in adolescents [13]. Given the high rate of digital insertion in this age group, it is important that adolescents have proper digital health literacy (DHL), so they can search, select, evaluate, interpret, and use health information found online (health 1.0) [14]. They should also have interactive skills such as posting health-related messages on the web and using telehealth applications and services (health 2.0) [15,16]. Digital Health Literacy Instrument (DHLI), developed in the Netherlands in 2017, is an instrument for measuring DHL, including the complete spectrum of eHealth skills (Health 1.0 and Health 2.0) and actual performance-based competencies [15,17]. DHLI has previously been tested for adolescents [4,18] and university students of some countries [19,20,21,22,23], demonstrating an effective mensuration of DHL, low cost, and easy application. This instrument was recently cross-culturally adapted for Brazilian adolescents [17]. However, it is necessary to verify the validity of this instrument for measuring this construct. Thus, this study aimed to evaluate the psychometric properties of the Portuguese version of DHLI-BrA for the public.

2. Materials and Methods

2.1. Study Design and Population

This is a methodological study evaluating the psychometric properties of the Digital Health Literacy Instrument adapted for Brazilian Adolescents (DHLI-BrA) [17]. It was conducted on a sample of 260 adolescents (aged 13 to 19) enrolled in five public schools in the Brazilian city of Belo Horizonte. The sample size was based on the recommendation of 2 to 20 individuals per instrument item to evaluate the properties [23]; then, considering 10 individuals per item, there is a minimum sample of 210 participants. Given the possibility of possible losses, the sample size was increased by 20% to 252 adolescents. The schools were randomly selected, considering their distribution throughout the municipality and their result in IDEB-2019 (Basic Education Development Index), a teaching quality indicator for Brazilian public schools. Five schools were selected, two among the six schools listed with the best performance in IDEB and three listed among the six as the worst-performing. In addition, the schools were distributed in 5 different locations in the municipality.

2.2. Eligibility Criteria

Literate adolescents of both sexes, Brazilian Portuguese native speakers, with access to the Internet, and who were present on the days of data collection were included. Adolescents aged 12 were excluded due to the 1998 Children’s Online Privacy Protection Act, which determines the minimum age of 13 for creating accounts on social networks and using digital services. In addition, adolescents who presented self-reported or school-reported (vision, hearing, or cognitive) problems that impaired their participation were excluded.

2.3. Instrument

The Digital Health Literacy Instrument measures digital health literacy. It is a self-reported scale with 21 items that measure the broad spectrum of the eHealth concept, which includes the use of health information on the Internet (health 1.0) and the use of recent applications with interactive technologies (health 2.0). Also, the instrument has 7 items that measure DHL based on the individual’s practical performance [15].
The original instrument is organized into seven skills (factors), with three items each: (1) operational skills—how to use the computer and Internet browsers, (2) navigation skills—how to navigate and find your way around the Web, (3) skills to search for information using correct search strategies, (4) skills to assess information reliability, (5) skills to determine online information relevance, (6) skills to include self-generated content, and (7) skills to protect and respect online privacy [15].
Self-reported items are answered using a 4-point scale (1 to 4), with options ranging from “very easy” to “very difficult” and from “never” to “almost always”, which score is performed invertedly: very easy/never = 4, fairly easy/sometimes = 3, reasonably difficult/often = 2, and very difficult/almost always = 1. The total score is obtained by an average of all responses, with higher scores representing a higher level of DHL. In addition, it is also possible to calculate scores for each of the instrument’s seven skills by averaging the items in each skill [15,17].
The performance-based items have five response options, with only one correct option (score = 1), three incorrect options (score = 0), and one “I don’t know” option (score = 0). The calculation of the total DHL score based on performance is done by adding up the correct answers [15,17]. The full version of the DHLI instrument in Brazilian Portuguese adapted for adolescents is available in the Supplementary Materials of this manuscript.

2.4. Data Collection

The data collection with adolescents and their guardians/caregivers was conducted between September and December of 2022. The guardians/caregivers responded to a semi-structured questionnaire about their sociodemographic aspects (age, kinship, education, and income) and about the adolescents’ information (birth order, changes in health, and medication use).
The collection took place at three separate times. Initially, the adolescents were instructed to answer the DHLI-BrA and Brazilian version of the eHealth Literacy Scale for Adolescents (eHEALS-BrA) instruments to measure digital health literacy. They also answered a semi-structured questionnaire to determine their demographic characteristics (gender, age, and education); general health (physical activity and self-assessment of general and oral health); access to the Internet (where the access happens, the main mean of access, use of mobile data, frequency of access, and use of social media); self-assessment of Internet skills; and search behavior for health information online (researched for a doctor/dentist, whether they followed influencers’ recommendations, whether they used the Internet to read/search for health information, appointments scheduling, use of health apps, research for symptoms, and questions to health professionals).
The eHEALS-BrA is a DHL measurement instrument composed of eight self-reported items. The answer options are organized on a 5-point Likert scale (1 to 5), with the total score ranging from 8 to 40 points and higher scores representing higher levels of DHL [14,24].
Secondly, the adolescents responded to the quality-of-life questionnaire, WHOQOL-Bref, and to the health literacy measurement instruments: NVS and REALMD-20. The WHOQOL-Bref is an abbreviated version of the World Health Organization Quality of Life (WHOQOL-100), an instrument for measuring the quality of life recommended by the World Health Organization. It consists of 26 questions with answer options organized on a Likert scale (1 to 5). The instrument covers 4 domains: physical, psychological, social, and environmental [25,26]. Higher scores represent a better quality of life for each instrument’s domain [25,26].
The Newest Vital Sign (NVS-BR) was used to measure HL, an instrument composed of 6 items that assess reading comprehension and numeracy through a simulation with the information on an ice cream label. Each item has a correct answer (1 point), and the total score can vary from 0 to 6 points, in which higher scores represent a better HL [27]. The 20-item Rapid Estimate Adult Literacy in Medicine and Dentistry (REALMD-20) consists of 20 medical and dental terms and assesses the ability to read and pronounce terminologies. Each correctly pronounced word receives 1 point, with higher scores representing a higher HL level [28].
The third moment of data collection happened after two weeks from the DHLI’s first application. Then, 25% of the adolescents were randomly selected to answer the DHLI-BrA instrument again to verify the stability of the instrument. All data collection stages were carried out individually and in a private space within the schools. The instruments were self-administered in printed version; only the NVS and REALDM-20 instruments were applied in an interview format by a single researcher, following the recommended methodologies for each instrument.

2.5. Statistical Analysis

Cronbach’s alpha coefficient (α) and McDonald’s omega (ω) were performed as measures of the internal consistency of self-related DHLI-BrA [29,30]. For performance-based items, the reliability was assessed using Cronbach’s alpha coefficient (α) and Kuder–Richardson KR20 [31,32]. Alpha was categorized as follows: 0.81 to 1.0—almost perfect, 0.61–0.80—substantial, 0.41 to 0.60—moderate, 0.21–0.40—reasonable, and 0–0.21—small [29]. The instrument stability was assessed by test–retest reliability after a period of two weeks by using the intraclass correlation coefficient (ICC). The values were categorized as follows: ≤0.40—weak, 0.41–0.60—moderate correlation, 0.61–0.80—good, and 0.81–1.00—excellent [33].
Exploratory factor analysis (EFA) was performed to evaluate the instrument’s dimensionality. The appropriateness of using factor analysis on the data set was assessed using the Kaiser–Meyer–Olkin test (>0.60) and Barlett’s sphericity test (p < 0.05). A simple factor solution structure based on reported eigenvalues above 1.0 was used, and the varimax rotation method was performed. The factor loadings were considered: >0.40 acceptable, >0.55 good, >0.63 very good, and >0.71 excellent [34].
Confirmatory factor analysis (CFA) was performed to test the hypothesis based on a theoretical framework and empirical research. The model fitness was evaluated using five indices: the goodness of fit of the chi-square test (χ2), the Comparative Fit Index (CFI), the Tucker Lewis index (TLI), the mean square error of approximation (RMSEA), and the standardized root mean square residual (SRMR). For χ2, p-values > 0.001 indicated a good fit. For the CFI and TLI, values ≥ 0.95 were considered excellent fits, while values between 0.90 and 0.95 indicated an acceptable fit. RMSEA values ≤ 0.05 indicate excellent model fit, while values between 0.05 and 0.08 suggest a good fit and values ≥ 0.10 a poor fit. For the SRMR values, <0.08 indicates a good fit [33,35].
The item response theory (IRT) analysis was conducted in RStudio version 4.2. The analysis was conducted after checking of the dimensionality structure (EFA and CFA). For performance-based items, due to the dichotomous nature of the responses, we used the two-parameter logistic (2PL) IRT model with the mirt package [36]. In this model, two parameters are evaluated. The first is the discrimination (α) parameter, which reflects how strongly the item is related to the latent construct, with values typically ranging from 0 to 3. Higher values indicate a stronger relationship between the item and the latent trait (theta). The second parameter is the difficulty (b) parameter, which indicates the points along the latent trait continuum where each response option has a 50% chance of endorsement. The 2PL model analyzed the parameters of the subjects and items that were shown in an item map. The interpretation of the infit/outfit is influenced by the sample size. In the present study (n = 260), values between 0.7 and 1.3 were considered acceptable [37,38].
Construct validity evidence was determined based on discriminant and convergent validity. Convergent and discriminant validity were investigated using the Spearman correlation test with sociodemographic variables (age and average family income), digital health literacy (eHEALS), and health literacy (NVS and REALMD-20). Furthermore, convergent validity was investigated through its correlation with quality of life (WHOQOL-Bref) and by comparing total scores with the variables self-assessment of general health and skills to use the Internet, frequency of Internet access, and behavior in searching for health information and using digital services (Mann–Whitney and Kruskal–Wallis U tests, p < 0.05).
All statistical tests were performed with SPSS Statistics 21.0 program (SPSS Inc., Chicago, IL, USA), Mplus software (Muthén & Muthén, Version 8.2, Los Angeles, CA, USA), and RStudio Version 4.2.

2.6. Ethical Aspects

Prior to this study, the authors of the original instrument were contacted and authorized its implementation [15]. This study was also approved by the Human Research Ethics Committee of the Federal University of Minas Gerais (#CAAE: 58603022.8.0000.5149). In accordance with the recommendations of Resolution 466/2012 of the National Health Council and the Declaration of Helsinki, all participants were informed about the objective of the study and signed an informed consent form (guardians and adolescents over 18 years of age) and an informed assent form (adolescents under 18 years of age).

3. Results

Two hundred and sixty adolescents with a mean age of 15.63 (±1.84) participated in this study: one hundred and forty-two (54.6%) were female and one hundred and twelve (45.4%) were male. A total of 139 adolescents (53.7%) attended elementary school II (up to 8 years of schooling), and 122 (48.0%) declared themselves as mixed race. In relation to family aspects, the mother was mostly the family provider (78.5%), and the average family income was BRL $2780.69 (USD $526.64). The average age of parents/tutors was 43.35 years (± 8.14), whose education was equal to or greater than 9 years of formal study (79.1%).
The average self-reported DHLI-BrA total score was 3.02 (±0.37; 2–3.86), and the average administration time was 7.59 min. The 21 self-reported items presented adequate internal consistency based on Cronbach’s alpha = 0.83 and Mc’Donald’s omega = 0.83. Table 1 describes the parameters of the 21 self-reported items from the DHLI-BrA, demonstrating the psychometric quality of the items and indicating that no item should be excluded from the instrument. The parameters presented include the average response value for each item on the scale, the scale variances, the correlation between each item and the total scale score (excluding that specific item), and the Cronbach’s alpha values if that specific item were to be removed.
Test–retest reliability analysis demonstrated excellent reproducibility [ICC = 0.906 (95% CI: 0.75–0.95, p < 0.001)]. Preliminary tests demonstrated that the data were adequate to perform an exploratory factor analysis of the 21 self-reported items: KMO = 0.84 and Barlett’s test of sphericity significance (p < 0.001).
In the EFA, six factors were found with values ≥ 1.0. However, in the original instrument, seven factors were identified for the 21 self-reported items. As to the theoretical framework, confirmatory factor analysis was conducted for the seven- and six-factor solutions, and the best-adjusted model was the one composed of six factors: χ2 = 229.173 (df = 174, p = 0.0032), CFI = 0.944, TLI = 0.933, RMSEA = 0.035 (CI: 0.021–0.047), and SRMR = 0.047. The factor loadings were >0.40 (Figure 1). Table 2 presents Cronbach’s alpha for each scale (0.53–0.76) and the correlations between the factors.
For the seven performance-based items, the average total score was 4.15 (±1.72, 0–7), and the average administration time was 10:28 min. Satisfactory reliability was observed (Cronbach’s alpha = 0.57 and KR20 = 0.56), and the detailed parameters of the DHLI-BrA performance-based items are described in Table 3, which demonstrate that all performance-based items should be maintained in the instrument. The assessment of test reliability retesting of the instrument demonstrated adequate reproducibility [ICC = 0.86 (95% CI: 0.76–0.92, p < 0.001)]. For the EFA evaluation, it was preliminarily observed that the data were adequate for its performance: KMO = 0.68 and Barlett’s sphericity test significance (p < 0.001).
In the EFA of the seven performance-based items, two factors were obtained with values ≥1.0. Based on the theoretical framework, confirmatory factor analysis was carried out for the one- and two-factor solutions. However, the two-factor solution presented an unsatisfactory grouping of questions. Then, the one-factor solution was better adjusted, with excellent indices, as demonstrated: χ2 = 17.901 (df = 14, p = 0.2113), CFI = 0.952, TLI = 0.927, RMSEA = 0.033, and SRMR = 0.038, indicating a good fit of the model. The factor loadings were >0.40 (0.432–0.679) for all instrument items (Figure 2).
In Table 4, we can see the adaptation values concerning the two-parameter logistic model for the seven performance-based items. Only item Q22 had values below the ideal values. In general, the items presented good adaptation values relating to the model, with central values between 0.7 and 1.3 for infit and outfit, verifying the one-dimensionality and confirming the final version of the instrument.
Figure 3 shows the skill level of the participants and the difficulty level of the items. It can be seen that most items are located between the −1 and +1 logit points. Items Q23 and Q25 were the easiest (difficulties −1.53 and −0.72 logits, respectively), while items Q26 and Q28 were the most difficult (difficulties 0.61 and 1.02 logits, respectively).
Figure 4 shows the item characteristic and item information curves plot. Item response category characteristic curves depicted the relationship between the level of performance-based digital health literacy and the probability of selecting the specific option for each item in the scale. The information curves showed that items Q22 and Q24 had the most information, indicating the importance of the response of this item in measuring the level of performance-based DHL.
The discriminant validity of the DHLI-BrA was measured by the correction between the total score and the adolescent’s age, as well as family income. Older adolescents with a higher family income had better self-reported and performance-based digital health literacy (p < 0.001) (Table 5).
The convergent validity of the self-reported DHLI-BrA was measured by the significant correlation with the scores of the digital health literacy instrument, eHEALS-BrA (r: 0.192, p = 0.002), and health literacy, REALMD-20 (r: 0.192, p = 0.002). For the performance-based items, convergent validity was demonstrated by correlation with the health literacy instruments, NVS (r: 0.472, p < 0.001) and REALMD-20 (r: 0.445, p < 0.001) (Table 5). The convergent validity of the self-reported items was demonstrated by their significant correlation with the perceived quality of life in the physical (r: 0.199, p = 0.001), psychological (r: 0.213, p = 0.001), and environmental domains (r: 0.183, p = 0.033) (Table 5). In addition, the convergent validity was supported by the association with using the Internet to take care of their health, such as searching for professionals, doctors, and dentists (p < 0.001) and scheduling appointments online (p = 0.014) (Table 6).
The predictive validity of the performance-based DHLI-BrA was demonstrated by the significant association with having a mobile connection to the Internet (p = 0.026) and a higher frequency of Internet use (p = 0.022) and, also, its association with searching for (p = 0.001) and reading (p = 0.002) health information online, using health apps (p = 0.040), searching for symptoms (p = 0.046), and publishing an evaluation/review about medical treatment (p < 0.001) (Table 6).
Later, a correlation between the total self-reported DHLI-BrA score and the total score of the performance-based items (r: 0.199, p = 0.001) was observed.

4. Discussion

This study showed that the 21 self-reported items and the 7 performance-based items of the DHLI-BrA are reliable and valid tools for measuring the DHL of Brazilian adolescents. Therefore, its use can be recommended for epidemiological studies and for health professionals to investigate patients’ individual abilities when using health information on the Internet.
Compared to the original instrument, the DHLI-BrA self-reported items demonstrated similar internal consistency (α = 0.87) and higher levels of test–retest reliability (ICC = 0.77) [15]. As the DHLI was developed recently (2017) [15], it has been adapted and validated for only a few countries and age groups so far [7,17,39,40,41]. Also, during the 2020 coronavirus pandemic, a reduced version of the DHLI was developed to measure DHL regarding COVID-19 in some populations [18,19,20,21,22]. Therefore, this study expands the evidence that the DHLI is an internally consistent and temporally stable measure for measuring DHL.
The original instrument performance-based items presented an internal consistency of α = 0.47, and the authors chose to interpret the items separately [15]. In our study, reliability was slightly higher but still considered moderate (α = 0.57). Based on recent criticism regarding the interpretation of Cronbach’s alpha, we decided to measure the one-dimensionality of the construct, since the small number of items can limit the alpha of an instrument, and making decisions based on a single value can be considered quite simplistic. It is necessary to evaluate other information such as the mean, correlation between items, and variance [42]. We observed that the one-dimensionality of the performance-based items presented satisfactory behavior.
In the analysis of item–total correlations, we noticed that the items that showed a good correlation with the total score had values close to or greater than 0.30. Lower values indicate that an item may not be correlated with the total scale score and should be removed [43]. In this study, self-reported items Q20 and Q21 had a correlation coefficient of 0.11 and 0.15, respectively; however, they were not excluded from the instrument, as the Cronbach’s alpha did not increase with their exclusion. Additionally, for the performance-based items, no outliers were observed.
The average DHLI-BrA score was 3.02 (±0.37), a value similar to those reported in the DHLI validation study for Canadian adults [3.11 (±0.87)] [15], a fact that may be justified by the great familiarity of adolescents with the use of technologies and by it being a phase in which their interest in health information begins [4,7]. For the performance-based items, the average total score was 4.15 (±1.72). It was not possible to compare this result, as this is the first study to measure the one-dimensionality of these items.
The correlation results confirmed the differences in the DHL levels between older adolescents and those with higher family incomes. This fact can be explained by the development and cognitive maturation of adolescents during this phase [8], as well as the development of skills to use the Internet [7] and greater exposure to health information throughout their lives [44]. Thus, family income can influence the health literacy of adolescents [37,38] and access to communication technologies by the population [45].
Functional health literacy, measured by REALMD-20, reflects the basic skills of reading and understanding the terms related to health [28]. In this study, functional HL was found to be moderately correlated with the total score of the performance-based items and weakly correlated with the self-reported items. Similar to the original instrument, we observed a moderate correlation of performance-based items with the total NVS score, which measures health literacy in a more comprehensive and practical way; that is, it assesses the individual’s ability to read, understand, interpret, and make decisions [27]. These results were expected as items based on the DHLI-BrA performance and require practical skills of interpretation and health-related decision-making in the digital environment [15].
Like the original instrument [15], we observed a moderate correlation between the self-reported DHLI-BrA and the eHEALS digital health literacy instrument. As to quality of life, there was a correlation between the best performance on the DHLI-BrA and the physical, psychological, and environment domains.
The better use of the Internet and better self-assessment of the skills to use it were associated with higher levels of digital health literacy. Our results corroborate previous findings in the literature [15,18]. For performance-based items, an association with variables related to eHealth was observed, such as searching and reading health information on the Internet, searching for symptoms, and using health-related applications.
This study provides important guidance on the psychometrics of the DHLI, but it has some limitations. The study involved elementary and high school students in the city of Belo Horizonte, located in Southeastern Brazil. Although schools distributed throughout the city and with different teaching quality classifications (IDEB) were selected, these adolescents may present some characteristics that differ from adolescents from other Brazilian regions. In addition, the digital world is evolving rapidly, and it is likely that some terms, websites, and applications may become outdated within a few years. Therefore, instruments such as DHLI-BrA need to be updated to reflect trends in a timely manner. Finally, we should look at self-reported DHLI items Q20 and Q21 with caution. These items make up the privacy dimension, but their adjustment parameters to the model are not ideal. Therefore, we suggest that future studies seek to validate instruments with better properties to measure this aspect of DHL in adolescents in Brazil.
It is worth highlighting the importance of this study for the safe recommendation of the DHLI-BrA as an instrument for measuring DHL in Brazilian adolescents. From a public health perspective, children and young people constitute a central population group for research and intervention in health literacy [46,47,48]. During youth, fundamental processes of cognitive, physical, and emotional development occur, and health-related behaviors and skills are incorporated [8,48,49]. As health literacy is a variable construct and can be acquired in a lifelong learning process, starting it during adolescence can bring benefits in the short and long term [48].

5. Conclusions

The DHLI-BrA demonstrated adequate psychometric properties to measure the self-reported, performance-based digital health literacy of Brazilian adolescents. It is important to highlight that, when using performance-based items, it is necessary to evaluate the main form of Internet access individually, as well as the suitability of the items to keep up with constant technological evolution. In addition, it is interesting that future studies will focus on adapting and evaluating the properties of the instrument for adolescents with disabilities, including those with visual, hearing, and cognitive impairments.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/ijerph21111458/s1, File S1 Digital Health Literacy Instrument—Versão Brasileira para adolescentes (DHLI-BrA).

Author Contributions

Conceptualization, M.C.F.B., A.F.G.-G., S.M.P. and F.d.M.F.; methodology, S.M.P. and F.d.M.F.; software, F.d.M.F., M.C.F.B. and R.T.F.; validation, M.C.F.B. and F.d.M.F.; formal analysis, M.C.F.B., R.T.F. and F.d.M.F.; investigation, M.C.F.B. and A.L.P.B.; resources, M.C.F.B. and F.d.M.F.; data curation, M.C.F.B. and A.L.P.B.; writing—original draft preparation, M.C.F.B.; writing—review and editing, A.F.G.-G., S.M.P. and F.d.M.F.; visualization, F.d.M.F.; supervision, A.F.G.-G. and S.M.P.; project administration, F.d.M.F.; funding acquisition, F.d.M.F. All authors have read and agreed to the published version of the manuscript.

Funding

This study was funded by the Universidade Federal de Minas Gerais (UFMG—Federal University of Minas Gerais) and the following Brazilian fostering agencies: Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES—Coordination for the Advancement of Higher Education Personnel), Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPQ—National Council of Scientific and Technological Development) (CNPq Nº 09/2022—Process: 309181/2022-4 and CNPq Nº 18/2021—Process: 406204/2021-7), and Fundação de Amparo à Pesquisa do Estado de Minas Gerais (FAPEMIG—Research Support Foundation of the State of Minas Gerais) (APQ-00360-22).

Institutional Review Board Statement

This study was also approved by the Human Research Ethics Committee of the Federal University of Minas Gerais (#CAAE: 58603022.8.0000.5149).

Informed Consent Statement

All participants were informed about the objective of the study and signed an informed consent form (guardians and adolescents over 18 years of age) and an informed assent form (adolescents under 18 years of age).

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors on request.

Acknowledgments

The authors would like to thank the Federal University of Minas Gerais (UFMG), the Belo Horizonte Municipal Department of Education, and the Minas Gerais State Department of Education for making this research possible. In conclusion, we thank all the adolescents and guardians who participated in this study.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Lobo, M.L.C.G. O uso das TICs como ferramenta de ensino-aprendizagem no Ensino Superior ASM. CAD Geogr. 2015, 25, 16–26. [Google Scholar] [CrossRef]
  2. Ferreira, E.Z.; Oliveira, A.M.N.; Medeiros, S.P.; Gomes, G.C.; Cezar-Vaz, M.R.; Ávila, J.A. Internet influence on the biopsychosocial health of adolescents: An integratitive review. Rev. Bras. Enferm. 2020, 73, e20180766. [Google Scholar] [CrossRef] [PubMed]
  3. Kostkova, P. Grand challenges in digital health. Front. Public Health 2015, 3, 134. [Google Scholar] [CrossRef]
  4. Park, E.; Kwon, M. Testing the Digital Health Literacy Instrument for Adolescents: Cognitive Interviews. J. Med. Internet Res. 2021, 23, e17856. [Google Scholar] [CrossRef]
  5. IBGE. Pesquisa Nacional por Amostra de Domicílios Contínua: Acesso à Internet e à Televisão e Posse de Telefone Móvel Celular Para uso Pessoal. 2019. Available online: https://www.ibge.gov.br/estatisticas/sociais/populacao/17270-pnadcontinua.html?edicao=30362 (accessed on 12 March 2022).
  6. Dunahee, M.; Lebo, H.; Digitalcenter. The World Internet Project International Report (Sixth Ed). 2016. Available online: https://www.digitalcenter.org/wp-content/uploads/2013/06/2015-World-Internet-Report.pdf (accessed on 12 March 2022).
  7. Park, E.; Kwon, M. Health-Related Internet Use by Children and Adolescents: Systematic Review. J. Med. Internet Res. 2018, 20, e120. [Google Scholar] [CrossRef] [PubMed]
  8. Crone, E.A.; Konijn, E.A. Media use and brain development during adolescence. Nat. Commun. 2018, 9, 588. [Google Scholar] [CrossRef]
  9. Wong, C.A.; Merchant, R.M.; Moreno, M.A. Using social media to engage adolescents and young adults with their health. Healthcare 2014, 2, 220–224. [Google Scholar] [CrossRef]
  10. Swire-Thompson, B.; Lazer, D. Public health and online misinformation: Challenges and recommendations. Annu. Rev. Public Health 2020, 41, 433–451. [Google Scholar] [CrossRef]
  11. Lotto, M.; Jorge, O.S.; Machado, M.A.A.M.; Cruvinel, T. Exploring online oral health misinformation: A content analysis. Braz. Oral Res. 2023, 37, e049. [Google Scholar] [CrossRef]
  12. Vosoughi, S.; Roy, D.; Aral, S. The spread of true and false news online. Science 2018, 359, 1146–1151. [Google Scholar] [CrossRef]
  13. Fleary, S.A.; Joseph, P.; Pappagianopoulos, J.E. Adolescent health literacy and health behaviors: A systematic review. J. Adolesc. 2018, 62, 116–127. [Google Scholar] [CrossRef] [PubMed]
  14. Norman, C.D.; Skinner, H.A. eHEALS: The eHealth Literacy Scale. J. Med. Internet Res. 2006, 8, e27. [Google Scholar] [CrossRef] [PubMed]
  15. Van Der Vaart, R.; Drossaert, C. Development of the Digital Health Literacy Instrument: Measuring a Broad Spectrum of Health 1.0 and Health 2.0 Skills. J. Med. Internet Res. 2017, 19, e27. [Google Scholar] [CrossRef] [PubMed]
  16. Van De Belt, T.H.; Engelen, L.J.; Berben, S.A.; Schoonhoven, L. Definition of Health 2.0 and Medicine 2.0: A systematic review. J. Med. Internet Res. 2010, 12, e18. [Google Scholar] [CrossRef]
  17. Barbosa, M.C.F.; Baldiotti, A.L.P.; Lopes, C.T.; Paiva, S.M.; Granville-Garcia, A.F.; Ferreira, F.M. Cross-cultural adaptation of the Digital Health Literacy Instrument (DHLI) for use on Brazilian Adolescents. Braz. Dent. J. 2023, 34, 104–114. [Google Scholar]
  18. Agormedah, E.K.; Quansah, F.; Ankomah, F.; Hagan, J.E., Jr.; Srem-Sai, M.; Abieraba, R.S.K.; Frimpong, J.B.; Schack, T. Assessing the validity of digital health literacy instrument for secondary school students in Ghana: The polychoric factor analytic approach. Front. Digit. Health 2022, 4, 968806. [Google Scholar] [CrossRef]
  19. Martins, S.; Augusto, C.; Martins, M.R.O.; José Silva, M.; Okan, O.; Dadaczynski, K.; Duarte, A.; Fronteira, I.; Ramos, N.; Rosário, R. Adaptation and validation of the Digital Health Literacy Instrument for Portuguese university students. Health Promot. J. Austr. 2022, 33, 390–398. [Google Scholar] [CrossRef]
  20. Rivadeneira, M.F.; Miranda-Velasco, M.J.; Arroyo, H.V.; Caicedo-Gallardo, J.D.; Salvador-Pinos, C. Digital Health Literacy Related to COVID-19: Validation and Implementation of a Questionnaire in Hispanic University Students. Int. J. Environ. Res. Public Health 2022, 19, 4092. [Google Scholar] [CrossRef]
  21. Lorini, C.; Velasco, V.; Bonaccorsi, G.; Dadaczynski, K.; Okan, O.; Zanobini, P.; Vecchio, L.P. Validation of the COVID-19 Digital Health Literacy Instrument in the Italian Language: A Cross-Sectional Study of Italian University Students. Int. J. Environ. Res. Public Health 2022, 19, 6247. [Google Scholar] [CrossRef]
  22. Chun, H.; Park, E.J.; Choi, S.K.; Yoon, H.; Okan, O.; Dadaczynski, K. Validating the Digital Health Literacy Instrument in Relation to COVID-19 Information (COVID-DHL-K) among South Korean Undergraduates. Int. J. Environ. Res. Public Health 2022, 19, 3437. [Google Scholar] [CrossRef]
  23. Anthoine, E.; Moret, L.; Regnault, A.; Sébille, V.; Hardouin, J.B. Sample size used to validate a scale: A review of publicationson newly-developed patient reported outcomes measures. Health Qual. Life Outcomes 2014, 12, 176. [Google Scholar] [CrossRef] [PubMed]
  24. Barbosa, M.C.F.; Baldiotti, A.L.P.; Dias, M.L.L.S.; Granville-Garcia, A.F.; Paiva, S.M.; Ferreira, F.M. Cross-cultural adaptation of the eHealth Literacy Instrument (eHEALS) for application in Brazilian Adolescents. Braz. Oral Res. 2024, 38, e094. [Google Scholar] [CrossRef] [PubMed]
  25. Fleck, M.P.A.; Louzada, S.; Xavier, M.; Chamovich, E.; Vieira, G.; Santos, L.; Pinzon, V. Aplicação da versão em português do instrumento abreviado de avaliação da qualidade de vida “WHOQOL-bref”. Rev. Saude Publica 2000, 34, 178–188. [Google Scholar] [CrossRef] [PubMed]
  26. The Whoqol Group. The word Health Organization Quality of Life Assessment (WHOQOL): Position paper from the Health Organization. Soc. Sci. Med. 1995, 41, 1403–1409. [Google Scholar] [CrossRef]
  27. Cruvinel, A.F.P.; Cusicanqui Méndez, D.A.; Campos Chaves, G.; Gutierres, E.; Lotto, M.; Marchini Oliveira, T.; Cruvinel, T. The Brazilian validation of a health literacy instrument: The newest vital sign. Acta Odontol. Scand. 2018, 76, 587–594. [Google Scholar] [CrossRef]
  28. Cruvinel, A.F.P.; Méndez, D.A.C.; Oliveira, J.G.; Gutierres, E.; Lotto, M.; Machado, M.A.A.M.; Oliveira, T.M.; Cruvinel, T. The Brazilian version of the 20-item rapid estimate of adult literacy in medicine and dentistry. Peer J. 2017, 5, e3744. [Google Scholar] [CrossRef]
  29. Landis, J.R.; Koch, G.G. The meadurement of observer agrément for categorical data. Biometrics 1977, 33, 159. [Google Scholar] [CrossRef]
  30. Dunn, T.J.; Baguley, T.; Brunsden, V. From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. Br. J. Psychol. 2014, 105, 399–412. [Google Scholar] [CrossRef]
  31. Kuder, G.F.; Richardson, M.W. The theory of the estimation of test reliability. Psychometrika 1937, 2, 151–160. [Google Scholar] [CrossRef]
  32. Revelle, W. Psych: Procedures for Personality and Psychological Research; Northwestern University: Evanston, IL, USA, 2023; Available online: https://CRAN.R-project.org/package=psych (accessed on 30 September 2023).
  33. Kline, R.B. Principles and Practice of Structural Equation Modeling, 4th ed.; Gu3ilford Press: New York, NY, USA, 2015. [Google Scholar]
  34. Norman, S. Biostatistics: The Bare Essentials, 3rd ed.; PMPH: Raleigh, NC, USA, 2008. [Google Scholar]
  35. Brown, T.A. Confirmatory Factor Analysis for Applied Research; Guilford Press: New York, NY, USA, 2006. [Google Scholar]
  36. Chalmers, P. Mirt: A Multidimensional Item Response Theory Package for the R Environment. J. Stat. Softw. 2012, 48, 1–29. [Google Scholar] [CrossRef]
  37. Bond, T.; Yan, Z.; Heene, M. Applying the Rasch Model: Fundamental Measurement in the Human Sciences, 4th ed.; Routledge: London, UK, 2020. [Google Scholar]
  38. Boone, W.J. Rasch Analysis for Instrument Development: Why, When, and How? CBE—Life Sci. Educ. 2016, 15, rm4. [Google Scholar] [CrossRef] [PubMed]
  39. Xie, L.; Mo, P.K.H. Comparison of eHealth Literacy Scale (eHEALS) and Digital Health Literacy Instrument (DHLI) in Assessing Electronic Health Literacy in Chinese Older Adults: A Mixed-Methods Approach. Int. J. Environ. Res. Public Health 2023, 20, 3293. [Google Scholar] [CrossRef] [PubMed]
  40. Miyawaki, R.; Kato, M.; Kawamura, Y.; Ishikawa, H.; Oka, K. [Developing a Japanese version of the Digital Health Literacy Instrument]. Nihon Koshu Eisei Zasshi 2024, 71, 3–14. (In Japanese) [Google Scholar] [PubMed]
  41. Xie, L.; Hu, H.; Lin, J.; Mo, P.K.H. Psychometric validation of the Chinese digital health literacy instrument among Chinese older adults who have internet use experience. Int. J. Older People Nurs. 2024, 19, e12568. [Google Scholar] [CrossRef]
  42. Ventura-León, J.; Peña-Calero, B.N. The world should not revolve around Cronbach’s alpha ≥ 0.70. Adicciones 2021, 33, 369–372, (In English, Spanish). [Google Scholar] [CrossRef]
  43. Field, A. Descobrindo a Estatística Usando o SPSS, 2nd ed.; Artmed: Porto Alegre, Brazil, 2009. [Google Scholar]
  44. Lima, L.C.M.; Neves, É.T.B.; Dutra, L.D.C.; Firmino, R.T.; Araújo, L.J.S.; Paiva, S.M.; Ferreira, F.M.; Granville-Garcia, A.F. Psychometric properties of BREALD-30 for assessing adolescents’ oral health literacy. Rev. Saude Publica 2019, 53, 53. [Google Scholar] [CrossRef]
  45. Lopes, R.T.; Neves, É.T.B.; Dutra, L.D.C.; Gomes, M.C.; Paiva, S.M.; Abreu, M.H.N.G.; Ferreira, F.M.; Granville-Garcia, A.F. Socioeconomic status and family functioning influence oral health literacy among adolescents. Rev. Saude Publica 2020, 54, 30. [Google Scholar] [CrossRef]
  46. Zarcadoolas, C.; Pleasant, A.; Greer, D.S. Understanding health literacy: An expanded model. Health Promot. Int. 2005, 20, 195–203. [Google Scholar] [CrossRef]
  47. Irwin, L.; Siddiqi, A.; Hertzman, C. Early Child Development: A Powerful Equalizer: Final Report for the World Health Organization’s Commission on Social Determinants of Health; University of British Columbia: Vancouver, Columbia, 2007. [Google Scholar]
  48. Bröder, J.; Okan, O.; Bauer, U.; Bruland, D.; Schlupp, S.; Bollweg, T.M.; Saboga-Nunes, L.; Bond, E.; Sørensen, K.; Bitzer, E.-M.; et al. Health literacy in childhood and youth: A systematic review of definitions and models. BMC Public Health 2017, 17, 361. [Google Scholar]
  49. Zhu, J.; Garin, C.M.; Qi, X.L.; Machado, A.; Wang, Z.; Hamed, S.B.; Stanford, T.R.; Salinas, E.; Whitlow, C.T.; Anderson, A.W.; et al. Brain structure and activity predicting cognitive maturation in adolescence. bioRxiv 2024, e608315, preprint. [Google Scholar]
Figure 1. Conceptual representation and factor loadings for the 6-factor solutions of the DHLI-BrA self-reported items.
Figure 1. Conceptual representation and factor loadings for the 6-factor solutions of the DHLI-BrA self-reported items.
Ijerph 21 01458 g001
Figure 2. Conceptual representation and factor loadings for the single-factor solution of the DHLI-BrA performance-based items.
Figure 2. Conceptual representation and factor loadings for the single-factor solution of the DHLI-BrA performance-based items.
Ijerph 21 01458 g002
Figure 3. Item map of the final version of the performance-based DHLI-BrA.
Figure 3. Item map of the final version of the performance-based DHLI-BrA.
Ijerph 21 01458 g003
Figure 4. Item response category characteristic curves and item information curves for the performance-based DHLI-BrA.
Figure 4. Item response category characteristic curves and item information curves for the performance-based DHLI-BrA.
Ijerph 21 01458 g004
Table 1. Scale means, scale variances, item–total correlation, and Cronbach’s alpha if item deleted from the DHLI-BrA self-reported items.
Table 1. Scale means, scale variances, item–total correlation, and Cronbach’s alpha if item deleted from the DHLI-BrA self-reported items.
Scale Mean If Item DeletedScale Variance If Item DeletedItem–Total CorrelationCronbach’s Alpha If Item Deleted
Q159.7158.20.380.83
Q259.7558.60.300.83
Q359.8857.80.360.83
Q460.9055.30.530.82
Q560.5953.50.610.81
Q660.8954.30.550.82
Q761.2453.80.560.82
Q860.7356.30.300.83
Q960.3554.80.500.82
Q1060.5355.60.420.82
Q1160.8255.90.390.83
Q1260.8755.30.400.83
Q1360.4757.70.370.83
Q1459.8757.20.340.83
Q1560.9656.40.400.83
Q1660.3155.00.510.83
Q1760.6554.20.470.83
Q1860.3856.20.380.83
Q1960.3557.50.310.83
Q2059.9760.10.110.84
Q2159.7959.80.150.83
Table 2. Cronbach’s alpha and correlations regarding the factors determined by CFA.
Table 2. Cronbach’s alpha and correlations regarding the factors determined by CFA.
Factors44Cronbach’s AlphaCorrelations
123456
1Operational skills0.5341
2Navigation skills0.5280.556 *1
3Skills to assess the reliability0.5980.620 *0.841 *1
4Skills to Search and determine the relevance for information0.7570.587 *0.577 *0.663 *1
5Skills to include self-generated0.6260.644 *0.645 *0.588 *0.689 *1
6Skills to protect privacy0.6240.1750.0490.0060.453*0.0931
* p < 0.001.
Table 3. Scale means, scale variances, item–total correlation, and Cronbach’s alpha if the item was deleted from the DHLI-BrA performance-based items.
Table 3. Scale means, scale variances, item–total correlation, and Cronbach’s alpha if the item was deleted from the DHLI-BrA performance-based items.
Scale Mean If Item DeletedScale Variance If Item DeletedItem–Total CorrelationCronbach’s Alpha If Item Deleted
Q223.4522.1970.4070.492
Q233.3942.4360.2550.548
Q243.5442.2130.3500.512
Q253.5332.3720.2360.557
Q263.7482.3660.2350.558
Q273.6172.2750.2900.537
Q283.6232.4600.3000.535
Table 4. Two-parameter model for performance-based DHLI-BrA.
Table 4. Two-parameter model for performance-based DHLI-BrA.
DiscriminationDifficultyOutfitInfit
Q221.827−0.7180.5130.694
Q230.856−1.5300.8460.938
Q241.233−0.4590.7370.803
Q250.762−0.7190.8850.911
Q260.7100.6110.8940.914
Q270.886−0.1820.8490.872
Q280.8581.0200.8480.893
Note. Infit = Weighted Mean Square Fit; Outfit = Unweighted Mean Square Fit.
Table 5. Spearman correlations between DHLI-BrA self-reported and performance-based with sociodemographic aspects, digital health literacy, health literacy, and quality of life.
Table 5. Spearman correlations between DHLI-BrA self-reported and performance-based with sociodemographic aspects, digital health literacy, health literacy, and quality of life.
VariablesDHLI-BrA
Self-Reported
DHLI-BrA
Performance-Based
rprp
Adolescent’s age0.267<0.0010.324<0.001
Average family income0.1670.0150.242<0.001
Digital Health Literacy–eHEALS-BrA 10.452<0.0010.0650.294
Health Literacy–NVS 20.0780.2150.472<0.001
Health Literacy–REALMD-20 30.1920.0020.445<0.001
Quality of life–physical0.1990.001−0.0590.346
Quality of life–psychological aspects0.2130.001−0.1010.106
Quality of life–social relationships0.0550.383−0.0710.257
Quality of life–environment0.1830.0330.0540.392
1 eHEALS-BrA: Version for Brazilian adolescents of the eHealth Literacy Scale; 2 NVS: Newest Vital Sign; 3 REALMD-20: 20-item Rapid Estimate Adult Literacy in Medicine and Dentistry. The significance level adopted was 5% (p < 0.05).
Table 6. Comparison of the mean (±SD) DHLI-BrA self-reported and performance-based scores with self-assessed general health and skills to use the Internet, access media, and search behavior for health information on the Internet.
Table 6. Comparison of the mean (±SD) DHLI-BrA self-reported and performance-based scores with self-assessed general health and skills to use the Internet, access media, and search behavior for health information on the Internet.
VariablesDHLI-BrA Self-ReportedpDHLI-BrA Performance-Basedp
General health self-assessmentVery Good3.14 (0.39)0.051 *3.90 (1.73)0.319 *
Good3.01 (0.37)4.32 (1.73)
Regular2.97 (0.34)4.04 (1.63)
Bad/Very Bad2.72 (0.51)3.70 (1.48)
Self-assessment of Internet usage skillsVery Good3.11 (0.34)0.002 *4.24 (1.78)0.012 *
Good2.97 (0.35)4.27 (1.69)
Bad2.87 (0.48)3.33 (1.41)
Frequency of Internet useEvery day3.02 (0.37)0.433 *4.22 (1.71)0.022 *
Almost all3.04 (0.42)3.50 (1.55)
Almost never2.78 (0.23)2.17 (1.15)
Do you have mobile Internet?Yes3.09 (0.36)0.002 *4.24 (1.79)0.026 *
Sometimes2.92 (0.36)4.19 (1.66)
No2.95 (0.40)3.32 (1.21)
Searched for a doctor/dentistYes3.07 (0.35)>0.0014.16 (1.75)0.705 #
No2.83 (0.39)4.11 (1.59)
Search for health information on the InternetYes3.03 (0.37)0.2434.26 (1.68)0.001 #
No2.91 (0.42)2.98 (1.69)
Schedule appointmentYes3.06 (0.38)0.0144.29 (1.72)0.090 #
No2.96 (0.36)3.97 (1.70)
Read health informationYes3.04 (0.37)0.2464.36 (1.66)0.002 #
No2.97 (0.37)3.59 (1.76)
Use health appsYes3.07 (0.39)0.0864.44 (1.76)0.040 #
No2.99 (0.36)3.99 (1.66)
Search for symptoms onlineYes3.01 (0.37)0.7284.24 (1.71)0.046 #
No3.03 (0.38)3.74 (1.70)
Post an evaluation/review about a medical treatmentYes3.08 (0.37)0.4022.77 (1.96)<0.001 #
No3.01 (0.37)4.290 (1.63)
# Kruskal–Wallis test/Mann–Whitney U test. * The significance level adopted was 5% (p < 0.05).
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Barbosa, M.C.F.; Baldiotti, A.L.P.; Firmino, R.T.; Paiva, S.M.; Granville-Garcia, A.F.; Ferreira, F.d.M. Evaluation of the Psychometric Properties of the Digital Health Literacy Instrument (DHLI-BrA) for Use in Brazilian Adolescents. Int. J. Environ. Res. Public Health 2024, 21, 1458. https://doi.org/10.3390/ijerph21111458

AMA Style

Barbosa MCF, Baldiotti ALP, Firmino RT, Paiva SM, Granville-Garcia AF, Ferreira FdM. Evaluation of the Psychometric Properties of the Digital Health Literacy Instrument (DHLI-BrA) for Use in Brazilian Adolescents. International Journal of Environmental Research and Public Health. 2024; 21(11):1458. https://doi.org/10.3390/ijerph21111458

Chicago/Turabian Style

Barbosa, Mariane Carolina Faria, Ana Luiza Peres Baldiotti, Ramon Targino Firmino, Saul Martins Paiva, Ana Flávia Granville-Garcia, and Fernanda de Morais Ferreira. 2024. "Evaluation of the Psychometric Properties of the Digital Health Literacy Instrument (DHLI-BrA) for Use in Brazilian Adolescents" International Journal of Environmental Research and Public Health 21, no. 11: 1458. https://doi.org/10.3390/ijerph21111458

APA Style

Barbosa, M. C. F., Baldiotti, A. L. P., Firmino, R. T., Paiva, S. M., Granville-Garcia, A. F., & Ferreira, F. d. M. (2024). Evaluation of the Psychometric Properties of the Digital Health Literacy Instrument (DHLI-BrA) for Use in Brazilian Adolescents. International Journal of Environmental Research and Public Health, 21(11), 1458. https://doi.org/10.3390/ijerph21111458

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop