Next Article in Journal
Are Black Sexual Minority Adults More Likely to Report Higher Levels of Psychological Distress than White Sexual Minority Adults? Findings from the 2013–2017 National Health Interview Survey
Next Article in Special Issue
Trainee Teachers’ Perceptions on Cyberbullying in Educational Contexts
Previous Article in Journal / Special Issue
Mothers’ Perceptions of the Phenomenon of Bullying among Young Children in South Korea
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Social Sciences
Volume 8
Issue 1
10.3390/socsci8010013
socsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Psychometric Properties of the CYBVICS Cyber-Victimization Scale and Its Relationship with Psychosocial Variables

by
Sofía Buelga
1,*,
Belén Martínez-Ferrer
2,
María-Jesús Cava
1 and
Jessica Ortega-Barón
3
1
Department of Social Psychology, Faculty of Psychology, University of Valencia, Valencia 46010, Spain
2
Department of Education and Social Psychology, University Pablo Olavide, Sevilla 41013, Spain
3
Department of Psychology of Education and Psychobiology, Faculty of Education, International University of la Rioja (UNIR), Logroño 26006, Spain
*
Author to whom correspondence should be addressed.
Soc. Sci. 2019, 8(1), 13; https://doi.org/10.3390/socsci8010013
Submission received: 23 November 2018 / Revised: 26 December 2018 / Accepted: 3 January 2019 / Published: 9 January 2019
(This article belongs to the Special Issue Family, Bullying and Cyberbullying)
Versions Notes

Abstract

:
The main goal of the present study was to analyze the psychometric properties of the revised version of the Adolescent Cyber-Victimization Scale (CYBVICS). This scale is composed of 18 items that assess direct and indirect cyber-victimization. Two subsamples participated in the present study. Sample 1 included 1318 adolescents (47.4% boys) from 12 to 16 years old (M = 13.89, SD = 1.32). Sample 2 was composed of 1188 adolescents (51.5% girls) from 12 to 16 years old (M = 14.19, SD = 1.80). First, an exploratory factor analysis was conducted on sample 1. Results yielded a bifactor structure: direct cyber-victimization and indirect cyber-victimization. To confirm the structure of the CYBVICS, we selected sample 2 to perform confirmatory factor analysis and test its convergent validity with theoretically related measures. The results supported the reliability and validity of the two-factor model. In addition, measurement invariance was established. Related to convergent validity, positive correlations between cyber-victimization and peer victimization, depressive symptoms, and offensive communication with the mother and the father were found. Moreover, negative correlations were found between cyber-victimization and open communication with the mother and the father and family self-esteem.

1. Introduction

Cyberbullying is defined as an intentional, aggressive, and repetitive behavior, where a person or group uses electronic devices (mainly the Internet and the smartphone) to bully a person who cannot defend him/herself (Aboujaoude et al. 2015; Buelga et al. 2017). The prevalence of this technological bullying by peers ranges between 5% and 72% (Athanasiou et al. 2018; Kowalski et al. 2014), with an average incidence rate of cyber-victimization of 23% in the adolescent population (Hamm et al. 2015). Therefore, there is considerable divergence in the results obtained in different scientific studies, probably due to the use of different methodological measures and strategies to evaluate cyberbullying and cyber-victimization (Buelga et al. 2012; Vivolo-Kantor et al. 2014). In recent years, cyberbullying has clearly become a growing problem among youth in every country in the world (Antoniadou et al. 2016; Lee and Shin 2017; Yubero et al. 2017; Zych et al. 2016).
As Lee and Shin (2017) suggest, this increase and spread of cyberbullying among youth can be attributed to numerous factors. One of the main reasons is the increasing availability and massive use of smartphones in adolescence (Ortega-Baron et al. 2017; Lee and Shin 2017). This tendency is confirmed in many countries. In Korea, more than 80% of students have a smartphone that they use daily to actively connect to a large number of smartphone apps (Lee and Lee 2013). In the United States, 95% of adolescents between 13 and 17 years old have a smartphone, and 45% of them claim to be online almost continually (Anderson and Jiang 2018). Of the European countries, Spain has the largest number of smartphones in the general population. In fact, 70% of young Spaniards have a smartphone at the age of 12, and 98% at the age of 14, and they use them to connect to popular social networks (Ditrendia Digital Marketing Trends 2016; Protégeles 2014).
Therefore, the smartphone is currently a key essential element in people’s daily lives, especially among adolescents (Del Río et al. 2017; Garaigordobil 2017; Navarro et al. 2016). Due to their comfortable and fast connectivity to immediate messaging services (e.g., WhatsApp) and social networks (e.g., Instagram, Facebook), these digital platforms can be used by young people not only to communicate positively with others, but also to easily harass peers publicly and virally.
The consequences of cyberbullying are so serious for the victim that it is considered a worldwide public health problem (Aboujaoude et al. 2015; Ferrara et al. 2018). Moreover, 20% of adolescent cyber-victims have expressed suicidal ideations (Van Geel et al. 2014). In many cases, the experience of cyber-victimization produces intense emotional distress in the victim (Larrañaga et al. 2016). In addition, unlike traditional victimization, parents frequently do not know about the victimization experienced by their child, which increases the victim’s feeling of loneliness (Buelga et al. 2016; Nocentini et al. 2018). Thus, authors such as Thompson et al. (2012) suggest that the accumulation of adverse life events (cumulative lifetime adversities) in adolescence is closely related to suicidal ideation (see Serafini et al. 2015).
Certainly, continuous technological advances promote the modification and appearance of new forms and modalities of cyberbullying behavior (Cross et al. 2015; Cohen-Almagor 2018) that should be evaluated through updated and validated tools with methodological rigor, in order to prevent and intervene in this social problem (Kowalski et al. 2014). For instance, recent studies have pointed out that cyberbullying behaviors could be classified into direct and indirect cyber-victimization behaviors (Antoniadou et al. 2016; Lohbeck and Petermann 2018). Direct cyber-victimization includes verbal-type behaviors and attacks directed toward the person (e.g., sending insulting messages directly to the victim). Indirect cyber-victimization includes behaviors and attacks oriented indirectly toward the person through the manipulation of photos and videos (e.g., gossiping behind the victim’s back).
From this perspective, taking into account the need to detect new expressions of cyberbullying in order to improve intervention in this field, the main purpose of this study is to validate the psychometric properties of the revised version of CYBVICS. This scale is a revised version of the CYBVIC scale (Buelga et al. 2012), which uses Willard’s classification (Willard 2007) to evaluate the frequency with which the adolescent is a target of cyber-victimization behaviors related to harassment, belittlement, invasion of privacy, identity theft, and social exclusion through the Internet, on the one hand, and through the smartphone, on the other.
The CYBVICS scale is a two-dimensional scale on the use of the smartphone and includes these cyber-aggressions and other new ones, grouped in two cyber-victimization modalities, direct and indirect. Direct cyber-victimization includes experiences of being victimized that involve direct attacks (e.g., “Someone insulted me or ridiculed me on social networks”) and social-type behaviors (e.g., “Someone removed or blocked me from groups so that I wouldn’t have any friends”). Indirect cyber-victimization includes experiences of being victimized that involve the manipulation of images (e.g., “Someone created or manipulated videos or photos of me”), identity theft (e.g., “Someone created a false profile with my personal data on the Internet”), or hacking (e.g., “Someone changed my password to social networks so I could not access them”; see Appendix A).
The structural validity of the instrument was analyzed through exploratory and confirmatory factor analyses, the multi-group analysis of two independent samples of adolescents, and an analysis to test its convergent validity with several indicators of psychosocial adjustment and wellbeing. This study will allow us to acquire up-to-date instruments for scientific investigation in order to prevent and intervene in this serious cyber-bullying problem among adolescents. Furthermore, this scale would be a useful tool for detecting cyber-victims in the school context.

2. Method

2.1. Participants

In this cross-sectional study, the participants were 2506 adolescents enrolled in Obligatory Secondary Education. They were distributed in two independent samples, from Andalucía and the Valencian Community (Spain). To select the participants, stratified cluster sampling was carried out. The sampling units were semi-private and public schools from Andalusian Region (sample 1) and the Valencian Community (sample 2). Previous analysis showed that the two samples were equivalent in terms of gender and academic grade.
Sample 1 was composed of 1318 adolescents (age range: 12–16 years, M = 13.89, SD = 1.32) of both sexes (52.6% girls). The sample distribution by academic grade in Secondary Education was as follows: 24.58% Grade 7, 27.39% Grade 8, 23.75% Grade 9, and 24.28% Grade 10.
Sample 2 was composed of 1188 adolescents (age range: 12–16 years, M = 14.19, SD = 1.80) of both sexes (48.5% boys). The sample distribution by academic grade was as follows: 30% Grade 7, 25.10% Grade 8, 21.5% Grade 9, and 23.40% Grade 10.

2.2. Measures

2.2.1. Cyber-Victimization Scale (CYBVICS)

This scale is an adaptation of the Adolescent Victimization through Mobile Phone and Internet Scale (CYBVIC; Buelga et al. 2010, 2012). This new scale is an updated version of the previous scale, given that, with the use of the smartphone, the distinction between the cellphone and Internet currently makes no sense. Moreover, in addition to the items on the original scale, this new scale includes eight other items. Thus, the Cyber-Victimization Scale consists of 18 self-reported items (see Appendix A) rated on a 5-point Likert-type scale ranging from 1 (never) to 5 (always). This scale measures the adolescent’s experience as a victim of cyberbullying in the past 12 months. A self-report measure was elaborated due to its greater capacity to recognize adolescents’ cyber-victim behaviors.
First, to adapt the questionnaire, we started with the items on the CYBVIC validated scale, and, after reviewing the literature and other scales used in the past three years (Antoniadou et al. 2016), a list of 23 cyber-victimization behaviors was elaborated. This initial questionnaire was administered to an experimental sample of 48 adolescents between 12 and 16 years old. The aim was to explore possible difficulties in comprehending the items and find out whether the adolescents would add to or eliminate any of the cyber-victimization behaviors from the questionnaire. Some adolescents thought one item from the list was difficult to understand, and that another item referred to a behavior that was not currently performed. At the same time, three experts in adolescence and cyber-bullying also examined the items’ clarity and thoroughness, coinciding with the adolescents. These judges also suggested suppressing another item, and so the questionnaire was composed of 20 items. Finally, two items that presented a saturation inferior to 0.20 in the Exploratory Factor Analysis (EFA) were eliminated, so that the final questionnaire contained 18 items.

2.2.2. Multidimensional Self-Concept Scale AF5 (AF5)

The AF5 (García and Musitu 1999) is a self-reported scale composed of 30 items that evaluate 5 dimensions of self-esteem: Academic, social, emotional, family, and physical. In this study, the Family Self-concept subscale was used, which is composed of 6 items that measure aspects related to adolescents’ perceptions of their affective and relational bonds with their families (e.g., “I feel loved by my parents”). Cronbach’s alpha reliability coefficient in this study was 0.73.

2.2.3. The Parent-Adolescent Communication Scale (PACS)

The PACS (Barnes and Olson 1982; Spanish Adaptation by Estévez et al. 2005) is a self-reported scale composed of 20 items rated on a 5-point Likert-type scale ranging from 1 (never) to 5 (always). Items measure the adolescent’s perception of the communication with his/her father and mother separately. This scale has three subscales for the father and three for the mother: Openness in Father/Mother Communication; Offensive Communication with Father/Mother, and Avoidant Communication with Father/Mother. In this study, the subscales of Openness in Father/Mother Communication (11 items) and Offensive Communication with Father/Mother (4 items) were used. The Cronbach’s alpha reliability coefficients in this study were: 0.91 and 0.90 for openness communication with the mother and father, respectively; and 0.71 and 0.78 for Offensive communication with the father and mother, respectively.

2.2.4. Center for Epidemiological Studies Depression Scale (CES-D)

The CES-D (Radloff 1977; Spanish Adaptation and Short form by Herrero and Meneses 2006) is a self-reported scale that consists of 7 items rated on a Likert-type scale ranging from 1 (never) to 4 (always) to assess the presence of depressive symptomatology in the past month. Cronbach’s alpha reliability coefficient in this study was 0.80.

2.2.5. Victimization Scale among Peers

This instrument is a self-reported scale consisting of 20 items rated from 1 (never) to 4 (always) that evaluate situations of direct and indirect victimization experienced by the adolescent in school (Mynard and Joseph 2000). Cronbach’s alpha reliability coefficient in this study was 0.79.

2.3. Procedure

The same procedure was followed in both samples. First, a letter was sent to the principals of the selected schools. Second, a seminar was held for the teachers to explain the research objectives and request parent authorization. Next, a letter describing the study was sent to the parents, asking for active authorization if they wanted their child to participate in the study (96% of the parents authorized their children’s participation in the study). Questionnaires were administered during a regular class period (55 min) with the supervision of trained researchers. Participation was voluntary and anonymous, and privacy was guaranteed. Few students refused to participate in the study (<2%). This study met the ethical requirements for research with human beings according to the fundamental principles established in the Declaration of Helsinki, and it was approved by the Ethics Committee of the University of Valencia (Protocol Number: H1456762885511).

2.4. Data Analysis

Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) were carried out separately in the two samples in order to obtain cross-validation of the scale’s factor structure (Izquierdo et al. 2014; Worthington and Whitttaker 2006). First, EFA was performed with the first sample, and the factors’ reliabilities were calculated. Previously, the suitability of this analysis was explored through the Bartlett sphericity test and the Kaiser-Mayer-Olkin (KMO) sample adequacy test. To extract the factors, the Principal Components method with Oblimin rotation was used. This rotation is used when it is estimated that the factors obtained are correlated. To select the factors, a self-value higher than 1 and a decrease in the Cattell sedimentation graphic were taken into account. The SPSS program (23) was used.
Next, CFA was carried out to contrast the factor structure obtained with the second sample, using the QS 6.1 program (Bentler and Wu 2005). The maximum robust verisimilitude estimation method was used for all the analyses, due to the lack of multivariate normality in the data (Mardia coefficient = 1573.56). Several fit indicators were used, in addition to the Satorra-Bentler chi-square: the comparative fit index (CFI), the Bentler-Bonett non-normative fit index (NNFI)—also called the Tucker-Lewis index (TLI)—and the Root Mean Square Error of Approximation (RMSEA). Values of CFI and TLI above 0.95 are considered a good fit, as well as RMSEA values below 0.50. Regarding the Chi-square, a proposed model is considered to fit the data well when the ratio between the Chi-square and the degrees of freedom is less than 3 (Bentler and Bonett 1980).

3. Results

3.1. Item Analysis

Table 1 presents the distribution of the item responses. The item with the highest mean score was item 2 (“They called me on my smartphone and hung up to bother or frighten me”), whereas the item with the lowest mean was item 9 (“They put my cellphone number on the Internet and said bad or false things about me so people would call me and get me into trouble.”). Kolmogorov-Smirnoff test results indicated that the items did not fit a normal distribution. The scores obtained on the asymmetrical and kurtosis indexes showed that all the items were distributed in a positive and leptokurtic way. Finally, the item-total correlation was positive and significant; therefore, all the items contribute in the same direction.

3.2. Factor Structure of the CYBVICS

First, Exploratory Factor Analysis was carried out in order to obtain the resulting factor structure. The results of the Kaiser-Meyer-Olkin test (0.95) and the Bartlett test (χ2 = 109048.45; gl = 153, p < 0.001) performed with the data from sample 1 indicated that the factor model was adequate to explain the data. The principal components analysis with oblimin rotation showed a two-factor structure that explained 53.08% of the variance.
Based on the semantic analysis of the items distributed in the factors and each item’s saturation in the factor (see Table 2), the following factor structure can be observed, coherent with theoretical arguments linked to cyber-victimization behaviors. The first factor (43.85% of the variance) refers to behaviors that involve the cyber-aggressor directing electronic communication towards the victim and provoking an immediate negative effect (e.g., phone calls, instant messaging, text, or multimedia messages etc.). Therefore, and following the proposal by Langos (2012), this factor has been referred to as direct cyber-victimization. The second factor (9.24% of the explained variance) contains items that refer to behaviors through which the cyberbully does not directly manage the communication and even avoids being identified. In addition, these behaviors involve greater premeditation. According to Langos (2012), this factor is referred to as indirect cyber-victimization. The correlation between the two factors is high and significant (r = 0.63, p < 0.001).
To confirm this structure, CFA was conducted in the second sample (see Table 2). The fit indexes for the proposed model indicated a good fit of the model and confirmed the scale’s factor structure (χ2 = 159.91, gl = 134, p > 0.05, CFI = 0.97, TLI = 0.97; RMSEA = 0.013 [0.00–0.02]). The internal consistency coefficients (Cronbach’s α) of these factors in the second sub-sample were also above 0.70. Furthermore, high and significant correlations were obtained between the two factors (see Table 3).

3.3. Test of Invariance

To evaluate the scale’s factor invariance, a multi-group analysis was carried out to confirm that the number of factors, the items’ saturations, and the correlations between the items were invariant in the two adolescent samples. Different nested models were estimated: (1) a first model without restrictions in which there is no imposed equality between the saturations and the correlations between the factors (model without restrictions); and (2) a second model that imposes equality between all the saturations and the correlations between the factors (model with restrictions) in the groups. These models were compared using the χ2 difference test (See Table 4). The results showed that the two models differed on one restriction. Once it was freed, the two models were equivalent.

3.4. Convergent Validity

Finally, the scale’s convergent validity was analyzed by considering the correlations between the global cyber-victimization index obtained with the CYBVICS scale and other theoretically related constructs, such as: open communication with the father and the mother, offensive communication with the father and the mother, family self-esteem, depressive symptomatology, and peer victimization (see Table 5).
The results showed significant positive and moderate correlations between cyber-victimization and offensive communication with the mother (r = 0.28, p < 0.01) and with the father (r = 0.23, p < 0.01). As expected, a strong and positive correlation was found between cyber-victimization and peer victimization (r = 042, p < 0.01) and depressive symptomatology (r = 0.46, p < 0.01). Furthermore, cyber-victimization showed significant and low negative correlations with open communication with the mother (r = −0.19, p < 0.01) and with the father (r = −0.18, p < 0.01), and moderate negative correlations with family self-esteem (r = −0.21, p < 0.01).

4. Discussion

The technological changes, along with the appearance and modification of new cyber-victimization behaviors, as well as the severe consequences of cyber-bullying for the victim’s psychological wellbeing (Ortega-Baron et al. 2016; Nocentini et al. 2018), suggest the need for updated and reliable instruments that allow its adequate measurement. In this regard, this study has found empirical evidence that supports the scale’s validity, factor structure, internal consistency, and convergent validity with theoretically related external variables.
Thus, regarding the CYBVICS’ factor structure, the results of the current study show a structure formed by two factors. This two-factor structure was also found for the original scale, which measures bullying through the smartphone, on the one hand, and bullying through the Internet, on the other. On the new scale, the two factors measure, in this case, direct cyber-victimization and indirect cyber-victimization, and they explain 53.08% of the variance. The first factor refers to cyber-aggressions oriented directly toward the victim, both verbal (i.e., insults on forums) and social (i.e., social exclusion from online groups) bullying. The second factor focuses more on indirect or instrumental cyber-aggressions that use tools to cause harm to the victim, such as creating a false profile of the victim, identity theft, or hacking his/her personal account.
These results are consistent with the two-factor structure of the CBVEQ-G scale by Antoniadou et al. (2016), who also classify their items as indirect aggressions and direct aggressions. These results are also congruent with previous literature on traditional bullying that points to the existence of these two dimensions of peer violence, referred to as direct or manifested aggressions and indirect or relational aggressions (Martínez-Ferrer et al. 2018; Mynard and Joseph 2000; Little et al. 2003).
Furthermore, the confirmatory factor analysis through the sample cross-validation procedure confirmed the scale’s validity. This two-factor structure was found to be the model that best fit the data, also verifying the factor invariance of the CYBVICS scale in two independent samples of adolescents. In addition, the items showed adequate reliability in terms of internal consistency, both by factor and in the total scale. The reliability of the scale with the addition of the new items, therefore, presents sufficient psychometric guarantees, also found for the original scale. Evidence was also obtained of our new instrument’s construct validity, given that significant correlations were found between the CYBVICS scale and the external variables considered: family communication with the mother and the father, family self-concept, depressive symptomatology, and traditional victimization.
Regarding these relationships between variables, our results show that cyber-victimization and traditional victimization are closely related, which was also found with the original scale and in previous literature (Alonso and Romero 2017; Cava and Buelga 2018; Mitchell and Jones 2015; Ortega-Baron et al. 2017). As Tokunaga (2010) and Kowalski et al. (2014) indicate, adolescents’ problems in the school context are transferred to and continue in Internet. Thus, for example, Hinduja and Patchin (2008) found that adolescents who had suffered from traditional bullying in the previous 6 months were 2.5 times more likely to experience cyberbullying as well. In addition, as in the original scale, the existence of a high relationship between cyber-victimization and depressive symptomatology can be observed. This result supports the large number of studies that show the negative consequences of victimization on the wellbeing and adjustment of the victim (Garaigordobil 2017; Navarro et al. 2018; Serafini et al. 2015; Yubero et al. 2017). In this victimization situation, the family plays an essential role in helping their child to get out of this situation (Garaigordobil and Machimbarrena 2017; Navarro et al. 2015; Nocentini et al. 2018). A problem arises when the family has no family resources to protect or help their child, making him/her more vulnerable to being cyber-victimized by his/her peers. Consistently, a close link has been found between a negative family atmosphere and a reduction in adolescents’ social and individual resources, which makes them an easier target for their peers’ aggressions (Cava et al. 2018; Lereya et al. 2013). In fact, our data show that the quality of the family climate is inversely related to cyber-bullying. Thus, a negative family climate with problematic communication patterns with the mother/father is related to cyber-victimization. However, when the communication patterns with the mother/father are open and the child feels valued and respected in the family, these variables are negatively associated with cyber-victimization, which seems to indicate that they play a protective role. Parents need to talk to their children to find out their worries, concerns, and motivations when navigating through the Internet (Buelga et al. 2016; Navarro et al. 2018). The children have to perceive that their parents are a source of unconditional social support that can help them to face any problem in the Internet. Some adolescents do not tell their parents they are being bullied because they are afraid of their reaction; of being punished because they performed risky behaviors in Internet, such as talking to strangers or sending photos with sexual content (sexting). Thus, it is essential for parents to foment family communication so that the child perceives that his/her parents provide their support in difficult situations. Therefore, the quality of the family climate, that is, the family emotional bond with relationships of parent-child trust and patterns of family communication, is one of the most important measures for preventing cyberbullying (Buelga et al. 2017; Moreno-Ruiz et al. 2019).
The current study also has some limitations. First, even though the sample used in this investigation is adequate for the analyses carried out (Cava et al. 2018; Kline 2015), it would be advisable to carry out subsequent validation analyses with larger samples of Spanish adolescents, as well as adolescent samples from other countries in the world, in order to take into account possible cultural differences in the instrument’s measurement. Moreover, the questionnaire’s validation was carried out in an urban population, so that it would be advisable to verify the results in a rural population, given that the conditions and the adolescents’ life style would differ. In addition, although this study provides evidence about important psychometric properties of the instrument (factor validity, convergent, and reliability), other psychometric aspects such as test-retest reliability were not considered and should be analyzed in future studies.
However, and despite these limitations, the current study guarantees the CYBVICS scale’s suitability, recommending its application in the adolescent population. The serious consequences of cyber-bullying for adolescents supports the need for in-depth studies with reliable instruments to prevent this serious worldwide problem. In this context, the family is still a fundamental socializing agent in promoting children’s healthy development and psychosocial adjustment.

Author Contributions

All the authors of the manuscript contributed equally to the research and writing of the present study.

Funding

This research was funded by the Ministry of Economy and Competitiveness of Spain, and the European Union though the European Regional Development Fund “A way of making Europe” [PSI2015-65683-P (MINECO/FEDER, UE)].

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A. CYBVICS. A

Adolescent Cyber-Victimization Scale
Below, you will see some behaviors that some boys and girls might do to intimidate or really bother you (not as a joke), through the cellphone, Internet, social networks, tablets, or WhatsApp in the past year.
1: Never2: Once or twice3: Few times (between 3 and 5)4: Several times (between 6 and 10)5. Many times (more than r0)
1. Someone insulted or ridiculed me in social networks or groups like WhatsApp to really hurt me. [Me han insultado o puesto en ridículo en redes sociales o en grupos como el WhatsApp para hacerme daño de verdad]
2. Someone called my cellphone and hung up to bother or frighten me. [Me han llamado al móvil y no han contestado para fastidiarme o asustarme]
3. Someone used threats to make me do things on the Internet or smartphone that I did not want to do (like recording myself on video, giving money, doing bad things). [ Me han obligado con amenazas a hacer cosas que no quería en Internet o por el móvil (como grabarme en video, dar dinero, hacer cosas malas)]
4. Someone told my secrets or revealed personal things about me in social networks or groups (WhatsApp, snapchat …). [Han contado secretos míos o revelado cosas personales mías sin mi permiso en redes sociales o en grupos (Whatsapp, snapchat …)]
5. To make fun of me, someone made or manipulated videos or photos of me and uploaded or distributed them on social networks or by smartphone. [Para burlarse de mí, han creado o manipulado videos o fotos mías, y las han subido o distribuido en redes sociales o por el móvil].
6. Someone logged into my profile or accounts and I could not avoid it. [Han entrado en mi perfil o en mis cuentas sin que yo pueda hacer nada]
7. Someone pretended to be me saying or doing bad things on the Internet. [Se han hecho pasar por mí para decir o hacer cosas malas en Internet]
8. Someone purposely created a webpage, a forum, or a group just to make fun of me and criticize me in front of everyone. [Han creado adrede una página, un foro o un grupo solo para meterse conmigo y criticarme delante de todos]
9. Someone put my cellphone number on the Internet and said bad or false things about me so that people would call me and get me into trouble. [Han puesto mi número de teléfono móvil en Internet diciendo cosas malas o falsas de mí para que me llamen y meterme en líos]
10. Someone took my smartphone and used it to send photos, videos, or mean messages to others to get me into trouble with them. [Me han cogido mi teléfono y desde mi móvil han enviado fotos, videos o mensajes malos a otros para meterme en problemas con los demás]
11. Someone criticized me or made fun of comments, photos, or videos I uploaded to social networks or groups like WhatsApp. [Han criticado o se han burlado de comentarios, fotos o videos que yo he subido en redes sociales o en grupos como el WhatsApp]
12. Someone created a false profile on the Internet with my personal data in order to impersonate me saying or doing bad things. [Han creado en Internet un perfil falso con mis datos personales para decir o hacer cosas malas, haciéndose pasar por mí]
13. Someone ignored and did not answer messages or things I shared in groups or social networks, just to make me feel bad. [Me han ignorado y no han contestado a mensajes o cosas que yo he puesto en grupos o en redes sociales para hacerme sentir mal]
14. Someone provoked me in social networks or groups by insulting or taunting me to make me angry and cause a big argument. [Me han provocado en redes sociales o en grupos con insultos y burlas para que yo me enfade mucho y que haya una gran discusión]
15. Someone eliminated or blocked me from groups to leave me without any friends. [Me han eliminado o bloqueado de grupos para dejarme sin amigos]
16. Someone stole my photos, videos, or private conversations and uploaded them or sent them to others. [Me han robado fotos, videos, conversaciones privadas, y las han subido o enviado a otros]
17. Someone changed my password to social networks so that I could not access them. [Me han cambiado la contraseña de las redes sociales para que no pueda entrar en ellas]
18. Someone sent me taunting messages to bother and annoy me. [Me han enviado mensajes de burla para fastidiarme y molestarme]

References

  1. Aboujaoude, Elias, Matthew Savage, Vladan Starcevic, and Wael Salame. 2015. Cyberbullying: Review of an old problem gone viral. Journal of Adolescent Health 57: 10–18. [Google Scholar] [CrossRef]
  2. Alonso, Cristina, and Estrella Romero. 2017. Aggressors and Victims in Bullying and Cyberbullying: A Study of Personality Profiles using the Five-Factor Model. The Spanish Journal of Psychology 20: E76. [Google Scholar] [CrossRef]
  3. Anderson, Monica, and Jingjing Jiang. 2018. Report Teens, Social Media & Technology. Washington: Pew Research Center. Available online: www.pewinternet.org/2018/05/31/teens-social-media-technology-2018 (accessed on 18 November 2018).
  4. Antoniadou, Nafsika, Constantinos M. Kokkinos, and Angelos Markos. 2016. Development, construct validation and measurement invariance of the Greek cyber-bullying/victimization experiences questionnaire (CBVEQ-G). Human Behavior 65: 380–90. [Google Scholar] [CrossRef]
  5. Athanasiou, Kalliope, Eirini Melegkovits, Elisabeth K. Andrie, Charalampos Magoulas, Chara K. Tzavara, Clive Richardson, and Artemis K. Tsitsika. 2018. Cross-national aspects of cyberbullying victimization among 14–17-year-old adolescents across seven European countries. BMC Public Health 18: 800. [Google Scholar] [CrossRef]
  6. Barnes, Howard, and David H. Olson. 1982. Parent adolescent communication scale. In Family Inventories: Inventories Used in a National Survey of Families across the Family Life Cycle. Edited by David H. Olson, Hamilton I. McCubbin, Howard Barnes, Andrea Larsen, Marla Muxen and y Marc Wilson. St. Paul: University of Minnesota Press, pp. 33–48. [Google Scholar]
  7. Bentler, Peter M., and Douglas G. Bonett. 1980. Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin 88: 588–66. [Google Scholar] [CrossRef]
  8. Bentler, Peter M., and Eric J. C. Wu. 2005. EQS 6.1 for Windows. Encino: Multivariate Software INC, pp. 1–26. ISBN 1-885898-03-7. [Google Scholar]
  9. Buelga, Sofía, María Jesús Cava, and Gonzalo Musitu. 2010. Cyberbullying: Victimización entre adolescentes a través del teléfono móvil y de Internet. Psicothema 22: 784–89. [Google Scholar]
  10. Buelga, Sofía, María Jesús Cava, and Gonzalo Musitu. 2012. Validación de la escala de victimización entre adolescentes a través del teléfono móvil y de Internet. Revista Panamericana de Salud Pública 32: 36–42. [Google Scholar] [CrossRef]
  11. Buelga, Sofía, Belén Martínez-Ferrer, and Gonzalo Musitu. 2016. Family relationships and cyberbullying. In Cyberbullying across the Globe: Gender, Family and Mental Health. Edited by Raúl Navarro, Santiago Yubero and Elisa Larrañaga. Basel: Springer International Publishing, pp. 94–114. ISBN 978-3-319-25552-1. [Google Scholar]
  12. Buelga, Sofía, Belén Martínez-Ferrer, and María-Jesús Cava. 2017. Differences in family climate and family communication among cyberbullies, cybervictims, and cyber bully–victims in adolescents. Computers in Human Behavior 76: 164–73. [Google Scholar] [CrossRef]
  13. Cava, María-Jesús, and Sofía Buelga. 2018. Propiedades psicométricas de la Escala de Victimización Escolar entre Iguales (VE-I). Revista Evaluar 18: 40–53. [Google Scholar]
  14. Cava, María-Jesús, Sofía Buelga, and Inés Tomás. 2018. Peer Victimization and Dating Violence Victimization: The Mediating Role of Loneliness, Depressed Mood, and Life Satisfaction. Journal of Interpersonal Violence. Advance online publication. [Google Scholar] [CrossRef]
  15. Cohen-Almagor, Raphael. 2018. Social responsibility on the Internet: Addressing the challenge of cyberbullying. Aggression and Violent Behavior 39: 42–52. [Google Scholar] [CrossRef]
  16. Cross, Donna, Amy Barnes, Alana Papageorgiou, Kate Hadwen, Lydia Hearn, and Leanne Lester. 2015. A social–ecological framework for understanding and reducing cyberbullying behaviours. Aggression and Violent Behavior 23: 109–17. [Google Scholar] [CrossRef]
  17. Del Río, María Isabel, Santiago Mendo, Benito del Barco, and Elena Felipe. 2017. Abuso del Móvil en Estudiantes Universitarios y Perfiles de victimización y agresión. Adicciones 29: 245–55. [Google Scholar] [CrossRef] [PubMed]
  18. Ditrendia Digital Marketing Trends. 2016. Informe Mobile en España y en el Mundo 2015. Available online: https://goo.gl/AGi5JF (accessed on 15 October 2018).
  19. Estévez, Estefanía, Gonzalo Musitu, and Juan Herrero. 2005. El rol de la comunicación familiar y del ajuste escolar en la salud mental del adolescente. Salud Mental 28: 81–89. [Google Scholar]
  20. Ferrara, Pietro, Francesca Ianniello, Alberto Villani, and Giovanni Corsello. 2018. Cyberbullying a modern form of bullying: Let’s talk about this health and social problem. Italian Journal of Pediatrics 44. [Google Scholar] [CrossRef]
  21. Garaigordobil, Maite. 2017. Antisocial behavior: Connection with bullying/cyberbullying and conflict resolution. Psychosocial Intervention 26: 47–54. [Google Scholar] [CrossRef]
  22. Garaigordobil, Maite, and Juan M. Machimbarrena. 2017. Stress, competence, and parental educational styles in victims and aggressors of bullying and cyberbullying. Psicothema 29: 335–40. [Google Scholar] [CrossRef] [PubMed]
  23. García, Fernando, and Gonzalo Musitu. 1999. AF5: Autoconcepto Forma 5. Madrid: Tea. [Google Scholar]
  24. Hamm, Michele P., Amanda S. Newton, Annabritt Chisholm, Jocelyn Shulhan, Andrea Milne, Purnima Sundar, and Lisa Hartling. 2015. Prevalence and effect of cyberbullying on children and young people: A scoping review of social media studies. JAMA Pediatrics 169: 770–77. [Google Scholar] [CrossRef] [PubMed]
  25. Herrero, Juan, and Julio Meneses. 2006. Short Web-based versions of the perceived stress (PSS) and Center for Epidemiological Studies-Depression (CESD) Scales: A comparison to pencil and paper responses among Internet users. Computers in Human Behavior 22: 830–46. [Google Scholar] [CrossRef]
  26. Hinduja, Sameer, and Justin W. Patchin. 2008. Cyberbullying: An exploratory analysis of factors related to offending and victimization. Deviant Behavior 29: 129–56. [Google Scholar] [CrossRef]
  27. Izquierdo, Isabel, Julio Olea, and Francisco José Abad. 2014. Exploratory factor analysis in validation studies: Uses and recommendations. Psicothema 26: 395–400. [Google Scholar] [CrossRef]
  28. Kline, Rex B. 2015. Principles and Practice of Structural Equation Modelling. New York: Guilford Publications, pp. 154–96. ISBN 9781462523344. [Google Scholar]
  29. Kowalski, Robin M., Gary W. Giumetti, Amber N. Schroeder, and Micah R. Lattanner. 2014. Bullying in the digital age: A critical review and meta-analysis of cyberbullying research among youth. Psychological Bulletin 140: 1073–137. [Google Scholar] [CrossRef]
  30. Langos, Colette. 2012. Cyberbullying: The challenge to define. Cyberpsychology, Behavior, and Social Networking 15: 285–89. [Google Scholar] [CrossRef]
  31. Larrañaga, Elisa, Santiago Yubero, Anastasio Ovejero, and Raúl Navarro. 2016. Loneliness, parent–child communication and cyberbullying victimization among Spanish youths. Computers in Human Behavior 65: 1–8. [Google Scholar] [CrossRef]
  32. Lee, Chang Ho, and Keyong S. Lee. 2013. Exploration of the impact of social media use on youths’ cyberbullying: Focusing on network characteristics. Studies on Korean Youth 24: 259–85. [Google Scholar]
  33. Lee, Chang Ho, and Namin Shin. 2017. Prevalence of cyberbullying and predictors of cyberbullying perpetration among Korean adolescents. Computers in Human Behavior 68: 352–58. [Google Scholar] [CrossRef]
  34. Lereya, Suzet Tanya, Muthanna Samara, and Dieter Wolke. 2013. Parenting behavior and the risk of becoming a victim and a bully/victim: A meta-analysis study. Child Abuse & Neglect 37: 1091–108. [Google Scholar]
  35. Little, Tood D., Christopher C. Henrich, Stephanie M. Jones, and Patricia. H. Hawley. 2003. Disentangling the “whys” from the “whats” of aggressive behaviour. International Journal of Behavioral Development 27: 122–33. [Google Scholar] [CrossRef]
  36. Lohbeck, Annette, and Franz Petermann. 2018. Cybervictimization, self-esteem, and social relationships among German secondary school students. Journal of School Violence 17: 472–86. [Google Scholar] [CrossRef]
  37. Martínez-Ferrer, Belén, David Moreno, and Gonzalo Musitu. 2018. Are adolescents engaged in the problematic use of social networking sites more involved in peer aggression and victimization? Frontiers in Psychology 9. [Google Scholar] [CrossRef]
  38. Mitchell, Kimberly J., and Lisa M. Jones. 2015. Cyberbullying and bullying must be studied within a broader peer victimization framework. Journal of Adolescent Health 56: 473–74. [Google Scholar] [CrossRef]
  39. Moreno-Ruiz, David, Belén Martinez-Ferrer, and Francisco García-Bacete. 2019. Parenting styles, cyberaggression, and cybervictimization among adolescents. Computers in Human Behavior 93: 252–59. [Google Scholar] [CrossRef]
  40. Mynard, Helen, and Stephen Joseph. 2000. Development of the multidimensional peer-victimization scale. Aggressive Behavior 26: 169–78. [Google Scholar] [CrossRef]
  41. Navarro, Raúl, Roberto Ruiz-Oliva, Elisa Larrañaga, and Santiago Yubero. 2015. The impact of cyberbullying and social bullying on optimism, global and school-related happiness and life satisfaction among 10-12-year-old schoolchildren. Applied Research in Quality of Life 10: 15–36. [Google Scholar] [CrossRef]
  42. Navarro, Raúl, Santiago Yubero, and Elisa Larrañaga, eds. 2016. Cyberbullying Across the Globe: Gender, Family and Mental Health. Switzerland: Springer International Publishing. [Google Scholar] [CrossRef]
  43. Navarro, Raúl, Santiago Yubero, and Larrañaga Elisa. 2018. Cyberbullying victimization and fatalism in adolescence: Resilience as a moderator. Children and Youth Services Review 84: 215–21. [Google Scholar] [CrossRef]
  44. Nocentini, Annalaura, Giada Fiorentini, Ludovica Di Paola, and Ersilia Menesini. 2018. Parents, family characteristics and bullying behavior: A systematic review. Aggression and Violent Behavior. [Google Scholar] [CrossRef]
  45. Ortega-Barón, Jessica, Sofía Buelga, and María Jesús Cava. 2016. The Influence of School Climate and Family Climate among Adolescents Victims of Cyberbullying. Comunicar 24: 57–65. [Google Scholar] [CrossRef] [Green Version]
  46. Ortega-Baron, Jessica, Sofía Buelga, María Jesús Cava, and Eva Torralba. 2017. School violence and attitude toward authority of student perpetrators of cyberbullying. Journal of Psychodidactics 22: 14–23. [Google Scholar] [CrossRef]
  47. Protégeles. 2014. Menores de Edad y Conectividad en España: Tablets y Smartphones. Available online: https://goo.gl/K4cvH9 (accessed on 21 November 2018).
  48. Radloff, Leonore S. 1977. The CES-D scale: A self report depression scale for research in the general population. Applied Psychological Measurement 1: 385–401. [Google Scholar] [CrossRef]
  49. Serafini, Gianluca, Caterina Muzio, Giulia Piccinini, Eirini Flouri, Gabriella Ferrigno, Maurizio Pompili, Paolo Girardi, and Mario Amore. 2015. Life adversities and suicidal behavior in young individuals: A systematic review. European Child & Adolescent Psychiatry 24: 1423–46. [Google Scholar] [CrossRef]
  50. Thompson, Richard, Alan J. Litrownik, Mark, D. Everson, Diana J. English, Howard Dubowitz, Laura Proctor, and Emalee G. Flaherty. 2012. Adverse experiences and suicidal ideation in adolescence: exploring the link using LONGSCAN samples. Psychology of Violence 2: 211–25. [Google Scholar] [CrossRef]
  51. Tokunaga, Robert S. 2010. Following you home from school: A critical review and synthesis of research on cyberbullying victimization. Computers in Human Behavior 26: 277–87. [Google Scholar] [CrossRef]
  52. Van Geel, Mitch, Paul Vedder, and Jenny Tanilon. 2014. Relationship between peer victimization, cyberbullying, and suicide in children and adolescents: A meta-analysis. JAMA Pediatrics 168: 435–42. [Google Scholar] [CrossRef]
  53. Vivolo-Kantor, Alana M., Brandi N. Martell, Kristin M. Holland, and Ruth Westby. 2014. A systematic review and content analysis of bullying and cyber-bullying measurement strategies. Aggression and Violent Behavior 19: 423–34. [Google Scholar] [CrossRef]
  54. Willard, Nancy E. 2007. The authority and responsibility of school officials in responding to cyberbullying. Journal of Adolescent Health 41: S64–S65. [Google Scholar] [CrossRef]
  55. Worthington, Roger L., and Tiffany A. Whitttaker. 2006. Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist 34: 806–38. [Google Scholar] [CrossRef]
  56. Yubero, Santiago, Raúl Navarro, María Elche, Elisa Larrañaga, and Anastasio Ovejero. 2017. Cyberbullying victimization in higher education: An exploratory analysis of its association with social and emotional factors among Spanish students. Computers in Human Behavior 75: 439–49. [Google Scholar] [CrossRef]
  57. Zych, Izabela, Rosario Ortega-Ruiz, and Inmaculada Marín-López. 2016. Cyberbullying: A systematic review of research, its prevalence and assessment issues in Spanish studies. Psicología Educativa 22: 5–18. [Google Scholar] [CrossRef]
Table
Table 1. Descriptive statistics for the CYBVICS.
Table 1. Descriptive statistics for the CYBVICS.
Frequency Distribution (%)MSDAsym.
(S.E. = 0.06)		Kurt.
(S.E. = 0.13)		K-Sri-t
ItemNeverOnce or TwiceFew Times (3–5)Several Times (6–10)Many Times (+10)
CV1873 (66.2)343 (26)60 (4.6)29 (1.4)23 (1.7)1.400.772.677.890.36 ***0.65
CV2749 (56.8)411 (30.1)92 (7)35 (2.7)31 (2.4)1.580.892.004.060.31 ***0.50
CV31260 (95.6)35 (2.7)13 (1)6 (0.5)4 (0.3)1.080.396.6750.710.46 ***0.51
CV41060 (80.4)167 (12.7)51 (3.9)23 (1.7)17 (1.3)1.340.732.899.080.38 ***0.67
CV51213 (92.1)72 (5.5)19 (1.4)9 (0.7)5 (0.4)1.130.474.9828.850.43 ***0.66
CV61196 (90.8)88 (6.7)18 (1.4)10 (8)6 (0.5)1.150.494.7026.000.42 ***0.52
CV71183 (89.8)96 (7.3)20 (1.5)12 (0.9)7 (0.5)1.170.524.4222.700.42 ***0.61
CV81240 (94)52 (3.9)15 (1.1)10 (0.8)1 (0.1)1.100.405.4333.290.44 ***0.60
CV91259 (95.5)44 (3.3)10 (0.8)4 (0.3)1 (0.1)1.070.326.5651.860.45 ***0.48
CV101178 (89.3)106 (8)24 (1.8)5 (0.4)5 (0.4)1.160.484.3023.140.42 ***0.60
CV111047 (79.4)185 (14)57 (4.3)19 (1.4)10 (0.8)1.340.682.738.480.38 ***0.71
CV121252 (94.9)39 (3)17 (1.3)7 (0.5)3 (0.2)1.090.406.0241.320.45 ***0.48
CV131075 (81.5)164 (12.4)47 (3.6)19 (1.4)13 (1)1.310.683.0210.310.38 ***0.66
CV14881 (66.8)312 (23.7)76 (5.8)38 (2.9)11 (0.8)1.410.782.365.560.37 ***0.64
CV151136 (86.2)113 (8.6)38 (2.9)16 (1.2)15 (1.1)1.250.673.5914.120.40 ***0.68
CV161231 (93.4)50 (3.8)16 (1.2)17 (1.3)4 (0.3)1.130.495.0928.010.43 ***0.65
CV171242 (94.3)47 (3.6)19 (1.4)3 (0.2)7 (0.5)1.100.435.9941.710.47 ***0.55
CV181164 (80.7)167 (12.7)42 (3.2)29 (2.2)16 (1.2)1.340.742.919.110.38 ***0.70
Note: M = mean; SD = standard deviation; Asym. = asymmetry; Kurt. = kurtosis; SE = standard error; K-S = Kolmogorov-Smirnoff; ri-t = corrected item–total correlation. *** p < 0.001.
Table
Table 2. Factor loadings for the EFA and CFA.
Table 2. Factor loadings for the EFA and CFA.
ItemEFACFA
F1F2F1
DCV		F2
ICV
CV10.760.450.66
CV20.600.370.50
CV30.420.64 0.71
CV40.750.530.68
CV50.580.73 0.69
CV60.410.68 0.63
CV70.520.70 0.67
CV80.520.70 0.75
CV90.340.67 0.72
CV100.530.68 0.69
CV110.790.540.75
CV120.330.69 0.69
CV130.770.450.73
CV140.770.430.77
CV150.760.540.73
CV160.560.74 0.70
CV170.400.74 0.65
CV180.830.470.77
% Explained Variance43.859.24
Note: DCV = Direct Cyber-Victimization; ICV = Indirect Cyber-Victimization; EFA = Exploratory Factor Analysis; CFA = Confirmatory Factor Analysis.
Table
Table 3. Descriptive statistics, factor correlations, and Cronbach’s alpha (in parentheses) for the 2-factor model.
Table 3. Descriptive statistics, factor correlations, and Cronbach’s alpha (in parentheses) for the 2-factor model.
MSDDCVICV
DCV1.230.60(0.92)
ICV1.370.750.66 **(0.89)
Note: DCV = Direct Cyber-Victimization, ICV = Indirect Cyber-Victimization; M = mean; SD = Standard Deviation. Alpha Global Scale = 0.88. ** p < 0.01.
Table
Table 4. Tests of measurement invariance across two samples for the two-factor model.
Table 4. Tests of measurement invariance across two samples for the two-factor model.
ModelDescriptionχ2dfComparison of Nested ModelsDifference S-B χ2Difference dfp
M1Restricted model362.0577284 ---
M2Model without restrictions343.3642268M1–M218.6916>0.05
Ma1 restriction unconstrained352.7305283Model 1a–Model 29.3715>0.05
Table
Table 5. Correlations among cyber-victimization, family communication, family self-esteem, depressive symptomatology, and peer victimization.
Table 5. Correlations among cyber-victimization, family communication, family self-esteem, depressive symptomatology, and peer victimization.
Cyber-VictimizationOpCMOfCMOpCFOfCFFS-EDepres Sympt.Peer Vict.
Cyber victimization1−0.19 **0.28 **−0.18 **0.23 **−0.21 **0.46 **0.42 **
Note: OpCM = Open Communication with the Mother; OfCM = Offensive Communication with the Mother; OpCF = Open Communication with the Father; OfCF = Offensive Communication with the Father; FS-E = Family Self-Esteem; Depres Sympt. = Depressive Symptoms; Peer Vict. = Peer Victimization. ** p < 0.01.

Share and Cite

MDPI and ACS Style

Buelga, S.; Martínez-Ferrer, B.; Cava, M.-J.; Ortega-Barón, J. Psychometric Properties of the CYBVICS Cyber-Victimization Scale and Its Relationship with Psychosocial Variables. Soc. Sci. 2019, 8, 13. https://doi.org/10.3390/socsci8010013

AMA Style

Buelga S, Martínez-Ferrer B, Cava M-J, Ortega-Barón J. Psychometric Properties of the CYBVICS Cyber-Victimization Scale and Its Relationship with Psychosocial Variables. Social Sciences. 2019; 8(1):13. https://doi.org/10.3390/socsci8010013

Chicago/Turabian Style

Buelga, Sofía, Belén Martínez-Ferrer, María-Jesús Cava, and Jessica Ortega-Barón. 2019. "Psychometric Properties of the CYBVICS Cyber-Victimization Scale and Its Relationship with Psychosocial Variables" Social Sciences 8, no. 1: 13. https://doi.org/10.3390/socsci8010013

APA Style

Buelga, S., Martínez-Ferrer, B., Cava, M.-J., & Ortega-Barón, J. (2019). Psychometric Properties of the CYBVICS Cyber-Victimization Scale and Its Relationship with Psychosocial Variables. Social Sciences, 8(1), 13. https://doi.org/10.3390/socsci8010013

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop