Construction of an Instrument for the Evaluation of the E ﬀ ects of Information and Communication Technologies among Young People

: The aim of this paper is to investigate the issue of access to Information and Communication Technologies (ICT) at younger ages, which is leading to dependency on mobile phones, video games, and compulsive aimless internet surﬁng—an issue that schools have been increasingly seeking to tackle. With the appearance of emerging technologies, and not forgetting those already established, an instrument is required that will adapt to new casuistry and help to design intervention programmes in accordance with present and future patterns of use, abuse, and addiction. Studies such as the one proposed here will provide data about the proﬁle of this population in order to improve programmes and inﬂuence the ICT policies rolled out by central and local governments. The chief aim of this paper is to construct and validate an instrument capable of evaluating problems experienced by young people in relation to technology use, abuse, and addiction within the programmes developed in Spain. The research design used in this study is mixed empirical, non-experimental, and sequential in nature in three stages: interviews conducted with 11 prevention professionals, group of 11 experts and pilot group of 30 participants in indicated prevention programmes. The ﬁndings of the study indicate that the instrument fulﬁlls the parameters established to be considered a systematic empirically sustainable instrument, since the young population needs to identify these patterns in order to understand and prevent risk behaviours associated with their use.


Introduction
The technological developments experienced over the past decade have brought forth a wide range of devices, applications, and tools designed for recreation, communication, and the services sector, becoming essential activities that facilitate everyday life. However, in some cases, their use has fuelled different types of repetitive, addictive or abusive behaviours, and this phenomenon has been studied within various disciplines such as psychology, education, and sociology [1][2][3][4][5].
Some studies have shown that access to Information and Communication Technologies (ICT) at increasingly young ages can make subjects more prone to inappropriate usage, triggering dependency on mobile phones, video games, compulsive aimless internet surfing, social media, or instant messaging apps.

Materials and Methods
The research design used is mixed empirical, nonexperimental and sequential in nature. It will explore relationships by associating and comparing data groups [29]. This design is rooted in the basic premise of a prior exploration, since there are no standardized measurement instruments available or a compendium of theoretically justified variables, prioritizing the compilation of qualitative data as the preliminary stage [30]. It will also help to improve research processes and products [31], providing quantifiable and contextual information [32] and allowing for the triangulation of results, the complementary nature of the phenomena studied, the discovery of singularities based on the profiles encountered, the sequencing of instruments designed, and the expansion of the study as we move through each of the stages [33].
Since this is a multi-stage study, the different stages designed to respond to the initially formulated research aim are as follows: • Stage 1 (Designing a first draft based on the information provided by in-depth interviews conducted with prevention professionals): In order to compile information to construct the instrument aimed at young people involved in indicated prevention programs run by the Proyecto Hombre Association, capable of evaluating possible technology use, abuse or addiction problems; initially, we sought to conduct semi-structured interviews with the programme officers, in order to compile information about the real needs of the professional team working directly with the target study group. Based on the analysis of this information, a first draft of the instrument was developed. • Stage 2 (Results of the procedure for validating the instruments by means of expert opinion): The tool developed in the previous stage was submitted for validation to a group of experts in the key issues of this project, gathering from them agreement indices for each of the evaluation elements proposed. The result of this activity provided a consensus of responses used to develop a second draft of the tool. • Stage 3 (Experimental application of the instrument to a pilot group): Having taken on board the recommendations of the experts consulted, the next stage that allowed us to develop the definitive instrument involved applying said instrument to a pilot group. The aim was to identify the reliability and validity of the measures, considering whether the questions were appropriate, whether the wording of the items was correct and comprehensible, whether the questions were of the right length, whether the answers were correctly categorized, whether there was any resistance or psychological barriers or rejection towards any of the questions, whether they were ordered logically, and whether the duration of the questionnaire was acceptable to respondents. This process gave shape to the definitive instrument.

Results
Below we set out the results of the different stages of this study, showing the process by which, the instrument was progressively constructed.

Stage 1: Designing a First Draft Based on the Information Provided by In-depth Interviews Conducted with Prevention Professionals
The aim of this first stage is to establish the core issues surrounding information and communication technology use, abuse, and addiction among the young people who take part in the indicated prevention programmes run by the Proyecto Hombre Association. Based on this evidence, the foundations can be laid to build an instrument to evaluate this problem systematically and empirically.
Semi-structured interviews were used to access this information, based on 13 questions designed to collate the following data: • Technological devices used by minors and young people within the indicated prevention programme. • Activities or tasks they carry out using these devices.

•
Knowledge possessed by programme officers about technology use, problematic use and addiction.

•
Indicators used to determine technology use, problematic use and addiction in the personal, family, social, education, and occupational or work dimensions.

•
Reasons that lead to use, problematic use, or addiction.

•
Profile of indicated prevention programme users.
• Attitude of the family to the consumption of technology.

•
Family information required to determine the problem. • Other necessary information.
The team of indicated prevention programme officers from the Proyecto Hombre Association who took part in this study was 11 women and 4 men from Alicante, Asturias, the Canary Islands, Malaga and Valladolid. With an average age of 38, and 9 years of professional experience on average, most of these professionals are trained in psychology, although some of them are also trained in primary education teaching or social work.
The interviews were incorporated into the qualitative analysis programme NVivo 11, using the interview questions as key elements when constituting the category tree shown in Table 1. Eight percent of the coded text is classified in the category "Devices used by the users". Ten percent of the text pertains to actions or tasks carried out when using the devices. Regarding technology use, problematic use and addiction, 6% of references were coded in this category. The weighting of indicators for the different dimensions (personal, social, family, education, occupational or work) is between 8% and 12%. Seven percent of the coded text is classified in the category "Profile of technology use in prevention programmes" and 8% in the "Family information required to specific the problem". Furthermore, the two categories with the lowest percentage weighting are related with family attitude, at 2%, and other information required to define the problem, at 4% of the coded references.
The instrument derived from this first stage is made up of a total of 10 analytical dimensions, 46 assessment elements, and 122 items. It is accompanied by a first round of elements not accounted for, the purpose of which is to identify the person providing information, and one last question about final observations referring to the tool.
Differentiation between elements and items stems from the fact that the evaluation elements are understood as units of observation that configure each analytical dimension, and the items are analytical units established within the evaluation elements as scaled and compulsory multiple-choice items.
The instrument is applied by means of a personal interview between indicated prevention programme officers and the user offering the information.

Stage 2: Results of the Procedure for Validating the Instruments by Means of Expert Opinion
The end product of the previous stage had to be validated by means of a suitable methodological procedure. In this instance, expert opinion [34] was the chosen method because, rather than being expressed quantitatively using an index or coefficient, it is estimated by means of a generally subjective or intersubjective opinion given by experts in the field. The purpose of this technique is to collate the opinions of people whose academic or professional background reflects their capacity to give evidence or critical assessments of the object of study [35], which enhances the validity of the content studied by seeking rational consensus [36].
When setting up this group, the selection criteria used brought together academics and professionals from the field of drug addiction prevention, ICT specialists, as well as experts in the design of instruments. A total of 10 people took part in the expert panel, as shown in Table 2. According to Skjong and Wentworth and Escobar [37] and Cuervo [35], the analysis process involved e-mailing the group an invitation to complete a validation protocol for the two instruments generated in the previous stage. The procedure entailed evaluating each of the elements of the tool aimed at users, assigning a score between 1 and 5 (1 indicating the minimum score and 5 the maximum score), in accordance with the following criteria:

•
Breadth of the content: fit of the question wording so that there is no redundancy and it is consistent with the response options.

•
Congruency: linkage and coherence of the items that make up the instrument. • Pertinence: correspondence between the content of the item and the dimension in which it will be used. Once the ten experts had given their general opinion when validating the instrument designed to be used with young users of indicated prevention programmes, we observed that all the parameters defined had scored highly (see Table 3). A breakdown of the evaluation data for each of the elements that make up the instrument confirmed this result. Furthermore, the consensus obtained between the groups of expert judges was high according to Aiken's V coefficient for the five stipulated criteria (V > 0.50) (see Table 4).  The output from this second stage was a tool comprising the 10 analytical dimensions from the first draft, increasing the evaluation elements to 50 and the number of items to 138. It is accompanied by a first round of elements not previously taken into consideration, the purpose of which is to identify the user who is providing information, and one final question relating to final observations about the tool.

Stage 3: Experimental Application of the Instrument to a Pilot Group
Having incorporated the recommendations of the experts to create the second draft of the instrument, the next stage of definitive construction was the experimental application to a pilot group of users participating in indicated prevention programmes run by Proyecto Hombre. The aim of this pilot test was to evaluate the consistency of the instrument (properties of the scale and its constituent elements) and its appropriateness to the object of measurement.
To study the psychometric properties of the instrument, the following analytical procedures were applied to the dimensions that, in their wording, incorporate elements configured by items from the scaled evaluation (the dimensions Description of family sphere and Availability of digital devices in the home are made up of elements of choice and are not eligible for validation): • Internal Consistency Analysis, in the sense of endowing the items with significance, in other words, ensuring that each of them measures a portion of the trait or characteristic studied. To this end, Cronbach's Alpha coefficient was used [38].

•
Analysis of the discrimination capacity of the elements to reinforce the one-dimensional nature of the test. Student's t-test was applied to the mean values of the established groups, indication of validity endorsed by García, Gil y Rodríguez [39].
The 30 people chosen for the pilot study are taking part in indicated prevention programmes at six centres run by Proyecto Hombre (Asturias, Catalonia, Madrid, Malaga, Melilla and Murcia) in a similar proportion in each of them (17.7%). 86.7% are male, and the remaining 13.3% are female. The age of the participants ranged from 14 to 22, and the majority of them had completed secondary education (73.3%). This number of people is valid to carry out this phase of the study according to McMillan and Schumacher [40].
Looking at the evaluation of the elements in the "Personal sphere" dimension, we see that the total value for Cronbach's Alpha in the scale (0.505) represents an acceptable correlation [41], an acceptable level of stability in the responses; hence, this part of the instrument presents signs of guaranteed reliability.
The discriminatory power of all the items in the test reinforces its one-dimensionality. To carry out this procedure, three closed items were chosen with ordinal response choices (response scale from 1 to 5) so that the sum total was recoded into three groups (Low, Medium and High): By applying Student's t-test for independent samples, we were able to establish the existence or non-existence of statistical difference (n.s. = 0.05) between the groups that score low and high in the selected elements. The results obtained using this test based on the 6 items belonging to this dimension and present in Table 5 show that 67% of the element possesses acceptable statistical discriminatory power, which indicates acceptable levels of validity in this dimension. The nondiscriminatory elements were maintained on account of their relevance in the instrument and in accordance with the suggestions made by the programme officers. The second dimension analysed, "ICT consumption habits", presented a low level of reliability, reflected in Cronbach's alpha coefficient, 0.27. Furthermore, discriminatory power revealed that only 25% of items have a valid discriminatory power (see Table 6). These results concluded that the dimension lacked acceptable reliability and validity criteria, and so we followed the recommendations of the prevention programme officers, by eliminating elements without discriminatory power and incorporating 13 new elements that would ensure the clarity and pertinence of the elements in the dimension according to the characteristics of the target population. Analysis of the 14 scaled items of the dimension "User's reasons for consuming ICT" revealed a guarantee of stability in the measure they offer, reflected in the Cronbach Alpha coefficient of 0.58. The application of the item discrimination test provided data that supported the observations proposed by the prevention programme officers participating in the study, since only 43% of items offered acceptable validity (see Table 7). The relevance of the contents processed did not lead to the suppression of any element but did lead to a redrafting of items where the discrimination was confusing. A Cronbach Alpha coefficient of 0.828 guaranteed the reliability of the 10 elements that make up the dimension "User's emotional management". Furthermore, the analytical test to establish the validity of these questions indicated that 70% offered valid content, leading us to revise the wording of three items (see Table 8). The reliability of the scaled items in the dimension "ICT in the family setting" when the subject is living in the family home is very high (Cronbach Alpha = 0.839). Sixty percent of these items possess an acceptable discriminatory power, which validates their inclusion in the instrument (see Table 9). Four of these items required further work based on the suggestions of the programme officers, who suggested that they should not be removed from the instrument but instead the wording needed to be revised to make them more understandable. The eight elements that make up the dimension "ICT in the social setting", once the relevant reliability test had been applied, contributed a global value of 0.890, clearly demonstrating their metric consistency. Furthermore, the application of the validity test showed that 100% of these items possess discriminatory power (see Table 10). The internal consistency of the nine scaled elements that describe the dimension "ICT in the education setting" was 0.880, scientifically guaranteeing their reliability. The validity of these items, measured by means of the corresponding item discriminatory test, showed that 89% measure the construct covered by this dimension, with just one of the items requiring revision. Furthermore, it should be noted that, at the request of the programme officers, a new element was incorporated into this dimension on account of its relevance for the subject studied (see Table 11). Finally, the eight elements that make up the dimension "ICT in the work setting" possess a high degree of reliability, as reflected in the Cronbach Alpha coefficient, 0.769, although in contrast these are not items that possess an acceptable level of discriminatory power, and accordingly all of them needed to be revised (see Table 12). However, it should be noted that only 5% of these prevention programmes participants are engaged in the employment market; hence, the results obtained understandable. Accordingly, the decision was made to maintain the elements as originally formulated. Following the same analytical dynamic as in the previous stages, the result of this third stage was a tool in the format of a personalised interview administered by prevention programme officers, made up of 10 analytical dimensions, 50 evaluation elements, and 156 items (see Table 13). As indicated previously, it is supplemented by a first round of elements, the purpose of which is to identify the user who is providing the information, along with an observations section in each dimension, and one final question related to final observations about the tool, all of which are not included in the total number of elements that make up in the final instrument (see Appendix A).

Discussion
Following the results set out regarding the construction and validation of the instrument created to identify technology use, abuse or addiction among the young people taking part in indicated prevention programmes run by the Proyecto Hombre Association throughout Spain, we can confirm that the instrument fulfills the parameters established to be classed as a systematic and empirically sustainable instrument, since the youth population needs to identify these patterns in order to understand and prevent risk behaviours associated with technology usage. In turn, the instrument must differentiate between the applications used and the use being made of them, in order to understand whether minors are using or abusing them [25]. This instrument adds to the contributions of López-Fenández, Freixa-Blanxart and Honrubia-Serrano's internet abuse scale for adolescents [27], detailing the devices used, the habits and reasons for consumption and the effects of use in the family, professional and emotional dimensions. There are direct connections to the instrument designed by Chen, Ho and Lwin [24] in the emotional and social effects, but the possible situations of bullying that arise with the use of these devices are not specified. Also, the scale designed by Peris, Maganto and Garaigordobil [42], focused on the use of social networks, prioritizes elements that are worked on in the instrument designed here such as the social use of devices and nomophobia.
The Jiménez, Alvarado and Llopis [43] instrument assesses the usefulness of ICT in the work of university students, the emotions that their use generates and the feelings of frustration that their absence generates. Understood by these elements as predictors of ICT addiction, the instrument referred to in this article considers those aspects in the adolescent population participating in the indicated prevention programmes. More than 28% of university students display risk behaviours with regard to the use of technology, and other variables need to be taken into consideration such as personality, family setting, and peer group [44]. The study conducted by [45] establishes that 90.6% of the population engages in controlled use of the internet, and just 9.4% have frequent problems.
However, the specificity of the adolescents attended in the indicated prevention programs for drug use and other risky behaviors and their relationship to the problematic use of ICT [46] and other behavioral problems or mismanagement of impulsivity [47] make it necessary to create an instrument adapted to new casuistic. Hence, this instrument facilitates the identification of use, abuse, and addiction profiles and their relationship with the technologies associated with these patterns.

Conclusions
By means of a mixed, nonsequential experimental study, three stages were developed with a view to achieving the aim of this research.
First, conducting semi-structured interviews with 15 programme officers working at Proyecto Hombre has shed light on risk factors in the family, education, and social dimensions, which can lead to problematic or addictive behaviour with regard to technological devices. This information was used to consolidate an initial draft of the evaluation instrument, comprising a total of 10 analytical dimensions, 46 evaluation elements, and 122 items.
Then, a panel of 10 experts in the evaluation of technology use, abuse, and addiction took part in the project. The results of the consensus analysis conducted allowed us to reshape the instrument into 10 analytical dimensions, 50 evaluation elements, and 138 items.
The third stage in the design of this tool involved the experimental application of this instrument to a pilot group of 30 users. The aim of this stage was to identify the reliability and validity indices of the instrument. Analysis of the information provided in this stage gave rise to the definitive instrument, made up of 10 analytical dimensions, 50 evaluation elements and 156 items. These dimensions are: description of the personal sphere, description of the family sphere, availability of digital devices in the home, ITC consumption habits, reasons for consuming ICT, description of emotional management, ICT in the family setting, ICT in the social setting, ICT in the education setting, and ICT in the work setting. The instrument was applied by means of a personalised interview conducted by prevention programme officers with users of the programme who provided the information.

Conflicts of Interest:
The authors declare no conflicts of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Appendix A Evaluation tool for problems of use, abuse or addiction of ICT
Property of Proyecto Hombre Association PERSONAL SPHERE DESCRIPTION

1.
What is the reason or reasons that led you to attend the youth program?
Point out how much time you spend on leisure per day: __ Less than 1 hour __ 1 or 2 hours __ 3 or 4 hours __ 5 or 6 hours __ More than 6 hours 3.
Value from 0 to 10 (0 = nothing and 10 = much) the accomplishment you do of the following activities: Have you attended or attend any other programs or received any therapy? __ Yes __ No 8.
If yes, please indicate which one: _____________________________________________ 9.
Indicate to what extent you identify yourself with the following statements (0 = nothing and 10 = much):  28. Indicate how much time you spend each day using technology: __ Less than 1 hour __ 1 or 2 hours __ 3 or 4 hours __ 5 or 6 hours __ More than 6 hours 29. Indicate which type of devices you use the most (0 = nothing and 10 = much):   43. Indicate to what extent you identify with each of the following statements (0 = nothing and 10 = much): Using technology due to a feeling of dissatisfaction with the relationship with my peer group 0 1 2 3 4 5 6 7 8 9 10 Using technology because they help to relate with others 0 1 2 3 4 5 6 7 8 9 10 Using technology due to fear of facing reality 0 1 2 3 4 5 6 7 8 9 10 Using technology due to fear of being excluded from my peer group 0 1 2 3 4 5 6 7 8 9 10 Using technology as a mean of coping with shyness 0 1 2 3 4 5 6 7 8 9 10 Using technology due to fear of not being up to date 0 1 2 3 4 5 6 7 8 9 10 Irritability over control over number of hours of technology use 0 1 2 3 4 5 6 7 8 9 10 Displays of aggression when number of hours of technology use are controlled 0 1 2 3 4 5 6 7 8 9 10 Fear of being without devices 0 1 2 3 4 5 6 7 8 9 10  45. Rate the extent to which the use of technologies affects coexistence (0 = nothing and 10 = much): The user does not speak with his/her father/tutor 0 1 2 3 4 5 6 7 8 9 10 The user fights with his/her father/tutor 0 1 2 3 4 5 6 7 8 9 10 The user is aggressive and rude towards his/her father/tutor 0 1 2 3 4 5 6 7 8 9 10 The user does not speak with his/her mother/tutor 0 1 2 3 4 5 6 7 8 9 10 The user fights with his/her mother/tutor 0 1 2 3 4 5 6 7 8 9 10 The user is aggressive and rude towards his/her mother/tutor 0 1 2 3 4 5 6 7 8 9 10 The user does not speak with his/her sibling(s) 0 1 2 3 4 5 6 7 8 9 10 The user fights with his/her sibling(s) 0 1 2 3 4 5 6 7 8 9 10 The user is aggressive and rude towards his/her sibling(s) 0 1 2 3 4 5 6 7 8 9 10 The user does not participate in family activities 0 1 2 3 4 5 6 7 8 9 10 In case of living with the partner 46. Rate the extent to which the use of technologies affects coexistence (0 = nothing and 10 = much): The user shows more interest in virtual than in physical relationships 0 1 2 3 4 5 6 7 8 9 10 The user enjoys virtual relations more than in physical relationships 0 1 2 3 4 5 6 7 8 9 10 The user invests more time in virtual relationships than in physical ones 0 1 2 3 4 5 6 7 8 9 10 The user shows their emotions more easily in the virtual world than in the physical world 0 1 2 3 4 5 6 7 8 9 10 The user says what they think more easily in the virtual world than in the physical world 0 1 2 3 4 5 6 7 8 9 10 The user pays more attention to virtual conversations than ones in the physical world 0 1 2 3 4 5 6 7 8 9 10 The user avoids conflicts by relating with others virtually 0 1 2 3 4 5 6 7 8 9 10 The user prefers to relate with others who share their love of