Below we set out the results of the different stages of this study, showing the process by which, the instrument was progressively constructed.
3.1. Stage 1: Designing a First Draft Based on the Information Provided by In-depth Interviews Conducted with Prevention Professionals
The aim of this first stage is to establish the core issues surrounding information and communication technology use, abuse, and addiction among the young people who take part in the indicated prevention programmes run by the Proyecto Hombre Association. Based on this evidence, the foundations can be laid to build an instrument to evaluate this problem systematically and empirically.
Semi-structured interviews were used to access this information, based on 13 questions designed to collate the following data:
Technological devices used by minors and young people within the indicated prevention programme.
Activities or tasks they carry out using these devices.
Knowledge possessed by programme officers about technology use, problematic use and addiction.
Indicators used to determine technology use, problematic use and addiction in the personal, family, social, education, and occupational or work dimensions.
Reasons that lead to use, problematic use, or addiction.
Profile of indicated prevention programme users.
Attitude of the family to the consumption of technology.
Family information required to determine the problem.
Other necessary information.
The team of indicated prevention programme officers from the Proyecto Hombre Association who took part in this study was 11 women and 4 men from Alicante, Asturias, the Canary Islands, Malaga and Valladolid. With an average age of 38, and 9 years of professional experience on average, most of these professionals are trained in psychology, although some of them are also trained in primary education teaching or social work.
The interviews were incorporated into the qualitative analysis programme NVivo 11, using the interview questions as key elements when constituting the category tree shown in Table 1
Eight percent of the coded text is classified in the category “Devices used by the users”. Ten percent of the text pertains to actions or tasks carried out when using the devices. Regarding technology use, problematic use and addiction, 6% of references were coded in this category. The weighting of indicators for the different dimensions (personal, social, family, education, occupational or work) is between 8% and 12%. Seven percent of the coded text is classified in the category “Profile of technology use in prevention programmes” and 8% in the “Family information required to specific the problem”. Furthermore, the two categories with the lowest percentage weighting are related with family attitude, at 2%, and other information required to define the problem, at 4% of the coded references.
The instrument derived from this first stage is made up of a total of 10 analytical dimensions, 46 assessment elements, and 122 items. It is accompanied by a first round of elements not accounted for, the purpose of which is to identify the person providing information, and one last question about final observations referring to the tool.
Differentiation between elements and items stems from the fact that the evaluation elements are understood as units of observation that configure each analytical dimension, and the items are analytical units established within the evaluation elements as scaled and compulsory multiple-choice items.
The instrument is applied by means of a personal interview between indicated prevention programme officers and the user offering the information.
3.2. Stage 2: Results of the Procedure for Validating the Instruments by Means of Expert Opinion
The end product of the previous stage had to be validated by means of a suitable methodological procedure. In this instance, expert opinion [34
] was the chosen method because, rather than being expressed quantitatively using an index or coefficient, it is estimated by means of a generally subjective or intersubjective opinion given by experts in the field. The purpose of this technique is to collate the opinions of people whose academic or professional background reflects their capacity to give evidence or critical assessments of the object of study [35
], which enhances the validity of the content studied by seeking rational consensus [36
When setting up this group, the selection criteria used brought together academics and professionals from the field of drug addiction prevention, ICT specialists, as well as experts in the design of instruments. A total of 10 people took part in the expert panel, as shown in Table 2
According to Skjong and Wentworth and Escobar [37
] and Cuervo [35
], the analysis process involved e-mailing the group an invitation to complete a validation protocol for the two instruments generated in the previous stage. The procedure entailed evaluating each of the elements of the tool aimed at users, assigning a score between 1 and 5 (1 indicating the minimum score and 5 the maximum score), in accordance with the following criteria:
Breadth of the content: fit of the question wording so that there is no redundancy and it is consistent with the response options.
Congruency: linkage and coherence of the items that make up the instrument.
Pertinence: correspondence between the content of the item and the dimension in which it will be used.
Precision: rigorousness with which words have been used when formulating each item in order to express what said item aims to measure.
Clarity: accuracy of the wording of each item, ensuring it is clear and easy to understand.
Once the ten experts had given their general opinion when validating the instrument designed to be used with young users of indicated prevention programmes, we observed that all the parameters defined had scored highly (see Table 3
A breakdown of the evaluation data for each of the elements that make up the instrument confirmed this result. Furthermore, the consensus obtained between the groups of expert judges was high according to Aiken’s V coefficient for the five stipulated criteria (V > 0.50) (see Table 4
The output from this second stage was a tool comprising the 10 analytical dimensions from the first draft, increasing the evaluation elements to 50 and the number of items to 138. It is accompanied by a first round of elements not previously taken into consideration, the purpose of which is to identify the user who is providing information, and one final question relating to final observations about the tool.
3.3. Stage 3: Experimental Application of the Instrument to a Pilot Group
Having incorporated the recommendations of the experts to create the second draft of the instrument, the next stage of definitive construction was the experimental application to a pilot group of users participating in indicated prevention programmes run by Proyecto Hombre. The aim of this pilot test was to evaluate the consistency of the instrument (properties of the scale and its constituent elements) and its appropriateness to the object of measurement.
To study the psychometric properties of the instrument, the following analytical procedures were applied to the dimensions that, in their wording, incorporate elements configured by items from the scaled evaluation (the dimensions Description of family sphere and Availability of digital devices in the home are made up of elements of choice and are not eligible for validation):
Internal Consistency Analysis, in the sense of endowing the items with significance, in other words, ensuring that each of them measures a portion of the trait or characteristic studied. To this end, Cronbach’s Alpha coefficient was used [38
Analysis of the discrimination capacity of the elements to reinforce the one-dimensional nature of the test. Student’s t
-test was applied to the mean values of the established groups, indication of validity endorsed by García, Gil y Rodríguez [39
The 30 people chosen for the pilot study are taking part in indicated prevention programmes at six centres run by Proyecto Hombre (Asturias, Catalonia, Madrid, Malaga, Melilla and Murcia) in a similar proportion in each of them (17.7%). 86.7% are male, and the remaining 13.3% are female. The age of the participants ranged from 14 to 22, and the majority of them had completed secondary education (73.3%). This number of people is valid to carry out this phase of the study according to McMillan and Schumacher [40
Looking at the evaluation of the elements in the “Personal sphere” dimension, we see that the total value for Cronbach’s Alpha in the scale (0.505) represents an acceptable correlation [41
], an acceptable level of stability in the responses; hence, this part of the instrument presents signs of guaranteed reliability.
The discriminatory power of all the items in the test reinforces its one-dimensionality. To carry out this procedure, three closed items were chosen with ordinal response choices (response scale from 1 to 5) so that the sum total was recoded into three groups (Low, Medium and High):
1 = Low group (minimum value, percentile 33): (6, 16)
2 = Medium group (percentile 34, percentile 66): (17, 23)
3 = High group (percentile 67, maximum value): (24, 35)
By applying Student’s t
-test for independent samples, we were able to establish the existence or non-existence of statistical difference (n.s. = 0.05) between the groups that score low and high in the selected elements. The results obtained using this test based on the 6 items belonging to this dimension and present in Table 5
show that 67% of the element possesses acceptable statistical discriminatory power, which indicates acceptable levels of validity in this dimension. The nondiscriminatory elements were maintained on account of their relevance in the instrument and in accordance with the suggestions made by the programme officers.
The second dimension analysed, “ICT consumption habits”, presented a low level of reliability, reflected in Cronbach’s alpha coefficient, 0.27. Furthermore, discriminatory power revealed that only 25% of items have a valid discriminatory power (see Table 6
). These results concluded that the dimension lacked acceptable reliability and validity criteria, and so we followed the recommendations of the prevention programme officers, by eliminating elements without discriminatory power and incorporating 13 new elements that would ensure the clarity and pertinence of the elements in the dimension according to the characteristics of the target population.
Analysis of the 14 scaled items of the dimension “User’s reasons for consuming ICT” revealed a guarantee of stability in the measure they offer, reflected in the Cronbach Alpha coefficient of 0.58. The application of the item discrimination test provided data that supported the observations proposed by the prevention programme officers participating in the study, since only 43% of items offered acceptable validity (see Table 7
). The relevance of the contents processed did not lead to the suppression of any element but did lead to a redrafting of items where the discrimination was confusing.
A Cronbach Alpha coefficient of 0.828 guaranteed the reliability of the 10 elements that make up the dimension “User’s emotional management”. Furthermore, the analytical test to establish the validity of these questions indicated that 70% offered valid content, leading us to revise the wording of three items (see Table 8
The reliability of the scaled items in the dimension “ICT in the family setting” when the subject is living in the family home is very high (Cronbach Alpha = 0.839). Sixty percent of these items possess an acceptable discriminatory power, which validates their inclusion in the instrument (see Table 9
). Four of these items required further work based on the suggestions of the programme officers, who suggested that they should not be removed from the instrument but instead the wording needed to be revised to make them more understandable.
The eight elements that make up the dimension “ICT in the social setting”, once the relevant reliability test had been applied, contributed a global value of 0.890, clearly demonstrating their metric consistency. Furthermore, the application of the validity test showed that 100% of these items possess discriminatory power (see Table 10
The internal consistency of the nine scaled elements that describe the dimension “ICT in the education setting” was 0.880, scientifically guaranteeing their reliability. The validity of these items, measured by means of the corresponding item discriminatory test, showed that 89% measure the construct covered by this dimension, with just one of the items requiring revision. Furthermore, it should be noted that, at the request of the programme officers, a new element was incorporated into this dimension on account of its relevance for the subject studied (see Table 11
Finally, the eight elements that make up the dimension “ICT in the work setting” possess a high degree of reliability, as reflected in the Cronbach Alpha coefficient, 0.769, although in contrast these are not items that possess an acceptable level of discriminatory power, and accordingly all of them needed to be revised (see Table 12
). However, it should be noted that only 5% of these prevention programmes participants are engaged in the employment market; hence, the results obtained understandable. Accordingly, the decision was made to maintain the elements as originally formulated.
Following the same analytical dynamic as in the previous stages, the result of this third stage was a tool in the format of a personalised interview administered by prevention programme officers, made up of 10 analytical dimensions, 50 evaluation elements, and 156 items (see Table 13
). As indicated previously, it is supplemented by a first round of elements, the purpose of which is to identify the user who is providing the information, along with an observations section in each dimension, and one final question related to final observations about the tool, all of which are not included in the total number of elements that make up in the final instrument (see Appendix A