Methodological Proposal for the Detection of the Composing Elements of Vulnerability Regarding Disinformation

Abstract

We live in a hyper-informed society that is constantly being fed with information stimuli. That information may not be correct, and society may be vulnerable to it. We present a methodological proposal with a mixed approach that allows the learning of the characteristics and weaknesses of news consumers in the face of disinformation. Said methodology moves away from the traditional model, and with it a new, much more complete and complex way of conducting discussion groups is carried out. The qualitative approach is carried out through the creation of an online community in which subjects are encouraged to participate in different activities and tests. On the other hand, in order to obtain quantitative data, a quasi-experimental survey where respondents are exposed to various stimuli created ad hoc, which seeks to measure the interest and credibility of different news items through an orthogonal design, is carried out. The use of this methodology will allow for an expansive and intensive approach to the knowledge of societal vulnerability factors, and with the subsequent results, a solid basis of disinformation can be established, which will allow for the development of a series of strategies to combat disinformation.

1. Introduction

Social science research evolves over the years, as have the discipline’s own objects of study. The rise of social networks, new media, technological advances, and other forms of media consumption, among other transformations, means that the analysis techniques traditionally used have to be updated and adapted to the current moment.

Currently, one of the topics that is arousing most interest in the scientific community is the analysis of disinformation and fake news in its broad spectrum, from its production to the effects it produces on citizens. Numerous authors study the factors that condition a person to be more or less vulnerable to disinformation.

The concept of vulnerability in this context can be understood as the weakness of consumers to identify manipulation (intentional or unintentional) by the media in sharing false or incorrect information (for further development of the concept of vulnerability, please refer to the studies conducted by Menczer and Hills [1], the report from the Council of Europe [2] or the research of the Data & Society group, where it is possible to highlight the one by Paris and Donovan [3]). To achieve this, different researchers use quantitative techniques, such as surveys, and/or qualitative techniques, such as in-depth interviews or focus groups. These types of techniques, which have a long history and a continuous presence in communication studies, have shown over time that they comply with the scientific rigour they are supposed to have if they are used correctly. In the case of surveys, they make it possible to reach a large yet superficial number of the population in an extensive yet superficial way. In the case of interviews or discussion groups, they go deeper into the subject with small profiles of the sample in a significant yet not representative way. As Wimmer and Dominick explain [4], these techniques prevent us from getting to know the individual in his or her different facets, such as his or her values, feelings and emotions in the face of misinformation. It should not be forgotten that a survey is often presented with closed or scalar questions that prevent the individual’s position from being developed more broadly. In the case of the focus group, where in less than two hours individuals have to present their opinions with the disadvantages that this entails, such as the negative effects of “Face-to-Face” group meetings, the monopolisation of the group by one of the members or the complexity of the analysis since, on many occasions it depends on the non-verbal reactions of the participants and also on their communication styles.

This article presents a methodological proposal that aims to make up for these shortcomings thanks to the structure of the tool, which through various aspects, such as anonymity, the iteration of different formulas when posing a problem so that decisions can be modified or argued, feedback with moderators and with the rest of the participants, intragroup homogeneity and intragroup heterogeneity of the participants, ensure the validity of the results. On the other hand, as will be seen below, different activities and processes are used to obtain information from individuals in a holistic way, covering a wide variety of fields and taking into account spatial and temporal dimensions (it lasts more than two months), so that chaos is not formed, and there are no individuals who monopolise the research.

Some works approach the study of vulnerability factors from inductive–deductive processes. Monteiro Borges and Rampazzo Gambarato [5] study the role of beliefs and behaviour on Facebook and their relationship with fake news, using a methodology that encompasses the qualitative conceptual study of Peircean semiotics, focusing on different concepts, such as reality or perception, to investigate the relationship between algorithms, fake news and transmedia journalism. Even within serious games, fake news and disinformation have been the subject of study by Gomez and Carillo [6].

Many authors deal with the study of cognitive skills, attitudes or cognitive biases, among other explanatory elements found at the base of vulnerability, from experimental or quasi-experimental research. This is the case for Saunders and MacLeod [7] in their search for the circumstances related to misinformation and its variation, depending on aspects such as memory, or that of the methodology followed by Dibbets and Meesters [8] to corroborate confirmation bias in children and young people. Although, these authors point out that, among the limitations of the experiment developed, the children who participated in it were forced to select one of the answer alternatives presented.

However, authors who have used the experimental method to study the effects of levels of political competence and media literacy on the detection of manipulated information point out its suitability, due to the need to test cause–effect hypotheses [9].

In these study processes, the use of tasks to assess different skills is common, as in Wineburg and McGrew [10], Wineburg et al. [11] or Nygren and Guath [12]. In some cases, participants are exposed to the evaluation of digital information in sessions conducted by the authors, such as Wineburg and McGrew [10], who recognize that any task that involves researchers creates an artificial environment that can distort what people usually do.

Other researchers combine the use of online and offline assessment tasks, such as McGrew et al. [13] in their evaluation of students’ civic online reasoning. In other cases, the use of these tasks is carried out through an online survey, in which the test items are included. Hatlevik et al. [14], to detect the factors that can influence how people navigate new information, facts and digital environments, used a survey with a digital competence test and a self-administered questionnaire. Nygren and Guath [12], in trying to establish the difficulties and abilities in determining the credibility of digital news, used an online survey with evidence elements designed by researchers in education and psychology, in collaboration with active teachers and in line with previous works [13,15], to ensure its internal validity and reliability. Although, they point out as a limitation that the sample is not random, and the survey is collected by teachers interested in signing up for their classes in order to participate. Instead, Kelly [16] in his study of people as biased information processors, conducted a nationally representative survey experiment combining two powerful research tools.

This paper aims to provide a methodological proposal that can be applied in subsequent research to achieve conclusive and profuse results regarding the vulnerability factors to misinformation, combining the validity and scientific rigor of the reviewed research, while increasing the benefits of traditional techniques and reducing the disadvantages that these may present. This methodological proposal has already been applied to the national project, “Study of the conditioning factors of misinformation and proposal of solutions against its impact based on the degrees of vulnerability of the groups analysed”, financed internationally by the Luca de Tena Foundation and the social network Facebook, within the framework of the public contest “Academic research on disinformation in Spain and approach to: anthropological, economic and sociological aspects that motivate it, history, expansion and current situation and proposals for solutions”, which had an endowment of 62,000 euros.

2. Contextualisation of the Proposal: What Does Disinformation Mean?

In order the understand how the methodological proposal works, it is necessary to explain how it works and what effects disinformation generates.

Disinformation has become a global phenomenon, to the point of talking about the post-truth era and the post-factual world [17]. The rise of social networks and their preponderance in the reconfiguration of public information spaces has led to citizens being overexposed to disinformation. The digital ecosystem in general, and social networks in particular, favour disinformation by increasing the potential audience of these messages and allowing for their re-broadcast [18]. This, together with the increase in connections between individuals and the speed of information transmission, allows an essential factor in its proliferation: virality. In addition, the increasing demand for news focused on emotional aspects, which is in turn more likely to be shared, also benefits disinformation [9]. On the other hand, this context prioritizes the immediacy of the information over reliability, affecting journalistic production routines [19,20] and increasing the difficulty of communication media to contrast much of the information they produce.

The High-Level Expert Group on Fake News and Online Disinformation of the European Commission recommends the use of the term disinformation when referring to bad informative practices [21]. The widespread expression “fake news” usually refers indistinctly to three concepts with different nuances: misinformation, disinformation and malinformation. Malinformation refers to information built on real data that, regardless of its informative relevance, is used as a weapon to attack people, organizations or states [18]. Misinformation and disinformation are distinguished by the intention of the sender in spreading false information, which in the case of disinformation is carried out with the knowledge that this information is disinformation. Misinformation is increasingly recurring with forwarded messages, especially on social networks, which focus on, among other issues, non-existent false public health alerts or simply on erroneous information [22]. On the other hand, an example of disinformation would be the creation of deceptions known as hoaxes [23].

Although there are authors who had not at first considered intentionality as a disinformative characteristic, pointing out only the misleading nature of the content [24], this element was later added, making it a common consideration as an indispensable intrinsic factor [25,26,27,28]. Other authors broaden the meaning of the term by also including the lack of information on certain topics from citizens [29,30].

Fallis points out that disinformation continues to be a type of information, misleading and intentional, with the ability to create false beliefs about the world or reality, and characterizes it with, among other things, the following distinctive notes [31]: it is usually a government or military activity; it is often the product of a carefully planned and technically sophisticated deception process; it may not come directly from the source attempting the deception; it can be widely distributed or, conversely, targeted at a specific group or organization. Disinformation, which ultimately produces an abuse of power [32], ends up harming the population by eroding their trust in institutions and the media [31].

The importance of its potential effects has captured the attention of European institutions [33,34], as has the determination of the causes of vulnerability to disinformation, the effects of which differing, among other factors, by educational levels, democratic culture or trust in institutions [33].

The disappearance of the gatekeeping function that was traditionally associated with mass communication has generated as a secondary effect a great difficulty, on the part of the recipients, to discern what is trustworthy and what is not [35]. The ability to assess the credibility of information is related to cognitive skills and attitudes. The lack of knowledge of the digital environment is very high [36], detecting a digital divide that reflects levels of education and social inequalities [12]. Similarly, previous beliefs, the coherence of the message and cognitive ability [12], as well as ideologies, age, memory and personality traits, can affect the evaluation of information [37]. All of which can be increased by the consumption occasion in which subjects may find themselves, perhaps overworked, tired or stressed, making it difficult to undertake effective evaluations [38].

Likewise, the cognitive biases used for the extension of informational disorders [39], such as confirmation bias in the evaluation of fake news, a natural tendency to blindly second messages related to one’s own beliefs [40], should be considered. This has been found to be true for all age groups [41]. The consonance of the falsehood with the beliefs and desires of the receiving subject reduces the possibility of questioning their falsehood [42], since beliefs are typically resistant to change. This is the case even in the face of data that contradict said beliefs, especially in cases with a strong ideological association [43,44].

On the other hand, indicators such as people’s perception of how often they find false information and their confidence in identifying it show differences when observed through sociodemographic analysis, revealing, for instance, that it is the youngest (between 15 and 24 years) and respondents with the highest educational levels who say they more frequently find fake news [45]. However, this confidence clashes with the position of some authors who have determined that especially young people may find it difficult to assess whether online information is reliable and to recognize disinformation [41,46].

In order to investigate disinformation in Spain, a systematic research methodology was built and endowed with economic resources that allow for articulating this central concern on disinformation. It aspires to contribute significant data to a heated debate with undoubted social impact.

This work presents a methodological proposal for the study of the constituent elements of vulnerability to disinformation. This methodology has already been applied in a research project that analyses the conditioning factors of disinformation and has been partially replicated in other investigations with positive results, which guarantees its validity and replicability.

3. Methodological Proposal

The proposed methodology is capable of shedding light on the characteristics of disinformation, with a special emphasis on the detection of vulnerable audiences and the subsequent formulation of actions whose implementation can help combat the effects of the phenomenon effectively.

The data obtained from the application of this methodology can provide systematic and reliable support for the actions proposed to combat the phenomenon of disinformation. The research design was planned in a way that approaches the problem from complementary methodologies and presents sufficiently broad coverage.

In this way, a work system is proposed to investigate the degree of vulnerability to disinformation concentrated on the independent variable of content consumption. The aforementioned system methodologically triangulates between quantitative and qualitative techniques (surveys and discussion groups in the Sensors online community, respectively) to increase the reliability of the results by approaching the phenomenon studied from different perspectives.

It is understood that the results that will be obtained from the two techniques will not only complement each other but will suggest similar conclusions, as other research has shown [47].

Thanks to the different techniques of the methodology being based on an exploratory approach, they can respond to a wide variety of research objectives. The following are examples:

• Learn the conditioning factors related to the effectiveness of disinformation.
• Study the profiles of the most affected audiences and identify their vulnerabilities.
• Catalogue possible actions that are specifically targeted and adapted to the different audiences identified.
• Determine key aspects in the construction of solutions to reduce the impact of disinformation.
• Establish whether educational level is associated with a greater susceptibility to disinformation.
• Clarify whether age is a factor related to a greater susceptibility to disinformation.
• Identify the incidence of other dependent variables of the subjects, namely:
  • Sociodemographic variables: level of income, social status, area of residence.
  • Intermediate or psychosocial variables: political orientation, strategies and thought patterns, attitude components, social attribution phenomena, cognitive dissonance, stereotypes, vulnerability to rumors, etc.
• Explain the effect of stimulus-dependent variables on susceptibility to disinformation:
  • Influence of the source, support and type of channel on the level of acceptance of disinformation.
  • Role of the information content and level of specificity of said information.
  • The effectiveness of disinformative content enhanced by the presence or lack thereof (along with the accumulation) of news values.

3.1. Methodological Design

A complex study with different phases has been designed to propose a method for a comprehensive study of disinformation. It is based on data production and cross-analysis tools. For this purpose, the qualitative and quantitative approaches are separated into two different studies, whose data can be pooled for validation and new results. Since, as suggested by Morse and Chung:

“[...] simultaneous or sequential triangulation of more than one qualitative method or combining qualitative and quantitative methods provides a more balanced perspective, moving toward holism [...] The use of multiple methods leans toward developing a systematic research program, with one study dictating the direction and nature of the next. In this way, the researcher may carefully identify and encompass the scope of the phenomena or project, with each study being complete in itself [...] With minimal overlap between these projects, but with each project validating and extending the previous, the results may be fit together to form an understanding of the concept” ([48], p. 18).

So that the results are confirmed by this comparison, as Hernandez Sampieri, Fernandez Collado and Baptista Lucio indicate, “the idea is that when a hypothesis or result survives the confrontation of different methods, it has a higher degree of validity than if it is tested by a single method” ([49], p. 789).

The proposed timeframe is as follows:

• Workshop or initial meeting, to agree on data and steps to be taken and to delve into objectives and define the appropriate timing.
• Qualitative phase: field work commencement.
• Quantitative phase: work process from qualitative field production to the launch of a quantitative questionnaire (cross-analysis approach and interphase triangulation).
• Integrated analysis of qualitative and quantitative data. Review of the state of the art in search of psychosocial models that explain the findings and phenomena uncovered.
• Final project meeting: project closing workshop and presentation of the results report.

3.2. Target Study Variables, Field Work and Research Phases

Regarding the sample cross variables, the following decisions are made to better address the susceptibility to being disinformed:

• Factors that condition susceptibility. The design by sample quotas that separately combine educational level, level of income, performance or not of paid work, etc. is suitable for this methodology. It is named social position.
• From here, the variables of age, sex, life cycle moment and social position (as an alternative to the typical social class variable) are identified. The combination of the aforementioned variables offers the necessary data to know the most susceptible categories to disinformation.

Finally, the complex cross-sectional variable of “disinformation effectiveness” is, at least, threefold:

• News values (presence or absence) in the content of the message, valence (+/−) and intensity.
• Effectiveness is understood as making false information credible in the terms set forth above. For effectiveness to be established, there must be, as a minimum, a successful process of established social influence—if not social power directly—and a high enough credibility factor associated with the messages for them to be accepted by individuals. All this is closely related to the elements of communication.
• Identification of the aforementioned elements of the communicative process and their role in the process of establishing influence: sender(s), receiver(s), channel/medium, code, referent, noise and feedback, amongst others.

After the initial workshop, the qualitative phase and the sample design that will best adapt to the object of study are defined, using innovative methodologies, such as web squared (a hybrid device with online and offline moments).

More specifically, the Sensors community and its platform are used together with other ethnographic work environments (mobile groups via WhatsApp).

This device of an online community plus smartphones multiplies the possibilities of traditional qualitative research since it allows:

• Provision of a group space and personal privacy.
• Exploration of motivations and meanings, as well as deep psychological keys (psychometric test application adapted to a private web environment).
• Learning from the informants, guidance of new steps by making activities more flexible and inclusion of objectives that arise in the course of the investigation. Thinking in a “crossed impact” mode with the participants and members of the research team with successive adaptations of the tool and collection of the reality of the participants in their own environment.
• Working both with the reflection of the participants and with the most lively, immediate game of response in group.
• Detection of trends and the according development of proposals.
• Provision of a strategic focus.
• Above all, the allowance of a quasi-experimental structure to be applied throughout the device, through:
  • Firstly, the exploration of the “opinion” of the participants on the matter both individually and as a group, and to encourage the exchange of perspectives, the subsequent elaboration, the reflection (discourse), etc.
  • After a reasonable period of time (a fortnight), begin to present informants with various news items (true or false) for their consideration without clarifying whether the content is true or false, in order to verify de facto, the real vulnerability to disinformation on the part of the sample (the behaviour).
  • This allows the contrast of the intentions and theoretical disposition of the sample to canards, errors, fakes, etc., with their actual coping behaviour, observing the inconsistencies between both variables, as well as subsequently guiding a purely experimental methodology in the quantitative phase to verify the first data obtained.

3.3. Qualitative Phase Specifications

The Sensors community is an online tool of reciprocal and multichannel impact that, creatively and continuously in the period that is established, launches different qualitative ways of doing research through different spaces, accessible through various interaction supports on/off: computer/tablet plus smartphone, etc. These spaces include:

• Ethnographic diaries (a space of intimacy/a space of identity): reflection is done intimately, but it is also a projective and symbolic space where participants develop storytelling, personal concerns, etc.
• In this space, stories of experiences are configured, thus encompassing the way in which subjects live and relate to the stimuli, phenomena or processes that are investigated.
• My wall (an intermediate window between the intimate and the private in the appearance of the social network Facebook): each individual has a wall on which they express themselves as they want, and to which the community can freely access and react to these communications.
• It is a space for self-pronunciation, a place for each informant to have a voice. The idea of self-expression before an eco-community is reinforced.
• Agora (community blog as an exploration workshop open to all): a true collaborative workshop on ideas, concepts, projects, communication, etc. In the agorae, experiences and opinions of the subjects’ day to day are shared and recreated. A fluid dynamic is constituted: a bonding energy emerges at the same time that content and relationship are shared. Agorae differentiated by segments are constructed each time it is necessary to have a selection of sample variables for specific activities.
• Test (parallel consultation): in this space small surveys, games or confidential tests are proposed where the participants respond individually. It allows the carrying out of specific activities and consultations, providing detailed information on those aspects to be investigated.
• WhatsApp (ethnographic pocket space that allows access whilst on the move, with a highly naturalized chat): through this app it is easy to naturalize a group with the informants; although they are in itinere, it allows a greater implantation and plasticity compared to Sensorsapp.

3.3.1. Implementation Process

The way of carrying out a global investigation involves a progressive blending of the different qualitative elements worked on, which is:

• Data production → continuous analysis of said data → input of learnings in subsequent steps → until the construction of the final quantitative phase.

In fact, the last tasks and qualitative analysis coincide in their final stretch with the beginning of the quantitative field, since they support from the base a strategic definition of the device, its clearly experimental approach, the definition of the variables to be manipulated and the news content tested with the questionnaire used.

Commencement of the qualitative phase is the beginning of field work on the Sensors platform and its continuity through different tasks, supports, channels and challenges:

• Once the qualitative phase is advanced, the data collected from an activity is analysed and guides the approach for the next activity, progressively integrating the different partial results and shaping the information production instruments.
• According to this dynamic, different test are launched in the form of consultations or study activities, many of them with an experimental cut, namely:
  • Test on thought patterns, self-completion → it reports thinking patterns and cognitive styles with which individuals deal with news and new data about their environment.
  • Agorae of group discussion divided according to different variables/sample segments on news of up to six different topics, including news with a high degree of “disinformation” → this provides group information, shared in small well-organized rooms of people who share a common score on significant variables (political ideology, usually).
  • Voluntary initiatives of strategy/voice or appeal to other informants in the intermediate spaces of communication (semi-public) that are the personal walls.
  • Here the dragging/summoning capacity of specific topics and people, as well as concerns of the moment which generate echo and contagion, content/topic trends, public opinion thermometer, etc., can be observed.
  • Individual information about the private sphere, including dreams, desires, attitudes, personal fears, etc., in a private and absolutely ethnographic space where the participants confess intimate realities that are difficult to access in other settings (as well as values, beliefs, etc.).
  • Confidential individual exercises in the ethnographic diaries with, on the one hand, a qualitative phase where an online community has been carried out using the Sensor platform and, on the other, a quantitative phase based on an online questionnaire.
• Progressive and spiral analysis of the data so obtained using the aforementioned tools with a cross or triangulation approach to shed light on the quantitative tools, their structure, design construction and base questionnaire to launch the field work.
• Integrated analysis of qualitative and quantitative data. Review of the scientific literature in search of psychosocial models that explain the findings and phenomena found.

3.3.2. Sample Design of the Online Community

The relevant variables for the sample design of this methodology, which are inspired both by the proposed objectives and the considerations that arise in the initial workshop, are the following:

• Sex: online community (preferably formed by 50% women and 50% men).
• Geographical area: capitals and areas of influence.
• Social class: according to social position, obtained according to income and educational levels (combining these variables with paid or unpaid work). When taking into account social position (disaggregating education level and household income level), we have a greater wealth of nuances, which multiplies the explanatory power.
• Life cycle moment: specific segments with more individuals in the family segments are chosen to collect the different casuistry (with and without children, young and older children, etc.).
  • Adolescents + youths.
  • Young stable couples without children aged 28–34 and couples with young children up to 12, aged 30–45.
  • Couples with children over 12 years of age between 44–55 years old.
  • Empty nest + 55 years old. Different political attitudes: distribution according to subjective statement on political ideology (left, centre-left, centre, centre-right, right and undefined). For the adolescent segment, the distribution according to political ideology is not considered; a group of its own is configured with this segment.
• Different employment situations (paid or unpaid job, pensioner, unemployed).
• Access to technological equipment: mobile phones or smartphones, normal television, computer, tablets, etc. Different levels of access to the internet and the media (on/off) were distinguished in the segments, as well as people who work and people who do not work.

3.3.3. Selection of Participants

The selection of participants for the community was carried out in three phases:

• Preselection of informants: First selection with the main segmentation variables. It is the first contact to accept participation in the online community. The participant is strongly involved in the process from the beginning.
• Interview/personal letter: It delves into lifestyle and key segmentation variables. This provides a prior qualitative knowledge of the participants of great value to the members of the research team.
• Task development: Preselected candidates will be validated for their expression skills through videos, collage, ethnographic diaries, storytelling, chats, etc. This process will last eight weeks.

From the beginning, participants are informed that there will be two or three dynamics per week.

3.3.4. Activities and Examinations

Some details and elements of the different activities and examinations require further specification:

• Moldes test: through the “test” tab, a test of cognitive/emotional strategies composed of different items is published to the participants where, individually and confidentially, they must answer a series of questions specifying their level of agreement (in a Likert scale from 1 to 5).
• Portrait-story: It is proposed that the participants describe a little more about their personal environment (way of being) and aspects of their reality. Through this activity we can further specify the political spectrum of the participants. It is carried out in the ethnographic diary of the participants; since this is a space of intimacy and identity, it is considered to be the most appropriate space on the platform.
• Different news items: News items from various sources and channels (Twitter, digital press, Instagram, WhatsApp...) are presented. Some of them are canards or false news items. This is done with the intention of knowing and understanding the reactions and attitudes towards possible untruthful items of news. Such pieces are published in the ethnographic diaries, since it is considered that, in this way, the participants can show their perspective and point of view without being conditioned by the rest of the informants.
• Agorae by thematic areas: Different agorae are published through which different topics are addressed (politics, sports, technology, society, environment, science, social networks, amongst others). Through these agorae, news pieces related to the subject in question are raised in order to know the types of channels in which the participants consider that the news piece may appear, the sources, as well as the dissemination of the news and interest in it. The agorae, in addition to being categorized by thematic areas, are also segmented by ideology (based on the data obtained in previous activities in relation to the political spectrum of the participants).
• News items proposed by the participants themselves: Simultaneously to all the proposed activities, the participants publish on their walls those news items that they considered interesting and choose to share with the rest of their peers. Such news pieces allow to better understand the interests of the participants, as well as the usual channels and sources they use when it comes to acquiring information.
• Portable groups via WhatsApp: For the period of a week, groups are held with informants through which issues related to information, disinformation, erroneous or partial information are openly raised. In them, the exchange of opinions and perceptions of the concept of disinformation is raised, as well as their concern and repercussion in relation to it.
• Scale of attitudes on issues related to national security, the progress of the economy, confidence in the future, the vision of human morality, progress, amongst others are used in order to contrast the results with the responses and attitudes expressed towards news items throughout the experience.

3.4. Quantitative Phase Specifications

Disinformation is a broad, complex and multifaceted field of study. However, from a quantitative perspective, it is essential to have concrete, well-defined and operationalizable concepts. Following this premise, the first step is to establish what is possible and what is not possible to measure. This implies, necessarily, leaving out of the scope of the study certain complex (if not impossible to be addressed with guarantees through quantitative methodology) aspects or dimensions. That is, in this phase the field of vision of the study must be reduced to ensure adequate observation.

In order to determine which factors make disinformation more or less credible, it is essential, firstly, to define these factors in order to measure and quantify their effect. In this sense, the qualitative phase helps, among many other things, to establish an exhaustive list of elements that affect the greater credibility of information.

The first task is to specify the possible scope of the quantitative phase by selecting, from the qualitative list, those factors that it is possible to use. Two criteria are taken into account in this selection: importance and measurement. That is, the factors whose effect on credibility are clearer are selected (disregarding others with a more residual weight), and which in turn were transferable to a questionnaire. Analysing the list extracted from the qualitative phase, it is possible to group the factors into three large dimensions:

• Those that have to do with the channel; understanding channel as the way in which an individual learns or finds out about the (dis)informative piece.
• Those that are related to the source. That is, those that are related to who originates/creates the news item (Inside this, we believe it is appropriate to explain the importance of including the ideological line as one of the factors to be analyzed, given that numerous studies have shown its relevance in influencing different groups [50,51], as well as the preference and predisposition to believe certain news if they appear in the media whose ideology is close to one’s own, as indicated by the Theory of Uses and Gratifications by Katz and Blumler [52]. The reality is that the reader interprets the news in one way or another according to the media from which it comes. This information is not offered during the experiment, so introducing the ideological line can make up for this lack).
• Those that have to do with content. That is, with what and how the (dis)information is told.

Under this scheme, the following factors are selected:

In total, ten different factors are measured that meet the two aforementioned selection criteria: they are relevant for the message to be more or less credible, and they are measurable.

The list of factors selected has, in turn, additional implications that affect the scope of the quantitative phase of the study. The items of (dis)information used in the measurement must fit said factors.

It is therefore necessary that the pieces used have a clearly identifiable source/origin. Consequently, canards such as the ones that arrive via WhatsApp and that, regardless of the sender, are anonymous, are left out.

In turn, it is necessary that they are (dis)informative pieces that have been (supposedly) published in some written medium (online or paper). Rumours that come from informal comments from the environment or false news from media such as television or radio are left out. To make the news items more realistic, respondents will read the headlines. Expressing television or radio news pieces in writing makes the situation too artificial. Consequently, it is decided to omit these media in order to guarantee a higher quality of the data. In addition, to achieve certain realism, it will be necessary to create videos and audios of the different news items, which is excessively complex.

3.4.1. Prospective Approach

Once the what (range) is defined, it is necessary to make decisions about the how (focus). From a methodological perspective, two possible paths are opened to face measurement:

• Analysing the past. That is, to analyse what respondents have already done: what news has caught their attention recently, how they have reacted to said news, what degree of credibility they gave it, etc. This alternative has some significant problems: the imprecision of memory, the effect of the theory of the spiral of silence or the prudent lie and, perhaps the most relevant, the lack of control over the specific news items in terms of the factors previously defined.
• Analysing the (possible) future. That is, to analyse what the respondents may do in the face of certain exposure to (dis)information. The main problem with this path is that respondents are placed before a fictitious (laboratory) situation. However, on the one hand, it allows control over the news items to which the interviewees are exposed and, on the other, the harmful effects of resorting to recollection are avoided.

After analysing the pros and cons of the two possible alternatives, it is more convenient to opt for the second path and confront the respondents with an experimental situation: they will be exposed to a series of stimuli (news pieces) and they will indicate, through the questionnaire, what their perception is of these and what behaviour they have before them.

3.4.2. Design of Stimuli for the Questionnaire

The following step, once the approach is decided, is to establish the stimuli that will be used in the measurement. At this point it is important to recall that one of the objectives of this methodology is to learn what factors affect the greater or lesser credibility of a news item. For this, there is a defined list of ten factors with their corresponding levels (

Table 1. Dimensions, factors and levels used.

Dimension	Factor	Level
Channel	Via	Social networks of popular public figure
		Social networks of non-popular public figure
		Social networks of unknown persona
		Nearby environment
		Direct source
	Repercussion	Much impact (likes, retweets, comments, etc.)
		Little impact
Source	Media	Online
		Traditional
	Track record/reputation	High reputation
		Reduced reputation
	Ideological line	Left
		Centre-left
		Centre-right
		Right
	Scope	Numerous readers
		Few readers
Content	Theme	Politics
		Technology
		Economy
		Health
	Specificity	Specific with data
		Non-specific with data
	Time frame	Breaking news
		Continuity news
	Style	Sensationalist
		Moderate

Source: own elaboration.

). The stimuli that are used, consequently, must be able to be characterized as univocally as possible in a level of each of these factors.

It is advisable not to resort to fake news already published, given the complexity of finding valid examples for all the necessary options. At the same time, the possibility that some respondents already know the news items is avoided and their spontaneous reaction to it is not measured. Therefore, it is necessary to use fake news ad hoc designed for the investigation. However, how many news items are necessary? All the possible combinations of the factors and the levels used suppose such a high number that it seems virtually impossible to handle them in a single investigation. In this situation, it is recommended to generate a fractional factorial design (

Table 2. Fractional factorial design example.

Via	Repercussion	Media	Track Record	Ideological Line	Scope	Theme	Specificity	Time Frame	Style
Direct source	Much impact	Traditional	Reduced reputation	Centre-left	Few readers	Health	Specific with data	Continuity news	Moderate
Social networks of non-popular public figure	Little impact	Traditional	Reduced reputation	Left	Numerous readers	Health	Non-specific with data	Breaking news	Sensationalist
Social networks of popular public figure	Much impact	Traditional	Reduced reputation	Centre-right	Numerous readers	Economy	Non-specific with data	Breaking news	Moderate
Social networks of non-popular public figure	Little impact	Traditional	High reputation	Left	Few readers	Economy	Non-specific with data	Breaking news	Moderate
Social networks of non-popular public figure	Little impact	Online	Reduced reputation	Centre-left	Few readers	Economy	Specific with data	Breaking news	Moderate
Social networks of popular public figure	Little impact	Traditional	High reputation	Right	Numerous readers	Health	Specific with data	Continuity news	Moderate
Social networks of popular public figure	Much impact	Online	Reduced reputation	Left	Numerous readers	Politics	Specific with data	Breaking news	Sensationalist
Nearby environment	Little impact	Traditional	Reduced reputation	Right	Few readers	Politics	Non-specific with data	Breaking news	Sensationalist
Social networks of non-popular public figure	Much impact	Traditional	Reduced reputation	Centre-right	Numerous readers	Politics	Specific with data	Continuity news	Moderate
Social networks of popular public figure	Much impact	Online	Reduced reputation	Right	Few readers	Health	Specific with data	Breaking news	Sensationalist
Social networks of unknown persona	Much impact	Online	High reputation	Right	Numerous readers	Economy	Specific with data	Breaking news	Moderate
Nearby environment	Much impact	Traditional	High reputation	Centre-left	Numerous readers	Economy	Specific with data	Continuity news	Sensationalist
Social networks of unknown persona	Little impact	Online	Reduced reputation	Centre-left	Few readers	Politics	Non-specific with data	Continuity news	Moderate
Social networks of unknown persona	Little impact	Traditional	Reduced reputation	Left	Numerous readers	Technology	Specific with data	Continuity news	Sensationalist
Social networks of popular public figure	Little impact	Online	High reputation	Centre-left	Numerous readers	Technology	Non-specific with data	Continuity news	Sensationalist
Social networks of unknown persona	Much impact	Traditional	High reputation	Centre-right	Few readers	Health	Non-specific with data	Breaking news	Sensationalist
Social networks of unknown persona	Little impact	Traditional	Reduced reputation	Right	Few readers	Economy	Specific with data	Continuity news	Sensationalist
Social networks of non-popular public figure	Much impact	Traditional	High reputation	Centre-right	Few readers	Technology	Specific with data	Continuity news	Sensationalist
Direct source	Little impact	Traditional	High reputation	Right	Numerous readers	Technology	Non-specific with data	Breaking news	Moderate
Social networks of non-popular public figure	Much impact	Online	High reputation	Right	Numerous readers	Politics	Non-specific with data	Continuity news	Moderate
Direct source	Little impact	Online	High reputation	Centre-right	Few readers	Politics	Specific with data	Breaking news	Sensationalist
Nearby environment	Little impact	Online	Reduced reputation	Centre-right	Numerous readers	Technology	Specific with data	Breaking news	Moderate
Social networks of unknown persona	Much impact	Traditional	High reputation	Centre-left	Numerous readers	Politics	Non-specific with data	Breaking news	Sensationalist
Social networks of popular public figure	Little impact	Online	High reputation	Centre-right	Few readers	Economy	Non-specific with data	Continuity news	Sensationalist
Social networks of non-popular public figure	Much impact	Online	Reduced reputation	Right	Few readers	Technology	Non-specific with data	Continuity news	Sensationalist
Social networks of unknown persona	Much impact	Online	High reputation	Left	Few readers	Technology	Specific with data	Breaking news	Moderate
Social networks of popular public figure	Little impact	Traditional	High reputation	Left	Few readers	Politics	Specific with data	Continuity news	Moderate
Social networks of unknown persona	Little impact	Online	Reduced reputation	Centre-right	Numerous readers	Health	Non-specific with data	Continuity news	Moderate
Social networks of popular public figure	Much impact	Traditional	Reduced reputation	Centre-left	Few readers	Technology	Non-specific with data	Breaking news	Moderate
Nearby environment	Much impact	Online	High reputation	Left	Few readers	Health	Non-specific with data	Continuity news	Moderate
Social networks of non-popular public figure	Little impact	Online	High reputation	Centre-left	Numerous readers	Health	Specific with data	Breaking news	Sensationalist
Direct source	Much impact	Online	Reduced reputation	Left	Numerous readers	Economy	Non-specific with data	Continuity news	Sensationalist

Source: own elaboration.

) that, ultimately, reduces the number of possible combinations to a more manageable one.

The generated design must meet the following characteristics:

• Orthogonality: a design is orthogonal when the number of times that a level of a factor is compared with all the levels of the rest of the factors is equal or proportional.
• Balance: a design is balanced when the different levels of each factor are shown the same number of times.
• Positional balance: there is positional balance when all the levels appear in the different positions a similar number of times.

For the remaindering factors, the information should be included in the most specific way possible and without identifying specific examples (

Table 3. Final example of a stimulus.

The COVID-19 vaccine is harmful to health. According to a WHO study, the vaccine will have more adverse effects than positive ones for the older population
In what media is the news item published?	An online newspaper
	Centre-right ideological line
	With a high reputation
	With few readers
Please imagine that you learn about this news item in the following manner:
Through a popular public figure/many followers share it on any of their social networks
The news item has high impact (It is shared, retweeted, sent and commented a lot)

Source: own elaboration.

). In this way, all respondents are provided with the same information. Hence, their interpretation is more controllable and less subjective than when talking about specific media.

It is not possible, for instance, to know how each respondent ranks a specific newspaper in the factors that are being measured. However, if it is directly indicated that it is an online newspaper with a centre-right-wing ideological line and with few readers, the information has a much more univocal and less interpretable character.

On the other hand, it is concluded that exposing each of the respondents to all possible stimuli is unwise. The duration of the interview would be longer than is advisable, causing fatigue in the respondent and, ultimately, making the quality of the responses and the information obtained inadequate.

To avoid this problem, it is recommended to expose each interviewee to six different stimuli. Different message rotations are designed for this, ensuring that each of them is composed of different topics and that each respondent sees the greatest variability of messages in relation to the factors being measured.

Additionally, another objective is addressed: to learn what elements influence the degree of vulnerability that individuals have with respect to the (dis)informative messages. To respond to this objective, the characteristics of each of the respondents must be measured in the questionnaire, in order to see how they relate to the credibility granted by the news tested: sex, age, educational level, exposure to the Internet, degree of interest and information with the topics to which the news refers, ideological position and socioeconomic level.

3.4.3. Weight of Factors According to Interest and Credibility

For every visualized stimulus, each respondent is asked their degree of interest, degree of credibility and their behaviour: would you expand on the information/click on the link/read it in full/share it on your social networks/send it to your private contacts/comment on it with your environment (friends, family, colleagues, etc.)?

The Conjoint procedure was used to determine the weight of each of the factors in the credibility and interest of the news. Conjoint analysis is a statistical procedure that allows the determination of the importance of a series of aspects or characteristics (in this case, the ten factors), without asking directly about those characteristics.

Initially, the Conjoint procedure was developed for its use in mathematical models of psychology and its application to marketing. In fact, its use was limited for many years, mainly for two reasons: the complexity when designing, executing and analysing a study of this type, and the low power of computers in the eighties and nineties, which prevented the analysis of moderately complex Conjoint studies.

In recent years, this technique has once again received the attention of researchers because, amongst other reasons, current computers can carry out complex analyses in minutes (or a few hours). The Conjoint procedure results in a utility score—called a partial contribution— for each level of the factors. These scores provide a quantitative measure that expresses the effect of each level on the credibility and interest of each news item.

3.4.4. Design of the Online Questionnaire

Finally, the technique chosen for this phase is an online panel survey. The decision to use this tool responds, fundamentally, to the need to show stimuli to the interviewees. The online interview enables respondents to read all the information with some confidence. In the case of conducting a telephone survey, it is more complex for the participants to assimilate all the stimuli and give quality responses.

At the same time, compared to other alternatives that also allow the interviewees to read the stimuli, the online interview allows a greater geographical dispersion, guaranteeing, in this sense, a more representative sample.

For the design of the sample, the variables that may a priori have a greater influence on the object of study will be taken into account: age and level of studies, namely. The distribution by sex and geographical area has also been controlled.

Therefore, a multistage sample is recommended, based on proportional and stratified random conglomerates, taking into account the idiosyncrasy of the country where the research is carried out that meets the quotas for sex, age and level of studies. This is performed considering the sampling error standards, where a confidence level of 95.5% is sought, presenting a real error of ±5.0% for the entire sample.

4. Discussion and Conclusions

This work provides a methodological proposal for the study of the vulnerability factors contributing to misinformation. The research design approaches the problem from complementary methodologies. The system methodologically triangulates between quantitative and qualitative techniques providing, as indicated by Hernandez Sampieri, Fernandez Collado and Baptista Lucio ([50], p. 791), greater confidence and validity to the results and greater sensitivity to the degrees of variation not perceptible with a single method.

We have worked with a qualitative technique through the Internet, located between the qualitative panel and the focus group, using an ad hoc qualitative tool, the Sensors methodological instrument, developed by the Analysis and Research group, a digital community subject to different variables, which integrates and represents different social sectors.

At the same time, it is combined with quasi-experimental research, in line with previous work in this field, such as Saunders and MacLeod [7]. The proposed methodology is aligned with the methodological construct followed by many other previous works, such as McGrew et al. [13], Hatlevik, Guðmundsdóttir and Loi [14] or Nygren and Guath [12], in which the use of tasks to evaluate different aspects, such as skills, is common, combined with the use of the survey and, in our case, with a clear bet for its representativeness, such as Kelly [16]. Another fundamental line is that the activity that involves researchers in the process is reduced to a minimum, avoiding creating the artificial environment indicated by Wineburg and McGrew [10].

However, previously existing content is not used, rather it has been developed to offer an ex profeso stimulus in which the specific levels of the proposed study factors are determined, unlike other works, such as the one cited by Nygren and Guath [12], in which the evaluated variables are native ads, unknown comments and scientific evidence. In our case, we design a set of stimuli uniquely characterized at one level for each of the factors. In addition, the established factors have already been used in other work, and some of which has already been published [53].

In short, the application of the methodological proposal results in an integrated analysis whose specification of data, operations, calculations and theoretical models consulted show that what is important for the work process is the following:

• The analysis of results is progressive and continuous during both phases, resorting to the cross-analysis of data of different nature (speech, psychometric tests, scanning of news items, news contributions from informants, spontaneous debates, personal ethnographic diaries, amongst others).
• The exhibition of the sample of fake news and true news is carried out with quasi-experimental criteria so that the stimuli (supports, sources, content and publication scope) are presented through an adequate rhythm and in a rotating model so the results can show construct validity.
• The presentation of results is carried out in an integrated way between some and other inputs.

This has been evidenced in its application to the aforementioned national project, “Study of the conditioning factors of misinformation and proposal of solutions against its impact based on the degrees of vulnerability of the groups analysed”, financed internationally by the Luca de Tena Foundation and the social network Facebook, within the framework of a public contest.

In addition to the data reflected in the report presented, the amount of information collected during the investigation is allowing new results to be obtained by crossing specific information from the different variables that show a series of synergies that had not been contemplated in the initial objectives and hypotheses.

The results achieved, and those that are being achieved with this new approach, make it clear that this methodology allows a holistic study of a polyhedral reality, such as disinformation.

Finally, with the results that can be obtained by applying this methodological proposal, some lines of action with which to deal with disinformation can be proposed:

• Create news verification networks in order to neutralize disinformation campaigns between the different social agents.
• Generate and promote the use of technologies based on artificial intelligence to help detect disinformative content.
• Develop media and digital literacy actions, as well as establishing measures against disinformation for the general public, especially focused on the most vulnerable groups in society.
• Instruct the new generations of journalists and media professionals on good praxis, emphasizing the control of factors that can condition vulnerability to disinformation.