Scale Development and Validation for Psychological Reactance to Health Promotion Messages
1
Department of Media & Communication, Pusan National University, Pusan 46241, Korea
2
Department of Communication, Drexel University, Philadelphia, PA 19104, USA
3
Department of Advertising and Public Relations, Chung-Ang University, Seoul 06974, Korea
*
Author to whom correspondence should be addressed.
Sustainability 2020, 12(14), 5816; https://doi.org/10.3390/su12145816
Submission received: 30 June 2020
/
Revised: 14 July 2020
/
Accepted: 16 July 2020
/
Published: 20 July 2020
(This article belongs to the Special Issue Physical Education, Health Promotion, and Sustainability)
Abstract
:According to the psychological reactance theory, psychological reactance is strongly associated with many adverse outcomes of health promotion messages. This is particularly pertinent when health messages are targeting young adults, as they resist freedom-threatening messages compared to other age groups. However, previous reactance measures either relied on the open-ended thought-listing procedure, or incorporated both antecedents as well as consequences of reactance and state reactance. This study aimed to develop and validate a comprehensive scale to measure the state of psychological reactance specifically toward health promotion messages. To this end, this study was situated in the context of an anti-binge drinking intervention targeting college students. A cross-sectional survey was conducted with 203 Singaporean undergraduate students. The dataset was analyzed by exploratory factor analysis, confirmatory factor analysis, and item analysis. The final 27 items were loaded on eight factors (anger, exaggeration, design derogation, authoritative tone, ineffectiveness, know-it-all attitude, jadedness, and source motive) that accounted for 78.53% of the variance. Each factor showed satisfactory reliability and validity (discriminant, convergent, and predictive). This study specified cognitive reactions by multiple dimensions and examined how they are intertwined with the affective dimension, which is represented by anger. The scale proposed herein will help researchers and practitioners develop sustainable health interventions.
1. Introduction
1.1. Purpose of the Study
Psychological reactance is a motivational state in which people become psychologically aroused by a threat to their behavioral freedom [1]. The underlying assumption of the psychological reactance theory (PRT) is that human beings are autonomous agents who refute persuasion attempts as a default state [2]. This explains why health intervention messages, which aim to produce a beneficial behavior (e.g., anti-binge drinking), sometimes produce the exact opposite behavior (e.g., excessive drinking). Ample studies have demonstrated that health messages, particularly those targeting young adults, can elicit the negative consequences of psychological reactance by focusing—either intentionally or unintentionally—on the message features that intensify reactance arousal (e.g., controlling language, negative or loss-message framing, and using graphic images) [3,4,5,6,7,8,9,10,11,12].
Along a similar line, several researchers have attempted to measure this concept of reactance [13,14,15,16], but few of these “one-off assessments” have been subsequently validated or applied in the health intervention field [17] (p. 289). This current study aims to propose and validate a comprehensive scale measuring psychological reactance toward health promotion messages by applying the scale to an anti-binge drinking intervention targeting college students.
Binge drinking, defined as the episodic or irregular consumption of excessive amounts of alcohol [18,19], is associated with dire consequences such as academic problems, sexual assault, drinking and driving, interpersonal conflicts, other substance abuses, and serious physical and health complications [20,21]. Many anti-binge drinking messages emphasize the negative results of binge drinking and persuade young adults to change their behaviors; however, this simultaneously carries the risk of aversive reactions such as message rejections, message derogation, and boomerang effects [9,22]. Considering that young audiences are characterized by an increasing desire to pursue autonomy and thus are prone to resist freedom-threatening messages [15], the assessment of psychological reactance is crucial in developing sustainable interventions for this population.
The present study situates itself with Singaporean young adults. According to a 2016/2017 National Population Health Survey conducted by the Health Ministry and Health Promotion Board in Singapore, the rate of binge drinking has been on the rise in Singapore [23]. The 2016 Singapore Mental Health Study estimated that 13.7% of Singaporean residents binge drink, and youths aged 18–34 years were twice as likely to binge drink compared to their older counterparts, according to a 2010 Institute for Mental Health study [24]. Young adults in Singapore are considered high-risk groups for binge drinking, which makes them an applicable target audience for this study’s purpose.
1.2. Psychological Reactance: Antecedents, Outcomes, and Measures
The four fundamental elements of PRT are freedom, threat to freedom, reactance, and the restoration of freedom [25]. Psychological reactance occurs in a two-step process, wherein individuals must first perceive that their behavioral freedom is threatened. Once they perceive that their freedom is threatened, they will sequentially enter two states: (1) the reactance state, in which they are psychologically aroused, and (2) the reactance restoration state, in which they exhibit aversive cognitive or affective responses (e.g., message derogation) and behavioral outcomes (e.g., boomerang effects such as engaging in restricted or similar behaviors).
Since Brehm [2] originally proposed psychological reactance as an immeasurable state of motivation, early PRT research assessed the outcomes of reactance rather than directly measuring reactance. However, over the past 15 years, several studies in the health communication domain have attempted to directly measure state reactance using various self-report scales [17]. While previous scales measuring state reactance are limited to one-off measurements [13,14,15,16], health intervention research has widely used the self-report measure suggested by Dillard and Shen [26].
Dillard and Shen [26] conceptualized state reactance as consisting of negative cognition and affect, which are intertwined with each other. Based on this intertwined model, studies [10,11,22,27] have assessed state reactance by measuring (1) recipients’ affect on a four-item anger scale (“annoyed; aggravated; irritated; angry”), and (2) negative cognitions using a thought-listing procedure. In the thought-listing procedure, recipients are asked to list their thoughts, and independent judges code their responses as positive, negative, or neutral; the total number of negative thoughts represents the negative cognition score.
Although many researchers have consistently used Dillard and Shen’s measure [26], it presents challenges that are common for measurements involving cognitive tasks, and its applicability in various health intervention contexts remains unresolved. First, the thought-listing protocols must be scored by independent judges, so this raises reliability concerns [28]; scoring the listed thoughts involves the judges’ subjective evaluation on the units, dimensions, and relevance of thoughts. Second, individuals may be unwilling or unable to record all their thoughts accurately [29]. Furthermore, individuals may report fewer thoughts with this method than with other methods [30], such as comprehensive questionnaires. Third, analyzing listed thoughts can also be arduous and time-consuming, which makes replicating this method in various settings difficult for researchers and practitioners [29]. Fourth, measuring reactance as an inseparable amalgam of anger and negative cognition requires a specific analysis method (e.g., Structural Equation Modeling), which may not be applicable for pilot studies or those with a small sample size. Furthermore, it requires further conceptual refinement because several researchers [6,7,31] have actually found support that the cognitive and affective components may represent separate factors.
Consequently, some researchers recently proposed a multiple-item scale to measure state reactance in the business context [32] and on cigarette-package warnings [33]. These studies are meaningful because they called for developing a more comprehensive and practical scale of state reactance. However, both scales have concerns about construct validity, because their items appear to measure antecedents of reactance (e.g., threat to freedom) rather than reactance per se [17]. Thus, based on the extensive literature on psychological reactance against health promotion efforts, this study aims to develop and validate a comprehensive scale with which to measure state reactance. The present study differentiates between and separates the antecedents (e.g., perceived threat to freedom) or consequences (e.g., reactance restoration) of reactance and state reactance in the measurement to improve the construct validity of measuring the state of psychological reactance. With this new scale, which offers convenient utility without the thought-listing process, we expect that psychological reactance can be better assessed and managed in various health intervention contexts.
2. Materials and Methods
2.1. Pre-Study
As an initial stage of developing a pool of psychological reactance scale items, a traditional thought-listing technique [29] was applied to collect broad negative responses to anti-binge drinking messages. In total, 156 undergraduate students (age: M = 21.2 years, SD = 1.30; gender: female = 78%; male = 22%) at a large Singaporean university participated in this open-ended survey. The participants were asked to read two anti-binge drinking campaign brochures and freely write down their thoughts while reading the brochures. The anti-binge drinking brochures from Health Promotion Board (HPB) Singapore contained information for college students about the risk of binge-drinking and tips on how to stay in control. Participants were asked to list their thoughts in several, separate boxes provided alongside the brochures, and they were rewarded SGD 5 for their participation. Two trained coders independently reviewed the collected thoughts to sort out negative responses from positive and neutral ones. Among the collections, the wordings of frequent responses were polished and converted into a full-sentence form for potential reactance items.
2.2. Survey Procedure and Sample
In total, 203 Singaporean undergraduate students (excluding those who participated in a pre-study) participated in this survey for a monetary reward of SGD 5 (The participant-item ratio was 7.5:1, meeting the minimum standard of 5:1 recommended by Carpenter [34]). The study took place at the university’s computer lab, and participants were first asked to read a written consent form briefly describing the purpose and nature (e.g., length, voluntary participation, confidentiality, potential risks and benefits of participation) of the study. After the participants signed the consent form, they were asked to read the same set of anti-binge drinking brochures from HPB Singapore in a print version and complete the questionnaire online. After completing the questionnaires, the participants were debriefed, thanked, and dismissed. Removing two cases with a missing value in the responses, a set of 201 responses was analyzed. The proportion of females (68.2%) was higher than that of males (31.3%), and their ages were in the range of 19–35 years, with an average of 22.32 years (SD = 2.19). The majority of respondents were Chinese (82.1%), followed by Indians (8.7%) and Malays (4.1%).
2.3. Instrument
The survey instrument included two major sections: (a) measures of potential reactance items after the exposure to anti-binge drinking brochures from HPB Singapore, and (b) measures of relevant concepts. First, an initial pool of psychological reactance measures was created based on the pre-study results, and from consulting the wording of questionnaires from previous PRT studies and relevant literature. Because many responses obtained in pre-study were incomplete or included multiple meanings, the statements were prepared in a more objective and clearer language for the main study. For instance, “yet another attempt by HPB—It has a lot of campaigns these days” and “feels like another attempt to manipulate young people” were reworded to the following items in the final version of the potential reactance scale: “I am bored by yet another campaign focusing on health topics (JA1).” “I feel numbed by the many attempts to educate the public about healthy living (JA2).” and “I feel jaded by yet another health campaign (JA3).” The candidate items were carefully reviewed to minimize any overlap with other relevant measures, such as those for the perceived threat to freedom, reactance trait, and boomerang effects (i.e., reactance restoration). Particularly, the current study aimed to differentiate psychological reactance from its antecedents (e.g., perceived threat to freedom) and consequences (e.g., reactance restoration); therefore, statements related to those were excluded. For example, statements relevant to perceived threat to freedom (“I am aware of my health, there is no need for someone to tell me what to do.” “There is no need for HPB to worry about my health. I am in control.” “I do not need the brochure to tell me what to do.”) and reactance restoration (“It made me think about drinking over the weekends. It created a sudden urge.” “I’m still drinking anyway.” “I’ve never tried drinking, now it is tempting.” “Drinking alcohol this weekend.”) were removed from the scale. In addition, following previous studies [26,35], four items were added to measure the level of anger (“when I was reading the anti-binge drinking brochures from HPB, I felt: annoyed; aggravated; irritated; or angry.’”). As a result, a batch of 27 items (4 items from the previous anger measure and the remaining 23 items from the pre-study) was used in the survey.
In the second part, some prior scales of relevant concepts were employed—namely, cynicism (7 items), issue involvement (3 items), trait reactance (11 items), perceived threat to freedom (4 items), reactance restoration (9 items), cognitive elaboration (3 items), and behavioral intention (3 items)—to examine the discriminant, convergent, and predictive validity. They were measured on 7-point scales, except for the behavioral intention measures, which were measured on a 0–100 scale (see the Appendix A for details). Concluding the survey, respondents reported their age, gender, race, and academic status.
3. Results
3.1. Structure Identification
With the 27 items, exploratory factor analysis (EFA) using principal axis factoring was performed with an oblique rotation method (i.e., promax). The number of factors was determined based on Kaiser’s rule (eigenvalue ≥ 1). A cut-off criterion of 0.32 was used for factor-loadings, as recommended by Carpenter [34]. As a result, the 27 items were successfully loaded onto eight factors with no cross-loaded items or two-item factors. As Table 1 shows, the eight factors explained 78.70% of the total variance, and they were named (1) anger (4 items), (2) exaggeration (4 items), (3) design derogation (3 items), (4) authoritative tone (4 items), (5) ineffectiveness (3 items), (6) know-it-all attitude (3 items), (7) jadedness (3 items), and (8) source motive (3 items). Table 2 indicates that all identified factors, except for the anger factor, showed substantial correlations with other factors (scores ranged from 0.17 to 0.56). The anger factor only correlated with authoritative tone (r = 0.344) and source motive factors (r = 0.158), which implies that negative emotions such as anger are mostly induced by negative responses to authoritative message tone and the message sender’s (or source’s) insincere motivation.
Additionally, second-order confirmatory factor analysis (CFA) was performed using the AMOS program to confirm the factor structure. The overall fit of the scale was acceptable (χ2 (316) = 684.240, p < 0.001; χ2/df = 2.165; GFI = 0.803, AGFI = 0.765; NFI = 0.837; CFI = 0.905; RMSEA = 0.076). As shown in Table 3 and Figure 1, standardized estimates of items loaded on each factor were satisfactory at 0.58–0.99, and all were statistically significant. All identified factors, other than anger (0.11), showed sufficient and significant standardized path estimates (exaggeration = 0.71, design derogation = 0.51, authoritative tone = 0.48, ineffectiveness = 0.71, know-it-all attitude = 0.60, jadedness = 0.72, and source motive = 0.48).
3.2. Item Analysis
As part of the item analysis, the 27 items were analyzed using “correlated item-total correlation” and “squared multiple correlation” (see Table 1). The former refers to the correlation between an item and the sum of the remaining items (should be ≥0.20), and the latter is an indicator of the association between an item and the remaining set of items [36]. Thus, if the squared multiple correlation approaches a value of 1, it is perfectly predicted by the remaining items, which means that the item has no unique contribution to the scale [36]. The scores of the correlated item-total correlation were in the range of 0.233–0.644 (except for the “angry” item at 0.073), and those of the squared multiple correlation were between 0.412 and 0.893.
3.3. Reliability of the Scale
The scale’s reliability was investigated using Cronbach’s α for internal consistency. Overall, the alpha scores of the eight factors were 0.915, 0.882, 0.939, 0.845, 0.907, 0.794, 0.876, and 0.669, respectively. Additionally, Cronbach’s α for the whole scale was 0.896, indicating satisfactory internal consistency (see Table 1). Although a slight increase (to 0.899 or 0.897) could be made by deleting one of the two items (i.e., “angry” and “I am already aware of the information given in the brochure.”), they were retained because the increase was not substantial, and their removal would produce a two-item factor, which is not recommended in scale development [34,36].
3.4. Validity of the Scale
The 27-item scale was correlated with other measures to test the discriminant validity, convergent validity, and predictive validity, as shown in Table 4. First, discriminant validity denotes whether two dissimilar constructs are distinct from each other. Cynicism showed significant correlations with two factors (authoritative tone: r = 0.163, p < 0.05; jadedness: r = 0.167, p < 0.05), and issue involvement was only significantly correlated with one factor (design derogation: r = 0.172, p < 0.05), supporting that psychological reactance is generally distinct from cynicism or issue involvement.
Second, convergent validity examines whether two similar constructs correspond with each other. In the current study, the eight identified factors were correlated with prior reactance scales and relevant concepts, such as the (a) reactance trait [37,38], (b) perceived threat to freedom [26], and (c) prior scale of psychological reactance toward health messages (e.g., [39]). These measures showed significant correlations with at least three factors of the proposed psychological reactance scale. In particular, reactance trait, which is considered a personal trait rather than situational state, appeared to be associated with authoritative tone (r = 0.246, p < 0.001), know-it-all attitude (r = 0.155, p < 0.05), jadedness (r = 0.250, p < 0.001), and source motive (r = 0.205, p < 0.01) factors. The perceived threat to freedom, as an antecedent of psychological reactance, showed strong significant correlations with the authoritative tone (r = 0.672, p < 0.001), anger (r = 0.382, p < 0.001), and exaggeration (r = 0.294, p < 0.001) factors. Prior scale of psychological reactance was correlated with five of the eight factors, but not with design derogation, ineffectiveness, and know-it-all attitude factors.
Third, predictive validity was tested by checking correlations with related behaviors and other consequences, such as reactance restoration (i.e., direct, vicarious, and related motivation), cognitive elaboration, and behavioral intention. The scale was most successful at predicting the level of cognitive elaboration and behavioral intention; six out of the eight factors showed significant negative relationships with cognitive elaboration and behavioral intention, meaning that greater psychological reactance suppresses cognitive elaborations and behavioral intentions to avoid binge-drinking. Moreover, most factors were significantly associated with three reactance restorations, confirming that psychological reactance to a message engenders boomerang effects. Among the eight factors, two (design derogation and know-it-all attitude) were consistently unrelated to the three dimensions of reactance restoration.
4. Discussion
Since its introduction in the mid-1960s, the PRT has provided health researchers and practitioners with a ground-breaking perspective as to why persuasion attempts will more likely fail than succeed, no matter the health message’s beneficial nature. The theory has helped them become mindful of how health promotion attempts could create less reactance among recipients—shedding light on the importance of language tone, message frame, and delivery modality, among others, to improve the potential of persuasion [8,13,22]. However, a common issue for reactance studies is related to the fact that administrating cognitive procedures (e.g., thought listing and subsequent coding of listed thoughts) is time- and cost-consuming and has volatile reliability, mainly due to its dependence on coder competence. Furthermore, recent studies have suggested that, given that reactance may not be an inseparable amalgamation of anger and negative cognitions, a multiple-item scale would provide great benefits for clarifying the different dimensions of psychological reactance and improving practical applications in a given context [6,7]. Accordingly, this current study aimed to develop and validate a self-report scale to measure the multiple constructs of psychological reactance to health promotion messages.
The results of the current study indicate that the final scale (27 items in eight factors) demonstrated high internal consistency and validity. The eight factors included a wide spectrum of psychological reactance from negative emotions (e.g., anger) to negative responses to message argument (e.g., exaggeration) and authoritative tone, design, effectiveness, and sources. Furthermore, an individual’s jadedness and know-it-all attitude also appeared to factor into psychological reactance, probably due to the widely conducted health campaigns in Singapore.
In summary, a major contribution of this study is the identification of the multifaceted nature of psychological reactance as it relates to its cognitive dimensions. This current scale improves the construct validity of the prior scales (e.g., [32,33]), which measure antecedents of reactance (e.g., threat to freedom) as a part of reactance per se. Also, previous scales have largely amalgamated the wide spectrum of cognitive dimensions as a single dimension by mostly focusing on negative responses to authoritative tones, controlling language, or exaggeration [17]. However, the present study extended the cognitive dimensions to other realms, such as design derogation, ineffectiveness, source motive, know-it-all attitude, and jadedness. In this regard, individuals may consider themselves “a superior message evaluator” who refutes health messages by criticizing design components, arguing for the ineffectiveness and source motive, and displaying jaded and know-it-all attitudes. Presumably, when faced with persuasive attempts, people may be triggered to exercise autonomy by self-persuading themselves that they are more knowledgeable and experienced regarding the subject matter. Grounded on these findings, future studies are expected to further investigate the underlying psychological mechanisms surrounding psychological reactance.
Besides, the contextualized nature of this study allows scholars and health practitioners to understand specified reactance responses. In other words, the scale developed in this study can identify how and which dimensions of psychological reactance lead to negative consequences in a given health context. For example, items such as anger, exaggeration, and source motive may be triggered for certain health issues (e.g., vaccination and social distancing to arrest the spread of COVID-19), whereas other health issues may trigger more nuanced dimensions of reactance such as ineffectiveness, jadedness, or a know-it-all attitude (e.g., binge-drinking). The present scale allows us to differentiate the varying magnitudes among the constructs that make up the psychological reactance depending on the relevant health issue.
Another interesting finding is that the “anger” factor was significantly correlated with only one (authoritative tone) out of the seven factors. This may be in line with Rains’ meta-analytic review of reactance research that found a weak correlation between anger and negative cognitions [40]. In this sense, this study offers added empirical evidence that the cognitive and affective components of psychological reactance may work independently from one another. Homing in on this finding, future studies should further examine what moderators may influence the magnitude of different dimensions of psychological reactance in health communication. Previous studies have shown how moderators play a role in the cognitive versus affective dimensions of psychological reactance. For example, Kim and Shin [6] found that the fear appeal in sexually transmitted disease (STD) prevention messages increased the level of anger arousal, whereas the message source type increased the level of negative cognitions. LaVoie et al. [7] found that different moderators, such as graphic images and individuals’ traits of reactance, influenced the magnitudes of the two indicators of psychological reactance differently. To this end, an examination of how different dimensions of psychological reactance are triggered by different message factors in various health contexts would be a meaningful extension to this research area.
We empirically tested discriminant, convergent, and predictive validity by examining the relationships with a wide range of relevant concepts. Specifically, an individual’s broad cynicism and issue involvement were not typically correlated with the identified factors, which supports satisfactory discriminant validity. Second, to see the convergent validity, the commonly used reactance trait scale [38] and the perceived threat to freedom scale [26] were correlated with the identified factors and showed substantial correlations. Finally, to examine the predictive validity, the items were correlated with three dimensions of reactance restoration (i.e., direct, vicarious, and related motivation), cognitive elaboration, and behavioral intention. The vast majority of the factors (5 or 6 out of 8 factors) were statistically associated with the outcome variables, confirming the sufficient predictive power of the scale proposed in the current study. In particular, the greater reactance that individuals perceived, the less likely they would be motivated to enact health-promoting behaviors. Moreover, corresponding to greater psychological reactance, reactance restoration increased and cognitive efforts decreased. Once again, the results suggest that reducing one’s psychological reactance is critical to improving a health message’s intended persuasive effects and minimizing the boomerang effects, which enables sustainable health interventions.
Overall, the final 27 items showed a satisfactory level of reliability and validity, but there was a little concern found over the “angry” item in the item analysis; the correlated item-total correlation score was 0.073, which did not meet the conventional 0.20 criterion. The four-item anger measures (angry, annoyed, irritated, and aggravated) have been widely accepted and applied in PRT studies. Although the current study retained the “angry” item, corresponding to previous studies [26,35], it uncovered the potential that the “angry” state was not particularly induced by health promotion messages (M = 1.73, SD = 1.10), compared to annoyed (M = 2.00, SD = 1.30), irritated (M = 1.97, SD = 1.29), and aggravated (M = 1.81, SD = 1.16). However, because the result might be due to the uniqueness of the topic (anti-binge drinking) or the population (Singaporean college students), future studies should revisit this topic.
5. Conclusions
Three general conclusions may be drawn from this study. First, the present study is a direct response to Rosenberg and Siegel’s call that scholars need to continue developing valid reactance measures that can examine the intertwined nature of psychological reactance [17]. Through our scale development process, the present study identified additional cognitive dimensions (e.g., design derogation, ineffectiveness, source motive, know-it-all attitude, and jadedness) that contribute to reactance arousal. With these new constructs in mind, further reliability and validity studies are necessary for the scale, and its utility and theoretical applicability should also be tested in various health promotion situations.
Second, it was found that the “anger” factor was significantly correlated exclusively with “authoritative tone.” This finding of the study offers additional empirical evidence that the cognitive and affective components of psychological reactance may work independently from one another and that the relationship between anger and negative cognition may be less substantial than historically thought [41]. It is crucial that future studies examine this relationship further to dissect whether the anger-negative cognition path is context-specific or influenced by additional moderators [6,7].
Lastly, we recognize that this study involved undergraduate students in Singapore, therefore limitations exist in its generalizability. To this end, future research should consider retesting this measurement using both different demographics (e.g., age, education level, health-awareness, etc.) and different health contexts to not only ensure generalizability of the scale, but to also contribute to the test–retest process of scale development to further enhance the robustness of the measurements’ validity and reliability.
Author Contributions
Conceptualization, H.J.K. and H.L.; methodology, H.J.K. and H.L.; software, H.H.; validation, H.J.K., H.L., and H.H.; formal analysis, H.H.; investigation, H.J.K. and H.L.; resources, H.J.K.; data curation, H.H.; writing—original draft preparation, H.J.K., H.L., and H.H.; writing—review and editing, H.J.K., H.L., and H.H.; visualization, H.H.; supervision, H.J.K.; project administration, H.J.K. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Conflicts of Interest
The authors declare no conflict of interest.
Appendix A
Table A1.
Measures of relevant concepts.
| Concepts | Measures | Sources | Cronbach’s α |
|---|---|---|---|
| Cynicism | “People will tell a lie if they can gain by it.” “People claim to have ethical standards regarding honesty and morality, but few stick to them when money is at stake.” “People pretend to care more about one another than they really do.” “It is pathetic to see an unselfish person in today’s world because so many people take advantage of him or her.” “Most people are just out for themselves.” “Most people inwardly dislike putting themselves out to help other people.” “Most people are not really honest by nature.” | [41] | 0.784 |
| Issue involvement | “To me, the issue of binge-drinking is …” irrelevant/relevant; uninvolving/involving; unimportant/important. | [42] | 0.892 |
| Trait reactance | “Regulations trigger a sense of resistance in me.” “I find contradicting others stimulating.” “When something is prohibited, I usually think, ‘That’s exactly what I am going to do.’” “I consider advice from others to be an intrusion.” “I become frustrated when I am unable to make free and independent decisions.” “It irritates me when someone points out things which are obvious to me.” “I become angry when my freedom of choice is restricted.” “Advice and recommendations usually induce me to do just the opposite.” “I resist the attempts of others to influence me.” “It makes me angry when another person is held up as a role model for me to follow.” “When someone forces me to do something, I feel like doing the opposite.” | [38] | 0.826 |
| Perceived threat to freedom | “The message threatened their freedom to choose.” “The message tries to manipulate them.” “The message tries to make a decision for them.” “The message tries to pressure them.” | [26] | 0.900 |
| Prior reactance scale | “The message triggered a sense of resistance in me.” “I feel the message to be an intrusion.” “I feel like resisting the attempts of the message to influence me.” “I found contradicting the message stimulating.” “While reading the message against binge drinking, I feel like doing the opposite.” “While reading the message prohibiting binge drinking, I felt like ‘That’s exactly what I am going to do.’” “The message against binge drinking induces me to do just the opposite.” | [39] | 0.897 |
| Reactance restoration | “Right now, I am … to get drunk.” (direct) “Right now, I am … to be around others who are drunk.” (vicarious) “Right now, I am … to do something totally rebellious.”(related) unmotivated/motivated; not encouraged/encouraged; not inspired/inspired | [35,43,44] | Direct: 0.982 Related: 0.985 Vicarious: 0.988 |
| Cognitive elaboration | “The amount of attention I paid to the anti-binge drinking brochure was…” very little/very much. “The extent to which I thought about the content of the anti-binge drinking brochure was …” very little/very much. “The extent to which I concentrated on its contents was …” very little/very much. | [45,46,47] | 0.918 |
| Behavioral intention | “The likelihood of me limiting my alcohol consumption within the next week is …” in 0–100. “The likelihood of me limiting my alcohol consumption within the next month is …” in 0–100. “The likelihood of me limiting my alcohol consumption within the next year is …” in 0–100. | [26] | 0.950 |
References
- Brehm, J.W. Psychological reactance: Theory and applications. ACR N. Am. Adv. 1989, 16, 72–75. [Google Scholar]
- Brehm, J.W. A theory of psychological reactance. In Organisation Change: A Comprehensive Reader; Burke, W.W., Lake, D.G., Paine, J.W., Eds.; Jossey-Bass: San Francisco, CA, USA, 1966; pp. 377–390. [Google Scholar]
- Bilandzic, H.; Busselle, R. Narrative persuasion. In the SAGE Handbook of Persuasion: Developments in Theory and Practice; Dillard, J.P., Shen, L., Eds.; Sage Publications, Inc.: Thousand Oaks, CA, USA, 2013; pp. 200–219. [Google Scholar]
- Campbell, R.G.; Babrow, A.S. The role of empathy in responses to persuasive risk communication: Overcoming resistance to HIV prevention messages. Health Commun. 2004, 16, 159–182. [Google Scholar] [CrossRef] [PubMed]
- Crano, W.; Alvaro, E.; Tan, C.; Siegel, J. Social mediation of persuasive media in adolescent substance prevention. Psychol. Addict. Behav. 2017, 31, 479–487. [Google Scholar] [CrossRef] [PubMed]
- Kim, H.; Shin, W. The effects of message source and fear appeal on young adults’ response to Sexually Transmitted Disease (STD) messages in Singapore. Asian J. Commun. 2018, 28, 185–204. [Google Scholar] [CrossRef]
- LaVoie, N.R.; Quick, B.L.; Riles, J.M.; Lambert, N.J. Are graphic cigarette warning labels an effective message strategy? A test of psychological reactance theory and source appraisal. Commun. Res. 2015, 44, 416–436. [Google Scholar] [CrossRef]
- Lee, H.; Cameron, G.T. Utilizing audiovisual and gain-framed messages to attenuate psychological reactance toward weight management health messages. Health Commun. 2017, 32, 72–81. [Google Scholar] [CrossRef]
- Lee, M.J.; Chen, Y.C. Underage drinkers’ responses to negative-restrictive versus proactive-nonrestrictive slogans in humorous anti-alcohol abuse messages: Are humorous responsible drinking campaign messages effective? J. Health Commun. 2013, 18, 354–368. [Google Scholar] [CrossRef]
- Quick, B.L.; Considine, J.R. Examining the use of forceful language when designing exercise persuasive messages for adults: A test of conceptualising reactance arousal as a two-step process. Health Commun. 2008, 23, 483–491. [Google Scholar] [CrossRef]
- Rains, S.A.; Turner, M. Psychological reactance and persuasive health communication: A test and extension of the intertwined model. Hum. Commun. Res. 2007, 33, 241–269. [Google Scholar] [CrossRef]
- Richards, A.S.; Banas, J.A. Inoculating against reactance to persuasive health messages. Health Commun. 2015, 30, 451–460. [Google Scholar] [CrossRef]
- Grandpre, J.; Alvaro, E.M.; Burgoon, M.; Miller, C.H.; Hall, J.R. Adolescent reactance and anti-smoking campaigns: A theoretical approach. Health Commun. 2003, 15, 349–366. [Google Scholar] [CrossRef] [PubMed]
- Lindsey, L.L.M. Anticipated guilt as behavioral motivation: An examination of appeals to help unknown others through bone marrow donation. Hum. Commun. Res. 2005, 31, 453–481. [Google Scholar] [CrossRef]
- Miller, C.H.; Lane, L.T.; Deatrick, L.M.; Young, A.M.; Potts, K.A. Psychological reactance and promotional health messages: The effects of controlling language, lexical concreteness, and the restoration of freedom. Hum. Commun. Res. 2007, 33, 219–240. [Google Scholar] [CrossRef]
- Norman, P.; Wrona-Clarke, A. Combining self-affirmation and implementation intentions to reduce heavy episodic drinking in university students. Psychol. Addict. Behav. 2016, 30, 434–441. [Google Scholar] [CrossRef] [Green Version]
- Rosenberg, B.D.; Siegel, J.T. A 50-year review of psychological reactance theory: Do not read this article. Motiv. Sci. 2018, 4, 281–300. [Google Scholar] [CrossRef] [Green Version]
- Cornelis, E.; Cauberghe, V.; De Pelsmacker, P. The inoculating effect of message sidedness on adolescents’ binge drinking intentions: The moderating role of issue involvement. J. Drug Issues 2014, 44, 254–268. [Google Scholar] [CrossRef]
- Dufur, M.; Parcel, T.; McKune, B. Capital and context: Using social capital at home and at school to predict child social adjustment. J. Health Soc. Behav. 2008, 49, 146–161. [Google Scholar] [CrossRef]
- Braitman, A.; Lau-Barraco, C. Descriptive norms but not harm reduction strategies as a mediator of personalized boosters after a computerized college drinking intervention. Alcoholism 2020, 44, 284–296. [Google Scholar] [CrossRef]
- Peterson, J. Effects of social capital on the culture of college drinking. J. Alcohol Drug Educ. 2019, 63, 50–69. [Google Scholar]
- Quick, B.L.; Bates, B.R. The use of gain- or loss-frame messages and efficacy appeals to dissuade excessive alcohol consumption among college students: A test of Psychological Reactance Theory. J. Health Commun. 2010, 15, 603–628. [Google Scholar] [CrossRef]
- Tay, T.F. Intake low here but binge drinking on the rise. Strait Times. 2018. Available online: https://www.straitstimes.com/singapore/intake-low-here-but-binge-drinking-on-the-rise (accessed on 29 June 2020).
- Lee, Y.; Wang, P.; Abdin, E.; Chang, S.; Shafie, S.; Sambasivam, R.; Tan, K.; Tan, C.; Heng, D.; Vaingankar, J.; et al. Prevalence of binge drinking and its association with mental health conditions and quality of life in Singapore. Addict. Behav. 2020, 100, 106114. [Google Scholar] [CrossRef]
- Brehm, J.W.; Brehm, S.S. Psychological Reactance—A Theory of Freedom and Control; Academic Press: New York, NY, USA, 1981. [Google Scholar]
- Dillard, J.P.; Shen, L. On the nature of reactance and its role in persuasive health communication. Commun. Monogr. 2005, 72, 144–168. [Google Scholar] [CrossRef]
- Quick, B.L.; Stephenson, M.T. Further evidence that psychological reactance can be modeled as a combination of anger and negative cognitions. Commun. Res. 2007, 34, 255–276. [Google Scholar] [CrossRef]
- Freeman, A.; Simon, K.; Beutler, L.; Arkowitz, H. Comprehensive Handbook of Cognitive Therapy; Springer Science & Business Media: New York, NY, USA, 1989. [Google Scholar]
- Cacioppo, J.T.; von Hippel, W.; Ernst, J.M. Mapping cognitive structures and processes through verbal content: The thought-listing technique. J. Consult. Clin. Psychol. 1997, 65, 928–940. [Google Scholar] [CrossRef]
- Blackwell, R.T.; Galassi, J.P.; Galassi, M.D.; Watson, T.E. Are cognitive assessment methods equal? A comparison of think aloud and thought listing. Cogn. Res. 1985, 9, 399–413. [Google Scholar] [CrossRef]
- Moyer-Gusé, E.; Nabi, R.L. Explaining the persuasive effects of narrative in an entertainment television program: Overcoming resistance to persuasion. Hum. Commun. Res. 2010, 36, 26–52. [Google Scholar] [CrossRef]
- Sittenthaler, S.; Traut-Mattausch, E.; Steindl, C.; Jonas, E. Salzburger State Reactance Scale (SSR Scale): Validation of a scale measuring state reactance. Z. Psychol. 2015, 223, 257–266. [Google Scholar] [CrossRef] [Green Version]
- Hall, M.; Sheeran, P.; Noar, S.; Ribisl, K.; Boynton, M.; Brewer, N. A brief measure of reactance to health warnings. J. Behav. Med. 2017, 40, 520–529. [Google Scholar] [CrossRef]
- Carpenter, S. Ten steps in scale development and reporting: A guide for researchers. Commun. Methods Meas. 2018, 12, 25–44. [Google Scholar] [CrossRef]
- Quick, B.L. What is the best measure of psychological reactance? An empirical test of two measures. Health Commun. 2012, 27, 1–9. [Google Scholar] [CrossRef]
- Murphy, K.R.; Davidshofer, C.O. Psychological Testing: Principles and Applications, 6th ed.; Prentice Hall: Upper Saddle River, JN, USA, 2005. [Google Scholar]
- Hong, S.M. Hong’s psychological reactance scale: A further factor analytic validation. Psychol. Rep. 1992, 70, 512–514. [Google Scholar] [CrossRef]
- Hong, S.M.; Faedda, S. Refinement of the Hong psychological reactance Scale. Educ. Psychol. Meas. 1996, 56, 173–182. [Google Scholar] [CrossRef]
- Stok, F.M.; de Vet, E.; de Wit, J.B.; Renner, B.; de Ridder, D.T. Communicating eating-related rules. Suggestions are more effective than restrictions. Appetite 2015, 86, 45–53. [Google Scholar] [CrossRef]
- Rains, S.A. The nature of psychological reactance revisited: A meta-analytic review. Hum. Commun. Res. 2013, 39, 47–73. [Google Scholar] [CrossRef]
- Kanter, D.L.; Mirvis, P.H. The Cynical Americans: Living and Working in an Age of Discontent and Disillusion; Jossey-Bass: San Francisco, CA, USA, 1989. [Google Scholar]
- Zaichkowsky, J.L. The personal involvement inventory: Reduction, revision, and application to advertising. J. Advert. 1994, 23, 59–70. [Google Scholar] [CrossRef]
- Quick, B.L.; Stephenson, M.T. Examining the role of trait reactance and sensation seeking on reactance-inducing messages, reactance, and reactance restoration. Hum. Commun. Res. 2008, 34, 448–476. [Google Scholar] [CrossRef]
- Quick, B.L.; Kim, D.K. Examining reactance and reactance restoration with South Korean Adolescents: A test of psychological reactance within a collectivist culture. Commun. Res. 2009, 36, 765–782. [Google Scholar] [CrossRef]
- Chow, C.W.C.; Luk, C.L. Effects of comparative advertising in high-and low-cognitive elaboration conditions. J. Advert. 2006, 35, 55–67. [Google Scholar] [CrossRef]
- Eveland, W.P., Jr. The cognitive mediation model of learning from the news: Evidence from non-election, off-year election, and presidential election contexts. Commun. Res. 2001, 28, 571–601. [Google Scholar] [CrossRef]
- Eveland, W.P., Jr.; Shah, D.V.; Kwak, N. Assessing causality in the cognitive mediation model: A panel study of motivations, information processing and learning during campaign 2000. Commun. Res. 2003, 30, 359–386. [Google Scholar] [CrossRef]
Figure 1.
Results of confirmatory factor analysis.
Table 1.
Results of exploratory factor analysis and item analysis.
| Factor | Factor Loading | Eigenvalue (% of Variance) | Cronbach’s α | Item Analysis | ||||
|---|---|---|---|---|---|---|---|---|
| Items | Mean | SD | Correlated Item-Total Correlation | Squared Multiple Correlation | Cronbach’s α If Item Deleted * | |||
| Anger | 7.835 (29.017) | 0.915 | ||||||
| Annoyed (AN1) | 0.943 | 2.00 | 1.300 | 0.282 | 0.893 | 0.896 | ||
| Irritated (AN2) | 0.898 | 1.97 | 1.286 | 0.294 | 0.876 | 0.896 | ||
| Aggravated (AN3) | 0.831 | 1.81 | 1.164 | 0.233 | 0.762 | 0.896 | ||
| Angry (AN4) | 0.739 | 1.73 | 1.104 | 0.073 | 0.611 | 0.899 | ||
| Exaggeration | 4.115 (15.241) | 0.882 | ||||||
| The message is overstated (EX1) | 0.910 | 3.30 | 1.493 | 0.644 | 0.802 | 0.888 | ||
| The message is exaggerated (EX2) | 0.900 | 3.21 | 1.402 | 0.596 | 0.799 | 0.890 | ||
| The arguments seem too far-fetched (EX3) | 0.870 | 3.21 | 1.424 | 0.640 | 0.713 | 0.889 | ||
| The arguments of the message seem weak (EX4) | 0.519 | 3.74 | 1.560 | 0.522 | 0.469 | 0.891 | ||
| Design derogation | 2.257 (8.360) | 0.939 | ||||||
| The design of the brochure is boring (DD1) | 0.940 | 4.48 | 1.597 | 0.439 | 0.828 | 0.893 | ||
| I do not like the design of the brochure (DD2) | 0.932 | 4.27 | 1.658 | 0.474 | 0.833 | 0.892 | ||
| The brochure looks unappealing (DD3) | 0.866 | 4.33 | 1.650 | 0.426 | 0.750 | 0.893 | ||
| Authoritative tone | 2.032 (7.527) | 0.845 | ||||||
| The brochure is trying to impose its views on me (AT1) | 0.970 | 3.20 | 1.454 | 0.550 | 0.706 | 0.891 | ||
| The brochure is authoritative (AT2) | 0.794 | 3.04 | 1.542 | 0.450 | 0.588 | 0.893 | ||
| I feel that the brochure is talking down to me (AT3) | 0.730 | 2.95 | 1.415 | 0.527 | 0.603 | 0.891 | ||
| The message tried to keep me from making up my own mind about drinking (AT4) | 0.548 | 3.02 | 1.554 | 0.327 | 0.480 | 0.896 | ||
| Ineffectiveness | 1.489 (5.515) | 0.907 | ||||||
| This brochure is not going to be effective at achieving its goals (IN1) | 0.927 | 4.54 | 1.404 | 0.608 | 0.882 | 0.889 | ||
| I do not think that the brochure is going to work (IN2) | 0.899 | 4.45 | 1.459 | 0.574 | 0.863 | 0.890 | ||
| I do not think that people will follow the advice from the message (IN3) | 0.742 | 4.54 | 1.439 | 0.493 | 0.597 | 0.892 | ||
| Know-it-all attitude | 1.414 (5.238) | 0.794 | ||||||
| The given information seems to be common knowledge (KIA1) | 0.944 | 4.51 | 1.456 | 0.526 | 0.713 | 0.891 | ||
| I am already aware of the information given in the brochure (KIA2) | 0.748 | 4.856 | 1.524 | 0.243 | 0.504 | 0.897 | ||
| The brochure seems to state the obvious (KIA3) | 0.580 | 4.49 | 1.517 | 0.549 | 0.627 | 0.891 | ||
| Jadedness | 1.084 (4.015) | 0.876 | ||||||
| I am bored by yet another campaign focusing on health topics (JA1) | 0.905 | 3.95 | 1.668 | 0.598 | 0.742 | 0.889 | ||
| I feel numbed by the many attempts to educate the public about healthy living (JA2) | 0.897 | 4.10 | 1.652 | 0.625 | 0.755 | 0.889 | ||
| I feel jaded by yet another health campaign (JA3) | 0.595 | 3.90 | 1.480 | 0.593 | 0.613 | 0.890 | ||
| Source motive | 1.022 (3.787) | 0.669 | ||||||
| HPB does not really care about the public (SM1) | 0.855 | 2.60 | 1.114 | 0.315 | 0.447 | 0.895 | ||
| HPB’s motivation may not be sincere (SM2) | 0.648 | 2.83 | 1.361 | 0.316 | 0.412 | 0.895 | ||
| It seems to me that HPB produced this brochure merely for the sake of producing it (SM3) | 0.378 | 3.48 | 1.397 | 0.515 | 0.468 | 0.891 | ||
Note. * Calculated based on Cronbach’s α of the scale in total (0.896).
Table 2.
Factor correlation matrix.
| Factor 1 (Anger) | Factor 2 (Exaggeration) | Factor 3 (Design Derogation) | Factor 4 (Authoritative Tone) | Factor 5 (Ineffectiveness) | Factor 6 (Know-It-All Attitude) | Factor 7 (Jadedness) | Factor 8 (Source Motive) | |
|---|---|---|---|---|---|---|---|---|
| Factor 1 | 1.000 | |||||||
| Factor 2 | 0.096 | 1.000 | ||||||
| Factor 3 | −0.038 | 0.281 | 1.000 | |||||
| Factor 4 | 0.348 | 0.495 | 0.102 | 1.000 | ||||
| Factor 5 | −0.110 | 0.563 | 0.501 | 0.215 | 1.000 | |||
| Factor 6 | −0.057 | 0.356 | 0.322 | 0.252 | 0.437 | 1.000 | ||
| Factor 7 | 0.080 | 0.478 | 0.312 | 0.387 | 0.444 | 0.545 | 1.000 | |
| Factor 8 | 0.156 | 0.304 | 0.168 | 0.330 | 0.182 | 0.309 | 0.347 | 1.000 |
Table 3.
Results of confirmatory factor analysis.
| Unstd. Estimate | Std. Estimate | S.E. | C.R. | p | |||
|---|---|---|---|---|---|---|---|
| Anger | ← | Reactance | 0.179 | 0.106 | 0.140 | 1.281 | 0.200 |
| Exaggeration | ← | Reactance | 1.893 | 0.705 | 0.403 | 4.699 | < 0.001 |
| Design derogation | ← | Reactance | 1.676 | 0.513 | 0.395 | 4.238 | < 0.001 |
| Authoritative tone | ← | Reactance | 1.000 | 0.483 | |||
| Ineffectiveness | ← | Reactance | 1.738 | 0.713 | 0.375 | 4.635 | < 0.001 |
| Know-it-all | ← | Reactance | 1.494 | 0.599 | 0.351 | 4.257 | < 0.001 |
| Jadedness | ← | Reactance | 2.424 | 0.717 | 0.509 | 4.760 | < 0.001 |
| Source motive | ← | Reactance | 0.920 | 0.475 | 0.260 | 3.539 | < 0.001 |
| AN1 | ← | Anger | 1.711 | 0.971 | 0.143 | 11.994 | < 0.001 |
| AN2 | ← | Anger | 1.638 | 0.940 | 0.139 | 11.793 | < 0.001 |
| AN3 | ← | Anger | 1.255 | 0.796 | 0.122 | 10.271 | < 0.001 |
| AN4 | ← | Anger | 1.000 | 0.668 | |||
| EX1 | ← | Exaggeration | 1.156 | 0.911 | 0.073 | 15.878 | < 0.001 |
| EX2 | ← | Exaggeration | 1.070 | 0.897 | 0.069 | 15.586 | < 0.001 |
| EX3 | ← | Exaggeration | 1.000 | 0.826 | |||
| EX4 | ← | Exaggeration | 0.825 | 0.621 | 0.088 | 9.412 | < 0.001 |
| DD1 | ← | Design derogation | 1.053 | 0.942 | 0.054 | 19.678 | < 0.001 |
| DD2 | ← | Design derogation | 1.091 | 0.941 | 0.056 | 19.634 | < 0.001 |
| DD3 | ← | Design derogation | 1.000 | 0.866 | |||
| AT1 | ← | Authoritative tone | 1.480 | 0.923 | 0.167 | 8.877 | < 0.001 |
| AT2 | ← | Authoritative tone | 1.318 | 0.774 | 0.160 | 8.249 | < 0.001 |
| AT3 | ← | Authoritative tone | 1.221 | 0.782 | 0.147 | 8.298 | < 0.001 |
| AT4 | ← | Authoritative tone | 1.000 | 0.583 | |||
| IN1 | ← | Ineffectiveness | 1.271 | 0.966 | 0.090 | 14.150 | < 0.001 |
| IN2 | ← | Ineffectiveness | 1.271 | 0.929 | 0.092 | 13.854 | < 0.001 |
| IN3 | ← | Ineffectiveness | 1.000 | 0.741 | |||
| KIA1 | ← | Know-it-all | 1.323 | 0.992 | 0.123 | 10.789 | < 0.001 |
| KIA2 | ← | Know-it-all | 0.866 | 0.621 | 0.098 | 8.833 | < 0.001 |
| KIA3 | ← | Know-it-all | 1.000 | 0.720 | |||
| JA1 | ← | Jadedness | 1.000 | 0.886 | |||
| JA2 | ← | Jadedness | 1.024 | 0.916 | 0.062 | 16.499 | < 0.001 |
| JA3 | ← | Jadedness | 0.722 | 0.721 | 0.060 | 11.998 | < 0.001 |
| SM1 | ← | Source motive | 0.989 | 0.751 | 0.159 | 6.199 | < 0.001 |
| SM2 | ← | Source motive | 1.017 | 0.633 | 0.166 | 6.121 | < 0.001 |
| SM3 | ← | Source motive | 1.000 | 0.606 |
Table 4.
Discriminant, convergent, and predictive validity (N = 201).
| Discriminant | Convergent Validity | Predictive Validity | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Cynicism | Involvement | Reactance Trait | Threat to Freedom | Prior Reactance Scale | RR Direct | RR Vicarious | RR Related | Elaboration | Behavioral Intention | |
| Factor 1 (Anger) | 0.131 | 0.105 | 0.100 | 0.382 *** | 0.418 *** | 0.216 ** | 0.198 ** | 0.264 *** | 0.027 | −0.131 |
| Factor 2 (Exaggeration) | 0.003 | −0.052 | 0.128 | 0.294 *** | 0.416 *** | 0.340 *** | 0.329 *** | 0.207 ** | −0.206 ** | −0.324 *** |
| Factor 3 (Design derogation) | −0.029 | 0.172 * | 0.092 | −0.024 | −0.028 | 0.099 | 0.111 | 0.030 | −0.205 ** | −0.088 |
| Factor 4 (Authoritative tone) | 0.163 * | 0.038 | 0.246 *** | 0.672 *** | 0.596 *** | 0.285 *** | 0.312 *** | 0.403 *** | −0.124 | −0.259 *** |
| Factor 5 (Ineffectiveness) | −0.061 | 0.016 | 0.099 | 0.123 | 0.136 | 0.209 ** | 0.187 ** | 0.087 | −0.283 *** | −0.155 * |
| Factor 6 (Know-it-all attitude) | 0.106 | −0.012 | 0.155 * | 0.006 | 0.049 | 0.098 | 0.128 | 0.001 | −0.178 * | −0.190 ** |
| Factor 7 (Jadedness) | 0.167 * | −0.071 | 0.250 *** | 0.101 | 0.296 *** | 0.250 *** | 0.196 ** | 0.184 ** | −0.240 *** | −0.321 *** |
| Factor 8 (Source motive) | 0.136 | 0.037 | 0.205 ** | 0.155 * | 0.251 *** | 0.205 ** | 0.186 ** | 0.244 *** | −0.141 * | −0.161 * |
Note. *** p < 0.001, ** p < 0.01, * p < 0.05; RR = Reactance restoration.
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
MDPI and ACS Style
Kim, H.J.; Lee, H.; Hong, H. Scale Development and Validation for Psychological Reactance to Health Promotion Messages. Sustainability 2020, 12, 5816. https://doi.org/10.3390/su12145816
AMA Style
Kim HJ, Lee H, Hong H. Scale Development and Validation for Psychological Reactance to Health Promotion Messages. Sustainability. 2020; 12(14):5816. https://doi.org/10.3390/su12145816
Chicago/Turabian StyleKim, Hyo Jung, Hyunmin Lee, and Hyehyun Hong. 2020. "Scale Development and Validation for Psychological Reactance to Health Promotion Messages" Sustainability 12, no. 14: 5816. https://doi.org/10.3390/su12145816
APA StyleKim, H. J., Lee, H., & Hong, H. (2020). Scale Development and Validation for Psychological Reactance to Health Promotion Messages. Sustainability, 12(14), 5816. https://doi.org/10.3390/su12145816
Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.
