Next Article in Journal
Protective Action of Carica papaya on β-Cells in Streptozotocin-Induced Diabetic Rats
Next Article in Special Issue
Occupational Stress: Preventing Suffering, Enhancing Wellbeing
Previous Article in Journal
Primary Care Physicians’ Beliefs and Practices Regarding E-Cigarette Use by Patients Who Smoke: A Qualitative Assessment
Previous Article in Special Issue
Associations of Extrinsic and Intrinsic Components of Work Stress with Health: A Systematic Review of Evidence on the Effort-Reward Imbalance Model
Open AccessArticle

Predictors of Workplace Bullying and Cyber-Bullying in New Zealand

1
School of Psychology, Massey University, Palmerston North 4442, New Zealand
2
School of Psychology, University of Waikato, Hamilton 3240, New Zealand
3
School of Psychology, University of Auckland, Auckland 1142, New Zealand
4
School of Management, Massey University, Palmerston North 0745, New Zealand
5
School of Management, RMIT University, Melbourne 3001, Australia
6
Griffith Business School, Griffith University, Queensland 4111, Australia
*
Author to whom correspondence should be addressed.
Academic Editor: Cary Cooper
Int. J. Environ. Res. Public Health 2016, 13(5), 448; https://doi.org/10.3390/ijerph13050448
Received: 8 March 2016 / Revised: 3 April 2016 / Accepted: 15 April 2016 / Published: 27 April 2016
(This article belongs to the Special Issue Occupational Stress, Human Health and Wellbeing)

Abstract

Background: The negative effects of in-person workplace bullying (WB) are well established. Less is known about cyber-bullying (CB), in which negative behaviours are mediated by technology. Drawing on the conservation of resources theory, the current research examined how individual and organisational factors were related to WB and CB at two time points three months apart. Methods: Data were collected by means of an online self-report survey. Eight hundred and twenty-six respondents (58% female, 42% male) provided data at both time points. Results: One hundred and twenty-three (15%) of participants had been bullied and 23 (2.8%) of participants had been cyber-bullied within the last six months. Women reported more WB, but not more CB, than men. Worse physical health, higher strain, more destructive leadership, more team conflict and less effective organisational strategies were associated with more WB. Managerial employees experienced more CB than non-managerial employees. Poor physical health, less organisational support and less effective organisational strategies were associated with more CB. Conclusion: Rates of CB were lower than those of WB, and very few participants reported experiencing CB without also experiencing WB. Both forms of bullying were associated with poorer work environments, indicating that, where bullying is occurring, the focus should be on organisational systems and processes.
Keywords: conservation of resources; bullying; cyber-bullying conservation of resources; bullying; cyber-bullying

1. Introduction

Workplace bullying is known to have severe negative effects on targets, witnesses and organisations. Targets and witnesses can experience stress, anxiety and poor physical and mental health [1,2,3,4], and these effects persist over time [5]. Targets risk being excluded from working life because of ill health, stigmatisation, victimisation or reputational damage [6]. Organisations experience poor employee productivity, as perpetrators are using employer resources and spending time on non-work activities [7], and targets and witnesses are likely to work less efficiently, take more time off, feel less committed to the organisation and be more likely to leave [2,8,9]. Despite the severe problems caused by workplace bullying, organisations have difficulty dealing with it. While policies, rules and guidelines are required, they are rarely sufficient: good role models, effective human resources systems [10], committed leadership [11] and a good workplace culture [12] are also essential to combat this complex psychosocial problem.
In the present study, bullying and cyber-bullying were defined as a situation where a person feels they have repeatedly been on the receiving end of negative actions from one or more other people, when it is difficult to defend themselves against these actions [13]. These negative actions could be physical (e.g., shoving) or non-physical (e.g., verbal abuse). A one-off incident is not defined as bullying (adapted from Lutgen-Sandvik et al. [13]). Note that this definition includes both ‘traditional’ face to face bullying (here called workplace bullying (WB)) and negative actions carried out online, known as cyber-bullying (CB).
WB is frequently associated with stressful and poorly-organised working conditions. Job insecurity [14], job stressors such as high workloads, role conflict and role ambiguity [15,16], organisational change [17] and lack of autonomy at work can create stress, uncertainty and opportunities for bullying [18,19,20]. Organisations may also condone and support bullying behaviour if the workplace culture over-emphasizes productivity, performance and competitiveness over wellbeing [17,21,22]. While extensive research has identified factors associated with WB [23], much less is known about the factors associated with CB.
The use of electronic media both within the workplace and outside is growing rapidly, associated with increased interest in work and non-work uses of technology. Building on the extensive work into WB, there is growing interest in bullying behaviour, which is mediated by technology. Cyber-bullying (CB) is defined as “inappropriate, unwanted social exchange behaviours initiated by a perpetrator via online or wireless communication technology and devices” [24]. Forms of CB include “anonymous, fraudulent, aggressive, unwanted messages, spreading rumours, hacking into email accounts, threats, harassment, attacks, unwanted phone calls, malicious, abusive messages” [25]. CB has important elements in common with WB. It is repeated and hurtful, it can be intense; it involves an imbalance of power; and it creates feelings of powerlessness [26]. The key difference between them, from this perspective, is that technology is used for CB. However, it has been argued that CB is distinct from, and potentially more harmful than, other forms of bullying, due to three features that WB does not share: potential for anonymity, publicness and access [26].
The potential anonymity of online interactions affects both perpetrators and targets. Cyber-bullies may feel protected and able to act with fewer constraints and, being remote from their targets, may not recognise or may be more able to ignore a target’s distress [3,27,28,29,30,31]. Targets may not know who is bullying them, leading to feelings of helplessness and invasion of privacy [25], fear [32] and being pursued, “haunted” and “hemmed in” [33], with few opportunities for redress [3]. At the other extreme, cyber-bullying can also be extremely public. A cyber-bully can publish comments, allegations, images and other material to an enormous audience almost instantaneously [3,31]. Audiences can include the target’s family, friends, neighbours and colleagues [30], and reputational harm can be done before a target becomes aware of it [30]. Access to targets is also far broader for the cyber-bully than the traditional bully. Cyber-bullies can potentially reach their targets at any time, in any location and by a wide range of means—email, phone, social networking sites and text messaging—making it difficult for targets to avoid bullies without giving up the technologies which, for many, are essential for work and family communication [31].
Like traditional bullying, cyber-bullying is related to higher stress, less optimism, worse job performance and more dissatisfaction [26,31,33,34,35]. Similar processes are likely to be involved as in traditional bullying: imbalance of power, the stigma associated with being targeted, unwillingness to report, fear of being labelled a “victim” [25,36]. In view of cyber-bullying’s anonymity, publicness and access, it has been proposed to have more severe impacts than traditional bullying, but there is little evidence to address this issue. The prevalence of CB is also relatively unknown. While the anonymity and relative immunity of cyber-bullies could facilitate CB, the prevalence of WB has so far been shown to be higher than CB [27]. Farley [34] found that 46.2% of a sample of trainee doctors had experienced at least one act of CB, while Perreault [37], in a survey of adult Internet users, found that 7% self-identified as having experienced CB. Privitera and Campbell [38] found that 34% of their sample of male union members had been bullied face-to-face, and 10.7% had been cyberbullied; all who experienced CB had also experienced face-to-face WB. There are indications that CB is less prevalent than WB and that it may not be creating ‘new’ targets, but instead is most likely to appear in conjunction with WB [32,39], providing an alternative means of attack, but these issues have yet to be resolved.

The Role of Resources in Preventing Bullying

The conservation of resources model [40] is useful in explaining the impacts of bullying on wellbeing. Resources are “personal characteristics, conditions or energies that are valued by the individual or that serve as a means for attainment of these objects, personal characteristics, conditions or energies” [41]. Resource drain and the inability to replenish resources can lead to stress, poor wellbeing and burnout [42]. Bullying can deplete targets’ resources as they use their time and energy to deal with the problem, which is likely to detract from their focus on work [5]. Resources that can be eroded by exposure to WB include social support, self-efficacy and optimism [43,44]. Loss of resources can lead to a spiral of future losses, inability to regain resources and decreasing wellbeing as employees become increasingly unable to regain resources, find themselves committing more and more resources to dealing with their situation [45]. While bullying can lead to a loss of resources, there is emerging evidence for a reciprocal process: low resources can be related to more bullying when, for example, poor health and high levels of strain make an employee an easier target for bullies [5]. While the body of longitudinal research into the effects of bullying on resources and vice versa is growing, it remains small, and the research exploring these issues in the context of cyber-bullying is even smaller. The aims of the present research were therefore to explore the relationship of personal and organisational factors with WB and CB across two time points. As the perpetrators of WB are frequently of higher organisational status than targets, reflecting the role of power in facilitating bullying [46], role (managerial/non-managerial) was included as a control variable in the analyses. The evidence for gender differences in bullying and cyber-bullying is mixed and few firm conclusions can be drawn; gender was included as a control variable in the analyses [35,47].
While bullying can give rise to a loss of resources, there is emerging evidence of a reciprocal process in which bullying gives rise to distress, which gives rise to further bullying [5,48,49]. Resource loss can lead to poor physical and mental health, which in turn reduce the ability to deal with work and other demands [44]. Employees who are performing poorly or are frequently absent may be bullied as a form of social control in response to organisational norm violation [50]. The outcome is an ongoing “loss spiral” of stress, illness, poor performance and absenteeism, leading to more bullying and further reducing targets’ opportunities to regain resources.
Hypothesis 1a:
Self-reported job performance at Time 1 will be negatively related to (i) WB and (ii) CB at Time 2.
Hypothesis 1b:
Physical health at Time 1 will be negatively related to (i) WB and (ii) CB at Time 2.
Hypothesis 1c:
Strain at Time 1 will be positively related to (i) WB and (ii) CB at Time 2.
Hypothesis 1d:
Hypothesis 1d: Self-reported absenteeism at Time 1 will be positively related to (i) WB and (ii) CB at Time 2.
As well as personal resources, there is substantial evidence that organisational factors act as resources that predict the occurrence of WB [15,51,52,53,54]. Poor leadership is strongly related to higher rates of bullying [55,56]. Destructive leadership undermines and/or sabotages the organization’s goals, tasks and resources, as well as the effectiveness and/or motivation, wellbeing or job satisfaction of employees [57]. Destructive leaders can be autocratic, ineffective, unethical, incompetent, inconsistent or overly political [58]. In contrast, fair and supportive leadership is associated with less bullying [16]. Ethical leadership is the “demonstration of normatively appropriate conduct through personal actions and interpersonal relationships” [59]. Ethical leaders display appropriate personal and interpersonal behaviours, are good role models and provide a fair and just work environment [59]. Such leaders are likely to foster supportive workplace climates and to prevent or effectively address conflict and bullying [56].
Hypothesis 2a:
Ethical leadership at Time 1 will be negatively related to (i) WB and (ii) CB at Time 2.
Hypothesis 2b:
Destructive leadership at Time 1 will be positively related to (i) WB and (ii) CB at Time 2.
Perceived organisational support (POS) is the perception by employees that the organization values their contribution and cares about their wellbeing [60,61]. POS is an organisational resource for employees both because it is valued and because it suggests that other resources will be forthcoming as needed. POS can reduce the likelihood of negative behaviours at work [53], reduce the effects of bullying on targets’ intentions to leave [9] and reduce the effects of negative behaviours on wellbeing and health [62]. Bullying and organisational support can co-exist, but rarely come from the same source. Targets may therefore be able to access some social resources to help address bullying [9].
Hypothesis 3:
POS at Time 1 will be negatively related to (i) WB and (ii) CB at Time 2.
Organisations require systems and strategies to manage WB and, increasingly, CB. Employer responses to reports of bullying have consistently been found to be inadequate and may range from helping the target, to doing nothing, to retaliating against the person reporting the bullying [63,64]. While organisations need to provide resources to address bullying, even potentially effective systems are of little value if employees are unaware of them, do not use them or do not trust them [65]. Employee perceptions that organisational anti-bullying strategies are effective are expected to be related to less WB and CB.
Hypothesis 4:
Perceived effectiveness of organisational strategies at Time 1 will be negatively related to (i) WB and (ii) CB at Time 2.
A positive team climate can provide resources, such as instrumental and emotional support, positive appraisal and feelings of belonging, which facilitate team performance and satisfaction [66]. In contrast, team conflict can arouse strong emotions and give rise to bullying behaviours aimed at achieving goals, emotional release or retaliation [67,68]. Members of a team experiencing conflict will have fewer resources to deal with bullying arising within or outside the team [69].
Hypothesis 5:
Team conflict at Time 1 will be positively related to (i) WB and (ii) CB at Time 2.

2. Method

Data were collected by means of online surveys using Qualtrics, a secure hosting site. To participate, individuals needed to be currently employed within New Zealand. Data on all variables were collected at two time points 3 months apart. In order to test the current hypotheses, predictor variables at Time 1 and criterion variables at Time 2 were the focus of analysis. Three months was selected as a timeframe suitable for exploring cross-time effects [70] and to minimize common method variance by separating the measurement of the predictor and criterion variables [71]. Ethical approval was obtained via a Low Risk Notification to the Massey University Human Ethics Committee (Approval Date 9 July 2014). .

2.1. Participants

All participants resided and worked in New Zealand at both Time 1 and Time 2. Time 1 participants comprised 991 men (40.9%) and 1421 women (58.6%); 12 (5%) did not provide this information). At Time 2, there were 349 men (42%) and 477 women (58%). The mean age at Time 2 was 50 years, and mean tenure was 6.5 years. Seventy-nine percent (1903) self-identified as New Zealand European 1903, 166 (6.8%) as other European, 167 (6.9%) as Maori or Cook Island Maori, 41 (1.7%) as Pasifika, 53 (2.2%) as Chinese, 53 (2.2%) as Indian and 138 (5.7%) identified with other ethnicities. In terms of job role, 293 (12.1%) were senior managers, 348 (14.4%) mid-level managers, 219 (9.0%) first-line supervisors and 1447 (59.7%) were non-managerial employees, and 117 (4.8%) did not provide this information. Role (coded 0 = non-managerial; 1 = managerial) and gender (0 = male, 1 = female) were entered as control variables in the regressions, to test the hypotheses outlined above. Evidence for the relationships between demographic variables and both WB and CB is inconsistent. Younger employees may be more likely to experience CB than older employees due to their more extensive involvement in online activities [72], but there is little evidence for this in the workplace. The relationships between age, WB and CB were explored. In addition, descriptive data on the perpetrators of WB and CB were examined.

2.2. Measures

Workplace bullying (WB) was measured using the 22-item Negative Acts Questionnaire-Revised (NAQ-R) [73]. This provides a list of 22 negative behaviours (e.g., “someone withholding information which affects your performance”), which are formulated in behavioural terms with no reference to the term bullying [74]. Respondents indicate how often they have experienced each behaviour over the previous 6 months (never = 0 to daily = 4). The mean score across negative acts was computed for each person (α = 0.93). In line with Leymann’s [75] criterion for identifying targets of WB, respondents were classified as “bullied” if they had experienced at least two of the negative behaviours weekly or more often over the past 6 months. Respondents who did not meet this criterion were classified as non-bullied.
Cyber-bullying (CB): To date there are few widely-used and well-validated measures of cyber-bullying that are suitable for use in the workplace, and many of those in use are based on the NAQ-R [34,38]. However, the NAQ-R does not include some behaviours that appear particularly relevant to CB (e.g., hacking of personal communications) while others (e.g., being given an excessive workload) do not seem distinctive to CB. Accordingly, we developed our own set of items, after reviewing a range of other measures. CB was measured using a list of 20 negative acts drawn from a range of sources (e.g., “I have received rude, insulting or offensive online communications by people at work”, 1 = never, 5 = daily).
Established criteria for classifying participants as cyber-bullied have yet to emerge. To enable comparisons with WB, the mean score of negative cyber-acts was computed (α = 0.82), then the criterion of having experienced at least two behaviours at least weekly for the last 6 months was applied. For both sets of questions, the negative acts questions were presented without asking participants about their perceptions of themselves as targets of either WB or CB.
Self-identified bullying was examined by providing participants with a definition of bullying (given above) and asking whether they considered themselves to have been bullied over the last 6 months (0 = “no” to 4 = “almost daily”). Those who gave any answer other than “no” were then asked who was responsible for the bullying.
To measure psychological strain, the 12-item version of the General Health Questionnaire (GHQ-12) [76,77] was used. Respondents indicated how often (0 = not at all to 3 = much more than usual) they had experienced each of 12 psychosocial symptoms in the previous 6 months, e.g., “felt constantly under strain”. Items were coded so that a higher score indicated more strain, and the mean score was calculated (α = 0.86).
Self-reported job performance was measured using a single item from Kessler et al. [78], “how would you rate … your own overall job performance on the days you have worked during the past 6 months?” (1 = “the worst performance anyone could have at your job”, 10 = “the performance of a top worker”).
Physical health was measured by 13 items adapted from Spector [79], e.g., “Over the past 6 months, how often have you experienced… An upset stomach or nausea” (Less than once a month or never = 1; several times per day = 5). Higher scores reflected poorer physical health (α = 0.82).
Absenteeism: The single-item measure used by Bentley et al. [80] asked approximately how many days over the previous 6 months respondents had been absent unofficially from work (0 = no days, 3 = 6–10 days).
Ethical leadership was measured with 10 items (e.g., “my boss listens to what employees have to say”, 1 = never to 5 = always; α = 0.89) [59].
Destructive leadership was measured with 20 items, e.g., “my boss has his/her head in the sand”, 1 = never to 5 = always; α = 0.95) [58].
Perceived organisational support (POS) was measured with 7 items from Djurkovic et al. [9], e.g., “my organisation strongly considers my goals and values” (1 = strongly disagree, 7 = strongly agree; α = 0.96).
The measure of team conflict used 8 items adapted from the intragroup conflict scale developed by Jehn [81] (e.g., “there is friction among members in my work unit”, 1 = never, 5 = always; α = 0.95).
To assess the perceived effectiveness of organisational strategies, participants were asked to think about what, if anything, their organisation had done to address bullying, whether face to face or online. Twenty items were derived from various sources and collated for this study (e.g., “a policy or procedure that defines workplace bullying and states that it is unacceptable”). For each item participants answered yes, no, don’t know or prefer not to answer, and for items endorsed “yes” they were then asked to rate the effectiveness of that item (1 = very ineffective, 4 = very effective).

2.3. Data Analysis

The WB and CB variables were non-normally distributed, with a positive skew of 2.32 (WB) and 8.45 (CB) and kurtosis of 6.81 (SE = 0.170) for WB and 105.94 (SE = 0.171) for CB. Log transformation produced a WB variable of reasonable symmetry (skew = 1.38; kurtosis = 1.63 (SE = 0.170), but the CB variable remained non-normally distributed (skew = 4.24; kurtosis 27.52; SE = 0.171). Regression and correlational analysis were run with transformed and non-transformed variables; no differences were observed in the patterns of findings, and so, the analyses using non-transformed variables are reported. Results of Harman’s single-factor test did not show a single factor accounting for the majority of covariance among the variables; in addition, predictor and criterion variables were separated in time, which can help to minimize common method variance.

3. Results

We examined whether participants who only took part in the first wave of data collection (n = 1589) differed in regard to demographic variables from those who completed both waves (n = 826; Table 1). The samples did not differ in regard to their demographic characteristics, except that fewer senior managers/executives completed the survey at both time points than at Time 1 alone (16.2% vs. 6.4%), while more non-managerial employees did so (59.7% vs. 62.7%; chi-square (3) = 46.7, p < 0.001).
Using the criterion of having experienced two or more negative acts at least weekly for at least six months, 123 (15%) of participants had been bullied and 23 (2.8%) of participants had been cyber-bullied. Six hundred and eighty-eight participants (84%) had experienced neither WB nor CB according to these criteria, whereas 20 (2%) had experienced both; 100 (12%) had experienced WB, but not cyber-bullying, and only three (<1%) had been cyber-bullied, but did not meet the criterion for WB.
When asked to self-identify as having been bullied (either CB or WB), 138 (16.79%) reported yes, rarely or now and then, and 14 (1.7%) reported that they had been bullied several times a week or almost daily. Of the 152 who self-identified as having been bullied rarely or more often, 47 (31%) said the perpetrator was a supervisor, employer or manager; 74 (48%) said a peer; 27 (17%) said a subordinate; and 26 (17%) said a client. Participants could endorse more than one response.
Looking at the two control variables of gender and role, compared to men, women reported worse physical health, more strain, destructive leadership, team conflict and WB (Table 1). There were no significant gender differences for self-reported performance, absenteeism, ethical leadership, POS, effectiveness of organisational strategies or CB. Compared to managers, non-managers reported worse physical health, more strain, less POS, less effective organisational strategies and less CB, but there were no differences in performance, absenteeism, leadership, team conflict or WB.
Regression analyses were conducted to investigate the personal and organisational factors anticipated to predict both WB and CB at Times 1 and 2.
There was mixed evidence for the role of personal resources in relation to WB (Table 2).
Job performance and absenteeism were unrelated to WB (Hypotheses 1a(i) and 1d(i)). However, those with worse physical health and higher strain at Time 1 experienced more bullying at Time 2 (Hypotheses 1b(i) and 1c(i)). There was stronger support for the importance of organisational factors in WB. While positive organisational resources, such as ethical leadership and POS, were not related to WB (Hypotheses 2a(i) and 3(i)), destructive leadership, more team conflict and less effective organisational strategies at Time 1 were associated with higher levels of Time 2 WB (Hypotheses 2b(i), 4(i) and 5(i)).
In relation to CB, role was a significant predictor, but not gender: managerial employees experienced more CB behaviours. Poor physical health was associated with more CB (Hypothesis 1b(ii)), but strain (Hypothesis 1c(ii)), performance (Hypothesis 1a(ii)) and absenteeism (Hypothesis 1d(ii)) were not. Ethical and destructive leadership (Hypotheses 2a(ii) and 2b(ii)), and team conflict (Hypothesis 5(ii)) were unrelated to CB. However, higher levels of POS and more effective organisational responses were related to lower levels of CB (Hypothesis 3(ii) and 4(ii), respectively).

4. Discussion

The findings of the present study provided little support for the argument that personal and role resources (seniority, performance) would be related to fewer experiences of bullying. Employees with worse health were likely to experience more bullying, but this was not related to self-reported poorer work performance or absenteeism; those in worse health may be bullied for other reasons, such as perceived vulnerability, and as has been well established, ongoing bullying is highly damaging to both physical and mental health. Rather than identifying personal resources as fostering bullying, the study reinforced the importance of organisational factors.
Destructive leadership and high levels of team conflict were related to WB, but not CB. That the relationship is only with WB and not CB is somewhat surprising. While poor leadership and teamwork are known to be related to increased bullying, it is not clear why they appear to play a smaller role in CB. CB is private and generally invisible to others, making the role of leadership problematic [24], but workplace bullying can also be covert and difficult to manage. Those in managerial positions were more likely than non-managers to experience CB, but not WB. The perceived anonymity of CB may allow employees to bully “upward” without the risks of doing so face to face, but this requires further examination. Workplaces with clear policies and norms of behaviour, in which conflict is managed and where employees trust the effectiveness of systems, should be those in which bullying is less likely to flourish [12,53,56]. The findings support this argument, as perceptions that organisational strategies were effective was related to less WB and CB. Reduced CB was also related to perceptions that the organisation was more supportive; a supportive organisational climate may create fewer reasons for employees to bully each other, or it may provide reassurance that such behaviour will be addressed if it does occur.
Rates of CB were much lower than those of WB, and very few participants reported experiencing CB without also experiencing WB. This is in line with other research, which has found that CB may not create “new” targets, but becomes an additional means by which bullies can reach their targets [27,39]. Whether being targeted in multiple ways creates greater impacts on targets is an issue worth exploring; the small numbers of those experiencing CB in this study did not allow this issue to be examined.

4.1. Limitations

Some limitations of the present study need to be considered. First is the use of self-report data. Employees with worse psychological health may both perceive their work situation in a negative way and report more experiences of bullying, and those who are bullied may see their work environment more negatively as a result [5]. However, research has consistently shown that both non-bullied and bullied participants, including witnesses, report poor work environments in workplaces where bullying takes place [8,53], which casts doubt on the idea that targets of bullying are particularly biased in the ways in which they see their workplaces.
A second concern is with the use of multiple-item lists of negative acts. These measures cannot capture all possible negative behaviours, and no data are available on behaviours that are not included. A related concern is that the measures focus on the frequency of behaviours, but not the severity of the different acts; some negative acts may have more impact than others [82]. This is an issue worth exploring. It is also worth noting that the measurement of CB is still developing; the measure used in the present study is new and requires further assessment.

4.2. Directions for Future Research

Bullying and cyber-bullying deserve further study. In particular, there is a need to examine whether the prevalence and types of negative behaviour change over time, as well as the short-term and long-term impacts within organisations, using longer time frames than in the present study. The interrelationships between WB and CB also require further examination. It is as yet unclear whether CB is more problematic across organisational boundaries or within organisations, given the reach and scope that online technologies provide. In-depth examination of WB and CB in the context of one organization’s policies and practices would also provide opportunities to examine how approaches to managing negative behaviours are enacted and the impacts that they have. Further analysis could use modelling approaches to explore the reciprocal relationships between the antecedents and consequences of WB and CB.

5. Conclusions

Recommendations for addressing WB tend to revolve around the development of organisational solutions, such as policy, procedures, guidelines, systems, training and “cultures of respect” [11,30,65], but bullying remains a problem in many workplaces. Targets are often unwilling to report bullying and rarely feel that doing so would be beneficial; policies are often poorly implemented [11], and HR practitioners are often not prepared to support employees when managers are accused of bullying [83].
Similar problems are likely to arise in relation to CB. While our data show that perceived effectiveness of organisational strategies was related to less bullying, few organisations have established codes of cyber conduct or formal or informal means of controlling online behaviour beyond policies about appropriate and inappropriate Internet use [3,84]. Policies need to be implemented and monitored, and monitoring of employees’ online activities raises concerns about privacy [3]. Network providers can monitor and manage abusive communications, but “censoring” of online activity is often resented by all participants, not just bullies [85].
Some suggested solutions are aimed at targets, such as raising awareness about the need for caution in online interactions [25], building resilience [11] or changing the use of technologies. Targets are unwilling to report that they are experiencing cyber-bullying [33,86] and are understandably reluctant to abandon the technology that is being used to bully them. Less drastic steps, such as ignoring or blocking unwanted contacts, changing passwords or phone numbers, limiting online work activities to working hours and not adding unknown people to contacts lists may have some effect [25]; but, these do not address the root causes, and cyber-bullies may find other ways to reach their targets or choose new targets.
One distinctive feature of CB is that online activity leaves a relatively permanent trail. Even anonymous communications may be traceable, provided there is policy, skill and willingness to engage with the issue [86]. Organisations may be able to identify CB and take appropriate steps. However, this requires effective policy, which defines bullying (face to face or online), clarifies standards of acceptable and unacceptable behaviours, is implemented, communicated and supported, offers support for affected employees and considers privacy issues, confidentiality and the implications of ongoing technological changes [24]. To date, however, there is evidence that organisations are often ineffective in dealing with allegations of both WB and CB [10,33]. Clearly, there is room for improvement in organisational practice to manage this costly workplace problem.

Acknowledgments

This study was supported by funding from Massey University, Auckland University of Technology and the University of Waikato.

Author Contributions

All authors designed the study and developed and pilot-tested the online survey. Dianne Gardner analysed the data and wrote the paper. Michael O’Driscoll, Helena D. Cooper-Thomas, Maree Roche and Tim Bentley edited the text.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Bartlett, J.E.; Bartlett, M.E. Workplace bullying: An integrative literature review. Adv. Dev. Hum. Resour. 2011, 13, 69–84. [Google Scholar]
  2. Nielsen, M.B.; Einarsen, S. Outcomes of exposure to workplace bullying: A meta-analytic review. Work Stress 2012, 26, 309–332. [Google Scholar] [CrossRef]
  3. West, B.; Foster, M.; Levin, A.; Edmison, J.; Robibero, D. Cyberbullying at work: In search of effective guidance. Laws 2014, 3, 598–617. [Google Scholar] [CrossRef]
  4. Van Heugten, K. Resilience as an underexplored outcome of workplace bullying. Qual. Health Res. 2013, 23, 291–301. [Google Scholar] [CrossRef] [PubMed]
  5. Rodriguez-Munoz, A.; Moreno-Jimenez, B.; Sanz-Verge, A.I. Reciprocal relations between workplace bullying, anxiety, and vigor: A two-wave longitudinal study. Anxiety Stress Coping Int. J. 2015, 28, 514–530. [Google Scholar] [CrossRef] [PubMed]
  6. Glambek, M.; Matthiesen, S.; Hetland, J.; Einarsen, S. Workplace bullying as an antecedent to job insecurity and intention to leave: A 6-month prospective study. Hum. Resour. Manag. J. 2014. [Google Scholar] [CrossRef]
  7. Weatherbee, T.G. Counterproductive use of technology at work: Information and communications technologies and cyberdeviancy. Hum. Resour. Manag. Rev. 2010, 20, 35–44. [Google Scholar] [CrossRef]
  8. Cooper-Thomas, H.; Bentley, T.; Catley, B.; Gardner, D.; O’Driscoll, M.; Trenberth, L. The impact of bullying on observers and targets. N. Z. J. Hum. Resour. Manag. 2014, 14, 82–95. [Google Scholar]
  9. Djurkovic, N.; McCormack, D.; Casimir, G. Workplace bullying and intention to leave: The moderating effect of percieved organisational support. Hum. Resour. Manag. J. 2008, 18, 405–422. [Google Scholar] [CrossRef]
  10. Woodrow, C.; Guest, D.E. When good HR gets bad results: Exploring the challenge of HR implementation in the case of workplace bullying. Hum. Resour. Manag. J. 2014, 24, 38–56. [Google Scholar] [CrossRef]
  11. Beirne, M.; Hunter, P. Workplace bullying and the challenge of pre-emptive management: A dual case analysis. Person. Rev. 2013, 42, 595–612. [Google Scholar] [CrossRef]
  12. Dollard, M.; Karasek, R. Building psychosocial safety climate: Evaluation of a socially coordinated par risk management stress prevention study. In Contemporary Occupational Health Psychology: Global Perspectives on Research and Practice; Houdmont, J., Leka, S., Eds.; Wiley-Blackwell: Chichester, UK, 2010; pp. 208–233. [Google Scholar]
  13. Lutgen-Sandvik, P.; Tracy, S.J.; Alberts, J.K. Burned by bullying in the american workplace: Prevalence, perception, degree and impact. J. Manag. Stud. 2007, 44, 837–862. [Google Scholar] [CrossRef]
  14. De Cuyper, N.; Baillien, E.; De Witte, H. Job insecurity, percieved employability and targets’ and perpetrators’ experiences of workplace bullying. Work Stress 2009, 23, 206–224. [Google Scholar] [CrossRef]
  15. Bowling, N.A.; Beehr, T.A. Workplace harassment from the victim’s perspective: A theoretical model and meta-analysis. J. Appl. Psychol. 2006, 91, 998–1012. [Google Scholar] [CrossRef] [PubMed]
  16. Hauge, L.J.; Einarsen, S.; Knardahl, S.; Lau, B.; Notelaers, G.; Skogstad, A. Leadership and role stressors as departmental level predictors of workplace bullying. Int. J. Stress Manag. 2011, 18, 305–323. [Google Scholar] [CrossRef]
  17. Baillien, E.; Neyens, I.; De Witte, H. Organizational correlates of workplace bullying in small- and medium-sized enterprises. Int. Small Bus. J. 2011, 29, 610–625. [Google Scholar] [CrossRef]
  18. Einarsen, S. The nature and causes of bullying at work. Int. J. Manpower 1999, 20, 16–27. [Google Scholar] [CrossRef]
  19. Hodson, R.; Roscigno, V.J.; Lopez, S.H. Chaos and the abuse of power. Work Occup. 2006, 33, 382–416. [Google Scholar] [CrossRef]
  20. Sweeney, P. Organizational chaos and relative powerlessness: Breeding ground for bullies? Acad. Manag. Perspect. 2007, 21, 77–78. [Google Scholar] [CrossRef]
  21. Ferris, G.; Zinko, R.; Brouer, R.L.; Buckley, M.R.; Harvey, M.G. Strategic bullying as a supplementary, balanced perspective on destructive leadership. Leadersh. Quart. 2007, 18, 195–206. [Google Scholar] [CrossRef]
  22. Sperry, L. Mobbing and bullying: The influence of individual, work group, and organizational dynamics on abusive workplace behavior. Consult. Psychol. J. Pract. Res. 2009, 61, 190–201. [Google Scholar] [CrossRef]
  23. Einarsen, S.; Hoel, H.; Zapf, D.; Cooper, C.L. Bullying and Harassment in the Workplace: Developments in Theory, Research, and Practice, 2nd ed.; Taylor & Francis: Boca Raton, FL, USA, 2011. [Google Scholar]
  24. Piotrowski, C. From workplace bullying to cyberbullying: The enigma of e-harassment in modern organizations. Organ. Dev. J. 2012, 30, 43–52. [Google Scholar]
  25. Grigg, D.W. Cyber-aggression: Definition and concept of cyberbullying. Aust. J. Guid. Couns. 2010, 20, 143–156. [Google Scholar] [CrossRef]
  26. Lawrence, F.J. Prevalence and Consequences of Negative Workplace Cyber Communications in the Australian Public Sector; Queensland University of Technology: Brisbane, Australia, 2015. [Google Scholar]
  27. Barlett, C.P. Anonymously hurting others online: The effect of anonymity on cyberbullying frequency. Psychol. Pop. Media Cult. 2015, 4, 70–79. [Google Scholar] [CrossRef]
  28. Coyne, I.; Chesney, T.; Logan, B.; Madden, N. Griefing in a virtual community: An exploratory survey of second life residents. J. Psychol. 2009, 217, 214–221. [Google Scholar] [CrossRef]
  29. McKenna, P. The rise of cyberbullying. New Sci. 2007, 195, 26–27. [Google Scholar]
  30. Sarkar, A. Step-by-step to stopping the cyber-bully: How organizations can help to tackle the menace. Hum. Resour. Manag. Int. Dig. 2015, 23, 31–33. [Google Scholar]
  31. Snyman, R.; Loh, J.M. Cyberbullying at work: The mediating role of optimism between cyberbullying and job outcomes. Comput. Hum. Behav. 2015, 53, 161–168. [Google Scholar]
  32. Ford, D.P. Virtual harassment: Media characteristics’ role in psychological health. J. Manag. Psychol. 2013, 28, 408–428. [Google Scholar]
  33. D’cruz, P.; Noronha, E. Navigating the extended reach: Target experiences of cyberbullying at work. Inf. Organ. 2013, 23, 324–343. [Google Scholar] [CrossRef]
  34. Farley, S.; Coyne, I.; Sprigg, C.; Axtell, C.; Subramanian, G. Exploring the impact of workplace cyberbullying on trainee doctors. Med. Educ. 2015, 49, 436–443. [Google Scholar] [CrossRef] [PubMed][Green Version]
  35. Giumetti, G.W.; McKibben, E.S.; Hatfield, A.L.; Schroeder, A.N.; Kowalski, R.M. Cyber [email protected] work: The new age of interpersonal deviance. Cyberpsychol. Behav. Soc. Netw. 2012, 15, 148–154. [Google Scholar] [CrossRef] [PubMed]
  36. Madan, A.O. Cyber aggression/cyber bullying and the dark triad: Effect on workplace behavior/performance. Int. J. Soc. Manag. Econ. Bus. Eng. 2014, 8, 1725–1730. [Google Scholar]
  37. Perreault, S. Self-Reported Intenert Victimization in Canada, 2009; Statistics Canada: Ottawa, ON, Canada, 2011. [Google Scholar]
  38. Privitera, C.; Campbell, M.A. Cyberbullying: The new face of workplace bullying? CyberPsychol. Behav. 2009, 12, 395–400. [Google Scholar] [CrossRef] [PubMed]
  39. Olweus, D. Cyberbullying: An overrated phenomenon? Eur. J. Dev. Psychol. 2012, 9, 520–538. [Google Scholar] [CrossRef]
  40. Hobfoll, S.E. The Ecology of Stress; Hemisphere: New York, NY, USA, 1988. [Google Scholar]
  41. Hobfoll, S.E. Conservation of resources: A new attempt at conceptualizing stress. Am. Psychol. 1989, 44, 513–524. [Google Scholar] [CrossRef] [PubMed]
  42. Gorgievski, M.J.; Hobfoll, S.E. Work can burn us out or fire us up: Conservation of resources in burnout and engagement. Handb. Stress Burn. Health Care 2008, 1, 7–22. [Google Scholar]
  43. Matthiesen, S.; Einarsen, S. Perpetrators and targets of bullying at work: Role stress and individual differences. Violence Vict. 2007, 22, 735–753. [Google Scholar] [CrossRef] [PubMed]
  44. Tuckey, M.R.; Neall, A.M. Workplace bullying erodes job and personal resources: Between-and within-person perspectives. J. Occup. Health Psychol. 2014, 19, 413–424. [Google Scholar] [CrossRef] [PubMed]
  45. Hobfoll, S.E.; Shirom, A. Stress and burnout in the workplace: Conservation of resources. Handb. Organ. Behav. 1993, 41–61. [Google Scholar]
  46. Tuckey, M.R.; Dollard, M.; Hosking, P.J.; Winefield, A.H. Workplace bullying: The role of psychosocial work environment factors. Int. J. Stress Manag. 2009, 16, 215–232. [Google Scholar] [CrossRef]
  47. Gardner, D.; Bentley, T.; Catley, B.; Cooper-Thomas, H.; O’Driscoll, M.; Trenberth, L. Ethnicity, workplace bullying, social support and psychological strain in Aotearoa/New Zealand. N. Z. J. Psychol. 2013, 42, 84–91. [Google Scholar]
  48. Finne, L.B.; Knardahl, S.; Lau, B. Workplace bullying and mental distress—A prospective study of Norwegian employees. Scand. J. Work Environ. Health 2011, 37, 276–287. [Google Scholar] [CrossRef] [PubMed]
  49. Nielsen, M.B.; Hetland, J.; Matthiesen, S.; Einarsen, S. Longitudinal relationships between workplace bullying and psychological distress. Scand. J. Work Environ. Health 2012, 38, 38–46. [Google Scholar] [CrossRef] [PubMed]
  50. Hoel, H.; Rayner, C.; Cooper, C.L. Workplace bullying. Int. Rev. Ind. Organ. Psychol. 1999, 14, 195–230. [Google Scholar]
  51. Balducci, C.; Fraccaroli, F.; Schaufeli, W.B. Workplace bullying and its relation with work characteristics, personality, and post-traumatic stress symptoms: An integrated model. Anxiety Stress Coping 2011, 24, 499–513. [Google Scholar] [CrossRef] [PubMed]
  52. Notelaers, G.; De Witte, H.; Einarsen, S. A job characteristics approach to explain workplace bullying. Eur. J. Work Organ. Psychol. 2009, 19, 487–504. [Google Scholar] [CrossRef]
  53. Skogstad, A.; Torsheim, T.; Einarsen, S. Testing the work environment hypothesis of bullying on a group level of analysis: Psychosocial factors as precursors of observed workplace bullying. Appl. Psychol. An Int. Rev. 2011. [Google Scholar] [CrossRef]
  54. Tuckey, M.R. Like a “Cancer“ of the Workplace, Bullying is a Symptom of Dysfunction. Available online: http://theconversation.com/like-a-cancer-of-the-workplace-bullying-is-a-symptom-of-dysfunction-43831 (accessed on 26 April 2016).
  55. Harold, C.M.; Holtz, B.C. The effects of passive leadership on workplace incivility. J. Organ. Behav. 2014. [Google Scholar] [CrossRef]
  56. Nielsen, M.B. Bullying in work groups: The impact of leadership. Scand. J. Psychol. 2012. [Google Scholar] [CrossRef]
  57. Einarsen, S.; Aasland, M.S.; Skogstad, A. Destructive leadership behavior: A definition and conceptual model. Leadersh. Quart. 2007, 18, 207–216. [Google Scholar]
  58. Shaw, J.B.; Erickson, A.; Harvey, M. A method for measuring destructive leadership and identifying types of destructive leaders in organizations. Leadersh. Quart. 2011, 22, 575–590. [Google Scholar]
  59. Brown, M.E.; Treviño, L.K.; Harrison, D.A. Ethical leadership: A social learning perspective for construct development and testing. Organ. Behav. Hum. Decis. Process. 2005, 97, 117–134. [Google Scholar]
  60. Eisenberger, R.; Huntingdon, R.; Hutchison, S.; Sowa, D. Perceived organizational support. J. Appl. Psychol. 1986, 71, 500–507. [Google Scholar] [CrossRef]
  61. Rhoades, L.; Eisenberger, R. Perceived organizational support: A review of the literature. J. Appl. Psychol. 2002, 87, 698–714. [Google Scholar] [CrossRef] [PubMed]
  62. Schat, A.C.H.; Kelloway, E.K. Reducing the adverse consequences of workplace aggression and violence: The buffering effects of organizational support. J. Occup. Health Psychol. 2003, 8, 110–122. [Google Scholar] [CrossRef] [PubMed]
  63. Namie, G.; Namie, R.U.S. Workplace bullying: Some basic considerations and consultation interventions. Consult. Psychol. J. Pract. Res. 2009, 61, 202–219. [Google Scholar] [CrossRef]
  64. Ferris, P. The role of the consulting psychologist in the prevention, detection and correction of bullying and mobbing in the workplace. Consult. Psychol. J. Pract. Res. 2009, 61, 169–189. [Google Scholar] [CrossRef]
  65. Hodgins, M.; MacCurtain, S.; Mannix-McNamara, P. Workplace bullying and incivility: A systematic review of interventions. Int. J. Workplace Health Manag. 2014, 7, 54–72. [Google Scholar] [CrossRef]
  66. Unden, A. Social support at work and its relationship to absenteeism. Work Stress 1996, 10, 46–61. [Google Scholar] [CrossRef]
  67. Ayoko, O.B.; Callan, V.J.; Härtel, C.E. Workplace conflict, bullying, and counterproductive behaviors. Int. J. Organ. Anal. 2003, 11, 283–301. [Google Scholar] [CrossRef]
  68. Brotheridge, C.M.; Lee, R.T. Examining the relationship between the perceived work environment and workplace bullying. Can. J. Commun. Ment. Health 2006, 25, 31–44. [Google Scholar] [CrossRef]
  69. Baillien, E.; Neyens, I.; De Witte, H. Organizational, team related and job related risk factors for workplace bullying, voilence and sexual harassment in the workplace: A qualitative study. Int. J. Organ. Behav. 2008, 13, 132–146. [Google Scholar]
  70. Kelloway, E.; Francis, L. Longitudinal research and data analysis. In Research Methods in Occupational Health Psychology: Measurement, Design, and Data Analysis; Sinclair, R.R., Wang, M., Tetrick, L.E., Eds.; Routledge: New York, NY, USA, 2013; pp. 374–394. [Google Scholar]
  71. Podsakoff, P.M.; Podsakoff, N.P.; Lee, J.-Y. Common method biases in behavioral research: A critical review of the literature and recommended remedies. J. Appl. Psychol. 2003, 88, 879–903. [Google Scholar] [CrossRef] [PubMed]
  72. Rainie, L. Digital “Natives” Invade the Workplace; Pew Internet & American Life Project: Washington, DC, USA, 2006. [Google Scholar]
  73. Einarsen, S.; Hoel, H.; Notelaers, G. Measuring exposure to bullying and harassment at work: Validity, factor structure and psychometric properties of the negative acts questionnaire-revised. Work Stress 2009, 23, 24–44. [Google Scholar] [CrossRef]
  74. Hauge, L.J.; Skogstad, A.; Einarsen, S. Relationships between stressful work environments and bullying: Results of a large representative study. Work Stress 2007, 21, 220–242. [Google Scholar] [CrossRef]
  75. Leymann, H. The content and development of mobbing at work. Eur. J. Work Organ. Psychol. 1996, 5, 165–185. [Google Scholar] [CrossRef]
  76. Goldberg, D.P. The Detection of Psychiatric Illness by Questionnaire; Oxford University Press: Oxford, UK, 1972. [Google Scholar]
  77. Goldberg, D.; Williams, P. A User’s Guide to the General health Questionnaire (GHQ); NFER-Nelson Publishing Company Limited: Windsor, UK, 1988. [Google Scholar]
  78. Kessler, R.C.; Barber, C.; Beck, A.; Berglund, P.; Cleary, P.D.; McKenas, D.K.; Pronk, N.; Simon, G.; Stang, P.E.; Ustun, T.B.; et al. Using the world health organization health and work performance questionnaire (hpq) to evaluate the indirect costs of illness. J. Occup. Environ. Med. 2004, 45, 156–174. [Google Scholar] [CrossRef]
  79. Spector, P.E.; Jex, S.M. Development of four self-report measures of job stressors and strain: Interpersonal conflict at work scale, organizational constraints scale, quantitative workload inventory, and physical symptoms inventory. J. Occup. Health Psychol. 1998, 3, 356–367. [Google Scholar] [CrossRef] [PubMed]
  80. Bentley, T.; Catley, B.; Cooper-Thomas, H.; Gardner, D.; O’Driscoll, M.; Trenberth, L. Understanding Stress and Bullying in New Zealand Workplaces: Final Report to OH&S Steering Committee; Health Research Council/New Zealand Department of Labour: Wellington, New Zealand, 2009; pp. 1–100. [Google Scholar]
  81. Jehn, K.A. A multimethod examinatin of the benefits and detriments of intragroup conflict. Adm. Sci. Quart. 1995, 40, 256–282. [Google Scholar] [CrossRef]
  82. Menesini, E.; Nocentini, A. Cyberbullying definition and measurement: Some critical considerations. J. Psychol. 2009, 217, 230–232. [Google Scholar] [CrossRef]
  83. Harrington, S.; Rayner, C.; Warren, S. Too hot to handle? Trust and human resource practitioners’ implementation of anti-bullying policy. Hum. Resour. Manag. J. 2012. [Google Scholar] [CrossRef]
  84. Polzer-Debruyne, A.M. Psychological and Workplace Attributes That Influence Personal Web Use (PWU); Massey University: Auckland, New Zealand, 2008. [Google Scholar]
  85. Van Laer, T. The means to justify the end: Combating cyber harassment in social media. J. Bus. Ethics 2014, 123, 85–98. [Google Scholar] [CrossRef]
  86. Hendricks, L.; Lumadue, R.; Waller, L.R. The evolution of bullying to cyber bullying: An overview of the best methods for implementing a cyber bullying preventive program. National Forum J. Counsel. Addict. 2012, 1, 1–9. [Google Scholar]
Table 1. Bivariate correlations between Time 1 variables and Time 2 workplace bullying and cyber-bullying. POS, perceived organisational support.
Table 1. Bivariate correlations between Time 1 variables and Time 2 workplace bullying and cyber-bullying. POS, perceived organisational support.
Variable NameMeanSD123456789101112
1.  Gender--
2.  Role--0.16 **
3.  Performance7.921.350.02−0.03
4.  Physical health1.630.500.19 **0.070 *−0.09 *
5.  Strain2.260.5160.09 *0.10 **−0.28 **0.43 **
6.  Absenteeism0.220.580.020.07−0.07 *0.12 **0.07
7.  Ethical leadership3.580.87−0.04−0.070.18 **−0.23 **−0.43 **−0.13 **
8.  Destructive Leadership1.910.810.09 *0.01−0.16 **0.23 **0.42 **0.11 **−0.75 **
9.  POS5.031.59−0.05−0.17 **0.20 **−0.27 **−0.51 **−0.10 **0.69 **−0.62 **
10.  Team conflict2.250.840.13 **0.01−0.08 *0.28 **0.33 **0.04−0.39 **0.48 **−0.42 **
11.  Effectiveness of org. strategies2.960.66−0.07−0.12 **0.17 **−0.17 **−0.44 **−0.13 **0.46 **−0.43 **0.52 **−0.39 **
12.  Workplace bullying T21.360.470.07 *−0.05−0.10 **0.34 **0.40 **0.09 **−0.41 **0.47 **−0.41 **0.47 **−0.38 **
13.  Cyber-bullying T21.100.25−0.00−0.13 **−0.010.16 **0.21 **0.07 *−0.18 **0.20 **−0.24 **0.19 **−0.23 **0.54 **
Notes: * p < 0.05; ** p < 0.01.
Table 2. Regression tests of hypotheses.
Table 2. Regression tests of hypotheses.
Variable NameWorkplace BullyingCyber-Bullying
BSE(B)BetaBSE(B)Beta
Gender−0.030.03−0.03−0.040.02−0.08
Role0.040.030.040.050.020.10 *
Performance0.000.010.000.010.010.07
Absenteeism0.050.030.07 *0.030.020.07
Physical health0.130.040.15 ***0.050.030.10 *
Strain0.100.040.11 *0.040.030.07
Ethical leadership0.010.030.020.030.020.08
Destructive leadership0.130.030.22 ***−0.000.02−0.01
POS−0.020.02−0.08−0.040.01−0.21 **
Team Conflict0.110.020.20 ***0.000.020.00
Effectiveness of org. responses−0.080.03−0.11 **−0.070.02−0.16 **
F30.64 *** 7.11 ***
R20.38 0.13
R2 adj.0.37 0.11
Notes: * p < 0.05; ** p < 0.01; *** p < 0.001.
Back to TopTop