# Cronbach’s Alpha under Insufficient Effort Responding: An Analytic Approach

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

#### Work Related to Cronbach’s Alpha under IER or Mixture Models

## 2. Mathematics for General Result

**V**is used to represent the valid distribution. Cronbach’s alpha is defined as

**V**, and $\overline{{\sigma}_{ijV}}$ to be the average covariance between distinct components of

**V**. Specifically,

**C**is used to represent the contaminating IER.

**M**, defined by

**M**is used to emphasize that it is a mixture of the valid and contaminating responses. Because W is either zero or one, each individual gives responses from one of the two response distributions. With probability p an individual will give contaminating responses, and with probability $1-p$ an individual will give valid responses.

**Lemma**

**1.**

**M**be defined as a mixture of

**V**and

**C**as in Equation (3), where p is the probability of observing a response from

**C**. Assume that the random quantities

**V**,

**C**, and W are independent. Then

- 1.
- $\overline{{\sigma}_{ijM}}=(1-p)\overline{{\sigma}_{ijV}}+p\overline{{\sigma}_{ijC}}+p(1-p)\overline{{\Delta}_{i}{\Delta}_{j}}$
- 2.
- $\overline{{\sigma}_{iM}^{2}}=(1-p)\overline{{\sigma}_{iV}^{2}}+p\overline{{\sigma}_{iC}^{2}}+p(1-p)\overline{{\Delta}_{i}^{2}}$

**Lemma**

**2.**

**V**is less than or equal to the average variance. Symbolically,

**Theorem**

**1.**

**V**and

**C**be multivariate distributions with k components representing potential responses to an instrument. Let W be a Bernoulli random variable with parameter p between zero and one. Define $\mathit{M}=(1-W)\mathit{V}+W\mathit{C}$ as a mixture of

**V**and

**C**. Assume that

**V**,

**C**, and W are independent. The behavior of Cronbach’s alpha under the mixture can be broken down into five categories.

- 1.
- Cronbach’s alpha does not change for any mixing proportion. ${\alpha}_{M}={\alpha}_{V}$ for all p.
- 2.
- Cronbach’s alpha inflates for any mixing proportion. ${\alpha}_{M}>{\alpha}_{V}$ for all p.
- 3.
- Cronbach’s alpha deflates for any mixing proportion. ${\alpha}_{M}<{\alpha}_{V}$ for all p.
- 4.
- Cronbach’s alpha inflates for small mixing proportions, but deflates for large mixing proportions. There is a value ${p}_{0}$ in the interval $(0,1)$ such that ${\alpha}_{M}>{\alpha}_{V}$ for $p<{p}_{0}$, but ${\alpha}_{M}<{\alpha}_{V}$ for $p>{p}_{0}$.
- 5.
- Cronbach’s alpha deflates for small mixing proportions, but inflates for large mixing proportions. There is a value ${p}_{0}$ in the interval $(0,1)$ such that ${\alpha}_{M}<{\alpha}_{V}$ for $p<{p}_{0}$, but ${\alpha}_{M}>{\alpha}_{V}$ for $p>{p}_{0}$.

**Proof.**

**M**.

- Case one is a trivial possibility when $a=b=0$. $f\left(p\right)=0$ for all p in $(0,1)$.
- Case two is $f\left(p\right)>0$ for all p in $(0,1)$. This has two subcases: if $a\ge 0$ and $b\ge 0$ (but not both $a=b=0$), or if $0<-a\le b$.
- Case three is $f\left(p\right)<0$ for all p in $(0,1)$. This has two subcases: if $a\le 0$ and $b\le 0$ (but not both $a=b=0$), or if $b\le -a<0$.
- Case four occurs when $0<b<-a$. The non-zero root ${p}_{0}$ lies in the interval $(0,1)$, meaning $f\left(p\right)$ changes sign at ${p}_{0}$. Because $a<0$ the function is concave down, so the bias changes from positive to negative.
- Case five occurs when $-a\le b<0$. The non-zero root ${p}_{0}$ is in the interval $(0,1)$, except now $a>0$ and the function is concave up. The bias changes from negative to positive.

**V**and

**C**in order to reproduce each case. Figure 2 shows, for each case, the bias of the simulation as a solid black line, the exact bias before discretization as a dotted blue line, and the function $f\left(p\right)$ as a dot-dash red line. To aid in seeing when the sign changes, there is a dashed line for the horizontal axis. The simulations used 10,000 respondents at each value of p.

**V**and

**C**in order to reproduce each case. A larger number of simulations was necessary to reduce the sampling variability and clearly see the bias in Cronbach’s alpha. We found 20,000 to be sufficient for all except case five, which used 100,000 simulations. The graphs for the binary case are not significantly different from the five-option case, and are omitted. This completes the proof. □

`alpha()`function in the

`psych`package [26] in R [27] produces a confidence interval for alpha and an analysis of how alpha will change if items are removed from this instrument.

**M**which implicitly depend on p, so the magnitude of the bias is a ratio of polynomials in p and is more difficult to analyze. Figure 2 makes it clear that $f\left(p\right)$ and the magnitude of the exact bias share the same roots and sign but potentially very different magnitudes. This is why the cases in Table 1 and Table 2 and Figure 2 do not differentiate between $f\left(p\right)$ being concave or convex; that property is not necessarily shared with the magnitude of the bias.

## 3. Discussion and Special Cases

- Random responding: Item responses are uniformly and independently chosen from those available.
- Straight-lining: The respondent chooses the same option for all items, either in an attempt to complete the instrument as quickly as possible or operating on the belief that all questions are sufficiently similar to the first. Different respondents may choose different options, but each respondent repeats their choice without deviation.

**Observation 1:**If all components of

**V**have a common mean, and all components of

**C**have a common mean, then the quadratic coefficient a cannot be positive. Cases one through four are still possible, but case five is not.

**Observation 2:**The forces pressuring Cronbach’s alpha to inflate are:

- Increasing the differences between means of
**V**and**C**when item validities have the same sign, - Increasing the ratio of average covariance to variance for the contaminating distribution,
- Decreasing the ratio of average covariance to variance for the valid distribution.

**C**increases, b becomes more positive, moving away from case three towards case two, possibly moving through cases four and five.

**V**decreases, a becomes more negative, which by itself would pressure alpha towards deflation, but b includes $-a$ in the sum, so b is becoming more positive. Also, the second term in b includes a negative average covariance of

**V**. Thus, b is increasing faster than a is decreasing, moving in the direction of case two and possibly case four.

**Observation 3:**If means are equal across items and distributions and contamination consists purely of random responses, then Cronbach’s alpha must deflate (except for the unusual case that ${\alpha}_{V}\le 0$). However, if the distributions have different means, either inflation or deflation is possible.

**V**and

**C**are not equal, then deflation is not guaranteed. Consider case two in Table 1 and Figure 2, in which the contaminating distribution has independent responses, yet alpha always inflates due to the difference in means. Case four also uses a contaminating distribution with independent observations, but whether alpha inflates or deflates depends on the exact mixing proportion p. Case two is noteworthy because both of the distributions contributing to the mixture have average covariances of zero (thus ${\alpha}_{V}={\alpha}_{C}=0$), yet the mixture has a positive value of Cronbach’s alpha. This scenario is discussed by Waller [16] as admittedly contrived and non-realistic, but useful as an example of the non-intuitive nature of reliability measures under mixtures.

**Observation 4:**Assume the valid distribution has a common mean and no questions use reverse keying. If contamination consists purely of straight-lining, then alpha is guaranteed to inflate.

**Observation 5:**If contamination is of a form that alternates between extremes, then case five is a possibility. Cronbach’s alpha deflates for small p, but inflates for larger p.

**Observation 6:**To investigate the effects of multiple types of IER occurring simultaneously, mixture models and the general covariance mixture theorem can be applied iteratively.

**C**are themselves a mixture of random, straight-lining, and perhaps other kinds of IER. Therefore the general covariance mixture theorem (Lemma 1 in the present paper) can be applied repeatedly to obtain the parameters of

**C**, at which point Theorem 1 can be applied to determine whether Cronbach’s alpha will deflate or inflate.

## 4. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## Appendix A. Proof of Lemma 2.

## References

- Osborne, J.W. Best Practices in Data Cleaning; SAGE Publications: Newcastle upon Tyne, UK, 2012. [Google Scholar]
- Huang, J.L.; Curran, P.G.; Keeney, J.; Poposki, E.M.; DeShon, R.P. Detecting and Deterring Insufficient Effort Responding to Surveys. J. Bus. Psychol.
**2012**, 27, 99–114. [Google Scholar] [CrossRef] - McGrath, R.E.; Mitchell, M.; Kim, B.H.; Hough, L. Evidence for response bias as a source of error variance in applied assessment. Psychol. Bull.
**2010**, 136, 450–470. [Google Scholar] [CrossRef] [PubMed] - Huang, J.L.; Liu, M.; Bowling, N.A. Insufficient effort responding: examining an insidious confound in survey data. J. Appl. Psychol.
**2015**, 100, 828–845. [Google Scholar] [CrossRef] [PubMed] - DeSimone, J.A.; DeSimone, A.J.; Harms, P.D.; Wood, D. The differential impacts of two forms of insufficient effort responding. Appl. Psychol.
**2018**, 67, 309–338. [Google Scholar] [CrossRef] - Credé, M. Random responding as a threat to the validity of effect size estimates in correlational research. Educ. Psychol. Meas.
**2010**, 70, 596–612. [Google Scholar] [CrossRef] - Holtzman, N.S.; Donnellan, M.B. A simulator of the degree to which random responding leads to biases in the correlations between two individual differences. Pers. Individ. Differ.
**2017**, 114, 187–192. [Google Scholar] [CrossRef] - Kam, C.C.S.; Meyer, J.P. How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organ. Res. Methods
**2015**, 18, 512–541. [Google Scholar] [CrossRef] - Schmitt, N.; Stuits, D.M. Factors defined by negatively keyed items: the result of careless respondents? Appl. Psychol. Meas.
**1985**, 9, 367–373. [Google Scholar] [CrossRef] - Cronbach, L.J. Coefficient Alpha and the Internal Structure of Tests. Psychometrika
**1951**, 16, 293–334. [Google Scholar] [CrossRef] - Fong, D.Y.; Ho, S.Y.; Lam, T.H. Evaluation of internal reliability in the presence of inconsistent responses. Health Qual. Life Outcomes
**2010**, 8, 1–10. [Google Scholar] [CrossRef] [PubMed] - Sijtsma, K. On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika
**2009**, 74, 107–120. [Google Scholar] [CrossRef] [PubMed] - McNeish, D.M. Thanks Coefficient Alpha, We’ll Take It From Here. Psychol. Methods
**2017**, 23, 412–433. [Google Scholar] [CrossRef] [PubMed] - Wertheimer, M.E. Identifying the types of insufficient effort responders. Master’s Thesis, Middle Tennessee State University, Murfreesboro, TN, USA, 2017. [Google Scholar]
- Attali, Y. Reliability of speeded number-right multiple-choice tests. Appl. Psychol. Meas.
**2005**, 29, 357–368. [Google Scholar] [CrossRef] - Waller, N.G. Commingled samples: A neglected source of bias in reliability analysis. Appl. Psychol. Meas.
**2008**, 32, 211–223. [Google Scholar] [CrossRef] - Kuder, G.F.; Richardson, M.W. The theory of the estimation of test reliability. Psychometrika
**1937**, 2, 151–160. [Google Scholar] [CrossRef] - Hoyt, C. Test relibability estimated by analysis of variance. Psychometrika
**1941**, 6, 153–160. [Google Scholar] [CrossRef] - Guttman, L. A basis for analyzing test-retest reliability. Psychometrika
**1945**, 10, 255–282. [Google Scholar] [CrossRef] [PubMed] - Lord, F.M.; Novick, M.R. Statistical Theories of Mental Test Scores; Addison-Wesley: Reading, MA, USA, 1968. [Google Scholar]
- Pinsoneault, T.B. Detecting random, partially random, and nonrandom Minnesota Multiphasic Personality Inventory-2 protocols. Psychol. Assess.
**2007**, 19, 159–164. [Google Scholar] [CrossRef] [PubMed] - Berry, D.T.R.; Wetter, M.W.; Baer, R.A.; Larsen, L.; Clark, C.; Monroe, K. MMPI-2 random responding indices: Validation using a self-report methodology. Psychol. Assess.
**1992**, 4, 340–345. [Google Scholar] [CrossRef] - Meehl, P.E. Psychodiagnosis: Selected Papers; University of Minnesota Press: Minneapolis, MN, USA, 1973; pp. 200–224. [Google Scholar]
- Waller, N.G.; Meehl, P.E. Multivariate Taxometric Procedures: Distinguishing Types from Continua; Sage: Thousand Oaks, CA, USA, 1998. [Google Scholar]
- Kreyszig, E. Introductory Functional Analysis with Applications; Wiley: Hoboken, NJ, USA, 1989. [Google Scholar]
- Revelle, W. Psych: Procedures for Psychological, Psychometric, and Personality Research; R Package Version 1.8.10; Northwestern University: Evanston, IL, USA, 2018. [Google Scholar]
- R Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2017. [Google Scholar]
- RStudio, Inc. Easy Web Applications in R. 2013. Available online: http://www.rstudio.com/shiny/ (accessed on 3 December 2018).
- Meijer, R.R.; Sijtsma, K. Methodology Review: Evaluating Person Fit. Appl. Psychol. Meas.
**2001**, 25, 107–135. [Google Scholar] [CrossRef] [Green Version] - Felt, J.M.; Castenada, R.; Tiemensma, J.; Depaoli, S. Using person fit statistics to detect outliers in survey research. Front. Psychol.
**2017**, 8. [Google Scholar] [CrossRef] [PubMed] - Embretson, S.E.; Reise, S.P. Item Response Theory for Psychologists; Lawrence Erlbaum Associates: Mahwah, NJ, USA, 2000. [Google Scholar]
- Curran, P.G. Methods for the detection of carelessly invalid responses in survey data. J. Exp. Soc. Psychol.
**2016**, 66, 4–19. [Google Scholar] [CrossRef] - Meade, A.W.; Craig, S.B. Identifying careless responses in survey data. Psychol. Methods
**2012**, 17, 437–455. [Google Scholar] [CrossRef] [PubMed] - Johnson, J.A. Ascertaining the validity of individual protocols from web-based personality inventories. J. Res. Personal.
**2005**, 39, 103–129. [Google Scholar] [CrossRef] - DeSimone, J.A.; Harms, P.D.; DeSimone, A.J. Best practice recommendations for data screening. J. Organ. Behav.
**2015**, 36, 171–181. [Google Scholar] [CrossRef] - Revelle, W. Hierarchical cluster analysis and the internal structure of tests. Multivar. Behav. Res.
**1979**, 14, 57–74. [Google Scholar] [CrossRef] [PubMed] - McDonald, R.P. Test Theory: A Unified Treatment; Lawrence Erlbaum Associates: Mahwah, NJ, USA, 1999. [Google Scholar]

**Figure 1.**Graphs showing the general behavior of cases two through five for $f\left(p\right)=a{p}^{2}+bp$, where p is the mixing proportion representing the probability of seeing a contaminating response. The sign of this function (positive or negative) on each region is equivalent to the sign of the bias of Cronbach’s alpha.

**Figure 2.**Graphs of $f\left(p\right)$, exact bias before discretization, and simulated bias after discretization for each case. The distributions described in Table 1 were used to produce these plots. In total, 10,000 respondents were simulated for each value of p.

**Table 1.**Summaries of

**V**and

**C**(before discretization) for producing each case in the context of a scale with five options.

Means of V | $\overline{{\mathit{\sigma}}_{\mathit{iV}}^{2}}$ | $\overline{{\mathit{\sigma}}_{\mathit{ijV}}}$ | Means of C | $\overline{{\mathit{\sigma}}_{\mathit{iC}}^{2}}$ | $\overline{{\mathit{\sigma}}_{\mathit{ijC}}}$ | |
---|---|---|---|---|---|---|

Case 1 | ${\mu}_{iV}=3$ for all items | 2 | 1 | ${\mu}_{iC}=3$ for all items | 2 | 1 |

Case 2 | ${\mu}_{iV}=2$ for all items | 1 | 0 | ${\mu}_{iC}=4$ for all items | 1 | 0 |

Case 3 | ${\mu}_{iV}=2$ for all items | 2 | 1.5 | ${\mu}_{iC}=4$ for all items | 6 | 0 |

Case 4 | ${\mu}_{iV}=2$ for all items | 1 | 0.5 | ${\mu}_{iC}=4$ for all items | 1 | 0 |

Case 5 | ${\mu}_{iV}=3$ for all items | 1 | 0.2 | ${\mu}_{iV}=1$ for odd items; | 1 | 0.8 |

${\mu}_{iV}=5$ for even items |

**Table 2.**Summaries of

**V**and

**C**(before discretization) for producing each case in the context of a scale with binary options.

Means of V | $\overline{{\mathit{\sigma}}_{\mathit{iV}}^{2}}$ | $\overline{{\mathit{\sigma}}_{\mathit{ijV}}}$ | Means of C | $\overline{{\mathit{\sigma}}_{\mathit{iC}}^{2}}$ | $\overline{{\mathit{\sigma}}_{\mathit{ijC}}}$ | |
---|---|---|---|---|---|---|

Case 1 | ${\mu}_{iV}=0.5$ for all items | 1 | 0.5 | ${\mu}_{iC}=0.5$ for all items | 1 | 0.5 |

Case 2 | ${\mu}_{iV}=0.4$ for all items | 1 | 0 | ${\mu}_{iC}=0.6$ for all items | 1 | 0.8 |

Case 3 | ${\mu}_{iV}=0.4$ for all items | 0.5 | 0.4 | ${\mu}_{iC}=0.6$ for all items | 1 | 0 |

Case 4 | ${\mu}_{iV}=0.2$ for all items | 0.3 | 0.1 | ${\mu}_{iC}=0.8$ for all items | 0.3 | 0 |

Case 5 | ${\mu}_{iV}=0.5$ for all items | 1 | 0.3 | ${\mu}_{iC}=0$ for odd items; | 0.5 | 0.2 |

${\mu}_{iC}=1$ for even items |

**Table 3.**An example of data simulated from case 2. Each class is inconsistent, with estimates of Cronbach’s alpha being 0.13 for the valid class and 0.063 for the contaminating class, yet the combined data set estimates alpha as 0.87.

Respondent | Response Class | Q1 | Q2 | Q3 | Q4 | Q5 |
---|---|---|---|---|---|---|

1 | Contaminating | 4 | 5 | 4 | 3 | 4 |

2 | Valid | 3 | 1 | 3 | 1 | 1 |

3 | Valid | 1 | 1 | 1 | 1 | 2 |

4 | Valid | 1 | 2 | 4 | 2 | 1 |

5 | Contaminating | 4 | 4 | 3 | 2 | 4 |

6 | Valid | 1 | 1 | 1 | 1 | 2 |

7 | Valid | 2 | 3 | 1 | 2 | 2 |

8 | Contaminating | 3 | 5 | 3 | 5 | 5 |

9 | Contaminating | 3 | 5 | 3 | 4 | 3 |

10 | Valid | 2 | 2 | 4 | 2 | 1 |

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Carden, S.W.; Camper, T.R.; Holtzman, N.S.
Cronbach’s Alpha under Insufficient Effort Responding: An Analytic Approach. *Stats* **2019**, *2*, 1-14.
https://doi.org/10.3390/stats2010001

**AMA Style**

Carden SW, Camper TR, Holtzman NS.
Cronbach’s Alpha under Insufficient Effort Responding: An Analytic Approach. *Stats*. 2019; 2(1):1-14.
https://doi.org/10.3390/stats2010001

**Chicago/Turabian Style**

Carden, Stephen W., Trevor R. Camper, and Nicholas S. Holtzman.
2019. "Cronbach’s Alpha under Insufficient Effort Responding: An Analytic Approach" *Stats* 2, no. 1: 1-14.
https://doi.org/10.3390/stats2010001