# Optimal Number of Choices in Rating Contexts

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Theoretical Characterization

## 3. Rounding Compression

## 4. Computational Simulations

Algorithm 1 Procedure for generating full scores in a uniform model |

Inputs: Number of scores n |

scoreSum $\leftarrow 0$ |

for $i=0:n\phantom{\rule{4pt}{0ex}}\mathbf{do}$ |

$r\leftarrow $ random(0,1) |

scores[i] $\leftarrow r$ |

scoresum = scoreSum $+r$ |

for $i=0:n\phantom{\rule{4pt}{0ex}}\mathbf{do}$ |

scores[i] = scores[i] / scoreSum |

Algorithm 2 Procedure for compressing scores | |

Inputs: scores[], number of total scores n, desired number of compressed scores k | |

$Z(n,k)\leftarrow \frac{n}{k}$ | ▹ Normalization |

for $i=0:n\phantom{\rule{4pt}{0ex}}\mathbf{do}$ | |

scoresCompressed $\left[\u230a\frac{i}{Z(n,k)}\u230b\right]$ += scores[i] |

Algorithm 3 Procedure for generating scores in a Gaussian model |

Inputs: Number of scores n, number of samples s, mean $\mu $, standard deviation $\sigma $ |

for $i=0:s\phantom{\rule{4pt}{0ex}}\mathbf{do}$ |

$r\leftarrow $ randomGaussian($\mu ,\sigma $) |

if $r<0$ then |

$r=0$ |

else if $r>n-1$ then |

$r\leftarrow n-1$ |

++scores[round(r)] |

for $i=0:n\phantom{\rule{4pt}{0ex}}\mathbf{do}$ |

scores[i] = scores[i] / s |

## 5. Experiments

#### 5.1. TripAdvisor Hotel Rating

#### 5.2. French Presidential Election

#### 5.3. Joke Recommender System

## 6. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Dubey, P.; Geanakoplos, J. Grading exams: 100, 99, 98, … or A, B, C? Games Econ. Behav.
**2010**, 69, 72–94. [Google Scholar] [CrossRef] - Likert, R.A. A Technique for Measurement of Attitudes. Arch. Psychol.
**1932**, 22, 140–155. [Google Scholar] - Matell, M.S.; Jacoby, J. Is there an optimal number of alternatives for Likert scale items? Study 1: Reliability and validity. Educ. Psychol. Meas.
**1971**, 31, 657–674. [Google Scholar] [CrossRef] - Wildt, A.R.; Mazis, M.B. Determinants of scale response: Label versus position. J. Mark. Res.
**1978**, 15, 261–267. [Google Scholar] [CrossRef] - Cox, E.P., III. The Optimal Number of Response Alternatives for a Scale: A Review. J. Mark. Res.
**1980**, 17, 407–442. [Google Scholar] [CrossRef] - Friedman, H.H.; Wilamowsky, Y.; Friedman, L.W. A comparison of balanced and unbalanced rating scales. Mid-Atl. J. Bus.
**1981**, 19, 1–7. [Google Scholar] - Garland, R. The mid-point on a rating scale: Is it desirable? Mark. Bull.
**1991**, 2, 66–70. [Google Scholar] - Matell, M.S.; Jacoby, J. Is there an optimal number of alternatives for Likert scale items? Effects of testing time and scale properties. J. Appl. Psychol.
**1972**, 56, 506–509. [Google Scholar] [CrossRef] - Sudman, S.; Bradburn, N.M.; Schwartz, N. Thinking about Answers: The Application of Cognitive Processes to Survey Methodology; Jossey-Bass Publishers: San Francisco, CA, USA, 1996. [Google Scholar]
- McKennell, A.C. Surveying attitude structures: A discussion of principles and procedures. Qual. Quant.
**1974**, 7, 203–294. [Google Scholar] [CrossRef] - Lewis, J.R.; Erdinç, O. User Experience Rating Scales with 7, 11, or 101 Points: Does It Matter? J. Usability Stud.
**2017**, 12, 73–91. [Google Scholar] - Preston, C.C.; Colman, A. Optimal Number of Response Categories in Rating Scales: Reliability, Validity, Discriminating Power, and Respondent Preferences. Acta Psychol.
**2000**, 104, 1–15. [Google Scholar] [CrossRef] - Lozano, L.; García-Cueto, E.; Muñiz, J. Effect of the Number of Response Categories on the Reliability and Validity of Rating Scales. Methodology
**2008**, 4, 73–79. [Google Scholar] [CrossRef] - Lehmann, D.R.; Hulbert, J. Are Three-point Scales Always Good Enough? J. Mark. Res.
**1972**, 9, 444. [Google Scholar] [CrossRef] - Alwin, D.F. Feeling Thermometers Versus 7-Point Scales: Which are Better? Sociol. Methods Res.
**1997**, 25, 318–340. [Google Scholar] [CrossRef] - Iyengar, S.S.; Lepper, M. When Choice is Demotivating: Can One Desire Too Much of a Good Thing? J. Personal. Soc. Psychol.
**2001**, 79, 995–1006. [Google Scholar] [CrossRef] - Fischhoff, B. Value Elicitation: Is There Anything in There? Am. Psychol.
**1991**, 46, 835–847. [Google Scholar] [CrossRef] - Kluver, D.; Nguyen, T.T.; Ekstrand, M.; Sen, S.; Riedl, J. How Many Bits Per Rating? In Proceedings of the Sixth ACM Conference on Recommender Systems, Dublin, Ireland, 9–13 September 2012; pp. 99–106. [Google Scholar] [CrossRef]
- Cosley, D.; Lam, S.K.; Albert, I.; Konstan, J.A.; Riedl, J. Is Seeing Believing?: How Recommender System Interfaces Affect Users’ Opinions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Gaithersburg, MD, USA, 15–17 March 2003; pp. 585–592. [Google Scholar] [CrossRef]
- Koren, Y.; Sill, J. OrdRec: An Ordinal Model for Predicting Personalized Item Rating Distributions. In Proceedings of the Fifth ACM Conference on Recommender Systems, Chicago, IL, USA, 23–27 October 2011; pp. 117–124. [Google Scholar] [CrossRef]
- Blédaité, L.; Ricci, F. Pairwise Preferences Elicitation and Exploitation for Conversational Collaborative Filtering. In Proceedings of the 26th ACM Conference on Hypertext & Social Media, Guzelyurt, Cyprus, 2–4 September 2015; pp. 231–236. [Google Scholar] [CrossRef]
- Kalloori, S. Pairwise Preferences and Recommender Systems. In Proceedings of the 22Nd International Conference on Intelligent User Interfaces Companion, Limassol, Cyprus, 13–16 March 2017; pp. 169–172. [Google Scholar] [CrossRef]
- Cordeiro de Amorim, R.; Hennig, C. Recovering the number of clusters in data sets with noise features using feature rescaling factors. Inf. Sci.
**2015**, 324, 126–145. [Google Scholar] [CrossRef] [Green Version] - Ming-Tso Chiang, M.; Mirkin, B. Intelligent Choice of the Number of Clusters in K-Means Clustering: An Experimental Study with Different Cluster Spreads. J. Classif.
**2010**, 27, 3–40. [Google Scholar] [CrossRef] [Green Version] - He, H.; Tan, Y. A dynamic genetic clustering algorithm for automatic choice of the number of clusters. In Proceedings of the 9th IEEE International Conference on Control and Automation (ICCA), Santiago, Chile, 19–21 December 2011. [Google Scholar]
- Pradel, B.; Usunier, N.; Gallinari, P. Ranking with Non-random Missing Ratings: Influence of Popularity and Positivity on Evaluation Metrics. In Proceedings of the Sixth ACM Conference on Recommender Systems, Santiago, Chile, 19–21 December 2012; pp. 147–154. [Google Scholar] [CrossRef]
- Mattei, N.; Walsh, T. PrefLib: A Library of Preference Data http://preflib.org. In Lecture Notes in Artificial Intelligence, Proceedings of the 3rd International Conference on Algorithmic Decision Theory (ADT 2013), Bruxelles, Belgium, 12–14 November 2013; Springer: Berlin/Heidelberg, Germany, 2013. [Google Scholar]
- Goldberg, K.; Roeder, T.; Gupta, D.; Perkins, C. Eigentaste: A constant time collaborative filtering algorithm. Inf. Retr.
**2001**, 4, 133–151. [Google Scholar] [CrossRef]

**Figure 3.**Example distribution where compressing with $k=2$ produces significantly lower error than $k=10$. The full distribution has mean 54.188, while the $k=2$ compression has mean 0.548 (54.253 after normalization) and the $k=10$ compression has mean 5.009 (55.009 after normalization). The normalized errors between the means were 0.906 for $k=10$ and 0.048 for $k=2$, yielding a difference of 0.859 in favor of $k=2$.

**Table 1.**Number of times each value of k in {2,3,4,5,10} produces minimal error and average error values, over 100,000 items generated according to both models.

2 | 3 | 4 | 5 | 10 | |
---|---|---|---|---|---|

Uniform # victories | 5564 | 9265 | 14,870 | 16,974 | 53,327 |

Uniform average error | 1.32 | 0.86 | 0.53 | 0.41 | 0.19 |

Gaussian # victories | 3025 | 7336 | 14,435 | 17,800 | 57,404 |

Gaussian average error | 1.14 | 0.59 | 0.30 | 0.22 | 0.10 |

2 | 3 | |
---|---|---|

Uniform number of victories | 36,805 | 63,195 |

Uniform average error | 1.31 | 0.86 |

Gaussian number of victories | 30,454 | 69,546 |

Gaussian average error | 1.13 | 0.58 |

2 | 10 | |
---|---|---|

Uniform number of victories | 8253 | 91,747 |

Uniform average error | 1.32 | 0.19 |

Gaussian number of victories | 4369 | 95,631 |

Gaussian average error | 1.13 | 0.10 |

**Table 4.**Number of times each value of k in {2,10} produces minimal error and average error values, over 100,000 items generated according to both models. For $k=10$, we only permitted scores between 3 and 6 (inclusive). If a score was below 3, we set it to be 3, and above 6 to 6.

2 | 10 | |
---|---|---|

Uniform number of victories | 32,250 | 67,750 |

Uniform average error | 1.31 | 0.74 |

Gaussian number of victories | 10,859 | 89,141 |

Gaussian average error | 1.13 | 0.20 |

**Table 5.**Number of times each value of k in {2,10} produces minimal error and average error values, over 100,000 items generated according to both generative models. For $k=10$, we only permitted scores between 3 and 7 (inclusive). If a score was below 3, we set it to be 3, and above 7 to 7.

2 | 10 | |
---|---|---|

Uniform number of victories | 93,226 | 6774 |

Uniform average error | 1.31 | 0.74 |

Gaussian number of victories | 54,459 | 45,541 |

Gaussian average error | 1.13 | 1.09 |

**Table 6.**Number of times each value of k produces minimal error and average error values, over 100,000 items generated according to both models with rounding compression.

2 | 3 | 4 | 5 | 10 | |
---|---|---|---|---|---|

Uniform # victories | 15,766 | 33,175 | 21,386 | 19,995 | 9678 |

Uniform average error | 0.78 | 0.47 | 0.55 | 0.52 | 0.50 |

Gaussian # victories | 13,262 | 64,870 | 10,331 | 9689 | 1848 |

Gaussian average error | 0.67 | 0.24 | 0.50 | 0.50 | 0.50 |

2 | 3 | |
---|---|---|

Uniform number of victories | 33,585 | 66,415 |

Uniform average error | 0.78 | 0.47 |

Gaussian number of victories | 18,307 | 81,693 |

Gaussian average error | 0.67 | 0.24 |

2 | 10 | |
---|---|---|

Uniform number of victories | 37,225 | 62,775 |

Uniform average error | 0.78 | 0.50 |

Gaussian number of victories | 37,897 | 62,103 |

Gaussian average error | 0.67 | 0.50 |

2 | 10 | |
---|---|---|

Uniform number of victories | 55,676 | 44,324 |

Uniform average error | 0.79 | 0.89 |

Gaussian number of victories | 24,128 | 75,872 |

Gaussian average error | 0.67 | 0.34 |

2 | 10 | |
---|---|---|

Uniform number of victories | 99,586 | 414 |

Uniform average error | 0.78 | 3.50 |

Gaussian number of victories | 95,692 | 4308 |

Gaussian average error | 0.67 | 1.45 |

Average Error | k = 2 | 3 | 4 |
---|---|---|---|

Overall | 1.04 | 0.31 | 0.15 |

Price | 1.07 | 0.27 | 0.14 |

Rooms | 1.06 | 0.32 | 0.16 |

Location | 1.47 | 0.42 | 0.16 |

Cleanliness | 1.43 | 0.40 | 0.16 |

Front Desk | 1.34 | 0.33 | 0.14 |

Service | 1.24 | 0.32 | 0.14 |

Business Service | 0.96 | 0.28 | 0.18 |

Minimal Error | k = 2 | 3 | 4 |
---|---|---|---|

Overall | 235 | 450 | 1157 |

Price | 181 | 518 | 1143 |

Rooms | 254 | 406 | 1182 |

Location | 111 | 231 | 1500 |

Cleanliness | 122 | 302 | 1418 |

Front Desk | 120 | 387 | 1335 |

Service | 140 | 403 | 1299 |

Business Service | 316 | 499 | 1027 |

# of Victories | k = 2 vs. 3 | 2 vs. 4 | 3 vs. 4 |
---|---|---|---|

Overall | 243, 1599 | 277, 1565 | 5, 1837 |

Price | 187, 1655 | 211, 1631 | 4, 1838 |

Rooms | 275, 1567 | 283, 1559 | 10, 1832 |

Location | 126, 1716 | 122, 1720 | 11, 1831 |

Cleanliness | 126, 1716 | 141, 1701 | 5, 1837 |

Front Desk | 130, 1712 | 133, 1709 | 8, 1834 |

Service | 153, 1689 | 152, 1690 | 11, 1831 |

Business Service | 368, 1474 | 329, 1513 | 22, 1820 |

Average Error | k = 2 | 3 | 4 |
---|---|---|---|

Overall | 0.50 | 0.28 | 0.15 |

Price | 0.48 | 0.31 | 0.15 |

Rooms | 0.48 | 0.30 | 0.16 |

Location | 0.63 | 0.41 | 0.22 |

Cleanliness | 0.6 | 0.4 | 0.21 |

Front Desk | 0.55 | 0.39 | 0.21 |

Service | 0.52 | 0.36 | 0.18 |

Business Service | 0.39 | 0.36 | 0.18 |

Minimal Error | k = 2 | 3 | 4 |
---|---|---|---|

Overall | 82 | 132 | 1628 |

Price | 92 | 74 | 1676 |

Rooms | 152 | 81 | 1609 |

Location | 93 | 52 | 1697 |

Cleanliness | 79 | 44 | 1719 |

Front Desk | 89 | 50 | 1703 |

Service | 102 | 29 | 1711 |

Business Service | 246 | 123 | 1473 |

# of Victories | k = 2 vs. 3 | 2 vs. 4 | 3 vs. 4 |
---|---|---|---|

Overall | 161, 1681 | 113, 1729 | 486, 1356 |

Price | 270, 1572 | 101, 1741 | 385, 1457 |

Rooms | 344, 1498 | 173, 1669 | 575, 1267 |

Location | 275, 1567 | 109, 1733 | 344, 1498 |

Cleanliness | 210, 1632 | 90, 1752 | 289, 1553 |

Front Desk | 380, 1462 | 95, 1747 | 332, 1510 |

Service | 358, 1484 | 109, 1733 | 399, 1443 |

Business Service | 870, 972 | 278, 1564 | 853, 989 |

Overall | Average error | 0.15, 0.21 |

# of victories | 1007, 835 | |

Price | Average error | 0.15, 0.17 |

# of victories | 955, 887 | |

Rooms | Average error | 0.15, 0.23 |

# of victories | 1076, 766 | |

Location | Average error | 0.22, 0.22 |

# of victories | 694, 1148 | |

Cleanliness | Average error | 0.21, 0.19 |

# of victories | 653, 1189 | |

Front Desk | Average error | 0.21, 0.17 |

# of victories | 662, 1180 | |

Service | Average error | 0.18, 0.18 |

# of victories | 827, 1015 | |

Business Service | Average error | 0.18, 0.31 |

# of victories | 1233, 609 |

Average Error | 2 | 3 | 4 | 5 | 8 | 10 |
---|---|---|---|---|---|---|

Francois Bayrou | 3.18 | 1.5 | 0.94 | 0.66 | 0.3 | 0.2 |

Olivier Besancenot | 1.7 | 0.8 | 0.5 | 0.35 | 0.16 | 0.1 |

Christine Boutin | 1.15 | 0.54 | 0.34 | 0.24 | 0.11 | 0.07 |

Jacques Cheminade | 0.64 | 0.3 | 0.19 | 0.13 | 0.06 | 0.04 |

Jean-Pierre Chevenement | 3.69 | 1.74 | 1.09 | 0.77 | 0.35 | 0.23 |

Jacques Chirac | 3.48 | 1.64 | 1.03 | 0.72 | 0.33 | 0.21 |

Robert Hue | 2.39 | 1.12 | 0.7 | 0.49 | 0.22 | 0.14 |

Lionel Jospin | 5.45 | 2.57 | 1.61 | 1.13 | 0.52 | 0.33 |

Arlette Laguiller | 2.2 | 1.04 | 0.65 | 0.46 | 0.21 | 0.13 |

Brice Lalonde | 1.53 | 0.72 | 0.45 | 0.32 | 0.14 | 0.09 |

Corine Lepage | 2.24 | 1.06 | 0.67 | 0.47 | 0.22 | 0.14 |

Jean-Marie Le Pen | 0.4 | 0.19 | 0.12 | 0.08 | 0.04 | 0.02 |

Alain Madelin | 1.93 | 0.91 | 0.57 | 0.4 | 0.18 | 0.12 |

Noel Mamere | 3.68 | 1.74 | 1.09 | 0.77 | 0.35 | 0.23 |

Bruno Maigret | 0.31 | 0.15 | 0.09 | 0.06 | 0.03 | 0.02 |

Average Error | 2 | 3 | 4 | 5 | 8 | 10 |
---|---|---|---|---|---|---|

Francois Bayrou | 1.65 | 0.73 | 0.91 | 0.75 | 0.48 | 0.62 |

Olivier Besancenot | 3.88 | 2.39 | 2.14 | 1.7 | 1.31 | 1.25 |

Christine Boutin | 3.87 | 2.39 | 1.84 | 1.5 | 0.9 | 0.86 |

Jacques Cheminade | 4.34 | 2.72 | 2.07 | 1.65 | 1.02 | 0.88 |

Jean-Pierre Chevenement | 1.47 | 0.65 | 1.2 | 0.82 | 0.55 | 0.61 |

Jacques Chirac | 1.64 | 1.0 | 1.13 | 0.88 | 0.55 | 0.64 |

Robert Hue | 2.51 | 1.27 | 1.14 | 1.09 | 0.67 | 0.77 |

Lionel Jospin | 0.33 | 0.49 | 0.87 | 0.67 | 0.51 | 0.63 |

Arlette Laguiller | 2.62 | 1.34 | 1.34 | 1.02 | 0.6 | 0.63 |

Brice Lalonde | 3.45 | 1.9 | 1.55 | 1.21 | 0.66 | 0.78 |

Corine Lepage | 2.89 | 1.59 | 1.56 | 1.16 | 0.79 | 0.87 |

Jean-Marie Le Pen | 4.92 | 3.26 | 2.55 | 2.06 | 1.39 | 1.2 |

Alain Madelin | 3.18 | 1.8 | 1.52 | 1.17 | 0.72 | 0.7 |

Noel Mamere | 2.02 | 1.55 | 1.77 | 1.44 | 1.29 | 1.41 |

Bruno Maigret | 4.88 | 3.23 | 2.46 | 1.99 | 1.28 | 1.1 |

Average Error | 2 | 3 | 4 | 5 | 10 |
---|---|---|---|---|---|

Joke 5 | 0.57 | 0.53 | 0.52 | 0.51 | 0.5 |

Joke 7 | 1.32 | 0.88 | 0.74 | 0.66 | 0.54 |

Joke 8 | 1.51 | 0.97 | 0.8 | 0.71 | 0.56 |

Joke 13 | 2.52 | 1.45 | 1.09 | 0.91 | 0.61 |

Joke 15 | 2.48 | 1.43 | 1.08 | 0.91 | 0.62 |

Joke 16 | 3.72 | 2.01 | 1.44 | 1.16 | 0.69 |

Joke 17 | 1.94 | 1.18 | 0.92 | 0.8 | 0.58 |

Joke 18 | 1.51 | 0.97 | 0.79 | 0.71 | 0.56 |

Joke 19 | 0.8 | 0.64 | 0.58 | 0.56 | 0.51 |

Joke 20 | 1.77 | 1.1 | 0.87 | 0.76 | 0.57 |

Average Error | 2 | 3 | 4 | 5 | 10 |
---|---|---|---|---|---|

Joke 5 | 0.48 | 0.47 | 0.48 | 0.47 | 0.48 |

Joke 7 | 1.2 | 1.2 | 1.2 | 1.2 | 1.2 |

Joke 8 | 1.44 | 1.43 | 1.42 | 1.43 | 1.42 |

Joke 13 | 2.43 | 2.43 | 2.43 | 2.42 | 2.42 |

Joke 15 | 2.34 | 2.34 | 2.33 | 2.33 | 2.33 |

Joke 16 | 3.59 | 3.58 | 3.57 | 3.57 | 3.57 |

Joke 17 | 1.84 | 1.82 | 1.82 | 1.81 | 1.81 |

Joke 18 | 1.45 | 1.44 | 1.44 | 1.44 | 1.44 |

Joke 19 | 0.72 | 0.72 | 0.71 | 0.71 | 0.71 |

Joke 20 | 1.65 | 1.63 | 1.63 | 1.63 | 1.63 |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Ganzfried, S.; Yusuf, F.B.
Optimal Number of Choices in Rating Contexts. *Big Data Cogn. Comput.* **2019**, *3*, 48.
https://doi.org/10.3390/bdcc3030048

**AMA Style**

Ganzfried S, Yusuf FB.
Optimal Number of Choices in Rating Contexts. *Big Data and Cognitive Computing*. 2019; 3(3):48.
https://doi.org/10.3390/bdcc3030048

**Chicago/Turabian Style**

Ganzfried, Sam, and Farzana Beente Yusuf.
2019. "Optimal Number of Choices in Rating Contexts" *Big Data and Cognitive Computing* 3, no. 3: 48.
https://doi.org/10.3390/bdcc3030048