Application of Resolution Regression and Resolution Graphs in Evaluating Probability Forecasts Generated Using Binary Choice Models

Abstract

Binary choice models are widely used in econometric modeling when the dependent variable corresponds to discrete outcomes. With appropriate decision rules, these models provide predictions of binary choices generated from predicted probabilities. The accuracy of these predictions in terms of classifying probabilities to events that occurred versus those that did not is a key issue. The use of expectation-prediction success at present is the standard method used to assess the accuracy of these predictions. However, this method is limited in its ability to correctly classify probabilities in the absence of appropriate predetermined cut-off levels. We propose alternative methods to classify probabilities generated through binary choice models, namely resolution graphs and resolution regressions that measure the ability to sort predicted probabilities against observed outcomes. Using probabilities generated from the use of logit models applied to purchasing decisions of various non-alcoholic beverages made by U.S. households, we compare probability sorting power using expectation-prediction success as well as resolution graphs and resolution regressions. Based on expectation-prediction success, the logit models performed better at classifying outcomes related to purchasing isotonic drinks, regular soft drinks, diet drinks, bottled water, and tea. Based on resolution regressions, the null hypothesis of perfect sorting of probabilities was rejected for all non-alcoholic beverages. Although the logit models generated upward-sloping resolution graphs as expected, they were relatively flat compared to the 45-degree perfect sorting line. Going forward, we recommend using resolution regression and resolution graphs to capture sorting of probabilities in addition to the conventional metrics used in ascertaining the ability of binary choice models to predict out-of-sample behavior.

1. Introduction

Qualitative choice models are widely used in econometric modeling when the dependent variable corresponds to discrete outcomes (Train, 2003). Binary logit and probit models, where the dependent variable corresponds to binary (0,1) outcomes, are popular among a wide range of1 qualitative choice models available. The use of the probit/logit2 analysis, particularly of binary choices, is well established in the economics and econometrics literature (Maddala, 1983; McFadden, 1984; Pindyck & Rubinfield, 1998). Once estimated, with appropriate decision rules, binary choice models are designed to provide predictions (probabilities) of the binary choices given a set of explanatory factors.

A key question relates to the accuracy of these predicted probabilities in terms of classifying them to events that occurred versus those that did not. Accuracy can be measured using a metric such as expectation-prediction success, which at present is the standard method to evaluate the predictive performance of qualitative choice models (Stock & Watson, 2007). With this metric, the percentage of correct (incorrect) predictions is calculated in comparison to the total number of predictions based on a predetermined probability cut-off level. However, the use of expectation-prediction success is limited in the ability to correctly classify the probabilities in the absence of appropriate predetermined cut-off levels3.

We propose resolution regression (also known as covariance regression) and resolution graphs (also known as covariance graphs), metrics that have not been used in the extant literature to measure the accuracy of predicted probabilities from binary choice models. Resolution graphs and resolution regression measure the ability of binary choice models to sort predicted probabilities against observed outcomes. To the best of our knowledge, our work is the first to evaluate forecasts developed from binary choice models (logit models in this study) using resolution regression and resolution graphs.

In this light, the general objective of this study is to propose alternative methods to evaluate the forecast performance of binary choice models. The specific objectives of this study are twofold:

(1)

to evaluate the out-of-sample prediction success of logit models using resolution regression and resolution graphs, and

(2)

to compare the use of resolution regression and resolution graphs with conventional prediction-success metrics that rely on predetermined cut-off values. To achieve these objectives4, we estimate logit models associated with purchases/non-purchases of non-alcoholic beverages made by U.S. households. We subsequently retrieve out-of-sample predicted probabilities of purchase and non-purchase. In turn, these predicted probabilities are then evaluated based on expectation-prediction success as well as using resolution regression and resolution graphs.

Based on sensitivity and specificity, the logit models performed better at classifying outcomes related to purchasing isotonic drinks, regular soft drinks, diet drinks, bottled water, and tea. The reverse was true for classifying outcomes related to not purchasing high-fat milk, low-fat milk, fruit drinks, fruit juices, and tea. Based on resolution regressions, the null hypothesis of perfect sorting of probabilities was rejected for all the non-alcoholic beverages considered. Although the respective logit models generated upward-sloping resolution graphs as expected, they were relatively flat compared to the 45-degree perfect sorting line. Except for isotonic drinks, the logit models were better at sorting forecasted probabilities associated with purchasing as opposed to not purchasing beverages. Based on our findings, we recommend using resolution regression and resolution graphs to capture sorting of probabilities in addition to the conventional metric used (expectation-prediction success) in ascertaining the ability of binary choice models to predict out-of-sample behavior.

2. Materials and Methods

To evaluate the effectiveness of the logit models to correctly classify the decision to purchase non-alcoholic beverages, we initially generate two random samples of observations. Then we estimate the respective logit models using one sample (the training sample) and reserve the data from the second sample (the testing sample) to obtain out-of-sample forecasted probabilities. In our analysis, we divide the sample of 7639 household-level observations associated with purchases or non-purchases of non-alcoholic beverages in half, generating two random samples of data based on the use of the SAS 9.2 Enterprise Minor data mining software (SAS Institute, Inc., 2022). The first sample, labeled as sample A (the training sample), contains 3820 household-level observations, and the second sample, labeled as sample B (the testing sample), contains 3819 household-level observations. Simply put, we use sample A to estimate the logit model dealing with the decision to buy or not to buy a particular non-alcoholic beverage. Next, using the estimated coefficients of the explanatory variables from the logit model based on sample A, we rely on data from sample B to generate out-of-sample forecast probabilities of purchase and non-purchase. That is, we use the estimated coefficients obtained from sample A and the observations associated with the explanatory variables from sample B to generate the out-of-sample forecast probabilities. Following Equation (1), we depict the estimated logit model using sample A observations.

$$Z=X^{\prime }\beta$$

(1)

where the cumulative distribution function, $F_L$, associated with the logistic distribution of $Z$ is shown in Equation (2) below.

$$P=F_L\left(Z_i\right)={\displaystyle \frac{e^{Z_i}}{(1+e^{Z_i})}}$$

(2)

where,

• $Z$ = vector of index values generated for each observation through regression of the binary choice variable on the vector of explanatory variables;
• $X$ = vector of explanatory variables in the logistic regression (price of each product and other socio-economic and demographic variables in this study);
• $\beta$ = vector of estimated regression coefficients associated with each explanatory variable;
• $P$ = associated probability generated through assuming the index variable, Z, has a logistic distribution;
• $F_L$ = cumulative distribution function.

Expectation-prediction success tables are two-by-two contingency tables concerning expected outcomes conditional on predicted probabilities. These tables provide the number of instances in which the purchase of the non-alcoholic beverage is correctly and incorrectly predicted, and the number of instances in which the non-purchase of the non-alcoholic beverage is correctly and incorrectly predicted. For classification purposes, the cut-off probability value often has been assumed to be 0.5 in the extant literature. Hence, if the predicted probability generated via the estimation of the logit model is greater than 0.5, that observation is predicted to be associated with the event that occurred (i.e., purchase of the non-alcoholic beverage, outcome index equals 1). On the other hand, if the predicted probability generated via the estimation of the logit model is less than 0.5, that observation is predicted to be associated with an event that did not occur (i.e., non-purchase of the non-alcoholic beverage, outcome index equals 0). The overall percentage of correct predictions can be used as a measure of out-of-sample forecast performance (ability to correctly classify decision outcomes).

We illustrate the measurement of expectation-prediction success in

Table 1. A two-by-two contingency table of expected-prediction success.

		Actual Outcome
		0	1
Predicted Outcome	0	True Negative	False Negative
	1	False Positive	True Positive

. The various entries in Table 1 are associated with a predetermined cut-off value in conjunction with the predicted probabilities generated from binary choice models. Entry TN (true negative) depicts the number of occurrences where the predicted and observed outcomes match, that is, where the outcome index equals 0 (not purchasing non-alcoholic beverages in our study). Similarly, entry TP (true positive) depicts the number of occurrences where the predicted and observed outcomes also match, that is, where the outcome index equals 1 (purchasing non-alcoholic beverages in our study). Entries FN (false negative) and FP (false positive) depict the number of occurrences of incorrect classifications. Entry FN deals with the number of times the predictions were made of not purchasing non-alcoholic beverages, but in fact purchases were made, and in like manner, entry FP deals with the number of times the predictions were made of purchasing non-alcoholic beverages, but in fact purchases were not made.

The number of correct out-of-sample predictions is given by $(TN+TP)$, the sum of the diagonal elements. The percentage of correct out-of-sample predictions is given by $\frac{(TN+TP)}{(TN+FN+FP+TP)}\ast 100\%$. The fraction of observations that are correctly predicted for which the outcome index equals 1 is called “sensitivity” in the literature and is depicted as ${\displaystyle \frac{TP}{(TP+FN)}}$. The fraction of observations that are correctly predicted for which the outcome index equals 0 is called “specificity” in the literature and is denoted by ${\displaystyle \frac{TN}{(TN+FP)}}$.

Another measure of forecast performance relates to the sum of the fraction of ones correctly predicted and the fraction of zeros correctly predicted, a number which should exceed unity if the classification of outcomes is of any value (Kennedy, 2003). In other words, the sum of sensitivity and specificity must exceed one.

In agreement with Greene (2012, p. 658), “in general any prediction rule will make two types of errors; it will incorrectly classify zeros as ones and ones as zeros.” For binary choice models, to the best of our knowledge, no benchmarks are provided regarding suitable prediction success rates or percentages of correct classifications. Without question, the choice of the cut-off probability level is critical to the decision to classify outcomes.

The choice of the cut-off probability level of 0.5 seems appropriate for discrete binary events that have realized relative frequencies close to 0.5 (the event occurs roughly 50% of the time). But suppose that the discrete binary events do not have realized frequencies of 50% or close to 50%. One example of this situation is the classification of a bull versus a bear market, wherein a bull market occurs 80% of the time, and a bear market occurs 20% of the time historically. This realized relative frequency can also be identified as the “market penetration.” In our analysis of non-alcoholic beverages, for example, the market penetration for bottled water is 70% (or probability of 0.7). Hence, the cut-off probability value that can be used to classify outcomes associated with the decision to purchase bottled water is 0.70 in this situation.

As noted previously, resolution is a metric of goodness of sorting power. In our work, we explicitly consider the ability of the logit models to sort probabilities into two classes, namely “high” probabilities associated with the purchase of non-alcoholic beverages and “low” probabilities associated with not purchasing non-alcoholic beverages. Ideally, we would like to see “high” probabilities associated with the purchase of a given non-alcoholic beverage, and “low” probabilities associated with the non-purchase of a given non-alcoholic beverage. Furthermore, for a perfect sorting model, we would like the probability to be close to 1 associated with the purchase of non-alcoholic beverages, and the probability to be close to 0 associated with the non-purchase of non-alcoholic beverages.

According to Sanders (1963), if any forecaster issues probability 1 for all events that occur (the purchase of non-alcoholic beverages) and probability 0 for all those events that do not occur (the non-purchase of non-alcoholic beverages), his/her sorting power is perfect (or he/she issues perfectly sorted probabilities). Dawid (1986) further stated that “it is unreasonable in general to expect perfect sorting, because perfect sorting is equivalent to absolutely correct or absolutely incorrect categorical forecasting.” Resolution, according to Yates (1982), is a metric of “goodness-of-sorting” power and can be measured using resolution regression. Put simply, with resolution regression, the predicted probabilities generated from the use of binary choice models are regressed as a function of the outcome index (0 and1). Mathematically, the Resolution regression is shown below in Equation (3).

$$Prob\left(Y\right|X)=\alpha +D\beta +e$$

(3)

where,

• $Y$ = predicted purchase probability generated from the binary choice model (logit model in this study);
• $X$ = vector of predictor variables used in the binary choice model (logit model in this study);
• $\alpha$ = intercept term;
• $D$ = observed outcome index (zero or one) associated with predicted probabilities of the binary choice model;
• $\beta$ = estimated parameter associated with the outcome index;
• $e$ = error term of the resolution regression. This error term is distributed with a non-linear Beta distribution.

Equation (3) is estimated using a maximum likelihood method, where the error term of the resolution regression (Equation (3)) is distributed Beta. The reason for not using the ordinary least squares (OLS) estimation method to estimate resolution regression is that, since the dependent variable is a bounded dependent variable where the bounds are 0 and 1, the estimates may not be efficient and consistent due to potential heteroscedasticity issues. Consequently, we use a “beta regression” to mitigate this problem and estimate the slope and intercept coefficients of the resolution regression. In this way, the slope and intercept coefficients are consistent and efficient under the assumption of beta-distributed errors in the resolution regression.

A joint test of the null hypothesis of the slope coefficient associated with the outcome index equal to one, and intercept equal to zero, can be carried out using a chi-square test. Note that the intercept coefficient ($\alpha$) captures the calibration bias, and the slope coefficient ($\beta$) captures the sorting slope. The joint hypothesis is shown below.

$$\begin{gathered} H_0:\beta =1,and\alpha =0 \\ {H}_1:\beta \ne 1,and\alpha \ne 0 \end{gathered}$$

(4)

Failing to reject this null hypothesis provides statistical evidence of perfect resolution (also known as perfect sorting) of probabilities for events that occurred and for probabilities of events that did not occur.

In addition, analysts can examine this sorting power visually by a plot of predicted probabilities on the y-axis and the outcome index on the x-axis. The line of perfect sorting coincides with a 45-degree line. In this case, all predicted probabilities are 0 when the outcome index equals 0, and all predicted probabilities are 1 when the outcome index equals 1. Some degree of sorting (or resolution) power of binary choice models is evident only in the case of upward-sloping graphs.

Resolution graphs provide visual representations of predicted probabilities and associated outcome indices, where a line can be drawn between the mean predicted probability associated with the outcome index of zero and the mean predicted probability associated with the outcome index of one. Upward-sloping graphs are expected for accurate classification of generated probabilities from binary choice models. This simple graphical representation of the classification of generated probabilities provides a visual aid that would go hand in hand with resolution regression, supported by statistical metrics. The statistical test of the hypothesis related to Equation (4) is characteristic only of resolution regression, compared to standard methods such as the use of expectation-prediction success tables, and other methods explained in footnote #3.

In our analysis, we initially construct resolution graphs and then regress forecasted probabilities as a function of the outcome index to determine their sorting power5. Any deviation of the slope coefficients from one and the intercept coefficients from zero jointly in these respective regressions would be characterized as poorly resolved probabilities. Additionally, out-of-sample prediction success tables are constructed based on the standard cut-off value commonly used in the literature (0.5) as well as cut-off values corresponding to the proportion of households that purchased ten non-alcoholic beverages (market penetration). With this information, we are positioned to compare and contrast the use of resolution regression and resolution graphs with the use of prediction-success metrics.

3. Results

The estimation of the respective logit models was done using a maximum likelihood procedure using SAS, version 9.4 (SAS Institute, Inc., 2022). The parameter estimates, standard errors, and p-values associated with the ten estimated logit models are provided in Appendix A. The set of explanatory variables considered was household socio-demographic factors, as well as the price of the respective non-alcoholic beverages. Nevertheless, the sole focus of our analysis is assessing the ability of the logit models to correctly classify the decision to purchase or not to purchase non-alcoholic beverages using conventional methods and two novel approaches, namely resolution graphs and resolution regressions. Consequently, we do not present a discussion dealing with the determination of the key socio-demographic factors or the impacts of price on affecting the probability of purchase of each non-alcoholic beverage. Suffice it to say that the set of explanatory variables considered was influential in affecting the likelihood or probability of purchasing or not purchasing each of the ten non-alcoholic beverages selected. The goodness-of-fit statistics, based on McFadden’s R², ranged from 0.0437 (tea) to 0.1415 (coffee). The magnitudes of the McFadden R² values are consistent with those reported in the economics literature.

As exhibited in

Table 2. Summary of expectation-prediction success using the 0.5 cut-off value and the market penetration cut-off value.

	0.5 Cut-Off Value			Market Penetration Cut-Off Value
Beverage Category	Sensitivity	Specificity	Sum of Sensitivity and Specificity	Sensitivity	Specificity	Sum of Sensitivity and Specificity
Isotonics	0.06	0.98	1.04	0.63	0.62	1.25
Regular Soft Drinks	1.00	0.00	1.00	0.68	0.64	1.35
Diet Soft Drinks	0.90	0.19	1.09	0.73	0.46	1.19
High-Fat Milk	1.00	0.00	1.00	0.51	0.71	1.22
Low-Fat Milk	0.87	0.27	1.15	0.55	0.67	1.22
Fruit Drinks	0.99	0.03	1.02	0.58	0.67	1.25
Fruit Juices	1.00	0.00	1.00	0.60	0.66	1.26
Bottled Water	0.98	0.07	1.05	0.66	0.54	1.20
Coffee	0.93	0.27	1.20	0.71	0.60	1.31
Tea	0.98	0.06	1.04	0.58	0.60	1.18

Source: Calculations made by the authors. Note: Sensitivity is the fraction of observations that are correctly predicted associated with the purchase of non-alcoholic beverages. Specificity is the fraction of observations that are correctly predicted associated with the non-purchase of non-alcoholic beverages. The sum of the sensitivity and the specificity is greater than 1 in all instances using the market penetration cut-off value. The higher the sum of sensitivity and specificity, the better the out-of-sample forecast performance of the logit model.

, the sensitivity, specificity, and the sum of sensitivity and specificity measures were calculated based on the out-of-sample probabilities generated using the 0.5 and the market penetration cut-off values for each beverage. For example, the market penetration value for diet soft drinks was calculated to be 65%. If the 0.5 cut-off was used to classify outcomes out-of-sample associated with the purchase or non-purchase of diet soft drinks, the sensitivity was calculated to be 0.90, and the specificity was calculated to be 0.19. However, if the market penetration was used as the cut-off value to classify the out-of-sample outcomes associated with the purchase or non-purchase of diet soft drinks, the sensitivity was calculated to be 0.73, and the specificity was calculated to be 0.46. The higher the sum of sensitivity and specificity, the better the out-of-sample forecast performance of the logit model concerning correct classifications. The respective sums of sensitivity and specificity were 1.09 and 1.19 for diet soft drinks. Hence, the better classification of outcomes occurred when the market penetration cut-off value was used. According to Table 2, this finding was perpetuated for each of the ten respective non-alcoholic beverages. Simply stated, the use of the market penetration cut-off yielded higher expectation-prediction success compared to the use of the 0.5 cut-off.

Further, the correct out-of-sample classification for purchases of non-alcoholic beverages ranged from 51% (high-fat milk) to 73% (diet soft drinks). On the other hand, the correct out-of-sample classification for non-purchases of non-alcoholic beverages varied from 46% (diet soft drinks) to 71% (high-fat milk). The logit models did a better job of classifying outcomes related to purchases of isotonic drinks, regular soft drinks, diet drinks, bottled water, and tea because their specificity measures exceeded their sensitivity measures. But the sensitivity measures exceeded the specificity measures for high-fat milk, low-fat milk, fruit drinks, fruit juices, and tea. In these instances, the logit models were better at classifying outcomes related to non-purchases of these beverage categories.

In Figure 1, we provide resolution graphs associated with the out-of-sample probability forecasts for each non-alcoholic beverage. As shown in Figure 1 as well as in

Table 3. Maximum Likelihood Estimates of the Parameters in the Resolution Regression (Beta Regression) and the Corresponding Mean Probabilities of Purchase and Non-Purchase.

Beverage Category	Intercept	Slope	Chi-Squared Statistics Associated with the Joint Test	Mean Probability for Purchases of Non-Alcoholic Beverages, Outcome Index = 1	Mean Probability for Non-Purchases of Non-Alcoholic Beverages, Outcome Index = 0
Regular Soft Drinks	1.8479 a (0.0338) b [0.0001] c	0.4185 (0.0355) [0.0001]	16,519.82 [0.0001]	0.91	0.84
Diet Soft Drinks	0.4527 (0.0150) [0.0001]	0.2356 (0.0177) [0.0001]	1994.99 [0.0001]	0.67	0.61
High-Fat Milk	1.2580 (0.0204) [0.0001]	0.2869 (0.0232) [0.0001]	6220.09 [0.0001]	0.82	0.77
Low-Fat Milk	0.2726 (0.0135) [0.0001]	0.2613 (0.0164) [0.0001]	2959.70 [0.0001]	0.63	0.57
Fruit Drinks	0.8761 (0.0169) [0.0001]	0.3191 (0.0197) [0.0001]	3039.86 [0.0001]	0.77	0.68
Fruit Juices	2.0894 (0.0497) [0.0001]	0.5715 (0.0514) [0.0001]	17,882.33 [0.0001]	0.93	0.89
Bottled Water	0.7128 (0.0147) [0.0001]	0.2471 (0.0174) [0.0001]	2343.81 [0.0001]	0.72	0.66
Coffee	0.5086 (0.0255) [0.0001]	0.6736 (0.0297) [0.0001]	539.81 [0.0001}	0.85	0.62
Tea	0.8160 (0.0146) [0.0001]	0.1991 (0.0168) [0.0001]	3127.16 [0.0001]	0.73	0.69
Isotonic Drinks	−1.3167 (0.0114) [0.0001]	0.3487 (0.0258) [0.0001]	20,539.25 [0.0001]	0.29	0.20

^a Estimated coefficient; ^b Estimated standard error; ^c p-value. Source: Estimation and calculations made by the authors. Note: All intercept and slope coefficients are statistically significant at the 0.01 level. The Chi-Square statistics correspond to the values associated with the joint test of the null hypothesis that the slope coefficient equals 1 and the intercept coefficient equals 0 (as shown in Equation (4)).

, we observed relatively high probabilities associated with the outcome index of one (indicative of purchasing non-alcoholic beverages) for regular soft drinks, diet soft drinks, high-fat milk, low-fat milk, fruit drinks, fruit juices, bottled water, coffee, and tea. Indeed, the mean probability values were 0.91, 0.67, 0.82, 0.63, 0.77, 0.93, 0.72, 0.85, and 0.73, respectively, for each of these beverages. However, for isotonics, we observed a mean probability value of 0.29 associated with the outcome index of 1.

In instances where purchases of a non-alcoholic beverage do not occur, we expect to observe relatively low mean probability values. Only in the case of isotonic drinks did we find this pattern. Consequently, the respective logit models underperformed in the sorting of out-of-sample forecast probabilities associated with the outcome index of zero (indicative of not purchasing non-alcoholic beverages), except for isotonic drinks. Additionally, we observed lower mean probabilities associated with the outcome index of zero for all non-alcoholic beverage categories compared to those associated with the outcome index of one. Bottom line, except for isotonic drinks, the logit models were better at sorting forecasted probabilities associated with purchasing as opposed to not purchasing non-alcoholic beverages.

The median values associated with the generated out-of-sample probabilities for each beverage are as follows: isotonic drinks 0.18, regular soft drinks 0.93, diet soft drinks 0.69, high-fat milk 0.82, low-fat milk 0.63, fruit drinks 0.78, fruit juices 0.95, bottled water 0.72, coffee 0.79, and tea 0.74. The 95% confidence intervals associated with each out-of-sample probability are as follows: isotonic drinks 0.0039, regular soft drinks 0.0019, diet soft drinks 0.0038, high-fat milk 0.0028, low-fat milk 0.0037, fruit drinks 0.0033, fruit juices 0.0018, bottled water 0.0033, coffee 0.0057, and tea 0.0030.

In addition, as exhibited in Table 3, each of the intercept coefficients of the resolution regressions for the set of non-alcoholic beverages was statistically different from zero, and each of the slope coefficients of the resolution regressions was statistically different from one at a level of significance of 0.01. Hence, as evidenced by the chi-squared statistics, the null hypothesis of perfect sorting of probabilities was rejected at a level of significance of 0.01. Although the respective logit models generated upward-sloping resolution graphs as expected, they were relatively flat compared to the 45-degree perfect sorting line.

4. Discussion

To demonstrate the application of resolution regression and resolution graphs, Nielsen Homescan scanner data concerning at-home purchasing behavior associated with ten non-alcoholic beverages in U.S. households were used in this analysis. Monthly household purchases (expenditure and quantity information) of regular soft drinks, diet (low-calorie) soft drinks, high-fat milk (defined as whole milk and 2% milk), low-fat milk (defined as 1% milk and skim milk), fruit drinks, fruit juices, bottled water, coffee, tea, and isotonic drinks (sports drinks) were obtained over the period January 2003 through December 20036 for 7639 U.S. households as part of a cooperative agreement with the Economic Research Service, U.S. Department of Agriculture.

The quantity data were standardized in terms of gallons per household for calendar year 2003, and expenditure data were expressed in terms of dollars per household for calendar year 2003. The prices associated with purchases were unit values derived as the ratio of expenditures (in dollars) to quantities (in gallons). However, some households may not have purchased non-alcoholic beverage products; in these situations, a weighted average unit value was generated by taking the ratio of the sum of total expenditures of all non-alcoholic beverages to the sum of quantities of all non-alcoholic beverages. This weighted average unit value was used as a proxy for the unobserved price of each non-alcoholic beverage considered in this study7. This weighted average unit value was positively correlated with the observed price (unit value) of each non-alcoholic beverage considered in this study8, justifying in part the appropriateness of the use of the weighted average unit value as a proxy for unobserved prices or unit values.

Demographic variables serve as the set of explanatory variables used in the respective logit models. Hill and Lynchehaun (2002) identified various cultural and socio-economic factors influencing consumer preferences, including age, ethnicity, income, education, gender, presence of children, region, and race. The detailed description of demographic categories of each variable used in this analysis is shown in

Table 4. Demographic characteristics considered in the binary logit models.

Demographic Characteristics	Categories
Age of Household Head	Less than 25 years
	25–29 years
	30–34 years
	35–44 years
	45–54 years
	55–64 years
	At least 65 years
Employment Status of Household Head	Household head employed full-time
	Household head employed part-time
	Household head not employed
Education Level of Household	Less than high school
	High school level
	Undergraduate level
	Post-college level
Region	Midwest
	South
	West
	East
Race of Household	Black
	Asian
	White
	Other
Hispanic Ethnicity of the Household Head	Hispanic Yes
	Hispanic No
Age and Presence of Children	Less than 6-years-old
	6–12-years-old
	13–17-years-old
	Less than 6-years-old and 6–12-years-old
	Less than 6-years-old and 13–17-years-old
	Less than 6-years-old, 6–12, and 13–17-years-old
	No children under 18 years
Gender of the Household Head	Household head male only
	Household head female only
	Households with both male and female
Poverty Status of the Household	Above poverty line of 185%
	Below poverty line of 185%

Source: Developed by authors.

. The demographic categories are age of the household head, employment status of the household head, education level of the household head, region where the household is located, race of the household head, Hispanic origin of the household head, age and presence of children, gender of the household head, and poverty status of the household (as a proxy for household income).

The information associated with the two random samples of households, Sample A and Sample B, is presented in

Table 5. Percentage of Households associated with Various Demographic Characteristics in Sample A and Sample B and Covariate Balance Test Results.

Demographic Characteristics	Categories	Percentage Sample A	Percentage Sample B	p-Value for the Balance Tests
Age of Household Head	25–29 years	2.28	2.57	0.40
	30–34 years	5.84	6.50	0.23
	35–44 years	21.80	20.56	0.18
	45–54 years	27.88	27.31	0.57
	55–64 years	22.62	23.88	0.18
	At least 65 years	19.40	18.80	0.50
Employment Status of Household Head	Employed full-time	45.97	44.88	0.33
	Employed part-time	15.84	17.02	0.15
Education Level of Household	High school level	25.24	23.25	0.04
	Undergraduate level	60.50	62.00	0.18
	Post-college level	10.92	11.02	0.88
Region	West	21.13	21.21	0.93
	Midwest	19.24	17.99	0.15
	South	38.53	39.17	0.54
Race of Household	Black	12.36	13.62	0.10
	Asian	2.60	3.14	0.15
	Other	6.10	6.78	0.22
Hispanic Ethnicity of the Household Head	Hispanic Yes	8.32	7.72	0.33
Age and Presence of Children	Less than 6-years-old 6-12-years-old 13-17-years-old Less than 6-years-old and 6-12-years-old Less than 6-years-old and 13-17-years-old 6-12 and 13-years-old Less than 6-years-old, 6-12, and 13-17-years-old	3.64 5.76 7.23 2.75 0.65 4.66 0.89 74.42	3.46 6.49 7.12 3.11 0.42 4.14 1.00 74.26	0.66 0.19 0.86 0.34 0.15 0.26 0.63
Gender of the Household Head	Household head male only	27.85	27.26	0.85
	Household head female only	10.52	10.66	0.55
Poverty Status of the Household	Below poverty line of 185%	13.64	13.12	0.52

Note: Sample mean difference test was conducted to compare the two samples, training (Sample A) and testing (Sample B). The null hypothesis is “there is no difference in sample means between Sample A and Sample B”. The cut-off p-value for the sample mean difference t-test is set at 0.01. Source: Developed by authors.

. The percentage of households in each demographic category is similar across the two samples of data. Balance tests of the covariates in Sample A (training) and Sample B (testing) are compared using a mean difference test (t-test). The associated p-values are also reported in Table 5. The covariates in Sample A and Sample B are not statistically different at the 0.05 significance level. As such, these results demonstrate a balance of the number of observations in the training (Sample A) and in the testing (Sample B).

In

Table 6. Number of households purchasing/not purchasing each beverage category and the accompanying market penetration.

	Sample A Number of Households			Sample B Number of Households
Beverage Category	Purchase	Non-Purchase	Market Penetration	Purchase	Non-Purchase	Market Penetration
Isotonic drinks	846	2974	22%	763	3056	20%
Regular Soft Drinks	3444	376	90%	3477	342	91%
Diet Soft Drinks	2494	1326	65%	2499	1320	65%
High-Fat Milk	3121	699	82%	3161	658	83%
Low-Fat Milk	2332	1488	61%	2440	1379	64%
Fruit Drinks	2866	954	75%	2923	896	77%
Fruit Juices	3555	265	93%	3584	235	94%
Bottled Water	2693	1127	70%	2681	1138	70%
Coffee	2812	1008	74%	2746	1073	72%
Tea	2753	1067	72%	2785	1034	73%

Source: Calculations made by the authors. Note: Market penetration is defined as the number of households that bought the beverage as a percentage of the total number of households in each sample. The total number of households in Sample A is 3820, and the total number of households in Sample B is 3819.

, we show the number of households that purchased or did not purchase a given non-alcoholic beverage, as well as the corresponding respective market penetration numbers. Isotonic drinks or sports drinks have the lowest market penetration at 20–22%, while fruit juices have the highest market penetration at 93–94%. The market penetrations for regular soft drinks, coffee and tea, high-fat milk, and low-fat milk are 90–91%, 72–74%, 82–83%, and 61–64%, respectively.

5. Conclusions, Limitations, and Future Work

Binary choice models are widely used in econometric modeling when the dependent variable corresponds to discrete outcomes. With appropriate decision rules, these models provide predictions of binary choices generated from predicted probabilities. The accuracy of these predictions in terms of classifying probabilities to events that occurred versus those that did not is a key issue. The use of expectation-prediction success at present is the standard method used to assess the accuracy of these predictions. However, this method is limited in its ability to correctly classify probabilities in the absence of appropriate predetermined cut-off levels. We propose alternative methods to classify probabilities generated through binary choice models, namely resolution graphs and resolution regressions that measure the ability to sort predicted probabilities against observed outcomes.

Using probabilities generated from the use of logit models applied to purchasing decisions of various non-alcoholic beverages, we compare probability sorting power using expectation-prediction success, resolution graphs, and resolution regression. Based on expectation-prediction success, the logit models performed better at classifying outcomes related to purchasing isotonic, regular soft drinks, diet drinks, bottled water, and tea. Based on resolution regressions, the null hypothesis of perfect sorting of probabilities was rejected for all non-alcoholic beverages. Although the logit models generated upward-sloping resolution graphs as expected, they were relatively flat compared to the 45-degree perfect sorting line. Bottom line, the use of resolution regression and resolution graphs yielded information not captured using prediction-success measures alone.

This additional evidence generated from resolution regressions and resolution graphs serves to complement the reporting of standard measures of prediction success. Going forward, for empirical analysis associated with binary choice models, we recommend that practitioners report the results from resolution regression and resolution graphs to capture sorting of probabilities in addition to the standard metrics used for the ability of the models to predict out-of-sample behavior.

This analysis was limited to purchase decisions made by U.S. households concerning ten non-alcoholic beverages for the calendar year 2003. Consequently, this analysis is tantamount to a case study. Since this study uses data from a representative sample of U.S. households for the calendar year 2003, this analysis does not capture the change in consumer behavior and market structure that has occurred in recent years. For future work, this study can be replicated using more current data, along with focusing on a myriad of other products purchased by households.