High-Dimensional Linear Preference Model

Abstract

Multiple-criteria decision analysis (MCDA) is a branch of operations research concerned with choices based on several quantifiable factors. Here, we focus on high-dimensional markets, in which each available product is described by a large number of features. Taking a probabilistic approach, we define a market as a collection of alternatives in a decision-making scenario governed by a linear utility function. Analytic approximations for the market share and its moments are derived in the limit of a large population and a large number of measured features. We identify a single parameter, termed the degree of subjectivity, that places markets on a continuous spectrum ranging from fully objective to fully subjective. At an intermediate value, the market is competitive in the sense that it maximizes the entropy of the market-share distribution. Empirical analysis of several real markets indicates that they can indeed be classified by this parameter, yielding predictable decision patterns and a unified, relative measure of competitiveness across markets. Simulations involving non-linear utility functions and a trained machine-learning classifier provide preliminary evidence that similar behavior may also arise beyond the linear model, suggesting that the degree of subjectivity may be useful as a diagnostic in some broader multi-feature decision problems.

1. Introduction

Decision theory is an active topic of research due to its direct ties with consumer-based economies, the job market, politics, and art. Studying the statistics of preference could act as a window into the dynamics of modern society, where the selection of clothes, media, activities, and careers is larger than ever [1,2]. In finance, a well-known axis of preference is the trade-off between risk and expected return, as investors need to decide how much risk they are willing to accept [3]. Decision theory is also relevant to classification methods in data science. In classification problems, decision trees or other machine learning (ML) methods are used to assign an object into one out of a given finite set of categories. One can think of the classifier as “deciding” which category to choose based on a list of features that describe the object.

To describe the multi-attribute utility model [4,5], consider a group of individuals, each asked to pick one alternative (e.g., product) out of several possible ones. Each individual makes their choice according to their own preferences. This may lead to different rankings (preference orders) of the alternatives. A common assumption in decision theory is that individuals weight the properties of products, which are summarized to a numerical score, called the utility function. Each individual then picks the alternative that provides them with the highest utility. Our main interest is in finding the market share of each alternative—the rate at which it is chosen. Intuitively, the market share should depend on the characteristics of the alternative itself, on the characteristics of other available alternatives, and on the distribution of preferences in the population.

A thorough introduction and examination of the multi-attribute utility model can be found in references [4,5]. A methodology for the practical application of discrete choice modeling can be found in reference [6], and examples for real applications are found in references [7,8]. Although the literature on discrete choice and multi-attribute utility is extensive (see [9,10,11]), much of it is concerned with the utility functions themselves, fitting the data, or about the process of making a choice, while we are more interested in the market statistics.

Another central approach in the decision theory literature comes from random utility theory, which assumes some unobserved choice preference [12,13]. In particular, the multinomial logit model [14] can predict choice in cases with taste heterogeneity using likelihood estimation methods [12,15]. For decision problems consisting of a high dimensional feature space, however, such methods may be impractical or more difficult to apply to real data. Information-theoretic models of bounded rationality present an approach to dealing with decision problems where information is not fully available by including entropy or information-processing costs as part of the problem itself [16,17].

Recent work has broadened random-utility and discrete-choice models through heterogeneous or unobserved choice sets and preferences [18], limited consideration [19], feature-dependent mixtures learned from incomplete preferences [20], and gradient-boosted utility functions that retain a random-utility interpretation [21]. These studies primarily address the identification, estimation, or prediction of heterogeneous choice mechanisms. In contrast, our focus is on the distribution of aggregate market shares induced by a given population of preferences in a many-feature limit.

Other decision-making frameworks represent uncertainty at the level of structured individual assessments: pure fuzzy-soft sets encode interactions among assessment parameters [22], whereas neutrosophic fuzzy-soft similarity methods retain membership, indeterminacy, and non-membership information [23].

Consider a market, here defined as a population of individuals who choose a single option out of a collection of available alternatives. Each alternative is described by a set of numerical features. The behavior arising from such markets seems to typically fall somewhere between fully objective and fully subjective. When the perceived utility of an alternative is not affected too much by the individual making the decision, we say that the choice is objective. This can happen if choices depend mostly on one factor, for example, when all products perform similarly, but one is cheaper. On the other end of the spectrum, when the utility varies significantly among individuals, we say that the choice is subjective. For example, choices related to politics or art are inherently subjective—beauty is in the eye of the beholder.

Between these two extremes lies what we refer to as a competitive market. A competitive market is characterized by giving alternatives equal opportunity. This is not necessarily translated into equal market share. Instead, in competitive markets, pairwise contests between random alternatives are maximally unpredictable.

The main goal of this paper is to show that, if the dimension is large enough, then under mild assumptions, the statistics of the market follow a limiting behavior that can be analyzed without having to consider specific model features. Simulations are used to generalize the analytical results. We also demonstrate the applicability of the main results to classification algorithms.

The paper is organized as follows. Section 2 defines an infinite-supply preference market along with a linear realization of the multi-attribute utility model and compares it with empirical data studies. We then simulate the model to get a basic understanding of its behavior. Some definitions and specifics are deferred to Section 4, where we derive analytic formulas for the expected market share and its variance in a random market. We highlight a key factor that naturally emerges from the structure of the model—a measure of ’objectiveness’ versus ’subjectiveness’ in choice behavior. Section 5 shows that a softmax choice rule can be integrated into the model with minimal effect on its statistics. Finally, Section 6 demonstrates that our results may be applicable to a larger class of decision problems, including non-linear classification methods. We provide an example from the classification of random symbols using a neural network. Some details and proofs are deferred to Appendix A and Appendix B. Appendix C explores the way alternatives separate the space of individual weights into segments. Appendix D presents initial results for deriving formulae for the distribution of market shares in a random market. Appendix E discusses some extensions into non-linear utility (see also [24]).

2. Linear Preference Model

Consider a market consisting of A alternatives. Each alternative is characterized by F numerical features. The population consists of I individuals, each of whom decides on one of the alternatives by maximizing a personalized linear utility function, which is a weighted combination of the F features [4,5]. For simplicity, we assume the following:

• Each alternative $a=1\dots A$ is characterized by various measurable features. The measurement vector of an alternative contains the measured values of all features for that alternative and is denoted as ${\overline{m}}_a=(m_a^1,\dots ,m_a^F)\in {\mathbb{R}}^F$.
• Each individual $i=1\dots I$ has a unique linear utility function, which assigns a real value to each alternative. Weight vectors are denoted as ${\overline{\omega }}_i=({\omega }_i^1,\dots ,{\omega }_i^F)\in {\mathbb{R}}^F$.
• Information about the alternatives is complete. Every individual picks from the same pool of alternatives, using the same measurements. All individuals see the same measurements of all features. In particular, there is no missing or hidden information.
• Individuals make their decision based only on the alternatives in the market, not based on choices of other individuals.
• The perceived value (utility) of alternative a for individual i is,

  $$v_{i,a}={\overline{\omega }}_i·{\overline{m}}_a={\displaystyle \sum _{f=1}^F}{\omega }_i^f{m}_a^f.$$

  (1)
• Each individual is assumed to choose the alternative with the highest value to them, i.e., ${arg\; max}_{a=1\dots A}{v}_{i,a}$.
• We assume independence across features and between weights and measurements. That is, for any $f\ne f^{\prime }$, ${\omega }_i^f$ and ${\omega }_i^{f^{\prime }}$ are independent, as well as $m_a^f$ and $m_a^{f^{\prime }}$, and any weight is independent of any measurement.

As an example, consider an individual seeking to purchase a vehicle and comparing multiple cars. Every car is described by various comparable numerical features such as price, speed, color, size, number of seats, running on gas/electricity, time between refueling/recharging, off-road capability, etc. Attributes such as price and gas consumption are relatively objective; however, a wealthy individual may assign a lower weight on costs, yet still care about the frequency of having to refuel. Other attributes may be subjective, as different individuals will look for different properties.

Due to the linearity of the model, certain situations are excluded from the model, such as those where the result is a non-linear mix of different components. In particular, while an individual may strongly favor two properties, they may not work well together, resulting in their combination being detrimental or causing diminishing returns. Such considerations cannot be modeled by a linear utility function. Additional linear model assumptions may not hold in some realistic scenarios [25]. For example, the assumption of full and coherent information on measurements, and on the existence of alternatives, is clearly non-trivial [26,27]. Recent heterogeneous-choice models allow the available or considered choice set itself to vary across individuals [18,19]. This source of variation is excluded from our base model and could otherwise be confounded with preference heterogeneity when only aggregate choices are observed. In some cases, such details can be taken into account using virtual measurements and weights. For example, adding an identity-specific feature that equals 1 for a specific alternative and 0 otherwise, with a large negative weight for an individual who does not know the alternative exists. Such approaches may lead to overfitting [28,29]. Finally, measurements and weights may be correlated. Some correlations may be reduced by principal component analysis.

The model is expected to be the most appropriate when choices aggregate many moderately important features, but may be less applicable if a single hard constraint dominates. We discuss this limitation further in Section 7.2.2.

2.1. Market Share

The market share of an alternative in a random market (i.e., random measurements and/or weights) is the average rate at which the alternative is chosen over the others. In other words, it is equal to the probability that a random individual i will pick alternative a out of all the available alternatives. There is an asymmetry within the structure of markets: when the number of individuals is large enough, due to the law of large numbers, adding more individuals does not change market shares as long as the underlying weight distributions remain the same. Adding new alternatives, on the other hand, adds competition to the existing alternatives, which reduces their total shares.

Often we will be interested in the market share of an alternative in a market where the rest of the alternatives are unknown. In such a case, the market share of alternative a can be defined as a random variable that, in the limit of $I\to \infty$, approaches the probability

$${\mathbb{T}}_a({\overline{m}}_a,{\mathit{M}}_a)=P_{{\overline{\omega }}_i}(v_{i,a}>v_{i,a^{\prime }},\forall a^{\prime }\ne a|{\overline{m}}_a).$$

(2)

Here, ${\mathit{M}}_a$ is a matrix in which the columns are the measurement vectors from the market, aside from ${\overline{m}}_a$. In this notation, we think of ${\overline{m}}_a$ as a fixed vector, i.e., the measurements of alternative a are known, but ${\mathit{M}}_a$ as random, i.e., the other measurements are random.

2.2. Market Types

To quantify the distinction between objective and subjective markets, we compare two components of the population’s preferences. The first is the shared component: the average direction in feature space toward which individuals tend to agree. It is given by the mean of a random weight vector, denoted as ${\overline{\mu }}_W$. The second is the heterogeneity component: the amount by which individual preferences vary around this average direction, and is given by the standard deviation of a weight vector, denoted as ${\overline{\sigma }}_W$.

Definition 1.

Degree of subjectivity. The market’s degree of subjectivity is defined as $\eta :=\Vert {\overline{\sigma }}_W\Vert /\Vert {\overline{\mu }}_W\Vert$. It ranges between zero for a fully objective market and infinity for a fully subjective market. It serves as a multivariate analogue to a coefficient of variation.

Intuitively, $\Vert {\overline{\sigma }}_W\Vert \ll \Vert {\overline{\mu }}_W\Vert$ implies that there are a few features that dominate the selection, and that variance in preference is limited. Here, individuals mostly look for the same thing in an alternative, i.e., the market is objective. On the other hand, if $\Vert {\overline{\sigma }}_W\Vert \gg \Vert {\overline{\mu }}_W\Vert$, the variance is large compared to the mean, and it is expected that every individual will choose differently. Accordingly, we call the market subjective. The special case $\eta =1$ will be shown to correspond to a competitive market, in which pairwise contests between random alternatives are maximally unpredictable. The surprising result is that as $I,F\to \infty$, the characterization of markets depends only on $\eta$.

To illustrate the meaning of the degree of subjectivity, consider the following analogy (equivalently, one may regard the varying daily conditions as playing the role of different realizations of individual preference). We come across a local chess club and learn that upsets do not happen frequently relative to the players’ Elo scores. If we pick two players at random to play many games, we can expect one of the pair to win most matches. This club corresponds to an objective market. In another club, a different chess opening is imposed each day, and different players specialize in different openings. If we pick two competitors at random, then on any given day one player may dominate their matches, but which player dominates depends strongly on the imposed opening. Across many randomly selected days, neither player has a systematic advantage. Such a club corresponds to a subjective market. Now consider a third club, somewhere between those two extremes. After picking two competitors at random, we observe their matches against one another over many randomly selected days. In this club, however, the eventual split of wins is maximally unpredictable. The proportion of games won by either player is uniformly distributed between 0 and 1 (for simplicity of the analogy, we ignore draws). Such a club reflects the maximal-entropy condition of a competitive market.

Definition 2.

Comparison distribution. Consider a simple market with only two alternatives, $A=2$. We refer to the distribution of the market share as a comparison distribution. As we will show later in the paper, this distribution reaches a stable limit as $F\to \infty$ of the form

$$P({\mathbb{T}}_a<t)=\Phi \left(\eta {\Phi }^{-1}\left(t\right)\right),$$

(3)

where Φ is the cumulative distribution function (CDF) of the standard normal distribution.

Definition 3.

Comparison entropy. The comparison entropy of a market with $A=2$ is defined as the differential entropy of its comparison distribution.

$$H=-\mathbb{E}[log{c}_{\eta }\left({\mathbb{T}}_1\right)],$$

(4)

where $c_{\eta }$ is the probability density function (PDF) of the comparison distribution for a market with degree of subjectivity η. Note that the comparison distribution is symmetric around $1/2$, and there is no difference between choosing ${\mathbb{T}}_1$ or ${\mathbb{T}}_2$ here.

Definition 4.

Competitive market. A market is said to be competitive when its comparison entropy is maximal, which implies maximal uncertainty when comparing two alternatives at random.

In Section 4.6, we prove Equation (3) and show that in the limit $I,F\to \infty$, the comparison entropy depends only on $\eta$, i.e., the degree of subjectivity of the market. Moreover, a market is competitive exactly when $\eta =1$. In this paper, we do not make any claims about the connection between competitiveness in a preference market and existing measures of competitiveness in economic theory. It is not clear how, or in what capacity, being competitive affects a preference market—for example, if time dynamics or pricing were introduced to the model. We leave the reader only with the maximal-entropy condition as intuition behind the name.

Note that the definition for comparison distribution can be generalized to $A>2$ by averaging the comparison distributions of all alternative pairs, thus enabling the same empirical analysis for larger markets.

3. Empirical Data and Simulations

In this section, we compare empirical data with simulations of the linear preference model. We do this in three ways. First, we demonstrate the behavior of simulated markets based on varying parameters, as well as different distribution types used for weights and measurements. Turning to real markets, we compare simulations of the linear model with example empirical market share data. Lastly, we look at empirical comparison distributions generated from full-ranking information, which is richer than knowing only the individual’s best alternative. The empirical results are summarized and sorted by comparison entropy in Table 1. Our goal is to demonstrate the applicability of the degree of subjectivity parameter $\eta$ in describing the full market share distribution in real markets.

Table 1. Summary of empirical preference data, sorted by comparison entropy.

Preference Market/Source of Data	$\mathit{\eta }$	Comparison Entropy	A	I
French approval ballots experiment	$1.26$	$-0.046$	16	409
Israeli city population	$1.3$	$-0.058$	100	9,662,037
Minneapolis mayoral elections	$1.56$	$-0.150$	35	80,101
Spotify top charts by country	$0.65$	$-0.253$	21	45
Jester joke set, high variance	$2.22$	$-0.399$	10	59,132
Israeli legislative elections	$2.3$	$-0.427$	13	4,690,442

Simulating the preference decision process provides an intuitive understanding of the model’s output and its implications. By generating random weights, $W\in {\mathbb{R}}^{F\times I}$, and measurements, $M\in {\mathbb{R}}^{F\times A}$, the product $W^{T}M$ determines the empirical market share ${\mathbb{T}}_a(W,M)$, which can be used as an approximation for the non-empirical version when using a large value for I.

Figure 1 shows the rank-frequency graphs (which are also the empirical distributions of market share) of preference markets generated via different types of distributions. As shown below in Section 4.2, we can set ${\mu }_M^f=0$ and ${\sigma }_M^f=1$ for all f. For simplicity, we use the same distribution for all weights and the same distribution for all measurements. The numerical results demonstrate the applicability of a central limit argument, establishing that when the number of features F is large, the market share distribution does not depend on the particular choice of distribution. Accordingly, for the rest of the paper simulations we will be using normal distributions for both weights and measurements.

Figure 1. Simulated random markets with uniform, normal, and log-normal distributions for both weights and measurements. (Left): a rank-frequency distribution corresponding to markets generated by a different distribution, with identical means and variances. (Right): the same data on a log–log scale.

Next, we analyze the distinction between objective and subjective decision problems within the model. A subjective problem can be thought of as one in which, for each individual who likes an aspect or element, there exists another individual who dislikes it as much. In terms of the model, we set ${\mu }_W^f=0$ for each f. Figure 2 shows the market share rank-frequency distribution that emerges in subjective and competitive markets, with varying market sizes. In subjective markets, the structure of rank-frequency distributions changes as the number of alternatives decreases, forcing individuals to select less preferred options. Conversely, when ${\mu }_W^f\gg {\sigma }_W^f$ for all f, individual preferences converge to a shared value function ${\sum }_{f=1}^F{\mu }_W^f{m}_a^f$ which would give rise to a single alternative getting the lion’s share of the market. In between these two extremes, we get a mixed decision problem with varying degrees of subjectivity. In contrast to subjective markets, markets that are more objective appear to retain their characteristic shape, even as the number of available alternatives decreases. When $\eta =\Vert {\overline{\sigma }}_W\Vert /\Vert {\overline{\mu }}_W\Vert =1$, the market is competitive, which means the comparison entropy is maximized (see Section 4.6).

Figure 2. Rank-frequency plots for preference markets generated by simulation. The (first row) shows small markets at size $A=30$, while the (second row) shows larger markets with $A=1000$, showing only the top alternatives. (Left column): subjective markets, generated by setting $\eta =\infty$. (Right column): competitive markets, generated by setting $\eta =1$. Smaller subjective markets have a distinct curvy shape, quite unlike their more objective counterparts.

Finally, we turn to study empirical examples of real markets. Finding static preference markets is challenging, as many real markets undergo certain dynamics that could influence the distribution of the measurement vectors in share-aware ways, which may not be possible to account for by a change in $\eta$ alone.

Figure 3 shows comparisons of the rank frequency plots generated from population and election data in Israel, with the rank frequency plots of randomly generated preference markets using different degrees of subjectivity for each, fitted to the data by ad-hoc means of generating random markets using various values of $\eta$.

Figure 3. (Left): Israeli 2022 legislative election results (election data taken from https://votes25.bechirot.gov.il/ (accessed on 5 September 2025)) (blue), compared to a simulated random market with $\eta =2.3$ (red). Higher $\eta$ indicates a more subjective market. (Right): Israeli city population data (Population data taken from the Israeli Central Bureau of Statistics at https://www.cbs.gov.il/en (accessed on 5 September 2025)) from 2022 (blue, top 100 shown only), compared to a simulated random market with $\eta =1.3$ (red).

Another source of data, which is more informative and sometimes easier to collect, comes from pairwise comparison polls. Figure 4 shows real data collected from online polls of various origins, alongside the approximate market share distribution in Equation (3). Higher values of $\eta$ imply more subjective markets, while lower values are tied to objective markets.

Figure 4. Empirical comparison distributions formed by counting ratios of population splits in pairwise comparisons. The values of $\eta$ were fit using a standard minimum Kolmogorov–Smirnov estimator. (A) Pairwise comparisons of joke rankings, including negative values. Part of the Jester data set [30] (https://goldberg.berkeley.edu/jester-data/ (accessed on 5 September 2025)). (B) Minneapolis mayoral elections from 2013, where a total of 35 candidates were voted for by ranked choice, (https://dataverse.harvard.edu/file.xhtml?fileId=11593264&version=25.0 (accessed on 5 September 2025)). (C) Elections generated by querying the 200 most listened to songs on Spotify, using their particular ratings in individual countries as a proxy for individual choosers (Preflib [31] dataset number 00047). (D) An experiment [32] conducted during the French presidential election of 2002, where voters engaged in approval voting could mark any number of candidates as a binary yes/no. For the purpose of comparison distribution, only cases where an individual marked one candidate as yes and another as no were taken into account (Preflib dataset number 00026).

4. Analytical Results

In this section, we derive the main analytical results. In particular, we provide a formula for the expected market share of an alternative in a random market.

4.1. Definitions

We start by properly defining the objects of study: preference markets and market shares.

Definition 5.

Random market. We define a market as a pair $(\mathit{W},\mathit{M})$, where $\mathit{W}\in {\mathbb{R}}^{F\times I}$ and $\mathit{M}\in {\mathbb{R}}^{F\times A}$ are random matrices containing all weights and measurements, respectively:

$$\mathit{W}=\left(\begin{array}{ccc}|& & |\\ {\overline{\omega }}_1& \dots & {\overline{\omega }}_I\\ |& & |\end{array}\right),\mathit{M}=\left(\begin{array}{ccc}|& & |\\ {\overline{m}}_1& \dots & {\overline{m}}_A\\ |& & |\end{array}\right).$$

(5)

The distribution of weight vectors is denoted as ${\mathcal{D}}_W$, and we assume that the individual weights are independent and identically distributed (IID). The marginal distributions are denoted ${\omega }_i^f\sim {\mathcal{D}}_W^f$, having finite means and standard deviations ${\mu }_W^f$ and ${\sigma }_W^f$, respectively, for each f. Note that weights may be negative and do not necessarily sum to 1. The distribution of measurements is denoted by ${\mathcal{D}}_M$, and we assume that the measurement vectors of different alternatives are IID. The marginal distributions are denoted by $m_a^f\sim {\mathcal{D}}_M^f$, having finite means and standard deviations ${\mu }_M^f$ and ${\sigma }_M^f$, respectively, for each f.

Furthermore, we denote the domains of ${\mathcal{D}}_W$ and ${\mathcal{D}}_M$ by ${\Omega }_W$ and ${\Omega }_M$, respectively. The distributions are both absolutely continuous with respect to the Lebesgue measures $dP\left({\overline{\omega }}_i\right)$ and $dP\left({\overline{m}}_a\right)$, respectively. Notice that these are distributions for a single vector, not a matrix (because they do not depend on i and a).

In addition to finite second moments, our high-dimensional market share approximations will require assuming some further mild conditions on higher moments, which roughly translate to ensuring that no single feature is solely responsible for a large portion of the decisions made. The model itself is defined regardless of these additional conditions, though it may appear more fruitful when they hold. For specifics, see Theorems A3 and A4, as well as the discussion in Section 7.2.2.

Definition 6.

Realized market. A realized market is a realization of $\mathit{M}$ alone, but not $\mathit{W}$, i.e., $\mathit{W}$ is still random. In other words, we know all the products in the market and their features. However, we do not know the particular individuals making their choices, only their statistics. We denote a realized market by M.

Below, we will also consider markets in which only one alternative, a, is realized, but not the rest of them (i.e., the measurements of all other alternatives are still random). For that, we may use a minor matrix, ${\mathit{M}}_a$, which has alternative a omitted,

$${\mathit{M}}_a:=\left(\begin{array}{cccccc}|& & |& |& & |\\ {\overline{m}}_1& \dots & {\overline{m}}_{a-1}& {\overline{m}}_{a+1}& \dots & {\overline{m}}_A\\ |& & |& |& & |\end{array}\right).$$

(6)

Definition 7.

Choice indicator. The choice indicator is a random variable defined as

$$c_{i,a}(W,M)=\left\{\begin{array}{cc}1\hfill & v_{i,a}>v_{i,a^{\prime }}\forall a^{\prime }\ne a\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(7)

In other words, $c_{i,a}=1$ if alternative a is the best choice for individual i. The choice indicator is dependent both on individual i’s weights, ${\overline{\omega }}_i$, and on the measurements of the entire market $\mathit{M}$. We assume that ${\sum }_{a=1}^A{c}_{i,a}=1$ with probability 1, meaning there are no ties. This limits us to markets with at least one continuous measurement, but we may still apply the model if the probability of a tie is negligible.

Definition 8.

Market Share. The market share of alternative a in a random market $(\mathit{W},\mathit{M})$ is the average rate at which the alternative is chosen over the others. In other words, it is equal to the probability that a random individual i will pick a out of all alternatives.

By realizing the weights as well as the measurements $\mathit{M}=M,\mathit{W}=W$, we can define the empirical market share of alternative a as

$${\mathbb{T}}_a(W,M):={\displaystyle \frac{1}{I}}{\displaystyle \sum _{i=1}^I}{c}_{i,a}(W,M).$$

(8)

As the number of individuals I becomes large, the law of large numbers (LLN) applies, and using realized weights as opposed to a random variable becomes unnecessary. The (non-empirical) market share of alternative a is then provided by

$${\mathbb{T}}_a\left(M\right):={\mathbb{E}}_{\mathit{W}}\left[{\mathbb{T}}_a(W,M)\right]=\underset{I\to \infty }{lim}{\displaystyle \frac{1}{I}}{\displaystyle \sum _{i=1}^I}{c}_{i,a}=P_{{\overline{\omega }}_i}(v_{i,a}>v_{i,a^{\prime }},\forall a^{\prime }\ne a|M).$$

(9)

Here, $P_X(·)$ and ${\mathbb{E}}_X[·]$ denote the probability and expectation with respect to a random variable or vector X, respectively. Notice that in (9), the left-hand side (LHS) does not depend on i, while the right-hand side (RHS) does. This is because ${\overline{\omega }}_i$ are IID.

There is an asymmetry within the structure of markets: when the number of individuals is large enough, adding more individuals (and, thus, weight vectors) does not change market shares, as long as the underlying weight distributions remain the same. Adding new alternatives, on the other hand (and, therefore, new measurement vectors), adds competition to existing alternatives.

For a random market, the market share of a single realized alternative a in an otherwise random market is a random variable dependent on ${\mathit{M}}_a$,

$${\mathbb{T}}_a({\overline{m}}_a,{\mathit{M}}_a)=P_{{\overline{\omega }}_i}(v_{i,a}>v_{i,a^{\prime }},\forall a^{\prime }\ne a|{\overline{m}}_a).$$

(10)

In this notation, we think of ${\overline{m}}_a$ as a fixed vector, i.e., the measurements of alternative a are known, but ${\mathit{M}}_a$ is random, i.e., the other measurements are random.

4.2. Standardizing the Model Parameters

In order to reduce the degrees of freedom when defining a random market, we define and single out the preference order, which solely determines the market share.

Definition 9.

Preference order. The preference order of a realized market $(\mathit{W},\mathit{M})$ is a distribution over permutation of length A, given by ${\pi }_{\sigma }=P_{{\overline{\omega }}_i}(v_{i,{\sigma }_1}>v_{i,{\sigma }_2}>\cdots >v_{i,{\sigma }_A})$ for every permutation $\sigma \in S_A$.

Remark 1.

The marginal distributions, and in turn the preference order, fully determine market shares. Any change in the market that leaves the preference order unchanged will also leave market shares unchanged. This allows us to derive a standardized form for a market.

There are certain transformations one can apply to both weight and measurement vectors without changing the market’s preference order. For example, by multiplying either each weight ${\omega }_i^f$ or each measurement $m_a^f$ by a constant $C>0$, or by multiplying all weights of a specific feature ${\omega }_i^f$ by a constant and dividing each corresponding measurement $m_a^f$ by the same constant.

An interesting asymmetry arises when one tries to add a constant value to a specific feature’s weight or measure:

$$v_{i,a}^{\prime }={\displaystyle \sum _{f=1}^F}({\omega }_i^f+C)m_a^f=v_{i,a}+C{\displaystyle \sum _{f=1}^F}{m}_a^f,$$

(11)

$$v_{i,a}^{\prime }={\displaystyle \sum _{f=1}^F}{\omega }_i^f(m_a^f+C)=v_{i,a}+C{\displaystyle \sum _{f=1}^F}{\omega }_i^f.$$

(12)

Since the first case depends on the alternative’s measurements ${\overline{m}}_a$, it could alter the preference order because it affects each alternative differently. The second case, however, depends only on the individual and therefore adds the same bias to the utility of all alternatives. Therefore, it does not affect the preference order.

Using these degrees of freedom, we can shift the distributions ${\mathcal{D}}_W^f$ and ${\mathcal{D}}_M^f$ such that ${\mu }_M^f=\mathbb{E}[m_a^f]=0$ and ${{\sigma }_M^f}^2=\mathbb{E}[{(m_a^f)}^2]=1$. This leaves us with one other degree of freedom, which we can use to normalize each weight vector. While this does not affect the preference order, it results in some loss of information on the relative contribution of features. Since, in this paper, we are mostly interested in the market share, unless stated explicitly, we will be working with normalized weights, $\Vert {\overline{\omega }}_i\Vert =1$.

Due to the parameter standardization, we can write

$$\Vert {\overline{\mu }}_W\Vert ={\displaystyle \frac{1}{\sqrt{1+{\eta }^2}}},\Vert {\overline{\sigma }}_W\Vert ={\displaystyle \frac{\eta }{\sqrt{1+{\eta }^2}}}.$$

(13)

4.3. Expected Market Share

We proceed to calculate the expected market share of a specific alternative in a random market.

Recall that the random variable ${\mathit{M}}_a$ is a minor of the measurement matrix $\mathit{M}$ created by omitting the column containing ${\overline{m}}_a$. As in Equation (10), we have

$${\mathbb{T}}_a({\overline{m}}_a,{\mathit{M}}_a)=P_{{\overline{\omega }}_i}(v_{i,a}>v_{i,a^{\prime }}\forall a^{\prime }\ne a|M).$$

(14)

If we condition over ${\overline{m}}_a$ and take expectation over ${\mathit{M}}_a$, we are left with a random variable over ${\overline{\omega }}_i$:

$$\begin{array}{cc}\hfill {\mathbb{E}}_{{\mathit{M}}_a}\left[{\mathbb{T}}_a\right|{\overline{m}}_a]& =\underset{{\Omega }_M^{A-1}}{\int }{P}_{{\overline{\omega }}_i}({\overline{\omega }}_i·{\overline{m}}_a>{\overline{\omega }}_i·{\overline{m}}_{a^{\prime }}\forall a^{\prime }\ne a|M)dP\left({\mathit{M}}_a\right)\hfill \\ \hfill & =\underset{{\Omega }_M^{A-1}}{\int }\underset{{\Omega }_W}{\int }{c}_{i,a}dP\left({\overline{\omega }}_i\right)dP\left({\mathit{M}}_a\right).\hfill \end{array}$$

(15)

Since the integrand is bounded, we can switch the order of integration as follows:

$$\underset{{\Omega }_W}{\int }\underset{{\Omega }_M^{A-1}}{\int }{c}_{i,a}dP\left({\mathit{M}}_a\right)dP\left({\overline{\omega }}_i\right)=\underset{{\Omega }_W}{\int }{P}_{{\mathit{M}}_a}({\overline{\omega }}_i·{\overline{m}}_a>{\overline{\omega }}_i·{\overline{m}}_{a^{\prime }}\forall a^{\prime }\ne a|{\overline{\omega }}_i,{\overline{m}}_a)dP\left({\overline{\omega }}_i\right).$$

(16)

Since ${\overline{m}}_{a^{\prime }}$ are independent, this expression can be simplified to

$$\underset{{\Omega }_W}{\int }{P}_{{\overline{m}}_{a^{\prime }}}{({\overline{\omega }}_i·{\overline{m}}_a>{\overline{\omega }}_i·{\overline{m}}_{a^{\prime }}|{\overline{\omega }}_i,{\overline{m}}_a)}^{A-1}dP\left({\overline{\omega }}_i\right).$$

(17)

Inside the parentheses we have F random variables ${\omega }_i^1{m}_{a^{\prime }}^1\dots {\omega }_i^F{m}_{a^{\prime }}^F$, each with mean 0 and variance ${({\omega }_i^f)}^2<\infty$. Under some mild conditions, the sum converges to a normal random variable. However, because we are looking to integrate over ${\Omega }_W$, the error might diminish impractically slowly. In order to bound this error, we need to show uniform convergence of ${\overline{\omega }}_i·{\overline{m}}_{a^{\prime }}$ over ${\Omega }_W$, which we can achieve by employing the Berry–Esseen theorem. The full statement and proof can be found in Appendix B; here, we only state the following version, which we need in order to derive the expected value.

Theorem 1.

Assume that ${\sum }_{f=1}^F\mathbb{E}\left[\right|{\omega }_i^f{|}^3]·\mathbb{E}[|m_{a^{\prime }}^f{|}^3]\underset{F\to \infty }{\to }0$ and ${\sum }_{f=1}^F\mathbb{E}[{({\omega }_i^f)}^4]\underset{F\to \infty }{\to }0$. Then, we have

$$\underset{F\to \infty }{lim}\left|{\int }_{{\Omega }_W}{P}_{{\overline{m}}_{a^{\prime }}}{\left({\overline{\omega }}_i·{\overline{m}}_{a^{\prime }}<{\overline{\omega }}_i·{\overline{m}}_a|{\overline{\omega }}_i\right)}^{A-1}dP\left({\overline{\omega }}_i\right)-{\int }_{{\Omega }_W}\Phi {\left({\overline{\omega }}_i·{\overline{m}}_a\right)}^{A-1}dP\left({\overline{\omega }}_i\right)\right|=0,$$

(18)

where Φ denotes the cumulative distribution function (CDF) of the standard normal distribution.

We now obtain

$$\underset{F\to \infty }{lim}\underset{{\Omega }_W}{\int }{P}_{{\overline{m}}_{a^{\prime }}}{({\overline{\omega }}_i·{\overline{m}}_a>{\overline{\omega }}_i·{\overline{m}}_{a^{\prime }}|{\overline{\omega }}_i,{\overline{m}}_a)}^{A-1}dP\left({\overline{\omega }}_i\right)=\underset{{\Omega }_W}{\int }\Phi {\left({\overline{\omega }}_i·{\overline{m}}_a\right)}^{A-1}dP\left({\overline{\omega }}_i\right).$$

(19)

We observe that the element within the CDF is yet another random vector dot product:

$${\overline{\omega }}_i·{\overline{m}}_a\underset{F\to \infty }{\stackrel{\mathrm{d}}{\to }}N({\overline{\mu }}_W·{\overline{m}}_a,{\overline{\sigma }}_W^2·{\overline{m}}_a^2),$$

(20)

where ${\overline{\sigma }}_W^2$ and ${\overline{m}}_a^2$ are provided by element-wise squaring of the original vector components. A proof of convergence in distribution can be found in Theorem A4. Assuming that $\sqrt{{\overline{\sigma }}_W^2·{\overline{m}}_a^2}$ does not vanish, we can change variables to obtain

$${\mathbb{E}}_{{\mathit{M}}_a}\left[{\mathbb{T}}_a\right|{\overline{m}}_a]\approx {\displaystyle \frac{1}{\sqrt{{\overline{\sigma }}_W^2·{\overline{m}}_a^2}}}{\displaystyle \underset{-\infty }{\overset{\infty }{\int }}}\Phi {\left(x\right)}^{A-1}\varphi \left({\displaystyle \frac{x-{\overline{\mu }}_W·{\overline{m}}_a}{\sqrt{{\overline{\sigma }}_W^2·{\overline{m}}_a^2}}}\right)dx.$$

(21)

We leave to Section 7.2.2 the discussion of the practical implications of the moment assumptions we had to take for Theorem 1. Equation (A8) also provides a finite-dimensional diagnostic for the first Gaussian approximation. In our notation, its error is controlled by the term ${\sum }_{f=1}^F\mathbb{E}\left[\right|{\omega }_i^f{|}^3\left]\mathbb{E}\right[|m_{a^{\prime }}^f{|}^3]$, together with the concentration of the weight-vector norm established in the proof. For moderate values of F, the error therefore depends on how quickly this term tends to zero. Figure 5 shows a comparison between the estimate and simulations. Here, we see our first motivation for normalizing the weights when we are only interested in the market share. Setting $\Vert {\overline{\omega }}_i\Vert =1$ does not change the preference order, and makes the use of A4 and A3 cleaner.

Figure 5. Comparison of the derived integral estimate of the expected market share and simulated data using normal distributions (top row) or log-normal distributions (lower row) for weights and measurements. (Left) Subjective market with $\eta =\infty$. (Right) Competitive market with $\eta =1$. In both cases, we set $F=30,m_a^f=2/\sqrt{F}$. In the log-normal case, the heavier tail in the distribution can cause the approximation to converge much slower.

4.4. Market Share Variance

We next consider the L’th moment of ${\mathbb{T}}_a({\overline{m}}_a,{\mathit{M}}_a)$.

$$\begin{array}{cc}\hfill {\mathbb{E}}_{{\mathit{M}}_a}\left[{\mathbb{T}}_a^L\right|{\overline{m}}_a]& =\underset{{\Omega }_M^{A-1}}{\int }{P}_{{\overline{\omega }}_i}{({\overline{\omega }}_i·{\overline{m}}_a>{\overline{\omega }}_i·{\overline{m}}_{a^{\prime }}\forall a^{\prime }\ne a|M)}^{L}dP\left({\mathit{M}}_a\right)\hfill \\ \hfill & =\underset{{\Omega }_M^{A-1}}{\int }{\left(\underset{{\Omega }_W}{\int }{c}_{i,a}dP\left({\overline{\omega }}_i\right)\right)}^{L}dP\left({\mathit{M}}_a\right).\hfill \end{array}$$

(22)

In order to switch the order of integration, we expand the power, each with an independent random weight vector:

$$\underset{{\Omega }_W^L}{\int }\underset{{\Omega }_M^{A-1}}{\int }{\displaystyle \prod _{l=1}^L}{\mathbb{1}}_{\{{\overline{\omega }}_l·{\overline{m}}_a>{\overline{\omega }}_l·{\overline{m}}_{a^{\prime }}\forall a^{\prime }\ne a\}}({\overline{m}}_a,M_a)dP\left({\mathit{M}}_a\right){\displaystyle \prod _{l=1}^L}dP\left({\overline{\omega }}_l\right).$$

(23)

Each inner product ${\overline{\omega }}_l·{\overline{m}}_{a^{\prime }}$ is again approximately normally distributed. However, unlike the $L=1$ case, they are now dependent. As a result, we need to consider them together as a multivariate normal distribution. By replacing the order of integration and integrating over $dP\left({\mathit{M}}_a\right)$, we obtain

$$\underset{{\Omega }_W^L}{\int }{P}_{\overline{m}}{({\overline{\omega }}_l·{\overline{m}}_a>{\overline{\omega }}_l·\overline{m},\forall 1\le l\le L|{\overline{m}}_a)}^{A-1}dP\left(\mathit{W}\right),$$

(24)

where

$$\mathit{W}=\left(\begin{array}{ccc}|& & |\\ {\overline{\omega }}_1& \dots & {\overline{\omega }}_L\\ |& & |\end{array}\right)\in {\mathbb{R}}^{F\times L}.$$

(25)

Define $x_l:={\overline{\omega }}_l·(\overline{m}-{\overline{m}}_a),\overline{x}=(x_1,\dots ,x_L)$. Recall that $\mathbb{E}[m_a^f]=0$ for all f. As $F\to \infty$, $\overline{x}\stackrel{\mathrm{d}}{\to }\mathcal{N}(-{\mathit{W}}^T{\overline{m}}_a,{\mathit{W}}^T\mathit{W})$. We can now write the L’th moment as

$$\underset{{\Omega }_W^L}{\int }{\left(\underset{{\mathbb{R}}_{-}^L}{\int }{\displaystyle \frac{exp(-\frac{1}{2}{(\overline{x}+{\mathit{W}}^T{\overline{m}}_a)}^T{\left({\mathit{W}}^T\mathit{W}\right)}^{-1}(\overline{x}+{\mathit{W}}^T{\overline{m}}_a))}{\sqrt{{\left(2\pi \right)}^L\left|{\mathit{W}}^T\mathit{W}\right|}}}d\overline{x}\right)}^{A-1}dP\left(\mathit{W}\right).$$

(26)

This integral is difficult to evaluate in general. We proceed to calculate the second moment of the market share, which encompasses information about the risk of inserting a known alternative into an unknown market. Observe that, given mild conditions, each product ${\overline{\omega }}_l·{\overline{\omega }}_{l^{\prime }}$ is normal as well. Particularly, for the second moment, $L=2$, we have

$$\left(\begin{array}{c}{z}_1\\ z_2\\ z_3\end{array}\right):=\left(\begin{array}{c}{\overline{\omega }}_1·{\overline{m}}_a\\ {\overline{\omega }}_2·{\overline{m}}_a\\ {\overline{\omega }}_1·{\overline{\omega }}_2\end{array}\right)\sim \mathcal{N}\left({\overline{\mu }}_2,{\Sigma }_2\right),$$

(27)

$${\overline{\mu }}_2=\left(\begin{array}{c}{\overline{\mu }}_W·{\overline{m}}_a\\ {\overline{\mu }}_W·{\overline{m}}_a\\ {\Vert {\overline{\mu }}_W\Vert }^2\end{array}\right),{\Sigma }_2=\left(\begin{array}{ccc}{\overline{\sigma }}_W^2·{\overline{m}}_a^2& 0& {\overline{\mu }}_W·({\overline{m}}_a\odot {\overline{\sigma }}_W^2)\\ 0& {\overline{\sigma }}_W^2·{\overline{m}}_a^2& {\overline{\mu }}_W·({\overline{m}}_a\odot {\overline{\sigma }}_W^2)\\ {\overline{\mu }}_W·({\overline{m}}_a\odot {\overline{\sigma }}_W^2)& {\overline{\mu }}_W·({\overline{m}}_a\odot {\overline{\sigma }}_W^2)& {\Vert {\overline{\sigma }}_W^2\Vert }^2+2{\overline{\mu }}_W^2·{\overline{\sigma }}_W^2\end{array}\right),$$

(28)

where ⊙ denotes element-wise vector product. By normalizing the weights, we obtain

$${\mathit{W}}^T\mathit{W}=\left(\begin{array}{cc}1& z_3\\ z_3& 1\end{array}\right).$$

(29)

Substituting $\overline{u}=\overline{x}+{\mathit{W}}^T{\overline{m}}_a$, the inner integral in (26) becomes

$${\Phi }_2(z_1,z_2,z_3):={\int }_{-\infty }^{z_1}{\int }_{-\infty }^{z_2}{\displaystyle \frac{exp(-\frac{1}{2}{\overline{u}}^T{\left({\mathit{W}}^T\mathit{W}\right)}^{-1}\overline{u})}{2\pi \sqrt{det\left({\mathit{W}}^T\mathit{W}\right)}}}d\overline{u}.$$

(30)

In other words, ${\Phi }_2(z_1,z_2,z_3)$ denotes the multivariate integral in two standard normal variables with a correlation coefficient of $z_3$, evaluated with limits as seen in Equation (30). Finally, we use the multivariate PDF for $\overline{z}$:

$$\mathbb{E}\left[{\mathbb{T}}_a^2\right|{\overline{m}}_a]=\underset{{\mathbb{R}}^2\times (-1,1)}{\int }{\displaystyle \frac{exp(-\frac{1}{2}{(\overline{z}-{\overline{\mu }}_2)}^T{\Sigma }_2^{-1}(\overline{z}-{\overline{\mu }}_2))}{\sqrt{{\left(2\pi \right)}^{3}det\left({\Sigma }_2\right)}}}{\Phi }_2{(z_1,z_2,z_3)}^{A-1}d\overline{z},$$

(31)

with ${\overline{\mu }}_2,{\Sigma }_2$ presented above in (28).

4.5. Conditioned Market Share Expectation

In this section, we derive estimates for the expected market share of alternative a in a random market, when the alternative is conditioned in a way that gives us some partial information about its measurement vector.

For the rest of the section, we will be working with normalized weights and applying Theorem A3 to obtain ${\overline{\omega }}_i·{\overline{m}}_a|{\overline{\omega }}_i\sim N(0,{\Vert {\overline{\omega }}_i\Vert }^2)=N(0,1)$. Note that, for $a\ne a^{\prime }$, we have $\mathrm{cov}({\overline{\omega }}_i·{\overline{m}}_a,{\overline{\omega }}_i·{\overline{m}}_{a^{\prime }}|{\overline{\omega }}_i)=0$. This implies that we can treat $v_{i,a}$ and each $v_{i,a^{\prime }}$ as independent normal variables inside the integral. We begin with a simple sanity check by deriving the mean market share ${\mathbb{E}}_{\mathit{M}}\left[{\mathbb{T}}_a\right]$, which has to equal $1/A$ due to symmetry:

$${\mathbb{E}}_{\mathit{M}}\left[{\mathbb{T}}_a\right]=\underset{{\Omega }_W}{\int }{P}_{\mathit{M}}(v_{i,a}>v_{i,a^{\prime }},\forall a^{\prime }\ne a|{\overline{\omega }}_i)dP\left({\overline{\omega }}_i\right)\approx {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}}\Phi {\left(x\right)}^{A-1}\varphi \left(x\right)dx={\displaystyle \frac{1}{A}}.$$

(32)

Our strategy henceforth will be to impose some restriction on ${\overline{m}}_a$ in the form of it belonging to an event R that has a non-zero measure, such that, given ${\overline{\omega }}_i$ and R, $v_{i,a}$ is normally distributed. Since ${\mathrm{cov}}_M({\overline{\omega }}_i·{\overline{m}}_a,{\overline{\omega }}_i·{\overline{m}}_{a^{\prime }}|{\overline{\omega }}_i,R)=0$, we can solve for the market share expectation ${\mathbb{E}}_{{\mathit{M}}_a}\left[{\mathbb{T}}_a\right|{\overline{m}}_a\in R]$ as we did in Section 4.3. As in Theorem A4, some requirements on the moments are needed to ensure uniform convergence over ${\Omega }_W$.

4.5.1. Conditioning on the Norm of the Measurement Vector

Here, we investigate what we can tell about the market share of an alternative, knowing only the norm of its measurement vector. If the alternative stands out in some way, even one that is not universally appreciated, how would that affect its market share? Assume that in our market, the measurements are symmetric in the sense that $\mathbb{E}[{(m_a^f)}^3]=0$, and that we have an alternative ${\overline{m}}_a$ with $\Vert {\overline{m}}_a\Vert =r\sqrt{F}$ for some $r>0$.

Consider ${\overline{\omega }}_i·{\overline{m}}_a|{\overline{\omega }}_i\sim N(0,1)$ and ${\Vert {\overline{m}}_a\Vert }^2\sim N(F,{\sum }_{f=1}^F\mathbb{E}[{(m_a^f)}^4]-1)$ as bivariate normal random variables. Their correlation is

$$\mathrm{cor}({\overline{\omega }}_i·{\overline{m}}_a|{\overline{\omega }}_i,{\Vert {\overline{m}}_a\Vert }^2)={\displaystyle \frac{{\sum }_{f=1}^F{\omega }_i^f\mathbb{E}[{(m_a^f)}^3]}{\sqrt{{\sum }_{f=1}^F\mathbb{E}[{(m_a^f)}^4]-1}}}.$$

(33)

This is a random variable which is dependent on ${\overline{\omega }}_i$. But since we are working with a symmetric market, it reduces to zero. Using the equations for bivariate normal variable conditioning,

$${\overline{\omega }}_i·{\overline{m}}_a|{\overline{\omega }}_i,\Vert {\overline{m}}_a\Vert =r\sqrt{F}\sim N(0,1).$$

(34)

At very large values of F, this limit does indeed hold, but a better approximation can be made using the law of large numbers (LLN) (in order to allow for the law of large numbers with our non identically generated $\mathbb{E}[{(m_a^f)}^3]$, we either need to assume additional mild conditions on the model generation (for example, independence of $\mathbb{E}[{({\omega }_i^f)}^2]$ and $\mathbb{E}[{(m_a^f)}^3]$), or use some special version of the LLN) to show that

$$\mathrm{var}({\overline{\omega }}_i·{\overline{m}}_a|{\overline{\omega }}_i,\Vert {\overline{m}}_a\Vert =r\sqrt{F})={\displaystyle \sum _{f=1}^F}{({\omega }_i^f)}^2\mathbb{E}[{(m_a^f)}^2|\Vert {\overline{m}}_a\Vert =r\sqrt{F}]\underset{F\to \infty }{\to }{r}^2.$$

(35)

Using ${\overline{\omega }}_i·{\overline{m}}_a\sim N(0,r^2)$ we have

$${\mathbb{E}}_{\mathit{M}}\left[{\mathbb{T}}_a\right|\Vert {\overline{m}}_a\Vert =r\sqrt{F}]\approx {\displaystyle \frac{1}{r}}{\displaystyle \underset{-\infty }{\overset{\infty }{\int }}}\Phi {\left(x\right)}^{A-1}\varphi \left({\displaystyle \frac{x}{r}}\right)dx.$$

(36)

Or, alternatively,

$${\mathbb{E}}_{\mathit{M}}\left[{\mathbb{T}}_a\right|\Vert {\overline{m}}_a\Vert =r]\approx {\displaystyle \frac{\sqrt{F}}{r}}{\displaystyle \underset{-\infty }{\overset{\infty }{\int }}}\Phi {\left(x\right)}^{A-1}\varphi \left({\displaystyle \frac{\sqrt{F}x}{r}}\right)dx.$$

(37)

Figure 6 demonstrates that the approximation is accurate when using a normal distribution for the measurements due to its zero skew.

Figure 6. A comparison of simulated data and the general estimate for the conditioned norm expected value ${\mathbb{E}}_{\mathit{M}}\left[{\mathbb{T}}_a\right|\Vert {\overline{m}}_a\Vert =\sqrt{F}+1]$ using normal distributions for weights and measurements (which is a symmetric distribution) and $F=30$.

4.5.2. Conditioning on the Value an Individual Assigns to an Alternative

We now consider another case where partial information is known about an alternative. Specifically, assume we have a single individual evaluate the information based on their preference (weight vector, denoted as $\overline{u}$), which is fully known. For simplicity, we take $\Vert \overline{u}\Vert =1$. We are interested in the conditional expectation of the market share of alternative a given $\overline{u}·{\overline{m}}_a=v$, ${\mathbb{E}}_{\mathit{M}}\left[{\mathbb{T}}_a\right|\overline{u}·{\overline{m}}_a=v]$. The idea here is that we can split the product ${\overline{\omega }}_i·{\overline{m}}_a$ into the linear projection and rejection (which is the perpendicular component) of vector ${\overline{\omega }}_i$ onto $\overline{u}$:

$${\mathbb{E}}_{\mathit{M}}\left[{\mathbb{T}}_a\right|\overline{u}·{\overline{m}}_a=v]=\underset{{\Omega }_W}{\int }P(({\mathrm{proj}}_{\overline{u}}{\overline{\omega }}_i+{\mathrm{oproj}}_{\overline{u}}{\overline{\omega }}_i)·{\overline{m}}_a>{\overline{\omega }}_i·{\overline{m}}_{a^{\prime }}\forall a^{\prime }\ne a|{\overline{\omega }}_i)dP\left({\overline{\omega }}_i\right),$$

(38)

where

$${\mathrm{proj}}_{\overline{u}}{\overline{\omega }}_i={\displaystyle \frac{\overline{u}·{\overline{\omega }}_i}{{\Vert \overline{u}\Vert }^2}}\overline{u}=(\overline{u}·{\overline{\omega }}_i)\overline{u},{\mathrm{oproj}}_{\overline{u}}{\overline{\omega }}_i={\overline{\omega }}_i-{\mathrm{proj}}_{\overline{u}}{\overline{\omega }}_i={\overline{\omega }}_i-(\overline{u}·{\overline{\omega }}_i)\overline{u}.$$

(39)

Now, since the vectors are orthogonal, we write $\Vert {\mathrm{oproj}}_{\overline{u}}{\overline{\omega }}_i\Vert =\sqrt{1-{(\overline{u}·{\overline{\omega }}_i)}^2}$ and notice that

$$\begin{array}{cc}\hfill \mathrm{cov}({\mathrm{proj}}_{\overline{u}}{\overline{\omega }}_i·{\overline{m}}_a,{\mathrm{oproj}}_{\overline{u}}{\overline{\omega }}_i·{\overline{m}}_a|{\overline{\omega }}_i)& =\mathbb{E}\left[{\displaystyle \sum _{f=1}^F}{m_a^f}^2{\left({\mathrm{proj}}_{\overline{u}}{\overline{\omega }}_i\right)}_f{\left({\mathrm{oproj}}_{\overline{u}}{\overline{\omega }}_i\right)}_f|{\overline{\omega }}_i\right]\hfill \\ \hfill & ={\mathrm{proj}}_{\overline{u}}{\overline{\omega }}_i·{\mathrm{oproj}}_{\overline{u}}{\overline{\omega }}_i=0.\hfill \end{array}$$

(40)

Hence, they are jointly normal. We can, therefore, write

$$({\overline{\omega }}_i·{\overline{m}}_a|{\overline{\omega }}_i,\overline{u}·{\overline{m}}_a=v)\sim N\left((\overline{u}·{\overline{\omega }}_i)v,1-{(\overline{u}·{\overline{\omega }}_i)}^2\right).$$

(41)

In this case, the means and variances of the value distribution are dependent but still normal for a given weight vector. We can solve this by going directly to the integral (38):

$$\begin{array}{cc}\hfill & \underset{{\Omega }_W}{\int }P((\overline{u}·{\overline{\omega }}_i)v+\sqrt{1-{(\overline{u}·{\overline{\omega }}_i)}^2}{Z}_a>Z_{a^{\prime }}\forall a^{\prime }\ne a)dP\left({\overline{\omega }}_i\right)\hfill \\ \hfill & =\underset{{\Omega }_W}{\int }{\displaystyle \frac{1}{\sqrt{1-{(\overline{u}·{\overline{\omega }}_i)}^2}}}{\displaystyle \underset{-\infty }{\overset{\infty }{\int }}}\Phi {\left(x\right)}^{A-1}\varphi \left({\displaystyle \frac{x-(\overline{u}·{\overline{\omega }}_i)v}{\sqrt{1-{(\overline{u}·{\overline{\omega }}_i)}^2}}}\right)dxdP\left({\overline{\omega }}_i\right).\hfill \end{array}$$

(42)

There are two specific cases we may be interested in:

• The ‘information’ vector has a shape that will lead to $\overline{u}·{\overline{\omega }}_i$ being distributed normally. This happens as long as the magnitude of the weights is not concentrated too much on a small subset of features.
• The case where $\overline{u}=e_f$, a standard basis vector, which allows us to check the effect of an isolated feature and its measurement on the alternative’s market share.

In the first case, $\overline{u}·{\overline{\omega }}_i\sim N(\overline{u}·{\overline{\mu }}_W,{\overline{u}}^2·{\overline{\sigma }}_W^2)$. Therefore,

$${\displaystyle \underset{-\infty }{\overset{\infty }{\int }}}{\displaystyle \underset{-1}{\overset{1}{\int }}}{\displaystyle \frac{1}{\sqrt{(1-z^2)({\overline{\sigma }}_W^2·{\overline{u}}^2))}}}\Phi {\left(x\right)}^{A-1}\varphi \left({\displaystyle \frac{x-vz}{\sqrt{1-z^2}}}\right)\varphi \left({\displaystyle \frac{z-{\overline{\mu }}_W·\overline{u}}{\sqrt{{\overline{\sigma }}_W^2·{\overline{u}}^2}}}\right)dzdx.$$

(43)

In the second case, we are going to need the distribution density function for ${\omega }_i^f$, denoted as ${\phi }_f\left(z\right)$:

$${\displaystyle \underset{-\infty }{\overset{\infty }{\int }}}{\displaystyle \underset{-1}{\overset{1}{\int }}}{\displaystyle \frac{1}{\sqrt{1-z^2}}}\Phi {\left(x\right)}^{A-1}\varphi \left({\displaystyle \frac{x-vz}{\sqrt{1-z^2}}}\right){\phi }_f\left(z\right)dzdx.$$

(44)

Figure 7 shows a comparison between simulations and approximations.

Figure 7. A comparison of simulated data and the estimate for the expected value of an alternative by conditioning on its subjective value for some test weight vector $\overline{u}$. (Top): $\overline{u}$ was picked as a random Gaussian vector. (Bottom): $\overline{u}=e_f$, which is akin to singling out a measurement $m_a^f$ and judging based on it alone. In both cases, $F=30$.

4.6. Distribution of Market Share

In this section, we derive a formula for the comparison distribution, which is the market share distribution of size $A=2$, and use it to calculate the comparison entropy. We also provide ad hoc bounds for the market share distribution of competitive markets.

Using the Gaussian approximation, finding a formula for $P({\mathbb{T}}_a<t)$ for $A=2$ is relatively simple since there are only two alternatives to compare:

$${\mathbb{T}}_a({\overline{m}}_a,{\overline{m}}_{a^{\prime }})=P(v_{i,a}>v_{i,a^{\prime }}|{\overline{m}}_a,{\overline{m}}_{a^{\prime }})=\Phi \left({\displaystyle \frac{{\overline{\mu }}_W·({\overline{m}}_a-{\overline{m}}_{a^{\prime }})}{\sqrt{{\overline{\sigma }}_W^2·{({\overline{m}}_a-{\overline{m}}_{a^{\prime }})}^2}}}\right).$$

(45)

Then,

$$P({\mathbb{T}}_a<t)=P\left({\displaystyle \frac{{\overline{\mu }}_W·({\overline{m}}_a-{\overline{m}}_{a^{\prime }})}{\sqrt{{\overline{\sigma }}_W^2·{({\overline{m}}_a-{\overline{m}}_{a^{\prime }})}^2}}}<{\Phi }^{-1}\left(t\right)\right)=\Phi \left(\eta {\Phi }^{-1}\left(t\right)\right).$$

(46)

We can see that when the market is subjective, as in $\eta =\Vert {\overline{\sigma }}_W\Vert /\Vert {\overline{\mu }}_W\Vert \to \infty$, the inner term will be close to $-\infty$ when $t<1/2$ and $+\infty$ when $t>1/2$. This implies that as $F\to \infty$, the CDF approaches a step function at $1/2$. When $\eta =1$, Equation (46) simplifies into $P({\mathbb{T}}_a<t)=t$, a uniform distribution, as can be seen in Figure 8.

Figure 8. CDFs of markets of size two. Insets show the segmentation of the market by chosen alternatives, where both alternatives and their respective segments are colored accordingly. Small dots represent individuals’ weight vectors, colored by their preferred alternative. Diamonds represent the alternatives. The boundary of each segment must go through the center of axes, which is marked with a plus. In a subjective market, two alternatives always segment the market into two parts going through the origin. As $F\to \infty$, the CDF of the subjective market distribution approaches a step function. In a competitive market with two alternatives, as $F\to \infty$, the distribution of market shares becomes uniform.

The fact that a market has a uniform distribution when comparing two alternatives is significant, as it implies that the competitive distribution maximizes the comparison entropy, which is how we originally defined the competitive distribution. In other words, a competitive market is one in which any two random alternatives segment the market in the least predictable way. From this fact, we draw the justification for calling it a competitive market.

Recall our definition of comparison entropy back in Section 2.2. In the context of the linear preference model, using Equation (46),

$$c_{\eta }\left(x\right)={\displaystyle \frac{\eta \varphi \left(\eta {\Phi }^{-1}\left(x\right)\right)}{\varphi \left({\Phi }^{-1}\left(x\right)\right)}},$$

(47)

$$\begin{array}{cc}\hfill H\left[\right(\mathit{W},\mathit{M}\left)\right]& =-{\int }_0^1{\displaystyle \frac{\eta \varphi \left(\eta {\Phi }^{-1}\left(x\right)\right)}{\varphi \left({\Phi }^{-1}\left(x\right)\right)}}log\left({\displaystyle \frac{\eta \varphi \left(\eta {\Phi }^{-1}\left(x\right)\right)}{\varphi \left({\Phi }^{-1}\left(x\right)\right)}}\right)dx.\hfill \end{array}$$

(48)

Substituting $z={\Phi }^{-1}\left(x\right)$,

$$\begin{array}{cc}\hfill H\left[\right(\mathit{W},\mathit{M}\left)\right]& =-{\int }_{-\infty }^{\infty }\eta \varphi \left(\eta z\right)\left(log\left(\eta \right)+log\left(\varphi \left(\eta z\right)\right)-log\left(\varphi \left(z\right)\right)\right)dz\hfill \\ \hfill & =-log\left(\eta \right)+{\displaystyle \frac{{\eta }^2-1}{2{\eta }^2}}.\hfill \end{array}$$

(49)

Maximizing $\left(49\right)$, we find that the maximal entropy distribution is obtained for $\eta =1$. Accordingly, this is our definition of a competitive market. This also follows on directly from the fact that the maximal entropy distribution among those defined on $[0,1]$ is uniform when $\eta =1$.

Based on Equation (47), we can also derive a maximum-likelihood estimator for $\eta$. Given n pairwise comparisons where $0<t_i<1$ of the population choose one of the alternatives, one can easily find the MLE to be

$$\widehat{\eta }=\sqrt{{\displaystyle \frac{n}{{\sum }_{i=1}^n{\Phi }^{-1}{\left(t_i\right)}^2}}}.$$

(50)

Using the same simulation as above, we investigate the market share distribution for larger values of A (Figure 9). The competitive distribution appears to be similar in shape to an $L^p$ circle with $p=A-1$. This would provide a distribution of

$$P({\mathbb{T}}_a<t)=(1-{(1-t)}^{{(A-1)}^{{\displaystyle \frac{1}{2}}}}{)}^{{(A-1)}^{-{\displaystyle \frac{1}{2}}}}.$$

(51)

Figure 9. Empirical cumulative distribution function of both subjective and competitive markets with $A=10$. For the competitive distribution, we provide two rough approximations. The true simulated value appears to be in between those two curves.

In fact, this approximation appears to work quite well as a lower bound (Figure 9), together with an upper bound provided by approximately

$$P({\mathbb{T}}_a<t)={(1-{(1-t)}^{{(A-1)}^{\gamma }})}^{{(A-1)}^{-\gamma }},$$

(52)

where $\gamma \approx 0.5772$ is the Euler–Mascheroni constant.

5. Imperfect Information and Softmax Choice

We now turn our attention to markets with incomplete information. We can model such markets by employing the softmax decision rule, instead of having individuals pick their best alternative with probability 1.

As shown in reference [14], multinomial logit choice probabilities arise from a random utility model in which each alternative is provided by $U_{i,a}=v_{i,a}+{ϵ}_{i,a}$, where the ${ϵ}_{i,a}$ are i.i.d. type I extreme value (Gumbel) random variables. We can work that into our case by introducing choice probabilities:

$$P_i\left(a\right)={\displaystyle \frac{e^{\beta v_{i,a}}}{{\sum }_{a^{\prime }=1}^A{e}^{\beta v_{i,a^{\prime }}}}},$$

(53)

where $0<\beta <\infty$ is the inverse-temperatureparameter, which resembles the amount of information available in the market. In the limit $\beta \to \infty$, we obtain our original best-choice decision rule. As seen in reference [33], the noise term ${ϵ}_{i,a}$ follows a Gumbel distribution with zero location and scale $1/\beta$. We can roughly approximate it with a normal distribution of the same variance, center it (the mean of the noise does not affect choice probabilities), and use it to approximate ${\eta }_{\mathrm{eff}}$ as a function of $\beta$:

$$v_{i,a}+{ϵ}_{i,a}\sim N\left(0,1+{\displaystyle \frac{{\pi }^2}{6{\beta }^2}}\right),$$

$${\eta }_{\mathrm{eff}}={\displaystyle \frac{\sqrt{{\Vert {\overline{\sigma }}_W\Vert }^2+\frac{{\pi }^2}{6{\beta }^2}}}{\Vert {\overline{\mu }}_W\Vert }}=\sqrt{{\eta }^2+{\displaystyle \frac{{\pi }^2(1+{\eta }^2)}{6{\beta }^2}}}.$$

(54)

Figure 10 shows the rank frequencies and comparison distributions in markets using the softmax choice rule, as well as the same plots for markets using the best choice rule with a modified ${\eta }_{\mathrm{eff}}$. The market structure itself stays mostly intact, suggesting that adding noise to the market, for example, by means of faulty information, or lack thereof, can masquerade as an inherent higher degree of subjectivity (for discussion on similar ideas, see [34]).

Figure 10. (Left) Rank frequency plots generated using the softmax choice rule with $\eta =0.5$ and a varying inverse-temperature $\beta$. (Right) Rank frequency plots averaged over many market realizations using the best value choice rule with the appropriate ${\eta }_{\mathrm{eff}}$. Small inconsistencies may arise due to approximating the Gumbel noise by the normal distribution. (Left inset): Comparison distribution CDF in each case. (Right inset): Log–log plot of the rank frequency plots.

This result suggests an underlying connection between systematic utility distributions generated by a linear preference model and the random utility theory framework presented in [12,15] and other works.

In Section 6, we discuss the model’s connection to neural networks and provide a short example of a preference market occurring in classification data in a natural manner. In Appendix C and Appendix D, we introduce market segmentation and present initial results in deriving a general formula for the market share distribution.

6. Relation to Classification Neural Networks

We present an example of a preference market arising naturally from a classification neural network by likening classification probabilities to utility. The purpose of this experiment is to compare theory with data and test the model predictions, specifically the importance of the parameter $\eta$ with non-linear utilities.

6.1. Problem Setup

First, a neural network with multiple hidden layers was trained (the network received an input of a 28 × 28 grayscale image, which was processed through two convolutional blocks, followed by a flat dense layer and then a softmax layer with 47 elements for the various characters. It uses ReLU activations, dropout for regularization, and batch normalization to stabilize training) to classify images into English letters and digits; henceforth, characters using the extended MNIST dataset [35]. We then generated synthetic symbols using a custom Python (https://www.python.org/) script that draws random scribbles with varying structural complexity (the symbols were drawn using a turtle script [36] with two basic operations: draw a line in a random direction or draw a circle arc in a random direction, starting from the last position in both cases. Angles and lengths were picked out of configurable distributions, alongside probabilities to draw a straight line or an arc. The number of lines drawn was the only parameter to be changed between the three data sets, ranging from 1–2 in the “few lines” set to 5–7 in the “many lines” set), henceforth, symbols. These synthetic symbols were passed through the trained classifier to determine which label, or character, they most closely resembled. Figure 11 shows some of the characters and symbols, along with their classifications by the neural network.

Figure 11. (Left) Example characters from the MNIST extended dataset. Above each character, the digit/letter that the classifier assigns to it is displayed. (Right) Randomly generated symbols from three distinct sets with varying amounts of lines. (Top) Few lines. (Center) Medium lines. (Bottom) Many lines.

One can think of a decision process as a machine that takes an input describing an individual (weights) and uses them to pick one of the alternatives [28]. In the classification neural networks studied here, characters play the role of alternatives and symbols as weights. The relevant features and measurements for each alternative are found by training the network. The process of classification corresponds to choosing the most appropriate character. In fact, if the classification network were to be composed of only the first and last layers, it would result in the match probability being a monotone transformation (typically softmax) of a linear product, resulting in the same preference order as that of a linear utility.

Figure 12 shows the rank-frequency plots of random symbols in each set. Intuitively, different symbols with many lines are often interpreted as different characters, which is analogous to a subjective market. In contrast, with a smaller number of lines, most symbols are interpreted as ’1’, ’L’, or ’t’, leading to a more objective market.

Figure 12. Rank-frequency plots of the random symbols generated in each set. Each set was generated with the same configurations, aside from the number of lines drawn. The number of lines drawn ranges from 1–2 in the “few lines” set and 5–7 in the “many lines” set.

6.2. Comparison Distribution

Because the final layer of a classification network is a full vector of probabilities, we can compare any two characters and derive the empirical comparison distribution for the market (Figure 13). By minimizing the Kolmogorov–Smirnov statistic on the comparison distribution, we can estimate values of $\eta$ as $0.81, 1.61$, and $2.32$ for the respective data sets. Note that these estimates are only empirical, since the model is defined for linear utility, which digit classification is unlikely to represent. Still, not only do we see that the empirical comparison distribution is shaped mostly as we expect it to be in a linear preference market, but we can also observe how a larger number of lines in the symbol set leads to a higher value of $\eta$, as in a larger degree of subjectivity in the market.

Figure 13. Empirical comparison distribution for each symbol set in blue. In black, in the background, we have displayed a sample of some set values of $\eta$ using Equation (46). The empirical comparison distribution was generated by creating a histogram from all possible pairs of alternatives (characters). For any given pair, we note the percentage of the population (symbols) that would pick one alternative over the other. We do this by comparing the respective values in the final vector of the neural network, i.e., the output layer, for each symbol. This gives rise to a distribution with symmetry around $1/2$, as the order of comparison does not matter.

In Section 7.3, we present an argument for why non-linear utility might still adhere to the shape of a linear preference model more often than expected.

7. Discussion

In this paper, we studied a high-dimensional probabilistic formulation of the linear multi-attribute utility model and investigated the statistical behavior of preference markets arising from it. While traditional approaches to MCDA typically emphasize the structure of individual utility functions and discrete choice modeling, our perspective shifts toward population-level outcomes.

To characterize the impact of variation in individual preferences, we introduced the parameter $\eta =\Vert {\overline{\sigma }}_W\Vert /\Vert {\overline{\mu }}_W\Vert$, the degree of subjectivity in a market. Markets with $\eta$ approaching zero exhibit largely objective behavior, where most individuals rank alternatives similarly. In contrast, when $\eta$ is large, markets become highly subjective, and choice outcomes depend strongly on individual differences. Of particular interest is the case $\eta =1$, which defines a competitive market. There, the comparison distribution becomes uniform, indicating maximal entropy and minimal predictability: an emergent balance between consensus and variation.

We then derived analytical approximations for the expected market share of a fixed alternative in a random market, as well as for its variance and several forms of conditional expectations. These results support our main observation that the single parameter $\eta$ classifies the distribution of market share among the alternatives.

Finally, we demonstrated the model’s applicability through a simulation involving randomly generated symbols and a trained neural network classifier. In this experiment, the neural network assigned labels to synthetic inputs, and the resulting distribution of classification outcomes naturally mirrored those of preference markets under the linear model. The similarity between empirical and simulated distributions suggests that the framework captures relevant features of high-dimensional decision processes and points to its potential use as a diagnostic tool in practical domains such as recommender systems, polling, and behavioral sciences.

7.1. Context and Repeated Surveying

Context is an important factor for preference. For example, asking individuals to pick their favorite fruit out of a list, which they may not have experienced in its entirety, could produce different results than first providing them with samples. In order to produce consistent and meaningful results when analyzing a number of decision-making processes, it is important that the context is consistent. This consistency is paramount for conducting a proper statistical analysis of preference, and while we took it for granted when defining the model, it is important to remember it is not trivial when working with real data and polls.

When the decision problem is consistent, each individual is met with the exact same scenario, and differences in individual preference alone cause differences in choice. The context can be thought of as an anchor, as discussed in [37].

An interesting idea is to transfer the implications of a change in context into a change in the individual’s preference. In theory, a probability distribution of contexts could then be transformed into a distribution of preference, the same as is possessed by the entire population (albeit being different parametrically).

One realization of this idea can be formed by conducting repeated surveys over time for a single individual, which may, in turn, approximate a cross-sectional survey under ergodicity assumptions.

7.2. Practical Limitations of the Model

7.2.1. Inherent Limitations of the Linear Preference Model

The most apparent limitation of modeling utility linearly is the inability to account for features that enhance or detract from one another. A fixed non-linear transformation of a single feature can be handled by transforming that measurement and its distribution (see Appendix E), but this does not account for interactions between features or individual-specific transformations (A simple example can be found in cooking: two ingredients may taste good separately, but not when mixed together). In Section 7.3, we provide an informal argument for how some non-linear utility functions may be represented using linear utility at the cost of introducing cross-weight correlations. Section 6 also presents one potentially non-linear case in which the empirical comparison distribution remained close to that predicted by the linear model. These observations suggest that the framework may remain useful in some non-linear settings, but we do not prove robustness to non-linear interactions or cross-weight correlations in this paper.

Aside from linearity, the model assumes complete and common information about the alternatives, decisions made independently by each individual, and an unlimited supply. Prices may be included as fixed measurements, but endogenous pricing, capacity constraints, and strategic or social interactions between decision-makers are not modeled. Section 5 relaxes deterministic choice by introducing random-utility noise through the softmax rule.

The model also describes a static market with a consistent context. In real markets, measurements, preferences, available alternatives, and consideration sets may change over time, while previous choices may themselves affect later decisions. Pooling observations across such changes can violate the assumed stability and independence of the underlying distributions. Context effects introduce a related limitation, discussed in Section 7.1: apparent differences in preference may instead reflect differences in how the same decision problem is presented or interpreted.

7.2.2. Limitations Arising from High-Dimensional Limit Assumptions

When deriving the equations for the expected market share in Theorem A3 (Appendix B), we had to take the assumption that

$${\displaystyle \sum _{f=1}^F}\mathbb{E}[{({\omega }_i^f)}^4]\underset{F\to \infty }{⟶}0,$$

as well as

$${\displaystyle \sum _{f=1}^F}\mathbb{E}\left[\right|{\omega }_i^f{|}^3]·\mathbb{E}[|m_{a^{\prime }}^f{|}^3]\underset{F\to \infty }{\to }0.$$

One of the major drawbacks of these conditions is that it becomes challenging to include distributions that are not sub-Gaussian.

The first condition essentially comes down to requiring that the magnitude of the weights is spread apart throughout the features and not clumped up too much in a small number of over-dominant features. It is also connected to the participation ratio of ${\overline{\sigma }}_W^2$, defined as ${\Vert {\overline{\sigma }}_W\Vert }_2^4/{\Vert {\overline{\sigma }}_W\Vert }_4^4$ in the case the vector’s elements are independent, and also sometimes referred to as the effective dimensionality of the vector. Indeed, since $\mathbb{E}[{({\omega }_i^f)}^4]\ge {({\sigma }_W^f)}^4$, the first condition forces this effective dimension to diverge whenever the total variance stays bounded away from zero; the converse additionally requires control of the fourth moments. In some scenarios, we might find that while F itself is large, the effective dimensionality is the real limiter to the model’s applicability.

The second condition can be thought of as related to deal breakers. In order to fail, the condition would require some features whose weights and/or measurements are highly skewed.

For example, take food allergies as a deal breaker for an individual choosing a meal from a menu in a restaurant. For some individuals, the weight of the deal breaker or allergen will be extraordinarily large, which will invalidate, in a sense, the usual form of decision making.

Additionally, when measuring the expected market share for some given alternative with known measurements ${\overline{m}}_a$, we had to further assume that

$$\underset{F}{sup}\left|{\displaystyle \frac{{\overline{m}}_a·{\overline{\mu }}_W\left(F\right)}{\sqrt{{\overline{m}}_a^2\left(F\right)·{\overline{\sigma }}_W^2\left(F\right)}}}\right|<\infty$$

on our limit of $F\to \infty$. The implication of this is that the expected value estimate starts failing for superstar alternatives—if the value of the alternative is sufficiently above the typical range for the other alternatives, the approximation might fail. This could become relevant in regimes where the size of the market A is comparable to, or grows with, the size of the population I. In such regimes, such as the population of cities, the top alternatives’ values could approach such limits due to the laws of extreme-value theory. The error itself would vanish as F grows, but it raises a question about what if F itself grows as a function of I. Such regimes with many (with respect to I) alternatives are an interesting topic in and of themselves, but they remain outside the scope of this paper.

7.2.3. Limitations of Empirical Interpretation

Linear additivity, independence, moment conditions, and a sufficiently large effective dimension are mathematical limitations of the present theory, since they are used to derive the limiting market-share laws. Other limitations primarily affect empirical interpretation. Within the model, $\eta$ measures preference heterogeneity relative to the shared component of preference. In observed choices, however, similar dispersion may arise from decision noise, incomplete information, limited consideration, changing context, or resource-limited information processing. Aggregate market shares alone do not generally distinguish between these mechanisms. An empirically fitted value of $\eta$ should, therefore, be interpreted as a model-conditional description of the observed comparison distribution, rather than as identifying preference heterogeneity as its unique cause. Separating these explanations would require additional structure or data, such as repeated choices, controlled variation in menus or information, or direct measurements of the decision process.

7.3. Non-Linearity and Cross-Weight Correlations

Looking back at Figure 13, one might suspect there is more going on behind the scenes to allow for a non-linear problem to so closely align with a linear model. We provide a non-rigorous argument for how one can attempt to model any analytic utility function using a linear model, assuming that correlations between the weights are negligible.

The key idea here is that instead of viewing the weights of individuals as the input layer of a multi-layered classification network, we instead focus on the second-to-last layer. The move from that layer to the next final layer exactly encapsulates discrete choice based on linear utility and softmax, as in the Logit model. This leads to a couple of subtle implications.

• In cases where the second-to-last layer is deep into the network, cross-weight correlations will be picked up along the network.
• There is no guarantee that the distribution of the weights in the second-to-last layer produces a variance vector with high enough dimensionality (see the discussion in the previous Section 7.2.2), which is a necessary condition for the high-dimensional limit to hold properly. Utility, even when composed from a multitude of feature-weights, may end up having a low effective dimension of complexity.
• The measurements become the edge weights of the neural network leading up to the final layer to each category (alternative). We may no longer be able to directly sample them, and the new measurements may be non-linear expressions of actual measurements.

In order to make sense of a preference market being generated directly for a utility function, we also require a concrete distribution on the alternatives, i.e., the neural network weights on the edges leading to the last layer. With that given, we are able to use PCA on the measurement space (modifying our new weights accordingly, further shifting their distribution), and finally, standardize the distributions as usual.

This leaves us with linear utility, but also with non-zero cross-weight correlations. As the non-linearity becomes ‘stronger’, one would expect the correlations to do so as well. Finally, we hypothesize that for linear models with low enough cross-weight correlations, one can bound the deviation from the correlation-free model we studied in this paper. This, however, is outside the scope of this paper and is left for future work to explore.