Diversification Is Not a Free Lunch

: This study analyzed the statement “diversification is a free lunch”. We empirically showed that diversification is only a free lunch under uncertainty or ignorance, confirming Warren Buffett’s “diversification is protection against ignorance”. Using historical returns of the S&P500 constituents illustrated that diversification not only decreased the risk but also the returns if the expected returns could be estimated. The findings of this study highlight that diversification reduces risk but that the risk reduction is not for free.


Introduction
Diversification is a fundamental concept in finance and is often loosely explained with "Don't put all your eggs in one basket".Importantly, diversification is frequently labeled a free lunch, and even "the only free lunch". 1 The statement seems to be at odds with the fundamental economic principle proposed by Nobel laureate Milton Friedman that "there is no such thing as a free lunch".In other words, nothing is for free. 2 Extending this idea to the concept of diversification means that the reduction in risk comes at the cost of lower returns and is thus not a free lunch.
This study aimed to solve the contradiction between the seemingly common and widely accepted belief that diversification is a free lunch and the fundamental economic principle that there is no such thing as a free lunch.We used theoretical arguments and an empirical analysis to demonstrate that there is no contradiction.
A simple example highlights that diversification is costly.The performance of a diversified portfolio of 500 stocks represented by the S&P500 increased four-fold between 2009 and 2023 compared with the performance of a single stock represented by Apple, which increased about 55 times over the same period.Obviously, this comparison is arbitrary and ex post.Ex ante, in 2009, it was not clear which stock would outperform the S&P500 over the following 14 years.However, it was clear in 2009 that the risk of any single stock would be higher than the risk of a diversified portfolio.This explains why the focus is commonly on risk, as the risk of a diversified portfolio is generally lower than the risk of any individual stock both ex ante and ex post.Importantly, the fact that the risk can be reduced through diversification does not automatically imply that this can be achieved without reducing the return, and thus, at no cost, implying a free lunch.Surprisingly, given the frequent claims of Markowitz that "Diversification is the only free lunch", Markowitz (1952) showed that if expected returns are used, diversification is never superior (p.89). 3  It is also important to note that diversification is not a free lunch if the Sharpe ratio increases for more diversified portfolios unless the increased Sharpe ratio is based on stable or higher returns.While Sharpe ratios can increase if the reduction in risk is greater than the reduction in the expected return, the increased Sharpe ratio cannot be labeled a free lunch, as it comes at the cost of a lower expected return.This paper illustrates that diversification can only be a free lunch if inefficient markets are assumed.Interestingly, the fundamental principle of a positive risk-return relationship would also be violated if diversification was a free lunch.Specifically, if diversification would lower the risk and not lower returns, risk and return would not be aligned.Given this reasoning and the explicit statements in Markowitz (1952) about diversification, we wonder whether Markowitz has actually ever said that "diversification is the only free lunch".Our skepticism is supported by the lack of any reference to an article, book, website, or recording as evidence.
The empirical part based on a sample of monthly share prices of the S&P500 constituents over 20 years showed that diversification is not a free lunch if the expected returns can be estimated.The analysis also identified two special conditions, uncertainty and ignorance, under which diversification is a free lunch for investors.We showed through simulations that such ignorance implies portfolios that had the same return but lower risk with an increasing number of assets in the portfolio.This condition supports the statement by Warren Buffett that "diversification is protection against ignorance".Importantly, while the expected returns did not fall with an increasing number of assets in a portfolio under investor ignorance, i.e., the investor knew nothing about the assets and picked them randomly, the expected returns were clearly lower compared with more reasonable alternatives, such as that expected returns could be estimated and were not unknown.Hence, ignorance whilst delivering a free lunch of diversification is costly.
The analysis further indicated that the statement "there is no such thing as a free lunch" (Friedman 1975) was based on the implicit and generally silent assumption of uncertainty or ignorance.If all the benefits and costs of an action are known, any reference to "there is no such thing as a free lunch" is redundant. 4 Given the importance of diversification for investing and risk management, it is not surprising that there is a large literature on diversification.However, we could not find any research that focused on the "free lunch of diversification" and analyzed its validity.In addition, we could not find any research that discussed the role of uncertainty or ignorance in the context of diversification.
Key papers on this topic are Markowitz (1952) and more recently Booth and Fama (1992), Chua et al. (2009), Choueifaty and Coignard (2008), Kinlaw et al. (2021), Rudin and Morgan (2006), Flores et al. (2017), andYeung et al. (2012), among many others.Choueifaty and Coignard (2008) studied maximum diversification and confirmed that constant proportion portfolios earn additional returns labeled diversification returns by Booth and Fama (1992).Chambers and Zdanowicz (2014) identified limitations on the diversification return if there is no mean reversion in the underlying assets.Rudin and Morgan (2006) created a portfolio diversification index and demonstrated a declining marginal value of diversification.They also used principal component analysis to evaluate the effective number of truly independent components within a portfolio.Chua et al. (2009) emphasized that upside diversification is undesirable.They proposed a portfolio construction technique that produces lower correlations on the downside and higher correlations on the upside than mean variance optimization portfolios.Kinlaw et al. (2021) reconsidered Chua et al. (2009)'s "myth of diversification".They argued that investors prefer diversification when a portfolio's main growth engine performs poorly and unification when it performs well.Yeung et al. (2012) identified the proponents of diversification, e.g., Harry Markowitz and William Sharpe, and concentrators, e.g., John Maynard Keynes and Warren Buffett.The authors analyzed mutual fund performance and found that concentrated portfolios outperformed diversified portfolios.Lozza (2001) and Jarrow (1986) analyzed stochastic dominance for portfolio choice and arbitrage, respectively.Tobin (1958) demonstrated that investors "pay for diversification" (see also Samuelson 1967), which supports this paper's main claim that "diversification is not a free lunch".
This study contributes to the literature by demonstrating that diversification is only a free lunch under special (and arguably extreme) assumptions of uncertainty or ignorance.Without uncertainty or ignorance but risk and some knowledge (as opposed to full ignorance), diversification is not a free lunch.
The findings of this study also contribute to the diversification puzzle (see Lozza et al. 2018;Statman 2004), as lower (sub-optimal) degrees of diversification can be explained with the fact that diversification comes at the cost of lower returns.Lozza et al. (2018) also showed that risk-seeking investors invest in a single stock and that different risk attitudes affect the level of diversification.Finally, Ibragimov (2007) demonstrated that extremely heavy-tailed risks or returns reverse the stylized fact that diversification is preferable. 5 The remainder of this paper is structured as follows: Section 2 provides the theoretical background and basis for the empirical analysis in Section 3. Section 4 summarizes the main results and provides concluding remarks.

Background
This section provides a theoretical background for the empirical analysis.If an investor selects the asset with the highest expected return from the set of investible assets, adding any other asset will reduce the expected return because the added asset has a lower expected return than the "highest expected return" asset.The fact that it will reduce the risk (if the correlation is smaller than 1 and the risk of the added asset is lower than the risk of the first asset) is the essence of diversification but this reduction in risk is not a free lunch because it comes at a cost.
Using the adage "Don't put all your eggs in one basket", if an investor puts all their eggs in two baskets, the risk will be lower but there is no free lunch, as two baskets must be more expensive than one basket.And the two baskets would need to be different, as if they were the same, the correlation would be one and there would be no diversification, and thus, no risk-reduction benefit.

Two-Asset Portfolios
Based on a two-asset portfolio, we show that diversification will always dilute returns unless both assets have the same expected return.In a second step, we demonstrate that diversification reduces risk if correlations are smaller than 1. 6 In a third step, we demonstrate that there is a free lunch on the inefficient part of the efficient frontier.
We do not explicitly discuss inefficient markets, e.g., the combination of two assets that have the same return but different risk or the same risk and different returns.In these cases, a combination of the assets, and thus, diversification would be a free lunch, but based on the assumption that markets are not efficient. 7We assumed that markets are efficient and that the no-arbitrage condition holds.

Returns
Under which conditions can diversification lead to a higher return than the "starting" asset 1 (R 1 )?For simplicity, we assumed that the one-asset portfolio consisting of asset 1 is diversified with an addition of asset 2 (R 2 ), forming a two-asset portfolio.We allowed for different weights denoted as w with 0 ≤ w ≤ 1.
The above set of equations show that the return of asset 1 can only be enhanced if asset 2 has a higher return than asset 1.This implies that diversification cannot be a free lunch unless a higher-return asset is added, which means that a sub-optimal set of assets was used as a starting point and higher-return assets are ignored.In other words, diversification is only a free lunch if higher-return assets are included in the second stage but were not considered in the first stage.
Note that this conclusion is based on the implicit assumption that the investor can precisely predict future returns or that the investor can rank assets correctly, even if the actual returns are different.If the investor is ignorant, no ranking can be performed.

Risk
Assume two assets with expected returns of 10% and 5% and standard deviations (risks) of 10% and 5%, respectively.If the correlation between the two assets is 0.5, an equally weighted portfolio would yield an expected return of 7.5% and a standard deviation (risk) of 6.61%.If the correlation between the two assets was 1, the risk of the portfolio would be 7.5%, and thus, a linear combination of the two assets' risks (0.5 × 10% + 0.5 × 5%).This simple example illustrates that the risk can be reduced under the rather weak assumption that the correlation between two assets is smaller than 1.However, the lower risk comes at a cost of a lower return.As mentioned above, similar returns of the two assets but different risks would imply inefficient markets, which is a rather strong assumption and premise to achieve a free lunch through diversification.

Efficient Frontier
Figure 1 presents the risk-return relationship for two assets with a correlation of zero for different weights w (1 − w) from 0 to 1.The graph shows that the risk can be reduced and the return can be enhanced, consistent with a free lunch of diversification on the inefficient part of the frontier only (below the dashed horizontal line).However, on the efficient part of the frontier (above the dashed horizontal line), any risk reduction implies a reduction in the expected return, inconsistent with the free lunch of diversification.Graphically, the claim that diversification is a free lunch would be represented by a line with a zero slope (parallel to the x-axis), implying that a reduction in risk can be achieved at a constant return, and thus, at no cost.A line with a negative slope would imply that the reduction in risk can be achieved at an increased return, and thus, at a negative cost.Efficient frontier and free lunch on an inefficient frontier.The graph displays the frontier of two assets with a correlation of zero (black line) and the frontier of two assets with similar risk and return with a correlation of one (red line).The inefficient part of the frontier (below the dashed horizontal line) is an area where the risk-return relationship is negative and where the risk can be reduced and the return can be enhanced, consistent with the free lunch of diversification.However, to achieve such an outcome, inefficient allocations must be entertained.The efficient frontier above the dashed line displays a positive risk-return relationship and does not entail the free lunch of diversification.
While the graph does not include short sales, leveraged positions through short sales would extend the efficient frontier on the right and enable higher returns and higher risk.Hence, short sales do not enhance returns at a constant level of risk, implying no "free lunch".This section demonstrates that for two-asset portfolios, there is no free lunch of diversification.The following section analyzes the question of whether diversification empirically entails a free lunch for portfolios of up to 500 stocks.

Empirical Analysis
This section presents the empirical analysis results of diversification based on log returns of the monthly prices of the S&P500 constituents from 31 December 1999 to 31 January 2022.We used all 500 constituents of the S&P500 for the analysis and assumed that an investor created differently sized, equally weighted portfolios drawing from this set of 500 stocks.
We considered three scenarios for the analysis of the costs and benefits of diversification to answer the question regarding whether diversification is a free lunch.
First, we assumed that investors can accurately predict future returns and, therefore, rank each asset (constituent of the S&P500) based on its expected return.Markowitz (1952) referred to this as the "expected return rule".Second, we assumed that investors were ignorant and randomly picked assets, and thus, not using any information about the assets or not knowing anything about the assets.Third, we assumed a scenario where investors used historical returns to predict future (expected) returns.Each scenario was assigned a specific label based on the information of investors.In the first scenario, investors had certainty about future returns; in the second scenario, investors had uncertainty about future returns; and in the third scenario, investors faced risk and could only imperfectly predict future returns.The distinction between uncertainty and risk was consistent with Knightian uncertainty (Knight 1921).We further analyzed the role of rebalancing by considering (i) no rebalancing of the portfolio over time and (ii) rebalancing every month to ensure equal weights of all assets.Rebalancing can be considered to enhance diversification, as it ensures equal weights (1/N) for all assets in the portfolio.No rebalancing allows weights to change significantly and could result in rather concentrated portfolios toward the end of the investment period.

Certainty-Expected Returns Rule
Table 1 and Figure 2a demonstrate the effect of diversification by increasing the number of assets considered in a portfolio from 1 to 5, 10, 20, 50, 100, 200, and 500.Given the assumed certainty about future expected returns, the assets were ranked and the highest returning asset was put in a portfolio with one asset.The next four-highest-ranked assets were included in the portfolio and equally weighted (1/N) to form the portfolio with five assets, and so forth.Due to the ranking of the assets based on their expected returns, the average returns of these portfolios fell with an increasing number of assets in a portfolio.
The table and the graphs clearly show that this also held empirically: the returns of the portfolios fell for an increasing number of assets in the portfolio, indicating a cost of diversification.As expected, the risk also fell, and fell by more than the return reflected in an increased Sharpe ratio 8 between portfolios of size 1 to 50.Portfolios of sizes 100, 200, and 500 had lower Sharpe ratios, suggesting that the optimal portfolio size was around 50 assets.The maximum Sharpe ratio was 50 for the non-rebalanced portfolio and 20 for the rebalanced portfolio.
While a focus on returns would suggest a portfolio with one asset only because it had the highest return, including the standard deviation or the risk of the portfolio into the decision may lead to portfolios with more than one asset or diversified portfolios.For example, if the return/risk ratio (Sharpe ratio) is used as a decision rule, porfolios with more than one asset are considered.Markowitz (1952) argued that this is the difference between speculating and investing (see p. 87).Relying exclusively on expected returns is speculating, whereas considering both expected returns and risk is investing.It is also noteworthy that the maximum return fell by more than the minimum return for an increasing number of assets in a portfolio.In other words, the upside risk fell by more than the downside risk, highlighting that the symmetry of the standard deviation was potentially hiding important information for investors.What diversification does is not only make portfolios more stable by reducing the risk of losses but also reduces the risk of gains.
We concluded that diversification is not a free lunch because the returns fell for more diversified portfolios.

Uncertainty-Ignorance
Table 2 and Figure 2b present the results for a scenario of uncertainty and ignorance, i.e., the investor did not know anything about any stock and randomly picked stocks.We used 1000 iterations of random picks for portfolios of size 1, size 5, size 10, etc., to calculate the total return, mean return, standard deviation of returns, and other statistics, as shown in Table 2.The graph is based on 10,000 iterations of randomly picked stocks and randomly picked months of the sample to calculate the equally weighted returns for each of these portfolios.For each portfolio size, the average return (median) and the distribution of these averages over all 10,000 draws is illustrated through boxplots.The results presented in the table are based on the full sample with no rebalancing and rebalancing.In contrast, the results presented in the graph are based on draws from the monthly return sample.The random draws from the sample distribution allow for the use of boxplots to illustrate the distribution of returns for each portfolio size.The number of iterations differ between the table and the figure because the use of the full sample and the calculation of the total return and average returns was computationally intensive, which is why only 1000 iterations were considered.
The boxplots and statistics show that the average return was stable across all considered portfolios and that the variation in these returns fell with an increased number of assets in the portfolio.Table 2 shows the variation in the average returns across all 500 stocks for portfolios of size 1 to portfolios of size 500.The relatively stable average return and the decreased risk represented by the standard deviation and the minimum and maximum returns across all portfolios was consistent with the free lunch associated with diversification.The larger the portfolio, and thus, the more diversified the portfolio was, the lower the risk.This was achieved with a constant return, and thus, no cost.In other words, since there was no cost due to diversification, diversification was a free lunch in this scenario.
Theoretically, it is not surprising that the average returns were similar for all portfolio sizes, as 1000 random draws of any number of assets will always yield an average return equal to the average of all 500 stocks, i.e., 0.007.The risk fell with increasing portfolio sizes because the assets were generally not perfectly correlated, with correlation coefficients smaller than 1.
Our finding is consistent with Warren Buffett's statement, "diversification is protection against ignorance". 9It seems that diversification indeed provides protection and that the risk is clearly lower for more-diversified portfolios than for less-diversified portfolios at no cost.
However, it is also important to note that the average return under uncertainty was lower than under certainty for all portfolio sizes, except the portfolio with all 500 stocks.Hence, diversification was a free lunch under uncertainty but not a free lunch compared with non-ignorance.

Risk-Estimate Future (Expected) Returns
Table 3 and Figure 2c present the results for a scenario in which investors estimated future returns based on historical returns with different (historical) sample sizes and then formed portfolios accordingly.The historical returns were assumed to be 36 months (3 years), 72 months (6 years), 108 months (9 years), and 144 months (12 years), as presented in panels A, B, C, and D in the table, respectively.
The figure only reports the results for 36 months.The results for 72, 108, and 144 months are displayed in Appendix A in Figure A1.Investors were assumed to use historical returns to predict future returns.The displayed results are based on 1000 draws from the predicted (out-of-sample) sample period.Specifically, the average returns were estimated based on the assumed historical data, e.g., 36 months.The returns were ranked and the assets allocated to the specific portfolios.We then used the future (out-of-sample) returns (in the 36-month case, the out-of-sample was from month 37 to the end of the sample) to calculate the total returns, risk, and minimum and maximum returns over the out-of-sample period., 5, 10, 20, 50, 100, 200, and 500 assets and for different histories of returns and corresponding estimation periods (36, 72, 108, and  This table shows that the average portfolio returns generally fell across all portfolio sizes and for all scenarios.
Remarkably, the order of returns was mainly preserved for the out-of-sample period.While the returns were lower than for the case of certainty, the forecasts still yielded higher returns than for the case of uncertainty and ignorance.
We concluded that diversification came at a cost of lower average returns despite the lower associated risk, which implies that diversification is not a free lunch.

Summary and Concluding Remarks
This study was motivated by the frequently stated claims that diversification is a free lunch.It contributes to the literature with an explicit analysis of this claim and by highlighting the cost of diversification and the role of uncertainty and ignorance.
We demonstrated that diversification generally came at a cost through lower returns, and thus, was not a free lunch.While the risk of diversified portfolios was clearly lower than that of less-diversified or undiversified portfolios, the return was generally also lower.There was only one exception.If the investor cannot know (uncertainty) or does not know (ignorance) the risk and return of stocks and consequently picks stocks randomly, diversification is a free lunch.Besides this exception and special case, "there is no such thing as a free lunch", as proposed by Milton Friedman.We also note that there seems to be no evidence that would support the claim that Harry Markowitz actually said that "diversification is the only free lunch".Furthermore, our findings are consistent with Warren Buffett's "diversification is protection against ignorance".In contrast, full knowledge makes diversification redundant.This may explain why entrepreneurs commonly have relatively concentrated wealth.
While most of our analysis was based on simulations with monthly share prices of all constituents of the S&P500, we also provide theoretical arguments why diversification is not a free lunch.For example, if diversification was a free lunch, it would violate the fundamental positive risk-return relationship in finance.Specifically, if risk can be reduced without a cost, risk and return are not positively aligned.We also show that diversification is a free lunch on the inefficient part of the frontier, which highlights that at least some "free lunch" claims may be based on the implicit assumption of inefficient markets.
We concluded that diversification is only a free lunch for investors who are ignorant or uninformed and for inefficient markets.This study can further help to explain why investors tend to use only a small number of assets to diversify and not a larger number of assets, which is commonly referred to as the "diversification puzzle".Transaction costs, and particularly the need to regularly rebalance a diversified portfolio, may also explain why investors hold rather undiversified portfolios.If such transaction costs were included, the case against the free lunch of diversification argument would be even stronger.
It may also be interesting to study the benefits and costs of diversification or diversity in other fields.For example, is the trade-off between returns and risk reduction for more diverse portfolios also evident in the returns and risks of biodiversity or diversity of the work force?We leave these questions for future research.

Conflicts of Interest:
The author declares no conflicts of interest.
Appendix A Table A1.Descriptive statistics of results as reported in Figure 2a-c.This table shows the average returns, SDs, minimums, maximums, and Sharpe ratios for portfolios with 1,5,10,20,50,100,200, and 500 assets and 10,000 draws.
Figure1.Efficient frontier and free lunch on an inefficient frontier.The graph displays the frontier of two assets with a correlation of zero (black line) and the frontier of two assets with similar risk and return with a correlation of one (red line).The inefficient part of the frontier (below the dashed horizontal line) is an area where the risk-return relationship is negative and where the risk can be reduced and the return can be enhanced, consistent with the free lunch of diversification.However, to achieve such an outcome, inefficient allocations must be entertained.The efficient frontier above the dashed line displays a positive risk-return relationship and does not entail the free lunch of diversification.
right−hand axis) (c) Risk-36 months Figure 2. Effect of diversification with no rebalancing.The figure shows the mean returns and the distributions of returns (vertical axis) for portfolios that comprised 1,5, 10, 20, 50, 100, 200, and 500 stocks.The graphs are based on 10,000 draws of monthly returns in each scenario (certainty, uncertainty, and risk).

Funding:
This research received no external funding.Data Availability Statement: Data is contained within the article.

Figure A1 .
Figure A1.Effect of diversification-risk scenario with no rebalancing.The figure shows the mean returns and the distributions of returns (vertical axis) for portfolios that comprised1, 5, 10, 20, 50, 100,  200, and 500 stocks.

Table 1 .
Certainty-expected returns rule.This table shows the average returns and risks of portfolios with 1, 5, 10, 20, 50, 100, 200, and 500 assets.Panel (A) shows the results of portfolios with no rebalancing and panel (B) shows the results of portfolios with monthly rebalancing.

Table 3 .
Risk-estimate future returns.This table shows the average returns, SDs, minimums, maximums, and Sharpe ratios for portfolios of sizes 1 144 months) to estimate future returns.Portfolios were not rebalanced over the investment horizon.