What Is The S&P 500's "Secret Sauce"?

[JustGotScammed]

We have all seen the data showing how few money managers and individuals are able to surpass the performance of the S&P 500. What is interesting, though, is that each component of the index is able to be purchased directly. What is it that is so special about grouping 500ish stocks approximately by market weight that is hard to beat? Theoretically, we should be able to isolate which aspect within the index delivers the results and makes it hard to beat, and then amend our strategies to own those stocks directly.

Maybe putting it differently, we often hear that 20% of a group delivers 80% of the results (Pareto). What is the "20%" that makes up the secret sauce of the S&P 500? If, Nvidia, Amazon, Microsoft, Apple, Meta, and Google or whatever weren't in the index for whatever reason, would it be bad? Are they the secret sauce?

[delamer]

Your theory appears to assume that the secret sauce (highest performing sector/companies) is consistent over long periods of time.

It isn’t and it isn’t predictable what will drive growth in the future. That’s why it’s better to buy the whole thing.

[gammalaser]

The Secret Sauce is the needle in the haystack. Don't bother looking for the needle. Buy the haystack.

[JustGotScammed]

Your theory appears to assume that the secret sauce (highest performing sector/companies) is consistent over long periods of time.

It isn’t and it isn’t predictable what will drive growth in the future. That’s why it’s better to buy the whole thing.

Why ~500 then? Why not Top 50? Top 3500?

[delamer]

Your theory appears to assume that the secret sauce (highest performing sector/companies) is consistent over long periods of time.

It isn’t and it isn’t predictable what will drive growth in the future. That’s why it’s better to buy the whole thing.

Why ~500 then? Why not Top 50? Top 3500?

3500 means you have a lot of companies in the index that are so small that they won’t affect a cap-weighted overall return.

50 may not be enough for adequate diversification.

It could be that the top 173 would have been optimal historically. Maybe someone’s studied it?

I can’t specifically defend the 500.

[BirdFood]

Theoretically, we should be able to isolate which aspect within the index delivers the results and makes it hard to beat, and then amend our strategies to own those stocks directly.

All the people managing actively managed funds are doing so on the assumption that they can figure out, and implement, the secret sauce.

And they’re usually wrong. Pretty close to always wrong.

So it’s not that no one has thought of the idea of secret sauce. They just can’t find it.

[alex_686]

Why ~500 then? Why not Top 50? Top 3500?

Nice round number. You could do the computations in a reasonable time on a 1960s vintage computer.

Are there better large cap indexes? Yes. CRISP is better theoretically. However S&P 500 has the first mover advantage. It has been around a long time and thus dominates.

[alex_686]

50 may not be enough for adequate diversification.

Historically you could get to 80% of the S&P 500 diversification with something between the high 20s to high 30s.

So less than 50.

[JustGotScammed]

Theoretically, we should be able to isolate which aspect within the index delivers the results and makes it hard to beat, and then amend our strategies to own those stocks directly.

All the people managing actively managed funds are doing so on the assumption that they can figure out, and implement, the secret sauce.

And they’re usually wrong. Pretty close to always wrong.

So it’s not that no one has thought of the idea of secret sauce. They just can’t find it.

I guess that's my question: (1) why does the index have it, but (2) the investors looking for it do not.

[mrspock]

Your theory appears to assume that the secret sauce (highest performing sector/companies) is consistent over long periods of time.

It isn’t and it isn’t predictable what will drive growth in the future. That’s why it’s better to buy the whole thing.

Why ~500 then? Why not Top 50? Top 3500?

Top 50, top 100, top 500+ all pretty correlated -> https://www.portfoliovisualizer.com/bac ... vJdTAkY7oc

Note: This doesn't backtest nearly far enough for my liking, but you get the general idea.

You'd do just fine with any of these, though top 50 would have more volatility (24% draw down vs. 18% for S&P 500). Folks by S&P 500 or similar funds because they are highly liquid and strike a good balance between risk and returns. If you want to take more risk, by all means, Top 50 would be a perfectly fine approach IMO. But I'd first ask: If you don't need to take the extra risk to achieve your financial goals, why take them?

[gougou]

Secret sauce is that there are a few stocks that greatly outperform the rest. For example, if you have an SP500 where 1 stock delivers 100% of the returns and 499 stocks are flat, and you have 500 stock-pickers who can pick 1 stock each, then the index has outperformed 499/500 stock-pickers.

[alex_686]

I guess that's my question: (1) why does the index have it, but (2) the investors looking for it do not.

While 500 is somewhat a round and arbitrary number it does approximate the market.

By definition, about 50% of active managers will do better, before fees - and 50% will do worse.

Why does the market in general produce returns? In part due to the Equity Risk Premium- risk assets should earn more than government bonds. In part because of the Efficient Market Hypothesis.

EMH basically says that because there are many active intelligent agents out there the market is fairly priced.

As such, I am a modest advocate of active management. There are niches and strategies that can pay off. But they are not simple nor foolproof.

As such, many active investors “have it”. Just not enough to differentiate themselves.

As a analogy, all NFL players “have it”. Are you smart enough to pick a winning team for the superbowl in August consistently for 8 straight years? Can you differentiate between luck and skill?

[John Z]

One big factor are the expenses. Having active managers trying to compete with an index hardly ever exceeds pure index performance. In the longer term even a slight increase in expenses sort of compounds and drags down the real performance longer term...

[delamer]

50 may not be enough for adequate diversification.

Historically you could get to 80% of the S&P 500 diversification with something between the high 20s to high 30s.

So less than 50.

I’m not clear what 80% of the diversification means.

[alex_686]

Historically you could get to 80% of the S&P 500 diversification with something between the high 20s to high 30s.

So less than 50.

I’m not clear what 80% of the diversification means.

You can get a portfolio where returns get a statistical power of 80% explication with the same volatility.

You have to add another 250 stocks to get to 90%.

So from a pragmatic exercise, 35 would be enough.

This was my daily work back in the 90s, back when adding a extra stock was kind of expensive. The rise of the magnificent 7 I suspect would modestly push the numbers up.

[arcticpineapplecorp.]

We have all seen the data showing how few money managers and individuals are able to surpass the performance of the S&P 500. What is interesting, though, is that each component of the index is able to be purchased directly. What is it that is so special about grouping 500ish stocks approximately by market weight that is hard to beat? Theoretically, we should be able to isolate which aspect within the index delivers the results and makes it hard to beat, and then amend our strategies to own those stocks directly.

Allan Roth wrote: The Case Against S&P500 Index Funds. After that article came out, Allan wrote a followup, John C. Bogle on the S&P 500 vs. the Total Stock Market in which he wrote:

In that column, I noted two flaws in this index fund and pointed to a better way, namely a Total US index fund. I received a note in the mail this week from - wait for it - John C. Bogle, the person who brought both of these index funds to the investing public. He also gave me permission to publish his note.

Hi Allan!
Of course you are right about these two flaws in the S&P 500, and I agree with your conclusion about "a better way."
But we're left to explain why, over history from 1928 on, the annual return in the S&P 500 has been 10.4 percent while the return on the Total Stock Market has been 10.2 percent (see the Little Book of Common Sense Investing).
Food for thought.
Best,
Jack

I'll confess, as any Boglehead will tell you, that it was pretty thrilling to get a hand written note from Jack Bogle. As always, he raises an interesting point by stating the S&P has outpaced the total U.S. stock market since 1928 by 0.2 percent annually. MoneyWatch writer, Nathan Hale, also pointed out a similar phenomenon. Food for thought, to be sure, and here's what it got me to thinking on the issue.

History of the S&P 500
There is no doubt in my mind that Mr. Bogle is giving the best and most unbiased view of S&P returns since 1928. Unfortunately, the data it's based upon is imperfect at best, since accurate data is hard to come by the older it becomes.

In 1928, the S&P index was comprised of only 90 companies. It didn't get to its current state of 500 companies until 1957. So comparing a total index with data that may contain inaccuracies to the S&P index, which contained a vastly different number of companies in that period, may yield some unintended consequences in the outcome...

So I don't know that there's anything inherent about 500 stocks. In fact you can see the total market has done either better than the S&P500 (depending on the timeframe) as Allan wrote in his article, as opposed to slightly worse as Jack wrote for a different time period. Basically the returns of both have been within spitting distance long term. Not to mention the fact that Jack eventually came to recommend the total stock market index fund rather than his own S&P500 index fund because he said the total market index fund was more "representative" of the total market. It's what caused me to switch from S&P500 to total stock market index fund (thank you Jack).

I'd say it's the market weighting that matters. When the positive skewness that Bessembinder writes about occurs, you get that growth in your portfolio. And when the others underperform, they become a smaller impact on your portfolio. Out of 25,300 companies since 1926...

The 1,092 top-performing companies, slightly more than 4% of the total, account for all of the net wealth creation. That is, the remaining 96% of companies whose common stock has appeared in the CRSP data collectively generate lifetime dollar gains that matched gains on one-month Treasury bills...

These results highlight the practical importance of positive skewness in the distribution of returns. The skewness in long horizon returns is attributable in part to the fact that monthly returns are skewed. It also reflects the possibly underappreciated fact that the compounding of
random returns induces positive skewness in the multi-period return distribution, and more so for stocks with more volatile returns. Researchers often assume that returns conform at least approximately to the normal distribution. However, even if returns were distributed normally at one-period horizon, the effects of compounding imply positive skewness at any longer horizon.

[BirdFood]

All the people managing actively managed funds are doing so on the assumption that they can figure out, and implement, the secret sauce.

And they’re usually wrong. Pretty close to always wrong.

So it’s not that no one has thought of the idea of secret sauce. They just can’t find it.

I guess that's my question: (1) why does the index have it, but (2) the investors looking for it do not.

I'm tentatively assuming that the "secret sauce" is a set of criteria that assembles a group of stocks that is likely to contain a subset that does extremely well. And that, therefore, the S&P 500 is the secret sauce.

Edited to add: That is, admittedly, simplistic. My main thought is that if there were a way to find the best stocks, more active managers would be succeeding.

[alex_686]

l
I'm tentatively assuming that the "secret sauce" is a set of criteria that assembles a group of stocks that is likely to contain a subset that does extremely well. And that, therefore, the S&P 500 is the secret sauce.

Yes, the secret is to pick the largest 500 stocks, which, depending on the way you slice things, is about 80 to 90% of the market.

Make minor adjustments for liquidity and free float - you don’t want the system to be gamed.

Honestly, not that much of a secret. The are lots of off-brand market cap indexes that do just as well.

[MedEngineer]

Why ~500 then? Why not Top 50? Top 3500?

Nice round number. You could do the computations in a reasonable time on a 1960s vintage computer.

If I recall the INDU (Dow Jones Industrial) was 30 stocks so you could use the 1896 vintage computer, pencil & paper

[Beensabu]

the secret sauce of the S&P 500?

buy and hold and don't rebalance

literally

[Thesaints]

Nice round number. You could do the computations in a reasonable time on a 1960s vintage computer.

If I recall the INDU (Dow Jones Industrial) was 30 stocks so you could use the 1896 vintage computer, pencil & paper

INDU is equal weight.
S&P chose 500 (but chose 400 for the midcap index, and 600 for the small cap) because it reduces the risk of leaving something out, that ends up generating outsized growth. Remember that inclusion in the index is not strictly by market cap only, which is done to maintain components stable and avoid friction from selling and buying.
Also 500 was kind of a soft spot in terms of transaction cost. More recent indexes feel less that problem and there you have the Russell 1000.

Today, with modern markets, at close to zero transaction cost, almost any number works and in fact the S&P7 greatly outperformed the S&P500 in the more recent past, to a point that the S&P21 covers half of the capitalization of the S&P500, i.e. ignoring any of the other 479 companies won’t change your return, one way or another

[alex_686]

Nice round number. You could do the computations in a reasonable time on a 1960s vintage computer.

If I recall the INDU (Dow Jones Industrial) was 30 stocks so you could use the 1896 vintage computer, pencil & paper

Critically, it was equal weighted back then. So yeah, adding 30 prices together was about the computational load of a chalkboard.

[alex_686]

Remember that inclusion in the index is not strictly by market cap only,

I think it is free float market rate? Can you point to a missing stock or other distortion?

Berkshire was a issue do to liquidity. Tesla was delayed due to dot.com anti churning laws. Spotify because it is not exactly a American company.

The rules are pretty clear and objective.

[Thesaints]

Sure. But the committee has to meet, decide a date, etc.
It would be crazy otherwise, with companies between 450 and 550 being one day in and the next out.

[gavinsiu]

My impression is that S&P500 is a sort of darwinian system where winner rise to the top and increase in weight, while loser shrink and disappear. Because is cap weighted, transaction is low compare to say a equal weighted fund. In addition, the fees are typically low so that the investor isn't facing headwind from expense.

Question do equal weight index do better or does cap weighted index do better?

[FundQuant]

You really only need about 25-50 holdings to get such diverse equity exposure that you'll roughly mimic the total market index. So the "secret sauce" is that it holds all the factors, all the sectors that always outperform by the most. You're not tilting to certain factors or sectors because you own them all so you capture all those outsized moves and allow the big winners to win via market cap rebalancing.

Most active managers are trying to time the sectors or factor that win and that's a losing battle in the long run.

[Kbg]

My impression is that S&P500 is a sort of darwinian system where winner rise to the top and increase in weight, while loser shrink and disappear. Because is cap weighted, transaction is low compare to say a equal weighted fund. In addition, the fees are typically low so that the investor isn't facing headwind from expense.

Question do equal weight index do better or does cap weighted index do better?

RSP vs SPY since RSP's inception: 10.98 vs 11.01 CAGR. RSP slightly more volatile, slightly more risky. Pretty much in the noise.

[babystep]

Your theory appears to assume that the secret sauce (highest performing sector/companies) is consistent over long periods of time.

It isn’t and it isn’t predictable what will drive growth in the future. That’s why it’s better to buy the whole thing.

Why ~500 then? Why not Top 50? Top 3500?

500 number itself is historical.

Idea is to cover the most, if not all, of the investable stock market in terms of the market cap. You will have a better coverage with 3500 and much less coverage with 50.

S&P 500 covers like 80-90% of the market. Why is that good enough?
1) 80-90% coverage is pretty good
2) Smaller cap stocks which are missing in the S&P 500 are not beating the S&P 500 so it is not even making any difference compared to holding say 3500 stocks. If rest of the missing 3000 stocks start beating the 500 then you will see S&P 500 slightly under-perform but notice the weight of those 3000 is very low so they have to outperform by large to make difference.

[alluringreality]

I'd say it's the market weighting that matters. When the positive skewness that Bessembinder writes about occurs, you get that growth in your portfolio.

The paper's methodology is not consistent with either market weight or equal weight, but both market weight and equal weight were used as references. Generally market weight and equal weight have outperformed the other across different time periods historically. If only a small portion of stocks outperform across a particular extended period, putting more assets into a limited number of holdings can potentially amount to relative risk going forward. The paper makes comments along the following lines, and the actual numbers tend to have changed depending on selected time period, such as shown in Table 3B.

The effects of positive skewness in the distribution of buy-and-hold returns can also be observed when comparing individual stocks returns to returns on market-wide benchmarks. At the decade horizon, only 37.3% of stocks have buy-and-hold returns that exceed the accumulated return to the value-weighted portfolio of all common stocks and just 33.6% outperform the accumulated return to the equal-weighted portfolio of all common stocks.

The paper makes a number of comments on skewness in relation to historical time horizon, and personally I find it interesting how skewness has varied depending on investment horizon, such as shown in the graphic from Petajisto included with the following.
https://awealthofcommonsense.com/2023/1 ... ck-market/

[Northern Flicker]

Your theory appears to assume that the secret sauce (highest performing sector/companies) is consistent over long periods of time.

It isn’t and it isn’t predictable what will drive growth in the future. That’s why it’s better to buy the whole thing.

Why ~500 then? Why not Top 50? Top 3500?

The secret sauce of the S&P500 and some other indices is that they can be implemented at low cost. Per SPIVA (S&P index vs active) data collected by S&P, about half of the portfolios of active managers typically beat their reference index in a given year. But that is before costs are factored in. Net of costs, a high percentage of active managers underperform in most years.

Many advocate for holding the total market, which is the portfolio indicated by analysis using the Capital Asset Pricing Model, a theoretical underpinning of index fund investing. Under CAPM, the market portfolio diversifies away all alpha and delivers the market return.

The reason the S&P500 does not just hold the 500 largest stocks is that S&P tries to keep the sector weighting reasonably close to the total market so that the S&P500 does not deviate too much from the total market portfolio.

[ReadyOrNot]

By some hypotheses, the market should approach the optimum. So anything that closely approximates the market should be close to the optimum. Anything that does not approximate the market has a tough time beating the market. So an active trader who differs
from the market is unlikely to do better. The S&P 500 approximates the total US market pretty well, so is likely to be close to the optimum.

Why suppose a secret sauce? Suppose you observe healthy people eating real food (minimally processed whole food); not too much; mostly plants. You decide to replace your food with meal replacement shakes which you can make with any refined, processed food supplements you want. You can leave out any unhealthy ingredients. What secret sauce would you add to make these meal replacement shakes better than real food?

[Lawrence of Suburbia]

Surely some credit should go to the unnamed(?) gnomes at S&P who decide now & again who's in the 500, and who's out?

[heyyou]

The 500 limit is a leftover from when adding machines were mechanical, before electric ones. The operator pulled the arm for each addition. Standard and Poore were two guys doing data accumulation who were publishing stock market info (i.e. total volume up or down, relative to yesterday's numbers), but when the market had grown to 500 businesses, it became difficult to accurately add all those closing stock prices between the market closing time and the evening news, so the smallest businesses in the stock market were then left out. The missing ones were such a small portion of the total dollar amount of transactions that they were inconsequential.

Per Google search, the Dow Jones 30 stock index is from 1896, and the S&P 500 is from 1957.

[Thesaints]

…but there were more than one thousand listed companies in 1957…

[alex_686]

…but there were more than one thousand listed companies in 1957…

There are problems with many of those companies. Many didn’t trade daily, so your index is going to contain stale data, so the index is going to have bad data. Further, with such low liquidity it would be expensive for multiple managers to pop in and out as external cash flows hit the market.

As a example, Berkshire A shares often had a daily trading value of under 10 in the 90s. The B shares were not much better. What manager would want to follow a index that they couldn’t replicate?

[deltaneutral83]

I can't recall where I saw this but it was something like 90% of companies listed don't make money and 94% don't beat treasuries. I think that means that 6 out of 100 publicly listed companies are responsible for the gains above treasuries. I'd be interested to see an update on that from the past 100 years if that data is available. Just a wild concept, it's even wild to me that I can buy every publicly traded company (or pretty much for all intents and purposes) in the US and pay basically nothing and sit back and read quarterly statements for 40-80 years (many younger people today will live to be over 100 I bet).

[Capster1]

There are lots of various opinions about how many stocks are needed to be diversified. However, I’ve seen many studies saying 50 stocks is enough (I don’t agree).
However, suppose that was true for large caps alone; why couldn’t you just buy XLG (Top 50 S&P stocks) and be done with it? It certainly captured the majority of the markets gains the last few years.
That’s really what they are talking about, how many stocks does it take to capture the majority of the returns of a specified index.

To me, it’s obvious it isn’t diversified overall because it’s nothing but large cap and heavily tech. So no diversification by size nor sector really…but if the goal is to match the S&P 500? In this case, to capture the secret sauce.

[nisiprius]

There's no "secret sauce," it's just a decent approximation to the total market. There's no magic in 500.

By 1957, electronic computers were common in larger businesses that could afford them. The choice of 500 stocks was a compromise based on a wish to calculate the index hourly, and a (correct) belief that 500 stocks was a good approximation to the total market. It was hoped to include 90% of the market by cap weight, and actually included 94%.

On the one hand, the selection rules for picking the 500, which are not strictly mechanical, impose a tiny tilt toward the quality factor, which might give it a tiny boost. On the other hand, the S&P 500 is so closely followed, and has so many dollars indexed to it, that the price of a stock gets a boost when it is added to the index, and therefore it is possible to front-run it by guessing which stocks will be added and when--which means the actors who do that are taking money away indexers and putting a tiny drag on the index.

Notice that the performance difference between funds tracking the S&P 500 (VFINX), the Vanguard Large-cap Index Fund (VLCAX, 493 stocks, CRSP US Large-cap Index), the Schwab 1000 Index Fund (SNXFX, 1000 stocks), and the Russell 1000 (VONE, 1000 stocks) is very small--and that the S&P 500 fund was not the top performer.

Source

[alluringreality]

I can't recall where I saw this but it was something like 90% of companies listed don't make money and 94% don't beat treasuries. I think that means that 6 out of 100 publicly listed companies are responsible for the gains above treasuries. I'd be interested to see an update on that from the past 100 years if that data is available. Just a wild concept, it's even wild to me that I can buy every publicly traded company (or pretty much for all intents and purposes) in the US and pay basically nothing and sit back and read quarterly statements for 40-80 years (many younger people today will live to be over 100 I bet).

How many stocks happen to outperform bills, or other references including market weight, likely depends on selected timeframes. The very long, historical numbers often relate with the following. There the numbers are often presented as aggregates. It takes a large amount of modestly-winning stocks to backfill all the multi-decade losers, so if just beating bills was the criteria their numbers would generally increase, at least relative to the 90% and 94% guesses. For example the infographic on the original paper suggests 38% of stocks beat Treasury bill returns by just moderate amounts across 1926 to 2017, which basically made up for the stated 58% that underperformed bills. It looks like there's a version covering the CRSP database from December 1925 to December 2023 published last year.
https://wpcarey.asu.edu/department-fina ... sury-bills

[deltaneutral83]

I can't recall where I saw this but it was something like 90% of companies listed don't make money and 94% don't beat treasuries. I think that means that 6 out of 100 publicly listed companies are responsible for the gains above treasuries. I'd be interested to see an update on that from the past 100 years if that data is available. Just a wild concept, it's even wild to me that I can buy every publicly traded company (or pretty much for all intents and purposes) in the US and pay basically nothing and sit back and read quarterly statements for 40-80 years (many younger people today will live to be over 100 I bet).

How many stocks happen to outperform bills, or other references including market weight, likely depends on selected timeframes. The very long, historical numbers often relate with the following. There the numbers are often presented as aggregates. It takes a large amount of modestly-winning stocks to backfill all the multi-decade losers, so if just beating bills was the criteria their numbers would generally increase, at least relative to the 90% and 94% guesses. For example the infographic on the original paper suggests 38% of stocks beat Treasury bill returns by just moderate amounts across 1926 to 2017, which basically made up for the stated 58% that underperformed bills. It looks like there's a version covering the CRSP database from December 1925 to December 2023 published last year.
https://wpcarey.asu.edu/department-fina ... sury-bills

Thank you, yes time frames on the treasuries would have a huge impact. I still think it's notable that on any scale, treasuries are going to return more with exponentially less risk (in fact, considered to be "risk free rate" often times) than well over 60% of listed equities. That has to give people pause for picking equities with publicly available information on the underlying.

[KEotSK66]

I don't know the mathematical details nor the method but the S&P 500 is a "representation" of the US economy.

[watchnerd]

We have all seen the data showing how few money managers and individuals are able to surpass the performance of the S&P 500. What is interesting, though, is that each component of the index is able to be purchased directly. What is it that is so special about grouping 500ish stocks approximately by market weight that is hard to beat? Theoretically, we should be able to isolate which aspect within the index delivers the results and makes it hard to beat, and then amend our strategies to own those stocks directly.

Maybe putting it differently, we often hear that 20% of a group delivers 80% of the results (Pareto). What is the "20%" that makes up the secret sauce of the S&P 500? If, Nvidia, Amazon, Microsoft, Apple, Meta, and Google or whatever weren't in the index for whatever reason, would it be bad? Are they the secret sauce?

The 500 is just a historical artifact.

It was the biggest number of stocks computers could track back when it was first invented.

[bertilak]

We have all seen the data showing how few money managers and individuals are able to surpass the performance of the S&P 500. What is interesting, though, is that each component of the index is able to be purchased directly. What is it that is so special about grouping 500ish stocks approximately by market weight that is hard to beat? Theoretically, we should be able to isolate which aspect within the index delivers the results and makes it hard to beat, and then amend our strategies to own those stocks directly.

Maybe putting it differently, we often hear that 20% of a group delivers 80% of the results (Pareto). What is the "20%" that makes up the secret sauce of the S&P 500? If, Nvidia, Amazon, Microsoft, Apple, Meta, and Google or whatever weren't in the index for whatever reason, would it be bad? Are they the secret sauce?

This is a job for GOOGLE! Try things like "how is the S&P 500 constructed?" and "goals of the S&P 500?"

The second one turned up an interesting Investopedia article; "https://www.investopedia.com/terms/s/sp500.asp." That article includes, among other things, a statement that may help answer your question about secret sauce:

"The S&P 500 is considered one of the best gauges of large U.S. stocks and even the entire equities market because of its depth and diversity. The "secret sauce" is summed up by Bogle: "don't try to find the needle in the haystack, buy the whole haystack. The S&P 500 is most of that haystack and is intended to represent the whole haystack. History shows it has done a pretty good job of that.

GOOGLE around for more.

[AlwaysLearningMore]

We have all seen the data showing how few money managers and individuals are able to surpass the performance of the S&P 500. What is interesting, though, is that each component of the index is able to be purchased directly. What is it that is so special about grouping 500ish stocks approximately by market weight that is hard to beat? Theoretically, we should be able to isolate which aspect within the index delivers the results and makes it hard to beat, and then amend our strategies to own those stocks directly.

Maybe putting it differently, we often hear that 20% of a group delivers 80% of the results (Pareto). What is the "20%" that makes up the secret sauce of the S&P 500? If, Nvidia, Amazon, Microsoft, Apple, Meta, and Google or whatever weren't in the index for whatever reason, would it be bad? Are they the secret sauce?

The 500 is just a historical artifact.

It was the biggest number of stocks computers could track back when it was first invented.

Nisi has brought this up a number of times

[AlwaysLearningMore]

IIRC companies included in the S&P 500 have to meet certain criteria, unlike a total stock market fund. For example:

Profitability: Positive earnings in the most recent quarter and sum of the last four consecutive quarters.
https://www.spglobal.com/spdji/en/docum ... ndices.pdf

[watchnerd]

The 500 is just a historical artifact.

It was the biggest number of stocks computers could track back when it was first invented.

Nisi has brought this up a number of times

As have multiple economic historians.

[alex_686]

IIRC companies included in the S&P 500 have to meet certain criteria, unlike a total stock market fund. For example:

Profitability: Positive earnings in the most recent quarter and sum of the last four consecutive quarters.
https://www.spglobal.com/spdji/en/docum ... ndices.pdf

That is only for inclusion. This filter was added after the dot.com bust. Back then a stock would get a 7% in value on the announcement that it was being added. There was a fair amount of stock manipulation and abuse around this.

After inclusion you never have to show profit again - technically speaking.

As such, this is a fairly light filter. It is also illustrative of the difference between a total index and a investable index.

[AlwaysLearningMore]

Nisi has brought this up a number of times

As have multiple economic historians.

Nice to hear that.

[AlwaysLearningMore]

IIRC companies included in the S&P 500 have to meet certain criteria, unlike a total stock market fund. For example:

Profitability: Positive earnings in the most recent quarter and sum of the last four consecutive quarters.
https://www.spglobal.com/spdji/en/docum ... ndices.pdf

That is only for inclusion. This filter was added after the dot.com bust. Back then a stock would get a 7% in value on the announcement that it was being added. There was a fair amount of stock manipulation and abuse around this.

After inclusion you never have to show profit again - technically speaking.

As such, this is a fairly light filter. It is also illustrative of the difference between a total index and a investable index.

Thank you for the information.

[alex_686]

Thank you for the information.

To extend a bit, I believe that the S&P 500 represents the largest 500 US public companies.

I am happy to go into the weeds. There are nuances and subjective opinions. The nuances are just the pragmatic respons to a complex world. I equate subjective to that of a sports referee- judgement calls must be made - but are the calls consistent?

For a bit more context, I have sat in a room of accountants reading prospects trying to determine if something is a stock or a bond and in what country the company exists in. Its harder than you think.