What Is The S&P 500's "Secret Sauce"?

[Northern Flicker]

Surely some credit should go to the unnamed(?) gnomes at S&P who decide now & again who's in the 500, and who's out?

These are not active decisions.

[Thesaints]

It was the biggest number of stocks computers could track back when it was first invented.

Doesn’t sound right, it was 1957, not 1857. Reading and adding up 500 integers didn’t take too much time.

[Thesaints]

That is only for inclusion. This filter was added after the dot.com bust.

Then how do you justify the fact that the companies on the tail end of the 500 were not replaced very frequently before 2001 ? Surely the difference in capitalization between #500 and #501 is less than the daily volatility.

[vfinx]

Surely some credit should go to the unnamed(?) gnomes at S&P who decide now & again who's in the 500, and who's out?

These are not active decisions.

This piqued my interest and I happened upon this article. I do not claim the source is credible.

Inside the S&P 500: An Active Committee

The S&P 500 is maintained by a committee of market professionals. We publish a detailed methodology document which includes guidelines for selecting stocks and other changes to the index. Unlike many other S&P Dow Jones Indices and the majority of indices offered by other index providers, there are no rigid or absolute rules for the S&P 500; the Index Committee have some discretion in selecting stocks or responding to market events.

People ask why we have a committee when other index providers manage with a rule book. Larry Swedroe, writing on ETF.COM, unearthed an old argument that having the Index Committee means that the S&P 500 is actively managed; he concluded that this isn’t a problem. Some years ago Bill Miller, then a well-known Legg Mason portfolio manager with a stellar record of beating the market, noted that the S&P 500’s track record of being very hard to beat suggested that active management can succeed and that the Index Committee were actually good active managers.

[dcabler]

Surely some credit should go to the unnamed(?) gnomes at S&P who decide now & again who's in the 500, and who's out?

These are not active decisions.

From: https://www.spglobal.com/spdji/tc/docum ... ndices.pdf

Page 12:
"Constituent Selection. Constituent selection is at the discretion of the Index Committee and is based on the eligibility criteria."

Cheers.

[alex_686]

These are not active decisions.

From: https://www.spglobal.com/spdji/tc/docum ... ndices.pdf

Page 12:
"Constituent Selection. Constituent selection is at the discretion of the Index Committee and is based on the eligibility criteria."

Cheers.

Yes, all indexes employ a index committee to handle subjective judgment calls. This falls into the same catalog as strike zones or roughing the quarterback. In this case you want actual large US companies, not companies that are technically large US companies. You don’t want to process to be abused or manipulated.

Quick question - is Spotify a US or European company? S&P and CRSP classify it differently.

[dcabler]

From: https://www.spglobal.com/spdji/tc/docum ... ndices.pdf

Page 12:
"Constituent Selection. Constituent selection is at the discretion of the Index Committee and is based on the eligibility criteria."

Cheers.

Yes, all indexes employ a index committee to handle subjective judgment calls. This falls into the same catalog as strike zones or roughing the quarterback. In this case you want actual large US companies, not companies that are technically large US companies. You don’t want to process to be abused or manipulated.

To that end, this article that describes what they don't do and what they actually do when big events happen: https://www.indexologyblog.com/2014/08/ ... d-in-2014/

Cheers.

[nisiprius]

...To extend a bit, I believe that the S&P 500 represents the largest 500 US public companies...

No, it doesn't. It omits some large-caps and includes some mid-caps.

Looking at VEXAX, the Extended Market Index Fund, which includes all the stocks that are not in the S&P 500, the three largest holdings as of 11/30/2024 are as follows. After each one, I put the market cap as determined from a simple web search right now.

Marvell Technology Inc (MRVL), $98.22 billion
Apollo Global Management Inc (APO) $93.23 billion
AppLovin Corp (APP) $106.88 billion

In VFIAX, the 500 Index fund, there are three holdings at the bottom that look anomalous so I'll ignore them. Going up just a little higher, I see:

Walgreens Boots Alliance Inc (WBA) $10.6 billion
Paramount Global (PARA) $7.47 billion
Fox Corp, FOX $21.46 billion.

So the S&P 500 spurns stocks with market caps of around $100 billion, but reaches down to include stocks with market caps of $7 to $20 billion. I don't think that's rounding error or fluctuation.

And the language S&P has used to describe the S&P 500 index is nuanced, and I don't think they have ever suggested that it is an attempt to include the 500 largest stocks. Vanguard uses the easily-misinterpreted language "The fund offers exposure to 500 of the largest U.S. companies." That's not "THE 500 largest." In fact it can be interpreted to say it does not include all of the 500 largest companies.

[GoldStar]

Your theory appears to assume that the secret sauce (highest performing sector/companies) is consistent over long periods of time.

It isn’t and it isn’t predictable what will drive growth in the future. That’s why it’s better to buy the whole thing.

Why ~500 then? Why not Top 50? Top 3500?

Compare VFIAX (s&p500) to VTSAX (total market - around 3600 holding I believe) and they track nearly identical.
Many of us hold VTSAX/VTI or similar to capture more of the market but with market weighting it barely makes a difference.

[Elysium]

Maybe putting it differently, we often hear that 20% of a group delivers 80% of the results (Pareto). What is the "20%" that makes up the secret sauce of the S&P 500? If, Nvidia, Amazon, Microsoft, Apple, Meta, and Google or whatever weren't in the index for whatever reason, would it be bad? Are they the secret sauce?

The S&P 500 index ultimately is the market index, although it is hand selected by a committee rather than rule based program, the rules are still same just that it takes a little bit extra time for the committee to add that stock, but not too long enough to create a void. For instance, Tesla was not initially part of the index, however as it started rising to the top of the Total US Mkt index, the S&P 500 decided to add it although it still didn't meet a few of the conditions for inclusion, such as 4 quarters of positive earnings (if memory serves right). Therefore, the initial exclusion of Tesla was not a problem as it was only a smaller share of index, not enough to create a difference, and as it rose to become more of an influence it got added.

Setting that aside for context, why then it is hard to beat he index? it's by design hard to beat the market when they are on a tear like we have witnessed in recent past, where the gains are all driven by top 10 companies. Because the top 10 companies also get a large weight as they continue going up in price, they influence the returns more heavily on the positive side, making it tough for money managers unless they are also copying the weight. Money managers often look at fundamentals and hold back on their weights, the index doesn't care. For instance, 2024 was one of the toughest year for money managers as most of them trailed the index, due to weighting from one or two stocks, NVDA and later TSLA which was driving the gains. Even if any money manager held these, but not in the same weight as index then they trailed.

The reverse will happen when markets will tank and then go sideways for prolonged periods without leadership, may be then low p/e or dividend paying companies or mid/small companies with lesser weighting in index will have higher returns, that's when money managers with different selections will find it easier to match or beat the index. The lost decade from 2000-09 is an example when value, mid cap, and small cap stocks were able to beat S&P 500 index. That cycle will happen again, and we will not hear these questions, then the question will be how to be more "diversified"

[grok87]

The S&P 500 index ultimately is the market index, although it is hand selected by a committee rather than rule based program, the rules are still same just that it takes a little bit extra time for the committee to add that stock, but not too long enough to create a void. For instance, Tesla was not initially part of the index, however as it started rising to the top of the Total US Mkt index, the S&P 500 decided to add it although it still didn't meet a few of the conditions for inclusion, such as 4 quarters of positive earnings (if memory serves right). Therefore, the initial exclusion of Tesla was not a problem as it was only a smaller share of index, not enough to create a difference, and as it rose to become more of an influence it got added.

yes i think you are remembering correctly

"To be eligible for S&P 500 index inclusion, a company should be a U.S. company, have a market capitalization of at least $xxx, be highly liquid, and have a public float of at least 10% of its shares outstanding. The company must also be profitable in its most recent quarter’s earnings, and the sum of its trailing four consecutive quarters’ earnings must be positive. "

[Carguy85]

Well VtSAX has about 3800 holdings and has almost identical returns to the sp500. Sometimes slightly more, sometimes slightly less. I noticed a year or so ago it seemed it was higher when tech was on a run.

[Carguy85]

Well VtSAX has about 3800 holdings and has almost identical returns to the sp500. Sometimes slightly more, sometimes slightly less. I noticed a year or so ago it seemed it was higher returns than SP500 when tech was on a run.

[nisiprius]

It was the biggest number of stocks computers could track back when it was first invented.

Doesn’t sound right, it was 1957, not 1857. Reading and adding up 500 integers didn’t take too much time.

It's right. One detail is that they wanted to calculate the index hourly.

I assume the problem was not calculating a weighted average of 500 numbers, but data transmission; in this case, from New York to Boston. It would have had to identify ticker symbols, know which should be included, get both price and shares outstanding, and deal with prices expressed as fractions like 20⅜.

This article says that the Datatron used 1,600 vacuum tubes, and the only storage was a magnetic drum memory for 4,000 ten-digit words, or about 20 kilobytes, or about 0.00002 gigabytes. So it took about 1/30th of a second to access a random location. There were tricks for speeding it up. On the face of it, the actual operation of adding 500 numbers might have taken 500/30 = 16 seconds based on storage access.

Ah, here's an article I missed last time I went hunting. Those are NOT magnetic tape drives, by the way--those are reels of punched paper tape.

New York Times, Feb 28, 1957, p. 35:

ELECTRONIC INDEX TO SERVE MARKET

Standard & Poor's to Offer Figures Hourly Based on 500 Common Stocks

Stock market statistics will catch up with the electronic age on Monday, when Standard & Poor's Corporation will begin to distribute its new hourly index of 500 New York Stock Exchange stocks....

The new indexes, for industrials, rails and utilities, will be available to investors a minute or so after any hour....

This has been made possible by the work of a group of young engineers under Dr. John Hansen of Melpar, Inc. of Boston. a subsidiary of the Westinghouse Air Brake Company. Two years ago they became fascinated with the possibilitics of the Datation electronic computer, made by the Burroughs Corporation....

The impulses that activate a Stock Exchange ticker are fed into this reperforator, which reproduces in the Boston laboratory the kind of perforated tape punched out by the girls in the Exchange ticker room. This is a six-channel tape and the Datatron uses a four-channel tape. So a converter had to be devised.

The tape then goes into the Datatron. This complex instrument has been taught to disregard extraneous information, such as Christmas greetings, welcomes to visiting celebrities and appeals for blood donors. It also cuts out bid-and-asked quotations and "cash" sales, recording only "regular way" deals and, what is more, only regular way deals in the 500 common stocks of the new index.

It remembers the most recent trade in each stock and can even adjust to the fact that the ticker drops digits when it falls too far behind the floor. other words, if the last sale was at 87¼ and the next appears in abbreviated form. as 7¼, the memory bank merely changes the fraction and "remembers" 87%....

Nontrivial. And apparently requiring a fair number of manual operations--ticker tape signal to paper tape to another paper tape to computer.

The article doesn't really explain why the limit was 500, though. Perhaps it had something to do with making result available "a minute or so after any hour." Perhaps there was some key calculation that actually took a minute. On the other hand, it may have been influenced by the fact that S&P had already been calculating a 480-stock average once a week.

[Northern Flicker]

That is only for inclusion. This filter was added after the dot.com bust.

Then how do you justify the fact that the companies on the tail end of the 500 were not replaced very frequently before 2001 ? Surely the difference in capitalization between #500 and #501 is less than the daily volatility.

Index changes were not uncommon before 2001. S&P announced a change on random Wednesdays at market close in those years.

[vv19]

Every 2 weeks, 10s of millions of American buy stocks through their 401K plan. A vast majority of it goes directly into the S&P 500 index. These investors don't care what the market is doing and what the macro news is. That's part of the secret sauce and why it is so hard to beat.

[Northern Flicker]

When the precursor of the S&P500 was expanded from 90 stocks to 500 stocks in 1957, creating the S&P500, smallcaps were much less liquid. They were traded over-the-counter without computerized communication between brokers to facilitate finding a trading partner in a timely manner. The S&P500 was implemented with profitability and liquidity screens so that it was representative of the "investable" market. I'm not sure when sector weight considerations were added, but the intent is for the index to be representative of the whole market.

In some ways, it is an obsolete index methodology. Smallcaps and midcaps are much more liquid so that investability screens are not needed for a large cap index. But it is quite pervasive, and there is so much $ indexed to the S&P 500, that it is here to stay.

[Northern Flicker]

Every 2 weeks, 10s of millions of American buy stocks through their 401K plan. A vast majority of it goes directly into the S&P 500 index. These investors don't care what the market is doing and what the macro news is. That's part of the secret sauce and why it is so hard to beat.

Active investors had the same struggles beating the market long before 401k's were in use. It is not correct that the "vast majority" of 401k assets is in S&P500 funds. In Table 1 of this report we see that, at the end of 2022, about 25% of private defined contribution assets were in individual stocks. Only 50% was in mutual funds of all types, which implies that (much) less than 50% is in S&P500 funds.

It also is interesting that there is significantly more assets in IRAs than in all types of defined contribution plans.

[Wwwdotcom]

Makes me think of this chart. It shows the minimal impact of Intel in 2024.
https://www.voronoiapp.com/markets/The- ... t-Cap-3552

[nisiprius]

Active investors had the same struggles beating the market long before 401k's were in use...

From 1975:

Gifted, determined, ambitious professionals have come into investment management in such large numbers during the past 30 years that it may no longer be feasible for any of them to profit from the errors of all the others sufficiently often and by sufficient magnitude to beat the market averages. Disagreeable data are streaming out of the computers of Becker securities and Merrill Lynch and all the other performance measurement firms. Over and over and over again, these facts and figures inform us that investment managers are failing to perform. Not only are the nation’s leading portfolio managers failing to produce positive absolute rates of return (after all, it's been a long, long bear market) but they are also failing to produce positive relative rates of return. Contrary to their oft articulated goal of outperforming the market averages, investment managers are not beating the market: The market is beating them.

-- The Financial Analyst's Journal, Vol. 31, No. 4, July/August 1975, 19-26

That's the secret sauce: if you just match the market, you beat the pros.

[Northern Flicker]

From a factor perspective, the S&P500 has a mild quality tilt, and tilts away from size. It even may have a slight value tilt.

https://www.portfoliovisualizer.com/fac ... 55WQTsvfcw

[BuglheadLuvsLondon]

50 may not be enough for adequate diversification.

Historically you could get to 80% of the S&P 500 diversification with something between the high 20s to high 30s.

Who wants only 80%? I don't want only 80%.

[Northern Flicker]

50 may not be enough for adequate diversification.

Historically you could get to 80% of the S&P 500 diversification with something between the high 20s to high 30s.

So less than 50.

Here is a study that found that at least 50 stocks generally are needed to diversify away security-specific risk, and 100 may be needed in the worst case.

http://eprints.utas.edu.au/17313/1/2013 ... cation.pdf

With careful sector diversification, I don't doubt that 80% of uncompensated risk can be diversified away with 20-30 stocks. But with ultra-low-cost index funds that are available, there is no need to take any idiosyncratic risk.