Two Centuries of U.S. Stock Market Indexes, 1802-2017

Discuss all general (i.e. non-personal) investing questions and issues, investing news, and theory.
Post Reply
User avatar
SimpleGift
Posts: 3121
Joined: Tue Feb 08, 2011 3:45 pm
Location: Central Oregon

Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by SimpleGift » Thu Apr 12, 2018 9:12 am

A fellow chart enthusiast recently sent me the graphic below, which was just published by Global Financial Data. It shows the historical breakdown of the U.S. stock market into 12 industry sectors over two centuries since 1800 — to which I've added the relevant U.S. stock market indexes at the top, covering each historical sub-period:
  • Image
    NOTE: Includes the 11 Global Industry Classification Standard sectors, plus an added transportation sector.
    Source: Adapted from Global Financial Data
What's striking is how thin and undiversified the indexes are before 1900. Starting in 1802, the Smith & Cole Index had just 6 bank stocks and a railway, adding more railroad stocks for a total of 27 companies by 1870 — all equal weighted and price only. The Cowles Index starts with 12 stocks in 1871 (a few railways, canals and banks) and grows to 350 stocks by 1927. The Cowles Index was value-weighted, included dividends, and was forerunner to the Standard & Poors Index series in 1928.

Bottom Line: It's hard to see how U.S. stock index data before 1900 can have much relevance for modern markets. Thoughts?

PS. If interested in more, this study is comprehensive: Indexes of U.S. Stock Prices from 1802-1987.
Cordially, Todd

gmaynardkrebs
Posts: 889
Joined: Sun Feb 10, 2008 11:48 am

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by gmaynardkrebs » Thu Apr 12, 2018 9:16 am

SimpleGift wrote:
Thu Apr 12, 2018 9:12 am
A fellow chart enthusiast recently sent me the graphic below, which was just published by Global Financial Data. It shows the historical breakdown of the U.S. stock market into 12 industry sectors over two centuries since 1800 — to which I've added the relevant U.S. stock market indexes at the top, covering each historical sub-period:
  • Image
    NOTE: Includes the 11 Global Industry Classification Standard sectors, plus an added transportation sector.
    Source: Adapted from Global Financial Data
What's striking is how thin and undiversified the indexes are before 1900. Starting in 1802, the Smith & Cole Index had just 6 bank stocks and a railway, adding more railroad stocks for a total of 27 companies by 1870 — all equal weighted and price only. The Cowles Index starts with 12 stocks in 1871 (a few railways, canals and banks) and grows to 350 stocks by 1927. The Cowles Index was value-weighted, included dividends, and was forerunner to the Standard & Poors Index series in 1928.

Bottom Line: It's hard to see how U.S. stock index data before 1900 can have much relevance for modern markets. Thoughts?

PS. If interested in more, this study is comprehensive: Indexes of U.S. Stock Prices from 1802-1987.
Markets are markets. Sure, it's relevant.

User avatar
SimpleGift
Posts: 3121
Joined: Tue Feb 08, 2011 3:45 pm
Location: Central Oregon

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by SimpleGift » Thu Apr 12, 2018 10:32 am

gmaynardkrebs wrote:
Thu Apr 12, 2018 9:16 am
Markets are markets. Sure, it's relevant.
A couple of concerns:
  • • As I understand it, there was no such thing as "the U.S. stock market" prior to about 1900. There were just regional stock exchanges in the major cities around the country that traded a handful of regional stocks — which were nearly all local banks and railroad companies early on.

    • The Smith & Cole Index and the Cowles Commission Index were not created contemporaneously with the stock markets that they studied. In other words, they were both created in the 1920s and 1930s by researchers going back through old regional newspapers from the 1800s, and copying out the bid/ask prices of the regional stock listings.

    • Dividend data prior to 1900 was mostly non-existent, so dividends were re-created by various estimation techniques.
So I'm still a bit skeptical of the pre-1900 stock index data, even when it's incorporated into well-respected studies and data sets from researchers like Robert Shiller and Jeremey Siegel. Incorporating data from the 1800s seems more like "financial archaeology" to me — which is quite interesting, but perhaps less relevant to today's modern global markets.
Last edited by SimpleGift on Thu Apr 12, 2018 11:10 am, edited 1 time in total.
Cordially, Todd

david1082b
Posts: 371
Joined: Fri Jun 09, 2017 12:35 pm

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by david1082b » Thu Apr 12, 2018 10:52 am

SimpleGift wrote:
Thu Apr 12, 2018 9:12 am
The Cowles Index was value-weighted, included dividends, and was forerunner to the Standard & Poors Index series in 1928.
It's a pity S&P didn't stick to only publishing an index with dividends included, it could have prevented so many people being misinformed about historical returns. Instead, today we have the endless running battle of people posting price charts of the S&P 500 and Dow to "prove" how bad stocks have returned, sometimes taking inflation away from the price index to boot. The total return index charts are hidden away for some bizarre reason.

User avatar
nisiprius
Advisory Board
Posts: 36646
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by nisiprius » Thu Apr 12, 2018 12:15 pm

The data before 1870 seem to be more or less garbage. The problem is that "the best available data" isn't necessarily good data. Jason Zweig wrote a devastating piece: Does stock market data really go back 200 years? (Jeremy Siegel responded in an article I can no longer access, here). The problem is that all the early data is full of survivorship bias, patchy, and not even close to representing the total market... and dividends were not well recorded and in charts like Siegel's basically replaced by guesstimates of what the average dividend might have been. Even the Cowles data, 1870-1939, have obvious problems which I discussed here.

Before 1920 about 1/6th of all stocks were traded on "the curb exchange," which later became the AMEX, and the curb exchange was so notoriously corrupt that the stock listings literally contained a warning label:
"It should be understood that no such reliability attaches to transactions on the "Curb" as to those on the regularly organized stock exchanges... it is out of the question for anyone to vouch for the absolute trustworthiness of the record of "Curb" transactions, and we give it for what it may be worth."
I agree with your SimpleGift's comments about early data being irrelevant to today even if they were accurate. I think the great works of the nineteenth century, the railroads, and the canals before them, were typically financed with bonds; stocks were a kind of exotic investment, not the mainstream way to raise capital. And US securities had such a bad reputation that Charles Dickens uses them in "A Christmas Carol" to epitomize unreliability:
This was a great relief, because “three days after sight of this First of Exchange pay to Mr. Ebenezer Scrooge or his order,” and so forth, would have become a mere United States’ security if there were no days to count by.
There are many watershed moments in the history of the stock market, and it is hard to believe that the behavior of the stock market was quantitatively the same before and after. People often seem unaware of the original meaning of "bull" and "bear." It didn't mean "optimist" and "pessimist." Before the SEC, stock market manipulation was assumed. It was what the stock market as about. "Bulls" mean people who were trying to make stocks go up, "bears" meant people who were trying to make stocks go down, and the game was all about trying to guess which side had how much in secret reserves from "syndicates" and which was going to be able to bankrupt the other.

The end of fixed commissions was another watershed. I don't have the slip but I remember paying something like $130 to Merrill Lynch to buy $4,000 worth of stock, and that doesn't event include a hidden fee of $0.125 per share for trading "odd lots." The creation of the 401(k) and millions of people automatically investing monthly in stocks was another.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

User avatar
nisiprius
Advisory Board
Posts: 36646
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by nisiprius » Thu Apr 12, 2018 12:24 pm

P.S. I don't think "S&P 90" index is right, by the way. I've been trying to unscramble this story for some time but still don't quite have it. There was a "Standard & Poor's Composite Index" that was calculated daily, and at some point another one with several hundreds that was calculated weekly. I think the Composite Index had 90 stocks when the S&P 500 was introduced in 1957, but I don't think it was constrained to have exactly 90.

You know that most of the short summary histories of "the S&P" must be sloppy and inaccurate, because of course there could not have been any "Standard and Poor's" index of any kind before 1941, because that's when Standard merged with Poor's.

There was more than one "Cowle's Commission Index" and it did not stop in 1927, it covered 1870-1938.

It would be mildly interesting to know the literal details of how the S&P indexes were calculated before the introduction of computers. The Cowles Commission literally used thirty or so desk calculators operated by student volunteers from Colorado College, under supervision. Heaven only knows what kind of cross-checking and "debugging" were involved.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

gmaynardkrebs
Posts: 889
Joined: Sun Feb 10, 2008 11:48 am

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by gmaynardkrebs » Thu Apr 12, 2018 12:30 pm

SimpleGift wrote:
Thu Apr 12, 2018 10:32 am
gmaynardkrebs wrote:
Thu Apr 12, 2018 9:16 am
Markets are markets. Sure, it's relevant.
A couple of concerns:
  • • As I understand it, there was no such thing as "the U.S. stock market" prior to about 1900. There were just regional stock exchanges in the major cities around the country that traded a handful of regional stocks — which were nearly all local banks and railroad companies early on.

    • The Smith & Cole Index and the Cowles Commission Index were not created contemporaneously with the stock markets that they studied. In other words, they were both created in the 1920s and 1930s by researchers going back through old regional newspapers from the 1800s, and copying out the bid/ask prices of the regional stock listings.

    • Dividend data prior to 1900 was mostly non-existent, so dividends were re-created by various estimation techniques.
So I'm still a bit skeptical of the pre-1900 stock index data, even when it's incorporated into well-respected studies and data sets from researchers like Robert Shiller and Jeremey Siegel. Incorporating data from the 1800s seems more like "financial archaeology" to me — which is quite interesting, but perhaps less relevant to today's modern global markets.
Ok, if the old data sets are inaccurate, incomplete, and unreliable, that's one thing. But if the data is accurate, it seems to me that it would still be useful, assuming that what was being traded was the same as what we call stocks today -- fractional ownership shares with a claim to all future cash flows of the corporation.

User avatar
SimpleGift
Posts: 3121
Joined: Tue Feb 08, 2011 3:45 pm
Location: Central Oregon

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by SimpleGift » Thu Apr 12, 2018 12:59 pm

nisiprius wrote:
Thu Apr 12, 2018 12:15 pm
People often seem unaware of the original meaning of "bull" and "bear." It didn't mean "optimist" and "pessimist." Before the SEC, stock market manipulation was assumed. It was what the stock market was about. "Bulls" mean people who were trying to make stocks go up, "bears" meant people who were trying to make stocks go down, and the game was all about trying to guess which side had how much in secret reserves from "syndicates" and which was going to be able to bankrupt the other.
Thank you for chiming in, nisiprius. You're definitely our resident expert on the history of U.S. stock indexes!

To further your point, I don't believe there were any State regulations or statutes about securities fraud until after about 1910. And of course Congress only created the Securities Act in 1933 and the Securities & Exchange Commission in 1934.

gmaynardkrebs wrote:
Thu Apr 12, 2018 12:30 pm
Ok, if the old data sets are inaccurate, incomplete, and unreliable, that's one thing.
From the article by Jason Zweig, linked in the post by nisiprius upthread:
Jason Zweig wrote:To be a good measure of stock returns, an index should be comprehensive (by including many stocks) and representative (by including the stocks commonly held by investors). The Smith and Cole indexes are neither, as the professors signaled in their 1935 book, Fluctuations in American Business. They cherry-picked their indexes by throwing out any stock that didn’t survive for the whole period, whose share prices were too hard to find or whose returns seemed “inflexible,” “erratic,” or “non-typical.”
So I wince when I see researchers drawing any kind of conclusions from the U.S. stock index data in the 1800s.
Cordially, Todd

User avatar
patrick013
Posts: 2365
Joined: Mon Jul 13, 2015 7:49 pm

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by patrick013 » Thu Apr 12, 2018 1:37 pm

Image

My contribution to the museum of stock prices.
Mostly Rails and very volatile looks like to me.

Right-click "view image" to enlarge for a better view.
age in bonds, buy-and-hold, 10 year business cycle

Retireby40
Posts: 10
Joined: Wed Apr 11, 2018 4:14 pm

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by Retireby40 » Thu Apr 12, 2018 1:40 pm

It is very interesting and I applaud the author for creating the data. However, let's be honest about how different the economy was that far back:

No Federal Reserve
No Federal Income Tax
No electric light bulbs
No indoor plumbing
No automobiles

I know some of those items seem like improvements :D It is difficult to draw valuable comparisons when the tax, trade, technology and regulatory environment was so different.

Shamb3
Posts: 25
Joined: Mon Apr 09, 2018 8:58 pm

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by Shamb3 » Thu Apr 12, 2018 2:09 pm

On a Discovery channel show I remember they said Vanderbilt was trying to buy a controlling share of the Erie railroad and they just printed more shares every time, devaluing all already sold shares. They were prosecuted, fled, bought off prosecutes, made a new deal, back-stabbed their partner and Vanderbilt made the railroad re-purchase his shares. Real similar times :beer

jbranx
Posts: 258
Joined: Thu Feb 09, 2017 6:57 pm

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by jbranx » Thu Apr 12, 2018 3:00 pm

nisiprius wrote:
Thu Apr 12, 2018 12:24 pm
P.S. I don't think "S&P 90" index is right, by the way. I've been trying to unscramble this story for some time but still don't quite have it. There was a "Standard & Poor's Composite Index" that was calculated daily, and at some point another one with several hundreds that was calculated weekly. I think the Composite Index had 90 stocks when the S&P 500 was introduced in 1957, but I don't think it was constrained to have exactly 90.

You know that most of the short summary histories of "the S&P" must be sloppy and inaccurate, because of course there could not have been any "Standard and Poor's" index of any kind before 1941, because that's when Standard merged with Poor's.

There was more than one "Cowle's Commission Index" and it did not stop in 1927, it covered 1870-1938.

It would be mildly interesting to know the literal details of how the S&P indexes were calculated before the introduction of computers. The Cowles Commission literally used thirty or so desk calculators operated by student volunteers from Colorado College, under supervision. Heaven only knows what kind of cross-checking and "debugging" were involved.
If I recall correctly, Standard Statistics (no S&P until 1941) did have another list of stocks, about 200, dating back further than the 90 that became the S&P Index. So, we aren't talking hundreds of stocks compiled and computed as an index. We're talking a mere 219 that S&P was tracking in some fashion, none from Amex and Nasdaq was not yet born. It was computed weekly, using a slide rule, and not until the 1960's did daily computerized calculations start by a predecessor to ADP, the official calculator for several decades until S&P developed its own calculations engines. Think Bunker-Ramo was the name of the old calculator. They had one of the first broker terminals with the real-time and closing prices for brokers. Edited: It was actually a firm named "Ultronics" that began real-time calculation of the 500 in 1962. The other index that I said was 200 was actually about 423 I see on a Wikipedia article that I cannot verify. I also recall that S&P made some other changes in 1947 but my memory lapses on what those were. Anyone looking at that list of 90 will find great companies like the Interboro Subway line in NYC! It was only one line I think of the system.

User avatar
SimpleGift
Posts: 3121
Joined: Tue Feb 08, 2011 3:45 pm
Location: Central Oregon

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by SimpleGift » Thu Apr 12, 2018 3:40 pm

nisiprius wrote:
Thu Apr 12, 2018 12:24 pm
P.S. I don't think "S&P 90" index is right, by the way. I've been trying to unscramble this story for some time but still don't quite have it. There was a "Standard & Poor's Composite Index" that was calculated daily, and at some point another one with several hundreds that was calculated weekly. I think the Composite Index had 90 stocks when the S&P 500 was introduced in 1957, but I don't think it was constrained to have exactly 90.
You're mostly correct. In researching this topic, I ran across this 2002 paper and the quoted text below (my bold):

An Analysis of the S&P 500 Index and Cowles’s Extensions: Price Indexes and Stock Returns, 1870–1999
Wilson & Jones wrote:It is important to understand clearly that there were actually two different S&P “Composite” Indexes. First, there is the weekly index (the “all stocks” index) as described above. Second, a daily index of 90 stocks, consisting of 50 industrials, 20 rails, and 20 utilities, was initiated in 1928 (and carried back weekly to December 1925). Going forward, this index of 90 stocks continued until March 1957, when the daily index was shifted to the 500 stocks.

In March 1957, S&P abandoned its historical weekly series with the statement, “To avoid confusion, Standard & Poor’s has standardized on its former daily price index . . . for the back record.” This caution by S&P was repeated in the biennial editions of their historical data (the “Blue Books”) for many years, but is not mentioned in recent editions. Simultaneously, S&P ceased estimating the S&P 90 Index. The decision to abandon the historical record of the broad-based weekly index in favor of the S&P 90, which ceased, has been the cause of much confusion.
According to this paper, the Ibbotson historical data series going back to 1926 is based on the S&P 90 daily data (starting in December 1925), rather than the more broadly-defined S&P weekly data (with 200-400 stocks). Interesting to know!
Cordially, Todd

jbranx
Posts: 258
Joined: Thu Feb 09, 2017 6:57 pm

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by jbranx » Thu Apr 12, 2018 3:49 pm

SimpleGift wrote:
Thu Apr 12, 2018 3:40 pm
nisiprius wrote:
Thu Apr 12, 2018 12:24 pm
P.S. I don't think "S&P 90" index is right, by the way. I've been trying to unscramble this story for some time but still don't quite have it. There was a "Standard & Poor's Composite Index" that was calculated daily, and at some point another one with several hundreds that was calculated weekly. I think the Composite Index had 90 stocks when the S&P 500 was introduced in 1957, but I don't think it was constrained to have exactly 90.
You're mostly correct. In researching this topic, I ran across this 2002 paper and the quoted text below (my bold):

An Analysis of the S&P 500 Index and Cowles’s Extensions: Price Indexes and Stock Returns, 1870–1999
Wilson & Jones wrote:It is important to understand clearly that there were actually two different S&P “Composite” Indexes. First, there is the weekly index (the “all stocks” index) as described above. Second, a daily index of 90 stocks, consisting of 50 industrials, 20 rails, and 20 utilities, was initiated in 1928 (and carried back weekly to December 1925). Going forward, this index of 90 stocks continued until March 1957, when the daily index was shifted to the 500 stocks.

In March 1957, S&P abandoned its historical weekly series with the statement, “To avoid confusion, Standard & Poor’s has standardized on its former daily price index . . . for the back record.” This caution by S&P was repeated in the biennial editions of their historical data (the “Blue Books”) for many years, but is not mentioned in recent editions. Simultaneously, S&P ceased estimating the S&P 90 Index. The decision to abandon the historical record of the broad-based weekly index in favor of the S&P 90, which ceased, has been the cause of much confusion.
According to this paper, the Ibbotson historical data series going back to 1926 is based on the S&P 90 daily data (starting in December 1925), rather than the more broadly-defined S&P weekly data (with 200-400 stocks). Interesting to know!
Here's some more info on the S&P index history from Reuters. I have somewhere a paper authored by David Blitzer of the index committee that gives more historical detail if I can find it. Note that Reuters is quoting a paper that was produced by S&P I believe:


Timeline: Key dates and milestones in the S&P 500's history
Caroline Valetkevitch
6 MIN READ

NEW YORK (Reuters) - Standard & Poor’s, initially known as the Standard Statistics Company, created its first stock market index in 1923. It consisted of the stocks of 233 companies and was computed weekly.

Three years later, it developed a 90-stock composite price index computed daily. That was expanded over the years.

On March 4, 1957, the Standard & Poor's 500 .INX .SPX was introduced.

The S&P 500 index has became synonymous with the term “U.S. stock market.” It is one of the leading benchmarks for the market, even though others, including the Russell and Wilshire indexes, are broader measures of the market. Still, investors use the S&P 500 as the main index to measure their portfolios’ performance against, with about $5.58 trillion benchmarked to the S&P 500.

The S&P’s 500 companies represent the U.S. market more broadly than the Dow Jones industrial average, which includes the stocks of only 30 companies.

American Telephone and Telegraph was the heaviest-weighted stock in the index in 1957. The company, now known as AT&T (T.N), is the 11th-largest company in the S&P 500 index.

Today the S&P 500 index has a total market cap of $14.6 trillion. Sixty-nine of the 500 original companies remain in the S&P 500 today.

Below are some key dates and milestones in the history of the S&P 500.

1923: Standard Statistics Company, as S&P was formerly known, develops its first stock market index consisting of the stocks of 233 U.S. companies, computed weekly.

1926: Standard Statistics creates a 90-stock composite price index computed daily.

March 4, 1957: The Standard & Poor’s 500 index is introduced, tracking the performance of the stocks of 500 leading U.S. companies. With a total market capitalization of $172 billion, the S&P 500 followed the performance of 425 industrial, 15 rail and 60 utility stocks.

1958: S&P 500 ends the year up 38.06 percent, its best year in terms of percentage gain.

June 4, 1968: S&P 500 closes above 100 for the first time.

August 31, 1976: Vanguard introduces the first retail index mutual fund, the Vanguard First Index Investment Trust, which tracks the S&P 500, allowing individual investors for the first time to buy into the broad market with a single purchase. The fund, now known as the Vanguard 500 Index Fund (VFINX.O), has $125 billion in assets.

April 21, 1982: The Chicago Mercantile Exchange begins trading futures based on the S&P 500.

July 1, 1983: Options contracts based on the S&P 500 index begin trading on the Chicago Board Options Exchange.

Oct 19, 1987: S&P 500 registers its worst daily percentage loss, falling 20.47 percent. The one-day crash, known as “Black Monday,” was blamed on program trading and those using a hedging strategy known as portfolio insurance. Despite the losses, the S&P 500 still ended up that year.

January 22, 1993: State Street’s Standard & Poor’s Depositary Receipts, or the SPDR S&P 500 (SPY.P), an exchange-traded fund that tracks the S&P 500’s performance, begins trading on the American Stock Exchange. It was the first ETF to trade in the United States. The first SPDR and the many variations that followed are commonly referred to as the “spiders.” The fund currently has about $133.8 billion in assets, making it the largest exchange-traded fund in terms of assets.

September 9, 1997: CME introduces the S&P E-mini futures, which is valued at $50 multiplied by the price of the S&P 500, or one-fifth of the size of the “big” S&P futures contract. It has since become the most heavily traded futures contract on the CME.

February 2, 1998: S&P 500 closes above 1,000 for the first time.

March 24, 2000: The S&P 500 index reaches an all-time intraday high of 1,552.87 during the dot-com bubble.

March 24, 2004: Trading begins in futures on the VIX .VIX, the CBOE Volatility Index measuring implied volatility of S&P 500 index options. The VIX is known as the market’s “fear gauge.” It tends to rise when stocks fall. It recently fell to levels not seen since April 2007.

Standard & Poor's Corp
2663.99
.SPXCHICAGO BOARD OPTIONS EXCHANGE
+21.80(+0.83%)
.SPX
.SPXT.NVFINX.OSPY.PAAPL.O
March-September 2005: The index is transitioned from simply market-value weighted to float adjusted, where the market capitalization is calculated using only the number of shares available for public trading.

October 9, 2007: Index closes at a record high of 1,565.15.

October 11, 2007: S&P 500 hits intraday record high of 1,576.09.

Oct 13, 2008: S&P 500 marks its best daily percentage gain, rising 11.58 percent. It also registers its largest single-day point increase of 104.13 points.

2008: For the year, S&P 500 falls 38.49 percent - its worst yearly percentage loss. In September 2008, Lehman Brothers collapsed as the financial crisis spread.

March 9, 2009: S&P 500 closes at 676.53, its closing low after the onset of the 2008 financial crisis and the Lehman Brothers’ bankruptcy.

August 20, 2012: Apple (AAPL.O) becomes the biggest U.S. company and takes over as market capitalization leader in the S&P 500, pushing Exxon Mobil (XOM.N) into the No. 2 spot. Since then, Exxon and Apple have gone back and forth between the two spots, but Apple is currently No. 1 with a market cap of about $416 billion. Exxon has a market cap of $404 billion.

March 28, 2013: S&P 500 ends above 1,569 - setting a record closing high and surpassing the previous milestone set in 2007.

Sources: S&P Dow Jones Indices Senior Index Analyst Howard Silverblatt, the Standard & Poor’s book, “Innovation & Evolution, The S&P 500,” CME, CBOE, Vanguard Group Inc, State Street Global Advisors, Thomson Reuters.

Reporting by Caroline Valetkevitch; Editing by Jan Paschal

Our Standards:The Thomson Reuters Trust Principles.
AppsNewslettersReuters PlusAdvertising GuidelinesCookiesTerms of UsePrivacy

jalbert
Posts: 3683
Joined: Fri Apr 10, 2015 12:29 am

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by jalbert » Thu Apr 12, 2018 4:02 pm

So I'm still a bit skeptical of the pre-1900 stock index data, even when it's incorporated into well-respected studies and data sets from researchers like Robert Shiller and Jeremey Siegel.
Visually, the 2017 chart endpoint also does not seem to match the current or recent past makeup of the S&P500, wherein Tech is the largest sector.
Risk is not a guarantor of return.

User avatar
JoMoney
Posts: 5868
Joined: Tue Jul 23, 2013 5:31 am

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by JoMoney » Thu Apr 12, 2018 4:13 pm

Some might find this S&P history 'timeline' interesting
https://web.archive.org/web/20061114131 ... _2006.html
"To achieve satisfactory investment results is easier than most people realize; to achieve superior results is harder than it looks." - Benjamin Graham

User avatar
triceratop
Moderator
Posts: 5654
Joined: Tue Aug 04, 2015 8:20 pm
Location: la la land

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by triceratop » Thu Apr 12, 2018 6:10 pm

david1082b wrote:
Thu Apr 12, 2018 10:52 am
SimpleGift wrote:
Thu Apr 12, 2018 9:12 am
The Cowles Index was value-weighted, included dividends, and was forerunner to the Standard & Poors Index series in 1928.
It's a pity S&P didn't stick to only publishing an index with dividends included, it could have prevented so many people being misinformed about historical returns. Instead, today we have the endless running battle of people posting price charts of the S&P 500 and Dow to "prove" how bad stocks have returned, sometimes taking inflation away from the price index to boot. The total return index charts are hidden away for some bizarre reason.
You might be interested in this paper: Reconsidering Returns, which deals with precisely this problem.
"To play the stock market is to play musical chairs under the chord progression of a bid-ask spread."

golfCaddy
Posts: 704
Joined: Wed Jan 10, 2018 10:02 pm

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by golfCaddy » Thu Apr 12, 2018 6:14 pm

nisiprius wrote:
Thu Apr 12, 2018 12:15 pm
The data before 1870 seem to be more or less garbage. The problem is that "the best available data" isn't necessarily good data. Jason Zweig wrote a devastating piece: Does stock market data really go back 200 years? (Jeremy Siegel responded in an article I can no longer access, here).
That Zweig piece is devastating,
Thus the indexes relied on by Prof. Siegel exclude 97% of all the stocks that existed in the earliest years of the U.S. market,
Any index that excludes 97% of stocks at the time might charitably be described as historical fiction.

User avatar
siamond
Posts: 4186
Joined: Mon May 28, 2012 5:50 am

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by siamond » Thu Apr 12, 2018 6:45 pm

SimpleGift wrote:
Thu Apr 12, 2018 9:12 am
Bottom Line: It's hard to see how U.S. stock index data before 1900 can have much relevance for modern markets. Thoughts?
For me, the primary relevance is that human nature doesn't really change. As such, speculative crises like the Tulip Mania and others, are very meaningful, even if they occurred hundreds of years ago. The simple fact that they kept re-appearing for nearly 500 years is enlightening. I don't know the future, but I can predict that speculative bubbles will happen again!

Besides that, I agree, the data series pre-1900 do not seem terribly reliable, nor relevant. For all the reasons that you, nisiprius and others articulated.

PS. this was a cool graph in the OP, the sector breakdown.

User avatar
SimpleGift
Posts: 3121
Joined: Tue Feb 08, 2011 3:45 pm
Location: Central Oregon

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by SimpleGift » Thu Apr 12, 2018 7:07 pm

triceratop wrote:
Thu Apr 12, 2018 6:10 pm
david1082b wrote:
Thu Apr 12, 2018 10:52 am
It's a pity S&P didn't stick to only publishing an index with dividends included, it could have prevented so many people being misinformed about historical returns. Instead, today we have the endless running battle of people posting price charts of the S&P 500 and Dow to "prove" how bad stocks have returned, sometimes taking inflation away from the price index to boot. The total return index charts are hidden away for some bizarre reason.
You might be interested in this paper: Reconsidering Returns, which deals with precisely this problem.
Thanks for the link, triceratop. This paper explains a lot about why, in the early 1900s, the emphasis when first constructing stock indexes was primarily on price changes and not total returns (by adding dividends, whether reinvested or not):
Hartzmark & Solomon wrote:This brief history of investment thought helps to explain the odd way that many stock indices are constructed. Early academic papers which constructed such indices, and companies which did the same, did not think they were constructing indices of returns. They thought they were constructing indices of prices.

Adjustments that seem obvious to a modern finance academic, such as reinvesting dividends, were not nearly so obvious when the subject specifically was prices. This tendency was almost surely exacerbated by the fact that when computations were being done by hand, there was considerable extra work involved to calculate periodic reinvestment values compared with just taking the ratio of starting and ending prices, and dividend information was often difficult to obtain.
Unfortunately, this historical legacy still lives on today, in the form of the widely published, price-only S&P 500 Index.
Cordially, Todd

protagonist
Posts: 5388
Joined: Sun Dec 26, 2010 12:47 pm

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by protagonist » Thu Apr 12, 2018 7:48 pm

SimpleGift wrote:
Thu Apr 12, 2018 9:12 am

Bottom Line: It's hard to see how U.S. stock index data before 1900 can have much relevance for modern markets. Thoughts?
"modern markets", like "modern art", won't be modern for long.

AlohaJoe
Posts: 3754
Joined: Mon Nov 26, 2007 2:00 pm
Location: Saigon, Vietnam

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by AlohaJoe » Thu Apr 12, 2018 8:01 pm

nisiprius wrote:
Thu Apr 12, 2018 12:24 pm
P.S. I don't think "S&P 90" index is right, by the way. I've been trying to unscramble this story for some time but still don't quite have it.
A while ago I made an attempt at unscrambling it some

About Cowles:
https://medium.com/@justusjp/making-of- ... d083a9b994

About Standard and Poors:
https://medium.com/@justusjp/making-of- ... 433561dd1b

AlohaJoe
Posts: 3754
Joined: Mon Nov 26, 2007 2:00 pm
Location: Saigon, Vietnam

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by AlohaJoe » Thu Apr 12, 2018 8:11 pm

golfCaddy wrote:
Thu Apr 12, 2018 6:14 pm
That Zweig piece is devastating,
Thus the indexes relied on by Prof. Siegel exclude 97% of all the stocks that existed in the earliest years of the U.S. market,
Any index that excludes 97% of stocks at the time might charitably be described as historical fiction.
The Dow Jones Industrial Average excludes far more than 97% of stocks and yet the historical record shows that it is still a surprisingly good indicator of the overall market.

I think it is right to take historical numbers with a grain of salt but we also have evidence that even small numbers of stocks can tell us quite a lot about the overall market, so I don't think throwing the baby out with the bathwater is necessarily the right response either.

learning_head
Posts: 836
Joined: Sat Apr 10, 2010 6:02 pm

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by learning_head » Thu Apr 12, 2018 8:23 pm

AlohaJoe wrote:
Thu Apr 12, 2018 8:11 pm
golfCaddy wrote:
Thu Apr 12, 2018 6:14 pm
That Zweig piece is devastating,
Thus the indexes relied on by Prof. Siegel exclude 97% of all the stocks that existed in the earliest years of the U.S. market,
Any index that excludes 97% of stocks at the time might charitably be described as historical fiction.
The Dow Jones Industrial Average excludes far more than 97% of stocks and yet the historical record shows that it is still a surprisingly good indicator of the overall market.

I think it is right to take historical numbers with a grain of salt but we also have evidence that even small numbers of stocks can tell us quite a lot about the overall market, so I don't think throwing the baby out with the bathwater is necessarily the right response either.
Dow Jones attempts to include "representative" stocks, seemingly quite successfully. So the question is whether 3% that were included were representative ones, or say just the ones that happened to survive, making it a whole different story... ?

I want to thank SimpleGift, Nisiprius and all other contributors for a great and very interesting discussion!

User avatar
nisiprius
Advisory Board
Posts: 36646
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by nisiprius » Thu Apr 12, 2018 8:33 pm

Two notes. First, yes, railroads were important, extremely important. (They might have been the dubious "United States' securities" Dickens was referring to, at some point there was a near-mania in England for US railroad stocks). Originally, or very near the beginning, there were two important Dow Jones averages (not "indices!"): the Dow Jones Railroad Average and the Dow Jones Industrial Average. The DJIA was not intended to represent "the market," it was intended to be used, together with and in relationship to the Dow Jones Railroad Average as part of a technical analysis system in which the relative movements of the two indices were indicators of something.

Second, with regard to human nature and "the market is the market," it depends on what exact uses you want to make of the data. It may tell you something about dynamics or fractal behavior or market psychology. I think it's ludicrous to treat it as quantitative, the mean is thus-and-such, the standard deviation is thus-and-such. Siegel makes much of the "extraordinary stability" of US stock market returns over two centuries, the almost straight line on the 200-year chart. In fact, until fairly recently people started to call it "Siegel's Constant," at one time said to be 7%, more recently 6.6%. The last edition of his book was in 2014, I need to inspect it and see if he has joined the consensus that it is going to be lower going forward.

But the point is that it is not legitimate to use pre-1870 data to prove the straightness of the line going back to 1802 if the data from 1802-1870 is price only and is based on one, single, overall estimate for dividends over the entire period. The line segment from 1802 to 1870 is going to bend down if you make a lower estimate, bend up if you make a higher estimate, and line up with the rest of the chart if you make one in between.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

User avatar
patrick013
Posts: 2365
Joined: Mon Jul 13, 2015 7:49 pm

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by patrick013 » Thu Apr 12, 2018 8:42 pm

nisiprius wrote:
Thu Apr 12, 2018 8:33 pm
But the point is that it is not legitimate to use pre-1870 data to prove the straightness of the line going back to 1802 if the data from 1802-1870 is price only and is based on one, single, overall estimate for dividends over the entire period. The line segment from 1802 to 1870 is going to bend down if you make a lower estimate, bend up if you make a higher estimate, and line up with the rest of the chart if you make one in between.
Well I think we can go post 1960 for good modern data. Wondering why
my stat program likes to start at 1960.
age in bonds, buy-and-hold, 10 year business cycle

golfCaddy
Posts: 704
Joined: Wed Jan 10, 2018 10:02 pm

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by golfCaddy » Thu Apr 12, 2018 9:17 pm

AlohaJoe wrote:
Thu Apr 12, 2018 8:11 pm
The Dow Jones Industrial Average excludes far more than 97% of stocks and yet the historical record shows that it is still a surprisingly good indicator of the overall market.

I think it is right to take historical numbers with a grain of salt but we also have evidence that even small numbers of stocks can tell us quite a lot about the overall market, so I don't think throwing the baby out with the bathwater is necessarily the right response either.
According to one online calculator, about half the long run return from the DJIA comes from price appreciation and half from dividends. So without dividend information, the DJIA does not provide a reliable measure of long term return. Then, the DJIA isn't a random selection of stocks. It represents the largest public companies the US. If Siegel had dividend data(instead of making it up) and if Cowles was selecting the largest stocks by market cap at the time(instead of survivorship bias and arbitrary judgments about erratic pricing), 30 stocks might not be terrible. Even then, my estimate is the DJIA total return beat VTSMX, by 1%/year, since the inception in 1992 of VTSMX.

AlohaJoe
Posts: 3754
Joined: Mon Nov 26, 2007 2:00 pm
Location: Saigon, Vietnam

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by AlohaJoe » Thu Apr 12, 2018 9:44 pm

golfCaddy wrote:
Thu Apr 12, 2018 9:17 pm

...and if Cowles was selecting the largest stocks by market cap at the time(instead of survivorship bias and arbitrary judgments about erratic pricing), 30 stocks might not be terrible.
I think you're mixing up Cowles and Smith&Cole (the latter is who Zweig is talking about).

G. William Schwert's paper "Indexes of Stock Market 1802-1987" also has a discussion about the Smith & Cole series, including comparing it to the Macaulay series for the years when there is overlap.

User avatar
siamond
Posts: 4186
Joined: Mon May 28, 2012 5:50 am

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by siamond » Thu Apr 12, 2018 10:00 pm

nisiprius wrote:
Thu Apr 12, 2018 8:33 pm
Second, with regard to human nature and "the market is the market," it depends on what exact uses you want to make of the data. It may tell you something about dynamics or fractal behavior or market psychology. I think it's ludicrous to treat it as quantitative, the mean is thus-and-such, the standard deviation is thus-and-such. Siegel makes much of the "extraordinary stability" of US stock market returns over two centuries, the almost straight line on the 200-year chart. In fact, until fairly recently people started to call it "Siegel's Constant," at one time said to be 7%, more recently 6.6%.
I fully agree with you. And there is an ever simpler proof of the fact that this 'constant' is a mix of fiction and coincidence. It's called a good old 'out of sample test'. Now that historical research on other countries made solid progress, as far as I know, other countries (or geographic theater) did NOT display such a straight line. If only because most of them got devastated by WW-II. Yes, ok, the US is a rather exceptional country, but still... case closed. Same goes for the most vaulted '4% rule', by the way.

User avatar
siamond
Posts: 4186
Joined: Mon May 28, 2012 5:50 am

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by siamond » Thu Apr 12, 2018 10:58 pm

May I suggest something... There are a lot of fascinating historical nuggets which have been posted in this thread.

It would be great if the most interesting facts could be captured in a historical section of the S&P 500 index wiki page. Just saying.

User avatar
SimpleGift
Posts: 3121
Joined: Tue Feb 08, 2011 3:45 pm
Location: Central Oregon

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by SimpleGift » Thu Apr 12, 2018 11:33 pm

Reading this thread, plus several of the linked papers, my initial skepticism about the accuracy of the early U.S. stock index data has only grown. Which made me wonder: What early index data do Dimson, Marsh and Staunton (arguably the best global data series today, unfortunately commerical) use for their United States stock returns series starting in 1900?

From their 2002 book, Triumph of the Optimists, a summary their U.S. data sources:
Dimson, Marsh & Staunton" wrote:Our series commences with the Wilson-Jones index data over 1900-1925. For 1926-61, we use the University of Chicago's Center for Research in Security Prices (CRSP) capitalization-weighted index of all New York Stock Exchange stocks. For 1962-70, we use the CRSP cap-weighted index of NYSE and Amex stocks. From 1971 onward we employ the Wilshire 5000 Index, which includes stocks listed on the NYSE, Amex, Nasdaq, and other exchanges. All indexes include reinvested dividends.
This paper has the full Wilson-Jones Index data (1870-1999) — which updated and corrected the Cowles monthly data from 1871. An interesting tidbit from this 2002 Wilson-Jones paper (my bold):
Wilson & Jones wrote:Cowles made an important contribution to the study of returns over long periods of time by extending the original S&P data backward in time to the beginning of 1871. In subsequent reports Cowles corrected errors in his data, which have not been picked up by most sources reporting this historical series. For example, although Standard & Poor’s reports their historical series based on Cowles’s data, S&P has consistently failed to use the subsequently corrected Cowles data. The uncorrected S&P series, in turn, has been used by Shiller and by Siegel, as well as others, in studying stock returns over long periods.
It'd be interesting to compare the Shiller annual data vs. the Wilson-Jones data to see how large these differences are.
Cordially, Todd

User avatar
nisiprius
Advisory Board
Posts: 36646
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by nisiprius » Fri Apr 13, 2018 12:39 pm

No, I am not an expert on the history of financial data, and there is a lot of interesting stuff in this thread I hadn't known.

With regard to the CRSP data, there is a story behind it. I presented some of it in a thread, Origins of the CRSP; what about that 1926 starting point?. Here are the key points.
  • Because of the influence of the CRSP's data set, an awful lot of studies start in 1926. The choice of 1926 appears to have been what might be called a "tasteful decision" by the founders of the CRSP. It is not an obviously biased decision, it is in fact in between a point (1929) that might have been chosen by someone trying to make stocks look bad, and one (1921) that might have been chosen by someone trying to make stocks look good. But it's still an arbitrary choice, and nowhere is the specific rationale for the choice it explained by the people who chose it.
  • The CRSP was founded for a specific purpose: to provide Merrill Lynch with data so that it could run big full-page infomercial-like ads saying that stocks, held for the long-term, were a prudent investment for individual retailer buyers. Now, that might be true--just as the claims presented in TV ads for expensive new prescription drugs may be true--but the CRSP was not founded as a purely academic endeavor. It was sponsored for commercial purposes. The first paper published by the center--"Rates of Return on Investments in Common Stocks," L. Fisher and J. H. Lorie, The Journal of Business, Vol. 37, No. 1 (Jan., 1964), pp. 1-21--notes that "This work is the first to emerge from the Center for Research in Security Prices (sponsored by Merrill Lynch, Pierce, Fenner & Smith Inc.)"
  • The paper says simply that "Monthly closing prices from the New York Stock Exchange from January, 1926 through December, 1960 have been placed on tape." I've read, but lost the reference, to a story about a "box of tapes" found somewhere in Merrill Lynch that they were going to throw out, until someone checked to see if CRSP wanted them. That may be apocryphal. Lorie describes the data acquisition process in some detail here. But it is clear that the data were originally recorded as news items, not with the intention of being a research-quality record--and that in particular, dividend data took a lot of work and cross-collating.
    Annual dividend guides that list publicly held companies in alphabetical order and that describe each dividend paid during the year are available for the period beginning in 1937. For earlier periods, quarterly guides are available. To collect the data, clerks were given cards with a coding form printed on them, a list of names and code numbers of listed companies, and a dividend guide. They filled out as many cards as there were cash dividends for listed companies. This information was then punched into the cards and the data were transcribed onto magnetic tape. For the last years of the study, the annual guides note the exchanges on which a stock was listed. For the earlier years they do not. Because it was so easy to make clerical errors, our method of collection could not be expected to produce a very complete list of dividends...
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

User avatar
siamond
Posts: 4186
Joined: Mon May 28, 2012 5:50 am

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by siamond » Fri Apr 13, 2018 12:42 pm

SimpleGift wrote:
Thu Apr 12, 2018 11:33 pm
It'd be interesting to compare the Shiller annual data vs. the Wilson-Jones data to see how large these differences are.
Check this thread... :wink:

User avatar
SimpleGift
Posts: 3121
Joined: Tue Feb 08, 2011 3:45 pm
Location: Central Oregon

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by SimpleGift » Fri Apr 13, 2018 1:26 pm

siamond wrote:
Fri Apr 13, 2018 12:42 pm
SimpleGift wrote:
Thu Apr 12, 2018 11:33 pm
It'd be interesting to compare the Shiller annual data vs. the Wilson-Jones data to see how large these differences are.
Check this thread...
Thank you, siamond. You just saved me from going down the rabbit hole of comparing the Shiller data set with the Wilson-Jones series. It appears that you and AlohaJoe have already explored that hole very nicely!

Seems that Dimson, Marsh and Staunton are now the only ones currently using the Wilson-Jones data in their U.S. return series (for the 1900-1925 period) — so they are likely still the gold standard for accuracy of long-term returns.
Cordially, Todd

User avatar
nisiprius
Advisory Board
Posts: 36646
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by nisiprius » Fri Apr 13, 2018 1:42 pm

Darn. Excellent thread and I posted in it and I'd completely forgotten all about it. Glad to be reminded of it.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

jalbert
Posts: 3683
Joined: Fri Apr 10, 2015 12:29 am

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by jalbert » Fri Apr 13, 2018 1:59 pm

Including the chart unquoted for larger size:

Image

And here is S&P500 sector data at end of Feb 2018:

Basic Materials 3.25%
Consumer Cyclical 12.05%
Financial Services 17.44%
Real Estate 2.01%
Communication Services 3.19%
Energy 5.70%
Industrials 11.49%
Technology 20.17%
Consumer Defensive 8.36%
Healthcare 14.02%
Utilities 2.80%

After comparing that to the end of the chart, is there a convincing reason to consider any of the chart data as being visually accurate?
Risk is not a guarantor of return.

User avatar
SimpleGift
Posts: 3121
Joined: Tue Feb 08, 2011 3:45 pm
Location: Central Oregon

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by SimpleGift » Fri Apr 13, 2018 2:13 pm

jalbert wrote:
Fri Apr 13, 2018 1:59 pm
After comparing that to the end of the chart, is there a convincing reason to consider any of the chart data as being visually accurate?
Jalbert, you may want to take this up with the chart creators, Global Financial Data, who discuss the historical background of this chart and their industry sector classifications in this blog post.

This is the only data source I've seen with sector breakdowns back to 1800 — so there's not many other references I'm aware of where one could compare the accuracy of their industry classifications.
Cordially, Todd

Nate79
Posts: 3486
Joined: Thu Aug 11, 2016 6:24 pm
Location: Portland, OR

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by Nate79 » Fri Apr 13, 2018 2:25 pm

jalbert wrote:
Fri Apr 13, 2018 1:59 pm
Including the chart unquoted for larger size:

Image

And here is S&P500 sector data at end of Feb 2018:

Basic Materials 3.25%
Consumer Cyclical 12.05%
Financial Services 17.44%
Real Estate 2.01%
Communication Services 3.19%
Energy 5.70%
Industrials 11.49%
Technology 20.17%
Consumer Defensive 8.36%
Healthcare 14.02%
Utilities 2.80%

After comparing that to the end of the chart, is there a convincing reason to consider any of the chart data as being visually accurate?
Looking thru the numbers I do not see the big difference from the chart. Perhaps you could put your guesstimate next to each to point out where you think the chart differs from the sector data that you list?

jalbert
Posts: 3683
Joined: Fri Apr 10, 2015 12:29 am

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by jalbert » Fri Apr 13, 2018 2:57 pm

Compare the right edge of the chart for finance and technology, which is clearly incorrect as Technology is a little larger but is much smaller in the diagram. Then industrials in the diagram is about the same size as technology despite the sector being about half the size.

(I originally had stronger objections due to confusion from real estate and consumer staples using virtually the same color).

The chart does show the migration of GDP from home production and small businesses to large corporations over the history of the country, and the rise and fall of the railroad industry for example.

But if you look at the finance sector in the early 1800’s the visual might lead you to think it completely dominated the economy. But it is less than 12 companies in the graph and GDP was mostly being generated outside of corporate structure at the time. Despite the very large visual for finance in 1800, the finance sector is certainly a larger percentage of GDP today than in 1800. The graph just reflects a percentage of stock capitalization, not total magnitude.

In the early 1800’s you could not diversify unsystematic risk. An investor would thus expect to be compensated for taking that risk, which would likely increase expected return and increase variance and risk.
Risk is not a guarantor of return.

Nate79
Posts: 3486
Joined: Thu Aug 11, 2016 6:24 pm
Location: Portland, OR

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by Nate79 » Fri Apr 13, 2018 3:13 pm

jalbert wrote:
Fri Apr 13, 2018 2:57 pm
Compare the right edge of the chart for finance and technology, which is clearly incorrect as Technology is a little larger but is much smaller in the diagram. Then industrials in the diagram is about the same size as technology despite the sector being about half the size.

(I originally had stronger objections due to confusion from real estate and consumer staples using virtually the same color).

The chart does show the migration of GDP from home production and small businesses to large corporations over the history of the country, and the rise and fall of the railroad industry for example.

But if you look at the finance sector in the early 1800’s the visual might lead you to think it completely dominated the economy. But it is less than 12 companies in the graph and GDP was mostly being generated outside of corporate structure at the time. Despite the very large visual for finance in 1800, the finance sector is certainly a larger percentage of GDP today than in 1800. The graph just reflects a percentage of stock capitalization, not total magnitude.

In the early 1800’s you could not diversify unsystematic risk. An investor would thus expect to be compensated for taking that risk, which would likely increase expected return and increase variance and risk.
Here is some more sector weight information for the S&P500. What's interesting is that even this data doesn't match what you list and say what Morningstar gives.

https://www.bespokepremium.com/think-bi ... eightings/

Here is another:
https://us.spindices.com/indices/equity/sp-500

Either way technology is way under represented in the posted chart.

BigMoneyNoWhammies
Posts: 125
Joined: Tue Jul 11, 2017 11:58 am

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by BigMoneyNoWhammies » Fri Apr 13, 2018 3:58 pm

gmaynardkrebs wrote:
Thu Apr 12, 2018 12:30 pm
SimpleGift wrote:
Thu Apr 12, 2018 10:32 am
gmaynardkrebs wrote:
Thu Apr 12, 2018 9:16 am
Markets are markets. Sure, it's relevant.
A couple of concerns:
  • • As I understand it, there was no such thing as "the U.S. stock market" prior to about 1900. There were just regional stock exchanges in the major cities around the country that traded a handful of regional stocks — which were nearly all local banks and railroad companies early on.

    • The Smith & Cole Index and the Cowles Commission Index were not created contemporaneously with the stock markets that they studied. In other words, they were both created in the 1920s and 1930s by researchers going back through old regional newspapers from the 1800s, and copying out the bid/ask prices of the regional stock listings.

    • Dividend data prior to 1900 was mostly non-existent, so dividends were re-created by various estimation techniques.
So I'm still a bit skeptical of the pre-1900 stock index data, even when it's incorporated into well-respected studies and data sets from researchers like Robert Shiller and Jeremey Siegel. Incorporating data from the 1800s seems more like "financial archaeology" to me — which is quite interesting, but perhaps less relevant to today's modern global markets.
Ok, if the old data sets are inaccurate, incomplete, and unreliable, that's one thing. But if the data is accurate, it seems to me that it would still be useful, assuming that what was being traded was the same as what we call stocks today -- fractional ownership shares with a claim to all future cash flows of the corporation.
Assuming it is accurate, which based on those here with far greater knowledge on the history than me seem to indicate is not the case, I don't think 2 century old stock information is all that informative for current and future markets for the simple reason that the conditions are entirely different: who invests, how many people are invested in the market, how much they invest, why they invest, which vehicles (401k, ira, etc) they use to do so, the level of automation of both markets and market actors, the drastic increase in regulation of markets, the sort of business the companies on the market are engaged in - all of these factors are wildly different now than the historical eras in question, and for me make utilizing century old or more market data basically useless in informing future indicators/market changes/investment decisions

jalbert
Posts: 3683
Joined: Fri Apr 10, 2015 12:29 am

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by jalbert » Fri Apr 13, 2018 4:00 pm

Here is some more sector weight information for the S&P500. What's interesting is that even this data doesn't match what you list and say what Morningstar gives.

https://www.bespokepremium.com/think-bi ... eightings/

Here is another:
https://us.spindices.com/indices/equity/sp-500

Either way technology is way under represented in the posted chart.
My posted data was from Morningstar.
Risk is not a guarantor of return.

snarlyjack
Posts: 780
Joined: Fri Aug 28, 2015 12:44 pm
Location: Montana

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by snarlyjack » Sat Apr 14, 2018 1:25 pm

Triceratop,

I read your study "Reconsidering Returns".

Thank You for posting the study.

I totality agree with the study & I think it might be
part of our whole problem. The price index's are not
created correctly.

For example (my account) I received approx. 270 additional
shares last year that do not show up in the total returns of the fund,
(only the price line shows up). If you take 270 x $33. = $8,910. that
is not showing up. Not to mention additional dividends on these 270
shares or any compounding of these shares.

No wonder people are confused & numerious arguments we have had.
Were finally on the correct path to seeing the true light.
This study is "Pure Gold"... Every One should read it...

https://papers.ssrn.com/sol3/papers.cfm ... id=3039507

Thank You!

User avatar
nisiprius
Advisory Board
Posts: 36646
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by nisiprius » Sat Apr 14, 2018 1:39 pm

I never even thought about the question of sector classification. There's a huge problem there, given that the GICS classification was not even created in 1999. So, this is "fantasy football," in that someone has, somehow, tried to classify companies in 1850 according to categories that were not codified until 1999.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

User avatar
SimpleGift
Posts: 3121
Joined: Tue Feb 08, 2011 3:45 pm
Location: Central Oregon

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by SimpleGift » Sat Apr 14, 2018 2:17 pm

nisiprius wrote:
Sat Apr 14, 2018 1:39 pm
So, this is "fantasy football," in that someone has, somehow, tried to classify companies in 1850 according to categories that were not codified until 1999.
Could be an issue post-1900 — but probably not much of one before 1900, when the researchers were dealing at most with several dozen bank, railroad and canal stocks. These fit pretty easily into the Finance and Transportation categories, which were over 80% of market cap throughout the 19th century.
Cordially, Todd

User avatar
triceratop
Moderator
Posts: 5654
Joined: Tue Aug 04, 2015 8:20 pm
Location: la la land

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by triceratop » Mon Apr 16, 2018 3:01 pm

snarlyjack wrote:
Sat Apr 14, 2018 1:25 pm
Triceratop,

I read your study "Reconsidering Returns".

Thank You for posting the study.

I totality agree with the study & I think it might be
part of our whole problem. The price index's are not
created correctly.

For example (my account) I received approx. 270 additional
shares last year that do not show up in the total returns of the fund,
(only the price line shows up). If you take 270 x $33. = $8,910. that
is not showing up. Not to mention additional dividends on these 270
shares or any compounding of these shares.

No wonder people are confused & numerious arguments we have had.
Were finally on the correct path to seeing the true light.
This study is "Pure Gold"... Every One should read it...

https://papers.ssrn.com/sol3/papers.cfm ... id=3039507

Thank You!
Nope, that is not what the article is about.

"For example (my account) I received approx. 270 additional
shares last year that do not show up in the total returns of the fund,
(only the price line shows up). "

No. If the price line shows up then you are not being showed the total returns of the fund, but instead the price return.

Our oft-repeated disagreements here are not about what this article discusses.
"To play the stock market is to play musical chairs under the chord progression of a bid-ask spread."

snarlyjack
Posts: 780
Joined: Fri Aug 28, 2015 12:44 pm
Location: Montana

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by snarlyjack » Mon Apr 16, 2018 3:34 pm

Triceratop,

Once again thank you for posting the study.

I understand exactly what you are saying & I
appreciate it. We have no problems...

I would like to post a article for your consideration.
Jack Bogle was mentioned in #4.

https://www.dividend-growth-stocks.com/ ... atter.html

CurlyDave
Posts: 699
Joined: Thu Jul 28, 2016 11:37 am

Re: Two Centuries of U.S. Stock Market Indexes, 1802-2017

Post by CurlyDave » Mon Apr 16, 2018 9:50 pm

nisiprius wrote:
Thu Apr 12, 2018 12:15 pm
... People often seem unaware of the original meaning of "bull" and "bear." It didn't mean "optimist" and "pessimist." Before the SEC, stock market manipulation was assumed. It was what the stock market as about. "Bulls" mean people who were trying to make stocks go up, "bears" meant people who were trying to make stocks go down, and the game was all about trying to guess which side had how much in secret reserves from "syndicates" and which was going to be able to bankrupt the other.

...The creation of the 401(k) and millions of people automatically investing monthly in stocks was another.
Looks to me like the creation of the 401(k) was a great victory for the bulls, pushing the demand curve into previously unimagined territory.

Post Reply