Alternative to Shiller's S&P 500 data series

Discuss all general (i.e. non-personal) investing questions and issues, investing news, and theory.
Post Reply
AlohaJoe
Posts: 2416
Joined: Mon Nov 26, 2007 2:00 pm
Location: Saigon, Vietnam

Alternative to Shiller's S&P 500 data series

Post by AlohaJoe » Sat Dec 10, 2016 9:55 am

At the request of siamond, I'm creating a separate thread about this.

Recently MachineGhost (who joined in 2009 but appears to have come out of hibernation!) pointed out a 2002 article by Wilson & Jones that was published in the Journal of Business that claims there are several flaws in the commonly used "S&P 500" data (as used by Ibbotson, Shiller, Siegel, Damodaran, and others) and provides a fix.

The full paper is here: http://www.jstor.org/stable/10.1086/339903

siamond is inquiring whether it is permissible to include their data in the Bogleheads/simba backtesting spreadsheet. The question remains, does their methodology make sense?

Prior work on bogleheads.

Long time readers may remember that in 2014, nisiprius started a thread where he raised a number of questions about the validity of the returns data before 1939: viewtopic.php?t=144112

More recent threads have also wondered about the validity of data before 1957: viewtopic.php?t=199104

This paper addresses some (but not all) of the concerns raised in those posts.

The differences in practice.

I'll start by illustrating that we're not exactly talking about trivial, small differences.

This shows the year by year difference between the new data (which I'll call W&J after the authors, Wilson & Jones) and Shiller/simba's data. You can see that it can differ by up to 10% in a single year. For instance, in 1928, Shiller says the "S&P 500" gained 46.41% but W&J say it only gained 39%. In 1934, Shiller says it lost -8.03% but W&J say it gained 3.02%.

Image

A tell tale chart, which shows the cumulative returns of the two, expressed as a ratio, makes visible how this difference would show up to an investor starting in 1871.

Image

The differences show up in other measures, such as P/E. The historical mean P/E of S&P 500 is 15.63. But W&J say it should be 14.28. W&J provides a variety of metrics showing how their index differs from others in Table 4. Overall they find their index has lower returns and higher volatility:

Total Percentage Change 1871-1999
S&P Historical: 32,077.47
W&J: 28,809.16

Standard Deviation
S&P Historical: 18.5176
W&J: 19.2442

Of particular note, they found that the Great Depression was worse than previously reported (with a P/E of 136, due to earnings being destroyed), while the late 1960s/early 1970s was better. (In fact, their data results in a slight increase to the "Safe Withdrawal Rate" of about .1%

What did they do?

They identify a number of claimed flaws in the original work.

Cowles originally his first edition in 1938. Errors were discovered and revised editions were published in 1939 and 1940. W&J claim that other indices only use the uncorrected first edition. They use the last version Cowles published.

Up until 1957, the S&P published a daily index based on the prices of around 90 stocks. This "S&P 90" was expanded in 1957 and became the S&P 500. Thus, from 1919 to 1956 the index only included 90 stocks. However, S&P also published a weekly index of around 400 stocks. (Calculating a daily index of 400 stocks was apparently too challenging before calculators & computers.) This "S&P 400" was discontinued in 1957 and eventually lost to the annals of time. W&J base their index on this weekly data of a broader set of companies, arguing that this is a clearly better data set.

Cowles' data is "averaged". He took the average of prices across a month instead of reporting the price at the start or end of the month. This reduces volatility.

Cowles used monthly arithmetic means of the S&P weekly indexes from 1918 through 1940 and the midpoint of the monthly high and low prices of stocks in constructing monthly indexes back to January 1871.


The authors provide a way to "de-average" Cowles' data by looking at DJIA data to calculate an adjustment. They use Cowles' calculation on the DJIA historical data to see the difference between the average and un-averaged prices. They then assume that is a good enough adjustment to unaverage the Cowles data. This produces a data series with (approximately) the same average but higher volatility.

They estimate monthly dividends (not just annual dividends), which allows dividends to be reinvested throughout the year and not just at year's end.

Why not just use this?

Well, their arguments sound reasonable enough. But of course they would, otherwise they wouldn't have published their paper :happy The bigger question is: why does Googling find virtually no references or follow up to this paper? Why do so many smart, respected authors no follow this paper's suggestions?

Does the paper make sense or it is all a bunch of mumbo jumbo, best ignored?

User avatar
nisiprius
Advisory Board
Posts: 33922
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Alternative to Shiller's S&P 500 data series

Post by nisiprius » Sat Dec 10, 2016 10:22 am

I can't speak to the specifics but I have long had serious doubts about the quality of the financial data that people use to make calculations to three and four decimal places of precision.

Even on something as basic as "the total return of the S&P 500 for 1985"--the most commonly quoted of all financial series, for the simplest possible time-span--a calendar year, from the days of computers and computer networks--I discovered a meaningful difference in the number printed in a trusted reference book, the Ibbotson SBBI Classic Yearbook, for 2005 and 2009: 0.3216 (i.e. 32.16%) versus 0.3173 (i.e. 31.73%). Only the first digit of the two numbers matches!

And I've ranted about the fact that virtually every source simply talks about "the S&P 500" even when the data series goes back to 1870, ignoring details like the S&P 500's not being created until 1957 and Standard & Poor's itself not being formed until 1941.

A much more fundamental problem is obvious when you read the introduction to the Cowles Commission's books, and has nothing to do with computations. I posted a detailed piece about this, Reliability of stock data prior to 1939? Executive summary: at the time of the Cowles study, they think they got about 90% of the NYSE. But the NYSE only accounted for about 67% of the stock capitalization of the United States, the two dozen or so regional and city exchanges were important, and I don't think Cowles even tried to include them.

A much more serious omission is the omission of the Curb Exchange, which accounted for about 1/6th of the U.S. stock capitalization. Cowles omitted it because the Curb (which later became the AMEX) was so rife with fraud that the newspaper that listed its stocks literally included a warning, "It should be understood that no such reliability attaches to transactions on the "Curb" as to those on the regularly organized stock exchanges... it is out of the question for anyone to vouch for the absolute trustworthiness of the record of "Curb" transactions, and we give it for what it may be worth." From the point of view of the Cowles Commission, it is obviously a situation of they would have been damned if they did, but since these stocks may have been the predecessors of some of the microcap, small-cap value, and tech startup categories of today, it seems to me they are also damned since they didn't.

That is to say, the Cowles data is pretty obviously not even qualitatively comparable to modern data, and certainly not quantitatively.

That's not in any way to demean the greatness of the achievement, nor the hard work of the thirty or so students at Colorado College who did the actual work using desk calculators.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

AlohaJoe
Posts: 2416
Joined: Mon Nov 26, 2007 2:00 pm
Location: Saigon, Vietnam

Re: Alternative to Shiller's S&P 500 data series

Post by AlohaJoe » Sat Dec 10, 2016 10:42 am

I already linked to your excellent detailed piece in the OP :D

User avatar
nisiprius
Advisory Board
Posts: 33922
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Alternative to Shiller's S&P 500 data series

Post by nisiprius » Sat Dec 10, 2016 10:51 am

I've just glanced at the paper. My initial reaction is that it is indeed important and seemingly overlooked. I am very inclined to trust their criticisms of the discordant and inconsistent data series that are customarily lumped together and incorrectly called "the S&P 500 back to 1870." Reading over their discussion it looks like a much more complete and detailed account of all the things that had already started to bother me, just from reading bits of the Cowles' report.

Whether it follows that their own "reconstructions" are correct and should simply be accepted by everyone as superseding the older datasets is another question. I certainly don't know, and "corrected" and "reconstructed" data has its own set of problems.

As to "why does Googling find virtually no references or follow up to this paper?" I'd shrug and say, probably because it's hugely inconvenient and people don't want to deal with it. I have my own pet set of "important things that nobody ever talks about," I think I'll add this one to my mental list.

Benoit Mandelbrot and Richard L. Hudson, The Misbehavior of Markets: A Fractal View of Financial Turbulence.

Pastor and Stambaugh, 2011, "Are Stocks Really Less Volatile in the Long Run?"
Last edited by nisiprius on Sat Dec 10, 2016 3:42 pm, edited 1 time in total.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

User avatar
nisiprius
Advisory Board
Posts: 33922
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Alternative to Shiller's S&P 500 data series

Post by nisiprius » Sat Dec 10, 2016 10:52 am

AlohaJoe wrote:I already linked to your excellent detailed piece in the OP :D
Indeed you did, thanks!
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

User avatar
siamond
Posts: 3323
Joined: Mon May 28, 2012 5:50 am

Re: Alternative to Shiller's S&P 500 data series

Post by siamond » Sat Dec 10, 2016 11:59 am

Terrific analysis, AlohaJoe, thank you so much for that.

One point to clarify: the data sources that were used by AlohaJoe for the 'regular' S&P 500 actually came from two sources:
- Prof. Shiller data series from 1871 to 1970
- S&P 500 itself from 1971 to 2015

To be more precise, the latter was extracted from Morningstar S&P 500 TR USD, and uses dividends reinvested monthly. And I was pleased to see that those numbers almost exactly matched the work from Wilson and Jones for this time period.

Also remember that the Shiller numbers have a bit of a quirk. If I am not mistaken, the price series is averaged for the whole month of January, instead of taking the value on January 1st, and THIS REMAINS TRUE FOR RECENT NUMBERS IN THE SERIES. This makes it consistent with the oldest numbers from Cowles, but creates a weird time lag, and artificially attenuates standard deviation math.

User avatar
siamond
Posts: 3323
Joined: Mon May 28, 2012 5:50 am

Re: Alternative to Shiller's S&P 500 data series

Post by siamond » Sat Dec 10, 2016 12:31 pm

nisiprius wrote:Whether it follows that their own "reconstructions" are correct and should simply be accepted by everyone as superseding the older datasets is another question. I certainly don't know, and "corrected" and "reconstructed" data has its own set of problems.

Yes, I agree. If we were to include this data series in the Simba spreadsheet, I would put it side to side with the series from Prof. Shiller, so that people can form their own opinions. We would have to make a choice for the pre-1971 years of the VFINX series though. I also just figured out that there is this S&P 500 TR USD(1936) data series, where dividends were computed as reinvested annually. Too much of a good thing... :wink:

But first things first, I sent an e-mail yesterday to Prof. Jones (Prof. Wilson passed away a few years ago, Prof Jones still teaches at the University of North Carolina) asking for authorization to reproduce, and didn't get an answer yet.

stlutz
Posts: 3968
Joined: Fri Jan 02, 2009 1:08 am

Re: Alternative to Shiller's S&P 500 data series

Post by stlutz » Sat Dec 10, 2016 12:41 pm

At least for the data from the 20s forward, it's important to note that the price portion of the alternate data series is not reconstructed but is calculated from actual index prices reported S&P at the time. Now the dividend portion is reconstructed, so I think the new series would provide a better estimate of volatility but the total return math is probably still a little fuzzy--not in the sense of being way off but in terms of 8% possibly have really been 8.2% or something like that.

FactualFran
Posts: 428
Joined: Sat Feb 21, 2015 2:29 pm

Re: Alternative to Shiller's S&P 500 data series

Post by FactualFran » Sat Dec 10, 2016 12:53 pm

siamond wrote:One point to clarify: the data sources that were used by AlohaJoe for the 'regular' S&P 500 actually came from two sources:
- Prof. Shiller data series from 1871 to 1970
- S&P 500 itself from 1971 to 2015

To be more precise, the latter was extracted from Morningstar S&P 500 TR USD, and uses dividends reinvested monthly. And I was pleased to see that those numbers almost exactly matched the work from Wilson and Jones for this time period.

A few years ago on the web there was a publicly accessible file with the month end S&P 500 Total Return index values. The file included "Source: Reprinted with permission from Standard & Poors" and "Monthly Dividend Reinvestment from Jan. 1970 to 1988, Swithced [sic] to Daily Reinvestment 1/4/88".

heyyou
Posts: 2716
Joined: Tue Feb 20, 2007 4:58 pm

Re: Alternative to Shiller's S&P 500 data series

Post by heyyou » Sat Dec 10, 2016 2:37 pm

We get future returns from markets that are hardly influenced by distant history. As Rod C has noted in his signature, tenths of a percentage are not predictive. Most assuredly, we learn from history, but not at that level of precision, especially in economic history.

When it was calculated that the light from stars behind the edge of the moon was bending, making them visible, did that affect anything about the future of your life?

Rodc
Posts: 13601
Joined: Tue Jun 26, 2007 9:46 am

Re: Alternative to Shiller's S&P 500 data series

Post by Rodc » Sat Dec 10, 2016 3:23 pm

Total Percentage Change 1871-1999
S&P Historical: 32,077.47
W&J: 28,809.16


Over 128 years they both returned 8.4% to the nearest tenth of a percent.

This is the problem with compounding for so long - trivial differences look large. Given the challenges of putting together estimates of returns that far back if anything it is surprising they are so close. And the reality is since these are just reconstructions, reality might be off by more than this trivial difference so there is no knowing which is most accurate or even if either is super accurate.

The differences in volatility are not very large either.

The year to year differences are perhaps more problematic.
We live a world with knowledge of the future markets has less than one significant figure. And people will still and always demand answers to three significant digits.

pkcrafter
Posts: 11979
Joined: Sun Mar 04, 2007 12:19 pm
Location: CA
Contact:

Re: Alternative to Shiller's S&P 500 data series

Post by pkcrafter » Sat Dec 10, 2016 3:33 pm

heyyou wrote:We get future returns from markets that are hardly influenced by distant history. As Rod C has noted in his signature, tenths of a percentage are not predictive. Most assuredly, we learn from history, but not at that level of precision, especially in economic history.

When it was calculated that the light from stars behind the edge of the moon was bending, making them visible, did that affect anything about the future of your life?


Thank you for putting this in perspective.

Any response from Shiller?


Paul
When times are good, investors tend to forget about risk and focus on opportunity. When times are bad, investors tend to forget about opportunity and focus on risk.

qwertyjazz
Posts: 1026
Joined: Tue Feb 23, 2016 4:24 am

Re: Alternative to Shiller's S&P 500 data series

Post by qwertyjazz » Sat Dec 10, 2016 5:53 pm

IRL if I were presented with this kind of question I would ask you to think this through your goals in the project to begin to understand what is 'better'
What concept are you trying to understand with the S&P 500?
What sort of errors (not in the data per se because there always are systematic ones) would lead to problems in the analyses using the data in making your conclusions?
What are the tradeoffs in the inherent biases of both methods of data collection?
Then I would try to look at some sensitivity analyses in using each data set.
I think your goals relate not to absolute gains but rather relative ones between types of investments (but you have to decide what is the point of the data). I think you are trying to capture what decisions in different areas could have been by some definition of risk optimal in a way.
Therefore the absolute number may not be what you are picking data on. But you have to decide your goals yourselves. The averaging techniques - stock choices in different eras by different standards etc etc mean like most things whatever you choose will be right and wrong. You may want to understand it in the context of why you are collecting the data though.
G.E. Box "All models are wrong, but some are useful."

User avatar
siamond
Posts: 3323
Joined: Mon May 28, 2012 5:50 am

Re: Alternative to Shiller's S&P 500 data series

Post by siamond » Sat Dec 10, 2016 10:18 pm

FactualFran wrote:
siamond wrote:One point to clarify: the data sources that were used by AlohaJoe for the 'regular' S&P 500 actually came from two sources:
- Prof. Shiller data series from 1871 to 1970
- S&P 500 itself from 1971 to 2015

To be more precise, the latter was extracted from Morningstar S&P 500 TR USD, and uses dividends reinvested monthly. And I was pleased to see that those numbers almost exactly matched the work from Wilson and Jones for this time period.

A few years ago on the web there was a publicly accessible file with the month end S&P 500 Total Return index values. The file included "Source: Reprinted with permission from Standard & Poors" and "Monthly Dividend Reinvestment from Jan. 1970 to 1988, Swithced [sic] to Daily Reinvestment 1/4/88".

Yes, you're right, I figured out where to get the S&P 500 TR (1989) 'daily dividends' data series, and am getting exactly the same numbers down to the third decimal than the 'monthly dividends' series, so S&P probably spliced them together. And those 1989+ numbers are very exactly the same as the 'large stocks' reported by SBBI/Ibbotson. Before that, small differences start to appear.

carolinaman
Posts: 2588
Joined: Wed Dec 28, 2011 9:56 am
Location: North Carolina

Re: Alternative to Shiller's S&P 500 data series

Post by carolinaman » Sun Dec 11, 2016 8:32 am

I am no expert in this data analysis but I often wonder about the validity of comparing data from 80 to 100 years ago. How comparable is this data to the current market? There have been so many changes over the years, i.e. accounting rules for one. It would seem that all of the changes over the years may be more significant than the variations discussed on this post. Perhaps someone can enlighten me.

User avatar
nisiprius
Advisory Board
Posts: 33922
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Alternative to Shiller's S&P 500 data series

Post by nisiprius » Sun Dec 11, 2016 9:04 am

carolinaman wrote:I am no expert in this data analysis but I often wonder about the validity of comparing data from 80 to 100 years ago. How comparable is this data to the current market? There have been so many changes over the years, i.e. accounting rules for one. It would seem that all of the changes over the years may be more significant than the variations discussed on this post. Perhaps someone can enlighten me.
I agree with you completely and more and more I am coming to think that the search for certainty through the examination of long periods of history is a fool's errand. To me, the proof of this is the Dimson, Marsh & al. statistics for the long-term historical average real return of (a) U.S. stocks and (b) the world ex-US, i.e. "international" stocks, from 1900 through 2015, 116 years of data.

Experts nowadays more or less universally recommend that U.S. investors hold an important portion of their stock allocation in international stocks, and the recommendations nowadays range from 20% to 50%. However, according to Dimson & al, over 116 years historically,

--U.S. stocks have returned 6.4% real
--ex-US have returned only 4.3% real.

That's a huge difference. But we would laugh at anybody who seriously gave it as a reason to expect international stocks to return 2% less going forward, because when we look at country-by-country data, nobody believes that we are looking at 116 years of "the same thing." We can clearly see how the unfolding of world history includes identifiable events that are so huge that they have a huge effect on even a 116-year average.

As I read up on financial history, most recently Charles R. Geisst's "Wall Street: A History," and learn things about U.S. history they weren't too clear on in school, I just get more and more impressed by how obviously there are numerous point events--policy decisions, enactment of laws--that obviously changed the qualitative nature of the U.S. stock market. Just to name one, interest on U.S. Treasuries was exempt from Federal income tax until 1941. Didn't that amount to a fundamental change in the nature of Treasury investments?
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

User avatar
siamond
Posts: 3323
Joined: Mon May 28, 2012 5:50 am

Re: Alternative to Shiller's S&P 500 data series

Post by siamond » Sun Dec 11, 2016 2:46 pm

nisiprius wrote:Experts nowadays more or less universally recommend that U.S. investors hold an important portion of their stock allocation in international stocks, and the recommendations nowadays range from 20% to 50%. However, according to Dimson & al, over 116 years historically,

--U.S. stocks have returned 6.4% real
--ex-US have returned only 4.3% real.

That's a huge difference. But we would laugh at anybody who seriously gave it as a reason to expect international stocks to return 2% less going forward, because when we look at country-by-country data, nobody believes that we are looking at 116 years of "the same thing." We can clearly see how the unfolding of world history includes identifiable events that are so huge that they have a huge effect on even a 116-year average.

Well, only a cursory look at the Dimson & al literature (incl. latest updates from Credit Suisse) shows that the US advantage was all concentrated in the first half of the 20th century - while Europe was ravaged by war. Since WW-II ended, in a time of relative peace, the performance of the US has been on par with the rest of the world, and also on par with an 'old star fading star' like the UK. In the mean time, bonds went all over the place (in real terms, which is what really matters) all over the world.

There IS value in looking at stock market history, if only to not be surprised by future events, and react appropriately (e.g. do nothing!). But yes, it does need to be taken with a big grain of salt. And point events do have a huge effect, that is for sure. Unavoidable conclusion: diversify stocks as much as one can. Less obvious conclusion: bonds are NOT as reliable and predictable as perceived by many.

Back to the specific topic of this thread, yes, sure, this isn't terribly actionable. Personally, I am curious about it, but this is much more intellectual curiosity than anything else. What puzzles me to no end is why S&P decided to use this S&P 90 series as a proxy for S&P 500 in the old days instead of the (apparently obvious) choice documented by Wilson and Jones?

MachineGhost
Posts: 107
Joined: Sat Feb 14, 2009 2:46 am

Re: Alternative to Shiller's S&P 500 data series

Post by MachineGhost » Sun Dec 11, 2016 3:40 pm

carolinaman wrote:I am no expert in this data analysis but I often wonder about the validity of comparing data from 80 to 100 years ago. How comparable is this data to the current market? There have been so many changes over the years, i.e. accounting rules for one. It would seem that all of the changes over the years may be more significant than the variations discussed on this post. Perhaps someone can enlighten me.


The bottom line is 400 stocks are more statistically significant and represent a broader swath of the economy akin to the S&P 500 than only 90 stocks.
Last edited by MachineGhost on Sun Dec 11, 2016 3:57 pm, edited 1 time in total.

MachineGhost
Posts: 107
Joined: Sat Feb 14, 2009 2:46 am

Re: Alternative to Shiller's S&P 500 data series

Post by MachineGhost » Sun Dec 11, 2016 3:48 pm

nisiprius wrote:Experts nowadays more or less universally recommend that U.S. investors hold an important portion of their stock allocation in international stocks, and the recommendations nowadays range from 20% to 50%. However, according to Dimson & al, over 116 years historically,


That's just due to sloppy recognition that valuation is not to be had in the U.S. in the aggregate. They also tend to ignore the currency effect which drives those foreign returns. They're basically advocating a USD hedge without realizing it. But a direct hedge for that purpose would be superior than relying on business cycle vagaries (which may all be synchronized now due to globalist offshoring). I think the currency hedged equity ETF's are interesting to consider though, but the hedging comes at a cost.
Last edited by MachineGhost on Sun Dec 11, 2016 3:52 pm, edited 1 time in total.

MachineGhost
Posts: 107
Joined: Sat Feb 14, 2009 2:46 am

Re: Alternative to Shiller's S&P 500 data series

Post by MachineGhost » Sun Dec 11, 2016 3:51 pm

siamond wrote:Back to the specific topic of this thread, yes, sure, this isn't terribly actionable. Personally, I am curious about it, but this is much more intellectual curiosity than anything else. What puzzles me to no end is why S&P decided to use this S&P 90 series as a proxy for S&P 500 in the old days instead of the (apparently obvious) choice documented by Wilson and Jones?


Maybe Wilson and Jones didn't give them perrmission? Or more likely its just big business politics, etc.. It's not as if the founder of S&P is still managing the company so his self-interest in being a top quality company is long gone. You guys expect too much from employee-based capitalism. :wink:

User avatar
siamond
Posts: 3323
Joined: Mon May 28, 2012 5:50 am

Re: Alternative to Shiller's S&P 500 data series

Post by siamond » Sun Dec 11, 2016 4:57 pm

MachineGhost wrote:
siamond wrote:What puzzles me to no end is why S&P decided to use this S&P 90 series as a proxy for S&P 500 in the old days instead of the (apparently obvious) choice documented by Wilson and Jones?

Maybe Wilson and Jones didn't give them permission? Or more likely its just big business politics, etc.. It's not as if the founder of S&P is still managing the company so his self-interest in being a top quality company is long gone. You guys expect too much from employee-based capitalism. :wink:

I didn't mean for S&P to reuse the Wilson & Jones work per se. I meant, why didn't S&P leverage its own "S&P 400" weekly data series, and went to S&P 90 numbers instead as a proxy for the S&P 500 history? There has to be a reason, and maybe it is rational, who knows... :wink:

MachineGhost
Posts: 107
Joined: Sat Feb 14, 2009 2:46 am

Re: Alternative to Shiller's S&P 500 data series

Post by MachineGhost » Sun Dec 11, 2016 6:12 pm

siamond wrote:I didn't mean for S&P to reuse the Wilson & Jones work per se. I meant, why didn't S&P leverage its own "S&P 400" weekly data series, and went to S&P 90 numbers instead as a proxy for the S&P 500 history? There has to be a reason, and maybe it is rational, who knows... :wink:


It seems to me what probably occured here was maintaining the "Composite Index" integrity which first started with 90 companies in 1926. Maybe at that time it wasn't expected that the S&P 90 would eventually become a dominant index by 1957. And maybe backfilling the weekly S&P 400 indexes into the "Composite Index" was technically unfeasible or not kosher to do in 1957. I think it sounds really bad to say: "Oh, we're going to expand our S&P 90 to 410 more stocks but while we're at it we're gonna invalidate all of its back history to 1923."

A bigger question may be who really started the 90, was it Poor or was it Standard? Perhaps the 90 was from Poor and the 400 from Standard. Oil and water?

User avatar
siamond
Posts: 3323
Joined: Mon May 28, 2012 5:50 am

Re: Alternative to Shiller's S&P 500 data series

Post by siamond » Sun Dec 11, 2016 10:53 pm

I assembled my own telltale about this. I elected to use SBBI 'Large Stocks' (i.e. S&P 500) as a reference, as it perfectly aligns with the SPWC series from Wilson and Jones post-1957.

You can see that Prof. Shiller's S&P 500 goes through some gyrations compared to SBBI (side-effect of the January averaging thing, I suspect) and also trends lower (probably a side-effect of the way I computed its annual total returns, only modeling annual dividends reinvestment). I also played with Tyler's Large-Caps series, just for fun.

As to SPWC, yes, it did plunge even deeper than the S&P 90 during the great depression. And then again post WW-II. Not a fun time.

Image

MachineGhost
Posts: 107
Joined: Sat Feb 14, 2009 2:46 am

Re: Alternative to Shiller's S&P 500 data series

Post by MachineGhost » Mon Dec 12, 2016 4:28 pm

Anyone know what Siegel is using before 1871 that he has the gall to charge $100 for (given that we know he uses that turdish Cowles data)?

http://www.jeremysiegel.com/index.cfm/f ... D/7743.cfm

EDIT: This guy is a real greed monger. He charges $30 alone just to give you an updated chart of the one below: http://www.jeremysiegel.com/index.cfm/f ... D/7689.cfm

Image

User avatar
siamond
Posts: 3323
Joined: Mon May 28, 2012 5:50 am

Re: Alternative to Shiller's S&P 500 data series

Post by siamond » Mon Dec 12, 2016 5:08 pm

MachineGhost wrote:Anyone know what Siegel is using before 1871 that he has the gall to charge $100 for (given that we know he uses that turdish Cowles data)?

http://www.jeremysiegel.com/index.cfm/f ... D/7743.cfm

You can find the answer for free (!) in this article from the same author. I am all for learning from backtesting, but data from the 19th century is a little too much for me...

pkcrafter
Posts: 11979
Joined: Sun Mar 04, 2007 12:19 pm
Location: CA
Contact:

Re: Alternative to Shiller's S&P 500 data series

Post by pkcrafter » Tue Dec 13, 2016 2:52 pm

Any evidence that Shiller ever commented on Wilson/Jones paper? Apparently he never modified his own data. Either he was not aware of their paper (hard to believe) or he does not think it's important.

Paul
When times are good, investors tend to forget about risk and focus on opportunity. When times are bad, investors tend to forget about opportunity and focus on risk.

MachineGhost
Posts: 107
Joined: Sat Feb 14, 2009 2:46 am

Re: Alternative to Shiller's S&P 500 data series

Post by MachineGhost » Tue Dec 13, 2016 6:40 pm

Shiller also needs to be asked if the "colleagues" he worked with includes Siegel and if he's providing Siegel's municipal/Treasury yield data pre-1920. I do know they are friends but nothing explicit has been made about this.

1871-1900 is "data from the 19th century". The problem with the data before 1871 appears to be a lack of diversity, so its pointless.

User avatar
siamond
Posts: 3323
Joined: Mon May 28, 2012 5:50 am

Re: Alternative to Shiller's S&P 500 data series

Post by siamond » Fri Dec 16, 2016 11:25 am

I ran a quick test about the 1/PE10 expected returns model, comparing the predictive power of the regular Shiller series with the SPWC data series from Wilson & Jones. I was vaguely hoping that the trajectory of expected returns vs actuals would be improved in the 1st half of the 20th century, compared to the Shiller model. Well, this wasn't convincing.

The CAGR of the expected returns series vs the SPWC actuals is actually a better match for long time periods, this is probably partly because of the monthly dividends reinvestment computation (while my math with Shiller uses annual reinvestment). But the correlation is definitely poorer, and the RMSE is slightly worse. Oh well, I tried...

User avatar
siamond
Posts: 3323
Joined: Mon May 28, 2012 5:50 am

Re: Alternative to Shiller's S&P 500 data series

Post by siamond » Wed Dec 21, 2016 6:17 pm

siamond wrote:But first things first, I sent an e-mail yesterday to Prof. Jones (Prof. Wilson passed away a few years ago, Prof Jones still teaches at the University of North Carolina) asking for authorization to reproduce, and didn't get an answer yet.

Well, I can't elicit an answer from the good professor. I made one last-ditch attempt via Wade Pfau, as he referenced the Wilson & Jones work in one of his articles a few years ago, alluding to a past e-mail exchange with Prof. Jones. Wade was kind enough to issue his own request (repeating my request!) to Prof. Jones, maybe academic to academic will work better, but I am not holding my breath.

It is too bad that this very interesting research didn't seem to generate much of a follow-up form other academics.

AlohaJoe
Posts: 2416
Joined: Mon Nov 26, 2007 2:00 pm
Location: Saigon, Vietnam

Re: Alternative to Shiller's S&P 500 data series

Post by AlohaJoe » Wed Dec 21, 2016 10:10 pm

siamond wrote:I am not holding my breath.

It is too bad that this very interesting research didn't seem to generate much of a follow-up form other academics.


Maybe we could do something like what is done for the Fama-French data. Include a note & a link with instructions on how to fill it into the spreadsheet for those who are interested in it for private use.

User avatar
siamond
Posts: 3323
Joined: Mon May 28, 2012 5:50 am

Re: Alternative to Shiller's S&P 500 data series

Post by siamond » Thu Dec 22, 2016 2:42 pm

AlohaJoe wrote:Maybe we could do something like what is done for the Fama-French data. Include a note & a link with instructions on how to fill it into the spreadsheet for those who are interested in it for private use.

Great idea. Done. In the Data_Misc tab.

User avatar
siamond
Posts: 3323
Joined: Mon May 28, 2012 5:50 am

Re: Alternative to Shiller's S&P 500 data series

Post by siamond » Mon Jan 23, 2017 2:39 pm

From the Simba thread, we have an interesting alternate avenue to explore...

AlohaJoe wrote:I discovered that Professor Schwert makes available both:

- Monthly stock returns 1802-1925: http://schwert.ssb.rochester.edu/mstock.htm
- Daily stock returns 1885-1962: http://schwert.ssb.rochester.edu/dstock.htm

Schwert's paper is older (1990) and takes a different approach than Wilson & Jones but he also tried to "unaverage" the data from Cowles.

Please note that downloading Prof. Schwert's data implies to agree with a very clear condition: "I agree not to reproduce and distribute the stock index data from Schwert's Journal of Business paper."

For personal use (and maybe a private customization of the Simba spreadsheet), this does seem quite interesting. Thanks for the pointer! Oh, and here is a link to Schwert's paper.

AlohaJoe
Posts: 2416
Joined: Mon Nov 26, 2007 2:00 pm
Location: Saigon, Vietnam

Re: Alternative to Shiller's S&P 500 data series

Post by AlohaJoe » Mon Jan 23, 2017 9:38 pm

siamond wrote:Please note that downloading Prof. Schwert's data implies to agree with a very clear condition: "I agree not to reproduce and distribute the stock index data from Schwert's Journal of Business paper."

For personal use (and maybe a private customization of the Simba spreadsheet), this does seem quite interesting. Thanks for the pointer! Oh, and here is a link to Schwert's paper.


He does have a note saying you can use it for academic purposes for $15 and to contact him to discuss commercial usage. I'm not sure which one the spreadsheet counts as, or whether it is worth bothering getting permission to include. The time period covered it not exactly one most people care about, given the lack of other assets at the time.

User avatar
siamond
Posts: 3323
Joined: Mon May 28, 2012 5:50 am

Re: Alternative to Shiller's S&P 500 data series

Post by siamond » Tue Jan 24, 2017 9:34 am

I think this is a great reference to cite (done in my working copy of the next Simba update!), but I would not go any further. The warning is pretty clear (and we're not academics nor commercial), and yes, we start Simba data series in 1871 and this is already quite ancient history (investment-wise)!

Post Reply