Source is Daily Kos.
S&P 500 Visualization: Worst. Year. Ever.
S&P 500 Visualization: Worst. Year. Ever.
I love this visualization of returns of the S&P 500. Who knew it's been around since 1825? And what a great illustration of the Normal Distribution.

Source is Daily Kos.
Source is Daily Kos.
- Henry Sadovsky
- Posts: 748
- Joined: Thu May 08, 2008 1:37 am
How normal is normal?
Hi Dan.
That is almost certainly not a normal distribution. You might take a look at: The Misbehavior of Markets: A Fractal View of Risk, Ruin & Reward by Benoit Mandelbrot.
H.
That is almost certainly not a normal distribution. You might take a look at: The Misbehavior of Markets: A Fractal View of Risk, Ruin & Reward by Benoit Mandelbrot.
H.
"What we can't say we can't say, and we can't whistle it either." |
Frank P. Ramsey" |
|
(f.k.a. Zalzel)
-
- Posts: 1292
- Joined: Mon Mar 31, 2008 9:26 am
Basically it's the "similar" that gets us. It makes real risks harder (impossible) to compute. On the other hand, from [the] graph we can see that the S&P 500 pays off more often than not, which is why it's still here, after all those years.mark500 wrote:That is almost certainly not a normal distribution.
Help me with statistics here. It looks similar to a bell curve.
"Simplicity is the ultimate sophistication."
Two things. I don't think that graphical depiction is centered on the mean. A minor nit.mark500 wrote:That is almost certainly not a normal distribution.
Help me with statistics here. It looks similar to a bell curve.
Notice in particular that the right side tail is rather different from the left side tail.
That said, if you play with a normal random number generator, and pick say 100 samples and plot hem, repeat, repeat, repeat, you see lots of graphs that are not all that normal looking. Do it with 1000 or 10,000 and things start to smooth out a lot better.
Also to the OP, the S&P 500 does not extend back to 1825. I'd have to look it up, but prior to about the 1950s those data are reconstructed by people looking back at stock records. My understanding that much before 1900, maybe 1920s those data reconstructions are pretty shaky.
Still, none of that invalidates the graph showing that is year is truly horrible by any historical measure.
We live a world with knowledge of the future markets has less than one significant figure. And people will still and always demand answers to three significant digits.
- jeffyscott
- Posts: 8267
- Joined: Tue Feb 27, 2007 9:12 am
- Location: Wisconsin
Not sure that looking at discrete calendar years is all that meaningful in putting declines in perspective. 1973-74 saw about a 50% decline as did 2000-2002, I don't know that having it all in one calendar year (hopefully) this time really makes much difference.
Time is your friend; impulse is your enemy. - John C. Bogle
-
- Posts: 2556
- Joined: Tue Feb 27, 2007 9:17 pm
- Location: San Jose, CA
It is similar to a bell curve.mark500 wrote:That is almost certainly not a normal distribution.
Help me with statistics here. It looks similar to a bell curve.
The OP called it an great illustration of a normal distribution. But it is really just a sample. It could be considered the sort of sample that one would get by sampling a normal distribution. Also, its the sort of sample that one would get from sampling any of an infinite number of other distributions.
The claim that it is almost certainly not a normal distribution is a claim about the underlying theoretical distribution from which the sample is drawn. But this has nothing to do with the sample, the sample does not prove it's not normal. It's a claim based on some other theory or observations.
-
- Posts: 2556
- Joined: Tue Feb 27, 2007 9:17 pm
- Location: San Jose, CA
I looked at it from 1950-2008 and it doesn't look normal, but a histogram is a poor way of judging normality.
Running an Anderson-Darling test gives a p-value of 0.442, which is consistent with the data being normal. However to disprove normality for something that is close to normal one normally needs more data points than ~50.
Running an Anderson-Darling test gives a p-value of 0.442, which is consistent with the data being normal. However to disprove normality for something that is close to normal one normally needs more data points than ~50.
PS With respect to the statistics, there are tests for normality and characterizations of properties of data distributions that reflect the shape of the distribution. In the end it is usually appropriate to find what sort of distribution is a good fit to the data, if any of the usual suspects is appropriate at all.
That data might not be normally distributed and that data might not be well described by any of a variety of mathematically tractable representations are two completely different issues. A different issue is the problem of how to represent rare events in a distribution that is generally well matched to a standard model overall. All of these problems are not novel and are much studied by those who need to deal with the relevant subjects. It may well be that success in this area when applied to markets leaves much to be desired.
That data might not be normally distributed and that data might not be well described by any of a variety of mathematically tractable representations are two completely different issues. A different issue is the problem of how to represent rare events in a distribution that is generally well matched to a standard model overall. All of these problems are not novel and are much studied by those who need to deal with the relevant subjects. It may well be that success in this area when applied to markets leaves much to be desired.
The Wikipedia says the 500 goes back to 1957. Everything before that is a reconstruction, or based on some similar index. There were S&P indexes before 1957, but not with 500 stocks.dbr wrote:According to the blogger the S&P goes back to 1825!?!?TheEternalVortex wrote:Where is the source for the data? The S&P500 doesn't go back that far...
I am not sure why the reconstructions stopped in 1825. Maybe there were just too few joint stock companies before that, or maybe there are no records.
- Rick Ferri
- Posts: 8933
- Joined: Mon Feb 26, 2007 11:40 am
- Location: Georgetown, TX. Twitter: @Rick_Ferri
- Contact:
Re: S&P 500 Visualization: Worst. Year. Ever.
[quote="Dan Kohn"]I love this visualization of returns of the S&P 500. Who knew it's been around since 1825? And what a great illustration of the Normal Distribution.
The graph supports two points of view that we’ve been hearing for quite sometime: 1) Gary Schilling, who has been right on every call he’s made throughout this horrible year, places a fair value of “600” on the S&P; 2) Myriad analysts keep insisting that at these levels we are being presented with a buying opportunity that will not present itself again in our lifetime.
The graph supports two points of view that we’ve been hearing for quite sometime: 1) Gary Schilling, who has been right on every call he’s made throughout this horrible year, places a fair value of “600” on the S&P; 2) Myriad analysts keep insisting that at these levels we are being presented with a buying opportunity that will not present itself again in our lifetime.
Re: S&P 500 Visualization: Worst. Year. Ever.
It would certainly be interesting to debate who has made the wisest decision between those dumping everything into TIPS and those dumping everything into equities. Should we have another Swedroe vs Ferriregmac wrote:Dan Kohn wrote:I love this visualization of returns of the S&P 500. Who knew it's been around since 1825? And what a great illustration of the Normal Distribution.
The graph supports two points of view that we’ve been hearing for quite sometime: 1) Gary Schilling, who has been right on every call he’s made throughout this horrible year, places a fair value of “600” on the S&P; 2) Myriad analysts keep insisting that at these levels we are being presented with a buying opportunity that will not present itself again in our lifetime.

Index is not down 50%
How did they arrive that the S&P 500 index is down 50% in 2008? According to M* the Vanguard S&P 500 Index fund is down about 41% year-to-date and also down about 41% over the last twelve months. Thru the end of November the Index was down almost 38%. Those are really bad numbers, but they are not 50%.
Bob K
Bob K
In finance risk is defined as uncertainty that is consequential (nontrivial). |
The two main methods of dealing with financial risk are the matching of assets to goals & diversifying.
Re: Index is not down 50%
It think that "-50" must mean "-40 to -50". Look at the X axis tick labels, that's all that makes sense.bobcat2 wrote:How did they arrive that the S&P 500 index is down 50% in 2008? According to M* the Vanguard S&P 500 Index fund is down about 41% year-to-date and also down about 41% over the last twelve months. Those are really bad numbers, but they are not 50%.
Bob K
Re: S&P 500 Visualization: Worst. Year. Ever.
Well, I think we got down to these P/E levels in 2003, and we will never see it again in our lifetimes? Is there an asteroid out there?regmac wrote:Dan Kohn wrote:I love this visualization of returns of the S&P 500. Who knew it's been around since 1825? And what a great illustration of the Normal Distribution.
The graph supports two points of view that we’ve been hearing for quite sometime: 1) Gary Schilling, who has been right on every call he’s made throughout this horrible year, places a fair value of “600” on the S&P; 2) Myriad analysts keep insisting that at these levels we are being presented with a buying opportunity that will not present itself again in our lifetime.
tadamsmar
Yes, you are right. -50% does not mean -50% or worse. Instead it means worse than -40%, but better than -50%. So if one draws the graph using end of November data, 2008 returns would not be in the left most column, but instead one column to the right. But if you draw the graph with data so far this week then 2008 gets dumped into the left most column. So where will it be by December 31? It could stay in the left column, move one column rightward, or find itself in a brand new -60% column. :lol:
Bob K
edit - It looks like the graph is correct (2008 worse than 1931) using returns from close of business on Monday, but incorrect as of close of business on Tuesday. Who knows how accurate by close of business today
Bob K
edit - It looks like the graph is correct (2008 worse than 1931) using returns from close of business on Monday, but incorrect as of close of business on Tuesday. Who knows how accurate by close of business today

Last edited by bobcat2 on Wed Dec 03, 2008 11:56 am, edited 1 time in total.
In finance risk is defined as uncertainty that is consequential (nontrivial). |
The two main methods of dealing with financial risk are the matching of assets to goals & diversifying.
- fluffyistaken
- Posts: 1435
- Joined: Fri Apr 04, 2008 1:32 pm
Re: S&P 500 Visualization: Worst. Year. Ever.
P/E is definitely lower now than in 2003: http://www2.standardandpoors.com/spf/xl ... EPSEST.XLStadamsmar wrote:regmac wrote:Well, I think we got down to these P/E levels in 2003, and we will never see it again in our lifetimes? Is there an asteroid out there?Dan Kohn wrote:I love this visualization of returns of the S&P 500. Who knew it's been around since 1825? And what a great illustration of the Normal Distribution.
The graph supports two points of view that we’ve been hearing for quite sometime: 1) Gary Schilling, who has been right on every call he’s made throughout this horrible year, places a fair value of “600” on the S&P; 2) Myriad analysts keep insisting that at these levels we are being presented with a buying opportunity that will not present itself again in our lifetime.
http://www.marketwatch.com/news/story/F ... 6DFD1EE%7D
The S&P's dividend yield now higher than the 10-year T-note's
The S&P's dividend yield now higher than the 10-year T-note's
For the first time since 1958, the dividend yield on the Standard & Poor's 500 index has risen above the interest rate on the 10-year Treasury note. As of Tuesday's close, for example, the S&P's dividend yield was 3.3%, while the 10-year T-Note was yielding 2.7% -- a spread of 0.6 percentage points in favor of stocks.
To put this in historical context, the spread since 1958 has averaged 3.7 percentage points in favor of bonds.
Cliff Asness, however, thinks he has come up with some clear answers. He is managing and founding principal at AQR Capital Management, a Greenwich, CT-based quantitative research firm. In the March/April 2000 issue of Financial Analysts Journal, Asness argued that neither the pre-1958 period nor the decades since are anomalous. On the contrary, he found that -- below the surface -- stock and bond yields have always been strongly positively correlated.
The reason that this strong correlation was hidden, according to Asness, is that investors' expectations have changed of the relative volatility of stocks and bonds. Prior to the last 50 years, investors expected the stock market's volatility to be much greater than bond market volatility. To entice investors to incur that greater volatility, the stock market had to provide a higher yield than bonds.
These expected relative volatilities changed early in the post-war period, according to Asness. And that is why -- beginning in 1958 -- the stock market's yield dropped below that of bonds, and stayed there for five decades.
The situation has reversed itself in recent months, according to Asness' theory, given stocks' extraordinary volatility.
To bet on whether stocks' dividend yield recent move above the T-Note yield will persist, therefore, you have to bet on whether investors will expect stocks' volatility, relative to bonds', to remain considerably higher than they have averaged over the last 50 years.
Note that the key element of this bet is investors' expectations. Asness points out that investors' memories live for a very long time. The memory of the Great Depression lingered for years after it ended, for example, which is one reason why stocks' dividend yields remained so high.
In contrast, the Baby Boom generation (at least possibly until now) had no traumatic memory similar to their parents' memory of the Great Depression. That in turn helps to explain why stocks' dividend yield slipped in the 1960s and beyond, relative to bonds' yields.
SURGEON GENERAL'S WARNING: Any overconfidence in your ability, willingness and need to take risk may be hazardous to your health.
Let's go for the record
Come one, people. How often do you get the chance to be part of something big like this? This is a rare opportunity--a once in a lifetime event.
So are we going to let 1931 beat us?
All they had back then was telephones, pencils, paper, ledgers and balance sheets.
We now have computers, high-speed networks, the internet, modern portfolio theory, ETFs, credit default swaps, and mortgage-backed securities.
We now have Financial Weapons of Mass Destruction--something that Wall Street professionals in 1931 couldn't have imagined in their wildest dreams!
I say go for the record. Let's bring it down 60% for 2008 !
Are you with me?
I said, are you with me?
So are we going to let 1931 beat us?
All they had back then was telephones, pencils, paper, ledgers and balance sheets.
We now have computers, high-speed networks, the internet, modern portfolio theory, ETFs, credit default swaps, and mortgage-backed securities.
We now have Financial Weapons of Mass Destruction--something that Wall Street professionals in 1931 couldn't have imagined in their wildest dreams!
I say go for the record. Let's bring it down 60% for 2008 !
Are you with me?
I said, are you with me?
- jeffyscott
- Posts: 8267
- Joined: Tue Feb 27, 2007 9:12 am
- Location: Wisconsin
Cliff Asness
I don't understand that comment. Why can't I simply buy stocks and happily collect the (higher than bond) dividends, without it being a "bet" that they will remain above bonds?To bet on whether stocks' dividend yield recent move above the T-Note yield will persist, therefore, you have to bet on whether investors will expect stocks' volatility, relative to bonds', to remain considerably higher than they have averaged over the last 50 years.
Time is your friend; impulse is your enemy. - John C. Bogle
Re: Let's go for the record
Am I the only one seeing the good part of the graph? When I look at the graph I see the Up years that follow the down years. Let's just make it through the next year or so then start reaping the reward.grayfox wrote:Come one, people. How often do you get the chance to be part of something big like this? This is a rare opportunity--a once in a lifetime event.
So are we going to let 1931 beat us?
...
BearWolf
- Christine_NM
- Posts: 2699
- Joined: Tue Feb 20, 2007 1:13 am
- Location: New Mexico
It's the uptick rule, which began after 1937 and ended in July 2007. 1931, 1937, and 2008 all are years with no uptick rule.
Without the uptick rule, volatility will continue and there will be no "back to normal" for the market.
Facing this, in order to maintain an overall portfolio at an unchanged risk tolerance then one must permanently reduce the equity allocation. Or reduce it till the SEC comes to its senses and restores the uptick rule.
I'm not sure buy and hold makes sense without an uptick rule. So far, it doesn't. Something like Taleb's 5-10% stock trading and 90-95% Treasuries is more appropriate for a riskier (more volatile) market.
Without the uptick rule, volatility will continue and there will be no "back to normal" for the market.
Facing this, in order to maintain an overall portfolio at an unchanged risk tolerance then one must permanently reduce the equity allocation. Or reduce it till the SEC comes to its senses and restores the uptick rule.
I'm not sure buy and hold makes sense without an uptick rule. So far, it doesn't. Something like Taleb's 5-10% stock trading and 90-95% Treasuries is more appropriate for a riskier (more volatile) market.
Re: Let's go for the record
I'm with you!grayfox wrote:Come one, people. How often do you get the chance to be part of something big like this? This is a rare opportunity--a once in a lifetime event.
So are we going to let 1931 beat us?
All they had back then was telephones, pencils, paper, ledgers and balance sheets.
We now have computers, high-speed networks, the internet, modern portfolio theory, ETFs, credit default swaps, and mortgage-backed securities.
We now have Financial Weapons of Mass Destruction--something that Wall Street professionals in 1931 couldn't have imagined in their wildest dreams!
I say go for the record. Let's bring it down 60% for 2008 !
Are you with me?
I said, are you with me?
If I'm going to go through this much pain, I want a record dang it!
Then we can have a record recovery.
We live a world with knowledge of the future markets has less than one significant figure. And people will still and always demand answers to three significant digits.
- jeffyscott
- Posts: 8267
- Joined: Tue Feb 27, 2007 9:12 am
- Location: Wisconsin
-
- Posts: 2556
- Joined: Tue Feb 27, 2007 9:17 pm
- Location: San Jose, CA
The existence of volatile years while the rule was in place contradicts your hypothesis.Christine_NM wrote:It's the uptick rule, which began after 1937 and ended in July 2007. 1931, 1937, and 2008 all are years with no uptick rule.
Without the uptick rule, volatility will continue and there will be no "back to normal" for the market.
Facing this, in order to maintain an overall portfolio at an unchanged risk tolerance then one must permanently reduce the equity allocation. Or reduce it till the SEC comes to its senses and restores the uptick rule.
I'm not sure buy and hold makes sense without an uptick rule. So far, it doesn't. Something like Taleb's 5-10% stock trading and 90-95% Treasuries is more appropriate for a riskier (more volatile) market.
And it would be difficult to take advantage of purchasing equities at those lower prices if you don't have a job because of the business downturn!jeffyscott wrote:I hope those of you begging for -60% for 2008 realize that this would mean 587 which would be -31% from yesterdays close.

SURGEON GENERAL'S WARNING: Any overconfidence in your ability, willingness and need to take risk may be hazardous to your health.
I'm willing to "win" by 0.01%jeffyscott wrote:I hope those of you begging for -60% for 2008 realize that this would mean 587 which would be -31% from yesterdays close.
We live a world with knowledge of the future markets has less than one significant figure. And people will still and always demand answers to three significant digits.
- Christine_NM
- Posts: 2699
- Joined: Tue Feb 20, 2007 1:13 am
- Location: New Mexico
No year with an uptick rule has been as volatile in a bad way as the three outliers, 31, 37 and 08. The uptick rule will not cure all ills but will reduce volatility, which is the common measure of risk.TheEternalVortex wrote:The existence of volatile years while the rule was in place contradicts your hypothesis.Christine_NM wrote:It's the uptick rule, which began after 1937 and ended in July 2007. 1931, 1937, and 2008 all are years with no uptick rule.
Without the uptick rule, volatility will continue and there will be no "back to normal" for the market.
Facing this, in order to maintain an overall portfolio at an unchanged risk tolerance then one must permanently reduce the equity allocation. Or reduce it till the SEC comes to its senses and restores the uptick rule.
I'm not sure buy and hold makes sense without an uptick rule. So far, it doesn't. Something like Taleb's 5-10% stock trading and 90-95% Treasuries is more appropriate for a riskier (more volatile) market.
It's like walking a high wire with a net vs. without a net. You might do the former, but not so much the latter. Ignoring the fact that the net has been removed while we are in midwalk (invested in stocks) seems unwise to me.
tadamsmar said:
I especially liked this one:The comments section of the link in the OP is interesting.
Anyone yearning for a good economics thread with politics and crude humor mixed in should check it out, the stuff we can't have here.
In particular, the photo of the guy carrying the "Jump you &%$s!" sign past the NYSE.
I kind of like the idea of throwing money at the problem till we get rid of it all. Money is the root of all evil. What we are doing is making sure future generations won't have to worry about it.
Re: Let's go for the record
They had airplanes and radios.grayfox wrote:Come one, people. How often do you get the chance to be part of something big like this? This is a rare opportunity--a once in a lifetime event.
So are we going to let 1931 beat us?
All they had back then was telephones, pencils, paper, ledgers and balance sheets.
We now have computers, high-speed networks, the internet, modern portfolio theory, ETFs, credit default swaps, and mortgage-backed securities.
We now have Financial Weapons of Mass Destruction--something that Wall Street professionals in 1931 couldn't have imagined in their wildest dreams!
I say go for the record. Let's bring it down 60% for 2008 !
Are you with me?
I said, are you with me?
What I don't like about the original graph in this post is the "arithmetic" as opposed to the "geometric" nature of it. There seems to be an implication that a "+10 to +20" year balances out a "-10 to -20" year. Or that the "+50 to +60" years even "surpass" the losses of the "-40 to -30" years. If the "x-axis" was spaced in a "log" fashion, then the "-50" block would have to be as far to the left as a "+100" block. A "-40" block would be "balanced" by a +67 block.
This would make those blocks on the left side, further to the left, and the "normalness" would be "less normal".
(That multiyear 1930-1932 "-90 to -80" block would need a "+500 to +1000" block on the right to counteract it!).
-Brad.
This would make those blocks on the left side, further to the left, and the "normalness" would be "less normal".
(That multiyear 1930-1932 "-90 to -80" block would need a "+500 to +1000" block on the right to counteract it!).
-Brad.
Well, the graph is currently skewed to the right so arguably it'd actually make it MORE normal to somehow overweight losses, but there's really no perfect way to do this sort of thing as even the height and width of the boxes affects the appearance. All of that said, my sense is that a lot of the regression techniques used in the econ papers we read assume normality without even bothering to check that assumption as did Eternal Vortex above (and in this case the assumption checks out OK).madsinger wrote:What I don't like about the original graph in this post is the "arithmetic" as opposed to the "geometric" nature of it. There seems to be an implication that a "+10 to +20" year balances out a "-10 to -20" year. Or that the "+50 to +60" years even "surpass" the losses of the "-40 to -30" years. If the "x-axis" was spaced in a "log" fashion, then the "-50" block would have to be as far to the left as a "+100" block. A "-40" block would be "balanced" by a +67 block.
This would make those blocks on the left side, further to the left, and the "normalness" would be "less normal".
(That multiyear 1930-1932 "-90 to -80" block would need a "+500 to +1000" block on the right to counteract it!).
-Brad.
All best,
Pete
-
- Posts: 343
- Joined: Mon Feb 18, 2008 4:46 pm
- Location: Somewhere between Appalachia and Atlantic
Re: Let's go for the record
I am with both of you. I hope that S&P stay under 800 for next couple of years, so that i can have a decent base of investment at good prices. I am willing to live with pain for next few years.Rodc wrote:I'm with you!grayfox wrote:Come one, people. How often do you get the chance to be part of something big like this? This is a rare opportunity--a once in a lifetime event.
So are we going to let 1931 beat us?
All they had back then was telephones, pencils, paper, ledgers and balance sheets.
We now have computers, high-speed networks, the internet, modern portfolio theory, ETFs, credit default swaps, and mortgage-backed securities.
We now have Financial Weapons of Mass Destruction--something that Wall Street professionals in 1931 couldn't have imagined in their wildest dreams!
I say go for the record. Let's bring it down 60% for 2008 !
Are you with me?
I said, are you with me?
If I'm going to go through this much pain, I want a record dang it!
Then we can have a record recovery.
Everything that you own, owns piece of you.
- Murray Boyd
- Posts: 794
- Joined: Mon Feb 19, 2007 6:00 pm
[NB: Before you take any of the following analysis seriously, be sure to read and understand the last paragraph.]mark500 wrote:Help me with statistics here. It looks similar to a bell curve.Henry Sadovsky wrote:That is almost certainly not a normal distribution.
As to whether the original data is a histogram derived from a normal distribution, I can offer no opinion. There simply aren't enough data points to tell, now that it's been reduced to a histogram. Almost any test of normality (and I note with approval that somebody tried Anderson-Darling

However, we can take a look at Shiller's dataset of S&P composite returns from 1871-present. Now, he's done a lot of fiddling around to get a continuous series of index price levels, dividend values, and inflation measures that far back. His web site will tell you all about it; for purposes of this post, I'm going to take him at face value and assume that a highly competent economist such as himself has done a credible job.
So I took his yearly data series and simplified it down until it contained just what we need: Year, Price, Dividend, and CPI. I loaded those into R and repeated his spreadsheet calculations to derive the total return, as well as real (inflation-adjusted) index price level, dividend payment, and thus a real return. At the end of that, I was armed with a time series of year, nominal return, and real return from 1871 - 2006. (I had to fiddle with the 2007 data to supply enough information to calculate returns through 2006; obviously I can't get the 2008 numbers yet.)
R script and input data available from me upon PM request.
Fine. Now, are the returns normally distributed or not? To answer that, I ran 8 different tests of normality on the nominal and real returns. I won't bother you about the details of the differences between the tests, and their different sensitivities, because the verdict is unanimous: the returns are normally distributed.
In the tables that follow, the thing to keep your eye on is the p-value of the test, in the rightmost column. That's the probability of seeing a dataset like yours, with the deviations from normality that eventually creep in, even if the source of the data is really a normal distribution (the "null hypothesis" of the tests). That is, you should only believe the data is not normal if p<0.05 or so. Otherwise you should believe it's normal.
Here's the result for nominal returns, showing the test results in a table and a quantile-quantile plot. In the QQ plot, normal data will lie along the blue diagonal line, so this is a visual confirmation of what the table says. As you can see, it's pretty convincing.
Code: Select all
Test Name Statistic Value P
========= ========= ===== =
Shapiro-Wilk normality W 0.9970 0.991
Robust Jarque Bera X-squared 0.0293 0.985
Standardized SJ SJ 0.0961 0.463
Anderson-Darling A 0.1240 0.987
Cramer-von Mises W 0.0185 0.979
Lilliefors (Kolmogorov-Smirnov) D 0.0379 0.903
Pearson chi-square P 18.4000 0.104
Shapiro-Francia W 0.9960 0.955
Mean = 10.5%
StdDev = 17.7%
N = 136

Here's the result for the real returns:
Code: Select all
Test Name Statistic Value P
========= ========= ===== =
Shapiro-Wilk normality W 0.9950 0.893
Robust Jarque Bera X-squared 0.0312 0.985
Standardized SJ SJ 0.1370 0.447
Anderson-Darling A 0.1540 0.956
Cramer-von Mises W 0.0192 0.974
Lilliefors (Kolmogorov-Smirnov) D 0.0385 0.892
Pearson chi-square P 7.8200 0.799
Shapiro-Francia W 0.9960 0.930
Mean = 8.25%
StdDev = 17.5%
N = 136

Of course, the real question everybody wants to ask is, "What does that tell us about how crappy this year is?" Nominal returns have a mean of 10.5% and a standard deviation of 17.7%. This year's VFINX (Vanguard 500 Index) return through November was -37.7%. That's in the tail with probability less than 0.323%; figure about once every 300 years.
A cautionary note: This analysis is meaning-free; it's what happens when a statistics nerd gets bored. Impressive as Shiller's data is, it simply does not adequately sample the world to predict a black swan. In fact, that's essentially what Taleb and others are saying: due to the fat-tailed nature of the return distributions, no amount of historical samples can adequately fit a model which includes black swans. So my confident statement that this is a "once every 300 years" market is hogwash (especially when derived from a model with only 136 years of history!

- Henry Sadovsky
- Posts: 748
- Joined: Thu May 08, 2008 1:37 am
Assessing for normality
.
Hi sgr000.
That was very interesting. Does not the use of yearly data unnecessarily limit the power of the analysis? My understanding of the fat-tail question is that it is not a conjecture, but that fat-tails are seen in the analysis of daily data. I don't have the data at hand, but I seem to recall something to the effect that if (U.S. equity) market returns were normally distributed, the probability of seeing days like 10/19/1987, and all the +/- >3% days we are seeing this year would be vanishingly small.
Hi sgr000.
That was very interesting. Does not the use of yearly data unnecessarily limit the power of the analysis? My understanding of the fat-tail question is that it is not a conjecture, but that fat-tails are seen in the analysis of daily data. I don't have the data at hand, but I seem to recall something to the effect that if (U.S. equity) market returns were normally distributed, the probability of seeing days like 10/19/1987, and all the +/- >3% days we are seeing this year would be vanishingly small.
"What we can't say we can't say, and we can't whistle it either." |
Frank P. Ramsey" |
|
(f.k.a. Zalzel)
Outstanding
sgr000, thanks for the outstanding work.
The best way would be to follow the link above to Shiller's yearly data spreadsheet. He calculates real returns only, but you can look at it in Excel to imitate a calculation for nominal returns if you want.grayfox wrote:Outstanding work! This forum never ceases to amaze me.
I have a request. Would it be possible to post a table of nominal return and real return for each year from 1871 to present?
Presumably Shiller is updating his data. I won't be updating this. So what you have below is just a one-shot deal; you're better off going with Shiller.
- P is the index price at the beginning of the year,
- D is the dividend paid throughout the year,
- CPI is the CPI level,
- RealP is the inflation-adjusted index price (using the current year's CPI),
- RealD is the inflation-adjusted dividend (using next year's CPI, essentially assuming with Shiller that dividends are paid at the end of the year),
- RealReturn is the real return: (RealPnextyear - RealP + RealD) / RealP,
- Return is the nominal return: (Pnextyear - P + D) / P
Code: Select all
Year P D CPI RealP RealD RealReturn Return
1871 4.44 0.26 12.460 72.11 4.159 0.1358000 0.153200
1872 4.86 0.30 12.650 77.74 4.693 0.0886200 0.113200
1873 5.11 0.33 12.940 79.94 5.400 0.0215900 -0.023480
1874 4.66 0.33 12.370 76.26 5.802 0.1228000 0.045060
1875 4.54 0.30 11.510 79.82 5.599 0.1128000 0.048460
1876 4.46 0.30 10.850 83.23 5.550 -0.1443000 -0.136800
1877 3.55 0.19 10.940 65.67 4.167 0.1488000 -0.030990
1878 3.25 0.18 9.229 71.28 4.402 0.2899000 0.156900
1879 3.58 0.20 8.278 87.54 4.052 0.2290000 0.483200
1880 5.11 0.26 9.990 103.50 5.587 0.3387000 0.262200
1881 6.19 0.32 9.419 133.00 6.362 -0.0672900 0.008078
1882 5.92 0.32 10.180 117.70 6.484 0.0551900 0.035470
1883 5.81 0.33 9.990 117.70 7.238 0.0265900 -0.051640
1884 5.18 0.31 9.229 113.60 7.581 -0.0206600 -0.121600
1885 4.24 0.24 8.278 103.70 6.078 0.3288000 0.283000
1886 5.20 0.22 7.992 131.70 5.572 0.1154000 0.115400
1887 5.58 0.25 7.992 141.30 6.044 -0.0488800 -0.003584
1888 5.31 0.23 8.373 128.40 5.825 0.0791900 0.030130
1889 5.24 0.22 7.992 132.70 5.850 0.1221000 0.068700
1890 5.38 0.22 7.612 143.10 5.708 -0.0824200 -0.059480
1891 4.84 0.22 7.802 125.60 6.078 0.2608000 0.183900
1892 5.51 0.24 7.326 152.20 6.152 -0.0150400 0.061710
1893 5.61 0.25 7.897 143.80 7.387 -0.0609300 -0.185400
1894 4.32 0.21 6.850 127.60 6.475 0.0772900 0.032410
1895 4.25 0.19 6.565 131.00 5.774 0.0344200 0.049410
1896 4.27 0.18 6.660 129.80 5.631 0.0607500 0.030440
1897 4.22 0.18 6.470 132.00 5.471 0.1648000 0.199100
1898 4.88 0.20 6.660 148.30 5.993 0.2688000 0.286900
1899 6.08 0.21 6.755 182.20 5.383 -0.1122000 0.037830
1900 6.10 0.30 7.897 156.40 7.879 0.2380000 0.208200
1901 7.07 0.32 7.707 185.70 8.202 0.1650000 0.193800
1902 8.12 0.33 7.897 208.10 7.715 -0.0126500 0.082510
1903 8.46 0.35 8.658 197.80 8.559 -0.1308000 -0.169000
1904 6.68 0.31 8.278 163.30 7.410 0.2790000 0.308400
1905 8.43 0.33 8.468 201.50 7.888 0.2100000 0.210000
1906 9.87 0.40 8.468 235.90 9.150 -0.0342800 0.009119
1907 9.56 0.44 8.849 218.70 10.290 -0.2207000 -0.237400
1908 6.85 0.40 8.658 160.10 9.053 0.3370000 0.381000
1909 9.06 0.44 8.944 205.00 9.001 0.0495000 0.161100
1910 10.08 0.47 9.895 206.20 10.310 0.0360100 -0.033730
1911 9.27 0.47 9.229 203.30 10.420 0.0452900 0.034520
1912 9.12 0.48 9.134 202.10 9.914 -0.0005083 0.072370
1913 9.30 0.48 9.800 192.10 9.716 -0.0674200 -0.048390
1914 8.37 0.42 10.000 169.40 8.417 -0.0655000 -0.056150
1915 7.48 0.43 10.100 149.90 8.369 0.2672000 0.304800
1916 9.33 0.56 10.400 181.60 9.688 -0.0348900 0.085740
1917 9.57 0.69 11.700 165.60 9.976 -0.3101000 -0.174500
1918 7.21 0.57 14.000 104.20 6.993 -0.0091200 0.167800
1919 7.85 0.53 16.500 96.30 5.559 0.0193700 0.192400
1920 8.83 0.51 19.300 92.61 5.433 -0.1234000 -0.137000
1921 7.11 0.46 19.000 75.75 5.510 0.2270000 0.091420
1922 7.30 0.51 16.900 87.43 6.145 0.2967000 0.289000
1923 8.90 0.53 16.800 107.20 6.201 0.0212900 0.051690
1924 8.83 0.55 17.300 103.30 6.435 0.2605000 0.260500
1925 10.58 0.60 17.300 123.80 6.785 0.2104000 0.252400
1926 12.65 0.69 17.900 143.00 7.981 0.1393000 0.113800
1927 13.40 0.77 17.500 155.00 9.009 0.3815000 0.365700
1928 17.53 0.85 17.300 205.10 10.060 0.4838000 0.466600
1929 24.86 0.97 17.100 294.30 11.480 -0.0876900 -0.087690
1930 21.71 0.98 17.100 257.00 12.480 -0.1598000 -0.218800
1931 15.98 0.82 15.900 203.40 11.610 -0.3654000 -0.429300
1932 8.30 0.50 14.300 117.50 7.846 0.0137000 -0.085540
1933 7.09 0.44 12.900 111.30 6.747 0.5135000 0.548700
1934 10.54 0.45 13.200 161.60 6.698 -0.1058000 -0.078750
1935 9.26 0.47 13.600 137.80 6.894 0.5144000 0.536700
1936 13.76 0.72 13.800 201.80 10.340 0.3024000 0.330700
1937 17.59 0.80 14.100 252.50 11.400 -0.3164000 -0.311500
1938 11.31 0.51 14.200 161.20 7.374 0.1667000 0.150300
1939 12.50 0.62 14.000 180.70 9.029 0.0410400 0.033600
1940 12.30 0.67 13.900 179.10 9.618 -0.1007000 -0.087800
1941 10.55 0.71 14.100 151.50 9.154 -0.1794000 -0.086260
1942 8.93 0.59 15.700 115.10 7.067 0.1110000 0.196000
1943 10.09 0.61 16.900 120.90 7.096 0.1994000 0.234900
1944 11.85 0.64 17.400 137.90 7.278 0.1656000 0.192400
1945 13.49 0.66 17.800 153.40 7.340 0.3543000 0.384700
1946 18.02 0.71 18.200 200.40 6.684 -0.2521000 -0.116500
1947 15.21 0.84 21.500 143.20 7.174 -0.0653900 0.030240
1948 14.83 0.93 23.700 126.70 7.844 0.0847200 0.098450
1949 15.36 1.14 24.000 129.50 9.819 0.1981000 0.173200
1950 16.88 1.47 23.500 145.40 11.710 0.2431000 0.343600
1951 21.21 1.41 25.400 169.00 10.770 0.1569000 0.207000
1952 24.19 1.41 26.500 184.80 10.730 0.1363000 0.140600
1953 26.18 1.45 26.600 199.20 10.910 0.0164200 0.027880
1954 25.46 1.54 26.900 191.60 11.670 0.4697000 0.458800
1955 35.60 1.64 26.700 269.90 12.390 0.2814000 0.286200
1956 44.15 1.74 26.800 333.50 12.760 0.0374300 0.068400
1957 45.43 1.79 27.600 333.20 12.670 -0.0885000 -0.055470
1958 41.12 1.75 28.600 291.00 12.210 0.3759000 0.395200
1959 55.62 1.83 29.000 388.20 12.640 0.0652100 0.076230
1960 58.03 1.95 29.300 400.90 13.250 0.0449000 0.062730
1961 59.72 2.02 29.800 405.60 13.630 0.1825000 0.190400
1962 69.07 2.13 30.000 466.00 14.180 -0.0400200 -0.027220
1963 65.06 2.28 30.400 433.20 14.940 0.1905000 0.210100
1964 76.45 2.50 30.900 500.80 16.220 0.1480000 0.159200
1965 86.12 2.72 31.200 558.70 17.310 0.0941500 0.115200
1966 93.32 2.87 31.800 594.00 17.660 -0.0955800 -0.064290
1967 84.45 2.92 32.900 519.60 17.330 0.1192000 0.160000
1968 95.04 3.07 34.100 564.20 17.460 0.0593600 0.106000
1969 102.00 3.16 35.600 580.20 16.920 -0.1373000 -0.083990
1970 90.31 3.14 37.800 483.60 15.970 0.0162100 0.069980
1971 93.49 3.07 39.800 475.50 15.120 0.1018000 0.137800
1972 103.30 3.15 41.100 508.70 14.970 0.1354000 0.176900
1973 118.40 3.38 42.600 562.70 14.680 -0.2320000 -0.159900
1974 96.11 3.60 46.600 417.50 13.990 -0.2912000 -0.207600
1975 72.56 3.68 52.100 281.90 13.400 0.2984000 0.385600
1976 96.86 4.05 55.600 352.60 14.010 0.0583600 0.113600
1977 103.80 4.67 58.500 359.20 15.120 -0.1442000 -0.085640
1978 90.25 5.07 62.500 292.30 15.030 0.0624100 0.161000
1979 99.71 5.65 68.300 295.50 14.700 0.0258900 0.168600
1980 110.90 6.16 77.800 288.50 14.330 0.1222000 0.254900
1981 133.00 6.63 87.000 309.40 14.230 -0.1403000 -0.068140
1982 117.30 6.87 94.300 251.70 14.220 0.2426000 0.288700
1983 144.30 7.09 97.800 298.60 14.080 0.1541000 0.202500
1984 166.40 7.53 101.900 330.50 14.450 0.0398900 0.076630
1985 171.60 7.90 105.500 329.30 14.590 0.2121000 0.259200
1986 208.20 8.28 109.600 384.50 15.070 0.2914000 0.310300
1987 264.50 8.81 111.200 481.50 15.410 -0.0578600 -0.019730
1988 250.50 9.73 115.700 438.20 16.260 0.1258000 0.178300
1989 285.40 11.05 121.100 477.10 17.560 0.1691000 0.229900
1990 340.00 12.10 127.400 540.20 18.200 -0.0600900 -0.006971
1991 325.50 12.20 134.600 489.50 17.880 0.2824000 0.315800
1992 416.10 12.38 138.100 609.90 17.570 0.0418300 0.075780
1993 435.20 12.58 142.600 617.80 17.420 0.0881900 0.115700
1994 473.00 13.18 146.200 654.90 17.750 -0.0160900 0.011500
1995 465.20 13.79 150.300 626.60 18.080 0.3144000 0.350300
1996 614.40 14.90 154.400 805.50 18.960 0.2338000 0.271300
1997 766.20 15.50 159.100 974.80 19.410 0.2578000 0.277500
1998 963.40 16.20 161.600 1207.00 19.960 0.2915000 0.313100
1999 1249.00 16.69 164.300 1538.00 20.010 0.1242000 0.155000
2000 1426.00 16.27 168.800 1709.00 18.810 -0.0889900 -0.054990
2001 1331.00 15.74 175.100 1539.00 17.990 -0.1413000 -0.131500
2002 1140.00 16.07 177.100 1303.00 17.900 -0.2205000 -0.200200
2003 895.80 17.39 181.700 998.00 19.010 0.2025000 0.225700
2004 1081.00 19.44 185.200 1181.00 20.630 0.0952200 0.127700
2005 1199.00 22.22 190.700 1273.00 22.680 0.0299100 0.070960
2006 1262.00 24.88 198.300 1288.00 24.880 0.1188000 0.142000
2007 1416.42 27.73 202.416 NA NA NA NA