S&P 500 Visualization: Worst. Year. Ever.

Discuss all general (i.e. non-personal) investing questions and issues, investing news, and theory.
User avatar
Topic Author
Dan Kohn
Posts: 1505
Joined: Tue Jun 26, 2007 12:15 am
Location: New York, NY
Contact:

S&P 500 Visualization: Worst. Year. Ever.

Post by Dan Kohn » Wed Dec 03, 2008 8:32 am

I love this visualization of returns of the S&P 500. Who knew it's been around since 1825? And what a great illustration of the Normal Distribution.

Image

Source is Daily Kos.

User avatar
Roverdog
Posts: 476
Joined: Fri May 02, 2008 11:39 am

Post by Roverdog » Wed Dec 03, 2008 8:42 am

Dan,

Thanks for the graph (and reference).

I give a few lectures in a financial planning elective here at school, and that's a keeper! :)

Bob

User avatar
Henry Sadovsky
Posts: 748
Joined: Thu May 08, 2008 1:37 am

How normal is normal?

Post by Henry Sadovsky » Wed Dec 03, 2008 8:57 am

Hi Dan.

That is almost certainly not a normal distribution. You might take a look at: The Misbehavior of Markets: A Fractal View of Risk, Ruin & Reward by Benoit Mandelbrot.

H.
"What we can't say we can't say, and we can't whistle it either." | Frank P. Ramsey" | | (f.k.a. Zalzel)

mark500
Posts: 392
Joined: Sat Oct 11, 2008 8:37 am

Post by mark500 » Wed Dec 03, 2008 9:18 am

That is almost certainly not a normal distribution.

Help me with statistics here. It looks similar to a bell curve.

superlight
Posts: 1292
Joined: Mon Mar 31, 2008 9:26 am

Post by superlight » Wed Dec 03, 2008 9:24 am

mark500 wrote:That is almost certainly not a normal distribution.

Help me with statistics here. It looks similar to a bell curve.
Basically it's the "similar" that gets us. It makes real risks harder (impossible) to compute. On the other hand, from [the] graph we can see that the S&P 500 pays off more often than not, which is why it's still here, after all those years.
"Simplicity is the ultimate sophistication."

Rodc
Posts: 13601
Joined: Tue Jun 26, 2007 9:46 am

Post by Rodc » Wed Dec 03, 2008 9:42 am

mark500 wrote:That is almost certainly not a normal distribution.

Help me with statistics here. It looks similar to a bell curve.
Two things. I don't think that graphical depiction is centered on the mean. A minor nit.

Notice in particular that the right side tail is rather different from the left side tail.

That said, if you play with a normal random number generator, and pick say 100 samples and plot hem, repeat, repeat, repeat, you see lots of graphs that are not all that normal looking. Do it with 1000 or 10,000 and things start to smooth out a lot better.

Also to the OP, the S&P 500 does not extend back to 1825. I'd have to look it up, but prior to about the 1950s those data are reconstructed by people looking back at stock records. My understanding that much before 1900, maybe 1920s those data reconstructions are pretty shaky.

Still, none of that invalidates the graph showing that is year is truly horrible by any historical measure.
We live a world with knowledge of the future markets has less than one significant figure. And people will still and always demand answers to three significant digits.

User avatar
stratton
Posts: 11082
Joined: Sun Mar 04, 2007 5:05 pm
Location: Puget Sound

Post by stratton » Wed Dec 03, 2008 9:54 am

Considering current daily volatility we could be at -25 or -30% by the end of the year. Or down more.

Paul

ziggy29
Posts: 925
Joined: Mon Mar 10, 2008 7:08 pm
Location: Southeast Texas Boonies

Post by ziggy29 » Wed Dec 03, 2008 9:56 am

stratton wrote:Considering current daily volatility we could be at -25 or -30% by the end of the year. Or down more.
And wait until the tax loss harvesting starts up in earnest...

User avatar
jeffyscott
Posts: 8267
Joined: Tue Feb 27, 2007 9:12 am
Location: Wisconsin

Post by jeffyscott » Wed Dec 03, 2008 10:13 am

Not sure that looking at discrete calendar years is all that meaningful in putting declines in perspective. 1973-74 saw about a 50% decline as did 2000-2002, I don't know that having it all in one calendar year (hopefully) this time really makes much difference.
Time is your friend; impulse is your enemy. - John C. Bogle

maywood
Posts: 256
Joined: Tue Jun 19, 2007 9:52 pm

Post by maywood » Wed Dec 03, 2008 10:22 am

Very interesting, thanks for posting!

The only good thing I could come up with was, if someone survived this year they should be able to survive anything!

Maywood

TheEternalVortex
Posts: 2556
Joined: Tue Feb 27, 2007 9:17 pm
Location: San Jose, CA

Post by TheEternalVortex » Wed Dec 03, 2008 10:44 am

Where is the source for the data? The S&P500 doesn't go back that far...

dbr
Posts: 30798
Joined: Sun Mar 04, 2007 9:50 am

Post by dbr » Wed Dec 03, 2008 10:49 am

TheEternalVortex wrote:Where is the source for the data? The S&P500 doesn't go back that far...
According to the blogger the S&P goes back to 1825!?!?

User avatar
tadamsmar
Posts: 8537
Joined: Mon May 07, 2007 12:33 pm

Post by tadamsmar » Wed Dec 03, 2008 10:49 am

mark500 wrote:That is almost certainly not a normal distribution.

Help me with statistics here. It looks similar to a bell curve.
It is similar to a bell curve.

The OP called it an great illustration of a normal distribution. But it is really just a sample. It could be considered the sort of sample that one would get by sampling a normal distribution. Also, its the sort of sample that one would get from sampling any of an infinite number of other distributions.

The claim that it is almost certainly not a normal distribution is a claim about the underlying theoretical distribution from which the sample is drawn. But this has nothing to do with the sample, the sample does not prove it's not normal. It's a claim based on some other theory or observations.

TheEternalVortex
Posts: 2556
Joined: Tue Feb 27, 2007 9:17 pm
Location: San Jose, CA

Post by TheEternalVortex » Wed Dec 03, 2008 10:52 am

I looked at it from 1950-2008 and it doesn't look normal, but a histogram is a poor way of judging normality.

Running an Anderson-Darling test gives a p-value of 0.442, which is consistent with the data being normal. However to disprove normality for something that is close to normal one normally needs more data points than ~50.

dbr
Posts: 30798
Joined: Sun Mar 04, 2007 9:50 am

Post by dbr » Wed Dec 03, 2008 10:56 am

PS With respect to the statistics, there are tests for normality and characterizations of properties of data distributions that reflect the shape of the distribution. In the end it is usually appropriate to find what sort of distribution is a good fit to the data, if any of the usual suspects is appropriate at all.

That data might not be normally distributed and that data might not be well described by any of a variety of mathematically tractable representations are two completely different issues. A different issue is the problem of how to represent rare events in a distribution that is generally well matched to a standard model overall. All of these problems are not novel and are much studied by those who need to deal with the relevant subjects. It may well be that success in this area when applied to markets leaves much to be desired.

User avatar
tadamsmar
Posts: 8537
Joined: Mon May 07, 2007 12:33 pm

Post by tadamsmar » Wed Dec 03, 2008 10:58 am

dbr wrote:
TheEternalVortex wrote:Where is the source for the data? The S&P500 doesn't go back that far...
According to the blogger the S&P goes back to 1825!?!?
The Wikipedia says the 500 goes back to 1957. Everything before that is a reconstruction, or based on some similar index. There were S&P indexes before 1957, but not with 500 stocks.

I am not sure why the reconstructions stopped in 1825. Maybe there were just too few joint stock companies before that, or maybe there are no records.

User avatar
Rick Ferri
Posts: 8933
Joined: Mon Feb 26, 2007 11:40 am
Location: Georgetown, TX. Twitter: @Rick_Ferri
Contact:

Post by Rick Ferri » Wed Dec 03, 2008 10:58 am

Dan,

Maybe the axiom that for every action there is an equal and opposite reaction holds true for the stock market next year.

Rick Ferri

regmac
Posts: 29
Joined: Mon Dec 17, 2007 10:46 am

Re: S&P 500 Visualization: Worst. Year. Ever.

Post by regmac » Wed Dec 03, 2008 10:59 am

[quote="Dan Kohn"]I love this visualization of returns of the S&P 500. Who knew it's been around since 1825? And what a great illustration of the Normal Distribution.

The graph supports two points of view that we’ve been hearing for quite sometime: 1) Gary Schilling, who has been right on every call he’s made throughout this horrible year, places a fair value of “600” on the S&P; 2) Myriad analysts keep insisting that at these levels we are being presented with a buying opportunity that will not present itself again in our lifetime.

User avatar
tadamsmar
Posts: 8537
Joined: Mon May 07, 2007 12:33 pm

Post by tadamsmar » Wed Dec 03, 2008 11:01 am

Rick Ferri wrote:Dan,

Maybe the axiom that for every action there is an equal and opposite reaction holds true for the stock market next year.

Rick Ferri
Yes, but what if the 1990's is the action, and this is the reaction?

dbr
Posts: 30798
Joined: Sun Mar 04, 2007 9:50 am

Re: S&P 500 Visualization: Worst. Year. Ever.

Post by dbr » Wed Dec 03, 2008 11:04 am

regmac wrote:
Dan Kohn wrote:I love this visualization of returns of the S&P 500. Who knew it's been around since 1825? And what a great illustration of the Normal Distribution.

The graph supports two points of view that we’ve been hearing for quite sometime: 1) Gary Schilling, who has been right on every call he’s made throughout this horrible year, places a fair value of “600” on the S&P; 2) Myriad analysts keep insisting that at these levels we are being presented with a buying opportunity that will not present itself again in our lifetime.
It would certainly be interesting to debate who has made the wisest decision between those dumping everything into TIPS and those dumping everything into equities. Should we have another Swedroe vs Ferri :P

User avatar
bobcat2
Posts: 5359
Joined: Tue Feb 20, 2007 3:27 pm
Location: just barely Outside the Beltway

Index is not down 50%

Post by bobcat2 » Wed Dec 03, 2008 11:22 am

How did they arrive that the S&P 500 index is down 50% in 2008? According to M* the Vanguard S&P 500 Index fund is down about 41% year-to-date and also down about 41% over the last twelve months. Thru the end of November the Index was down almost 38%. Those are really bad numbers, but they are not 50%.

Bob K
In finance risk is defined as uncertainty that is consequential (nontrivial). | The two main methods of dealing with financial risk are the matching of assets to goals & diversifying.

User avatar
tadamsmar
Posts: 8537
Joined: Mon May 07, 2007 12:33 pm

Re: Index is not down 50%

Post by tadamsmar » Wed Dec 03, 2008 11:27 am

bobcat2 wrote:How did they arrive that the S&P 500 index is down 50% in 2008? According to M* the Vanguard S&P 500 Index fund is down about 41% year-to-date and also down about 41% over the last twelve months. Those are really bad numbers, but they are not 50%.

Bob K
It think that "-50" must mean "-40 to -50". Look at the X axis tick labels, that's all that makes sense.

User avatar
tadamsmar
Posts: 8537
Joined: Mon May 07, 2007 12:33 pm

Re: S&P 500 Visualization: Worst. Year. Ever.

Post by tadamsmar » Wed Dec 03, 2008 11:31 am

regmac wrote:
Dan Kohn wrote:I love this visualization of returns of the S&P 500. Who knew it's been around since 1825? And what a great illustration of the Normal Distribution.

The graph supports two points of view that we’ve been hearing for quite sometime: 1) Gary Schilling, who has been right on every call he’s made throughout this horrible year, places a fair value of “600” on the S&P; 2) Myriad analysts keep insisting that at these levels we are being presented with a buying opportunity that will not present itself again in our lifetime.
Well, I think we got down to these P/E levels in 2003, and we will never see it again in our lifetimes? Is there an asteroid out there?

User avatar
bobcat2
Posts: 5359
Joined: Tue Feb 20, 2007 3:27 pm
Location: just barely Outside the Beltway

tadamsmar

Post by bobcat2 » Wed Dec 03, 2008 11:49 am

Yes, you are right. -50% does not mean -50% or worse. Instead it means worse than -40%, but better than -50%. So if one draws the graph using end of November data, 2008 returns would not be in the left most column, but instead one column to the right. But if you draw the graph with data so far this week then 2008 gets dumped into the left most column. So where will it be by December 31? It could stay in the left column, move one column rightward, or find itself in a brand new -60% column. :lol:

Bob K

edit - It looks like the graph is correct (2008 worse than 1931) using returns from close of business on Monday, but incorrect as of close of business on Tuesday. Who knows how accurate by close of business today :?:
Last edited by bobcat2 on Wed Dec 03, 2008 11:56 am, edited 1 time in total.
In finance risk is defined as uncertainty that is consequential (nontrivial). | The two main methods of dealing with financial risk are the matching of assets to goals & diversifying.

User avatar
fluffyistaken
Posts: 1435
Joined: Fri Apr 04, 2008 1:32 pm

Re: S&P 500 Visualization: Worst. Year. Ever.

Post by fluffyistaken » Wed Dec 03, 2008 11:52 am

tadamsmar wrote:
regmac wrote:
Dan Kohn wrote:I love this visualization of returns of the S&P 500. Who knew it's been around since 1825? And what a great illustration of the Normal Distribution.

The graph supports two points of view that we’ve been hearing for quite sometime: 1) Gary Schilling, who has been right on every call he’s made throughout this horrible year, places a fair value of “600” on the S&P; 2) Myriad analysts keep insisting that at these levels we are being presented with a buying opportunity that will not present itself again in our lifetime.
Well, I think we got down to these P/E levels in 2003, and we will never see it again in our lifetimes? Is there an asteroid out there?
P/E is definitely lower now than in 2003: http://www2.standardandpoors.com/spf/xl ... EPSEST.XLS

User avatar
tadamsmar
Posts: 8537
Joined: Mon May 07, 2007 12:33 pm

Post by tadamsmar » Wed Dec 03, 2008 1:52 pm

The comments section of the link in the OP is interesting.

Anyone yearning for a good economics thread with politics and crude humor mixed in should check it out, the stuff we can't have here.

In particular, the photo of the guy carrying the "Jump you &%$s!" sign past the NYSE.

User avatar
Kenster1
Posts: 3187
Joined: Wed Feb 28, 2007 10:50 pm

Post by Kenster1 » Wed Dec 03, 2008 1:58 pm

http://www.marketwatch.com/news/story/F ... 6DFD1EE%7D

The S&P's dividend yield now higher than the 10-year T-note's
For the first time since 1958, the dividend yield on the Standard & Poor's 500 index has risen above the interest rate on the 10-year Treasury note. As of Tuesday's close, for example, the S&P's dividend yield was 3.3%, while the 10-year T-Note was yielding 2.7% -- a spread of 0.6 percentage points in favor of stocks.

To put this in historical context, the spread since 1958 has averaged 3.7 percentage points in favor of bonds.
Cliff Asness, however, thinks he has come up with some clear answers. He is managing and founding principal at AQR Capital Management, a Greenwich, CT-based quantitative research firm. In the March/April 2000 issue of Financial Analysts Journal, Asness argued that neither the pre-1958 period nor the decades since are anomalous. On the contrary, he found that -- below the surface -- stock and bond yields have always been strongly positively correlated.

The reason that this strong correlation was hidden, according to Asness, is that investors' expectations have changed of the relative volatility of stocks and bonds. Prior to the last 50 years, investors expected the stock market's volatility to be much greater than bond market volatility. To entice investors to incur that greater volatility, the stock market had to provide a higher yield than bonds.

These expected relative volatilities changed early in the post-war period, according to Asness. And that is why -- beginning in 1958 -- the stock market's yield dropped below that of bonds, and stayed there for five decades.

The situation has reversed itself in recent months, according to Asness' theory, given stocks' extraordinary volatility.

To bet on whether stocks' dividend yield recent move above the T-Note yield will persist, therefore, you have to bet on whether investors will expect stocks' volatility, relative to bonds', to remain considerably higher than they have averaged over the last 50 years.

Note that the key element of this bet is investors' expectations. Asness points out that investors' memories live for a very long time. The memory of the Great Depression lingered for years after it ended, for example, which is one reason why stocks' dividend yields remained so high.

In contrast, the Baby Boom generation (at least possibly until now) had no traumatic memory similar to their parents' memory of the Great Depression. That in turn helps to explain why stocks' dividend yield slipped in the 1960s and beyond, relative to bonds' yields.
SURGEON GENERAL'S WARNING: Any overconfidence in your ability, willingness and need to take risk may be hazardous to your health.

User avatar
grayfox
Posts: 5187
Joined: Sat Sep 15, 2007 4:30 am

Let's go for the record

Post by grayfox » Wed Dec 03, 2008 2:09 pm

Come one, people. How often do you get the chance to be part of something big like this? This is a rare opportunity--a once in a lifetime event.

So are we going to let 1931 beat us?

All they had back then was telephones, pencils, paper, ledgers and balance sheets.

We now have computers, high-speed networks, the internet, modern portfolio theory, ETFs, credit default swaps, and mortgage-backed securities.
We now have Financial Weapons of Mass Destruction--something that Wall Street professionals in 1931 couldn't have imagined in their wildest dreams!

I say go for the record. Let's bring it down 60% for 2008 !

Are you with me?
I said, are you with me?

User avatar
jeffyscott
Posts: 8267
Joined: Tue Feb 27, 2007 9:12 am
Location: Wisconsin

Post by jeffyscott » Wed Dec 03, 2008 2:16 pm

Cliff Asness
To bet on whether stocks' dividend yield recent move above the T-Note yield will persist, therefore, you have to bet on whether investors will expect stocks' volatility, relative to bonds', to remain considerably higher than they have averaged over the last 50 years.
I don't understand that comment. Why can't I simply buy stocks and happily collect the (higher than bond) dividends, without it being a "bet" that they will remain above bonds?
Time is your friend; impulse is your enemy. - John C. Bogle

User avatar
bearwolf
Posts: 1718
Joined: Sun May 18, 2008 12:34 am
Location: Oklahoma

Re: Let's go for the record

Post by bearwolf » Wed Dec 03, 2008 2:18 pm

grayfox wrote:Come one, people. How often do you get the chance to be part of something big like this? This is a rare opportunity--a once in a lifetime event.

So are we going to let 1931 beat us?

...
Am I the only one seeing the good part of the graph? When I look at the graph I see the Up years that follow the down years. Let's just make it through the next year or so then start reaping the reward.

BearWolf

User avatar
Christine_NM
Posts: 2699
Joined: Tue Feb 20, 2007 1:13 am
Location: New Mexico

Post by Christine_NM » Wed Dec 03, 2008 2:26 pm

It's the uptick rule, which began after 1937 and ended in July 2007. 1931, 1937, and 2008 all are years with no uptick rule.

Without the uptick rule, volatility will continue and there will be no "back to normal" for the market.

Facing this, in order to maintain an overall portfolio at an unchanged risk tolerance then one must permanently reduce the equity allocation. Or reduce it till the SEC comes to its senses and restores the uptick rule.

I'm not sure buy and hold makes sense without an uptick rule. So far, it doesn't. Something like Taleb's 5-10% stock trading and 90-95% Treasuries is more appropriate for a riskier (more volatile) market.

Rodc
Posts: 13601
Joined: Tue Jun 26, 2007 9:46 am

Re: Let's go for the record

Post by Rodc » Wed Dec 03, 2008 2:29 pm

grayfox wrote:Come one, people. How often do you get the chance to be part of something big like this? This is a rare opportunity--a once in a lifetime event.

So are we going to let 1931 beat us?

All they had back then was telephones, pencils, paper, ledgers and balance sheets.

We now have computers, high-speed networks, the internet, modern portfolio theory, ETFs, credit default swaps, and mortgage-backed securities.
We now have Financial Weapons of Mass Destruction--something that Wall Street professionals in 1931 couldn't have imagined in their wildest dreams!

I say go for the record. Let's bring it down 60% for 2008 !

Are you with me?
I said, are you with me?
I'm with you!

If I'm going to go through this much pain, I want a record dang it!

Then we can have a record recovery.
We live a world with knowledge of the future markets has less than one significant figure. And people will still and always demand answers to three significant digits.

User avatar
jeffyscott
Posts: 8267
Joined: Tue Feb 27, 2007 9:12 am
Location: Wisconsin

Post by jeffyscott » Wed Dec 03, 2008 2:55 pm

I hope those of you begging for -60% for 2008 realize that this would mean 587 which would be -31% from yesterdays close.
Time is your friend; impulse is your enemy. - John C. Bogle

TheEternalVortex
Posts: 2556
Joined: Tue Feb 27, 2007 9:17 pm
Location: San Jose, CA

Post by TheEternalVortex » Wed Dec 03, 2008 2:59 pm

Christine_NM wrote:It's the uptick rule, which began after 1937 and ended in July 2007. 1931, 1937, and 2008 all are years with no uptick rule.

Without the uptick rule, volatility will continue and there will be no "back to normal" for the market.

Facing this, in order to maintain an overall portfolio at an unchanged risk tolerance then one must permanently reduce the equity allocation. Or reduce it till the SEC comes to its senses and restores the uptick rule.

I'm not sure buy and hold makes sense without an uptick rule. So far, it doesn't. Something like Taleb's 5-10% stock trading and 90-95% Treasuries is more appropriate for a riskier (more volatile) market.
The existence of volatile years while the rule was in place contradicts your hypothesis.

User avatar
Kenster1
Posts: 3187
Joined: Wed Feb 28, 2007 10:50 pm

Post by Kenster1 » Wed Dec 03, 2008 3:21 pm

jeffyscott wrote:I hope those of you begging for -60% for 2008 realize that this would mean 587 which would be -31% from yesterdays close.
And it would be difficult to take advantage of purchasing equities at those lower prices if you don't have a job because of the business downturn! :shock:
SURGEON GENERAL'S WARNING: Any overconfidence in your ability, willingness and need to take risk may be hazardous to your health.

Rodc
Posts: 13601
Joined: Tue Jun 26, 2007 9:46 am

Post by Rodc » Wed Dec 03, 2008 3:24 pm

jeffyscott wrote:I hope those of you begging for -60% for 2008 realize that this would mean 587 which would be -31% from yesterdays close.
I'm willing to "win" by 0.01%
We live a world with knowledge of the future markets has less than one significant figure. And people will still and always demand answers to three significant digits.

User avatar
Christine_NM
Posts: 2699
Joined: Tue Feb 20, 2007 1:13 am
Location: New Mexico

Post by Christine_NM » Wed Dec 03, 2008 3:46 pm

TheEternalVortex wrote:
Christine_NM wrote:It's the uptick rule, which began after 1937 and ended in July 2007. 1931, 1937, and 2008 all are years with no uptick rule.

Without the uptick rule, volatility will continue and there will be no "back to normal" for the market.

Facing this, in order to maintain an overall portfolio at an unchanged risk tolerance then one must permanently reduce the equity allocation. Or reduce it till the SEC comes to its senses and restores the uptick rule.

I'm not sure buy and hold makes sense without an uptick rule. So far, it doesn't. Something like Taleb's 5-10% stock trading and 90-95% Treasuries is more appropriate for a riskier (more volatile) market.
The existence of volatile years while the rule was in place contradicts your hypothesis.
No year with an uptick rule has been as volatile in a bad way as the three outliers, 31, 37 and 08. The uptick rule will not cure all ills but will reduce volatility, which is the common measure of risk.

It's like walking a high wire with a net vs. without a net. You might do the former, but not so much the latter. Ignoring the fact that the net has been removed while we are in midwalk (invested in stocks) seems unwise to me.

Pacific
Posts: 1321
Joined: Tue Mar 06, 2007 8:19 pm
Location: Lost in the middle of the Pacific

Post by Pacific » Wed Dec 03, 2008 4:18 pm

tadamsmar said:
The comments section of the link in the OP is interesting.

Anyone yearning for a good economics thread with politics and crude humor mixed in should check it out, the stuff we can't have here.

In particular, the photo of the guy carrying the "Jump you &%$s!" sign past the NYSE.
I especially liked this one:
I kind of like the idea of throwing money at the problem till we get rid of it all. Money is the root of all evil. What we are doing is making sure future generations won't have to worry about it.

User avatar
tadamsmar
Posts: 8537
Joined: Mon May 07, 2007 12:33 pm

Re: Let's go for the record

Post by tadamsmar » Wed Dec 03, 2008 4:31 pm

grayfox wrote:Come one, people. How often do you get the chance to be part of something big like this? This is a rare opportunity--a once in a lifetime event.

So are we going to let 1931 beat us?

All they had back then was telephones, pencils, paper, ledgers and balance sheets.

We now have computers, high-speed networks, the internet, modern portfolio theory, ETFs, credit default swaps, and mortgage-backed securities.
We now have Financial Weapons of Mass Destruction--something that Wall Street professionals in 1931 couldn't have imagined in their wildest dreams!

I say go for the record. Let's bring it down 60% for 2008 !

Are you with me?
I said, are you with me?
They had airplanes and radios.

User avatar
madsinger
Posts: 951
Joined: Sat Sep 29, 2007 3:26 pm

Post by madsinger » Wed Dec 03, 2008 4:34 pm

What I don't like about the original graph in this post is the "arithmetic" as opposed to the "geometric" nature of it. There seems to be an implication that a "+10 to +20" year balances out a "-10 to -20" year. Or that the "+50 to +60" years even "surpass" the losses of the "-40 to -30" years. If the "x-axis" was spaced in a "log" fashion, then the "-50" block would have to be as far to the left as a "+100" block. A "-40" block would be "balanced" by a +67 block.

This would make those blocks on the left side, further to the left, and the "normalness" would be "less normal".

(That multiyear 1930-1932 "-90 to -80" block would need a "+500 to +1000" block on the right to counteract it!).

-Brad.

peter71
Posts: 3768
Joined: Tue Jul 24, 2007 8:28 pm

Post by peter71 » Wed Dec 03, 2008 4:51 pm

madsinger wrote:What I don't like about the original graph in this post is the "arithmetic" as opposed to the "geometric" nature of it. There seems to be an implication that a "+10 to +20" year balances out a "-10 to -20" year. Or that the "+50 to +60" years even "surpass" the losses of the "-40 to -30" years. If the "x-axis" was spaced in a "log" fashion, then the "-50" block would have to be as far to the left as a "+100" block. A "-40" block would be "balanced" by a +67 block.

This would make those blocks on the left side, further to the left, and the "normalness" would be "less normal".

(That multiyear 1930-1932 "-90 to -80" block would need a "+500 to +1000" block on the right to counteract it!).

-Brad.
Well, the graph is currently skewed to the right so arguably it'd actually make it MORE normal to somehow overweight losses, but there's really no perfect way to do this sort of thing as even the height and width of the boxes affects the appearance. All of that said, my sense is that a lot of the regression techniques used in the econ papers we read assume normality without even bothering to check that assumption as did Eternal Vortex above (and in this case the assumption checks out OK).

All best,
Pete

Bharat
Posts: 343
Joined: Mon Feb 18, 2008 4:46 pm
Location: Somewhere between Appalachia and Atlantic

Re: Let's go for the record

Post by Bharat » Wed Dec 03, 2008 5:00 pm

Rodc wrote:
grayfox wrote:Come one, people. How often do you get the chance to be part of something big like this? This is a rare opportunity--a once in a lifetime event.

So are we going to let 1931 beat us?

All they had back then was telephones, pencils, paper, ledgers and balance sheets.

We now have computers, high-speed networks, the internet, modern portfolio theory, ETFs, credit default swaps, and mortgage-backed securities.
We now have Financial Weapons of Mass Destruction--something that Wall Street professionals in 1931 couldn't have imagined in their wildest dreams!

I say go for the record. Let's bring it down 60% for 2008 !

Are you with me?
I said, are you with me?
I'm with you!

If I'm going to go through this much pain, I want a record dang it!

Then we can have a record recovery.
I am with both of you. I hope that S&P stay under 800 for next couple of years, so that i can have a decent base of investment at good prices. I am willing to live with pain for next few years.
Everything that you own, owns piece of you.

User avatar
grayfox
Posts: 5187
Joined: Sat Sep 15, 2007 4:30 am

Post by grayfox » Thu Dec 04, 2008 1:42 am

Business idea for the recession: Make T-shirts that say:

I Lived Through The Crash of 2008
And All I Have Left is this Lousy T-Shirt

yobria
Posts: 5978
Joined: Mon Feb 19, 2007 11:58 pm
Location: SF CA USA

Post by yobria » Thu Dec 04, 2008 2:02 am

grayfox wrote:Business idea for the recession: Make T-shirts that say:

I Lived Through The Crash of 2008
And All I Have Left is this Lousy T-Shirt
Not sure that'd be appropriate - we all lost our shirts in 2008!

Nick

User avatar
Murray Boyd
Posts: 794
Joined: Mon Feb 19, 2007 6:00 pm

Post by Murray Boyd » Thu Dec 04, 2008 3:00 am

How can anyone know what the P/E is right now, what with the E crashing and all?

User avatar
sgr000
Posts: 365
Joined: Wed Feb 28, 2007 12:21 pm
Location: Boston
Contact:

Post by sgr000 » Fri Dec 05, 2008 10:44 am

mark500 wrote:
Henry Sadovsky wrote:That is almost certainly not a normal distribution.
Help me with statistics here. It looks similar to a bell curve.
[NB: Before you take any of the following analysis seriously, be sure to read and understand the last paragraph.]

As to whether the original data is a histogram derived from a normal distribution, I can offer no opinion. There simply aren't enough data points to tell, now that it's been reduced to a histogram. Almost any test of normality (and I note with approval that somebody tried Anderson-Darling :)) won't have enough to achieve significance if it were not normal.

However, we can take a look at Shiller's dataset of S&P composite returns from 1871-present. Now, he's done a lot of fiddling around to get a continuous series of index price levels, dividend values, and inflation measures that far back. His web site will tell you all about it; for purposes of this post, I'm going to take him at face value and assume that a highly competent economist such as himself has done a credible job.

So I took his yearly data series and simplified it down until it contained just what we need: Year, Price, Dividend, and CPI. I loaded those into R and repeated his spreadsheet calculations to derive the total return, as well as real (inflation-adjusted) index price level, dividend payment, and thus a real return. At the end of that, I was armed with a time series of year, nominal return, and real return from 1871 - 2006. (I had to fiddle with the 2007 data to supply enough information to calculate returns through 2006; obviously I can't get the 2008 numbers yet.)

R script and input data available from me upon PM request.

Fine. Now, are the returns normally distributed or not? To answer that, I ran 8 different tests of normality on the nominal and real returns. I won't bother you about the details of the differences between the tests, and their different sensitivities, because the verdict is unanimous: the returns are normally distributed.

In the tables that follow, the thing to keep your eye on is the p-value of the test, in the rightmost column. That's the probability of seeing a dataset like yours, with the deviations from normality that eventually creep in, even if the source of the data is really a normal distribution (the "null hypothesis" of the tests). That is, you should only believe the data is not normal if p<0.05 or so. Otherwise you should believe it's normal.

Here's the result for nominal returns, showing the test results in a table and a quantile-quantile plot. In the QQ plot, normal data will lie along the blue diagonal line, so this is a visual confirmation of what the table says. As you can see, it's pretty convincing.

Code: Select all

Test Name                       Statistic  Value  P
=========                       =========  =====  =
Shapiro-Wilk normality          W          0.9970 0.991
Robust Jarque Bera              X-squared  0.0293 0.985
Standardized SJ                 SJ         0.0961 0.463
Anderson-Darling                A          0.1240 0.987
Cramer-von Mises                W          0.0185 0.979
Lilliefors (Kolmogorov-Smirnov) D          0.0379 0.903
Pearson chi-square              P         18.4000 0.104
Shapiro-Francia                 W          0.9960 0.955

Mean   = 10.5%
StdDev = 17.7%
N      = 136
Image

Here's the result for the real returns:

Code: Select all

Test Name                       Statistic  Value  P
=========                       =========  =====  =
Shapiro-Wilk normality          W          0.9950 0.893
Robust Jarque Bera              X-squared  0.0312 0.985
Standardized SJ                 SJ         0.1370 0.447
Anderson-Darling                A          0.1540 0.956
Cramer-von Mises                W          0.0192 0.974
Lilliefors (Kolmogorov-Smirnov) D          0.0385 0.892
Pearson chi-square              P          7.8200 0.799
Shapiro-Francia                 W          0.9960 0.930

Mean   = 8.25%
StdDev = 17.5%
N      = 136
Image

Of course, the real question everybody wants to ask is, "What does that tell us about how crappy this year is?" Nominal returns have a mean of 10.5% and a standard deviation of 17.7%. This year's VFINX (Vanguard 500 Index) return through November was -37.7%. That's in the tail with probability less than 0.323%; figure about once every 300 years.

A cautionary note: This analysis is meaning-free; it's what happens when a statistics nerd gets bored. Impressive as Shiller's data is, it simply does not adequately sample the world to predict a black swan. In fact, that's essentially what Taleb and others are saying: due to the fat-tailed nature of the return distributions, no amount of historical samples can adequately fit a model which includes black swans. So my confident statement that this is a "once every 300 years" market is hogwash (especially when derived from a model with only 136 years of history! :().

User avatar
grayfox
Posts: 5187
Joined: Sat Sep 15, 2007 4:30 am

Post by grayfox » Fri Dec 05, 2008 10:55 am

Outstanding work! This forum never ceases to amaze me.

I have a request. Would it be possible to post a table of nominal return and real return for each year from 1871 to present?

User avatar
Henry Sadovsky
Posts: 748
Joined: Thu May 08, 2008 1:37 am

Assessing for normality

Post by Henry Sadovsky » Fri Dec 05, 2008 11:06 am

.
Hi sgr000.

That was very interesting. Does not the use of yearly data unnecessarily limit the power of the analysis? My understanding of the fat-tail question is that it is not a conjecture, but that fat-tails are seen in the analysis of daily data. I don't have the data at hand, but I seem to recall something to the effect that if (U.S. equity) market returns were normally distributed, the probability of seeing days like 10/19/1987, and all the +/- >3% days we are seeing this year would be vanishingly small.
"What we can't say we can't say, and we can't whistle it either." | Frank P. Ramsey" | | (f.k.a. Zalzel)

User avatar
Topic Author
Dan Kohn
Posts: 1505
Joined: Tue Jun 26, 2007 12:15 am
Location: New York, NY
Contact:

Outstanding

Post by Dan Kohn » Fri Dec 05, 2008 11:08 am

sgr000, thanks for the outstanding work.

User avatar
sgr000
Posts: 365
Joined: Wed Feb 28, 2007 12:21 pm
Location: Boston
Contact:

Post by sgr000 » Fri Dec 05, 2008 11:23 am

grayfox wrote:Outstanding work! This forum never ceases to amaze me.

I have a request. Would it be possible to post a table of nominal return and real return for each year from 1871 to present?
The best way would be to follow the link above to Shiller's yearly data spreadsheet. He calculates real returns only, but you can look at it in Excel to imitate a calculation for nominal returns if you want.

Presumably Shiller is updating his data. I won't be updating this. So what you have below is just a one-shot deal; you're better off going with Shiller.
  • P is the index price at the beginning of the year,
  • D is the dividend paid throughout the year,
  • CPI is the CPI level,
  • RealP is the inflation-adjusted index price (using the current year's CPI),
  • RealD is the inflation-adjusted dividend (using next year's CPI, essentially assuming with Shiller that dividends are paid at the end of the year),
  • RealReturn is the real return: (RealPnextyear - RealP + RealD) / RealP,
  • Return is the nominal return: (Pnextyear - P + D) / P

Code: Select all

Year       P     D     CPI   RealP  RealD RealReturn    Return
1871    4.44  0.26  12.460   72.11  4.159  0.1358000  0.153200
1872    4.86  0.30  12.650   77.74  4.693  0.0886200  0.113200
1873    5.11  0.33  12.940   79.94  5.400  0.0215900 -0.023480
1874    4.66  0.33  12.370   76.26  5.802  0.1228000  0.045060
1875    4.54  0.30  11.510   79.82  5.599  0.1128000  0.048460
1876    4.46  0.30  10.850   83.23  5.550 -0.1443000 -0.136800
1877    3.55  0.19  10.940   65.67  4.167  0.1488000 -0.030990
1878    3.25  0.18   9.229   71.28  4.402  0.2899000  0.156900
1879    3.58  0.20   8.278   87.54  4.052  0.2290000  0.483200
1880    5.11  0.26   9.990  103.50  5.587  0.3387000  0.262200
1881    6.19  0.32   9.419  133.00  6.362 -0.0672900  0.008078
1882    5.92  0.32  10.180  117.70  6.484  0.0551900  0.035470
1883    5.81  0.33   9.990  117.70  7.238  0.0265900 -0.051640
1884    5.18  0.31   9.229  113.60  7.581 -0.0206600 -0.121600
1885    4.24  0.24   8.278  103.70  6.078  0.3288000  0.283000
1886    5.20  0.22   7.992  131.70  5.572  0.1154000  0.115400
1887    5.58  0.25   7.992  141.30  6.044 -0.0488800 -0.003584
1888    5.31  0.23   8.373  128.40  5.825  0.0791900  0.030130
1889    5.24  0.22   7.992  132.70  5.850  0.1221000  0.068700
1890    5.38  0.22   7.612  143.10  5.708 -0.0824200 -0.059480
1891    4.84  0.22   7.802  125.60  6.078  0.2608000  0.183900
1892    5.51  0.24   7.326  152.20  6.152 -0.0150400  0.061710
1893    5.61  0.25   7.897  143.80  7.387 -0.0609300 -0.185400
1894    4.32  0.21   6.850  127.60  6.475  0.0772900  0.032410
1895    4.25  0.19   6.565  131.00  5.774  0.0344200  0.049410
1896    4.27  0.18   6.660  129.80  5.631  0.0607500  0.030440
1897    4.22  0.18   6.470  132.00  5.471  0.1648000  0.199100
1898    4.88  0.20   6.660  148.30  5.993  0.2688000  0.286900
1899    6.08  0.21   6.755  182.20  5.383 -0.1122000  0.037830
1900    6.10  0.30   7.897  156.40  7.879  0.2380000  0.208200
1901    7.07  0.32   7.707  185.70  8.202  0.1650000  0.193800
1902    8.12  0.33   7.897  208.10  7.715 -0.0126500  0.082510
1903    8.46  0.35   8.658  197.80  8.559 -0.1308000 -0.169000
1904    6.68  0.31   8.278  163.30  7.410  0.2790000  0.308400
1905    8.43  0.33   8.468  201.50  7.888  0.2100000  0.210000
1906    9.87  0.40   8.468  235.90  9.150 -0.0342800  0.009119
1907    9.56  0.44   8.849  218.70 10.290 -0.2207000 -0.237400
1908    6.85  0.40   8.658  160.10  9.053  0.3370000  0.381000
1909    9.06  0.44   8.944  205.00  9.001  0.0495000  0.161100
1910   10.08  0.47   9.895  206.20 10.310  0.0360100 -0.033730
1911    9.27  0.47   9.229  203.30 10.420  0.0452900  0.034520
1912    9.12  0.48   9.134  202.10  9.914 -0.0005083  0.072370
1913    9.30  0.48   9.800  192.10  9.716 -0.0674200 -0.048390
1914    8.37  0.42  10.000  169.40  8.417 -0.0655000 -0.056150
1915    7.48  0.43  10.100  149.90  8.369  0.2672000  0.304800
1916    9.33  0.56  10.400  181.60  9.688 -0.0348900  0.085740
1917    9.57  0.69  11.700  165.60  9.976 -0.3101000 -0.174500
1918    7.21  0.57  14.000  104.20  6.993 -0.0091200  0.167800
1919    7.85  0.53  16.500   96.30  5.559  0.0193700  0.192400
1920    8.83  0.51  19.300   92.61  5.433 -0.1234000 -0.137000
1921    7.11  0.46  19.000   75.75  5.510  0.2270000  0.091420
1922    7.30  0.51  16.900   87.43  6.145  0.2967000  0.289000
1923    8.90  0.53  16.800  107.20  6.201  0.0212900  0.051690
1924    8.83  0.55  17.300  103.30  6.435  0.2605000  0.260500
1925   10.58  0.60  17.300  123.80  6.785  0.2104000  0.252400
1926   12.65  0.69  17.900  143.00  7.981  0.1393000  0.113800
1927   13.40  0.77  17.500  155.00  9.009  0.3815000  0.365700
1928   17.53  0.85  17.300  205.10 10.060  0.4838000  0.466600
1929   24.86  0.97  17.100  294.30 11.480 -0.0876900 -0.087690
1930   21.71  0.98  17.100  257.00 12.480 -0.1598000 -0.218800
1931   15.98  0.82  15.900  203.40 11.610 -0.3654000 -0.429300
1932    8.30  0.50  14.300  117.50  7.846  0.0137000 -0.085540
1933    7.09  0.44  12.900  111.30  6.747  0.5135000  0.548700
1934   10.54  0.45  13.200  161.60  6.698 -0.1058000 -0.078750
1935    9.26  0.47  13.600  137.80  6.894  0.5144000  0.536700
1936   13.76  0.72  13.800  201.80 10.340  0.3024000  0.330700
1937   17.59  0.80  14.100  252.50 11.400 -0.3164000 -0.311500
1938   11.31  0.51  14.200  161.20  7.374  0.1667000  0.150300
1939   12.50  0.62  14.000  180.70  9.029  0.0410400  0.033600
1940   12.30  0.67  13.900  179.10  9.618 -0.1007000 -0.087800
1941   10.55  0.71  14.100  151.50  9.154 -0.1794000 -0.086260
1942    8.93  0.59  15.700  115.10  7.067  0.1110000  0.196000
1943   10.09  0.61  16.900  120.90  7.096  0.1994000  0.234900
1944   11.85  0.64  17.400  137.90  7.278  0.1656000  0.192400
1945   13.49  0.66  17.800  153.40  7.340  0.3543000  0.384700
1946   18.02  0.71  18.200  200.40  6.684 -0.2521000 -0.116500
1947   15.21  0.84  21.500  143.20  7.174 -0.0653900  0.030240
1948   14.83  0.93  23.700  126.70  7.844  0.0847200  0.098450
1949   15.36  1.14  24.000  129.50  9.819  0.1981000  0.173200
1950   16.88  1.47  23.500  145.40 11.710  0.2431000  0.343600
1951   21.21  1.41  25.400  169.00 10.770  0.1569000  0.207000
1952   24.19  1.41  26.500  184.80 10.730  0.1363000  0.140600
1953   26.18  1.45  26.600  199.20 10.910  0.0164200  0.027880
1954   25.46  1.54  26.900  191.60 11.670  0.4697000  0.458800
1955   35.60  1.64  26.700  269.90 12.390  0.2814000  0.286200
1956   44.15  1.74  26.800  333.50 12.760  0.0374300  0.068400
1957   45.43  1.79  27.600  333.20 12.670 -0.0885000 -0.055470
1958   41.12  1.75  28.600  291.00 12.210  0.3759000  0.395200
1959   55.62  1.83  29.000  388.20 12.640  0.0652100  0.076230
1960   58.03  1.95  29.300  400.90 13.250  0.0449000  0.062730
1961   59.72  2.02  29.800  405.60 13.630  0.1825000  0.190400
1962   69.07  2.13  30.000  466.00 14.180 -0.0400200 -0.027220
1963   65.06  2.28  30.400  433.20 14.940  0.1905000  0.210100
1964   76.45  2.50  30.900  500.80 16.220  0.1480000  0.159200
1965   86.12  2.72  31.200  558.70 17.310  0.0941500  0.115200
1966   93.32  2.87  31.800  594.00 17.660 -0.0955800 -0.064290
1967   84.45  2.92  32.900  519.60 17.330  0.1192000  0.160000
1968   95.04  3.07  34.100  564.20 17.460  0.0593600  0.106000
1969  102.00  3.16  35.600  580.20 16.920 -0.1373000 -0.083990
1970   90.31  3.14  37.800  483.60 15.970  0.0162100  0.069980
1971   93.49  3.07  39.800  475.50 15.120  0.1018000  0.137800
1972  103.30  3.15  41.100  508.70 14.970  0.1354000  0.176900
1973  118.40  3.38  42.600  562.70 14.680 -0.2320000 -0.159900
1974   96.11  3.60  46.600  417.50 13.990 -0.2912000 -0.207600
1975   72.56  3.68  52.100  281.90 13.400  0.2984000  0.385600
1976   96.86  4.05  55.600  352.60 14.010  0.0583600  0.113600
1977  103.80  4.67  58.500  359.20 15.120 -0.1442000 -0.085640
1978   90.25  5.07  62.500  292.30 15.030  0.0624100  0.161000
1979   99.71  5.65  68.300  295.50 14.700  0.0258900  0.168600
1980  110.90  6.16  77.800  288.50 14.330  0.1222000  0.254900
1981  133.00  6.63  87.000  309.40 14.230 -0.1403000 -0.068140
1982  117.30  6.87  94.300  251.70 14.220  0.2426000  0.288700
1983  144.30  7.09  97.800  298.60 14.080  0.1541000  0.202500
1984  166.40  7.53 101.900  330.50 14.450  0.0398900  0.076630
1985  171.60  7.90 105.500  329.30 14.590  0.2121000  0.259200
1986  208.20  8.28 109.600  384.50 15.070  0.2914000  0.310300
1987  264.50  8.81 111.200  481.50 15.410 -0.0578600 -0.019730
1988  250.50  9.73 115.700  438.20 16.260  0.1258000  0.178300
1989  285.40 11.05 121.100  477.10 17.560  0.1691000  0.229900
1990  340.00 12.10 127.400  540.20 18.200 -0.0600900 -0.006971
1991  325.50 12.20 134.600  489.50 17.880  0.2824000  0.315800
1992  416.10 12.38 138.100  609.90 17.570  0.0418300  0.075780
1993  435.20 12.58 142.600  617.80 17.420  0.0881900  0.115700
1994  473.00 13.18 146.200  654.90 17.750 -0.0160900  0.011500
1995  465.20 13.79 150.300  626.60 18.080  0.3144000  0.350300
1996  614.40 14.90 154.400  805.50 18.960  0.2338000  0.271300
1997  766.20 15.50 159.100  974.80 19.410  0.2578000  0.277500
1998  963.40 16.20 161.600 1207.00 19.960  0.2915000  0.313100
1999 1249.00 16.69 164.300 1538.00 20.010  0.1242000  0.155000
2000 1426.00 16.27 168.800 1709.00 18.810 -0.0889900 -0.054990
2001 1331.00 15.74 175.100 1539.00 17.990 -0.1413000 -0.131500
2002 1140.00 16.07 177.100 1303.00 17.900 -0.2205000 -0.200200
2003  895.80 17.39 181.700  998.00 19.010  0.2025000  0.225700
2004 1081.00 19.44 185.200 1181.00 20.630  0.0952200  0.127700
2005 1199.00 22.22 190.700 1273.00 22.680  0.0299100  0.070960
2006 1262.00 24.88 198.300 1288.00 24.880  0.1188000  0.142000
2007 1416.42 27.73 202.416 NA      NA      NA         NA

Post Reply