Vanguard risk tolerance survey blunder.

Discuss all general (i.e. non-personal) investing questions and issues, investing news, and theory.
Post Reply
*3!4!/5!
Posts: 1256
Joined: Fri Sep 02, 2016 1:47 pm

Vanguard risk tolerance survey blunder.

Post by *3!4!/5! » Fri Jun 16, 2017 1:42 am

This question has been in Vanguard risk tolerance survey for several years.
https://personal.vanguard.com/us/FundsInvQuestionnaire
Vanguard wrote:Q: The chart below shows the greatest 1-year loss and the highest 1-year gain on 3 different hypothetical investments of $10,000. Given the potential gain or loss in any 1 year, I would invest my money in:

[_]A (loss of $164, gain of $593)

[_]B (loss of $1,020, gain of $1,921)

[_]C (loss of $3,639, gain of $4,229)
(There is a bar graph figure, but I don't know how to display it.)

Obviously this question is designed to probe the investor's feelings about risk and return, and the options are listed in increasing order of risk. But what about expected return? Well, there's not enough information, but a natural simple presumption is that the two listed return amounts are equally likely (flip a coin).

So you would get expected return

A ($593-$164)/2=$214.5
B ($1,921-$1,020)/2=$450.5
C ($4,229-$3,639)/2=$295

In other words option B has lower risk and higher expected return than option C!
(If you use geometric mean, option C is actually a long term loser.)

It's a ridiculous question, and it's astonishing that they've left in unchanged for years. What were they thinking? How is an astute investor supposed to answer a question like this?

jjface
Posts: 2462
Joined: Thu Mar 19, 2015 6:18 pm

Re: Vanguard risk tolerance survey blunder.

Post by jjface » Fri Jun 16, 2017 3:55 am

Its testing your maximum risk tolerance. Obviously C means you can either gain the most or lose the most.

Your calculation also assume all outcomes are equally likely but they may well not be.

*3!4!/5!
Posts: 1256
Joined: Fri Sep 02, 2016 1:47 pm

Re: Vanguard risk tolerance survey blunder.

Post by *3!4!/5! » Fri Jun 16, 2017 4:08 am

jjface wrote:Its testing your maximum risk tolerance. Obviously C means you can either gain the most or lose the most.

Your calculation also assume all outcomes are equally likely but they may well not be.
I already said that. But the question is obviously deeply flawed.

ResearchMed
Posts: 5474
Joined: Fri Dec 26, 2008 11:25 pm

Re: Vanguard risk tolerance survey blunder.

Post by ResearchMed » Fri Jun 16, 2017 4:13 am

jjface wrote:Its testing your maximum risk tolerance. Obviously C means you can either gain the most or lose the most.

Your calculation also assume all outcomes are equally likely but they may well not be.
I think *3!4!/5!'s calculations would also hold if there were a "bell curve" type of distribution, but clustered around a middle outcome.
But that's not important for this purpose.

More important is that it's probably the case that for most novice investors, they wouldn't know, understand, or be thinking much about how the outcomes might be "distributed".

I'd suspect that novice investors would be affected by more gross measures of possible outcomes, such as "max loss or gain", even though they probably wouldn't understand how unlikely either actually is (and this is not unimportant).

But as jjface mentions, these extreme figures are likely to get at some "gut level" reaction, and that's also likely to be what causes someone to jump ship at the worst time, or at least "a bad time".

Also, the juxtaposition of low/med/high "min/max" outcomes might also help novice investors grasp at least slightly that there's "no free lunch": a strategy's "potential profit magnitude" is somehow "linked" to a "potential loss magnitude".

I have no idea (obviously!) how much research has been put into how well this type of question/modeling represents how novice investors actually view such things.

RM
This signature is a placebo. You are in the control group.

rkhusky
Posts: 4488
Joined: Thu Aug 18, 2011 8:09 pm

Re: Vanguard risk tolerance survey blunder.

Post by rkhusky » Fri Jun 16, 2017 7:01 am

This looks to me like the potential numbers for approximately 20/80, 50/50 and 80/20 portfolios. Perhaps Vanguard used real portfolios to come up with the numbers, rather than some model distribution.

fourwheelcycle
Posts: 240
Joined: Sun May 25, 2014 5:55 pm

Re: Vanguard risk tolerance survey blunder.

Post by fourwheelcycle » Fri Jun 16, 2017 7:16 am

I have taken the survey several times over the years, and I have always answered "B"; not because I do any math analysis, but because on a gut level B looks like the best mix of potential gain vs. potential loss.

I have always thought the question meant "Are you conservative, and you would like to stick with a limited upside in return for a limited downside, are you middle of the road, willing to take a little more risk to gain a moderate upside in return for a moderate downside, or are you a high risk taker, willing to risk a large downside for a large upside.

I see myself as a middle of the road risk taker. Plus, it seemed "B" offered the best eyeball ratio of potential returns to potential losses. The OP's even-probability-weighted math shows my eyeball assessment was correct.

User avatar
nisiprius
Advisory Board
Posts: 34319
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Vanguard risk tolerance survey blunder.

Post by nisiprius » Fri Jun 16, 2017 7:26 am

I haven't seen a risk tolerance survey that wasn't deeply flawed. The best that can be said about them is that if you've never invested in securities before and are used to bank accounts, it at least rubs your nose into the idea that there could be losses and the ballpark sizes your losses could be.

I've slowly come to the cynical conclusion that none of these surveys is seriously intended to measure risk tolerance for your own benefit, but are just there to keep advisors out of trouble by putting you on the record as accepting a moderate degree of risk so you can't say they recommended unsuitable investments.

One of the things I think is dishonest, is that I think anybody who lived through 2008-2009--regardless of what they did--experienced it as about a 50% loss in stocks (as in the joke of the day, "My 401(k) is now a 201(k).") A 50% loss is psychologically important. I've noticed that these questionnaires never bring up the possibility of a 50% loss. Here in the forum, and IIRC, in The Bogleheads' Guide to Investing--before 2008-2009--50% was, in fact, the hypothetical number that was suggested for assessing risk tolerance. Of course, 50% is by no means the worst that's occurred, but there have been four such declines in the last ninety years so it's well within the range of things that "really happens," and that an investor should consider as a serious possibility.

But these "risk tolerance" questionnaires seem to me to avoid it. They never say "how would you feel about losing half your money," it's always some crazy-precise, randomish, bloodless number like--in Vanguard's case--"over 31%."

How would I feel about losing "over 31%?" Well, gee. I'd be cool with 30%, but 32%, no way? I mean, seriously. Who has any particular psychological feeling for "31%?"

If you look at the version of the questionnaire in their "Get a Recommendation" tool, all they actually do with this is put you in one of four categories, roughly corresponding to those of the four LifeStrategy funds. They could just ask you "are you conservative, moderately conservative, moderately aggressive, or aggressive?" I do think that what they do is considerably better than just asking you to pick one of four phrases, but it comes down to the same thing.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

User avatar
nisiprius
Advisory Board
Posts: 34319
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Vanguard risk tolerance survey blunder.

Post by nisiprius » Fri Jun 16, 2017 7:37 am

fourwheelcycle wrote:...I have always answered "B"; not because I do any math analysis, but because on a gut level B looks like the best mix of potential gain vs. potential loss...
This brings up a much more serious flaw in these things, which is the effect of framing or and/or what the behavioral economists call "anchoring" from the set of choices provided.

Faced with those three choices, most of us--at some level, conscious or not--are going to perceive B as "what most people do" or "normal" or "sensible," and will think long and hard before choosing an "unusual" or "extreme" choice.

I encountered a very clear example of this in a "retirement workbook" exercise I went through in my employer's 401(k) provider's half-day-free-pizza retirement education session, back in 1997. The worksheet was to calculate needed savings rates. They asked us to estimate our own expected, annualized, real return for our investments, and read a result form a table based on that rate of return. The table contained three columns. The three columns were headed 5%, 7%, and 9%. And the 7% column, the whole column, was printed in boldface!

I mean, seriously. One of these questionnaires could include a question, "Which of these would you like to do on your next vacation?

a) "Skydive."
b) "Walk a tightrope across Niagara Falls."
c) "Try to jump Snake River Canyon in a replica of Evel Knievel's SkyCycle K-2."

I'll bet most people would pick "b."

P.S. The behavioral economists have shown that people actually connect any number they've seen recently with the choices they make, even if it obviously has no logical connection at all. That is, they asked people spin a Wheel of Fortune with numbers on it from 1 to 100, and then a few minutes later asked people to estimate or guess how many nations in Africa were part of the United Nations, and if they spun a higher number they guessed that more nations were members.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

pkcrafter
Posts: 12161
Joined: Sun Mar 04, 2007 12:19 pm
Location: CA
Contact:

Re: Vanguard risk tolerance survey blunder.

Post by pkcrafter » Fri Jun 16, 2017 8:58 am

Expected return can't be derived from these numbers. They aren't about return, but more about volatility, so you have to add them, not subtract.

From the article:
The maximum gain or loss on an investment is impossible to predict. The ranges shown in the chart are hypothetical and are designed solely to gauge an investor's risk tolerance.
These questionnaires are notoriously poor because they don't deal with real emotions.

Paul
When times are good, investors tend to forget about risk and focus on opportunity. When times are bad, investors tend to forget about opportunity and focus on risk.

User avatar
whaleknives
Posts: 1210
Joined: Sun Jun 24, 2012 7:19 pm

Re: Vanguard risk tolerance survey blunder.

Post by whaleknives » Fri Jun 16, 2017 10:53 am

nisiprius wrote:. . . I mean, seriously. One of these questionnaires could include a question, "Which of these would you like to do on your next vacation?
a) "Skydive."
b) "Walk a tightrope across Niagara Falls."
c) "Try to jump Snake River Canyon in a replica of Evel Knievel's SkyCycle K-2."
I'll bet most people would pick "b."
Your example is flawed. I already know I can skydive much better than walk a tightrope or fly a motorcycle. :D
"I'm an indexer. I own the market. And I'm happy." (John Bogle, "BusinessWeek", 8/17/07) ☕ Maritime signal flag W - Whiskey: "I require medical assistance."

User avatar
bottlecap
Posts: 5048
Joined: Tue Mar 06, 2007 11:21 pm
Location: Tennessee

Re: Vanguard risk tolerance survey blunder.

Post by bottlecap » Fri Jun 16, 2017 12:28 pm

I disagree with your assumption that the chances are equally likely. The question asks about risk v. reward, I don't know how you can make that assumption. Nothing in the question asked about what kind of average return you want.

You also have too much time on your hands!

JT

*3!4!/5!
Posts: 1256
Joined: Fri Sep 02, 2016 1:47 pm

Re: Vanguard risk tolerance survey blunder.

Post by *3!4!/5! » Fri Jun 16, 2017 9:48 pm

rkhusky wrote:This looks to me like the potential numbers for approximately 20/80, 50/50 and 80/20 portfolios. Perhaps Vanguard used real portfolios to come up with the numbers, rather than some model distribution.
Yes, that's also the only explanation I could ever think of for how this question could have come about. But it yields a question where, for a wide range of interpretations to fill in the missing information (of which I just gave one interpretation as an example), option B has lower risk and higher expected return than option C. That's actually quite astonishing when you'd expect a generic survey like this should unambiguously portray a tradeoff between risk and return.
nisiprius wrote:I haven't seen a risk tolerance survey that wasn't deeply flawed. ...

I've slowly come to the cynical conclusion that none of these surveys is seriously intended to measure risk tolerance for your own benefit, but are just there to keep advisors out of trouble by putting you on the record as accepting a moderate degree of risk so you can't say they recommended unsuitable investments. ...
Right, but the question I mentioned provides a gratuitous additional flaw, over and above some of the unavoidable flaws.

And it's true that these surveys are somewhat to cover the advisors (though I believe Vanguard is genuinely trying to have a helpful website) (and I also believe honest advisors shouldn't blamed for things out of their control, such as market volatility) but they could definitely improve the survey to serve the purpose they want.

Also think about what wishes an investor may be expressing by choosing option C (more risk, less return) instead of option B. They could be saying they want to split between stocks and casino table games. Or they could be saying they want to stay invested in a highly leveraged (or inverse) stock ETF while it inevitably compounds towards zero. An advisor could carry out these self-destructive wishes in accordance with the survey results.

There's another flawed question in the Vanguard survey that asks if you would sell your bonds if they dropped 4%. They fail to distinguish between the Nervous Nellies who are bailing out at a loss, and the aggressive risk-taking investors who are selling the bonds to load up on more of the stocks that have dropped 50%.
bottlecap wrote:I disagree with your assumption that the chances are equally likely. The question asks about risk v. reward, I don't know how you can make that assumption. Nothing in the question asked about what kind of average return you want.
You have the logic exactly backwards. Pointing out the flaw in Vanguard's survey question doesn't rely on any specific assumption or interpretation. Instead, the mere existence of one (indeed many) reasonable interpretations that make option C have more risk and less return than option B, means that Vanguard's survey question has failed to unambiguously exclude that type of comparison, and that's a serious problem.
bottlecap wrote:You also have too much time on your hands!
It only took me a couple of seconds to recognize the flaw in the question.

User avatar
grabiner
Advisory Board
Posts: 20973
Joined: Tue Feb 20, 2007 11:58 pm
Location: Columbia, MD

Re: Vanguard risk tolerance survey blunder.

Post by grabiner » Fri Jun 16, 2017 10:00 pm

*3!4!/5! wrote:This question has been in Vanguard risk tolerance survey for several years.
https://personal.vanguard.com/us/FundsInvQuestionnaire
Vanguard wrote:Q: The chart below shows the greatest 1-year loss and the highest 1-year gain on 3 different hypothetical investments of $10,000. Given the potential gain or loss in any 1 year, I would invest my money in:

[_]A (loss of $164, gain of $593)

[_]B (loss of $1,020, gain of $1,921)

[_]C (loss of $3,639, gain of $4,229)
(There is a bar graph figure, but I don't know how to display it.)

Obviously this question is designed to probe the investor's feelings about risk and return, and the options are listed in increasing order of risk. But what about expected return? Well, there's not enough information, but a natural simple presumption is that the two listed return amounts are equally likely (flip a coin).

So you would get expected return

A ($593-$164)/2=$214.5
B ($1,921-$1,020)/2=$450.5
C ($4,229-$3,639)/2=$295

In other words option B has lower risk and higher expected return than option C!
I don't assume a symmetric distribution; B looks like the best and worst annual returns for bonds, and C for stocks (which lost 37% in 2008, close to the $3639 loss). I don't know what A is; it could be real returns on cash.
David Grabiner

*3!4!/5!
Posts: 1256
Joined: Fri Sep 02, 2016 1:47 pm

Re: Vanguard risk tolerance survey blunder.

Post by *3!4!/5! » Fri Jun 16, 2017 11:25 pm

grabiner wrote:
*3!4!/5! wrote:This question has been in Vanguard risk tolerance survey for several years.
https://personal.vanguard.com/us/FundsInvQuestionnaire
Vanguard wrote:Q: The chart below shows the greatest 1-year loss and the highest 1-year gain on 3 different hypothetical investments of $10,000. Given the potential gain or loss in any 1 year, I would invest my money in:

[_]A (loss of $164, gain of $593)

[_]B (loss of $1,020, gain of $1,921)

[_]C (loss of $3,639, gain of $4,229)
(There is a bar graph figure, but I don't know how to display it.)

Obviously this question is designed to probe the investor's feelings about risk and return, and the options are listed in increasing order of risk. But what about expected return? Well, there's not enough information, but a natural simple presumption is that the two listed return amounts are equally likely (flip a coin).

So you would get expected return

A ($593-$164)/2=$214.5
B ($1,921-$1,020)/2=$450.5
C ($4,229-$3,639)/2=$295

In other words option B has lower risk and higher expected return than option C!
I don't assume a symmetric distribution; B looks like the best and worst annual returns for bonds, and C for stocks (which lost 37% in 2008, close to the $3639 loss). I don't know what A is; it could be real returns on cash.
I don't literally personally assume that either. It's just a demonstration that Vanguard's survey question fails to unambiguously provide options portraying the usual tradeoff between risk and return.

You can reverse-engineer the question to try to figure how they came up with the question. But they just should have come up with a question that probed the investor desire to take more risk for more expected reward, without having an option that can reasonably be construed as having more risk for less expected reward. Also look at the question "From September 2008 through October 2008, bonds lost nearly 4%. If I owned a bond investment that lost almost 4% in 2 months, I would:". That's messed up too, so Vanguard really hasn't done a good job of designing a basic risk tolerance survey, when it shouldn't be that hard to do.

*3!4!/5!
Posts: 1256
Joined: Fri Sep 02, 2016 1:47 pm

Re: Vanguard risk tolerance survey blunder.

Post by *3!4!/5! » Fri Jun 16, 2017 11:27 pm

Here's a useful link
"Vanguard portfolio allocation models"
https://personal.vanguard.com/us/insigh ... llocations

rkhusky
Posts: 4488
Joined: Thu Aug 18, 2011 8:09 pm

Re: Vanguard risk tolerance survey blunder.

Post by rkhusky » Sun Jun 18, 2017 7:46 pm

*3!4!/5! wrote:Here's a useful link
"Vanguard portfolio allocation models"
https://personal.vanguard.com/us/insigh ... llocations
Which of these has the best risk-adjusted return according to your model?

*3!4!/5!
Posts: 1256
Joined: Fri Sep 02, 2016 1:47 pm

Re: Vanguard risk tolerance survey blunder.

Post by *3!4!/5! » Sun Jun 18, 2017 9:29 pm

rkhusky wrote:
*3!4!/5! wrote:Here's a useful link
"Vanguard portfolio allocation models"
https://personal.vanguard.com/us/insigh ... llocations
Which of these has the best risk-adjusted return according to your model?
BYOM. :happy

User avatar
nedsaid
Posts: 8821
Joined: Fri Nov 23, 2012 12:33 pm

Re: Vanguard risk tolerance survey blunder.

Post by nedsaid » Sun Jun 18, 2017 10:17 pm

My feeling is that the surveys tend to overestimate an individual's risk tolerance, thus the recommendations will inevitably lead to a more aggressive investment portfolio than the client might be comfortable with. In my own experience, I took the risk survey at my favorite mutual fund company. The survey recommended an aggressive portfolio when what I really wanted was a moderate risk portfolio. It seems like if you take the surveys recommendation and slice 10% to 15% off the recommended stock allocation, you are probably closer to your true risk tolerance.
A fool and his money are good for business.

GoldenFinch
Posts: 1384
Joined: Mon Nov 10, 2014 11:34 pm

Re: Vanguard risk tolerance survey blunder.

Post by GoldenFinch » Mon Jun 19, 2017 6:51 am

*3!4!/5! wrote:
rkhusky wrote:
*3!4!/5! wrote:Here's a useful link
"Vanguard portfolio allocation models"
https://personal.vanguard.com/us/insigh ... llocations
Which of these has the best risk-adjusted return according to your model?
BYOM. :happy
BYOM = Bring Your Own Metric?

*3!4!/5!
Posts: 1256
Joined: Fri Sep 02, 2016 1:47 pm

Re: Vanguard risk tolerance survey blunder.

Post by *3!4!/5! » Mon Jun 19, 2017 2:50 pm

GoldenFinch wrote:
*3!4!/5! wrote:
rkhusky wrote:
*3!4!/5! wrote:Here's a useful link
"Vanguard portfolio allocation models"
https://personal.vanguard.com/us/insigh ... llocations
Which of these has the best risk-adjusted return according to your model?
BYOM. :happy
BYOM = Bring Your Own Metric?
Bring Your Own Model

Thesaints
Posts: 931
Joined: Tue Jun 20, 2017 12:25 am

Re: Vanguard risk tolerance survey blunder.

Post by Thesaints » Tue Jun 20, 2017 12:46 am

Those numbers are nothing else than the (more or less) max return and max loss for stock, bond and "cash" investments, once diversifiable risk has been eliminated.
If the OP thinks that the rational choice must be then he has concluded that investing in bonds is better than investing in stocks no matter what.

*3!4!/5!
Posts: 1256
Joined: Fri Sep 02, 2016 1:47 pm

Re: Vanguard risk tolerance survey blunder.

Post by *3!4!/5! » Tue Jun 20, 2017 7:19 am

Thesaints wrote:Those numbers are nothing else than the (more or less) max return and max loss for stock, bond and "cash" investments, once diversifiable risk has been eliminated.
If the OP thinks that the rational choice must be then he has concluded that investing in bonds is better than investing in stocks no matter what.
Try rethinking at a higher level.

User avatar
bottlecap
Posts: 5048
Joined: Tue Mar 06, 2007 11:21 pm
Location: Tennessee

Re: Vanguard risk tolerance survey blunder.

Post by bottlecap » Tue Jun 20, 2017 9:02 am

*3!4!/5! wrote:
Thesaints wrote:Those numbers are nothing else than the (more or less) max return and max loss for stock, bond and "cash" investments, once diversifiable risk has been eliminated.
If the OP thinks that the rational choice must be then he has concluded that investing in bonds is better than investing in stocks no matter what.
Try rethinking at a higher level.


Be nice.

The bottom line is that this question in Vanguard survey is an instrument designed to assess risk tolerance as measured by volatility.

You think it should be a question that measures the investor's desire to take more risk for more expected reward, and should not include an option that "could" be construed as having more risk for less expected reward - if you make a bunch of assumptions.

The assumptions are that:

1) the numbers given will be assumed to be the average gain (or loss) even though they say they are not;

2) that the years of gains and losses will be equal and will occur with such regularly that they can be averaged to determine expected return;

3) that a novice investor taking this survey would make these assumptions and perform an average expected return calculation;

4) this novice investor would wrongly conclude that there is not a relationship between risk and reward; and,

5) this would lead to monumental, colossal, and devastating consequences with respect novice investor's portfolio.

Most don't agree with these assumptions or conclusions and recognize that no question in a blunt instrument is going to be perfect. Most don't think a novice would make these assumptions or even attempt the calculations.

Vanguard response would probably be:

"It's a question about risk tolerance, not a question about desired returns. Anyone who would know enough to begin making the calculations you're making would not make the assumptions you're making. The figures used are not made up and based on actual historical data. Other areas in the survey address the investor's desire and need for a certain return."

Nonetheless, if you feel so strongly about this and can't see it any other way, I suggested alerting Vanguard. Perhaps you can make a suggestion that it more to your liking.

JT

Thesaints
Posts: 931
Joined: Tue Jun 20, 2017 12:25 am

Re: Vanguard risk tolerance survey blunder.

Post by Thesaints » Tue Jun 20, 2017 10:21 am

*3!4!/5! wrote:
Thesaints wrote:Those numbers are nothing else than the (more or less) max return and max loss for stock, bond and "cash" investments, once diversifiable risk has been eliminated.
If the OP thinks that the rational choice must be then he has concluded that investing in bonds is better than investing in stocks no matter what.
Try rethinking at a higher level.


At a higher level, Vanguard question is if you feel you have the nerve to invest in cash, bonds, or stocks.
You have calculated that bonds have a higher risk-normalized return, therefore your calculation is not correct.

*3!4!/5!
Posts: 1256
Joined: Fri Sep 02, 2016 1:47 pm

Re: Vanguard risk tolerance survey blunder.

Post by *3!4!/5! » Tue Jun 20, 2017 1:21 pm

At a basic level, some people are just not grasping the logic (or the math). It's frustrating. To me, it's obvious that it's a flawed survey. Vanguard is a large organization, and inevitably some jobs are not done by their best people.

Thesaints
Posts: 931
Joined: Tue Jun 20, 2017 12:25 am

Re: Vanguard risk tolerance survey blunder.

Post by Thesaints » Tue Jun 20, 2017 1:27 pm

*3!4!/5! wrote:At a basic level, some people are just not grasping the logic (or the math). It's frustrating. To me, it's obvious that it's a flawed survey. Vanguard is a large organization, and inevitably some jobs are not done by their best people.
I'm sorry, but to calculate the expected return of the three options you necessarily had to assume a probability distribution. You cannot calculate the expected return only based on higher loss and higher gain, otherwise betting on a single number at the roulette would have a higher expected return than betting on black/red.

In fact Vanguard's question does not involve expected return at all.

User avatar
zaboomafoozarg
Posts: 1853
Joined: Sun Jun 12, 2011 12:34 pm

Re: Vanguard risk tolerance survey blunder.

Post by zaboomafoozarg » Tue Jun 20, 2017 10:38 pm

*3!4!/5! wrote:How is an astute investor supposed to answer a question like this?
I've heard before that astute investors don't take risk tolerance surveys to measure their risk tolerance, they take them to make fun of the surveys...

... and the people who make the surveys, and the people who take the surveys without recognizing the glaring blunders that anyone with a rudimentary understanding of high finance should easily spot.

Thesaints
Posts: 931
Joined: Tue Jun 20, 2017 12:25 am

Re: Vanguard risk tolerance survey blunder.

Post by Thesaints » Wed Jun 21, 2017 12:44 am

How would you assess risk tolerance ?
Granted a single question can only give a rudimentary answer and since it refers to a 10k capital those answers have completely different meaning than if it referred to a, let's say, 1 million capital.

In any case, the OP simply took a question about returns volatility and read it as one about expected returns.

Ari
Posts: 450
Joined: Sat May 23, 2015 6:59 am

Re: Vanguard risk tolerance survey blunder.

Post by Ari » Wed Jun 21, 2017 3:31 am

I agree that the question is obviously flawed. If I don't assume equal likelihood or make some other simple assumption, how could I possibly provide an answer? Perhaps option B has an expected return of -1000 USD? Without expected return, it's impossible to answer the question. Is Vanguard thinking that my preference in investment vehicles is irrelevant of expected return? Of course not. Thus, I will have to guess it to be able to answer. The two simplest guesses seem to be either assuming identical expected return (in which case A is clearly superior) or assuming equal likelihood of best and worst options (in which case B is clearly superior to C).

It's a badly designed question.
All in, all the time.

Money Market
Posts: 43
Joined: Thu Mar 16, 2017 10:36 am

Re: Vanguard risk tolerance survey blunder.

Post by Money Market » Wed Jun 21, 2017 4:18 am

nedsaid wrote:My feeling is that the surveys tend to overestimate an individual's risk tolerance, thus the recommendations will inevitably lead to a more aggressive investment portfolio than the client might be comfortable with. In my own experience, I took the risk survey at my favorite mutual fund company. The survey recommended an aggressive portfolio when what I really wanted was a moderate risk portfolio. It seems like if you take the surveys recommendation and slice 10% to 15% off the recommended stock allocation, you are probably closer to your true risk tolerance.
You are absolutely correct. They keep giving me 100%, but I'm comfortable at 85/15.

Thesaints
Posts: 931
Joined: Tue Jun 20, 2017 12:25 am

Re: Vanguard risk tolerance survey blunder.

Post by Thesaints » Wed Jun 21, 2017 4:52 am

Ari wrote:I agree that the question is obviously flawed. If I don't assume equal likelihood or make some other simple assumption, how could I possibly provide an answer? Perhaps option B has an expected return of -1000 USD? Without expected return, it's impossible to answer the question. Is Vanguard thinking that my preference in investment vehicles is irrelevant of expected return? Of course not. Thus, I will have to guess it to be able to answer. The two simplest guesses seem to be either assuming identical expected return (in which case A is clearly superior) or assuming equal likelihood of best and worst options (in which case B is clearly superior to C).

It's a badly designed question.
It is not a question about expected returns, but about volatility. More precisely not even volatility, but about tails. You can answer without knowing the particular distribution of possible outcomes.
We do this all the time in everyday life, for instance when we avoid activities with a particularly negative possible outcome (viz. death). We don't know the probability distribution, but the fact itself that death is a possible (i.e. not impossible) outcome is enough information.

The only real problem with the question is that it refers to a $10,000 investment. If I'm worth 5 millions, what happen to such an investment bears no relevance for me and all three answer correspond to a perceived risk close to zero. The situation is very different if I'm worth $50,000.

User avatar
bottlecap
Posts: 5048
Joined: Tue Mar 06, 2007 11:21 pm
Location: Tennessee

Re: Vanguard risk tolerance survey blunder.

Post by bottlecap » Wed Jun 21, 2017 6:30 am

We're talking about an 11 question survey that is not at all billed or intended as a teaching tool.

The question at issue asks about the highest one year gain or loss and further contains the caveat:

"*The maximum gain or loss on an investment is impossible to predict. The ranges shown in the chart are hypothetical and are designed solely to gauge an investor's risk tolerance."

It contains a graph (that I'm unable to produce here) that suggests (crudely), the less volatility you are willing to take, the less potential gains you are likely to make.

Another question also suggests there is a similar relationship ("5. Generally, I prefer investments with little or no fluctuation in value, and I'm willing to accept the lower return associated with these investments.").

Vanguard's "Getting started in Investing" section - which is intended to educate - is replete with explanations of the risk/reward spectrum, including a statement that "...there's typically a direct relationship between the amount of risk involved in an investment and the potential amount of money it could make."

This is one question in a risk tolerance survey that Vanguard states is hypothetical and is designed solely to gauge an investor's risk tolerance. That is, please don't rely on this question for anything else.

We understand your argument, but just don't think it's a likely concern given this context, nor is this likely to be the end of some novice investor's world.

JT

sschullo
Posts: 2310
Joined: Sun Apr 01, 2007 8:25 am
Location: Rancho Mirage, CA
Contact:

Re: Vanguard risk tolerance survey blunder.

Post by sschullo » Wed Jun 21, 2017 7:15 am

nedsaid wrote:My feeling is that the surveys tend to overestimate an individual's risk tolerance, thus the recommendations will inevitably lead to a more aggressive investment portfolio than the client might be comfortable with. In my own experience, I took the risk survey at my favorite mutual fund company. The survey recommended an aggressive portfolio when what I really wanted was a moderate risk portfolio. It seems like if you take the surveys recommendation and slice 10% to 15% off the recommended stock allocation, you are probably closer to your true risk tolerance.
I agree ned. And people fill them out in the comfortable office of their FA or online at home at the very moment the markets are soaring. Heck newbies can be seduced into a more aggressive allocation in a comfortable environment. And especially while drinking your favorite brew. :- )

After careful construction of your low-cost VG diversified portfolio, a more genuine test of risk tolerance is when the market tanks, and you see and FEEL the consequences of your plan. If you bail out of your carefully constructed diversified plan, you took on too much risk. If you stayed the course and experienced that the market and your portfolio recovered, you at spot-on with an accurate risk tolerance.

If you bail or feel terrible, not all is lost. While it's humbling and painful, it's a valuable learning tool that no book, formula, model, Excel program or algorithm can teach you. Risk tolerance surveys assume that people know themselves and act according in that comfortable setting! On the other hand, one learns great things from the pain of loss, mistakes, and miscalculations (Just ask any entrepreneur the many mistakes they committed before they become successful). You can readjust your allocation as you learn about yourself. When it comes to risk tolerance, it's about discovering how your head and your emotions REACT, not what goes on "out there."
Public School K-12 Educators: "Ask NOT what your annuity sales person can do for you, ask what you can do to be a Do-It-Yourselfer (DIY)."

User avatar
bottlecap
Posts: 5048
Joined: Tue Mar 06, 2007 11:21 pm
Location: Tennessee

Re: Vanguard risk tolerance survey blunder.

Post by bottlecap » Wed Jun 21, 2017 9:37 am

sschullo wrote:
nedsaid wrote:My feeling is that the surveys tend to overestimate an individual's risk tolerance, thus the recommendations will inevitably lead to a more aggressive investment portfolio than the client might be comfortable with. In my own experience, I took the risk survey at my favorite mutual fund company. The survey recommended an aggressive portfolio when what I really wanted was a moderate risk portfolio. It seems like if you take the surveys recommendation and slice 10% to 15% off the recommended stock allocation, you are probably closer to your true risk tolerance.
I agree ned.
I agree, too. Any "blunder" is in the guidance provided.

I took the survey this morning (over coffee...). I was honest, but erred on the side of being conservative.

It suggested and 80/20 allocation. My family's actual allocation is more along the lines of 65/35.

While I'll admit our allocation might be a bit more conservative than it could be and that we could probably even handle an 80/20 allocation, I think how Vanguard treats the responses to the questions in the survey overstates the novice investor's ability to handle volatility.

Most people are probably overconfident in their ability to handle volatility and not sell and give up after a severe downturn.

The questions in the survey ask about this ability, but seem to take the "surveyee" at their word. That's probably a mistake and the survey should err on the side of caution. Reducing the recommended stock allocation by about 10 or 15% sounds about right.

Perhaps that should be the new answer to people who come to the forum to ask about their risk tolerance!

JT

*3!4!/5!
Posts: 1256
Joined: Fri Sep 02, 2016 1:47 pm

Re: Vanguard risk tolerance survey blunder.

Post by *3!4!/5! » Wed Jun 21, 2017 1:25 pm

Thesaints wrote:It is not a question about expected returns, but about volatility.
How could you possibly know something like that?
bottlecap wrote:It contains a graph (that I'm unable to produce here) that suggests (crudely), the less volatility you are willing to take, the less potential gains you are likely to make.
What?! No! It totally fails to do that. That's the whole point of this thread.

Thesaints
Posts: 931
Joined: Tue Jun 20, 2017 12:25 am

Re: Vanguard risk tolerance survey blunder.

Post by Thesaints » Wed Jun 21, 2017 2:59 pm

*3!4!/5! wrote:
Thesaints wrote:It is not a question about expected returns, but about volatility.
How could you possibly know something like that?
I know because it is written in the question text:
The chart below shows the greatest 1-year loss and the highest 1-year gain on 3 different hypothetical investments of $10,000. Given the potential gain or loss in any 1 year, I would invest my money...
Given max gain and loss decide which investment is better suited for you.

alex_686
Posts: 2584
Joined: Mon Feb 09, 2015 2:39 pm

Re: Vanguard risk tolerance survey blunder.

Post by alex_686 » Wed Jun 21, 2017 3:32 pm

*3!4!/5! wrote:I already said that. But the question is obviously deeply flawed.
It is not flawed - It is exactly right. The flaw is that you assume that investors will weight gains the same as losses. This is not true for both logical and behavioral reasons.

For a logical reason, take a person who has retired and must live on their savings. A large loss will mean they won't be able to meet their retirement goals and living standards Choice A is the right choice for them.

For cognitive defects, behavioral issues are the hardest to overcome. Education rarely works. Industry lore is that one should accommodate this behavioral issue rather than fix the customer.

This is a challenging subject. It tripped up Milton Friedman. Try looking up "Loss Aversion". Daniel Kahneman's discovered the principle and got a Nobel for it. His book "Thinking Fast, Thinking Slow" is a good introductory read. So is Michael Lewis "The Undoing Project" is another good introductory read.

User avatar
saltycaper
Posts: 2195
Joined: Thu Apr 24, 2014 8:47 pm
Location: The Tower

Re: Vanguard risk tolerance survey blunder.

Post by saltycaper » Wed Jun 21, 2017 3:57 pm

I agree the question is a poor one. I recall the question from when I took the survey some time ago, and it confused me. I didn't know what it was I was trying to calculate in my mind; not what it was called, not even whether I was making a sensible calculation. It just struck me that "C" was not only an answer unsuitable for me, but also an answer that didn't seem to make sense at all. I wound up choosing "B". Like many others commented, I thought the risk tolerance Vanguard assigned to me was way too high.
"I guess I should warn you, if I turn out to be particularly clear, you've probably misunderstood what I've said." --Alan Greenspan

User avatar
bottlecap
Posts: 5048
Joined: Tue Mar 06, 2007 11:21 pm
Location: Tennessee

Re: Vanguard risk tolerance survey blunder.

Post by bottlecap » Wed Jun 21, 2017 4:33 pm

*3!4!/5! wrote:
Thesaints wrote:It is not a question about expected returns, but about volatility.
How could you possibly know something like that?
Because the question explicitly states that it is "solely designed to gauge the investor's risk tolerance."

JT

Thesaints
Posts: 931
Joined: Tue Jun 20, 2017 12:25 am

Re: Vanguard risk tolerance survey blunder.

Post by Thesaints » Wed Jun 21, 2017 4:38 pm

I suggest people try to answer the question when possible outcomes do not refer to a 10k investment, but to their own net invested worth.
If you've got a million, multiply all amounts by 100, for instance.
There are no calculations at all to be made. Just look at max gains and losses involved and go with your guts.
That's the right spirit.

alex_686
Posts: 2584
Joined: Mon Feb 09, 2015 2:39 pm

Re: Vanguard risk tolerance survey blunder.

Post by alex_686 » Wed Jun 21, 2017 5:00 pm

nisiprius wrote:I haven't seen a risk tolerance survey that wasn't deeply flawed. The best that can be said about them is that if you've never invested in securities before and are used to bank accounts, it at least rubs your nose into the idea that there could be losses and the ballpark sizes your losses could be.


I would agree with you, but is there a better alternative out there? Willingness to take risk is one of the most important things about developing a ISP and to the success of the plan. It is also notoriously hard to measure. I would argue that a flawed method - and knowing what those flaws are - is better than a no method.
nisiprius wrote:But these "risk tolerance" questionnaires seem to me to avoid it. They never say "how would you feel about losing half your money," it's always some crazy-precise, randomish, bloodless number like--in Vanguard's case--"over 31%."
Yeah - this does not work. People are very good at lying to themselves when you ask them a direct question. From my personal experience, when somebody says they can stand a 50% drop in their portfolio they will sell out of the market when it drops by 10%. Score one for the behavioral economist. You get better results when you ask people to make gut checks. It is one of those cases where congestive defects in calculating expected return tend to map over to one's risk aversion.
Last edited by alex_686 on Wed Jun 21, 2017 5:43 pm, edited 1 time in total.

frankbrenowitz
Posts: 54
Joined: Wed Nov 19, 2014 9:53 pm

Re: Vanguard risk tolerance survey blunder.

Post by frankbrenowitz » Wed Jun 21, 2017 5:16 pm

My planning spreadsheet contains historical and current portfolio values, with potential loss percentages attached to the current values by asset class. I know to the penny how much my equities will decline with a 50% drop in the market. It keeps the numbers real and informs my asset allocation.

Post Reply