## Calibrating your Confidence and Why It Matters for Investing

- PrettyCoolWorkshop
**Posts:**192**Joined:**Tue Aug 11, 2015 8:44 pm

### Calibrating your Confidence and Why It Matters for Investing

You may or may not have heard of the concept of "calibrating your confidence." It is a formal process to determine whether you successfully predict outcomes of events in proportion to how likely you believe these outcomes to occur. On another level, it also lets you know whether your model of the world is providing you useful, actionable, accurate information.

You can imagine formalizing your predictions as follows:

1. "I have 70% confidence that the S&P500 will close at a higher level on August 6, 2020, than on August 6, 2019"

2. "I have 30% confidence that my incumbent state senator will win his reelection campaign."

3. "I have 5% confidence that Seabiscuit will win the triple crown."

Let's imagine a histogram with your confidence from 0-100% in bins of 10% on the x axis, and your measured predictive success, based off of all the predictions you have ever stated, on the y-axis. If somebody has perfectly calibrated confidence, in the long run, their 10% confident predictions would have been right 10% of the time, their 20% confident predictions would have been right 20% of the time, etc, all the way up to 100%.

A hypothetical person with perfectly calibrated confidence would be an excellent gambling handicapper, and an excellent stock picker. He would know whether odds for a particular bet were favorable, and he would be able to trust his intuitions. Although he may only know that a particular bet has only a 20% chance of paying off, he would know whether it is worth taking, as long as the payoff is at least 5x the wager.

An underconfident person would fail to notice or act on favorable bets.

An overconfident person will take many bets that don't end up paying off. Psychological research has shown that humans, as a whole, err on the side of overconfidence.

Bogleheads, by diversifying, ascribe to a low-confidence strategy. They do not claim to have strong predictive skill or act in a way that demonstrates intent to make strong predictions. Highly active traders ascribe to a high-confidence strategy, where they try (and may fail or succeed) to out-predict the market.

Here is a toy to try out this experiment:

http://confidence.success-equation.com/

You can imagine formalizing your predictions as follows:

1. "I have 70% confidence that the S&P500 will close at a higher level on August 6, 2020, than on August 6, 2019"

2. "I have 30% confidence that my incumbent state senator will win his reelection campaign."

3. "I have 5% confidence that Seabiscuit will win the triple crown."

Let's imagine a histogram with your confidence from 0-100% in bins of 10% on the x axis, and your measured predictive success, based off of all the predictions you have ever stated, on the y-axis. If somebody has perfectly calibrated confidence, in the long run, their 10% confident predictions would have been right 10% of the time, their 20% confident predictions would have been right 20% of the time, etc, all the way up to 100%.

A hypothetical person with perfectly calibrated confidence would be an excellent gambling handicapper, and an excellent stock picker. He would know whether odds for a particular bet were favorable, and he would be able to trust his intuitions. Although he may only know that a particular bet has only a 20% chance of paying off, he would know whether it is worth taking, as long as the payoff is at least 5x the wager.

An underconfident person would fail to notice or act on favorable bets.

An overconfident person will take many bets that don't end up paying off. Psychological research has shown that humans, as a whole, err on the side of overconfidence.

Bogleheads, by diversifying, ascribe to a low-confidence strategy. They do not claim to have strong predictive skill or act in a way that demonstrates intent to make strong predictions. Highly active traders ascribe to a high-confidence strategy, where they try (and may fail or succeed) to out-predict the market.

Here is a toy to try out this experiment:

http://confidence.success-equation.com/

Be greedy and fearful. All the time.

### Re: Calibrating your Confidence and Why It Matters for Investing

PrettyCoolWorkshop wrote: ↑Mon Aug 05, 2019 9:03 amA hypothetical person with perfectly calibrated confidence would be an excellent gambling handicapper, and an excellent stock picker.

In a way, this is true. Of course, an "excellent stock picker" would likely be one who never picks any stocks at all.

Bogleheads, by diversifying, ascribe to a low-confidence strategy. They do not claim to have strong predictive skill or act in a way that demonstrates intent to make strong predictions. Highly active traders ascribe to a high-confidence strategy, where they try (and may fail or succeed) to out-predict the market.

I am not sure I agree with this assessment. In fact, I have been a highly "confident" investor my whole life and consider myself a Boglehead. I am highly confident that over the extremely long term stocks will outperform bonds. So far, so good. Highly active traders very often have the same exact level of confidence, just in difference instances. As such, Bogleheads, in my opinion, have a much better calibrated confidence, in my opinion, not less confidence.

On the whole though I do agree that the average human believe that he or she is above average. This results in a miscalibration of confidence in some aspects of most lives. Money still flows into active funds despite stacks and stacks of reports indicating underperformance.

### Re: Calibrating your Confidence and Why It Matters for Investing

OP are you trying to pick a fight here?

- nisiprius
- Advisory Board
**Posts:**39432**Joined:**Thu Jul 26, 2007 9:33 am**Location:**The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

### Re: Calibrating your Confidence and Why It Matters for Investing

I can't resist the temptation...

I'm not sure how it relates to investing, and I'm not sure I agree with PrettyCoolWorkshop's characterizations, but I don't see any evidence of either gross overconfidenceCalibration score

Mean confidence: 70.60%

Actual percent correct: 76.00%

You want your mean confidence and actual score to be as close as possible.

Mean confidence on correct answers: 74.47%

Mean confidence on incorrect answers: 58.33%

You want your mean confidence to be low for incorrect answers and high for correct answers.

Quiz score

38 correct out of 50 questions answered (76.00%)

16 correct out of 25 questions answered with low (50 or 60%) confidence (64.00%)

6 correct out of 9 questions answered with medium (70% or 80%) confidence (66.67%)

16 correct out of 16 questions answered with high (90 or 100%) confidence (100.00%)

**or**gross underconfidence. I don't think the the Bogleheads investment philosophy is necessariy a "low confidence strategy," I think it might well be an accurate or realistic confidence strategy.
Last edited by nisiprius on Mon Aug 05, 2019 6:05 pm, edited 1 time in total.

Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

### Re: Calibrating your Confidence and Why It Matters for Investing

Um, no. That isn't at all the correct way to determine whether to take a wager. I'd review risk adjusted return if I were you...PrettyCoolWorkshop wrote: ↑Mon Aug 05, 2019 9:03 amAlthough he may only know that a particular bet has only a 20% chance of paying off, he would know whether it is worth taking, as long as the payoff is at least 5x the wager.

### Re: Calibrating your Confidence and Why It Matters for Investing

you have it backwardsPrettyCoolWorkshop wrote: ↑Mon Aug 05, 2019 9:03 am

Bogleheads, by diversifying, ascribe to a low-confidence strategy.

Three-Fund Portfolio: FSPSX - FXAIX - FXNAX (with slight tilt of CD - CASH - Canned Beans - Rice - Bottled Water)

### Re: Calibrating your Confidence and Why It Matters for Investing

Who came up with this concept and the equation, i.e., who's behind this? How did they define "confidence"? Did they come up with the Bogleheads' example or did you?PrettyCoolWorkshop wrote: ↑Mon Aug 05, 2019 9:03 amYou may or may not have heard of the concept of "calibrating your confidence." It is a formal process to determine whether you successfully predict outcomes of events in proportion to how likely you believe these outcomes to occur. On another level, it also lets you know whether your model of the world is providing you useful, actionable, accurate information.

http://confidence.success-equation.com/

John Bogle on his often bumpy road to low-cost indexing: "When a door closes, if you look long enough and hard enough, if you're strong enough, you'll find a window that opens."

### Re: Calibrating your Confidence and Why It Matters for Investing

*Calibration score*

Mean confidence: 72.20%

Actual percent correct: 76.00%

You want your mean confidence and actual score to be as close as possible.

Mean confidence on correct answers: 76.32%

Mean confidence on incorrect answers: 59.17%

You want your mean confidence to be low for incorrect answers and high for correct answers.

Quiz score

38 correct out of 50 questions answered (76.00%)

12 correct out of 21 questions answered with low (50 or 60%) confidence (57.14%)

10 correct out of 12 questions answered with medium (70% or 80%) confidence (83.33%)

16 correct out of 17 questions answered with high (90 or 100%) confidence (94.12%)

Mean confidence: 72.20%

Actual percent correct: 76.00%

You want your mean confidence and actual score to be as close as possible.

Mean confidence on correct answers: 76.32%

Mean confidence on incorrect answers: 59.17%

You want your mean confidence to be low for incorrect answers and high for correct answers.

Quiz score

38 correct out of 50 questions answered (76.00%)

12 correct out of 21 questions answered with low (50 or 60%) confidence (57.14%)

10 correct out of 12 questions answered with medium (70% or 80%) confidence (83.33%)

16 correct out of 17 questions answered with high (90 or 100%) confidence (94.12%)

I think by some criteria I'm equally right, but more finely calibrated, as Nisiprius. And only slightly overconfident (about it).

- PrettyCoolWorkshop
**Posts:**192**Joined:**Tue Aug 11, 2015 8:44 pm

### Re: Calibrating your Confidence and Why It Matters for Investing

This is a more fine grained distinction than I laid out in my original post, and I think it's worthwhile to break it down further. I think it is difficult to disentangle these four things:I am not sure I agree with this assessment. In fact, I have been a highly "confident" investor my whole life and consider myself a Boglehead. I am highly confident that over the extremely long term stocks will outperform bonds. So far, so good. Highly active traders very often have the same exact level of confidence, just in difference instances. As such, Bogleheads, in my opinion, have a much better calibrated confidence, in my opinion, not less confidence.

1. How often you like to make predictions (informally, highly confident people often tend to make and act on more predictions than other people)

2. The likelihood you believe your predicted results will occur (formally, how confident you are)

3. Your predictive success (how often your predicted outcomes actually do occur)

4. How well calibrated you are (whether #2 matches #3)

We can both agree that Bogleheads prefer to act on investment predictions less often than active traders do. On the other hand, many Bogleheads think it is very likely they will outperform the active traders. So far the evidence shows that this is a well-calibrated belief for most Bogleheads.

Nope, I index like most of the rest of the Bogleheads. I'm not claiming to be on the side of stock pickers. I think most of them are overconfident, and that is why most of them do not outperform in the long run. I would like to encourage Bogleheads to perform an exercise to test their own forward-looking predictive success by recording some 1-year long predictions and checking back in on them in due time. For some exceptionally good predicters, they might consider slowly ramping up a "play money" account, conditional on continued predictive success, measured as beating their chosen reference index fund or reference portfolio. However, becoming a picker is not the primary purpose of the exercise. The exercise should mainly help your mental clarity with regards to the long-term nature of investing, how one should make observations over time, and under what ordinary extraordinary circumstances it is worth it to no longer "hold the course." If the zombie apocalypse actually began to occur, I doubt everyone would cling to their portfolios quite the same as they do today.OP are you trying to pick a fight here?

At a high level, you should always try to weight your own predictions vs. that of the sometimes-emotional Mr. Market. You personally know that you are subject to error, and that everyone else is subject to error as well. Calibrating your confidence, in the long run, will help you know when it is worth it to try to out-guess your peers. It is rarely a good course of action to try to out-guess the collective wisdom of millions. However, if extraordinary circumstances do strike, those who have considered their possibility can outmaneuver others. This applies in life as well as investing.

I've hopefully cleared things up with #1-4 above.you have it backwardsBogleheads, by diversifying, ascribe to a low-confidence strategy.

"Confidence Intervals" are an accepted practice in statistics, and using the concept on an everyday basis is a popular concept among the online rationalist community. There's a bit more discussion at the link below.Who came up with this concept and the equation, i.e., who's behind this? How did they define "confidence"? Did they come up with the Bogleheads' example or did you?

https://www.lesswrong.com/posts/225697e ... confidence

https://www.lesswrong.com/posts/ofSYgmM ... alibration

Be greedy and fearful. All the time.

### Re: Calibrating your Confidence and Why It Matters for Investing

Here is background on the rationalist movement and the link:PrettyCoolWorkshop wrote: ↑Tue Aug 06, 2019 10:02 am..."Confidence Intervals" are an accepted practice in statistics, and using the concept on an everyday basis is a popular concept among the online rationalist community. There's a bit more discussion at the link below.Who came up with this concept and the equation, i.e., who's behind this? How did they define "confidence"? Did they come up with the Bogleheads' example or did you?

https://www.lesswrong.com/posts/225697e ... confidence

https://www.lesswrong.com/posts/ofSYgmM ... alibration

https://wiki.lesswrong.com/wiki/Rationalist_movementThe rationalist movement, rationality community,[1] rationalsphere or rationalistsphere[2] represents a set of modes of bayesian thinking from self-described rationalists or 'aspiring rationalists' typically associated with the Less Wrong diaspora and their associated communities.

John Bogle on his often bumpy road to low-cost indexing: "When a door closes, if you look long enough and hard enough, if you're strong enough, you'll find a window that opens."