Calibrating your Confidence and Why It Matters for Investing

Discuss all general (i.e. non-personal) investing questions and issues, investing news, and theory.
Post Reply
User avatar
Topic Author
PrettyCoolWorkshop
Posts: 192
Joined: Tue Aug 11, 2015 8:44 pm

Calibrating your Confidence and Why It Matters for Investing

Post by PrettyCoolWorkshop » Mon Aug 05, 2019 9:03 am

You may or may not have heard of the concept of "calibrating your confidence." It is a formal process to determine whether you successfully predict outcomes of events in proportion to how likely you believe these outcomes to occur. On another level, it also lets you know whether your model of the world is providing you useful, actionable, accurate information.

You can imagine formalizing your predictions as follows:
1. "I have 70% confidence that the S&P500 will close at a higher level on August 6, 2020, than on August 6, 2019"
2. "I have 30% confidence that my incumbent state senator will win his reelection campaign."
3. "I have 5% confidence that Seabiscuit will win the triple crown."


Let's imagine a histogram with your confidence from 0-100% in bins of 10% on the x axis, and your measured predictive success, based off of all the predictions you have ever stated, on the y-axis. If somebody has perfectly calibrated confidence, in the long run, their 10% confident predictions would have been right 10% of the time, their 20% confident predictions would have been right 20% of the time, etc, all the way up to 100%.

A hypothetical person with perfectly calibrated confidence would be an excellent gambling handicapper, and an excellent stock picker. He would know whether odds for a particular bet were favorable, and he would be able to trust his intuitions. Although he may only know that a particular bet has only a 20% chance of paying off, he would know whether it is worth taking, as long as the payoff is at least 5x the wager.

An underconfident person would fail to notice or act on favorable bets.

An overconfident person will take many bets that don't end up paying off. Psychological research has shown that humans, as a whole, err on the side of overconfidence.

Bogleheads, by diversifying, ascribe to a low-confidence strategy. They do not claim to have strong predictive skill or act in a way that demonstrates intent to make strong predictions. Highly active traders ascribe to a high-confidence strategy, where they try (and may fail or succeed) to out-predict the market.

Here is a toy to try out this experiment:
http://confidence.success-equation.com/
Be greedy and fearful. All the time.

megabad
Posts: 2524
Joined: Fri Jun 01, 2018 4:00 pm

Re: Calibrating your Confidence and Why It Matters for Investing

Post by megabad » Mon Aug 05, 2019 4:38 pm

PrettyCoolWorkshop wrote:
Mon Aug 05, 2019 9:03 am
A hypothetical person with perfectly calibrated confidence would be an excellent gambling handicapper, and an excellent stock picker.
In a way, this is true. Of course, an "excellent stock picker" would likely be one who never picks any stocks at all.

Bogleheads, by diversifying, ascribe to a low-confidence strategy. They do not claim to have strong predictive skill or act in a way that demonstrates intent to make strong predictions. Highly active traders ascribe to a high-confidence strategy, where they try (and may fail or succeed) to out-predict the market.
I am not sure I agree with this assessment. In fact, I have been a highly "confident" investor my whole life and consider myself a Boglehead. I am highly confident that over the extremely long term stocks will outperform bonds. So far, so good. Highly active traders very often have the same exact level of confidence, just in difference instances. As such, Bogleheads, in my opinion, have a much better calibrated confidence, in my opinion, not less confidence.

On the whole though I do agree that the average human believe that he or she is above average. This results in a miscalibration of confidence in some aspects of most lives. Money still flows into active funds despite stacks and stacks of reports indicating underperformance.

Dottie57
Posts: 7203
Joined: Thu May 19, 2016 5:43 pm
Location: Earth Northern Hemisphere

Re: Calibrating your Confidence and Why It Matters for Investing

Post by Dottie57 » Mon Aug 05, 2019 4:46 pm

OP are you trying to pick a fight here?

User avatar
nisiprius
Advisory Board
Posts: 39486
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Calibrating your Confidence and Why It Matters for Investing

Post by nisiprius » Mon Aug 05, 2019 5:40 pm

I can't resist the temptation...
Calibration score
Mean confidence: 70.60%
Actual percent correct: 76.00%

You want your mean confidence and actual score to be as close as possible.

Mean confidence on correct answers: 74.47%
Mean confidence on incorrect answers: 58.33%

You want your mean confidence to be low for incorrect answers and high for correct answers.

Quiz score
38 correct out of 50 questions answered (76.00%)
16 correct out of 25 questions answered with low (50 or 60%) confidence (64.00%)
6 correct out of 9 questions answered with medium (70% or 80%) confidence (66.67%)
16 correct out of 16 questions answered with high (90 or 100%) confidence (100.00%)

Image
I'm not sure how it relates to investing, and I'm not sure I agree with PrettyCoolWorkshop's characterizations, but I don't see any evidence of either gross overconfidence or gross underconfidence. I don't think the the Bogleheads investment philosophy is necessariy a "low confidence strategy," I think it might well be an accurate or realistic confidence strategy.
Last edited by nisiprius on Mon Aug 05, 2019 6:05 pm, edited 1 time in total.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

thx1138
Posts: 997
Joined: Fri Jul 12, 2013 2:14 pm

Re: Calibrating your Confidence and Why It Matters for Investing

Post by thx1138 » Mon Aug 05, 2019 5:51 pm

PrettyCoolWorkshop wrote:
Mon Aug 05, 2019 9:03 am
Although he may only know that a particular bet has only a 20% chance of paying off, he would know whether it is worth taking, as long as the payoff is at least 5x the wager.
Um, no. That isn't at all the correct way to determine whether to take a wager. I'd review risk adjusted return if I were you...

cashboy
Posts: 154
Joined: Tue Sep 11, 2018 5:03 pm
Location: USA

Re: Calibrating your Confidence and Why It Matters for Investing

Post by cashboy » Mon Aug 05, 2019 6:21 pm

PrettyCoolWorkshop wrote:
Mon Aug 05, 2019 9:03 am


Bogleheads, by diversifying, ascribe to a low-confidence strategy.
you have it backwards

:D
Three-Fund Portfolio: FSPSX - FXAIX - FXNAX (with slight tilt of CD - CASH - Canned Beans - Rice - Bottled Water)

Fallible
Posts: 7037
Joined: Fri Nov 27, 2009 4:44 pm
Contact:

Re: Calibrating your Confidence and Why It Matters for Investing

Post by Fallible » Mon Aug 05, 2019 6:51 pm

PrettyCoolWorkshop wrote:
Mon Aug 05, 2019 9:03 am
You may or may not have heard of the concept of "calibrating your confidence." It is a formal process to determine whether you successfully predict outcomes of events in proportion to how likely you believe these outcomes to occur. On another level, it also lets you know whether your model of the world is providing you useful, actionable, accurate information.
http://confidence.success-equation.com/
Who came up with this concept and the equation, i.e., who's behind this? How did they define "confidence"? Did they come up with the Bogleheads' example or did you?
John Bogle on his often bumpy road to low-cost indexing: "When a door closes, if you look long enough and hard enough, if you're strong enough, you'll find a window that opens."

dorster
Posts: 19
Joined: Tue Aug 15, 2017 4:20 pm

Re: Calibrating your Confidence and Why It Matters for Investing

Post by dorster » Mon Aug 05, 2019 7:12 pm

Calibration score
Mean confidence: 72.20%
Actual percent correct: 76.00%
You want your mean confidence and actual score to be as close as possible.

Mean confidence on correct answers: 76.32%
Mean confidence on incorrect answers: 59.17%
You want your mean confidence to be low for incorrect answers and high for correct answers.

Quiz score
38 correct out of 50 questions answered (76.00%)
12 correct out of 21 questions answered with low (50 or 60%) confidence (57.14%)
10 correct out of 12 questions answered with medium (70% or 80%) confidence (83.33%)
16 correct out of 17 questions answered with high (90 or 100%) confidence (94.12%)


I think by some criteria I'm equally right, but more finely calibrated, as Nisiprius. And only slightly overconfident (about it).

User avatar
Topic Author
PrettyCoolWorkshop
Posts: 192
Joined: Tue Aug 11, 2015 8:44 pm

Re: Calibrating your Confidence and Why It Matters for Investing

Post by PrettyCoolWorkshop » Tue Aug 06, 2019 10:02 am

I am not sure I agree with this assessment. In fact, I have been a highly "confident" investor my whole life and consider myself a Boglehead. I am highly confident that over the extremely long term stocks will outperform bonds. So far, so good. Highly active traders very often have the same exact level of confidence, just in difference instances. As such, Bogleheads, in my opinion, have a much better calibrated confidence, in my opinion, not less confidence.
This is a more fine grained distinction than I laid out in my original post, and I think it's worthwhile to break it down further. I think it is difficult to disentangle these four things:

1. How often you like to make predictions (informally, highly confident people often tend to make and act on more predictions than other people)
2. The likelihood you believe your predicted results will occur (formally, how confident you are)
3. Your predictive success (how often your predicted outcomes actually do occur)
4. How well calibrated you are (whether #2 matches #3)

We can both agree that Bogleheads prefer to act on investment predictions less often than active traders do. On the other hand, many Bogleheads think it is very likely they will outperform the active traders. So far the evidence shows that this is a well-calibrated belief for most Bogleheads.
OP are you trying to pick a fight here?
Nope, I index like most of the rest of the Bogleheads. I'm not claiming to be on the side of stock pickers. I think most of them are overconfident, and that is why most of them do not outperform in the long run. I would like to encourage Bogleheads to perform an exercise to test their own forward-looking predictive success by recording some 1-year long predictions and checking back in on them in due time. For some exceptionally good predicters, they might consider slowly ramping up a "play money" account, conditional on continued predictive success, measured as beating their chosen reference index fund or reference portfolio. However, becoming a picker is not the primary purpose of the exercise. The exercise should mainly help your mental clarity with regards to the long-term nature of investing, how one should make observations over time, and under what ordinary extraordinary circumstances it is worth it to no longer "hold the course." If the zombie apocalypse actually began to occur, I doubt everyone would cling to their portfolios quite the same as they do today.

At a high level, you should always try to weight your own predictions vs. that of the sometimes-emotional Mr. Market. You personally know that you are subject to error, and that everyone else is subject to error as well. Calibrating your confidence, in the long run, will help you know when it is worth it to try to out-guess your peers. It is rarely a good course of action to try to out-guess the collective wisdom of millions. However, if extraordinary circumstances do strike, those who have considered their possibility can outmaneuver others. This applies in life as well as investing.
Bogleheads, by diversifying, ascribe to a low-confidence strategy.
you have it backwards

:D
I've hopefully cleared things up with #1-4 above.
Who came up with this concept and the equation, i.e., who's behind this? How did they define "confidence"? Did they come up with the Bogleheads' example or did you?
"Confidence Intervals" are an accepted practice in statistics, and using the concept on an everyday basis is a popular concept among the online rationalist community. There's a bit more discussion at the link below.

https://www.lesswrong.com/posts/225697e ... confidence
https://www.lesswrong.com/posts/ofSYgmM ... alibration
Be greedy and fearful. All the time.

Fallible
Posts: 7037
Joined: Fri Nov 27, 2009 4:44 pm
Contact:

Re: Calibrating your Confidence and Why It Matters for Investing

Post by Fallible » Tue Aug 06, 2019 11:05 am

PrettyCoolWorkshop wrote:
Tue Aug 06, 2019 10:02 am
...
Who came up with this concept and the equation, i.e., who's behind this? How did they define "confidence"? Did they come up with the Bogleheads' example or did you?
"Confidence Intervals" are an accepted practice in statistics, and using the concept on an everyday basis is a popular concept among the online rationalist community. There's a bit more discussion at the link below.

https://www.lesswrong.com/posts/225697e ... confidence
https://www.lesswrong.com/posts/ofSYgmM ... alibration
Here is background on the rationalist movement and the link:
The rationalist movement, rationality community,[1] rationalsphere or rationalistsphere[2] represents a set of modes of bayesian thinking from self-described rationalists or 'aspiring rationalists' typically associated with the Less Wrong diaspora and their associated communities.
https://wiki.lesswrong.com/wiki/Rationalist_movement
John Bogle on his often bumpy road to low-cost indexing: "When a door closes, if you look long enough and hard enough, if you're strong enough, you'll find a window that opens."

Post Reply