Testing validity of Monte Carlo Simulators

Non-investing personal finance issues including insurance, credit, real estate, taxes, employment and legal issues such as trusts and wills
Post Reply
Topic Author
10YearPlan
Posts: 250
Joined: Tue Dec 22, 2015 2:58 pm

Testing validity of Monte Carlo Simulators

Post by 10YearPlan » Wed Sep 26, 2018 10:54 am

Anyone compare results from Monte Carlo simulators to actual performance in retirement? By design, I know they are meant to provide a broad range of results, which of course is helpful. But of those already retired, is anyone tracking how close to the low, high or average your portfolio is performing?

I feel pretty comfortable with a 90% accuracy rate, but just trying to figure out how far out some of the projections really are. Any insight is helpful!

heerekj1
Posts: 111
Joined: Sat Jun 28, 2014 8:42 am

Re: Testing validity of Monte Carlo Simulators

Post by heerekj1 » Wed Sep 26, 2018 11:16 am

I did not trust what others were providing so I wrote my own simulator. It basically confirmed what others have done. That said, I don't understand how one could confirm that their individual performance would match the simulation. As you note they produce a wide range of results and you choose the probability that you can live with.

I was looking at the probability of having money left over at the end of 25-30 years. How would people report their result?

User avatar
vineviz
Posts: 5391
Joined: Tue May 15, 2018 1:55 pm

Re: Testing validity of Monte Carlo Simulators

Post by vineviz » Wed Sep 26, 2018 12:04 pm

10YearPlan wrote:
Wed Sep 26, 2018 10:54 am
Anyone compare results from Monte Carlo simulators to actual performance in retirement? By design, I know they are meant to provide a broad range of results, which of course is helpful. But of those already retired, is anyone tracking how close to the low, high or average your portfolio is performing?
Pretty much by definition, the results of a Monte Carlo simulator will be perfectly valid (assuming it is competently built).

They are not predicting the future, after all, but are merely calculating a set of mathematical probabilities GIVEN a certain set of inputs.

Those inputs (expected returns, expected variance, and expected correlation) will determine the output. Unfortunately, it is up to the investor to use they judgement abut what those inputs should be.
"Far more money has been lost by investors preparing for corrections than has been lost in corrections themselves." ~~ Peter Lynch

michaeljc70
Posts: 5828
Joined: Thu Oct 15, 2015 3:53 pm

Re: Testing validity of Monte Carlo Simulators

Post by michaeljc70 » Wed Sep 26, 2018 12:12 pm

heerekj1 wrote:
Wed Sep 26, 2018 11:16 am
I did not trust what others were providing so I wrote my own simulator. It basically confirmed what others have done. That said, I don't understand how one could confirm that their individual performance would match the simulation. As you note they produce a wide range of results and you choose the probability that you can live with.

I was looking at the probability of having money left over at the end of 25-30 years. How would people report their result?
I agree. Unless you opt for 100% success, there is always a chance for failure. And even if you used 100% success rate, that assumes returns like we've had in the past which could vary. I guess I can post on here in 10,20,30 and 40 years if I am eating cat food.

Topic Author
10YearPlan
Posts: 250
Joined: Tue Dec 22, 2015 2:58 pm

Re: Testing validity of Monte Carlo Simulators

Post by 10YearPlan » Wed Sep 26, 2018 12:28 pm

I guess I should clarify. Since I know none of us can come back from the dead and tell us what was left the day they departed, what I was getting at was...if you are already retired, how close (or not) did the simulators come to predicting your actual situation (to date). Does that make sense?

elliott908
Posts: 3
Joined: Mon Apr 13, 2015 11:35 am

Re: Testing validity of Monte Carlo Simulators

Post by elliott908 » Wed Sep 26, 2018 12:37 pm

It totally makes sense, and I hope that some will chime in here as I'm curious as well.

Shallowpockets
Posts: 1432
Joined: Fri Nov 20, 2015 10:26 am

Re: Testing validity of Monte Carlo Simulators

Post by Shallowpockets » Wed Sep 26, 2018 12:50 pm

I have a result based on significantly below market average and a monthly withdrawal rate about 20% higher than I actually need. Using these parameters I feel I have a buffer built into the simulation. Therefore since retired, only 2.5 years, I am well under the actual numbers of drawdown, but well over the numbers in further accrual of assets.

I just looked at the analysis today and did not input any asset changes. I use the Fidelity retirement simulater. This does not have access to all my asset info so I must update those, which I do at end of the year. However I do an overall asset compilation throughout the year to see how it all goes. Sometimes I do it after a great week in the market and sometimes after a bad week. This way I have a reasonable feedback on fluctuations that might affect me. Thus far the fluctuation is not of any significance.

I do not know how my results with Fidelity correspond with Firecalc. I did look there once and results were similar. I believe that you can shoot for what you can accrue and then make your decisions. Since I am retired and felt it was OK at that time, I am certainly not going back. Once you make the decision, the simulation is of concern only if you were cutting it too close at retirement.

People are always posting here trying to find the perfect number for their retirement. You make your choices and good feedback in a Monte Carlo simulator helps with that choice. Whether or not it continues to hold true through the years is based on what is the input/changes on your end as well as the what data they used that comprised the makeup of the Monte Carlo model.

mindbogle
Posts: 140
Joined: Sun Feb 10, 2013 11:28 am

Re: Testing validity of Monte Carlo Simulators

Post by mindbogle » Wed Sep 26, 2018 1:08 pm

10YearPlan wrote:
Wed Sep 26, 2018 12:28 pm
I guess I should clarify. Since I know none of us can come back from the dead and tell us what was left the day they departed, what I was getting at was...if you are already retired, how close (or not) did the simulators come to predicting your actual situation (to date). Does that make sense?
I don't think it makes a lot of sense. As mentioned above, MC simulators don't "predict" the future. They are used to better understand uncertainty - to provide insight into the range of possible outcomes and probabilities. A typical simulation might result in thousands of outcomes, limited only by computer time.

And as said in a previous reply above, the "validity" of MC results depend on the future relevance of assumed market dynamics and of the user chosen inputs (forward-looking return distributions, parameter cross-correlations, etc). Not sure how one would back-test those.

Regards, MB

bradpevans
Posts: 598
Joined: Sun Apr 08, 2018 1:09 pm

Re: Testing validity of Monte Carlo Simulators

Post by bradpevans » Wed Sep 26, 2018 1:25 pm

mindbogle wrote:
Wed Sep 26, 2018 1:08 pm
10YearPlan wrote:
Wed Sep 26, 2018 12:28 pm
I guess I should clarify. Since I know none of us can come back from the dead and tell us what was left the day they departed, what I was getting at was...if you are already retired, how close (or not) did the simulators come to predicting your actual situation (to date). Does that make sense?
I don't think it makes a lot of sense. As mentioned above, MC simulators don't "predict" the future. They are used to better understand uncertainty - to provide insight into the range of possible outcomes and probabilities. A typical simulation might result in thousands of outcomes, limited only by computer time.

And as said in a previous reply above, the "validity" of MC results depend on the future relevance of assumed market dynamics and of the user chosen inputs (forward-looking return distributions, parameter cross-correlations, etc). Not sure how one would back-test those.

Regards, MB
I tend to agree - Monte Carlo allows for hundreds or thousand of scenarios, based on assumptions.

The actual performance is just that: the actual performance.

What *might* be useful is to compare the actual with the assumptions.
If the recent history is well outside of what the monte carlo assumptions,
then perhaps the assumptions could/should be modified.

GuyFromGeorgia
Posts: 139
Joined: Sat Dec 29, 2012 12:16 pm

Re: Testing validity of Monte Carlo Simulators

Post by GuyFromGeorgia » Wed Sep 26, 2018 1:54 pm

One thing to keep in mind is that Monte Carlo simulations are used to provide confidence bands. You are 95% likely to live out x # of years with a Y% safe withdrawal rate. probability does not equal actuality. In engineering you are taught fairly early that everything has a tolerance and everything has a breaking point. A typical piece of wood might be able to withstand 300 pounds of force before breaking. The best wood in this bundle can withstand 400 pounds before breaking. But the worst wood in the bundle is only able to withstand 200 pounds of force. For a nightstand or bookcase maybe it doesn't matter that it has a design requirement of being rated to 250 pounds. 1) you're unlike to have worst case wood, 2) you're unlikely to stress the product to that point anyways, and 3) if you do have failure, it is fairly inconsequential.

Here's the thing, with your retirement: 1) you don't know what the market will be, 2) you don't know what your health or other speed bumps in the road will be, and 3) running out of money the absolute worst.

The other thing is that most people don't have a strong enough understanding of what confidence bands mean. In statistics I believe they are called confidence intervals. There is a lot of math that goes into it. Fundamentally, the higher your probability of being correct means the wider your window is. For example, if I ask you how many pretzels are in a standard bag from the chip aisle, provide me a 95% confidence band. Go ahead, do the experiment. Now give me 90% band. The 90% band should be narrower than the 95% because you have to catch fewer outliers. Now I have no clue how many pretzels are in there, so I'm going so say between 100-500. If the actual number is in there, I win. If it is actually 98 or 502, close enough. Someone else might have guessed 200-300. Not a bad guess. If it lands at 198 or 302, close enough. If it lands at 100 or 400, you end up being pretty far off from a statistical perspective. Ok, we can all somewhat relate to chips. What's your 95% confidence band for how many oranges are grown in the state of California? As you can imagine, this is quite complicated, as is retirement savings. That is why I like how Bogleheads have a conservative approach. 3% SWR instead of 4%. Focus on spending levels, not income levels as a metric prior to retirement. Good luck!

dknightd
Posts: 1878
Joined: Wed Mar 07, 2018 11:57 am

Re: Testing validity of Monte Carlo Simulators

Post by dknightd » Wed Sep 26, 2018 3:00 pm

The reality is that Monte Carlo simulations are always open to interpretation and error.
Most use historical returns to provide average returns and standard deviation of returns.
Then a random number is picked for each year of the simulation.
At least that is my understanding.

I'm not sure returns from one year to the next are completely random. But I think that is what Monte Carlo simulation assume (I could be wrong!)

Basing chance of success using historical returns (i.e. what happened if you retire in year x, or y) may or may not be better.

Both methods assume history is a guide to future returns. One method assumes each year is random. The other method assumes that previous patterns will occur again.

I'd say both methods are useful. Neither is perfect.

cheezit
Posts: 261
Joined: Sat Jul 14, 2018 7:28 pm

Re: Testing validity of Monte Carlo Simulators

Post by cheezit » Wed Sep 26, 2018 3:48 pm

vineviz wrote:
Wed Sep 26, 2018 12:04 pm
Pretty much by definition, the results of a Monte Carlo simulator will be perfectly valid (assuming it is competently built).

They are not predicting the future, after all, but are merely calculating a set of mathematical probabilities GIVEN a certain set of inputs.
Sed contra, the probabilities a Monte Carlo simulator spits out will only be accurate if the assumptions baked into the model are true. If any of the assumptions are wrong, the probability distribution function it generates will be wrong.

Eg. the simulator may assume that monthly stock market returns can be accurately modeled as an independent Gaussian random variable with a mean equal to the average of the last 40 years' worth of monthly market returns and a standard deviation equal to the standard deviation of the last 40 years' worth of monthly market returns. If it turns out that monthly returns aren't independent, or that they aren't well-modeled by a Gaussian, the probabilities and confidence bands that you get from the simulation will be wrong. Furthermore, even if the basic model is right and monthly returns are indeed well modeled as Gaussian independent random variables, the output will still be bad if the sample period chosen to pick the random variable's mean and SD turns out to be non-representative.

There are also various sources of out-of-model risk that are not represented by the simulator, but those are probably beyond the scope of this forum.

This isn't to say that Monte Carlo simulations aren't useful (especially since central limit theorem makes the assumption of normally distributed returns a good default!), but rather that we should be cognizant of their limitations when using them, just as we should use tools like backtesting with our "eyes open" so to speak.

Jordan4FI
Posts: 190
Joined: Mon May 28, 2018 11:00 am
Location: Honduras

Re: Testing validity of Monte Carlo Simulators

Post by Jordan4FI » Wed Sep 26, 2018 3:54 pm

Not much help with this post, but there is a new monte carlo simulaiton out there that was shared in the ChooseFI community on facebook. I tried to find it again, but no luck at this time. It also added in the probability of dying using trends of death at that age from recent history. Most the time I input my numbers/expected numbers, I had a greater chance of dying then running out of money. That is a huge part that many forget about. We just seem to assume that since we put in the calculator that we will live to 80/85/90/95 that it will be so... For most of us, that will not be the case, so 100/95/90/85% success could very well be over estimating...

If I find that site for the calculator, I will try to get back here and share..

afan
Posts: 4408
Joined: Sun Jul 25, 2010 4:01 pm

Re: Testing validity of Monte Carlo Simulators

Post by afan » Wed Sep 26, 2018 4:20 pm

The simulations usually are based on an assumption of a normal distribution. Stock returns are not normal. With more work one can alter the input distribution. Just as one does not know the mean return for the future, one does not know the future distribution of returns. So a MC simulation will return results based on the inputs. For that they are very reliable. But one has no way of knowing whether those inputs are correct.

For planning it gives an estimate of the range of possible outcomes, for the inputs used.
We don't know how to beat the market on a risk-adjusted basis, and we don't know anyone that does know either | --Swedroe | We assume that markets are efficient, that prices are right | --Fama

wolf359
Posts: 1882
Joined: Sun Mar 15, 2015 8:47 am

Re: Testing validity of Monte Carlo Simulators

Post by wolf359 » Wed Sep 26, 2018 4:40 pm

There's a computing concept called, "Garbage in, garbage out."

If the assumptions you put in are flawed, then the simulation you build based on those assumptions are also flawed. You have no way of know if the assumptions are flawed until you live them. Black swan events that nobody could predict can and will happen.

Life insurance is priced according to actuarial tables. Using the law of large numbers, life insurance companies can predict the lifespan of a population based on specific risk factors. They can't predict your specific lifespan, but they know how long someone of your age, health, and lifestyle will generally live.

I treat Monte Carlo simulators as the retirement version of actuarial tables. Based on the assumptions we have of how the markets will behave, we run a large number of simulations and come up with predictions. The results should still be used as an estimate, however. 100% probability in a Monte Carlo simulator does not mean guaranteed. Something outside your assumptions could occur. Fraud, elder abuse, a communist revolution that nationalizes all assets, being on the losing end of a general war, a larger than normal natural disaster, or an unprecedented economic catastrophe could occur. You always have to be flexible.

Treat Monte Carlo simulators with a grain of salt. It can help you build a solid plan, but there are no guarantees.

User avatar
tennisplyr
Posts: 2358
Joined: Tue Jan 28, 2014 1:53 pm
Location: Sarasota, FL

Re: Testing validity of Monte Carlo Simulators

Post by tennisplyr » Thu Sep 27, 2018 6:49 am

Since these simulators are always based on past performance and any one of us could always be an outlyer, not sure what I could definitively learn from this exercise.
Those who move forward with a happy spirit will find that things always work out.

Shallowpockets
Posts: 1432
Joined: Fri Nov 20, 2015 10:26 am

Re: Testing validity of Monte Carlo Simulators

Post by Shallowpockets » Thu Sep 27, 2018 7:19 am

People seem very skeptical of MC simulations. Garbage in, garbage out. Past results mean nothing going forward.
Do you really believe that there is no effort on the part of those that create the financial MC at, say Fidelity, are wholly lacking in finesse and integrity that they create an MC model that is wrong? That their model does not have enough algorithms or incorrect data, or purposely skewed data to achieve a result to trick you into believing what comes out the other end?
Is there an MC simulation that would satisfy a BH based on index funds! Is that possible?
Or because past stock market results are no indicator of future results, are all models essentially just an exercise in curiosity?

carolinaman
Posts: 3889
Joined: Wed Dec 28, 2011 9:56 am
Location: North Carolina

Re: Testing validity of Monte Carlo Simulators

Post by carolinaman » Thu Sep 27, 2018 8:35 am

wolf359 wrote:
Wed Sep 26, 2018 4:40 pm
There's a computing concept called, "Garbage in, garbage out."

If the assumptions you put in are flawed, then the simulation you build based on those assumptions are also flawed. You have no way of know if the assumptions are flawed until you live them. Black swan events that nobody could predict can and will happen.

Life insurance is priced according to actuarial tables. Using the law of large numbers, life insurance companies can predict the lifespan of a population based on specific risk factors. They can't predict your specific lifespan, but they know how long someone of your age, health, and lifestyle will generally live.

I treat Monte Carlo simulators as the retirement version of actuarial tables. Based on the assumptions we have of how the markets will behave, we run a large number of simulations and come up with predictions. The results should still be used as an estimate, however. 100% probability in a Monte Carlo simulator does not mean guaranteed. Something outside your assumptions could occur. Fraud, elder abuse, a communist revolution that nationalizes all assets, being on the losing end of a general war, a larger than normal natural disaster, or an unprecedented economic catastrophe could occur. You always have to be flexible.

Treat Monte Carlo simulators with a grain of salt. It can help you build a solid plan, but there are no guarantees.
+1. Amen to GIGO.

Monte Carlo simulators use historical averages. Most "experts" today predict below average returns for the next X years because market valuations are so high. If that proves true, returns over the next X years are likely to be below average. Depending on how long that is, your results may well be below average.

I have been retired 8 years. The biggest factor in our retirement has been our spending in retirement. We have actually spent less than I thought but we have also had some major unplanned and unexpected expenses: major dental work, home maintenance and repairs and others. We have also given some money to our children and also one grandchild to help with her college tuition.

The market has been great so far and has exceeded our expectations. Conversely, I have a non COLA pension which represents a major part of our retirement income. Historically it had kept pace with inflation due to investment income but I have had less than 1% COLA since retirement and expect none going forward. My original assumption in planning was that it would keep pace with inflation. I do not expect any pension COLAs now and that is a big bogey regarding my original planning.

My major point is that there are other things that can negatively affect your retirement aside from the MC simulation.

mindbogle
Posts: 140
Joined: Sun Feb 10, 2013 11:28 am

Re: Testing validity of Monte Carlo Simulators

Post by mindbogle » Thu Sep 27, 2018 9:16 am

tennisplyr wrote:
Thu Sep 27, 2018 6:49 am
Since these simulators are always based on past performance and any one of us could always be an outlyer, not sure what I could definitively learn from this exercise.
There is nothing inherent about MC analysis that requires a basis on past performance. Since MC simulation is concerned with understanding the range of FUTURE outcomes, forward-looking asset return distributions, cross-correlations, and depending on how sophisticated the model, serial correlations, are needed. The user should do his/her homework to know if he/she is comfortable with the assumptions being used in the model. Past performance is SOMETIMES used as a proxy for future performance. Would you be comfortable with that assumption? If not, then you probably won't be comfortable with a model that uses it. As always, do your due diligence.

The fact that the future is uncertain and that any one of us could realize an "outlier" future path is the very reason why many find MC analysis useful. What are the chances that I will get served an outlier? What is the probability that I will get served an acceptable path? Should I change my current plan to increase my probability of a successful path? There is no prediction magic in MC analysis, and again, the conclusions you make from it are only as good as the reasonableness of the model and supplied inputs.

MB

Admiral
Posts: 2488
Joined: Mon Oct 27, 2014 12:35 pm

Re: Testing validity of Monte Carlo Simulators

Post by Admiral » Thu Sep 27, 2018 10:39 am

Use a MC to see a RANGE of possible outcomes. That's the best way to look at results. It's not predicting the future, just showing what MAY happen under various conditions. Change your inputs (high and low) and then err on the side of safety.

User avatar
snackdog
Posts: 806
Joined: Wed Nov 12, 2014 4:57 am

Re: Testing validity of Monte Carlo Simulators

Post by snackdog » Thu Sep 27, 2018 11:07 am

A more insightful question might be: for those who have been retired more than 20 years and had a plan, how far has actual deviated from plan?

I think you will find most did not have a plan or the plan was simply to spend their monthly pension plus SS and that is what they did. If finances got tight, they sold or mortgaged real estate or borrowed from family.

For the few who had a plan based on investments, you will find some who got lucky or unlucky with investments, some who got lucky or unlucky with major expenses (healthcare, lawsuits, family), variance in inheritance windfalls, variance in property valuation performance and of course variance in longevity.

Just think about the people you know personally. My parents had heaps of hardwired financial and healthcare benefits from government careers, plus investments on the side, plus my mother still chooses to work at 83 because she loves it.

My aunt's husband died and left her a bit strapped so my grandfather bought her a house. He bought his brother a couple cars. He lived off a pension and quite a few savings bonds he accumulated.

Some people I know who divorced late in life ended up needing to work into their 70s to make ends meet. They have frugal lives these days but they seem happy.

I know a couple people who retired with lots of investments in their 40s and never worked again. I also know several who were back at work full time within 5-8 years and are still at it 20 years later, either out of boredom or financial necessity.

I do know a couple people who went from lavish millionaire lifestyles to destitute by over-spending. One died and one is trying to sell real estate.

GuyFromGeorgia
Posts: 139
Joined: Sat Dec 29, 2012 12:16 pm

Re: Testing validity of Monte Carlo Simulators

Post by GuyFromGeorgia » Thu Sep 27, 2018 11:39 am

Shallowpockets wrote:
Thu Sep 27, 2018 7:19 am
People seem very skeptical of MC simulations. Garbage in, garbage out. Past results mean nothing going forward.
Do you really believe that there is no effort on the part of those that create the financial MC at, say Fidelity, are wholly lacking in finesse and integrity that they create an MC model that is wrong? That their model does not have enough algorithms or incorrect data, or purposely skewed data to achieve a result to trick you into believing what comes out the other end?
Is there an MC simulation that would satisfy a BH based on index funds! Is that possible?
Or because past stock market results are no indicator of future results, are all models essentially just an exercise in curiosity?
Not so much that people are skeptical of MC, but the data sets being used. More than likely the future stock market gains will be about 8% per year. But that only gives you a 51% probability of it being correct (as I stated "more than likely"). To reach 90% or 95% confidence in your model you have to include a lot more data. Where does that data come from? What assumptions are you making? Are there any data sets that exist today that predict the future? That's the point. Nobody is saying that the financial people at Fidelity are incompetent or purposefully being deceitful. We can however, say that no financial advisor can predict the future and some set of assumptions have to be made. When a fallible human makes assumptions, there is risk that maybe they got something wrong.

User avatar
Mursili
Posts: 103
Joined: Tue May 15, 2012 8:32 pm

Re: Testing validity of Monte Carlo Simulators

Post by Mursili » Thu Sep 27, 2018 11:44 am

Shallowpockets wrote:
Thu Sep 27, 2018 7:19 am
People seem very skeptical of MC simulations. Garbage in, garbage out. Past results mean nothing going forward.
Do you really believe that there is no effort on the part of those that create the financial MC at, say Fidelity, are wholly lacking in finesse and integrity that they create an MC model that is wrong?
I firmly believe that the creators of the MC models believe in their models. The model results are "correct" given their underlying assumptions. The problem that I see people on this forum discussing is what happens when those assumptions are wrong. We need to question the very nature of the model if we want to fully understand the results. There is not a sufficient amount of that in our society (in my opinion, of course).

Some would say one of the proximate causes of the 2008 financial meltdown was believing in a model - apparently a quite elegant model - that showed that you could bundle securities and make a lower-risk security. It turns out that one of the underlying assumptions of that model was not actually the situation that played out.
When it comes to havoc, no one wreaks like me! - Dr. Heinz Doofenshmirtz

Admiral
Posts: 2488
Joined: Mon Oct 27, 2014 12:35 pm

Re: Testing validity of Monte Carlo Simulators

Post by Admiral » Thu Sep 27, 2018 11:53 am

I think that the problem (if it can be said to be a "problem") is that MC simulators don't account for human behavior and psychology.

Just because a model says that one can safely withdraw 4.5% in perpetuity and not deplete a portfolio doesn't mean that Joe Retire who left the workforce in November, 2008 is going to do that. He would likely make emotion-driven choices about what he thinks is a wise and sustainable withdrawal strategy (as we all would.)

So, nobody is going to blindly follow some computer model about what is sustainable. They will do whatever meets with their level of comfort.

mindbogle
Posts: 140
Joined: Sun Feb 10, 2013 11:28 am

Re: Testing validity of Monte Carlo Simulators

Post by mindbogle » Thu Sep 27, 2018 12:43 pm

Admiral wrote:
Thu Sep 27, 2018 11:53 am
I think that the problem (if it can be said to be a "problem") is that MC simulators don't account for human behavior and psychology.

Just because a model says that one can safely withdraw 4.5% in perpetuity and not deplete a portfolio doesn't mean that Joe Retire who left the workforce in November, 2008 is going to do that. He would likely make emotion-driven choices about what he thinks is a wise and sustainable withdrawal strategy (as we all would.)

So, nobody is going to blindly follow some computer model about what is sustainable. They will do whatever meets with their level of comfort.
Yep - I don't think it would be wise to blindly follow a computer model neither. That would be a problem with the user of such a model, not the model itself.

I am comfortably retired and have a high degree of confidence that my plan is sustainable. When I was evaluating my decision to retire or not, I researched market history, built my own MC models, performed scenario analysis (picked a few bad luck what-if scenarios and see if I could afford to immunize against those), and read a few books on behavioral economics to better understand and avoid my own innate bad instincts when it comes to finances. MC analysis was only one insightful tool that helped to move me towards a decision that I am comfortable with.

MB

Shallowpockets
Posts: 1432
Joined: Fri Nov 20, 2015 10:26 am

Re: Testing validity of Monte Carlo Simulators

Post by Shallowpockets » Thu Sep 27, 2018 12:46 pm

There is skepticism. No doubt about it. People dismiss any model that tries to give some sort of forecast to your money. Data bad, what data, forgot some data, data insufficient. That is skepticism.
So, I pose this question. What do you use to help you forecast your financial future? Remember, others have dismissed the past as not indicative of the future. They have dismissed validity because you cannot account for black swan events, or upsets in your own life. So where are we then? Nothing seems to be all inclusive of everything possible. All is simply assumptions and you cannot rely on that. Worse, they are other peoples assumptions. Even if they are professionals.

Other than the BH three fund there are no absolutes (tongue in cheek). Therefore all going forward is uncertain and there is really no point in trying to project out anything. Anything could happen not accounted for in any scenario. Bad data, poor projections, black swans, events we have no control over.
We have absolutely nothing. We must go forward based on intuition. No plans should be made. There is nothing but what we personally believe in as our data. How objective is that?

User avatar
Mursili
Posts: 103
Joined: Tue May 15, 2012 8:32 pm

Re: Testing validity of Monte Carlo Simulators

Post by Mursili » Thu Sep 27, 2018 3:28 pm

If we knew the answer, then the traffic on this forum would be much smaller as people would just point out the "right" answer.

Life is more complicated than that. People can be skeptical and still use information from the past - as long as they understand that that is not how the future will play out, but it is the best guess we have today. You will have to remain cognizant of the world and manage your situation as it evolves into the future.
When it comes to havoc, no one wreaks like me! - Dr. Heinz Doofenshmirtz

User avatar
Hyperborea
Posts: 824
Joined: Sat Apr 15, 2017 10:31 am
Location: Japan

Re: Testing validity of Monte Carlo Simulators

Post by Hyperborea » Thu Sep 27, 2018 11:37 pm

Shallowpockets wrote:
Thu Sep 27, 2018 12:46 pm
There is skepticism. No doubt about it. People dismiss any model that tries to give some sort of forecast to your money. Data bad, what data, forgot some data, data insufficient. That is skepticism.
So, I pose this question. What do you use to help you forecast your financial future?
I don't have any problem with the data in most of the MC simulations. The problem that I have is with the model in general. It assumes far too many things which don't appear to be true about the markets - time independence (this year's returns have no relation to the sequence leading to this year), asset independence (inflation, bond returns, equity returns have no relation to each other), and the shape of the curve used for the returns. Using a MC sim to predict your finances is about as useful as using physics models that assume perfectly elastic collisions and frictionless planes in doing accident reconstruction.

The real world data is probably the best that we've got to use and probably always will be. Outside of serious black swan events where your financial assets wouldn't be of use anyways, picking a reasonable allocation and withdrawal rate that survives for the duration you are planning for with historical data is probably the best that we can do.
It’s hard to win an argument with a smart person, it's damn near impossible to win an argument with a stupid person. - Bill Murray

mindbogle
Posts: 140
Joined: Sun Feb 10, 2013 11:28 am

Re: Testing validity of Monte Carlo Simulators

Post by mindbogle » Fri Sep 28, 2018 9:12 am

Hyperborea wrote:
Thu Sep 27, 2018 11:37 pm
I don't have any problem with the data in most of the MC simulations. The problem that I have is with the model in general. It assumes far too many things which don't appear to be true about the markets - time independence (this year's returns have no relation to the sequence leading to this year), asset independence (inflation, bond returns, equity returns have no relation to each other), and the shape of the curve used for the returns. Using a MC sim to predict your finances is about as useful as using physics models that assume perfectly elastic collisions and frictionless planes in doing accident reconstruction.
Most of the "problems" that you mention are not inherent in MC models. Many if not most retirement MC models assume some degree of asset return cross-correlation. Some, including the ones that I have built, assume a degree of asset return time dependence (serial correlation). And there are some, including the ones that I have built, that assume non-normal return distributions. So the MC model in "general" is capable of handling all of that.

But if you are looking for "truth", then MC won't get you there. It is a model, just like any physics model. Its not truth.

And as has been said many times before in this thread, MC models in particular are not prediction models. They are used to better understand uncertainty and risk and are used widely in many industries for improving descision quality.

MB

protagonist
Posts: 6049
Joined: Sun Dec 26, 2010 12:47 pm

Re: Testing validity of Monte Carlo Simulators

Post by protagonist » Fri Sep 28, 2018 9:47 am

vineviz wrote:
Wed Sep 26, 2018 12:04 pm

Those inputs (expected returns, expected variance, and expected correlation) will determine the output. Unfortunately, it is up to the investor to use they judgement abut what those inputs should be.
Correct. And when it comes to predicting what the world will look like in 20 or 30 or (gasp!) 50 or more years, the investor is clueless. What inputs should you use?

If you told me that my life (or even just my finances) would be what it is today 30 years ago I would have laughed hysterically. 50 years ago I would have written you off as a nut job.

Which is why I would say, don't waste your time with simulators, unless you just want to stoke the flames of your worst fears. You would do just as well to go to fortune tellers- maybe better, since they will probably accent the positive and thus you will avoid climbing a wall of worry the rest of your life. Just live and invest wisely (as many here do), and leave the rest up to fate. It's the best you can do anyway. The future is....well....the future.

Admiral
Posts: 2488
Joined: Mon Oct 27, 2014 12:35 pm

Re: Testing validity of Monte Carlo Simulators

Post by Admiral » Fri Sep 28, 2018 11:07 am

protagonist wrote:
Fri Sep 28, 2018 9:47 am
vineviz wrote:
Wed Sep 26, 2018 12:04 pm

Those inputs (expected returns, expected variance, and expected correlation) will determine the output. Unfortunately, it is up to the investor to use they judgement abut what those inputs should be.
Correct. And when it comes to predicting what the world will look like in 20 or 30 or (gasp!) 50 or more years, the investor is clueless. What inputs should you use?

If you told me that my life (or even just my finances) would be what it is today 30 years ago I would have laughed hysterically. 50 years ago I would have written you off as a nut job.

Which is why I would say, don't waste your time with simulators, unless you just want to stoke the flames of your worst fears. You would do just as well to go to fortune tellers- maybe better, since they will probably accent the positive and thus you will avoid climbing a wall of worry the rest of your life. Just live and invest wisely (as many here do), and leave the rest up to fate. It's the best you can do anyway. The future is....well....the future.
I actually think this is totally wrong. OK maybe not totally: there are black swan-type events that are beyond the range of predictability (meteor, nuclear annihilation, etc).

There's a wide chasm between predicting the future (your "fortune teller") and planning for the future. To plan for the future, what tool do we have? We have the past. That's it. That doesn't mean the future will be just like the past, but the future, like the past, will have a range of outcomes. If one plans for a future within a previous general range of outcomes, that strikes me as prudent.

Some here plan on a worst-case scenario of 2% returns for the next 30 years. That's an eventuality that is possible but highly unlikely. Others say 8%, because that's the S&P average in the record books. That's optimistic, and also possible, but also probably unlikely. Returns are likely to be somewhere above inflation and below a top-end estimate like 8%. Good MC sims will offer a median value, a low 10%, and a high 10%. It's up to the user to invest based on this range of outcomes and what is acceptable in terms of their needs and wants.

MathWizard
Posts: 3636
Joined: Tue Jul 26, 2011 1:35 pm

Re: Testing validity of Monte Carlo Simulators

Post by MathWizard » Fri Sep 28, 2018 11:18 am

I am doubtful of the extremes, but if you are looking at a 90% confidence window, that may
not matter.

The problem with using random numbers is that there should be no "memory" in the numbers.

For example, people often assume that if a fair die has rolled a 6 nine times in a row that the
probability of the next roll being a 6 is less than 1/6th. That is incorrect. If it is a fair die, the
probability remains 1/6th on each roll, independent of past results.

I do not believe that returns are random, or even that the deviation from an average rate of return
is random. That is, I believe both in momentum and in reversion to the mean. I believe in momentum because of
human behavior "This is stock is going up, I should get into it."
I believe in reversion to the mean, because if not, it is not the mean. But that means that their is a higher probability
of lower prices when returns are above the mean, and a higher probability of returns when returns are below the mean.

mindbogle
Posts: 140
Joined: Sun Feb 10, 2013 11:28 am

Re: Testing validity of Monte Carlo Simulators

Post by mindbogle » Fri Sep 28, 2018 11:28 am

protagonist wrote:
Fri Sep 28, 2018 9:47 am
Which is why I would say, don't waste your time with simulators, unless you just want to stoke the flames of your worst fears. You would do just as well to go to fortune tellers- maybe better, since they will probably accent the positive and thus you will avoid climbing a wall of worry the rest of your life. Just live and invest wisely (as many here do), and leave the rest up to fate. It's the best you can do anyway. The future is....well....the future.
MC analysis is used to help inform decisions. In this context, the decision might be "can I retire now?", or "how much longer do I have to work to retire with a high degree of confidence?", or "how much investment risk can or should I take to achieve my goals?". Its only one tool that could be used to help inform these. Again, its not a predication tool, its not a truth box, and its usefulness is only as good as the reasonableness of input parameters and assumptions used for model dynamics. None of these caveats invalidate MC analysis as a useful tool.

People have different financial circumstances, views about the future, and aversions to risk. Some, in order to feel confident in making decisions, need to do alot of research and study, and maybe tinkering, and others can be confident with alot less, such as advice from a trusted advisor, or following a generally accepted Boglehead rule of thumb or principle. I would not blanket recommend that everyone needs to use MC analysis to make high quality and high confidence decisions. But it can be useful for some (and definitely more so than a fortune teller!). Its just a tool - its not a panacea.

I used MC analysis along several other tools to help inform my retirement timing and investment strategy. I do not know with 100% confidence that my plan will ultimately result in a successful outcome, but MC analysis was instrumental in giving me comfortable margin of confidence. No wall of worry here.
MathWizard wrote:
Fri Sep 28, 2018 11:18 am
The problem with using random numbers is that there should be no "memory" in the numbers.
The dynamic of reversion to the mean in simulation can be modeled with "random numbers", several ways. Imposing a negative serial correlation on the simulated return sequences is the simplest.

MB

protagonist
Posts: 6049
Joined: Sun Dec 26, 2010 12:47 pm

Re: Testing validity of Monte Carlo Simulators

Post by protagonist » Sun Sep 30, 2018 10:41 am

wolf359 wrote:
Wed Sep 26, 2018 4:40 pm

Life insurance is priced according to actuarial tables. Using the law of large numbers, life insurance companies can predict the lifespan of a population based on specific risk factors. They can't predict your specific lifespan, but they know how long someone of your age, health, and lifestyle will generally live.

I treat Monte Carlo simulators as the retirement version of actuarial tables.
The difference is, that with actuarial tables, we actually know things about human lifespan and its limits. We also have large amounts of strong, scientifically and statistically verifiable information about what diseases affect humans at what ages, chances of recovery, survival rates, etc. We know, for instance, the chance of a 66 year old nonsmoking male developing lung cancer within the next year, or 10 years, and if he does, the chances of him surviving for 1 year, 5 years, 10 years, 30 years (including standard deviations)

Yes, there may be breakthroughs, nuclear war, whatever, which we cannot predict....

But can we predict the chances of a 66 year old experiencing a major market crash within the next 1, 5, 10 or 20 years, and if so, how bad it will be, and chances of recovery in 1 year, 10 years, even 30 years? If you know, please give me the numbers along with standard deviations, because I am 66 and I would really like to know.

2015
Posts: 2906
Joined: Mon Feb 10, 2014 2:32 pm

Re: Testing validity of Monte Carlo Simulators

Post by 2015 » Sun Sep 30, 2018 11:29 am

I decided I didn't want to deal with what I viewed as all of the nonsense surrounding the 4% rule, spending guidelines, guardrails, handicap entrances, etc., relying on the "accuracy, reliability, GIGO, GoGo, NoGo, GoWithTheFlow affect, etc." of any calculator. Less appealing was having to be enslaved to reading every new thread, publication, chart, graph, technical analysis, or discharge that seems to come out daily on these matters (I wanted to get a life).

A very strong influence on my thinking is everything I've studied and read on behavioral factors being far superior to determining investor outcomes (for better or worse) than any clever portfolio or investment strategy. Also relevant for me is the real life behavioral impact the 2008 economic impact had on investors. Reality (as in how people really act, versus discussions regarding how they should or do act) has a much stronger punch to the face than the most clever of theories. Finally, my involved reading on the impact of luck in life, that investing is mostly about Luck, and not about Skill, made me want to take action to bring my financial life from the Luck to the Skill side of the Luck/Skill Outcomes Continuum. I wanted to take my financial life from existing in an open, complex, adaptive system with unknown, unknowable, and therefore uncontrollable inputs, to a closed system that was far more knowable/controllable. Like taking investing from a geological to a physics system. In this way, I could mimic Buffett/Munger and make all financial decisions based strictly on Microeconomics, virtually devoid of all Macroeconomic considerations.

For these reasons, this year I completed transitioning to liability matching, with all of my essential/discretionary expenses covered for the the rest of my life. Outside of my liability matched PF, I gauge my Risk Portfolio's performance against Fidelity's significantly below market conditions, more out of curiosity than anything else. My thinking is still evolving on whether I'll use longinvest's most excellent VPW tool to spend more money out of the Risk Portfolio on toys, lap dances, etc., as I'm not sure I'll need to. We'll see.

I can't describe the level of peace of mind liability matching has brought into my life, but I'm in the decumulation phase, so if anyone else is in the accumulation phase I can understand why their thinking might be different.

Post Reply