Is backtesting worthless? Are there ANY insights you can get from it?

Discuss all general (i.e. non-personal) investing questions and issues, investing news, and theory.
Topic Author
JustinR
Posts: 1147
Joined: Tue Apr 27, 2010 11:43 pm

Is backtesting worthless? Are there ANY insights you can get from it?

Post by JustinR » Fri Mar 30, 2018 4:04 am

Whenever backtesting is brought up people remind us that the past performance isn't indicative of future results.

Doesn't that mean that backtesting is worthless, then?

So what's the purpose of backtesting? Does it have any value? What are some actual insights you can get from backtesting an allocation?

jts
Posts: 8
Joined: Wed Mar 07, 2018 5:41 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by jts » Fri Mar 30, 2018 4:50 am

We know that the future is unpredictable but backtesting will allow insight into the vulnerabilities or volatile nature of some holdings or strategies. Improper use or over reliance can be dangerous. There are still fundamental considerations that have to be made, diversification, risk tolerance and all of the areas that should normally be considered. For instance, the losses of the Total Stock fund in 2000, 2001 & 2002 then 08 right after it recovered, kept this fund with little or no gain for 10-12 years. That does not mean it should not be considered in a diversified portfolio but it does show a potential weakness that should be evaluated as part of your overall assets. If you are very sensitive to declines, then that should be considered as a possibility in your allocations. This is the same as when we consider sequence of return losses at or near retirement or use standard deviation measures or any one of many considerations from the past.

sambb
Posts: 2688
Joined: Sun Mar 10, 2013 3:31 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by sambb » Fri Mar 30, 2018 6:28 am

Yes it is worthless. Look at the japanese market for some examples.
Backtesting is for comfort. No one can predict the future.

Dandy
Posts: 6056
Joined: Sun Apr 25, 2010 7:42 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by Dandy » Fri Mar 30, 2018 6:49 am

I think backtesting can be a useful tool but it can easily be misused or misunderstood. How far back are you backtesting? Are you including the great depression when there was about a 90% drop in equities? Does going back that far make sense? Are you looking at how a 60/40 allocation has performed over the last 30 years? If so, remember there has been a 30 year bull market in bonds that can't easily be repeated in the intermediate term. Does that mean intermediate bonds may not be as strong in providing portfolio stability? Does that matter?

I think it is useful to see some of the benefits of diversity e.g. how adding/changing fixed income and international exposure can work to improve overall results. The operative word is "can". The future may track close to the past patterns or not so you need to avoid a set it and forget it attitude. Anyone think there would be many countries experiencing negative interest rates? Backtesting helps determine what is a reasonable approach not a certain one.

User avatar
fortyofforty
Posts: 1732
Joined: Wed Mar 31, 2010 12:33 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by fortyofforty » Fri Mar 30, 2018 6:57 am

Past performance is no guarantee of future results. I think the key is "guarantee". You can certainly get some indication of future results from past results. If not, we would not suspect stocks will provide us higher returns than bonds. We would not suspect emerging market stocks will be more volatile than large cap value stocks. We would not be able to guess that bonds will lose value as interest rates move up in response to the actions of the Federal Reserve.

We really have nothing else, in many cases. We have to look at the performance of bonds during periods of rising Fed Funds rates, for one example, and infer that, if the Fed continues to raise rates, the underlying bonds held by our funds will decline in value. We have to look at the volatility of emerging market stocks in the past and infer that their volatility will continue to be greater than large cap value stocks.

After seeing the past, we create rules and justifications to match our observations. If they prove to be wrong, we invent new rules and new justifications, or declare that there was some unforeseen factor that influenced the outcome.
Indexing works, not because of magic, but because of math. | Diligentia. Vis. Celeritas. - Jeff Cooper | Original Vanguard Diehard

Call_Me_Op
Posts: 7426
Joined: Mon Sep 07, 2009 2:57 pm
Location: Milky Way

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by Call_Me_Op » Fri Mar 30, 2018 7:05 am

JustinR wrote:
Fri Mar 30, 2018 4:04 am
Whenever backtesting is brought up people remind us that the past performance isn't indicative of future results.
One needs to be careful in interpreting that statement. Of course it is not useless - but it can be misused. If backtesting is done over a sufficiently long time period - particularly if used to illustrate relative real returns of asset classes, it can be quite useful.
Best regards, -Op | | "In the middle of difficulty lies opportunity." Einstein

bigred77
Posts: 2028
Joined: Sat Jun 11, 2011 4:53 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by bigred77 » Fri Mar 30, 2018 7:09 am

I find backtesting to be very useful.

Just keep in mind it’s not appropriate to define absolute statements or rules based on backtesting.

“I’m using a 60/40 AA because historical data shows this allocation had superior risk adjusted returns” - Good

“I’m using a 60/40 AA because I need 4% annual real returns and this allocation always delivers at least 4% over 20 year periods. The historical data proves that” - Bad

scone
Posts: 1457
Joined: Wed Jul 11, 2012 4:46 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by scone » Fri Mar 30, 2018 7:16 am

I feel I've learned a great deal from backtesting, especially comparing the various asset types. Not just the obvious lessons that bonds and stocks behave very differently, but something about how these assets behave in combination with each other. Years ago, I read some of Bill Bernstein's work, where he showed that you can combine two volatile assets and get a less volatile result, which seemed so unlikely to me, I could hardly believe it. But backtesting proved that he is right.

Second, backtesting has taught me that similar allocations are so similar, there's no need to try for a "perfect" portfolio-- there is none, and the "imperfect" portfolios we have are good enough and fit for purpose. Tiny tweaks are often a waste of effort. Portfolio Visualizer has been especially valuable in this regard, because I can see the results of three asset allocation variations at once.

Backtesting also tells me something about new funds that come on the market. I can satisfy my curiosity, investigate them in depth, and see how they mix with my current funds. I can "try before I buy," and so far, I haven't bought any shiny new "toys!"

Despite all the problems with data, backtesting offers a way for me to experiment, to test out the assertions that people make on Bogleheads. This is especially valuable for me, since I'm not the sort of person who takes things on faith, and I seldom buy in to "arguments from authority." Overall, backtesting has made me a better investor: more confident, less inclined to doubt and regret, and much calmer in the face of the inevitable gyrations in the markets. It's also a lot of fun.

Thanks to the Bogleheads, and many others, for doing the work to make the backtesting data and software available. :beer
"My bond allocation is the amount of money that I cannot afford to lose." -- Taylor Larimore

User avatar
Watty
Posts: 18053
Joined: Wed Oct 10, 2007 3:55 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by Watty » Fri Mar 30, 2018 7:24 am

JustinR wrote:
Fri Mar 30, 2018 4:04 am
So what's the purpose of backtesting? Does it have any value? What are some actual insights you can get from backtesting an allocation?
It is subtle but there is a difference between testing something to see if it is a "good plan" or a "bad plan"

Backtesting is better at telling you when you have a "bad plan" that you should reconsider.

For example the 4% safe withdrawal studies use past performance for backtesting. This is still very useful since if you start with a 7% that would have been risky in the past so you might want to avoid doing that.

longinvest
Posts: 4085
Joined: Sat Aug 11, 2012 8:44 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by longinvest » Fri Mar 30, 2018 7:28 am

Backtesting is an awesome tool for finding counterexamples to false claims. Here's an example: viewtopic.php?f=10&t=226467&p=3510693&h ... x#p3510693

When it comes to predicting the future, backtesting is effectively worthless.
Bogleheads investment philosophy | One-ETF global balanced index portfolio | VPW

SGM
Posts: 3062
Joined: Wed Mar 23, 2011 4:46 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by SGM » Fri Mar 30, 2018 8:13 am

Back-testing can be used to prove anything you want to prove. Small changes in the dates or how the data is presented can change the message. I don't remember if it was a basic accounting or economy text book that had as a first assignment to check a newspaper for a graph and determine how the data was manipulated and what was the message the author was trying to present.

User avatar
czeckers
Posts: 1025
Joined: Thu May 17, 2007 3:49 pm
Location: Upstate NY

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by czeckers » Fri Mar 30, 2018 8:14 am

The primary usefulness of backtesting is to look at the performance of a specific portfolio under various market conditions.

For example, look at the inflationary period of 1972-1980, deflation 1930-1933 and prosperity of the 1990's. If you have a portfolio that behaves reasonably well under all of those circumstances, you have a winner.

On the other hand, the way many people use backtesting to optimize a portfolio based on past performance of very long time periods can be very dangerous because the results tend to be heavily skewed by recent performance and result in people chasing past performance.
The Espresso portfolio: | | 20% US Total Mkt, 20% Small Cap Value, 10% US REIT, 10% Developed Int'l, 10% EM, 30% Inter-term US Treas | | "A journey of a thousand miles begins with a single step."

rbaldini
Posts: 1074
Joined: Mon Mar 23, 2015 3:20 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by rbaldini » Fri Mar 30, 2018 8:38 am

If the future is *something* like the past (which most people believe), and you're very careful about how you do the statistics, then you can get something out of it.

If the past were completely irrelevant to the future, the study of history would be little more than an intellectual curiosity. Most of us think it's more than that.

User avatar
saltycaper
Posts: 2650
Joined: Thu Apr 24, 2014 8:47 pm
Location: The Tower

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by saltycaper » Fri Mar 30, 2018 9:17 am

JustinR wrote:
Fri Mar 30, 2018 4:04 am
Whenever backtesting is brought up people remind us that the past performance isn't indicative of future results.
It is not worthless, not even to the people who make this claim. When someone says this, oftentimes what they really mean is backtesting is not valid for supporting any argument other than their own.
Quod vitae sectabor iter?

User avatar
willthrill81
Posts: 15086
Joined: Thu Jan 26, 2017 3:17 pm
Location: USA

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by willthrill81 » Fri Mar 30, 2018 9:22 am

If backtesting were useless, how many would be invested in equities?
saltycaper wrote:
Fri Mar 30, 2018 9:17 am
JustinR wrote:
Fri Mar 30, 2018 4:04 am
Whenever backtesting is brought up people remind us that the past performance isn't indicative of future results.
It is not worthless, not even to the people who make this claim. When someone says this, oftentimes what they really mean is backtesting is not valid for supporting any argument other than their own.
:thumbsup
“It's a dangerous business, Frodo, going out your door. You step onto the road, and if you don't keep your feet, there's no knowing where you might be swept off to.” J.R.R. Tolkien,The Lord of the Rings

dbr
Posts: 31196
Joined: Sun Mar 04, 2007 9:50 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by dbr » Fri Mar 30, 2018 9:25 am

rbaldini wrote:
Fri Mar 30, 2018 8:38 am
If the future is *something* like the past (which most people believe), and you're very careful about how you do the statistics, then you can get something out of it.

If the past were completely irrelevant to the future, the study of history would be little more than an intellectual curiosity. Most of us think it's more than that.
This is the most correct response. Lot's can be learned about how investments might behave. Generally there is little to be gained trying to scheme ways to beat the market, time the market, or devise plans that try to be more certain than the nature of investments allows. That "past results . . ." phrase should never be uttered as it is usually meaningless and misapplied.

Marketman
Posts: 196
Joined: Sun Feb 04, 2018 2:24 pm
Location: Texas

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by Marketman » Fri Mar 30, 2018 9:25 am

I believe that the past gives good incites into future stock market risk but no so much into future return. To some degree, I believe this is true of bonds as well, but the case is not as strong as with stocks.

2015
Posts: 2906
Joined: Mon Feb 10, 2014 2:32 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by 2015 » Fri Mar 30, 2018 9:53 am

saltycaper wrote:
Fri Mar 30, 2018 9:17 am
JustinR wrote:
Fri Mar 30, 2018 4:04 am
Whenever backtesting is brought up people remind us that the past performance isn't indicative of future results.
It is not worthless, not even to the people who make this claim. When someone says this, oftentimes what they really mean is backtesting is not valid for supporting any argument other than their own.
+1
I've personally found backtesting good for illustrative purposes. Backtesting also helps a lot of bloggers pay their mortgage and get new clients.

alex_686
Posts: 5151
Joined: Mon Feb 09, 2015 2:39 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by alex_686 » Fri Mar 30, 2018 10:10 am

Backtesting is an exercise in statistics. Most people badly use statistics.

My personal crusade is people who use correlations over long periods of time or use it to examine markets during time of crisis. Correlation calculations assume constant volatility. Change the volatility over the period being tested and you get junk numbers. I can get a positive correlation on independent rolls of fair dice if I am allowed to change the volatility.

Many people who backtest assume that the future will be like the past and that things will mean revert. It does not. The are secular changes in the market. Tax law and investor preferences change, favoring stock buy backs over dividends in one era. I am confident that stocks have a higher rate of return and risk over bonds. However that level has changed from decade to decade. International stocks used to have a low correlation to domestic and offer good diversification. Now, International stocks are dominated by multinational corporations and face the same global factors and US multinational corporations.

You can test to see how the world worked in the past. You can find statistically significant driving factors. Then you need to use wisdom, logic, and skill to forecast how these factors will preform in the future.

User avatar
siamond
Posts: 5128
Joined: Mon May 28, 2012 5:50 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by siamond » Fri Mar 30, 2018 10:13 am

This is one of those topics where people thinking in black & white terms tend to miss the boat. For sure, there are many challenges with backtesting, and one can draw many hasty 'conclusions' that may not be valid. This doesn't mean that one should throw the 'worthless' baby out with the bathwater... :shock:

There are a couple of uses for backtesting which are straightforward:
- as previously mentioned, backtesting can disprove a false claim, this is very handy
=> as an example, carefully looking at past data can help to defuse the effects of recency bias, success bias, and more biases.
- backtesting can teach you a lot about past stressful situations, help you understand that a current situation (e.g. big bear or big bull market) isn't that exceptional, that staying the course did work in similar (or worse) conditions, hence providing much needed reinforcement in a time of duress.

Besides those straightforward points, the key is to understand that the past cannot predict the future (in black & white terms) while the future is very likely to rhyme with the past (shades of grey)... Once you get that, and exercise great care when interpreting results, then yes, backtesting can be very useful indeed. It's the usual premise of science, make a hypothesis validated with empirical data AND rooted on a durable common sense explanation, then shake the tree as much as possible, then either refine or discard the hypothesis...

Now, what is very frustrating is the lack of publicly available historical data, notably outside the US. Which severely limits the ability to shake the tree, i.e. to perform 'out of sample' testing of one hypothesis or another. I really wish academics would do a much better job of sharing available data sources in a public & open manner. Here is the historical data assembled by the Bogleheads community, and various tools allowing to analyze it for your personal use.

As the good Dr. Bernstein wisely said, "You simply can’t learn enough about market history".

alex_686
Posts: 5151
Joined: Mon Feb 09, 2015 2:39 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by alex_686 » Fri Mar 30, 2018 10:28 am

siamond wrote:
Fri Mar 30, 2018 10:13 am
Now, what is very frustrating is the lack of publicly available historical data, notably outside the US. Which severely limits the ability to shake the tree, i.e. to perform 'out of sample' testing of one hypothesis or another. I really wish academics would do a much better job of sharing available data sources in a public & open manner.
2 points on this.

First, the actually quality of historical data is pretty poor. Often we only have opening and closing prices, maybe highs and lows. Are these prices of high quality, with lots of trades around them? Was the high just 1 weird trade? Was it just one guy trading back and forth to manipulate prices?

Second, most of the data is privately generated and is copyrighted. S&P created and maintains its index. It will give away some stuff for free but the good stuff costs extra. Sigh.

User avatar
nisiprius
Advisory Board
Posts: 39737
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by nisiprius » Fri Mar 30, 2018 10:34 am

Backtesting is not worthless. It's like the question "is investing in the stock market gambling?" It has a gambling element. You can't ignore the gambling element. But it's not the same as "gambling."

The problem with backtesting is that it is worth much less than people are apt to think. But I feel it is quite worthwhile and that I get useful insights from it. What I don't get from it is a formula for beating the market with certainty, or a proof that my retirement savings will hit some desired number by some desired date.

Here are some problems with it.

1) Because financial data typically contains episodic runs of somewhat consistent behavior that can easily last five or ten years or more, followed by a fairly big change to a different run... any past average effectively contains the equivalent of a very small number of observations. "The stock market from 1926 through 2017" is not an average of 102 things, it's an average of maybe seven or eight things. This means that even under the best circumstances, the ability to predict, let's say, how your retirement savings will do over the next thirty years, based on the last thirty years of data, is very weak.

2) One of the things that backtesting does show is that the variability of past data is huge. I made a posting on Endpoint sensitivity of "historic" data in which I looked perhaps the most commonly cited number of all: "the historic past returns of the US stock market" over periods of many decades. That number, averaged over more than 70 years, can come out anywhere from 9% to 11% depending on fairly small changes in where you put the endpoints. The next time you see some statement like "During the 20th century, the stock market returned an average of 10.4%" remember that three places of precision is nonsense. Quite possibly the exact number from 1/1/1900 through 12/31/1999 was 10.4%, not 10.3% or 10.5%, but really you should be interpreting any number like that as having an error range of at least ±1%.

3) Everyone wants certainty. Unfortunately, backtesting is much more valid for demonstrating uncertainty than for finding certainty. Backtesting is most useful in disproving things. For example, if someone says "Dividend stocks provide downside protection," and backtesting shows that the Vanguard High Dividend Yield Stock Index Fund, VHDYX, fell more than the Total Stock Index Fund, VTSMX during 2008-2009, that disproves--or at leasts casts heavy doubt on the real-life usefulness--of the claim.
Image
Similarly, if someone says "when stocks go down, bonds usually go up," backtesting will show that, as stated, that statement is wrong (or simplistic or potentially misleading).

4) The usefulness of backtesting is only high if you, and your data, are close to being new to the testing process. There is a terrible problem with repeated backtesting, where you don't even keep a record of the number of tests you are making, and where each test is based on seeking to improve something that didn't come out just quite right in the last backtest. This problem has been recognized forever in science, although it is only recently that scientists have begun to appreciate how bad it is. Decades ago statistics professors warned against the perils of "torturing your data until it tells you what you want to hear." Now they talk about "p-hacking" which is pretty much the same thing.

If someone admits to having making repeated refinements of a strategy over time, and then demonstrates backtesting of the current version of the strategy, that backtesting is next to worthless. It might be worth looking at if they were quite careful to tell you how many times, and in which years, they had published a specific, testable strategy, and shown you the results of both the original version of the strategy and the later versions. I can't think offhand of anyone who has done this. (Maybe Bill Schultheis and the "Coffeehouse Portfolio?")

5) Backtesting is quite useful in lending credence to things which are, unfortunately, so basic that people don't really care much about having them confirmed. Let me give you an example. This is a valid example in the sense that I, personally, have never looked at this question before, nor have I seen anybody else's charts or statistics relating to it.

Hypothesis: stocks are more volatile than intermediate-term government bonds.

Hypothesis in form to be tested: if volatility is measured by standard deviation of annual returns, over successive ten-year periods starting with 1926-1935, stocks are represented by the SBBI Large-Company Stocks series, and bonds are represented by the SBBI Intermediate-Term Government Bond series, will almost always turn out to have been higher for stocks.

Actual results of a backtest which I've never done before, or ever seen done:

Code: Select all

Year	Large stocks	Int gov bonds
1926-1935	33.5%	3.6%
1936-1945	23.9%	1.6%
1946-1955	18.1%	1.2%
1956-1965	16.5%	4.2%
1966-1975	19.7%	4.8%
1976-1985	14.2%	8.7%
1986-1995	13.8%	6.8%
1996-2005	19.5%	5.2%
2006-2015	19.0%	5.0%
In this case, backtesting shows that large stocks had higher volatility than intermediate-term government bonds 9 decades out of 9.

An online calculator for the Mann-Whitney nonparametric U test says:
The U-value is 0. The critical value of U at p < .01 is 14. Therefore, the result is significant at p < .01.

The Z-Score is 3.53209. The p-value is .00021. The result is significant at p < .01.
In order for this to be valid, this has to be the only test I've ever made, and I have to have made it not really know anything about how it would turn out. In this case, although as always anything could happen, I feel that backtesting gives not only convincing evidence that stocks are more volatile than intermediate-term government bonds, I personally would be willing to place a fairly large bet that over the next ten years stocks will be more volatile than intermediate-term government.

(An aside to those who've taken statistics classes: have you ever noticed that financial writers are always showing "t-stats," i.e. the parametric Student's "t" value, and never the Mann-Whitney "u" value, which doesn't assume a normal distribution? I don't think I've ever seen a financial writer cite a nonparametric test, even though most people will state quite plainly that financial data do not follow a normal distribution. In other words... I think they cheat).
Last edited by nisiprius on Fri Mar 30, 2018 10:46 am, edited 1 time in total.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.

User avatar
siamond
Posts: 5128
Joined: Mon May 28, 2012 5:50 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by siamond » Fri Mar 30, 2018 10:46 am

alex_686 wrote:
Fri Mar 30, 2018 10:28 am
First, the actually quality of historical data is pretty poor. Often we only have opening and closing prices, maybe highs and lows. Are these prices of high quality, with lots of trades around them? Was the high just 1 weird trade? Was it just one guy trading back and forth to manipulate prices?

Second, most of the data is privately generated and is copyrighted. S&P created and maintains its index. It will give away some stuff for free but the good stuff costs extra. Sigh.
You're right, but... A LOT of stuff has also been generated by researchers without relying on indexers and copyrighted data, coming back to the data source of individual stocks, which is entirely public data. But then some of those researchers somehow lost their academic soul, and try to make a buck out of it, charging a price high enough to discourage most private individuals. I will not mention names, but I'm sure you can guess... Luckily, in the US, Prof. Shiller and Prof. Fama were much more open, and we should all be extremely grateful to them. And also to the Federal Reserve of St Louis, and its mountain of publicly available data.

User avatar
siamond
Posts: 5128
Joined: Mon May 28, 2012 5:50 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by siamond » Fri Mar 30, 2018 11:05 am

nisiprius wrote:
Fri Mar 30, 2018 10:34 am
Hypothesis: stocks are more volatile than intermediate-term government bonds. [...] Actual results of a backtest which I've never done before, or ever seen done [...] In this case, backtesting shows that large stocks had higher volatility than intermediate-term government bonds 9 decades out of 9.
Just for kicks, I just ran a very similar test with the Simba spreadsheet (the latest version allows to run various tests on a series of fixed-duration cycles). Each vertical shows the standard deviation, based on annual nominal returns, for the 10 years period starting from the year on the X axis. Then I ran it with 20 years periods, just for fun. Seems to me your hypothesis is fairly solid (common sense + empirical), although I am pretty sure that running it with UK data would display less convincing results (we can side-track in a dedicated thread, if you're interested to pursue this).

Image

Image

Note that our data sources are different, the SBBI data is a tad different from the data series used in Simba, for various reasons we discussed in the past.

dcabler
Posts: 1192
Joined: Wed Feb 19, 2014 11:30 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by dcabler » Fri Mar 30, 2018 11:15 am

My thoughts
1. Yeah, it's useful. But only in the sense of pretty big ideas: For example, in the very long term, stocks and bonds have low correlation. In the short term, they can become anti-correlated or correlated.
2. There's never enough data and never will be. So many things you can invest in didn't even exist 20 years ago or, in some cases, even 10 years ago.
3. I personally used it as one of many tools to help design my AA. But it's more like a chainsaw and less like a scalpel. No sense in trying to be hyper-accurate given #1 and #2 above.
3. Investing still ends up involving a little bit of math and a lot of leap-of-faith, in my opinion. And on this one because of #1, #2, and #3 above, we will continue to have endless debates here on BH about tilting, including international in our AA, using EM, factors, etc. Only after t->infinity will we have our answers. But frankly, I don't intend to live that long. :sharebeer

Dead Man Walking
Posts: 900
Joined: Wed Nov 07, 2007 6:51 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by Dead Man Walking » Sat Mar 31, 2018 12:00 am

I'm too pragmatic to accept the theoretical backtesting used by most researchers. If they want to sell me on their hypothesis, it must be proven using funds that are available to the general public. Of course, survivorship bias must be considered when using mutual funds. Survivorship bias may also be relevant when researching the entire market - some stocks don't survive. Many index funds have a short history when compared to the history of the market; however, they may be more relevant to retail investors who are determining their investment plans. Although style drift might be a problem, active funds could be used to document an hypothesis in a broad sense.

DMW

User avatar
fortyofforty
Posts: 1732
Joined: Wed Mar 31, 2010 12:33 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by fortyofforty » Sat Mar 31, 2018 8:27 am

Isn't the vaunted "Monte Carlo Simulation" nothing more than glorified backtesting? Without that "tool" I wouldn't have any idea how to allocate my assets.
Indexing works, not because of magic, but because of math. | Diligentia. Vis. Celeritas. - Jeff Cooper | Original Vanguard Diehard

User avatar
willthrill81
Posts: 15086
Joined: Thu Jan 26, 2017 3:17 pm
Location: USA

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by willthrill81 » Sat Mar 31, 2018 5:33 pm

The entire body of safe withdrawal rate research is based on backtested data.
“It's a dangerous business, Frodo, going out your door. You step onto the road, and if you don't keep your feet, there's no knowing where you might be swept off to.” J.R.R. Tolkien,The Lord of the Rings

alex_686
Posts: 5151
Joined: Mon Feb 09, 2015 2:39 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by alex_686 » Sat Mar 31, 2018 8:36 pm

fortyofforty wrote:
Sat Mar 31, 2018 8:27 am
Isn't the vaunted "Monte Carlo Simulation" nothing more than glorified backtesting? Without that "tool" I wouldn't have any idea how to allocate my assets.
No. A Monte Carlo simulation is loaded with a range of scenarios of your own choosing. If you want a normal distribution you can have a normal distribution. If you want skewness and fat tails, you state what you want. You can load in any type off senerio you want.

This is complex. Many web sites just slap together some historical run or some or lazy implementation.

Dominic
Posts: 211
Joined: Sat Jul 02, 2016 11:36 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by Dominic » Sat Mar 31, 2018 8:56 pm

It's not worthless unless you make it worthless. That is, don't try to find the optimal slice-and-dice portfolio with 25 asset classes by backtesting it, because you're never going to come close to replicating those results.

What is okay is to look back and examine how realistic portfolios behaved. How did a three-fund portfolio compare to a four-fund portfolio with REITs added, or to a portfolio that has a tilt to small-cap value? The future won't be exactly like the past, but you can see what tracking error looked like for those portfolios, what drawdowns looked like, etc., and then you have an idea of whether you'd be comfortable holding said portfolio given its riskiness.

Even then, backtesting is just one part of choosing an asset allocation. Don't hold a portfolio purely because it performed well in the past; you should also have a reasonable explanation as to why it should perform well in the future.

pascalwager
Posts: 1620
Joined: Mon Oct 31, 2011 8:36 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by pascalwager » Sat Mar 31, 2018 10:00 pm

willthrill81 wrote:
Sat Mar 31, 2018 5:33 pm
The entire body of safe withdrawal rate research is based on backtested data.
Thanks for pointing this out. I'm trying to personally relinquish ideas, beliefs and understandings based on back-testing and I missed this one. I suspect back-testing largely exists to enable advisors to develop, promote and sell particular portfolios.

Now I need to think some more about how to invest without back-testing derived information.

User avatar
willthrill81
Posts: 15086
Joined: Thu Jan 26, 2017 3:17 pm
Location: USA

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by willthrill81 » Sat Mar 31, 2018 10:08 pm

pascalwager wrote:
Sat Mar 31, 2018 10:00 pm
willthrill81 wrote:
Sat Mar 31, 2018 5:33 pm
The entire body of safe withdrawal rate research is based on backtested data.
Thanks for pointing this out. I'm trying to personally relinquish ideas, beliefs and understandings based on back-testing and I missed this one. I suspect back-testing largely exists to enable advisors to develop, promote and sell particular portfolios.

Now I need to think some more about how to invest without back-testing derived information.
If you put no stock in backtested results (pun intended), I really don't know how you will determine how much income you can reasonably expect to get from an invested portfolio.

I also don't know how you would determine your AA.
“It's a dangerous business, Frodo, going out your door. You step onto the road, and if you don't keep your feet, there's no knowing where you might be swept off to.” J.R.R. Tolkien,The Lord of the Rings

Topic Author
JustinR
Posts: 1147
Joined: Tue Apr 27, 2010 11:43 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by JustinR » Sat Mar 31, 2018 10:18 pm

Dominic wrote:
Sat Mar 31, 2018 8:56 pm
It's not worthless unless you make it worthless. That is, don't try to find the optimal slice-and-dice portfolio with 25 asset classes by backtesting it, because you're never going to come close to replicating those results.

What is okay is to look back and examine how realistic portfolios behaved. How did a three-fund portfolio compare to a four-fund portfolio with REITs added, or to a portfolio that has a tilt to small-cap value? The future won't be exactly like the past, but you can see what tracking error looked like for those portfolios, what drawdowns looked like, etc., and then you have an idea of whether you'd be comfortable holding said portfolio given its riskiness.

Even then, backtesting is just one part of choosing an asset allocation. Don't hold a portfolio purely because it performed well in the past; you should also have a reasonable explanation as to why it should perform well in the future.
Isn't the only difference between your first and second example is that the first example had 25 funds while your second example had 3/4?

Why is it ok for the 3/4 fund example but not 25?

Dominic
Posts: 211
Joined: Sat Jul 02, 2016 11:36 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by Dominic » Sat Mar 31, 2018 10:50 pm

JustinR wrote:
Sat Mar 31, 2018 10:18 pm
Dominic wrote:
Sat Mar 31, 2018 8:56 pm
It's not worthless unless you make it worthless. That is, don't try to find the optimal slice-and-dice portfolio with 25 asset classes by backtesting it, because you're never going to come close to replicating those results.

What is okay is to look back and examine how realistic portfolios behaved. How did a three-fund portfolio compare to a four-fund portfolio with REITs added, or to a portfolio that has a tilt to small-cap value? The future won't be exactly like the past, but you can see what tracking error looked like for those portfolios, what drawdowns looked like, etc., and then you have an idea of whether you'd be comfortable holding said portfolio given its riskiness.

Even then, backtesting is just one part of choosing an asset allocation. Don't hold a portfolio purely because it performed well in the past; you should also have a reasonable explanation as to why it should perform well in the future.
Isn't the only difference between your first and second example is that the first example had 25 funds while your second example had 3/4?

Why is it ok for the 3/4 fund example but not 25?
If you start packing in asset classes and finding the most efficient portfolio, you're essentially data mining. You add more variables so that an algorithm can squeeze out some historical gains that can't reasonably be expected to continue going forward.

msk
Posts: 1443
Joined: Mon Aug 15, 2016 10:40 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by msk » Sat Mar 31, 2018 11:07 pm

IMHO backtesting is useful (what else is there?!) for long horizons and gauging probabilities, e.g. we know:

Stocks long term do better than bonds
Trade and Industry have returned >5% p.a. real terms over the past 300 years
Between 1966 and 2016 (50 years) in the USA stocks have appreciated 6.6% p.a. and inflation 4% p.a., both compounded, and dividends have averaged 3% p.a. Note that this is line with the 300 year history.
Every market collapse was unpredictable except through hindsight.

So, conclusions:
Time in market trounces market timing
We worry too much. The next market collapse is neither predictable as to timing and nor will it drag us down to nil net worth. We will recover. Provided one lives long enough.

golfCaddy
Posts: 728
Joined: Wed Jan 10, 2018 10:02 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by golfCaddy » Sat Mar 31, 2018 11:07 pm

willthrill81 wrote:
Fri Mar 30, 2018 9:22 am
If backtesting were useless, how many would be invested in equities?
Historically, people did invest in equities, even without back-testing. When people were investing in stocks in the 1920s, they did so without the benefit of decades of historical data. The Composite index(a S&P predecessor) wasn't created until 1923.

Northern Flicker
Posts: 5055
Joined: Fri Apr 10, 2015 12:29 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by Northern Flicker » Sat Mar 31, 2018 11:13 pm

One appropriate use of backtesting is to understand risk of an asset class. If a risk showed up in the past, it is not just a theoretical risk. That doesn't mean it will materialize in the future, but it still is a risk salient to the asset class.
Index fund investor since 1987.

User avatar
nedsaid
Posts: 12971
Joined: Fri Nov 23, 2012 12:33 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by nedsaid » Sat Mar 31, 2018 11:39 pm

JustinR wrote:
Fri Mar 30, 2018 4:04 am
Whenever backtesting is brought up people remind us that the past performance isn't indicative of future results.

Doesn't that mean that backtesting is worthless, then?

So what's the purpose of backtesting? Does it have any value? What are some actual insights you can get from backtesting an allocation?
I wouldn't say that backtesting is useless. It is just that history and markets rhyme and not repeat. For example, we know that bear markets show up from time to time but the causes of each bear market are somewhat different. So what might have helped your portfolio during the last bear market might not work next time around.

Pretty much, it helps me that if an investing factor discovered by researching past data has a behavioral explanation, I am much more inclined to believe that such a factor will persist into the future. In other words, human nature and human behavior really don't change over time. People are people. Pretty much, if people made behavioral errors in the past, they will make the same behavioral errors in the future. The old saying about greed and fear.

Where this gets into problems is the belief that investing is like a math equation to be solved or an engineering problem to be fixed. Investing is more than quantitative, there is a huge behavioral element in there. Once you think you have it all figured out, the wild card of human emotion gets in there and blows it all up.

Backtesting is useful as long as you understand that it has its limitations. First, the economy and the markets are dynamic and what worked in the past may not work the same exact way in the future. Or may stop working altogether. Second, the emotions of greed and fear have never been repealed. Markets can be irrationally optimistic or irrationally pessimistic.

The problem is that market data exists in the past. By learning market history, we get a sense of the ebbs and flows of the market and what drives it. If certain things worked over and over again in the past, there are pretty good odds that they will work in the future. Even if we know that something will happen, we don't know when it will happen. What I am trying to say, is that by studying the past you get a sense of what will likely happen in the future but the future will still hold some surprises.

So use it as a guide realizing that the guidance is not infallible.
A fool and his money are good for business.

User avatar
willthrill81
Posts: 15086
Joined: Thu Jan 26, 2017 3:17 pm
Location: USA

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by willthrill81 » Sun Apr 01, 2018 12:10 am

golfCaddy wrote:
Sat Mar 31, 2018 11:07 pm
willthrill81 wrote:
Fri Mar 30, 2018 9:22 am
If backtesting were useless, how many would be invested in equities?
Historically, people did invest in equities, even without back-testing. When people were investing in stocks in the 1920s, they did so without the benefit of decades of historical data. The Composite index(a S&P predecessor) wasn't created until 1923.
I never said that no one would, but I seriously doubt that as many would today if there was no historic record.
“It's a dangerous business, Frodo, going out your door. You step onto the road, and if you don't keep your feet, there's no knowing where you might be swept off to.” J.R.R. Tolkien,The Lord of the Rings

User avatar
HomerJ
Posts: 13559
Joined: Fri Jun 06, 2008 12:50 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by HomerJ » Sun Apr 01, 2018 12:43 am

Watty wrote:
Fri Mar 30, 2018 7:24 am
Backtesting is better at telling you when you have a "bad plan" that you should reconsider.
This. Every investment model that has ever been used at one point back-tested well. And yet many of them failed going forward.

So a lot of false positives.

But back-testing can indeed show that a model failed in the past. It can be used to reject a model, but you can't claim a model will definitely work going forward based on back-testing.

User avatar
HomerJ
Posts: 13559
Joined: Fri Jun 06, 2008 12:50 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by HomerJ » Sun Apr 01, 2018 12:46 am

nedsaid wrote:
Sat Mar 31, 2018 11:39 pm
JustinR wrote:
Fri Mar 30, 2018 4:04 am
Whenever backtesting is brought up people remind us that the past performance isn't indicative of future results.

Doesn't that mean that backtesting is worthless, then?

So what's the purpose of backtesting? Does it have any value? What are some actual insights you can get from backtesting an allocation?
I wouldn't say that backtesting is useless. It is just that history and markets rhyme and not repeat. For example, we know that bear markets show up from time to time but the causes of each bear market are somewhat different. So what might have helped your portfolio during the last bear market might not work next time around.

Pretty much, it helps me that if an investing factor discovered by researching past data has a behavioral explanation, I am much more inclined to believe that such a factor will persist into the future. In other words, human nature and human behavior really don't change over time. People are people. Pretty much, if people made behavioral errors in the past, they will make the same behavioral errors in the future. The old saying about greed and fear.

Where this gets into problems is the belief that investing is like a math equation to be solved or an engineering problem to be fixed. Investing is more than quantitative, there is a huge behavioral element in there. Once you think you have it all figured out, the wild card of human emotion gets in there and blows it all up.

Backtesting is useful as long as you understand that it has its limitations. First, the economy and the markets are dynamic and what worked in the past may not work the same exact way in the future. Or may stop working altogether. Second, the emotions of greed and fear have never been repealed. Markets can be irrationally optimistic or irrationally pessimistic.

The problem is that market data exists in the past. By learning market history, we get a sense of the ebbs and flows of the market and what drives it. If certain things worked over and over again in the past, there are pretty good odds that they will work in the future. Even if we know that something will happen, we don't know when it will happen. What I am trying to say, is that by studying the past you get a sense of what will likely happen in the future but the future will still hold some surprises.

So use it as a guide realizing that the guidance is not infallible.
This is a really excellent post nedsaid.

foo.c
Posts: 62
Joined: Sat Jul 15, 2017 4:55 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by foo.c » Sun Apr 01, 2018 7:30 am

Among other things, it's a good way to waste time while you're waiting for your forward testing to finish.

User avatar
fortyofforty
Posts: 1732
Joined: Wed Mar 31, 2010 12:33 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by fortyofforty » Sun Apr 01, 2018 7:37 am

alex_686 wrote:
Sat Mar 31, 2018 8:36 pm
fortyofforty wrote:
Sat Mar 31, 2018 8:27 am
Isn't the vaunted "Monte Carlo Simulation" nothing more than glorified backtesting? Without that "tool" I wouldn't have any idea how to allocate my assets.
No. A Monte Carlo simulation is loaded with a range of scenarios of your own choosing. If you want a normal distribution you can have a normal distribution. If you want skewness and fat tails, you state what you want. You can load in any type off senerio you want.

This is complex. Many web sites just slap together some historical run or some or lazy implementation.
But the range of numbers almost all financial analysts use are based on what has been seen. The choices of what values to enter are made by first looking backward. Normal distribution is "normal" only based on what we've seen in the past. Tails are only "fat" or not based on historical performance. Things can be viewed as "skewed" only in light of what's already happened, extrapolated to account for current conditions, to make inferences and draw conclusions. Probabilities of outcome are based first on history.
Indexing works, not because of magic, but because of math. | Diligentia. Vis. Celeritas. - Jeff Cooper | Original Vanguard Diehard

User avatar
tennisplyr
Posts: 2397
Joined: Tue Jan 28, 2014 1:53 pm
Location: Sarasota, FL

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by tennisplyr » Sun Apr 01, 2018 7:42 am

I look at it as one piece of information out of many, many pieces of information I might use when making an investment decision....certainly not as a predictive piece.
Those who move forward with a happy spirit will find that things always work out.

selters
Posts: 658
Joined: Thu Feb 27, 2014 9:26 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by selters » Sun Apr 01, 2018 7:47 am

If backtesting is worthless, is there any investment knowledge out there that is not worthless?

User avatar
siamond
Posts: 5128
Joined: Mon May 28, 2012 5:50 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by siamond » Sun Apr 01, 2018 9:15 am

nedsaid wrote:
Sat Mar 31, 2018 11:39 pm
I wouldn't say that backtesting is useless. It is just that history and markets rhyme and not repeat.
[...]
So use it as a guide realizing that the guidance is not infallible.
I tried to say something similar a few posts ago, but your post was much better worded. 100% agreed.

longinvest
Posts: 4085
Joined: Sat Aug 11, 2012 8:44 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by longinvest » Sun Apr 01, 2018 9:24 am

selters wrote:
Sun Apr 01, 2018 7:47 am
If backtesting is worthless, is there any investment knowledge out there that is not worthless?
I don't agree that backtesting is worthless in general, only that it's worthless to predict future returns. It's useful for other things.

But, it's important to know some investment knowledge doesn't rely on backtesting:
  • The law of supply and demand.
  • The corollary that stocks must return more than bonds often enough, otherwise the price of stocks would drop along with demand, increasing their future returns, until the price became attractive enough.
  • There are also theorems, like The Arithmetic of Active Management.
So, it's possible to select an asset allocation without relying on backtesting.

As other have said, backtesting can be useful to learn about risks that have shown up in the past, yet it would be a mistake to think that all risks have already shown up in the relatively little amount of historical data (of variable quality) that we have access to.
Bogleheads investment philosophy | One-ETF global balanced index portfolio | VPW

dbr
Posts: 31196
Joined: Sun Mar 04, 2007 9:50 am

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by dbr » Sun Apr 01, 2018 9:41 am

Things like "backtesting is useless" "past results don't predict future results" "this fund is going to 'beat' that fund" etc., etc. are thoughtless oversimplifications of investing. Investing is both simple and very complex depending on what it is one is discussing at the time and information about investing, truths about investing, are also both simple and very complex depending on what one is talking about. This is not different from anything else that goes on in the world.

alex_686
Posts: 5151
Joined: Mon Feb 09, 2015 2:39 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by alex_686 » Sun Apr 01, 2018 9:50 am

fortyofforty wrote:
Sun Apr 01, 2018 7:37 am
But the range of numbers almost all financial analysts use are based on what has been seen. The choices of what values to enter are made by first looking backward. Normal distribution is "normal" only based on what we've seen in the past. Tails are only "fat" or not based on historical performance. Things can be viewed as "skewed" only in light of what's already happened, extrapolated to account for current conditions, to make inferences and draw conclusions. Probabilities of outcome are based first on history.
Are they? 2 points.

The first point is on models in general. When a model spits out a number how confident are you about those numbers? When a Monte Carlo says there is a 5% chance of failure how accurate is that 5%? The problems with most models are tail events, black swans, etc. They don't happen with normal frequency. One can estimate fat tails but those estimates have historically had huge errors. Monte Carole testing is an answer. We can use historical estimates - but the frequency and depth don't conform to spastically theory. Or we can project ahead using theory and logic.

Which takes me to my second point. We know the future is not going to be like the past. We are not going to have another Great Depression or 70s stagflation. We know enough about that events to sidestep them. No, we are going to blow up in a entirely new way. I will point for the Secular Stagnation theory which suggests that we are going to have historically low returns for the next 20 years. I believe in the theory. We can input this hypothetical world instead of a historical world.

I like economic history. I majored in college in it and I read deeply. There is much to be learned. I am also keeenly aware of its limitations.

User avatar
fortyofforty
Posts: 1732
Joined: Wed Mar 31, 2010 12:33 pm

Re: Is backtesting worthless? Are there ANY insights you can get from it?

Post by fortyofforty » Sun Apr 01, 2018 10:09 am

alex_686 wrote:
Sun Apr 01, 2018 9:50 am
fortyofforty wrote:
Sun Apr 01, 2018 7:37 am
But the range of numbers almost all financial analysts use are based on what has been seen. The choices of what values to enter are made by first looking backward. Normal distribution is "normal" only based on what we've seen in the past. Tails are only "fat" or not based on historical performance. Things can be viewed as "skewed" only in light of what's already happened, extrapolated to account for current conditions, to make inferences and draw conclusions. Probabilities of outcome are based first on history.
Are they? 2 points.
Yes, they are.
alex_686 wrote:
Sun Apr 01, 2018 9:50 am
The first point is on models in general. When a model spits out a number how confident are you about those numbers? When a Monte Carlo says there is a 5% chance of failure how accurate is that 5%? The problems with most models are tail events, black swans, etc. They don't happen with normal frequency. One can estimate fat tails but those estimates have historically had huge errors. Monte Carole testing is an answer. We can use historical estimates - but the frequency and depth don't conform to spastically theory. Or we can project ahead using theory and logic.
This is to say they haven't happened with normal frequency. Backtesting. Theory and logic are grounded in backtesting, which allows us to be somewhat confident in future projections. Economic theories are very good at explaining why something happened, and rather less good at predicting what will happen.
alex_686 wrote:
Sun Apr 01, 2018 9:50 am
Which takes me to my second point. We know the future is not going to be like the past. We are not going to have another Great Depression or 70s stagflation. We know enough about that events to sidestep them. No, we are going to blow up in a entirely new way. I will point for the Secular Stagnation theory which suggests that we are going to have historically low returns for the next 20 years. I believe in the theory. We can input this hypothetical world instead of a historical world.
We think we will not have another Great Depression or 70s stagflation. We hope the financial experts making macroeconomic decisions are wise enough not to repeat the same mistakes that allowed those events to occur. We don't know we won't. We believe we won't.
alex_686 wrote:
Sun Apr 01, 2018 9:50 am
I like economic history. I majored in college in it and I read deeply. There is much to be learned. I am also keeenly aware of its limitations.
Nothing you presented changes anything about my contention that backtesting is the essence of asset allocation models. I use it. You use it. Every financial expert uses it. Every useful Monte Carlo Simulation is based upon it. That doesn't make it wrong. That simply makes it reality. It's all we have.

Here's a thought experiment: ignore all financial history. Try to make decisions regarding asset allocation for a specific goal or time period without resorting to what happened in the past. It's well-nigh impossible.
Indexing works, not because of magic, but because of math. | Diligentia. Vis. Celeritas. - Jeff Cooper | Original Vanguard Diehard

Post Reply