http://www.scientificamerican.com/artic ... ways-wrongWhen it comes to assigning blame for the current economic doldrums, the quants who build the complicated mathematic financial risk models, and the traders who rely on them, deserve their share of the blame. [See “A Formula For Economic Calamity” in the November 2011 issue]. But what if there were a way to come up with simpler models that perfectly reflected reality? And what if we had perfect financial data to plug into them?
Incredibly, even under those utterly unrealizable conditions, we'd still get bad predictions from models.
The reason is that current methods used to “calibrate” models often render them inaccurate.
Almost all models have parameters that have to be adjusted to make a model applicable to the specific conditions to which it's being applied--the spring constant in Hooke's law, for example, or the resistance in an electrical circuit. Calibrating a complex model for which parameters can't be directly measured usually involves taking historical data, and, enlisting various computational techniques, adjusting the parameters so that the model would have "predicted" that historical data. At that point the model is considered calibrated, and should predict in theory what will happen going forward.
Carter had initially used arbitrary parameters in his perfect model to generate perfect data, but now, in order to assess his model in a realistic way, he threw those parameters out and used standard calibration techniques to match his perfect model to his perfect data. It was supposed to be a formality--he assumed, reasonably, that the process would simply produce the same parameters that had been used to produce the data in the first place. But it didn't. It turned out that there were many different sets of parameters that seemed to fit the historical data. And that made sense, he realized--given a mathematical expression with many terms and parameters in it, and thus many different ways to add up to the same single result, you'd expect there to be different ways to tweak the parameters so that they can produce similar sets of data over some limited time period.
I'm sure this is no surprise to the finance professionals and academics who frequent this forum, but as a layman I thought this was interesting.
I had been advised, and had accepted, that expectations of ex-ante returns of a portfolio which is optimized based upon backtesting is bullplop. I guess I hadn't questioned why, exactly, that was so. This article lit up the lightbulb above my head, so I thought I'd share.