John C. Bogle exposed? No way! [Investing paper retracted]

[Allan Roth]

It's been 15 years since the Journal of Financial Planning awarded the winner of their annual call-for-papers competition under the "academic" category for a paper entitled "Passive Investing: The Emperor Exposed." Jack was that emperor and the paper showed all previous studies were wrong and that 2/3 of active funds bested passive S&P 500 index funds by more than two percent annually.

https://www.financialplanningassociatio ... T05JFP.pdf

One call to the data provider, Lipper, confirmed the fatal flaws in this so called academic study but, although I challenged, the Journal's anonymous appeals committee refused to make a five-minute call to Lipper to confirm and upheld the paper. Enough pressure was put on the Journal where they eventually made that call and retracted the paper.

https://www.financialplanningassociatio ... or-exposed

It's still the only paper ever retracted by the journal but other so-called academic papers try to disprove arithmetic. We don't have John C. Bogle any longer but millions of us continue to benefit for the index funds he brought us.

[livesoft]

In many journals, retracted papers found in online archives have a much more prominent watermark than the tiny box used by JFP.

See also: https://retractionwatch.com/

[000]

Well, the good news is most investors (including me before reading your post) have never heard of The Journal of Financial Planning.

[Allan Roth]

Well, the good news is most investors (including me before reading your post) have never heard of The Journal of Financial Planning.

It's a publication by the Financial Planning Association for members who at typically CFPs. People like respect like David Blanchett at Morningstar publish some great work in this Journal.

I can tell you for sure that Jack read that piece about him. It was Jack's assistant that showed it to me. And Barron's just published a piece referencing a study that active bested passive.

[nisiprius]

I appreciate the yeoman work you did in getting Ric Edelman to sort of kind of not quite retract his crazy claim that Vanguard index funds cost investors 1.4%/year in hidden expenses. Although I think you said he started making that claim again?

It's too much to hope for but I do wish that the SPIVA reports also contained percentages of active funds underperforming, not the costless index itself, but actual mutual funds or ETFs based on that index. It could even be an S&P promotional tool as they could use a point of choosing funds and ETFs tracking S&P indexes.

[jeffyscott]

One call to the data provider, Lipper, confirmed the fatal flaws in this so called academic study but, although I challenged, the Journal's anonymous appeals committee refused to make a five-minute call to Lipper to confirm and upheld the paper. Enough pressure was put on the Journal where they eventually made that call and retracted the paper.

What were the fatal flaws? Just that the data did not actually exist? The retraction just says: "conclusions were based on data that did not match the data presented in the paper".

[Robot Monster]

I thought this thread was going to be salacious, based on the subject title. Mildly disappointed it's not.

[neurosphere]

One call to the data provider, Lipper, confirmed the fatal flaws in this so called academic study but, although I challenged, the Journal's anonymous appeals committee refused to make a five-minute call to Lipper to confirm and upheld the paper. Enough pressure was put on the Journal where they eventually made that call and retracted the paper.

What were the fatal flaws? Just that the data did not actually exist? The retraction just says: "conclusions were based on data that did not match the data presented in the paper".

I had the same question. Was the analysis accurate but the data set flawed? Was the author motivated to use this data set (e.g. intentional because it would lead to the desired conclusion).

I've seen several cases where a retracted paper leads to a NEW publication with the opposite conclusion. For example when all of the methods are solid apart from one issue (a contaminated reagent, a calculation error) identified after publication, which leads to a new one. I guess the author couldn't get a paper published which says "uh I've confirmed yet again that passive investing beats active investing".

[hnd]

They came for him then and now that he's gone they'll come for him even harder. Now that indexing is approximately 50% of the mutual fund market. Their sliver of take is shrinking.

[bertilak]

One call to the data provider, Lipper, confirmed the fatal flaws in this so called academic study but, although I challenged, the Journal's anonymous appeals committee refused to make a five-minute call to Lipper to confirm and upheld the paper. Enough pressure was put on the Journal where they eventually made that call and retracted the paper.

What were the fatal flaws? Just that the data did not actually exist? The retraction just says: "conclusions were based on data that did not match the data presented in the paper".

I also would like to know.

[cheezit]

I appreciate the yeoman work you did in getting Ric Edelman to sort of kind of not quite retract his crazy claim that Vanguard index funds cost investors 1.4%/year in hidden expenses. Although I think you said he started making that claim again?

It's too much to hope for but I do wish that the SPIVA reports also contained percentages of active funds underperforming, not the costless index itself, but actual mutual funds or ETFs based on that index. It could even be an S&P promotional tool as they could use a point of choosing funds and ETFs tracking S&P indexes.

How could he seriously make this claim? The tracking error numbers aren't exactly hidden.

[mptness]

Thanks. I am a big fan of all your writing and appreciate reading you posts on this forum. Bogle's words: " You get what you don't pay for" are supported by years of SPIVA Reports, but fighting words to your average Financial Planner.

[Greenman72]

Well, the good news is most investors (including me before reading your post) have never heard of The Journal of Financial Planning.

It is one of the few academic, peer-reviewed journals in the industry.

I have never read the Journal of Petroleum Engineering, but I still fill my car with gasoline.

[kupuna]

Allan, you are a soldier of the truth and we are all better off because of it.

TOM

[aristotelian]

Claudia Sahm was on Masters in Business podcast (Barry Ritholz) talking about replication problems in Economics. https://www.bloomberg.com/news/audio/20 ... cy-podcast

There is a growing cottage industry in academia of replication studies that shows pretty much across fields a large portion of studies fail replication and overstate their results.

[nzahir]

Not reading this, but are they comparing apples to apples or not (likely not)?

Are they comparing tech, large growth, and small value vs large blend?

[Doctor Rhythm]

Claudia Sahm was on Masters in Business podcast (Barry Ritholz) talking about replication problems in Economics. https://www.bloomberg.com/news/audio/20 ... cy-podcast

There is a growing cottage industry in academia of replication studies that shows pretty much across fields a large portion of studies fail replication and overstate their results.

Replication is mainly an issue with experimental research under controlled conditions. I didn’t listen to the podcast, but curious how this applies to macroeconomics/finance - where it seems to be more about analysis of real world data, without an expectation of reproducibility given lack of controllable conditions. It sounds more like a badly chosen data set or improper analysis. Not my area of specialty, so just wondering.

[Allan Roth]

One call to the data provider, Lipper, confirmed the fatal flaws in this so called academic study but, although I challenged, the Journal's anonymous appeals committee refused to make a five-minute call to Lipper to confirm and upheld the paper. Enough pressure was put on the Journal where they eventually made that call and retracted the paper.

What were the fatal flaws? Just that the data did not actually exist? The retraction just says: "conclusions were based on data that did not match the data presented in the paper".

According to Lipper, the data request was for then current 2004 data and weights at that time. So it excluded 40 years of funds that went out of business and, even more importantly, weighted the funds based on the then current assets. So naturally the active funds that performed the best garnered more assets and were weighted more heavily even though they may have been tiny for most of that period.

So it's a combination of extreme survivor bias and extreme weighting bias.

[aristotelian]

Claudia Sahm was on Masters in Business podcast (Barry Ritholz) talking about replication problems in Economics. https://www.bloomberg.com/news/audio/20 ... cy-podcast

There is a growing cottage industry in academia of replication studies that shows pretty much across fields a large portion of studies fail replication and overstate their results.

Replication is mainly an issue with experimental research under controlled conditions. I didn’t listen to the podcast, but curious how this applies to macroeconomics/finance - where it seems to be more about analysis of real world data, without an expectation of reproducibility given lack of controllable conditions. It sounds more like a badly chosen data set or improper analysis. Not my area of specialty, so just wondering.

Sorry, Econ is not my field so I was not listening for those details. I believe this is the paper she references in the podcast. According to her, it was desk-rejected from all the top field journals.
https://www.federalreserve.gov/econresd ... 083pap.pdf

[neurosphere]

One call to the data provider, Lipper, confirmed the fatal flaws in this so called academic study but, although I challenged, the Journal's anonymous appeals committee refused to make a five-minute call to Lipper to confirm and upheld the paper. Enough pressure was put on the Journal where they eventually made that call and retracted the paper.

What were the fatal flaws? Just that the data did not actually exist? The retraction just says: "conclusions were based on data that did not match the data presented in the paper".

According to Lipper, the data request was for then current 2004 data and weights at that time. So it excluded 40 years of funds that went out of business and, even more importantly, weighted the funds based on the then current assets. So naturally the active funds that performed the best garnered more assets and were weighted more heavily even though they may have been tiny for most of that period.

So it's a combination of extreme survivor bias and extreme weighting bias.

Thanks Allan, in the end, the problem seems obvious in retrospect.

I've been trying for a long time to find public or cheap data that lists closed funds by fund family. Do those exist? Or are they proprietary data sets that cost a lot of money to obtain? I have a feeling there are a ton of folks on BH with the education, knowledge, and experience to do fun little bias-free research.

[000]

It's a publication by the Financial Planning Association for members who at typically CFPs. People like respect like David Blanchett at Morningstar publish some great work in this Journal.

I can tell you for sure that Jack read that piece about him. It was Jack's assistant that showed it to me. And Barron's just published a piece referencing a study that active bested passive.

I wonder why the journal is still held in such esteem after the events you describe.

Well, the good news is most investors (including me before reading your post) have never heard of The Journal of Financial Planning.

It is one of the few academic, peer-reviewed journals in the industry.

I have never read the Journal of Petroleum Engineering, but I still fill my car with gasoline.

Well, you've made my point for me.

The vast majority of people on Earth (including me) do not make life decisions based on academic publications.

A fraudulent article in the Journal of Petroleum Engineering claiming premium gas is better for my car engine won't induce me to buy premium.

And I certainly won't pay a GUM fee to a CGP to pick fuels for my engine.

[peetsperk]

I noticed that Wikipedia notes many accomplishments of Christopher Carosa, but makes no reference to the retraction. To bring balance, should this be noted?

[hnd]

I noticed that Wikipedia notes many accomplishments of Christopher Carosa, but makes no reference to the retraction. To bring balance, should this be noted?

In 2005, the Journal of Financial Planning named Carosa as the winner of their annual call-for-papers competition under the "academic" category. His paper introduced the "snapshot-in-time" anomaly as it pertains to measuring investment performance.[16][17] In 2006, Allan Roth formally challenged and claimed there was a math error, but the Appeal Committee did not find enough evidence in their review to recommend retracting and stated that it "came down to methodology rather than mathematics."[18] After further inquires made by Journal and Financial Planning Association staff and no follow-up research provided by Carosa, the paper was retracted by the Journal in May 2007 for possible flaws.[19] To date, it is the only paper ever retracted by the Journal of Financial Planning per Carley Schulaka, editor of the Journal.[verification needed]

[jeffyscott]

One call to the data provider, Lipper, confirmed the fatal flaws in this so called academic study but, although I challenged, the Journal's anonymous appeals committee refused to make a five-minute call to Lipper to confirm and upheld the paper. Enough pressure was put on the Journal where they eventually made that call and retracted the paper.

What were the fatal flaws? Just that the data did not actually exist? The retraction just says: "conclusions were based on data that did not match the data presented in the paper".

According to Lipper, the data request was for then current 2004 data and weights at that time. So it excluded 40 years of funds that went out of business and, even more importantly, weighted the funds based on the then current assets. So naturally the active funds that performed the best garnered more assets and were weighted more heavily even though they may have been tiny for most of that period.

So it's a combination of extreme survivor bias and extreme weighting bias.

Thanks for those details. Survivorship bias occurred to me, but the paper (apparently mistakenly) claimed the data included closed and merged funds. Maybe they were "included" but then weighted at 0, since they had $0 as of 2004.

I would guess that asset weighting correctly would still help the managed fund results a bit, since larger funds tend to have lower ERs, if nothing else.

[AlwaysLearningMore]

Claudia Sahm was on Masters in Business podcast (Barry Ritholz) talking about replication problems in Economics. https://www.bloomberg.com/news/audio/20 ... cy-podcast

Do economists routinely perform double-blind experiments to test hypotheses, and are those double-blind experiments replicated by other researchers (i.e., science)?

For those who feel this is a 'no true Scotsman appeal to purity,' I say: Eaconamairean Albannach (mar an co-obraichean anns a h-uile àite) bu mhath leotha a bhith air am faicinn mar 'fìor luchd-saidheans.' Gu mì-fhortanach, leis nach eil iad a ’dèanamh deuchainnean fo smachd dà-dall tha iad fhathast air an ainmeachadh mar‘ luchd-saidheans sòisealta. ’ Bidh ceimigearan, fiosaig agus bith-eòlaichean luath airson na h-eadar-dhealachaidhean a chomharrachadh.
Bidh ceimigearan, fiosaig agus bith-eòlaichean gu luath a ’comharrachadh na h-eadar-dhealachaidhean eadar an dòigh-obrach aca an aghaidh eaconamaichean.
Tha fìor shaidheans air a chomharrachadh le smachd, mionaideachd, reusantachd, caochladairean fo smachd, agus ro-innseachd. Tha saidheans sòisealta an taobh eile air a choileanadh le caochladairean neo-fhaicsinneach no neo-riaghlaidh, agus bidh e a ’dèiligeadh ri faireachdainnean agus giùlan dhaoine.

[Doctor Rhythm]

Claudia Sahm was on Masters in Business podcast (Barry Ritholz) talking about replication problems in Economics. https://www.bloomberg.com/news/audio/20 ... cy-podcast

Do economists routinely perform double-blind experiments to test hypotheses, and are those double-blind experiments replicated by other researchers (i.e., science)?

This is essentially what I was asking above. I think the term “replication” problem was being used broadly to refer to any problem with external validity.

[Allan Roth]

I noticed that Wikipedia notes many accomplishments of Christopher Carosa, but makes no reference to the retraction. To bring balance, should this be noted?

When I first noticed the lack of the mention for the retraction several months ago, I made the following edit to mention the retraction which reads:

In 2005, the Journal of Financial Planning named Carosa as the winner of their annual call-for-papers competition under the "academic" category. His paper introduced the "snapshot-in-time" anomaly as it pertains to measuring investment performance.[16][17] In 2006, Allan Roth formally challenged and claimed there was a math error, but the Appeal Committee did not find enough evidence in their review to recommend retracting and stated that it "came down to methodology rather than mathematics."[18] After further inquires made by Journal and Financial Planning Association staff and no follow-up research provided by Carosa, the paper was retracted by the Journal in May 2007 for possible flaws.[19] To date, it is the only paper ever retracted by the Journal of Financial Planning per Carly Schulaka, editor of the Journal.[verification needed]

Carly Schulaka told me to the best for her knowledge, it was the only paper ever to be retracted by the journal

[afan]

Just to keep the record straight, Journal of Financial Planning is very far from a top academic journal. Based on what it publishes and the editorial standards, it would be difficult to describe it as an academic journal at all.

List of the top academic economics and finance journals. I sorted by H index but you can choose your own metric.

https://www.scimagojr.com/journalrank.p ... h&ord=desc

It goes from 1 (Journal of Finance) to 282. Journal of Financial Planning is not on the list. That may mean it has some citations, just fewer than the 282nd journal. Or that it has none at all.