Shadow Government Statistics Indicator Re Inflation

Discuss all general (i.e. non-personal) investing questions and issues, investing news, and theory.
Post Reply
Topic Author
Dagwood
Posts: 1026
Joined: Mon Mar 02, 2009 5:26 pm
Location: MD

Shadow Government Statistics Indicator Re Inflation

Post by Dagwood »

I found this news article and thought it was interesting:

http://finance.yahoo.com/news/Inflation ... 6.html?x=0

The claim is that inflation calculated under the "old method" is closer to 10%. The insinuation is that this is a more accurate measure of inflation today.

I would be interested in hearing people's views as to the pros / cons of the older methodology as compared to the newer methodology. However, this topic can violate forum rules if people turn it into a surrogate for debating popular economic / political policy issues, so please refrain from so doing. Instead, let's try to keep the discussion to why one methodology or another is better. I am trying to give myself the ability to look at these things more critically, not looking to incite a political debate. And if someone wanders off the reservation of what is permitted here, let's just ignore it. Thanks in advance.
User avatar
Barry Barnitz
Wiki Admin
Posts: 3227
Joined: Mon Feb 19, 2007 10:42 pm
Contact:

Billion Prices Project @ MIT

Post by Barry Barnitz »

MIT also produces a look at inflation:

The Billion Prices Project @ MIT

Methodology

Data collection: our data are collected every day from online retailers using a software that scans the underlying code in public webpages and stores the relevant price information in a database. The resulting dataset contains daily prices on the full array of products sold by these retailers. Our data include information on product descriptions, package sizes, brands, special characteristics (e.g. “organic”), and whether the item is on sale or price control.

Daily Online Price Index Computation: The daily online index is an average of individual price changes across multiple categories and retailers. The index uses a basket of goods that changes over time as products appear and disappear from a retailer’s webpage. It is updated on a daily basis and leveraged to estimate annual and monthly inflation. This index is not designed to forecast official inflation announcements, but to provide real-time information on major inflation trends.

Monthly Inflation: The monthly inflation rate is the percentage change between the average of the daily online price index of the last 30 days and the average of the previous month. For example, on the last day of September 2010, we compared the average of the daily index between September 1st and September 30th to the average of the daily index between August 1st and August 31st. On the last day of each month, the value of our monthly inflation is equivalent to the monthly statistic reported by official offices.

Annual Inflation: The annual inflation rate is the percentage change between the average of the daily online price index of the last 30 days and the average for the same period a year ago. For example, on the last day of September 2010, we compare the average of the daily index between September 1st and September 30th 2010 to the average of the daily index between September 1st and September 30th 2009. On the last day of each month, the value of our annual inflation is equivalent to the annual (year-to-year) statistic reported by official offices.
regards,
Additional administrative tasks: Financial Page bogleheads.org. blog; finiki the Canadian wiki; The Bogle Center for Financial Literacy site; Wiki Bogleheads® España.
Rodc
Posts: 13601
Joined: Tue Jun 26, 2007 9:46 am

Post by Rodc »

Just to note that the search function will provide a huge amount of data on what folks here about think about this topic. It has been beat to death many times.

Everything from these folks wear tin foil hats to of course the Feds cook the books.
We live a world with knowledge of the future markets has less than one significant figure. And people will still and always demand answers to three significant digits.
User avatar
Random Musings
Posts: 5792
Joined: Thu Feb 22, 2007 4:24 pm
Location: Pennsylvania

Post by Random Musings »

Barry Barnitz wrote:
MIT also produces a look at inflation:

The Billion Prices Project @ MIT
Thanks for the link - I recall that work being done. What is interesting is the high inflation rates currently hitting Argentina, Russia and Venezuela. However, most of the pricing "tracks" pretty well with CPI's.

Except for Argentina - perhaps something a big fishy is going on there.

RM
User avatar
nisiprius
Advisory Board
Posts: 41964
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Post by nisiprius »

See the Wiki article Accuracy of the CPI for an overview.

I think the Shadowstats site is contemptible, for these reasons:

a) They do not give their data or state their methodology. There is no way to check what they post. It's simply an assertion.

b) Shadowstats claims that their SGS Alternate Inflation curve "reflects the CPI as if it were calculated using the methodologies in place in 1980." This is false on the face of it; they cannot be doing what they say they are doing. Prior to 1983, the CPI was calculated using actual housing prices; after 1983 it was changed and based on owner's equivalent rent. As noted by matt in 2010 (see below), if the SGS Alternate Inflation curve really used 1980 methodologies, the housing price collapse of 2006-2008 should have caused a much larger drop in the SGS Alternative Inflation curve than it did in the official CPI index, and the distance between the two curves should have narrowed. Instead, they stayed parallel.

c) The Bureau of Labor Statistics provides CPI-RS which is a recalculation of data from 1978 to the present using the new methodology. There's a difference, and the newer method calculates a lower number, but only by about 0.45% per year. If you believe the old method was the correct one, then the official figures understate inflation, but only about 0.45% per year, not the 6%-8% claimed by ShadowStats.
Last edited by nisiprius on Tue Apr 19, 2011 1:52 pm, edited 2 times in total.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.
grayfox
Posts: 5569
Joined: Sat Sep 15, 2007 4:30 am

Re: Billion Prices Project @ MIT

Post by grayfox »

Barry Barnitz wrote:MIT also produces a look at inflation:

The Billion Prices Project @ MIT
That is pretty interesting. I read that Google is trying to do something similar. I am in favor of alternative measures of inflation, as long as they actually look at prices and performing actual calculations. [As opposed to certain charlatans, that publish an alternative price index showing 10% inflation without, by their own admission, collecting any data or doing any meaningful calculation. Whenever I see someone site that source for inflation I always think "Another gullible buffoon!" and then discount everything the person has to say on any topic.]
User avatar
bobcat2
Posts: 5436
Joined: Tue Feb 20, 2007 3:27 pm
Location: just barely Outside the Beltway

Post by bobcat2 »

I follow the index provided by the Billion Prices Price Index @ MIT to see if it is tracking the CPI. But it is important to recognize the limitations of this online price index. In particular it tells us very little about price changes in services as opposed to price changes in goods. Notice this caveat posted at the Billion Prices Price Index site.
These indexes are designed to provide real-time information on major inflation trends, not to forecast official inflation announcements. We are constantly adding new categories of goods, but we do not cover 100% of CPI goods and services. The price of services, in particular, are not easy to find online and therefore are not included in our statistics.
The main advantage of this index is the numbers come out daily practically in real time, rather than the lag in the CPI, which comes out monthly with about a 3 week lag after the end of a month.

BobK
In finance risk is defined as uncertainty that is consequential (nontrivial). | The two main methods of dealing with financial risk are the matching of assets to goals & diversifying.
matt
Posts: 2305
Joined: Sun Mar 04, 2007 3:47 pm

Post by matt »

Reposting what I wrote on April 1, 2010. (Fitting that we were discussing this topic on April Fool's day!) http://www.bogleheads.org/forum/viewtop ... ht=#700479
matt wrote:I think the CPI is a little bogus. I think ShadowStats is massively bogus.

Consider that one of the main arguments from ShadowStats prior to the housing bust was that owners equivalent rent (OER) was substituted for home prices, greatly subduing the CPI over the late 90’s up to 2006. But then housing prices tumbled and "true" inflation, according to what ShadowStats was saying back then, should have turned deeply deflationary, whereas CPI only dipped modestly because rents held relatively steady.

So on the chart here (http://www.shadowstats.com/alternate_da ... ion-charts), we should see a period in 2007 or 2008 where the SGS Alternate CPI drops below the Official CPI-U. At the very least, the gap between the two should close significantly. But what do we see? No matter what was happening with home prices in relation to OER, the Official CPI-U is dramatically lower than SGS Alternate CPI each year.

So I'm calling bulls**t. My guess is that ShadowStats is massaging (perhaps manhandling is the more appropriate term) the data so that no matter what happens, its version of inflation will always come out substantially higher. I also think it is preying on the fact that many people A) aren’t good at math, and B) don’t trust the Government. The idea that CPI-U has understated inflation by 5-10% per year for the past 20+years is demonstrably false.
Update for the year that has passed since then: housing continues to be very deflationary, yet Shadowstats chart still shows absolutely no evidence of this in its chart. I really wouldn't be surprised if they just make up the numbers.
User avatar
bobcat2
Posts: 5436
Joined: Tue Feb 20, 2007 3:27 pm
Location: just barely Outside the Beltway

Post by bobcat2 »

Looking at the graph of both the MIT daily goods numbers and the monthly CPI for the US at the MIT site shows a spike in goods inflation in February and early March that is now retreating in the MIT daily data. OTOH the CPI monthly index is still moving up for the latest monthly data. What the daily MIT goods price index appears to suggest is that the CPI will decelerate in the next couple of months.

BobK
In finance risk is defined as uncertainty that is consequential (nontrivial). | The two main methods of dealing with financial risk are the matching of assets to goals & diversifying.
Topic Author
Dagwood
Posts: 1026
Joined: Mon Mar 02, 2009 5:26 pm
Location: MD

Post by Dagwood »

nisiprius wrote:See the Wiki article Accuracy of the CPI for an overview.

. . .

b) Shadowstats claims that their SGS Alternate Inflation curve "reflects the CPI as if it were calculated using the methodologies in place in 1980." This is false on the face of it; they cannot be doing what they say they are doing. Prior to 1983, the CPI was calculated using actual housing prices; after 1983 it was changed and based on owner's equivalent rent. If the SGS Alternate Inflation curve really used 1980 methodologies, the housing price collapse of 2006-2008 should have caused a much larger drop in the SGS Alternative Inflation curve than it did in the official CPI index, and the distance between the two curves should have narrowed. Instead, they stayed parallel.

. . .

Thank you nisiprius and Barry. I've seen the wiki but wanted specific feedback on the methodology (or perhaps the lack thereof . . . ) of this particular source, and you've provided it. Sometimes I think that there is a tendency today, due to the wide availability of media, for sources to market themselves by making very bold statements and then they figure they can deal with the substance at some later point.
Last edited by Dagwood on Tue Apr 19, 2011 12:59 pm, edited 1 time in total.
grayfox
Posts: 5569
Joined: Sat Sep 15, 2007 4:30 am

Post by grayfox »

matt wrote:Reposting what I wrote on April 1, 2010. (Fitting that we were discussing this topic on April Fool's day!) http://www.bogleheads.org/forum/viewtop ... ht=#700479
matt wrote:I think the CPI is a little bogus. I think ShadowStats is massively bogus.

Consider that one of the main arguments from ShadowStats prior to the housing bust was that owners equivalent rent (OER) was substituted for home prices, greatly subduing the CPI over the late 90’s up to 2006. But then housing prices tumbled and "true" inflation, according to what ShadowStats was saying back then, should have turned deeply deflationary, whereas CPI only dipped modestly because rents held relatively steady.

So on the chart here (http://www.shadowstats.com/alternate_da ... ion-charts), we should see a period in 2007 or 2008 where the SGS Alternate CPI drops below the Official CPI-U. At the very least, the gap between the two should close significantly. But what do we see? No matter what was happening with home prices in relation to OER, the Official CPI-U is dramatically lower than SGS Alternate CPI each year.

So I'm calling bulls**t. My guess is that ShadowStats is massaging (perhaps manhandling is the more appropriate term) the data so that no matter what happens, its version of inflation will always come out substantially higher. I also think it is preying on the fact that many people A) aren’t good at math, and B) don’t trust the Government. The idea that CPI-U has understated inflation by 5-10% per year for the past 20+years is demonstrably false.
Update for the year that has passed since then: housing continues to be very deflationary, yet Shadowstats chart still shows absolutely no evidence of this in its chart. I really wouldn't be surprised if they just make up the numbers.
There was a blogger who did calculate CPI using house prices instead of imputed rents. He took the CPI components and removed imputed rent and replaced it with Case-Shiller Housing Index. Sure enough, it showed really high inflation during the housing bubble years and then deflation for 2007-2010. Mish Shedlock wrote about it a few times.

As far as the other guy, he was challenged by some economists to show his CPI calculation so they could check it. His answer: there was nothing to show because there is no calculation. UCSD Professor James Hamilton reported on this on Econobrowser.

Here is how he comes up with 10% inflation. Every time the CPI was changed over the years, he came up with an estimate of how much it would reduce CPI. For instance when BLS went from house prices to imputed rents, maybe he estimated -2.5% reduction in inflation measurement. Then some other change -1%. Another change -0.5%. He adds up all these changes and they come to say -7%. Then when the BLS publishes CPI and says inflation is 2.8%, he adds 7% to it and the alternative measure he reports is 9.8% He doesn't collect any price data over time and weight various categories or anything. Just add an offset to the BLS inflation rate. :lol:

It is so bogus as to be laughable. That is why I laugh whenever I see him taken seriously by so many gullible buffoons like on CNBC, blogs, message boards.
matt
Posts: 2305
Joined: Sun Mar 04, 2007 3:47 pm

Post by matt »

grayfox wrote:There was a blogger who did calculate CPI using house prices instead of imputed rents. He took the CPI components and removed imputed rent and replaced it with Case-Shiller Housing Index. Sure enough, it showed really high inflation during the housing bubble years and then deflation for 2007-2010. Mish Shedlock wrote about it a few times.
A quick search finds that Floyd Norris had an article about this, verifying what we should expect to see: http://www.nytimes.com/2011/04/02/busin ... .html?_r=1. I'm sure others have done similar work. (Floyd's article was posted on April 1, 2011. This inflation discussion really is just a big Fool's game, after all.)
Here is how he comes up with 10% inflation... Every time the CPI was changed over the years, he came up with an estimate of how much it would reduce CPI. For instance when BLS went from house prices to imputed rents, maybe he estimated -2.5% reduction in inflation measurement. Then some other change -1%. Another change -0.5%. He adds up all these changes and they come to say -7%. Then when the BLS publishes CPI and says inflation is 2.8%, he adds 7% to it and the alternative measure he reports is 9.8% He doesn't collect any price data over time and weight various categories or anything. Just add an offset to the BLS inflation rate.
Are you saying that John Williams actually admitted that this is how he does it? He doesn't use any actual price data?
grayfox
Posts: 5569
Joined: Sat Sep 15, 2007 4:30 am

Post by grayfox »

Shadowstats Debunked
Shadowstats responds
I'm not going back and recalculating the CPI. All I'm doing is going back to the government's estimates of what the effect would be and using that as an ad factor to the reported statistics.
FredPeterson
Posts: 1025
Joined: Fri Jul 10, 2009 8:41 pm
Location: Madison, WI

Post by FredPeterson »

So the author says Shadowstats is debunked by something the BLS wrote.

There must be some logical fallacy term about "debunking" something using the very data that is being argued against in the first place.
User avatar
nisiprius
Advisory Board
Posts: 41964
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Post by nisiprius »

Apologies to matt for not crediting him as an early poster of the insight about Shadowstats' failure to reflect the housing price crash.
grayfox wrote:Shadowstats Debunked
Shadowstats responds
I'm not going back and recalculating the CPI. All I'm doing is going back to the government's estimates of what the effect would be and using that as an ad factor to the reported statistics.
You know what? I think of myself has having a pretty good idea for caveats, asides, footnotes, fine print, and weaseling... but I entirely missed the qualification in Shadowstat's description: "The CPI on the Alternate Data Series tab here reflects the CPI as if it were calculated using the methodologies in place in 1980." I misread it as "The CPI on the Alternate Data Series tab here reflects the CPI as calculated using the methodologies in place in 1980."

It still doesn't explain why the Alternate Data Series doesn't reflect the effect of the housing price crash.

I guess I will treat the Alternate Data Series as if it were bogus.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.
matt
Posts: 2305
Joined: Sun Mar 04, 2007 3:47 pm

Post by matt »

nisiprius wrote:"The CPI on the Alternate Data Series tab here reflects the CPI as if it were calculated using the methodologies in place in 1980." I misread it as "The CPI on the Alternate Data Series tab here reflects the CPI as calculated using the methodologies in place in 1980."
I'm not sure I see the difference in meaning between these two sentences. What is clear is that you, I, and many others actually thought that Williams was taking all of the the individual item price data from the BLS (and perhaps other sources for housing, etc.) and actually attempting to calculate what the older versions of CPI would produce today. We obviously gave him too much credit as he was not attempting this challenging task at all. He just grabbed some old estimates on the impact to calculation changes and has assumed them from then to infinity. No wonder his conclusions are so completely wrong; he's not even trying to do it right!

I hope his subscribers enjoy the $175 per year they contribute to this nonsense.
User avatar
nisiprius
Advisory Board
Posts: 41964
Joined: Thu Jul 26, 2007 9:33 am
Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry

Post by nisiprius »

matt wrote:
nisiprius wrote:"The CPI on the Alternate Data Series tab here reflects the CPI as if it were calculated using the methodologies in place in 1980." I misread it as "The CPI on the Alternate Data Series tab here reflects the CPI as calculated using the methodologies in place in 1980."
I'm not sure I see the difference in meaning between these two sentences.
Perhaps my irony was too subtle. :)
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.
Post Reply