Scheduled Maintenance: The site will be offline Tuesday, January 14, at 8:00 PM Eastern (01:00 UTC) for a forum software update. The update should take less than 1 hour.

Historical Bond Returns - From Rates to Returns [Bond Fund Simulator]

Discuss all general (i.e. non-personal) investing questions and issues, investing news, and theory.
Post Reply
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Historical Bond Returns - From Rates to Returns [Bond Fund Simulator]

Post by longinvest »

Added: Here are permanent links to the spreadsheet: Here's a thread to discuss how to properly calculate historical bond returns using Prof. Shiller's historical interest rate data (which goes back to 1871).

To summarize the problem: Shiller provides long-term (10-year government bond), and short-term (1-year government bill) historical rates. While we can reasonably expect a 1-year bill to deliver a 1-year total return equal to its interest rate (when bought at par), the same cannot be said of the 1-year total return of a 10-year bond. After one year, the total return of a 10-year bond is not only due to its coupon, but also to the change in value of its principal, as the bond has become a 9-year bond in a new rate environment, yet its coupon has not changed.

Now for the solution.

Naive Approach

Some people derive an annual total return for a 10-year bond as follows. They assume that the bond is bought at par and sold one year later, in a new rate environment. As the bond has a 9-year maturity, one year later, and 9-year rates are not available, they approximate the price of the bond using the new 10-year rate. Here's the actual calculation.

RB: 10-year rate when the bond is bought.
RS: 10-year rate when the bond is sold, one year later.
PV( interest_rate, number_payments, payment, future_value ): Financial function that calculates the present value of an investment.

1_year_bond_return = (coupon + sell_price) / buy_price = ( (1 X RB) - PV( RS, 9, 1 X RB, 1 ) ) / 1

Note that the PV() function returns a negative number, representing an amount paid for the investment, whereas payment and future_value are positive numbers, representing amounts received from the investment.

So, what's the problem, you'll ask me? Actually there are two problems.

The first one is obvious. As we use a 10-year rate to approximate a 9-year rate, we introduce an error in our calculation. In a rate environment where shorter-term rates are usually lower than longer-term rates (this has been the case since the 1930s), this leads the above formula to underestimate the value of the bond when it is sold. So, it's not too bad. Maybe the actual 1-year return using a proper 9-year rate would have been a tad higher.

The second problem is more serious. If we were to calculate the 1-year return of a 10-year bond using Shiller's historical rates and use them to represent historical bond returns, we would effectively be looking at the historical returns of a very special type of bond fund: a bond fund which buys a single 10-year bond each year and, one year later, receives a coupon, sells the now 9-year bond, and uses the proceeds to buy the new 10-year bond. Such a bond fund would have a high duration, much higher than the usual bond funds used by Bogleheads (e.g. short or intermediate-term bond funds). Higher duration implies higher volatility in the investor's safe part of his portfolio. That's not what we want to model.

A Simple Improvement

In the VPW backtesting spreadsheet, I tried to correct for the high duration of the naive approach by using a set of two bonds: a 10-year bond and a 1-year bill. So, the VPW's hypothetical bond fund works like this: each year, the fund uses half of its money to buy a 10-year bond and the other half to buy a 1-year bill. One year later, the fund receives two coupons, the capital of the bill, and sells the now 9-year bond on the market. It then uses the money to buy a new pair of bonds. The average maturity of the pair, at the beginning of the year, is (10 years + 1 year) / 2 = 5.5 years. At the end of the year, it's 4.5 years, for an overall average maturity of 5 years. Its duration* is a little lower, and thus in line with the duration of typical Bogleheads bond funds.

* Duration represents the weighted average maturity of all payments (coupons and principals), and is thus necessarily lower than average maturity for a fund of bonds with coupons.

Calculating an annual return for the bond-bill pair is simple: you take the average of the 1-year return of the bond and the rate of the bill.

1_year_bond_bill_return = (1_year_bond_return + bill_rate) / 2

You'll tell me: There ya go! You've got what you need. Leave me alone. Right?

But, I'm still annoyed. The bond-bill pair is not how a bond fund usually works. To top it off, it accumulates calculation errors due to the unavailability of 9-year rates.

A Better and Self-Correcting Approach

Remember the error due to using 10-year rates in lieu of 9-year rates? What if we could build a model where any such error would be guaranteed to be corrected over time? What if this self-correcting model was much closer to how bond funds actually work?

Here's my proposal.

We have both 10-year bond rates and 1-year bill rates. This means that we can precisely model the cash flows of a 10 year bond bought at par and sold one year prior to maturity. Here's an example:

In 1960, the 10-year bond rate was 4.72% and in 1969 the 1-year bill rate was 8.05%. So, in 1960, we could buy a 10-year government bond for $100 with an annual coupon of $4.72. In 1969, this bond still had a $4.72 coupon and $100 par value but had only one year left until maturity in a 8.05% 1-year bill environment. The value of this bond, in 1969, was thus:

value = -PV( 8.05, 1, 4.72, 100 ) = $96.92

To double check this, just consider that if you pay $96.92 and one year later you receive $4.72 (coupon) and $100 (principal), your return was: ($4.72 + $100) / $95.92 - 1 = 8.05%.

So, here is the precise sequence of cash flows (nothing is approximated):

1960: -$100.00
1961: $4.72
1962: $4.72
1963: $4.72
1964: $4.72
1965: $4.72
1966: $4.72
1967: $4.72
1968: $4.72
1969: $4.72 + $96.92 = $101.64

Neat, isn't it?

Now, we can go one step further. Like we could calculate the precise cash flows of a bond bought in 1960 and sold in 1969, we could also calculate the precise cash flows of a bond bought in 1961 and sold in 1970. And so on.

In other words, we could model a bond fund that worked like a bond ladder, except that it sold its bonds one year prior to maturity, like real-life bond funds do. This bond fund would, at the start of each year, contain 9 bonds maturing in 10, 9, 8, ..., 2 years. One year later the fund would sell its 1-year bond, combine the proceeds of the sale with all received coupons to buy a new 10-year bond.

What's nice about this model is that nothing is approximate. All coupons are reinvested at a known rate (the current 10-year rate at the time of reinvestment). All cash flows are thus precise, based on available historical data.

But, this still does not give us annual return data; it only gives us the internal cash flows of the bond fund. :annoyed

To calculate annual returns, we would need to calculate the present value of the bond fund, every year. To do that, we would need the historical 9-year, 8-year, ..., 2-year rates. But we don't.

What can we do? We can calculate approximate 9-year, 8-year, ..., 2-year rates and use them to do our present value calculations.

Won't that introduce errors in the calculated present value of the bond fund, every year? Yes, it will. But, the thing is that any error will necessarily be compensated by offsetting errors in later years. This is because all internal cash flows are precise. It simply cannot be otherwise. Don't you love mathematics? :happy

In other words, if we use approximate 9-year, 8-year, ..., 2-year rates to derive annual returns for our bond fund, any error will self-correct over time, within 9 years, to be precise. But other temporary errors will have been added for newer bonds, so, we'll never have exact historical annual returns at any point in time, but when looking at cumulative returns over longer periods, returns will become increasingly precise.

How do we come up with such approximate rates? I propose to simply use a linear approximation:

2_year_rate = 1_year_rate + (10_year_rate - 1_year_rate) X (2 - 1) / (10 - 1)
3_year_rate = 1_year_rate + (10_year_rate - 1_year_rate) X (3 - 1) / (10 - 1)
4_year_rate = 1_year_rate + (10_year_rate - 1_year_rate) X (4 - 1) / (10 - 1)
5_year_rate = 1_year_rate + (10_year_rate - 1_year_rate) X (5 - 1) / (10 - 1)
6_year_rate = 1_year_rate + (10_year_rate - 1_year_rate) X (6 - 1) / (10 - 1)
7_year_rate = 1_year_rate + (10_year_rate - 1_year_rate) X (7 - 1) / (10 - 1)
8_year_rate = 1_year_rate + (10_year_rate - 1_year_rate) X (8 - 1) / (10 - 1)
9_year_rate = 1_year_rate + (10_year_rate - 1_year_rate) X (9 - 1) / (10 - 1)

In summary, I am confident that such self-correcting returns would be of higher quality than those of the naive approach (as used by cFIREsim) or of the bond-bill pair approach (as used by the VPW backtesting spreadsheet).

I have implemented this in a spreadsheet and intend to include it in the next revision of the VPW backtesting spreadsheet.

Here are my questions:
  1. Is there any comment about the proposed approach?
  2. Would there be any interest to add such calculated pre-1972 bond returns, as well as calculated stock returns, based on Prof. Shiller's data set, into Simba's backtesting spreadsheet?
Last edited by longinvest on Mon Dec 16, 2024 3:23 pm, edited 10 times in total.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
User avatar
LadyGeek
Site Admin
Posts: 100596
Joined: Sat Dec 20, 2008 4:34 pm
Location: Philadelphia
Contact:

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by LadyGeek »

For the record: So, what's the problem?
longinvest wrote:So, what's the problem, you'll ask me? ...
This approach seems like a moving average. You start with a 9-year error, which goes to zero as you approach the 10 year reference period.

Why is using Shiller's index better than other methods? I'm not sure I understand the application to bond returns.
Wiki To some, the glass is half full. To others, the glass is half empty. To an engineer, it's twice the size it needs to be.
User avatar
siamond
Posts: 6050
Joined: Mon May 28, 2012 5:50 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by siamond »

longinvest wrote:Here are my questions:
  1. Is there any comment about the proposed approach?
  2. Would there be any interest to add such calculated pre-1972 bond returns, as well as calculated stock returns, based on Prof. Shiller's data set, into Simba's backtesting spreadsheet?
Big YES to question #2. It would be really useful to have a single reference we agree on, and can refer to for stocks & bonds long-term historical returns, and related stats.

Actually, I'd go one step further, and suggest that we format it in such a way that it is really easy for people with access to other types of historical data (e.g. UK returns, SBBI/Ibbotson US returns, French-Fama factor-oriented returns, DMS dataset, etc) to plug such numbers in their own version of this oh-so-useful spreadsheet.
Last edited by siamond on Sat Dec 12, 2015 3:58 pm, edited 1 time in total.
User avatar
gordoni2
Posts: 213
Joined: Wed Aug 15, 2007 12:20 am
Contact:

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by gordoni2 »

Shiller's bond interest rate data are monthly, so it is possible to compute monthly instead of annual bond returns, and then collapse the monthly returns down to annual returns. You would then be using a 10 year rate in your calculation, instead of a 9 year 11 month rate, and the difference would be negligible. I do something like this in AACalc.

They possibly should be, but I bet a majority of people aren't holding just Treasury bonds, but a mixture of treasury (17%:proportion of household holdings), agency (8%), municipal (25%), and corporate bonds (50%). They will thus experience different returns, durations, and volatilities than 10 year constant maturity Treasuries. It is hard to know what to do to make things more reasonable. In AACalc I punted, and applied a +0.7% increase in returns over the 10 year constant maturity returns and a 10% increase in volatility in order to get returns indicative of the US investment grade bond universe. I put a lot of thought into this. The precise values I used are hard to justify, but I feel they are roughly correct.
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

LadyGeek,

As usual, you're asking very good questions. Let me try to answer them in reverse order.
LadyGeek wrote:Why is using Shiller's index better than other methods? I'm not sure I understand the application to bond returns.
It's not a question of method, but a question of having historical data for a longer time span! Simba's spreadsheet only contains data for the period spanning from 1972 to 2014, for a total of 43 years. Shiller's data set spans from 1871 to 2012, for a total of 142 years.

While I highly prefer the Simba's spreadsheet data, as it is based (mostly) on published actual after-fee mutual fund annual returns, and I wouldn't replace this data with returns computed from rates. I think that adding 1871-1971 returns into Simba's spreadsheet would be a great addition, giving a data set covering a total of 144 years. It's not much (that's less than 5 data points to backtest a 30-year retirement), but it's more than the current data (that contains less than 1.5 data point to backtest a 30-year retirement).
LadyGeek wrote:This approach seems like a moving average. You start with a 9-year error, which goes to zero as you approach the 10 year reference period.
No, it's technically better. For one thing, we have actual 1-year rates, so we don't need to wait until the 10th year. The error is 0 on the 9th year. I've just gained one year of precision over your estimate; Yay! :wink:

The other point is that the yield curve is not random. So, using a linear approximation, while far from perfect, cannot stray very far from reality.

I invite you to look at the yield curve animation at http://stockcharts.com/freecharts/yieldcurve.php . Imagine that you knew the exact 1-year and the exact 10-year rate at any point of the animation. How far is a linear approximation of intermediate rates from reality?

So, while results won't be exact, they should be good enough for running good enough backtests. Isn't that the whole objective?
Last edited by longinvest on Sat Dec 12, 2015 4:46 pm, edited 1 time in total.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

gordoni2 wrote:Shiller's bond interest rate data are monthly, so it is possible to compute monthly instead of annual bond returns, and then collapse the monthly returns down to annual returns. You would then be using a 10 year rate in your calculation, instead of a 9 year 11 month rate, and the difference would be negligible. I do something like this in AACalc.
I do my calculations using his easier-to-use annual data sheet: http://www.econ.yale.edu/~shiller/data/chapt26.xls

It would be possible to go down to the monthly level, but he tends to provide yearly averages on a monthly basis, which is kind of tricky to use correctly. I really prefer the simplicity of his annual spreadsheet.

I don't think that going all the trouble to use his monthly data would make a big improvement.
gordoni2 wrote:They possibly should be, but I bet a majority of people aren't holding just Treasury bonds, but a mixture of treasury (17%:proportion of household holdings), agency (8%), municipal (25%), and corporate bonds (50%). They will thus experience different returns, durations, and volatilities than 10 year constant maturity Treasuries.
I agree with the underlined parts.

Note that I am proposing to emulate a bond fund, not a 10 year constant maturity Treasury! The later is what cFIREsim does. As I explained in my OP, the VPW backtesting spreadsheet simulates a bond/bill pair to reduce the average maturity and duration to something more sensible for a Boglehead.

My proposed bond fund has a variable average maturity not too far from 5.5 years. (It starts at (2 + 3 + ... + 10) / 9 = 6 years and ends at (1 + 2 + ... + 9) / 9 = 5 years). Actually, as the bonds are not equal, where longer-term bonds are usually bigger (bought every year using not only matured capital but also the whole set of coupons), the weighted average maturity is likely somewhat higher, but the duration should end up not too far from 5 or 5.5. That is quite representative of the average duration of bond funds used by Bogleheads.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
User avatar
siamond
Posts: 6050
Joined: Mon May 28, 2012 5:50 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by siamond »

longinvest wrote:I do my calculations using his easier-to-use annual data sheet: http://www.econ.yale.edu/~shiller/data/chapt26.xls
Would be cool if you could use Pr Shiller latest and greatest: http://www.econ.yale.edu/~shiller/data/ie_data.xls

Yes, this is monthly, but it is really not hard to get to the annual level, much simpler than the rest of the work you've already accomplished. This would also make it easier to update every now and then.
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

siamond wrote:Would be cool if you could use Pr Shiller latest and greatest: http://www.econ.yale.edu/~shiller/data/ie_data.xls

Yes, this is monthly, but it is really not hard to get to the annual level, much simpler than the rest of the work you've already accomplished. This would also make it easier to update every now and then.
We only need to do this for pre-1972 years. Why would we ever use calculated returns in a post 1986 year, now that we have a real bond fund's actual returns, with tracking errors and all? I mean Vanguard's Total Bond Market Index Fund (VBMFX), of course.

Anyway, I can provide my spreadsheet. Replacing the historical annual rates is easy enough. The spreadsheet will automatically compute new returns out of them.

Shiller has so conveniently provided annual data for 1871-2012. I'm just not sure I could do a better job than Shiller himself to calculate historical annual rates.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

If anybody is interested, here's a temporary link the Bond Fund Returns From Shiller Rates spreadsheet. The spreadsheet was developed using LibreOffice and is in its native format.

Added: Here are permanent links to the spreadsheet:
Last edited by longinvest on Wed Dec 16, 2015 12:13 pm, edited 4 times in total.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
User avatar
LadyGeek
Site Admin
Posts: 100596
Joined: Sat Dec 20, 2008 4:34 pm
Location: Philadelphia
Contact:

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by LadyGeek »

longinvest wrote:
LadyGeek wrote:This approach seems like a moving average. You start with a 9-year error, which goes to zero as you approach the 10 year reference period.
No, it's technically better. For one thing, we have actual 1-year rates, so we don't need to wait until the 10th year. The error is 0 on the 9th year. I've just gained one year of precision over your estimate; Yay! :wink:

The other point is that the yield curve is not random. So, using a linear approximation, while far from perfect, cannot stray very far from reality.

I invite you to look at the yield curve animation at http://stockcharts.com/freecharts/yieldcurve.php . Imagine that you knew the exact 1-year and the exact 10-year rate at any point of the animation. How far is a linear approximation of intermediate rates from reality?

So, while results won't be exact, they should be good enough for running good enough backtests. Isn't that the whole objective?
OK, then let's look at this as periodic sampling. Annual results will miss inflection points and increase your error.

The animated yield curve provides a good example. In Dynamic Yield Curve (your link), set the right-side red bar to the 2009 yield curve minimum (2 March 2009). Set the trail length to maximum. Observe the range of the yield curve at 10 Y is approximately 1 %.

Slide the trail length to the left (minimum) and observe the range of the yield curve at 10 Y is 0% (single point). Reducing your time period (increasing your periodic sampling) will reduce your error.

gordoni2 is suggesting to start with monthly data, which should improve accuracy. Let's think about the theory first, difficulty of implementation is next.

What level of accuracy is needed to be "good enough"?
Wiki To some, the glass is half full. To others, the glass is half empty. To an engineer, it's twice the size it needs to be.
User avatar
LadyGeek
Site Admin
Posts: 100596
Joined: Sat Dec 20, 2008 4:34 pm
Location: Philadelphia
Contact:

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by LadyGeek »

longinvest wrote:If anybody is interested, here's a temporary link the Bond Fund Returns From Shiller Rates spreadsheet. The spreadsheet was developed using LibreOffice and is in its native format.
OK, now I see how much data this is. Perhaps annual isn't so bad.

If this is a one-time calculation, perhaps it would be more efficient to analyze this in "R", then post the results in the spreadsheet? I'm not volunteering, but making a suggestion.
Wiki To some, the glass is half full. To others, the glass is half empty. To an engineer, it's twice the size it needs to be.
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

LadyGeek wrote:OK, then let's look at this as periodic sampling. Annual results will miss inflection points and increase your error.
You're right. But the annual rates I'm using are annual averages calculated by Shiller; they're not a single sample at a specific date each year.

Even his monthly data (in his newer spreadsheet) are annual averages. So, you don't actually get monthly samples, out of them. To use them properly can get quite tricky. I'm just trying to avoid a headache, if possible.

I'm already assuming bonds with annual coupons, requiring 9 columns per bond calculation. If you want to get down to monthly investments with semi-annual coupons, you'll not only have to approximate annual rates, but monthly rates, like in 13-months bond, 14-months bond, etc.

It's feasible, but it would be a pain for what I suspect is not much gain. We already have so many assumptions, just in how the fund itself is constructed, that our calculated returns will necessarily be different from real-life.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
User avatar
siamond
Posts: 6050
Joined: Mon May 28, 2012 5:50 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by siamond »

longinvest wrote:
siamond wrote:Would be cool if you could use Pr Shiller latest and greatest: http://www.econ.yale.edu/~shiller/data/ie_data.xls

Yes, this is monthly, but it is really not hard to get to the annual level, much simpler than the rest of the work you've already accomplished. This would also make it easier to update every now and then.
We only need to do this for pre-1972 years. Why would we ever use calculated returns in a post 1986 year, now that we have a real bond fund's actual returns, with tracking errors and all? I mean Vanguard's Total Bond Market Index Fund (VBMFX), of course.

Anyway, I can provide my spreadsheet. Replacing the historical annual rates is easy enough. The spreadsheet will automatically compute new returns out of them.

Shiller has so conveniently provided annual data for 1871-2012. I'm just not sure I could do a better job than Shiller himself to calculate historical annual rates.
In most cases, I would agree with you, using real-life returns from available bond funds is preferable. But in some cases, one might not want to mix apples and oranges. So I'd rather provide the full data derived from Shiller without guessing how it will be used. Then if people want to use such data for pre-1972 and regular funds data for 1972+, well, that's up to them.

Case in point: what we're doing right now! As it would be great to compare your results (Shiller-derived) with actual returns (either from SBBI, or from Vanguard), up to 2014 (or later when available). And yes, I am working on this... :wink:

It's really easy to address, I'll send you the 2012 to 2014 Shiller-derived annual returns if you wish, after double-checking I'm doing the math like Shiller does.

PS. also some historical numbers are occasional amended, due to new historical findings from CRSP. There was a case like that, recently.
User avatar
siamond
Posts: 6050
Joined: Mon May 28, 2012 5:50 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by siamond »

Actually, as I was doing such comparison with actuals (using either Simba or SBBI), I stumbled upon a small snag.

I wasn't too eager to use VBMFX, because this is a total-bonds fund, not a treasury fund. Luckily, the Simba spreadsheet provides the returns for VFITX, which is about treasuries. But then it is labeled as "5 years". And indeed, I checked on the Vanguard site, and that is the typical maturity (5.6 years to be exact) and duration (5.2 years to be exact).

This made me ponder what does "Intermediate Term" exactly mean? Coming back to my SBBI book, they define it in the same way as Vanguard, I quote, "maturity near 5 years". And yet, Shiller only provides the historical 10 years treasury rate. Hm. :?

PS. actually, the OP's original technique in VPW ended up with a 5.5 years duration, I believe. What about the new computation, I suspect it's actually 10 years duration, isn't it?
Last edited by siamond on Sat Dec 12, 2015 7:47 pm, edited 1 time in total.
User avatar
LadyGeek
Site Admin
Posts: 100596
Joined: Sat Dec 20, 2008 4:34 pm
Location: Philadelphia
Contact:

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by LadyGeek »

longinvest wrote:But the annual rates I'm using are annual averages calculated by Shiller; they're not a single sample at a specific date each year.
^^^ The key to the entire problem is solved by adding one word - "averages" (or at least better understood).

For new investors, the wiki has some background info: Robert Shiller, which leads to his Home Page.

The Online Data section provides a complete explanation of his averaging methods. The spreadsheet used by longinvest is U.S. Stock Markets 1871-Present and CAPE Ratio.

Shiller also makes available long term stock, bond, interest rate and consumption data.

I compared the Long Interest Rate "Data" in both spreadsheets. One spreadsheet (US Stock markets...) uses "Long Interest Rate GS10", the other (long term stock...) uses "Long Government Bond Yield 10 Year Post 53". I don't understand what these ratings mean, but the rates do not match each other (compare 1871+). Does it matter?

"Long Government Bond Yield 10 Year Post 53" is shown in longinvest's spreadsheet, so perhaps the rates came from the 2nd spreadsheet.
Wiki To some, the glass is half full. To others, the glass is half empty. To an engineer, it's twice the size it needs to be.
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

siamond wrote:PS. actually, the OP's original technique in VPW ended up with a 5.5 years duration, I believe. What about the new computation, I suspect it's actually 10 years duration, isn't it?
No, of course not! Duration must be lower than the average maturity of the fund.

There are more than one definition for duration, all pretty close to each other, but with subtle differences. The Macaulay duration is defined as: The weighted average term to maturity of the cash flows from a bond. The weight of each cash flow is determined by dividing the present value of the cash flow by the price, and is a measure of bond price volatility with respect to interest rates. (Investopedia)

If that sounds like gibberish (it used to, for me), let's simply say that duration is the average maturity of all of a bond's payments. If we take a single 2-year bond, for example, we know that it has three payments: one coupon payment in 1 year (small), another in 2 years (small), and a principal payment in 2 years (big). So, if we were to calculate the (weighted) average maturity of all payments of this 2-year bond, we'd get something like 1.9x years (depending on the size of the coupon). For a bond fund, we'd have to make a somewhat more elaborate calculation, but it is sure that the duration will necessarily be shorter than the average maturity of the fund. (The average maturity of a fund measures the weighted average maturity of its bonds, not of its payments).

As an approximation of the average maturity of our bond fund, we can calculate an average maturity assuming equal-size bonds. We know that, at the start of the year, the fund contains 9 bonds, with maturities ranging from 10 years to 2 years. The average is thus: (10 + 9 + ... + 2) / 9 = 54 / 9 = 6 years. At the end of the year, all 9 bonds will have lost one year, so the average will be 5. So, on average, during the year, the bond fund had an average maturity of: (6 + 5) / 2 = 5.5 years. As I said, duration must be lower than average maturity.

Another definition of duration (relatively equivalent) is the loss in value if all rates went up by 1%. Using the spreadsheet I provided, it is quite easy to calculate this. If I take the year 2012 as an example. The 10-year rate is 1.97% and the 1-year rate is 0%, and intermediate rates are a linear approximation based on these two rates. The bond fund's value is $492,597.74. If I increase both rates to 2.97% and 1%, respectively, and let the spreadsheet approximate the intermediate rates and compute the fund's value, it gets down to $470,357.01. That's a loss of (470,357.01 / 492,597.74) - 1 = -4.5%. In other words, we can say that in 2012, the fund's duration is approximately 4.5 years, much, much lower than your 10 years!

I do believe that this bond fund is a much better model for deriving historical bond returns that the bond/bill pair used in the current VPW spreadsheet. The model is much closer to how real-life bond funds work.

Side thought

When you think about the bond market as a whole, you realize that its has to resemble the proposed model. All bond investors, in aggregate, hold all existing bonds. When they receive coupon and principal payments, they can't, in aggregate, reinvest them into these existing bonds! So, if they decide to reinvest principal and coupon payments in bonds, they have to invest them into new bonds!
Last edited by longinvest on Sat Dec 12, 2015 11:38 pm, edited 2 times in total.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

LadyGeek wrote:"Long Government Bond Yield 10 Year Post 53" is shown in longinvest's spreadsheet, so perhaps the rates came from the 2nd spreadsheet.
Yes, I do use the second spreadsheet, because it provides annual data. It is much easier to use than the weirdly provided annual-averages per month of Shiller's newer data set.

Actually, I just went ahead and downloaded the newer data set. This reminded me of why I chose to use the older data set.

I was confused. It's not true that using the annual averages of the new sheet is more difficult. It's actually quite simple: you simply use the January number of every year and you are done. (You can compare the two spreadsheets, and you'll see it).

The real reason I use the older data set is because it provides 1-year rates, which are missing in the new data set. This allows me to calculate the selling price of bonds one year prior to maturity, without any need to approximate any rate, in my bond fund model. As I explained in the OP, all cash flows are precise and don't depend on any approximate rate*.

* There's a small exception. In order to start the fund, in 1871, approximate cash flows are needed. Also, in 2012, 8 bonds must be sold on the open market, and this requires approximate rates. Other than that, all other cash flows during the 140+ years are precise.
Last edited by longinvest on Sat Dec 12, 2015 11:28 pm, edited 3 times in total.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
User avatar
siamond
Posts: 6050
Joined: Mon May 28, 2012 5:50 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by siamond »

longinvest wrote:So, on average, during the year, the bond fund had an average maturity of: (6 + 5) / 2 = 5.5 years. As I said, duration must be lower than average maturity.
I actually meant 'maturity' in my previous post instead of 'duration', my fingers got a hold of myself (excuses, excuses!). Still, thanks for the explanation. Ok, then, we have a good match between the average maturity of your 'bond fund' and the way SBBI and Vanguard define "IT" bonds.

Will show some numbers comparing actuals to this 'bond fund' tomorrow.
User avatar
siamond
Posts: 6050
Joined: Mon May 28, 2012 5:50 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by siamond »

LadyGeek wrote:I compared the Long Interest Rate "Data" in both spreadsheets. One spreadsheet (US Stock markets...) uses "Long Interest Rate GS10", the other (long term stock...) uses "Long Government Bond Yield 10 Year Post 53". I don't understand what these ratings mean, but the rates do not match each other (compare 1871+). Does it matter?
Ladygeek, yes, they do match. The numbers in the annual spreadsheet (the one longinvest used) correspond to the January numbers in the monthly spreadsheet (ie_data). It's just that the former isn't updated as often as the latter. And by the way, I provided the latest January numbers (from the monthly spreadsheet) to longinvest, so that he can easily extend his model by a couple of years.
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

siamond wrote:And by the way, I provided the latest January numbers (from the monthly spreadsheet) to longinvest, so that he can easily extend his model by a couple of years.
Did you find 1-year rates? I don't seem to find them. :(
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
User avatar
siamond
Posts: 6050
Joined: Mon May 28, 2012 5:50 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by siamond »

longinvest wrote:
siamond wrote:And by the way, I provided the latest January numbers (from the monthly spreadsheet) to longinvest, so that he can easily extend his model by a couple of years.
Did you find 1-year rates? I don't seem to find them. :(
Well, I was just struggling with that. They are not in the ie-data spreadsheet, which is annoying. I thought that this would be easily addressed by checking multpl.com, which has all the Shiller data and much more, updated on a daily basis.
http://www.multpl.com/1-year-treasury-rate/table

BUT.

The 10-yrs treasury rate numbers do match the Shiller numbers, but NOT the 1-yr treasury rate (from the annual spreadsheet). Call me a bit baffled, orders of magnitude are similar, but this is always off by a bit, and sometimes by half a point. :confused

PS. anyhoo, recent values are so close to zero and so flat that using multpl.com will not introduce any meaningful error, I suspect.
User avatar
Kevin M
Posts: 16281
Joined: Mon Jun 29, 2009 3:24 pm
Contact:

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by Kevin M »

Seems like a reasonable approach. Despite various possible criticisms, it does the intended job of providing an approximation of something that is closer to an intermediate-term Treasury fund.

I don't think there's anything wrong with using the "naive" approach if you want to approximate historical 10-year bond returns, but as noted, this is not typical of a Boglehead bond holding.

Kevin
If I make a calculation error, #Cruncher probably will let me know.
User avatar
Kevin M
Posts: 16281
Joined: Mon Jun 29, 2009 3:24 pm
Contact:

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by Kevin M »

longinvest wrote: Did you find 1-year rates? I don't seem to find them. :(
Do Shiller's rates match CMT rates from FRED? If not, why not?

Why not just use 1-year CMT from FRED for more recent years?

Kevin
If I make a calculation error, #Cruncher probably will let me know.
User avatar
LadyGeek
Site Admin
Posts: 100596
Joined: Sat Dec 20, 2008 4:34 pm
Location: Philadelphia
Contact:

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by LadyGeek »

To help with the acronyms:

FRED = Federal Reserve Economic Data - FRED - St. Louis Fed

CMT = Constant Maturity Rate

Helpful link: CMT Search Results - FRED - St. Louis Fed
Wiki To some, the glass is half full. To others, the glass is half empty. To an engineer, it's twice the size it needs to be.
User avatar
JoMoney
Posts: 16260
Joined: Tue Jul 23, 2013 5:31 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by JoMoney »

You can get Ibbotson annual returns data on Corporate or Government bonds back to 1926 in the DFA Matrix book
https://www.ifa.com/book-library/3408/D ... ook__2015/
Or even Morningstar
IA SBBI US IT Govt TR USD
IA SBBI US LT Govt TR USD
"To achieve satisfactory investment results is easier than most people realize; to achieve superior results is harder than it looks." - Benjamin Graham
User avatar
siamond
Posts: 6050
Joined: Mon May 28, 2012 5:50 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by siamond »

I did the comparison with actuals, using either my SBBI book (1926+) or the Simba (1972+). Longinvest suggested that I assemble a "telltale chart" (something Mr Bogle likes to use), here it is:

Image

Image

I also assembled the usual metrics, R2, RMSE and CAGR delta. Without boring you with details, the R2/RMSE are ok without being stellar, while the CAGR is really close (slightly underestimated). Overall, a definite improvement from the VPW previous math.

PS. see this seminal article from John Bogle on how a Telltale chart works. It's basically the ratio between cumulative returns of the two quantities being compared: https://personal.vanguard.com/bogle_sit ... 20626.html
User avatar
siamond
Posts: 6050
Joined: Mon May 28, 2012 5:50 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by siamond »

JoMoney wrote:You can get Ibbotson annual returns data on Corporate or Government bonds back to 1926 in the DFA Matrix book
https://www.ifa.com/book-library/3408/D ... ook__2015/
Actually, it only includes LT Corporate or Government bonds, but unfortunately not Intermediate Term Government bonds. Or am I missing something?
JoMoney wrote:Or even Morningstar
IA SBBI US IT Govt TR USD
Aaah, that works... I just checked a few old annual numbers from my SBBI book, and a few recent ones, and it works... Admittedly tedious to get them one at a time from Morningstar, but... does that mean we can use such numbers without any IPR issue with the SBBI editor? Not too sure the Morningstar copyright allows something like that for our own purposes? the Bogleheads site is non-commercial, but not exactly private, right?

Copyright
All Information available through "Morningstar.com" is the property of Morningstar, Inc. or its Information Providers and is protected by copyright and intellectual property laws. All rights reserved.

You may not reproduce, retransmit, disseminate, sell, publish, broadcast, nor shall the Information be used in connection with creating, promoting, trading, marketing Investment Products without the express written consent of Morningstar, Inc or its Information Providers. You are entitled to use the Information it contains for your private, non-commercial use only.
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

Siamond,

Nice charts!

I'm actually surprised at how close the fund matches SBBI intermediate government bonds in the 1982-2014 period. It must be a side effect of the yield curve during the period, as the composition of the funds are different, given that an intermediate fund wouldn't normally be holding bonds with less than 3 years to maturity.

Visually, the fund model seems to deliver representative-enough returns, for a Boglehead-type bond fund (intermediate). I think that we can easily agree that it is more representative than a constant-maturity 10-year treasury fund or a bond/bill pair. Would be nice if it was in Simba's spreadsheet, so that I could simplify the VPW spreadsheet, and leave the complex calculations to the Simba spreadsheet people. ;)
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
User avatar
LadyGeek
Site Admin
Posts: 100596
Joined: Sat Dec 20, 2008 4:34 pm
Location: Philadelphia
Contact:

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by LadyGeek »

With scaling in mind, the worst-case delta between those charts is 0.3% - the "outlier" is VPW I-Bonds.

Here is the forum's copyright notice: Quoting outside material

The wiki's copyright notice is here:

Image

In both cases, material that is not your own needs to respect the copyright of the external site.
Wiki To some, the glass is half full. To others, the glass is half empty. To an engineer, it's twice the size it needs to be.
User avatar
siamond
Posts: 6050
Joined: Mon May 28, 2012 5:50 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by siamond »

LadyGeek wrote:In both cases, material that is not your own needs to respect the copyright of the external site.
Yes, I agree, my question was a tad rhetorical... Maybe Morningstar doesn't care too much, but (as we discussed in the past), SBBI/Ibbotson does care for sure, and they are the real source here. So I think that the attempt being made here to derive useful return numbers from Shiller (and maybe from FRED as Kevin suggested) remains valid.
User avatar
siamond
Posts: 6050
Joined: Mon May 28, 2012 5:50 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by siamond »

Kevin M wrote:
longinvest wrote: Did you find 1-year rates? I don't seem to find them. :(
Do Shiller's rates match CMT rates from FRED? If not, why not?

Why not just use 1-year CMT from FRED for more recent years?

Kevin
Shiller's 10-yrs rates do match the (January) monthly values from Fred. Shiller's 1-yr rates do NOT (don't know why).

So I followed Kevin's advice and used the 1-year from FRED for 1954+, and plugged that into longinvest's spreadsheet, replacing corresponding numbers from Shiller. Lo and behold, this nicely improves the comparison with the actuals.

The telltale charts are a tad better, although it's not that noticeable, but the key metrics definitely improve (R2 is now hovering around 0.92/0.93, RMSE decreased by half a point or so, and even the CAGR accuracy improved a tad further). COOL.
User avatar
JoMoney
Posts: 16260
Joined: Tue Jul 23, 2013 5:31 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by JoMoney »

Public response to a PM because others might be interested
siamond wrote:... How did you find those series? Searching the Morningstar site by keywords (e.g. SBBI) doesn't seem to work well...
First thing you need is the URL to get a Morningstar Growth chart:

Code: Select all

http://quotes.morningstar.com/chart/fund/chart.action?t=XXXXX
The XXXXX should be replaced with a stock (or Mutual Fund) ticker or the Morningstar ID for some index or data you're looking to build a chart for.
For example, the below URL should take you to a growth chart (including dividends) for PG stock - "Procter & Gamble Co" shares

Code: Select all

http://quotes.morningstar.com/chart/fund/chart.action?t=PG
Stock or mutual fund tickers are pretty easy to come by, the Morningstar codes for various index series are a little trickier.
If you do Google searches you can find documents with lists of the codes/ID for various indices like in this document:
http://morningstardirect.morningstar.co ... 290509.xls

If you have some idea what you're looking for, an easier way to get the code is to go to Morninstar's homepage, just type in some keywords like "S&P" or "Wilshire" or "Ibbotson" and it should start narrowing it down.
Image
The below URL should take you to a "Wilshire 5000 Total Mkt TR USD" Growth Chart

Code: Select all

http://quotes.morningstar.com/chart/fund/chart.action?t=FOUSA08WXH
The more advanced step, is if you want to use the data from the chart in your own spreadsheets without manually typing in the dates and manually typing the values. To do that you would need to use your browsers debugger tools and watch as the chart loads to find the link to the script pulling the chart data. For someone with somewhat advanced web skillz it's quite doable. It will spit out all the dates and divisor values somewhat cleanly, but you still might need some more advanced understanding to make a script or use excel or some program to parse the data and clean it up to a format that is easily usable in a spreadsheet... I've done it using some rudimentary javascript on my Chromebook. It doesn't take a lot of work if you're already familiar with scripting and techy stuff... but is more than I want to go through step by step...

Code: Select all

http://mschart.morningstar.com/chartweb/defaultChart?type=getcc&secids=FOUSA08WXH;XI&dataid=117&startdate=1900-01-01&enddate=2015-12-14&currency=&format=1&adjusment=-1&callback=jQuery16408879255673382431_1450084579168&_=1450084608816
"To achieve satisfactory investment results is easier than most people realize; to achieve superior results is harder than it looks." - Benjamin Graham
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

Deleted.

See Kevin M's post, instead: viewtopic.php?f=10&t=179425&p=2719088#p2719066
Last edited by longinvest on Mon Dec 14, 2015 1:57 pm, edited 2 times in total.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

siamond wrote:
Kevin M wrote: Do Shiller's rates match CMT rates from FRED? If not, why not?

Why not just use 1-year CMT from FRED for more recent years?

Kevin
Shiller's 10-yrs rates do match the (January) monthly values from Fred. Shiller's 1-yr rates do NOT (don't know why).
This is puzzling, indeed. Why don't Shiller's 1-year rates match CMT rates from FRED? Anybody has an idea?
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
User avatar
Kevin M
Posts: 16281
Joined: Mon Jun 29, 2009 3:24 pm
Contact:

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by Kevin M »

longinvest wrote: Why all this? Just to explain why duration is expressed in years, instead of as a percentage. Yet, we always hear that duration tells us about the loss (a percentage) when interest rates go up. This did intrigue me for a long time.
I didn't review all of your math, but the proper units/dimensions for the relevant duration measures are different.

Macaulay duration is the weighted average maturity of cash flows, and the units are years (dimension is time).

Modified duration is percent change in price per percentage point change in yield, so you could say that the units are %/pp. According to Wikipedia, modified duration is formally a semi-elasticity, the percent change in price for a unit change in yield, which I guess ends up having units of percent, or at least justifies expressing it this way.

Macaulay duration and modified duration are very close in value, especially at low interest rates. Given the uncertainty in the ways we typically use duration, I don't think it matters much which you use.

Vanguard expresses duration with units of years, yet they define it using the modified duration definition. A few years ago I messaged my flagship representative, pointed out the discrepancy, and asked which duration measure they actually were showing, and I was never able to get a clear answer.

I admit that I sometimes just avoid the issue by not stating units when I refer to duration.

Kevin
If I make a calculation error, #Cruncher probably will let me know.
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

Kevin M wrote: I didn't review all of your math, but the proper units/dimensions for the relevant duration measures are different.
OK. I erased my post.

Wikipedia gives the precise formula for the relation between the Macaulay duration and the Modified duration.
Last edited by longinvest on Mon Dec 14, 2015 11:57 am, edited 2 times in total.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
User avatar
Kevin M
Posts: 16281
Joined: Mon Jun 29, 2009 3:24 pm
Contact:

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by Kevin M »

longinvest wrote: This is puzzling, indeed. Why don't Shiller's 1-year rates match CMT rates from FRED? Anybody has an idea?
For you Shiller data hounds, doesn't Shiller define his data methods somewhere?

The CMT yields are derived using a published method (at least in summary). Shiller could very well have come up with his own method, but it is puzzling that he'd use the Treasury/FRED method for 10-year maturity and something else for 1-year maturity. FRED does not provide data as far back as Shiller, so Shiller must have used a different method for older data, but it seems that using a reliable, public source like FRED for newer data makes sense.

Hmm, in looking over the CMT method, I see this:
Between August 6, 2004 and June 2, 2008, to reduce volatility in the 1-year Treasury Constant Maturity (CMT) rate, and due to the fact that there were no on-the-run issues between 6-months and 2-years, Treasury used an additional input to insure that the 1-year CMT rate was consistent with on-the-run yields on either side of it's maturity range. Thus, Treasury interpolated between the secondary bond equivalent yield on the most recently auctioned 26-week bill and the secondary market yield on the most recently auctioned 2-year note and inputted the resulting yield as an additional knot point for the derivation of the daily Treasury Yield Curve. The result of that step was that the 1-year CMT was generally the same as the interpolated rate during that time period. As of June 3, 2008, the interpolated yield was dropped as a yield curve input and the on-the-run 52-week bill was added as an input knot point in the quasi-cubic hermite spline algorithm and resulting yield curve.
Also note this:
Treasury does not provide the computer formulation of our quasi-cubic hermite spline yield curve derivation program. However, we have found that most researchers have been able to reasonably match our results using alternative cubic spline formulas.
Kevin
If I make a calculation error, #Cruncher probably will let me know.
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

Kevin M wrote:
longinvest wrote: This is puzzling, indeed. Why don't Shiller's 1-year rates match CMT rates from FRED? Anybody has an idea?
For you Shiller data hounds, doesn't Shiller define his data methods somewhere?
Maybe I should have read what prof. Shiller wrote in the spreadsheet itself:
The Federal Reserve Board discontinued its 6-month commercial paper rate series August 1997.
After that, the 6-month Certificate of Deposit rate, secondary market, is used,
The series R is the result of converting the January and July rates into an annual yield.
So, Shiller's 1-year rates (Series R) are the result of a calculation based on 6-month commercial paper which has credit risk, and thus probably higher rates.

They are not 1-year government bond rates!
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

I guess that we should also investigate what the long rates (RLONG) really are, before 1953, just to make sure we're not abusing the data too much.

In his spreadsheet, Shiller writes:
RLONG is the 10-year Treasury after 1953
Before 1953, it is government bond yields from Sidney Homer A History of Interest Rates.
See Shiller, Irrational Exuberance 2nd Edition, 2005 for description of data
Anybody has Sidney Homer's A History of Interest Rates on his nightstand? And, of course, Shiller's Irrational Exuberance 2nd Edition?
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
Clive
Posts: 1950
Joined: Sat Jun 13, 2009 5:49 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by Clive »

longinvest wrote:In other words, we could model a bond fund that worked like a bond ladder, except that it sold its bonds one year prior to maturity, like real-life bond funds do.
Is there any evidence that selling a year prior to maturity (and incurring trading costs) is better than holding to maturity (avoiding trading costs)?
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

Clive wrote: Is there any evidence that selling a year prior to maturity (and incurring trading costs) is better than holding to maturity (avoiding trading costs)?
The spreadsheet can easily be modified to keep bonds until maturity.

Yet, we still need some historical short-term interest rates to approximate the yield curve. If 1-year rates are not exact, keeping the bonds until maturity (in the model) will be required to preserve the precise cash flows property which guarantees self-correcting returns*.

* As long as we have trustworthy 10-year rates, of course.

Maybe looking at the yield differential between FRED's 1-year rates and Shiller's 1-year rates, in years where both are available, could give us ideas to reasonably transform Shiller's older 1-year commercial paper rates into approximate 1-year government bond rates.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
User avatar
siamond
Posts: 6050
Joined: Mon May 28, 2012 5:50 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by siamond »

I don't have Sydney Homer's book, but I finally found my copy of Irrational Exuberance (2nd edition). Here is the most detailed extract about his description (p232).

Image

According to the index, Irrational Exuberance actually only briefly discusses short-term interest rates, the main reference is on p197 (plus note 9 on page 265), mostly to make the point that stocks rarely underperform such rates over a decade. Still there is an interesting hint on note 9 (and no, I don't have this other book from Pr Shiller, "Market Volatility"). Note that the SBBI book actually not not provide interest rates per se, but does provide US treasury bills returns (IPR-protected data).

Image

PS. oh, and in the Excel spreadsheet, I couldn't understand the reference to Chapter 26 (no such chapter in the book), but this is actually a reference to the other book! This was indeed stated in the first few lines of the spreadsheet: An Update of Data shown in Chapter 26 of Market Volatility, R. Shiller, MIT Press, 1989, and Irrational Exuberance, Princeton 2005.

PS2. Google Books only provides a few pages for Market Volatility (which seems especially "heady" stuff!), nothing useful. On Amazon, one can preview p279, which does elaborate a bit on the various 'sources of data' used for Chapter 15. The book is available used for less than 10 bucks if anybody is interested... I'll pass!
User avatar
Kevin M
Posts: 16281
Joined: Mon Jun 29, 2009 3:24 pm
Contact:

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by Kevin M »

I neglected to answer this question:
longinvest wrote:Would there be any interest to add such calculated pre-1972 bond returns, as well as calculated stock returns, based on Prof. Shiller's data set, into Simba's backtesting spreadsheet?
Personally, no, since I can't load that spreadsheet into Google Sheets directly, but a big yes to making the data easily available to users of any spreadsheet platform (including Google Sheets). I can load your bond returns spreadsheet directly into Google Sheets no problem. The problem with the Simba spreadsheet is the size.

Since I care mainly about the data, and not the backtesting functionality (which I can do using PortfolioVisualizer), it would be nice to make any data used by such humongous spreadsheets available separately. Of course I can just grab the 1972-> data from the PortfolioVisualizer website, but would be nice to have efforts such as yours to add data result in data that is more widely accessible.

Kevin
If I make a calculation error, #Cruncher probably will let me know.
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

siamond wrote:I don't have Sydney Homer's book, but I finally found my copy of Irrational Exuberance (2nd edition). Here is the most detailed extract about his description (p232).

Image

According to the index, Irrational Exuberance actually only briefly discusses short-term interest rates, the main reference is on p197 (plus note 9 on page 265), mostly to make the point that stocks rarely underperform such rates over a decade. Still there is an interesting hint on note 9 (and no, I don't have this other book from Pr Shiller, "Market Volatility"). Note that the SBBI book actually not not provide interest rates per se, but does provide US treasury bills returns (IPR-protected data).

Image

PS. oh, and in the Excel spreadsheet, I couldn't understand the reference to Chapter 26 (no such chapter in the book), but this is actually a reference to the other book! This was indeed stated in the first few lines of the spreadsheet: An Update of Data shown in Chapter 26 of Market Volatility, R. Shiller, MIT Press, 1989, and Irrational Exuberance, Princeton 2005.

PS2. Google Books only provides a few pages for Market Volatility (which seems especially "heady" stuff!), nothing useful. On Amazon, one can preview p279, which does elaborate a bit on the various 'sources of data' used for Chapter 15. The book is available used for less than 10 bucks if anybody is interested... I'll pass!
Siamond,

These are terrific findings. My takeaway is that we can rely on Shiller's 10-year rates, for "precision", but not on the 1-year ones.

I modeled a 10-rung ladder that keeps its bonds until maturity, to get back the nice self-correcting property (lost to 1-year rate discrepancies for the fund model). I didn't use the FRED rates, nor apply a spread on the commercial paper 1-year rates to correct for credit risk.

Here's what I get when I draw a telltale chart of the 10-rung ladder vs our previous fund:

Image

The ladder has been lagging the fund since the 80s. This will surely show up when adding FRED rates and comparing the ladder to SBBI intermediate bonds over the period.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

It's fun to have a bond fund simulator.

Here's the internal growth chart of the our 9-rung bond fund. Each bond is bought and kept for 9 years. It is then sold and the proceeds are combined with the 9 coupons (of 9 bonds) to buy a new 10-year bond.

Image

Notice how each rung grows mostly in its 9th year, when coupons are added to its value to buy a new bond.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

longinvest wrote:It's fun to have a bond fund simulator.

Here's the internal growth chart of the our 9-rung bond fund. Each bond is bought and kept for 9 years. It is then sold and the proceeds are combined with the 9 coupons (of 9 bonds) to buy a new 10-year bond.

Image

Notice how each rung grows mostly in its 9th year, when coupons are added to its value to buy a new bond.
The self-correcting property is actually illustrated in this chart. As long as we have precise 10-years and 1-year rates, the 9th year "peaks" of each rung are exact values (no approximation). The remaining parts of a rung's trajectory are calculated using a (not so badly) approximated yield curve.

As we now know that Shiller's 1-year rates are not representative of government bills, holding bonds until maturity would restore this "exact peaks" property.
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
Rodc
Posts: 13601
Joined: Tue Jun 26, 2007 9:46 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by Rodc »

Very interesting.

Not sure it matters much, but if one is going to go this effort, does it make sense to add in an estimate of ER and additional trading costs?

For well run stock mutual funds the ER and other costs are so low they can be offset by things like security lending so the net is essentially zero.

Is that true for bond funds as well?
We live a world with knowledge of the future markets has less than one significant figure. And people will still and always demand answers to three significant digits.
User avatar
Kevin M
Posts: 16281
Joined: Mon Jun 29, 2009 3:24 pm
Contact:

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by Kevin M »

So are you going to use 1-year CMT rates from FRED where available (since 1953)? That will at least give you better quality results for more recent time periods.

Kevin
If I make a calculation error, #Cruncher probably will let me know.
Topic Author
longinvest
Posts: 5884
Joined: Sat Aug 11, 2012 8:44 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by longinvest »

Kevin M wrote:So are you going to use 1-year CMT rates from FRED where available (since 1953)? That will at least give you better quality results for more recent time periods.

Kevin
Actually, as long as we're allowed to publish them in the spreadsheet, I don't see why we wouldn't use as many rates as FRED makes available along the yield curve: 1 year, 2 years, 3 years, 5 years, 7 years, and 10 years. This way, our yield curve approximation (for 4, 6, 8, and 9-year CMT) should be incredibly close to historical reality.

OK, I went looking, and here's what they write:
I would like to publish data found in FRED. What is the appropriate citation to use?

All FRED data series are freely available for personal use, and most are also freely available for use in publications and presentations if cited appropriately. But please note that some FRED data series are owned by third parties and subject to copyright restrictions. Please review the notes for any series you wish to publish or reprint.

For publicly available data: FRED now offers a suggested citation under the "Cite" tab. If you use FRED for research papers, publications, or presentations, please cite it using the information in that tab. FRED is also compatible with the Zotero citation tool and may be used with that tool.
For instance, on the 1-Year Treasury Constant Maturity Rate page, we find:
Suggested Citation

Board of Governors of the Federal Reserve System (US), 1-Year Treasury Constant Maturity Rate [DGS1], retrieved from FRED, Federal Reserve Bank of St. Louis https://research.stlouisfed.org/fred2/series/DGS1/, December 14, 2015.
In all cases, we don't need to reproduce the entire set of daily data, we only need one number per year. So, we'll have to do it using proper citations, but it's allowed. Great!
Variable Percentage Withdrawal (bogleheads.org/wiki/VPW) | One-Fund Portfolio (bogleheads.org/forum/viewtopic.php?t=287967)
User avatar
siamond
Posts: 6050
Joined: Mon May 28, 2012 5:50 am

Re: Historical Bond Returns - Shiller: From Rates to Returns

Post by siamond »

Rodc wrote:Not sure it matters much, but if one is going to go this effort, does it make sense to add in an estimate of ER and additional trading costs?
The idea is to add new tabs to the Simba spreadsheet for pre-1972 returns, while keeping the same general logic for the spreadsheet, which already uses ER estimates for similar situations. So the answer should be YES, but no need to do it at the level of modeling discussed on this thread.
Post Reply