What are the merits of market cap weighting for bond funds?

[Bernmaster]

Dear Bogleheads,

Regarding equity funds I believe to understand the many advantages of market cap weighting. However, I do not understand the advantages of market cap weighting for bond funds. My questions are as follows:

1. According to my understanding one big advantage is the reduction in turnover which minimizes transaction costs. This appears not to be the case for bond funds. Weights automatically adjust to price changes but bonds constantly fluctuate in and out. How much does this turnover cost fund holders in return?

2. Another fact I'm struggling with is that in contrast to stocks the return of bonds is known at the purchase date if held to maturity (barring default). This appears not to be the case for bond funds. How predictive is the "yield to maturity" or "yield to worst" figure depicted in the fund prospectus of actual fund performance if held for the average duration of the fund?

3. Another point I struggle with is the "investment grade" requirement which forces the fund to sell bonds that get downgraded below BBB rating. I believe the bond funds are also forced to sell bonds when they reach less than one year of duration. To my understanding this is a drag on performance but can it be quantified?

4. Could you please explain to me why market cap weighting works for bonds? Does a market cap weighted bond fund maximize risk-adjusted returns? Is the market portfolio for bonds "special" from a risk-return perspective?

5. Even John Bogle seemed to have struggled with the the concept of market cap weighting for bonds which is evident from his suggestions in "The Little Book of Common Sense Investing" on how to improve returns by adding more corporate bonds. Is the credit and term risk of the bonds in a Total Bond Market fund appropriately compensated for and do the funds capture these risk premia?

Thank you very much for taking the time to answer my questions !

[Geologist]

Item 3: why is selling bonds downgraded below investment grade a drag on performance (as opposed to continuing to hold them)? Bonds that are downgraded go down in market value (and they are likely to do so even before the downgrade). This is reflected in the NAV. Selling them doesn't change this. Are you arguing you would be better off with an active manager who supposedly would know how to avoid such bonds?

I am also not sure I understand how selling bonds a year before maturity is a drag on performance, so you should explain.

[JoMoney]

FWIW, your title says "bond funds", but your question is more about a specific bond index.
There are lots of bond funds, and even bond indexes that don't market cap, that don't have "investment grade" requirements, and may only focus on specific terms, credit risks.
I wouldn't say a Total Bond Index is a bad choice, but it's always seemed an odd choice to me for anything but the most generic option of it being a low-cost way to own a roughly intermediate term bond fund, that's not really entirely 'intermediate'. The idea that it may not have to sell bonds as the maturity lengths grow shorter (since it's not only intermediate bonds) may have some benefit, the turnover on Total Bond fund is lower than many comparable intermediate bond funds.

[rkhusky]

Are bond funds really cap-weighted? The fact that Total Bond has fixed allocations to corporate and government bonds would suggest that it’s not. And the high turnover of most bond funds would suggest that they are not either. Perhaps they are cap-weighted in some ways but not in others.

[JoMoney]

... The fact that Total Bond has fixed allocations to corporate and government bonds would suggest that it’s not...

I don't believe that's true. One of John Bogle's complaints about the total bond index was the growing allocation to government bonds (at a time where government bonds had exceptionally lower yield spread relative to corporate bonds) and suggested adding additional corporate bonds to it.

[rkhusky]

... The fact that Total Bond has fixed allocations to corporate and government bonds would suggest that it’s not...

I don't believe that's true. One of John Bogle's complaints about the total bond index was the growing allocation to government bonds (at a time where government bonds had exceptionally lower yield spread relative to corporate bonds) and suggested adding additional corporate bonds to it.

It’s been about 67/33 for as long as I’ve been watching it. Was there a time when it was significantly different?

[JoMoney]

... The fact that Total Bond has fixed allocations to corporate and government bonds would suggest that it’s not...

I don't believe that's true. One of John Bogle's complaints about the total bond index was the growing allocation to government bonds (at a time where government bonds had exceptionally lower yield spread relative to corporate bonds) and suggested adding additional corporate bonds to it.

It’s been 65/35 - 70/30 for as long as I’ve been watching it. Was there a time when it was significantly different?

I think it's varied by a wider margin than that, more like 60-70% governments, the makeup of those "government" bonds has changed over time, but regardless of that, even what you suggested in the variance is NOT a "fixed allocation".

FWIW, here's what the allocation looked like at the end of Dec. 2007:
https://www.sec.gov/Archives/edgar/data ... dfinal.htm

An here's what it is currently:
https://investor.vanguard.com/investmen ... omposition

[rkhusky]

... The fact that Total Bond has fixed allocations to corporate and government bonds would suggest that it’s not...

I don't believe that's true. One of John Bogle's complaints about the total bond index was the growing allocation to government bonds (at a time where government bonds had exceptionally lower yield spread relative to corporate bonds) and suggested adding additional corporate bonds to it.

It’s been 65/35 - 70/30 for as long as I’ve been watching it. Was there a time when it was significantly different?

I think it's varied by a wider margin than that, more like 60-70% governments, the makeup of those "government" bonds has changed over time, but regardless of that, even what you suggested in the variance is NOT a "fixed allocation".

FWIW, here's what the allocation looked like at the end of Dec. 2007:
https://www.sec.gov/Archives/edgar/data ... dfinal.htm

An here's what it is currently:
https://investor.vanguard.com/investmen ... omposition

I don’t really know if the index is fixed or not. I just assumed it was because it stayed relatively constant. If the index does have a fixed allocation to government vs corporate, it doesn’t mean that a fund has to match that exactly all the time. So, the slight variation shown above isn’t strong evidence for a market cap index.

There is an article in the wiki about the index, but I don’t see much evidence for cap weight there:
https://www.bogleheads.org/wiki/Bloombe ... Bond_Index

[Geologist]

This topic contains a graph (in the 5th post by SimpleGift Nov 19) viewtopic.php?t=295034 that shows the variation in composition of the Total Bond Index over time. It certainly has not been a constant (or even relatively constant over time).

One thing I will say about John Bogle’s recommendations for more corporates in the 2010’s is that the Index had roughly the same percentage of Treasuries (or more) in the mid-1990’s and he was recommending the Total Bond Index itself in his books published then. There is nothing wrong with changing your mind, but I think he needed to explain why and I never saw such an explanation. I personally think he was yield chasing: he thought the yield of Treasuries in the 2010’s were too low and wanted to add bonds with higher yields. I didn't find that an impressive argument.

[rkhusky]

This topic contains a graph (in the 5th post by SimpleGift Nov 19) viewtopic.php?t=295034 that shows the variation in composition of the Total Bond Index over time. It certainly has not been a constant (or even relatively constant over time).

Although I think I read that the index was developed in 1986, but could have been simulated further back. But it does show significant variation in the early years, even with the Government/Corporate mix. I didn’t start following until the 2000’s, where that mix has been fairly constant.

[Geologist]

This topic contains a graph (in the 5th post by SimpleGift Nov 19) viewtopic.php?t=295034 that shows the variation in composition of the Total Bond Index over time. It certainly has not been a constant (or even relatively constant over time).

Although I think I read that the index was developed in 1986, but could have been simulated further back. But it does show significant variation in the early years, even with the Government/Corporate mix. I didn’t start following until the 2000’s, where that mix has been fairly constant.

I don't think so. The fraction of straight Treasuries changed dramatically between 2002-2007 (20 to mid 20 percents) and subsequently (much higher). That is what really drove Bogle's comments on increasing corporates the 2010's.

[rkhusky]

This topic contains a graph (in the 5th post by SimpleGift Nov 19) viewtopic.php?t=295034 that shows the variation in composition of the Total Bond Index over time. It certainly has not been a constant (or even relatively constant over time).

Although I think I read that the index was developed in 1986, but could have been simulated further back. But it does show significant variation in the early years, even with the Government/Corporate mix. I didn’t start following until the 2000’s, where that mix has been fairly constant.

I don't think so. The fraction of straight Treasuries changed dramatically between 2002-2007 (20 to mid 20 percents) and subsequently (much higher). That is what really drove Bogle's comments on increasing corporates the 2010's.

Treasuries had their lowest allocation in 2002-2007. MBS were substantially larger. Corporates have stayed pretty steady.

[nisiprius]

1) Wikipedia says the Bloomberg US Aggregate Bond Index is

or the Agg, is a broad base, market capitalization-weighted bond market index representing intermediate term investment grade bonds traded in the United States.

I have not been able to find a Bloomberg source for "market capitalization-weighted" but see below.

2) These days I can't find the factsheets I used to be able to find, but I saved one from 2017. It shows that the composition is not fixed and has continually changed.



The factsheet doesn't use the exact phrase "market capitalization-weighted." I'm not sure what this means:

For each index, Bloomberg maintains two universes of securities: the Returns (Backward) and the Projected (Forward) Universes. The composition of the Returns Universe is rebalanced at each month-end and represents the fixed set of bonds on which index returns are calculated for the next month. The Projected Universe is a forward-looking projection that changes daily to reflect issues dropping out of and entering the index but is not used for return calculations. On the last business day of the month (the rebalancing date), the composition of the latest Projected Universe becomes the Returns Universe for the following month.

3) When John C. Bogle began saying things like we need to fix the bond index, the reason he gave was not intuition or personal taste. He had a reason:

...the fact of the matter is in that $16 trillion or so just of Treasuries alone here, what we are dealing with is huge amounts that are held not by U.S. investors, but foreign investors. And if you look at the data, you see that about $5.5 trillion of that $16 trillion is held by China and Japan and a couple of other countries, that's irrelevant to the U.S. investor in my opinion.... When we look at how U.S. investors do, the government position in the index should be about half of what it is maybe a third of what is. So, we've got to fix the index.

He might have been right, he might have been wrong, but he had a well-articulated reason.

[nisiprius]

One rationale for market-cap weighting is completely independent of what kind of market it is: stocks, bonds, whatever. All that is necessary is that there be a market... with the usual kinds of idealized economics assumptions, that traders are rational, and that trading within the market is frictionless. Under these assumptions, it can be shown that "the market portfolio," the actual set of all securities held by all investors within the market, is optimum in the sense of having the highest risk-adjusted return.

One statement of this is given here: The market portfolio. A non-mathematical explanation is given by William F. Sharpe here. And a celebrity endorsement, if you will, showing general acceptance of the idea, is given by Jeremy Siegel, in Stocks for the Long Run 5/E:

[Market capitalization-weighted portfolios,] under the assumptions of an efficient market, give investors the "best" trade-off between risk and return. This means that for any given risk level, these capitalization-weighted portfolios give the highest returns; and for any given return, these portfolios give the lowest risk. This property is called mean-variance efficiency. But the assumptions under which these desirable properties prevail are very stringent...

You can think of reasons why "Wall Street" would want to attack market cap-weighted indexing. So there are all sorts of indirect challenges. "It works for US stocks, but not in emerging markets." "It works in bull markets, but not in bear markets." "It used to work, but it's not going to work any more," etc. etc. I file "It works for stocks, but not for bonds" in the same drawer.

However, it is true that there is no real unified, integrated "bond market"--no pillared "New York Bond Exchange"--so the case for bond index funds isn't as clean as the case for stock index funds.

[rkhusky]

If Total Bond is cap weighted, presumably it is so not only by type, but also by maturity and rating. But I wonder if that would also apply to issuer for corporate bonds, or is sampling used a lot in that space.

There are so many more variations in bonds compared to stocks that cap weighting for bonds doesn’t have the same properties, such as low turnover.

[Bernmaster]

However, it is true that there is no real unified, integrated "bond market"--no pillared "New York Bond Exchange"--so the case for bond index funds isn't as clean as the case for stock index funds.

Thank you for your answers everyone! The above is probably the primary cause for my uncertainty regarding bond index funds. The construction framework of the index appears arbitrary compared to the corresponding stock funds. My feeling is that the index construction surely is good enough from a practical perspective (i.e. investing your money) but from an academical perspective it is slightly dissatisfying.

[jjj_22]

the actual set of all securities held by all investors within the market, is optimum in the sense of having the highest risk-adjusted return.

Highest risk adjusted return over what period of time, though?

I'm thinking, there's lots of discussion on this board about the duration of your bond holdings and the timeline on which you expect to need the money. This seem generally stated as, "if you need the money in X years, your bond holdings should have a duration of (roughly) X." But the duration of the whole cap weighted bond market is ... set by how long corporations and governments are issuing debt for, which from a bond investor's point of view seems totally arbitrary -- why let that average determine the duration of my own bond holdings?

Like, if I want to keep my money safe and get it back in 3 years, investing in the total bond market (which I think has a duration of like 6-7 years, last time I checked) is riskier than investing in, say, treasuries maturing 3 years out.

[Thesaints]

What are the merits of market cap weighting for bond funds?

Close to none.
Bonds terms are very well defined. Choose those the fit your own needs the best. It does not matter if some big market participant is buying different bonds, fitting their needs. Why should you imitate them ?

The situation with stocks is completely different, contract terms thee are close to none and therefore all stocks share common characteristics. You want to use indexes and market caps to average out the extreme indetermination of that market.

[MnD]

Globally, trillions of dollars are invested in US Treasury debt not because it's the best yield/risk proposition but because it's the only market capable of absorbing these amount of funds.

So the US enjoys a lowered yield benefit from that but the individual investor suffers a penalty.
In a nutshell that's why index bond funds are mid-packers in contrast to equity index funds which outperform virtually all competitors.

[djm2001]

Globally, trillions of dollars are invested in US Treasury debt [...] because it's the only market capable of absorbing these amount of funds.

Thought-provoking claim. Do you have a good reference/citation for this? I'd like to know more...

[Bernmaster]

What are the merits of market cap weighting for bond funds?

Close to none.
Bonds terms are very well defined. Choose those the fit your own needs the best. It does not matter if some big market participant is buying different bonds, fitting their needs. Why should you imitate them ?

The situation with stocks is completely different, contract terms thee are close to none and therefore all stocks share common characteristics. You want to use indexes and market caps to average out the extreme indetermination of that market.

Thank you for your comment - I agree 100%. The fitting my needs part is especially relevant here. Honestly, maybe I’m not much of a planner but anticipating/predicting needs that arise in 6-7 years (duration of the cap weighted bond funds) with any precision appears difficult. I’m not an insurance company and circumstances change.

[nisiprius]

Source



Red: performance of market-cap-weighted Vanguard Total Bond Market Index Fund (VBTLX).

Blue: performance of the Janus Henderson Global Unconstrained Bond Fund (JUCIX) while under the personal management of Bill Gross, often called the "bond king" and regarded as a genius in active bond selection.

Sharpe's "arithmetic of active management" works on all asset classes. The collective performance of active managers must equal the market minus expenses.

One "merit of market cap weighted bond index funds" is avoiding manager risk. Of course if you can identify the managers that have skill, you can do better with an active fund. But in 2014, would you have really dared to say that Bill Gross did not have skill? It might not have been surprising if JUCIX had tied or simply failed to beat VBTLX, but how many people realized that, by choosing Bill Gross, they were risking getting a return of +0.24% instead of +2.00%?

You might think that actively-managed plain vanilla core bonds funds from recognized providers would be reasonably free from manager risk, but even that isn't true. The Oppenheimer Core Bond Fund (OPIGX) was a mainstream fund offered in 401(k) plans:

[nisiprius]

I just compared passive versus active, arguing that you can't ignore manager risk in actively managed bond funds. But I guess that isn't the real question.

Is anybody aware of any alternative indexes, and funds or ETFs tracking them, other than market cap weighted? It occurs to me that I haven't heard much about "fundamental indexing" in bonds, or "smart beta" in bonds, or "equal-weighted" bond indexes.

What are the alternative indexes to market cap weighting, and what is their track record? What passive bond strategies are claimed to be superior to market cap weighting?

[djm2001]

1. According to my understanding one big advantage is the reduction in turnover which minimizes transaction costs. This appears not to be the case for bond funds. Weights automatically adjust to price changes but bonds constantly fluctuate in and out. How much does this turnover cost fund holders in return?

Any buy-and-hold strategy minimizes turnover (and so the low turnover advantage is not limited to market weighting per se)... so long as assets don't enter or leave the market in question.

In fact, stock index funds also suffer a similar issue, albeit to a smaller degree, as stock units enter and leave the stock market in question (e.g., companies issuing new stocks or doing buybacks, new companies entering the S&P500 or being delisted).

If the underlying assets mature, such as bonds do, then holding the asset class over longer than the maturity will necessitate buying new issues when the olds ones mature. Presumably bond funds can mitigate turnover by using fund inflows and outflows as rebalancing opportunities (just as an individual investor would rebalance using new income or by selling appropriately if they needed to sell anyway to meet cashflow needs). Either way, this is not specifically an issue with market weighting. Rather, it's an issue with holding an asset class longer than the maturity of the assets.

2. Another fact I'm struggling with is that in contrast to stocks the return of bonds is known at the purchase date if held to maturity (barring default). This appears not to be the case for bond funds. How predictive is the "yield to maturity" or "yield to worst" figure depicted in the fund prospectus of actual fund performance if held for the average duration of the fund?

There are two separate points you bring up here.

1. The first is that the payoff predictability of bonds is greater than that of stocks. We pay for this predictability with lower expected return. This point is more about stocks vs bonds. It is not specifically about bond funds or market weighting.
2. The second point is that the published statistics of bond funds are not as precise as individual bonds since they are aggregate / averaged statistics. They are lossy statistical summaries of the distribution of bonds in the fund. This is not related to market weighting or even bond funds, specifically. Most fund-level statistic (stock or bond) will be somewhat lossy, informationally speaking (e.g., PE ratio of S&P500).

3. Another point I struggle with is the "investment grade" requirement which forces the fund to sell bonds that get downgraded below BBB rating. I believe the bond funds are also forced to sell bonds when they reach less than one year of duration. To my understanding this is a drag on performance but can it be quantified?

This issue is not exactly unique to bonds. Stock indexes also have criteria that are subjective judgement calls (e.g., minimum market cap, minimum trading volume). That said, I agree that the BBB rating cutoff of the Bloomberg aggregate bond index (AGG) is annoying. Personally, I hold junk bonds (at market weight) to partially fill in that gap. AGG doesn't include TIPS or municipal bonds either (for tax management reasons, I think). Again, this is not an issue with market cap weighting, but rather to any fund (bond or stock) that has arbitrary quality thresholds that assets can drift across. E.g., an equal-weighted fund with quality thresholds would have the same issues (perhaps even more so).

4. Could you please explain to me why market cap weighting works for bonds? Does a market cap weighted bond fund maximize risk-adjusted returns? Is the market portfolio for bonds "special" from a risk-return perspective?

This is the heart of the question. In the simplistic mean-variance model used by Modern Portfolio Theory and its cousins, the market portfolio (consisting of all competitively traded, easily exchangeable assets which includes both stocks and bonds), lies on the efficient frontier and is the so-called tangency portfolio. And all investors hold a linear combination of the risk-free asset and the market portfolio. There are numerous assumptions that are made in the simple theory (e.g., single-period returns, simplistic risk model, borrowing at the risk-free rate). But there are some analogous results for more general models, such as long-term investing and multi-factor risk models.

There are varying views on what assets comprise the "universe of investable assets" with respect to market-cap weighting. E.g., should one include government bonds since both lenders and ultimately borrowers are the public themselves, and so is net public interest zero? And what about illiquid investments like private equity and debt, and real estate? William Sharpe has a good discussion on this in "Components of the Market Portfolio" section of Chapter 7 of his RISMAT book. That entire chapter is fantastic and worth reading.

Some people take a rigid definition of "market" (e.g., limit to a single exchange) whereas others consider all assets that can be bought and sold relatively easily (and thus be easily exchanged across multiple exchanges).

Anyway, my overall point here is that if we are looking at efficient frontier or risk-adjusted return optimization, we need to consider all assets that the public has access to. Market weighting is efficient only when all publicly accessible options are included. And even when it is efficient, it may not be optimal for an individual's particular consumption needs... Keep in mind that one's consumption stream is basically a negative asset, and so theoretically an individual investor should be looking at efficiency not just across all investment assets but also with their own personal negative asset (consumption) and positive asset (income/human capital).

But it's important to note that market weighting has additional advantages beyond finding an efficient portfolio that hold even when only a subset of the market is considered -- e.g., minimizing turnover (modulo assets entering and leaving the system), lower management fees and taxes, low exposure to information asymmetry (because we minimize trading), doing as well as the average actively managed dollar (see Sharpe's article on The Arithmetic of Active Management). These advantages hold for bonds same as they hold for other asset classes. And this is why many investors use market weighting for bonds even if they don't hold their stock vs bond ratio at market weight. (I think this paragraph is the TL;DR answer to your overall question.)

Does a market cap weighted bond fund maximize risk-adjusted returns? Is the market portfolio for bonds "special" from a risk-return perspective?

No, but neither does it do so for stocks. A market-weighted portfolio lies on the efficient frontier only when the all alternative assets are considered (including both bonds and stocks). If you omit some alternatives that the market has access to, you no longer can rely on market weighting to optimize anything. Other benefits of market weighting listed above still hold, however.

But if you want to have the efficient frontier theoretical justification of on your side, you need to hold market weights across both stocks and bonds (and anything else you believe to be part of the publicly investable universe). And if you want to adjust for personal circumstances, you'd build a liability matching portfolio to transform your personal income and consumption stream into that of the dollar-weighted average investor, and then invest your remaining money in the market portfolio.

5. Even John Bogle seemed to have struggled with the the concept of market cap weighting for bonds which is evident from his suggestions in "The Little Book of Common Sense Investing" on how to improve returns by adding more corporate bonds. Is the credit and term risk of the bonds in a Total Bond Market fund appropriately compensated for and do the funds capture these risk premia?

The question of inclusion of government bonds (as opposed to corporate bonds) within a bond index is an interesting question. Sharpe discusses this in RISMAT Chapter 7 that I linked above. Read it, and decide for yourself where you stand. Personally, I take the same position as Sharpe that government debt is an investment in future human capital (via future taxpayer earnings).

For corporate bonds, some interesting food for thought is the Modigliani-Miller theorem which says that, frictions aside, public claim to a companies earnings are independent of its capital structure (i.e., whether it chooses to fund its activities through issuing stocks or bonds) and dividend policy. This might justify holding both stocks and corporate bonds together at market weight.

Another wrinkle to consider is international bonds and the inescapable currency hedging that comes built into most international bond funds. But keep in mind that currency hedging affects only return, but not market weight. Personally, I just use global market weighting and hope that the built-in currency hedging in the fund's return mitigates the need for home-country bias in the global asset weighting.

[telemark]

AGG excludes TIPS apparently for historical reasons: TIPS were first offered in 1997 while AGG dates back to 1979.

For alternative indexes, David Swensen recommends combining a TIPS index with a Treasuries index in his Yale portfolio (although this is a slice and dice approach and thus anathema to some Bogleheads). It's been a while since I read his book, but as I recall the reasons are lack of credit risk and a lower correlation to stocks.

Correcting myself: the present form of AGG dates from 1986, when mortgage-backed securities were added.

[grabiner]

AGG excludes TIPS apparently for historical reasons: TIPS were first offered in 1997 while AGG dates back to 1979.

Correcting myself: the present form of AGG dates from 1986, when mortgage-backed securities were added.

But it also makes sense from an investment standpoint. TIPS and other bonds are fundamentally different, so it is unlikely that an investor would want to hold the two in the market weight. The appropriate TIPS weight for an investor depends on that investor's sensitivity to inflation risk.

In contrast, corporate and Treasury bonds are similar investments, so there is nothing inherently wrong with holding them at the market weight.

[Bernmaster]

1. According to my understanding one big advantage is the reduction in turnover which minimizes transaction costs. This appears not to be the case for bond funds. Weights automatically adjust to price changes but bonds constantly fluctuate in and out. How much does this turnover cost fund holders in return?

Any buy-and-hold strategy minimizes turnover (and so the low turnover advantage is not limited to market weighting per se)... so long as assets don't enter or leave the market in question.

In fact, stock index funds also suffer a similar issue, albeit to a smaller degree, as stock units enter and leave the stock market in question (e.g., companies issuing new stocks or doing buybacks, new companies entering the S&P500 or being delisted).

If the underlying assets mature, such as bonds do, then holding the asset class over longer than the maturity will necessitate buying new issues when the olds ones mature. Presumably bond funds can mitigate turnover by using fund inflows and outflows as rebalancing opportunities (just as an individual investor would rebalance using new income or by selling appropriately if they needed to sell anyway to meet cashflow needs). Either way, this is not specifically an issue with market weighting. Rather, it's an issue with holding an asset class longer than the maturity of the assets.

2. Another fact I'm struggling with is that in contrast to stocks the return of bonds is known at the purchase date if held to maturity (barring default). This appears not to be the case for bond funds. How predictive is the "yield to maturity" or "yield to worst" figure depicted in the fund prospectus of actual fund performance if held for the average duration of the fund?

There are two separate points you bring up here.

1. The first is that the payoff predictability of bonds is greater than that of stocks. We pay for this predictability with lower expected return. This point is more about stocks vs bonds. It is not specifically about bond funds or market weighting.
2. The second point is that the published statistics of bond funds are not as precise as individual bonds since they are aggregate / averaged statistics. They are lossy statistical summaries of the distribution of bonds in the fund. This is not related to market weighting or even bond funds, specifically. Most fund-level statistic (stock or bond) will be somewhat lossy, informationally speaking (e.g., PE ratio of S&P500).

3. Another point I struggle with is the "investment grade" requirement which forces the fund to sell bonds that get downgraded below BBB rating. I believe the bond funds are also forced to sell bonds when they reach less than one year of duration. To my understanding this is a drag on performance but can it be quantified?

This issue is not exactly unique to bonds. Stock indexes also have criteria that are subjective judgement calls (e.g., minimum market cap, minimum trading volume). That said, I agree that the BBB rating cutoff of the Bloomberg aggregate bond index (AGG) is annoying. Personally, I hold junk bonds (at market weight) to partially fill in that gap. AGG doesn't include TIPS or municipal bonds either (for tax management reasons, I think). Again, this is not an issue with market cap weighting, but rather to any fund (bond or stock) that has arbitrary quality thresholds that assets can drift across. E.g., an equal-weighted fund with quality thresholds would have the same issues (perhaps even more so).

4. Could you please explain to me why market cap weighting works for bonds? Does a market cap weighted bond fund maximize risk-adjusted returns? Is the market portfolio for bonds "special" from a risk-return perspective?

This is the heart of the question. In the simplistic mean-variance model used by Modern Portfolio Theory and its cousins, the market portfolio (consisting of all competitively traded, easily exchangeable assets which includes both stocks and bonds), lies on the efficient frontier and is the so-called tangency portfolio. And all investors hold a linear combination of the risk-free asset and the market portfolio. There are numerous assumptions that are made in the simple theory (e.g., single-period returns, simplistic risk model, borrowing at the risk-free rate). But there are some analogous results for more general models, such as long-term investing and multi-factor risk models.

There are varying views on what assets comprise the "universe of investable assets" with respect to market-cap weighting. E.g., should one include government bonds since both lenders and ultimately borrowers are the public themselves, and so is net public interest zero? And what about illiquid investments like private equity and debt, and real estate? William Sharpe has a good discussion on this in "Components of the Market Portfolio" section of Chapter 7 of his RISMAT book. That entire chapter is fantastic and worth reading.

Some people take a rigid definition of "market" (e.g., limit to a single exchange) whereas others consider all assets that can be bought and sold relatively easily (and thus be easily exchanged across multiple exchanges).

Anyway, my overall point here is that if we are looking at efficient frontier or risk-adjusted return optimization, we need to consider all assets that the public has access to. Market weighting is efficient only when all publicly accessible options are included. And even when it is efficient, it may not be optimal for an individual's particular consumption needs... Keep in mind that one's consumption stream is basically a negative asset, and so theoretically an individual investor should be looking at efficiency not just across all investment assets but also with their own personal negative asset (consumption) and positive asset (income/human capital).

But it's important to note that market weighting has additional advantages beyond finding an efficient portfolio that hold even when only a subset of the market is considered -- e.g., minimizing turnover (modulo assets entering and leaving the system), lower management fees and taxes, low exposure to information asymmetry (because we minimize trading), doing as well as the average actively managed dollar (see Sharpe's article on The Arithmetic of Active Management). These advantages hold for bonds same as they hold for other asset classes. And this is why many investors use market weighting for bonds even if they don't hold their stock vs bond ratio at market weight. (I think this paragraph is the TL;DR answer to your overall question.)

Does a market cap weighted bond fund maximize risk-adjusted returns? Is the market portfolio for bonds "special" from a risk-return perspective?

No, but neither does it do so for stocks. A market-weighted portfolio lies on the efficient frontier only when the all alternative assets are considered (including both bonds and stocks). If you omit some alternatives that the market has access to, you no longer can rely on market weighting to optimize anything. Other benefits of market weighting listed above still hold, however.

But if you want to have the efficient frontier theoretical justification of on your side, you need to hold market weights across both stocks and bonds (and anything else you believe to be part of the publicly investable universe). And if you want to adjust for personal circumstances, you'd build a liability matching portfolio to transform your personal income and consumption stream into that of the dollar-weighted average investor, and then invest your remaining money in the market portfolio.

5. Even John Bogle seemed to have struggled with the the concept of market cap weighting for bonds which is evident from his suggestions in "The Little Book of Common Sense Investing" on how to improve returns by adding more corporate bonds. Is the credit and term risk of the bonds in a Total Bond Market fund appropriately compensated for and do the funds capture these risk premia?

The question of inclusion of government bonds (as opposed to corporate bonds) within a bond index is an interesting question. Sharpe discusses this in RISMAT Chapter 7 that I linked above. Read it, and decide for yourself where you stand. Personally, I take the same position as Sharpe that government debt is an investment in future human capital (via future taxpayer earnings).

For corporate bonds, some interesting food for thought is the Modigliani-Miller theorem which says that, frictions aside, public claim to a companies earnings are independent of its capital structure (i.e., whether it chooses to fund its activities through issuing stocks or bonds) and dividend policy. This might justify holding both stocks and corporate bonds together at market weight.

Another wrinkle to consider is international bonds and the inescapable currency hedging that comes built into most international bond funds. But keep in mind that currency hedging affects only return, but not market weight. Personally, I just use global market weighting and hope that the built-in currency hedging in the fund's return mitigates the need for home-country bias in the global asset weighting.

Thank you very much for your comprehensive and extremely helpful answer! I really appreciate it! The linked articles are also very informative!

[Trance]

As to point three of having to sell bonds that fall below investment grade, there are some ETF's out there that are "Universal" bond funds like IUSB from iShares that include all of the same holdings as BND (the "total" market aggregate bond index") but also hold high risk bonds at market weight. So there is no need to sell the fallen angels when the credit falls under the investment grade. It only constitutes about 6.25% of the index.