Stock market indexing

From Bogleheads

The growing market acceptance of index based investing through the media of traditional mutual funds and exchange traded funds[note 1] has resulted in a proliferation of index providers, attracted to the business prospects of licensing their indexes to investment companies and seeing increasing royalty and licensing fee revenue from the growing asset base.[1] For investors this growth has led to an ever expanding universe of available index funds. However, the different methodologies that index providers utilize in measuring and carving up the stock market leads to a dispersion of returns that can be considerable over short and intermediate term time frames.[2]

Benchmark indexes

The benchmark indexes created by the major index providers share a number of characteristics.

Benchmark index strategy box

These characteristics place benchmark indexes in the Passive security selection/ Capitalization security weighting segment of the Index Strategy Box (see Figure 1.) The index weighting of securities is based on the market capitalization of the companies included in the index.

Benchmark Index Strategy Box.png
Figure 1. Index Strategy Box[3]

Benchmark index style box

In general, the stock market is composed of 3 levels of market capitalization and 3 styles, resulting in a 3 x 3 "style" box which includes large cap, mid cap and small cap stocks, divided among value, blend, and growth stocks. This is commonly represented in a style box as illustrated below:

Table 1. Total Market Index Style Box (%)
Value Blend Growth
Large Cap LV LB LG
Mid Cap MV MB MG
Small Cap SV SB SG

One may also see an expanded breakdown of market cap ranges to include Mega cap, Micro cap, and Nano cap stocks.

Table 2. Market Capitalization
Size Characterization Capitalization Range
Mega capitalization stocks Over $200 billion dollars
Large capitalization stocks Between $10 billion dollar and $200 billion dollars
Mid capitalization stocks Between $2 billion and $10 billion dollars
Small capitalization stocks Between $300 million and $2 billion dollars
Micro capitalization stocks Between $30 million and $375 million dollars
Nano capitalization stocks Below $50 million dollars

In addition to index provider companies, the Center for Research in Security Prices CRSP[4] has since 1960 provided the academic community with market return data extending from 1926 to the present. CRSP offers Historical Cap-Based Portfolio data tracking micro, small, mid and large-cap stock. For the cap-based portfolios, CRSP ranks all NYSE companies by market capitalization and then divides them into ten equally populated portfolios. Amex and NASDAQ stocks are then placed into the deciles determined by the NYSE breakpoints, based on their market capitalization. CRSP portfolios 1-2 represent large cap stocks, portfolios 3-5 represent mid-caps, portfolios 6-8 represent small caps, and portfolios 9-10 represent a benchmark of micro-cap stocks.[5]

Note: since 2010, CRSP also provides its own investable capitalization-based indexes, which are now tracked by index funds like Vanguard, and defined in a different way than the historical cap-based portfolios (see Index Methodology).

Major index providers

The market coverage of major index providers is supplied in Figure 2. below.

Figure 2. Market Coverage of Major U.S. Index Providers*[6]
Stock Benchmarks - Major US Index Providers.png
* Size of colored bars are not to scale.

Notes: Although each provider listed has an index intended to mirror the entire stock market, including the micro-cap segment, investors typically use the index that encompasses only large-, mid-, and small-cap securities to represent the broad U.S. market. Those indexes are shown in the light blue bars for each index provider.

Indices and funds

Index types

Usually, an index provider supplies three different index returns for its market segment:[7]

  • A Price Index (Price, PR) which does not include dividends or dividend reinvestment.
  • A Gross Dividend Index (Total Return, TR) which includes all dividends reinvested.
  • A Net Dividend Index (Net Return, NR) which recognizes that the dividends are subject to tax and not received by the investor. Only the remaining portion of the dividend (net of tax) can be reinvested.

For benchmarking international indices, the Net Return index is most commonly used. The Morningstar Index Returns[8] identifies indices by type.[note 2]

US total market

The first index to measure the entirety of the US public market, the Wilshire 5000 Index, was established in 1974 by Wilshire Associates.[9] Russell introduced the Russell 3000 Index, measuring 98% of the US Market, in 1984.[10] S&P created its first broad based index of the US market in 1994 with the S&P Composite 1500 index, comprised of the S&P 500, S&P 400, and S&P 600 indices. The index measures approximately 85% of the market. The first retail index fund based on a total market index became available in 1992.

Index Inception Dates and Market Coverage
Index Provider Inception Backdate data Market Coverage
Wilshire 1974 1971 100%
Russell 1984 1984 98%
Dow Jones 1991 1987 100%
Morningstar 1998 1998 97%
MSCI 2004 2004 99.5%
S&P 2006 2006 (?) 100%
CRSP 2001 2002 100% (?)

The advent of the twenty first century saw the expansion of index providers in the US market, as well as a steady broadening of provider indexes to encompass the total market. Russell added 1000 micro cap stocks to its Russell 3000 index to create its broadest index, the Russell 3000e (extended), measuring 99% of the US market. In 2006, S&P created its S&P Completion Index, which in combination with the S&P 500 index produced the S&P Total Market Index. MSCI, Dow Jones, and Morningstar also created broad based total market indexes in the first decade of the twenty first century. (See Figure 1. for a chart of index provider market coverage.)

At present, only Russell provides total market style indexes (Russell 3000 Value and Russell 3000 Growth).

The similar breadth of index provider market coverage in total market indexes leads to very low turnover of stocks[11] and in a low range of investment return dispersion, especially over long holding periods (see US total market index returns for data).[note 3] The differential in return is most often attributed to the breadth of micro cap stocks in a total market index. For example, Morningstar (97%), Russell (98%), and MSCI (99.5%) total market indexes hold lower ranges of micro cap market coverage than does the Wilshire 5000 index (99.9%).

US Total
Stock Market

US Total

US large and mid cap

Size indices

Large Cap Index Inception Dates and Market Coverage
Index Provider Inception Backdate data Market Coverage Holdings
S&P 1954 1926 75% 1-500
Russell 1984 1984 92% 1-1000
MSCI 1996 1992 86% 1-750
Morningstar 1998 1998 70% (?)
Dow Jones 2005 1991 86% 1-750
CRSP 2001 2002 85% (?)

Mid Cap Index Inception Dates and Market Coverage
Index Provider Inception Backdate data Market Coverage Holdings
Russell 1984 1984 27% 200-1000
S&P 1991 1991 7% 501-900
MSCI 1996 1992 15% 301-750
Morningstar 1998 1998 20% (?)
Dow Jones 2005 1991 (?) 501-1000
CRSP 2001 2002 15% (?)

US small cap

Russell introduced the first small cap index (the Russell 2000) in 1984. Other index providers created small cap indexes in the last decade of the twentieth century: S&P in 1992; MSCI in 1996; Morningstar 1998. The first retail small cap index fund, based on the Russell 2000 index, was initiated in 1989. More institutional investors' small cap allocation assets are benchmarked to the Russell 2000 index than to any other provider.[12]

Size indices

Small Cap Index Inception Dates and Market Coverage
Index Provider Inception Backdate data Market Coverage Holdings
Russell 1984 1984 8% 1,001-3,000
S&P 1994 1994 3% 901-1,500
MSCI 1996 1992 12% 751-2,500
Morningstar 1998 1998 7% (?)
Dow Jones 2005 1991 12% 751-2,500
CRSP 2001 2002 13% (?)


Style indices

US Size and Style

US Style

US sector

US Sector

US Sector

  • US Sector Fund

International and global

International Index Market Coverage[14]
Index Provider Large Cap Mid Cap Small Cap
Dow Jones Top 85% Top 80% - 90% (overlaps large and small) 85% - 100%
FTSE > 72% between 72% and 92% < 92%
MSCI Top 70% +/- 5% between 70% and 85% +/- 5% between 85% and 99%
S&P Top 70% Top 70% - 85% 85% - 100%

and Global

and Global

Index methodology

Index providers differ in the ways they select and weight securities in the benchmarks’ return calculation. There are three key issues:[15]

  1. Security selection and weighting
  2. Calculation of returns
  3. Ongoing index maintenance

In their attempts to make their indexes more easily investable for index funds, index providers have adopted free-float[16] weighting and buffer zones to their methodologies. Free float weighting eliminates non-trading shares in a company's capital base. These can include cross-ownership of shares by other companies, government owned shares, privately held shares, and other restricted shares. A company's free-float weighting will reflect the actual amount of shares available for public investment.

All equity-index providers periodically adjust membership to reflect market changes as well as those made necessary by corporate actions such as mergers or spin-offs.[17] Buffer zones were introduced to reduce the amount of turnover incurred as stocks migrate across market capitalization and style boundaries. Reduced turnover helps reduce transaction costs for funds tracking an index.

Each major index provider has a different methodology for determining buffer zones. For example:

A summary of methodologies is shown in the table below.

Equity Index Methodology for Major U.S. Index Providers[18]
  Standard & Poor's Dow Jones Russell MSCI CRSP FTSE
Weighting methodology Float-adjusted market capitalization. Full market capitalization (float-adjusted market capitalization also available). Full market capitalization, adjusted for free float only after inclusion in index. Full market capitalization, adjusted for free float before inclusion in index. Float-adjusted market capitalization. Float-adjusted market capitalization.
Construction methodology Based on criteria established by S&P committee. Rules-based. Rules-based. Rules-based. Rules-based. Rules-based.
Rebalancing methodology Changes made on an as-needed basis. No annual or semiannual reconstitution. Quarterly, to coincide with review of Dow Jones U.S. Total Stock Market Index. Annual rebalance in June, subject to capitalization banding (+/- 2.5%) as of 2007. Quarterly review (two full reviews, two partial reviews), with security migration subject to "buffer zones." Quarterly rebalancing, with security migration subject to “packeting.”[note 4] Semi-annual review in June and December, subject to capitalization buffers around large, mid-, and

small-cap indexes.

Number of securities S&P Total Market Index includes all listed securities. DJ U.S. Total Stock Market Index includes all U.S. equity securities with readily available prices. Russell 3000E Index represents approximately 99% of U.S. equity market. MSCI US Broad Market Index targets 99.5% of cumulative full market capitalization of U.S. equity universe. CRSP US Total Market Index targets 99.5% of cumulative full-market capitalization of U.S. equity


FTSE US Total Market Index captures more than 99.5% of U.S. stock market.
Subcomponents of Total Market Index include fixed number of stocks (e.g., S&P 500 contains 500 stocks). Subcomponents of U.S. Total Stock Market Index include fixed number of stocks (e.g., DJ Large-Cap Index contains 750 stocks). Subcomponents of 3000E Index include fixed number of stocks (e.g., Russell 3000 Index includes 3,000 stocks). Subcomponents of US Broad Market Index include fixed number of stocks (e.g., MSCI US Prime Market 750 Index contains 750 stocks). Subcomponents of US Total Market Index based on market-capitalization targets: Largest 70% constitute mega cap index; next 15%, mid-cap index; final 13.5%, small-cap index. Subcomponents of US Total Market Index based on market capitalization targets: largest 70% constitute large-cap index; next 20%, mid-cap index; final 9.5%, small-cap index.
Security makeup Securities that reflect the U.S. equity markets and, through the markets, the U.S. economy. Equity issues, common stocks, and REITs. Publicly traded companies in the United States with market cap greater than $30 million. U.S. common stocks, REITs, and select non-U.S.- domiciled stocks trading in the United States. U.S. equities traded on NYSE, AMEX, NASDAQ, or NYSE Arca. Publically traded companies in United States with an investable market cap exceeding $20 million including REITs, Master Limited Partnerships, Royalty Trusts, American Depository Receipts, and securities listed on OTC Bulletin Boards.
Liquidity requirements Ratio of annual dollar value traded to float-adjusted market cap should be 1.00 or greater. Company should trade a minimum of 250,000 shares in each of the six months leading up to the evaluation date. Stock must meet minimum threshold for trading volume. Securities must trade above $1 per share on May 31. Measured along two dimensions: the level of stock price and a relative criterion known as Annualized Traded Value Ratio (ATVR). Total market capitalization must be greater than $10 million, float shares must be greater than

10% of total shares, there are trading volume requirements, and a security must not have ten sequential days without trading volume.

Securities must maintain at least $10 million in market capitalization as of review dates.
Growth criteria • Trailing three-year earnings per share (EPS) growth.
• Three-year sales growth.
• Momentum—12-month percentage price change.
• Forward P/E ratio.
• Projected earnings growth.
• Price/book ratio.
• Dividend yield.
• Trailing revenue growth.
• Trailing earnings growth.
• Price/book ratio.
• Long-term earnings growth mean forecast by Institutional Brokers Estimate System (IBES).
• Long- and short-term projected EPS growth rates.
• Current internal growth rate.
• Long-term historical EPS growth trend.
• Long-term historical sales/share growth trend.
• Consensus long-term growth forecast from I/B/E/S.
• Consensus one-year forecast from I/B/E/S.
• Three-year growth in earnings per share.
• Three-year growth in sales per share.
• Current investment-to-assets ratio.
• Return on assets.
• Trailing three-year sales growth.
• Trailing three-year EPS growth.
• Two-year forward sales growth.
• Two-year forward EPS growth.
• Return on equity times retention ratio.
Value criteria • Price/book ratio.
• Price/earnings ratio.
• Price/sales ratio.
• Forward P/E ratio.
• Projected earnings growth.
• Price/book ratio.
• Dividend yield.
• Trailing revenue growth.
• Trailing earnings growth.
• Price/book ratio.
• Long-term earnings growth mean forecast by Institutional Brokers Estimate System (IBES).
• Price/book ratio.
• 12-month forward EPS.
• Dividend yield.
• Book/price ratio.
• Earnings per share consensus forecast one year ahead divided by stock price.
• Most recent EPS divided by stock price.
• Dividend/price ratio.
• Sales/price ratio.
• Price/book ratio.
• Sales/price ratio.
• Dividend yield.
• Cash flow/price ratio.
Growth and value crossover Style indexes- limited overlap between growth and value indexes. Pure style indexes- stocks placed in either growth or value index. 50% of stocks by float-adjusted market cap are in growth index and 50% are in value index. Securities may be classified proportionately in both growth and value indexes. Securities may be classified proportionately in both growth and value indexes. Securities may be classified proportionately in both growth and value indexes. Securities may be classified proportionately in both growth and value indexes.

Alternative methodologies

Although most indexes use market capitalization as a basis for fund selection, there are other types of indexes.


  1. As of year-end 2008, 368 index funds managed total assets of $604 billion. By the end of 2008, the total number of index-based and actively managed ETFs had grown to 728, and total net assets were $531 billion Of these, 12 were actively managed ETFs with less than $250 million in total net assets-- 2009 ICI Factbook p.33; p.40.
  2. Note the MSCI international index designator of "NR", such as MSCI EAFE NR USD. MSCI also supplies indices in local currency (LCL) and US Dollars (USD). MSCI Equity Indices Overview
  3. The data spreadsheet on this page includes data for the Russell 3000 Growth and Russell 3000 Value Indexes
  4. CRSP’s methodology introduces the concept of “packeting,” which cushions movement between adjacent indexes and allows holdings to be shared between two indexes of the same family. At reconstitution, if a company has moved significantly through a breakpoint into the core of an adjacent index, a “packet” of 50% of the total holdings of the company is moved between indexes. The objective is to maximize style purity and minimize turnover.

See also


External links

Index research