Stock market indexing

The growing market acceptance of index based investing through the media of traditional mutual funds and exchange traded funds has resulted in a proliferation of index providers, attracted to the business prospects of licensing their indexes to investment companies and seeing increasing royalty and licensing fee revenue from the growing asset base. For investors this growth has led to an ever expanding universe of available index funds. However, the different methodologies that index providers utilize in measuring and carving up the stock market leads to a dispersion of returns that can be considerable over short and intermediate term time frames.

Benchmark indexes
The benchmark indexes created by the major index providers share a number of characteristics.

Benchmark index strategy box
These characteristics place benchmark indexes in the Passive security selection/ Capitalization security weighting segment of the Index Strategy Box (see Figure 1.) The index weighting of securities is based on the market capitalization of the companies included in the index.



Figure 1. Index Strategy Box
 * }

Benchmark index style box
In general, the stock market is composed of 3 levels of market capitalization and 3  styles, resulting in a 3 x 3 "style" box which includes large cap, mid cap and small cap stocks, divided among value, blend, and growth stocks. This is commonly represented in a style box as illustrated below:

One may also see an expanded breakdown of market cap ranges to include Mega cap, Micro cap, and Nano cap stocks.

In addition to index provider companies, the Center for Research in Security Prices CRSP has since 1960 provided the academic community with market return data extending from 1926 to the present. CRSP offers Historical Cap-Based Portfolio data tracking micro, small, mid and large-cap stock. For the cap-based portfolios, CRSP ranks all NYSE companies by market capitalization and then divides them into ten equally populated portfolios. Amex and NASDAQ stocks are then placed into the deciles determined by the NYSE breakpoints, based on their market capitalization. CRSP portfolios 1-2 represent large cap stocks, portfolios 3-5 represent mid-caps, portfolios 6-8 represent small caps, and portfolios 9-10 represent a benchmark of micro-cap stocks.

Note: since 2010, CRSP also provides its own investable capitalization-based indexes, which are now tracked by index funds like Vanguard, and defined in a different way than the historical cap-based portfolios (see Index Methodology).

Major index providers
The market coverage of major index providers is supplied in Figure 2. below.

Index types
Usually, an index provider supplies three different index returns for its market segment:


 * A Price Index (Price, PR) which does not include dividends or dividend reinvestment.
 * A Gross Dividend Index (Total Return, TR) which includes all dividends reinvested.
 * A Net Dividend Index (Net Return, NR) which recognizes that the dividends are subject to tax and not received by the investor. Only the remaining portion of the dividend (net of tax) can be reinvested.

For benchmarking international indices, the Net Return index is most commonly used. The Morningstar Index Returns identifies indices by type.

US total market
The first index to measure the entirety of the US public market, the Wilshire 5000 Index, was established in 1974 by Wilshire Associates. Russell introduced the Russell 3000 Index, measuring 98% of the US Market, in 1984. S&P created its first broad based index of the US market in 1994 with the S&P Composite 1500 index, comprised of the S&P 500, S&P 400, and S&P 600 indices. The index measures approximately 85% of the market. The first retail index fund based on a total market index became available in 1992.

The advent of the twenty first century saw the expansion of index providers in the US market, as well as a steady broadening of provider indexes to encompass the total market. Russell added 1000 micro cap stocks to its Russell 3000 index to create its broadest index, the Russell 3000e (extended), measuring 99% of the US market. In 2006, S&P created its S&P Completion Index, which in combination with the S&P 500 index produced the S&P Total Market Index. MSCI, Dow Jones, and Morningstar also created broad based total market indexes in the first decade of the twenty first century. (See Figure 1. for a chart of index provider market coverage.)

At present, only Russell provides total market style indexes (Russell 3000 Value and Russell 3000 Growth).

The similar breadth of index provider market coverage in total market indexes leads to very low turnover of stocks and in a low range of investment return dispersion, especially over long holding periods (see US total market index returns for data). The differential in return is most often attributed to the breadth of micro cap stocks in a total market index. For example, Morningstar (97%), Russell (98%), and MSCI (99.5%) total market indexes hold lower ranges of micro cap market coverage than does the Wilshire 5000 index (99.9%). 

Size indices
 

US small cap
Russell introduced the first small cap index (the Russell 2000) in 1984. Other index providers created small cap indexes in the last decade of the twentieth century: S&P in 1992; MSCI in 1996; Morningstar 1998. The first retail small cap index fund, based on the Russell 2000 index, was initiated in 1989. More institutional investors' small cap allocation assets are benchmarked to the Russell 2000 index than to any other provider.

Size indices


International and global


Index methodology
Index providers differ in the ways they select and weight securities in the benchmarks’ return calculation. There are three key issues:
 * 1) Security selection and weighting
 * 2) Calculation of returns
 * 3) Ongoing index maintenance

In their attempts to make their indexes more easily investable for index funds, index providers have adopted free-float weighting and buffer zones to their methodologies. Free float weighting eliminates non-trading shares in a company's capital base. These can include cross-ownership of shares by other companies, government owned shares, privately held shares, and other restricted shares. A company's free-float weighting will reflect the actual amount of shares available for public investment.

All equity-index providers periodically adjust membership to reflect market changes as well as those made necessary by corporate actions such as mergers or spin-offs. Buffer zones were introduced to reduce the amount of turnover incurred as stocks migrate across market capitalization and style boundaries. Reduced turnover helps reduce transaction costs for funds tracking an index.

Each major index provider has a different methodology for determining buffer zones. For example:
 * CRSP: CRSP U.S. Equity Indexes Methodology Guide
 * Dow Jones: Dow Jones Size Segment and Style Indexes Methodology
 * FTSE: FTSE UK Index Series Ground rules
 * MSCI: MSCI US Equity Indices Methodology (Nov 11)
 * Russell: Russell US Indexes Construction and Methodology
 * Wilshire: Wilshire U.S. Style Indexes: A Methodology Overview

A summary of methodologies is shown in the table below.

Alternative methodologies
Although most indexes use market capitalization as a basis for fund selection, there are other types of indexes.

Index research

 * Yahoo finance indices - Every major US and world index with real-time charts, list of components, and historical prices
 * MSCI Index Definitions - Includes market holdidays (worldwide), MSCI Base Dates, MSCI Earliest Index History Dates
 * MSCI Resource Center - MSCI fact sheets, methodologies, publications
 * Russell Index Performance Calculator | FTSE Russell, this tool allows you to view total returns for any of Russell's US equity indexes over customizable date ranges. Data starting from January 1, 1995.

Tutorial

 * Index Investing: What Is An Index?, from Investopedia