Historical return on stocks may be a lot lower than first thought

Sounds as if there are some real basic problems with Jeremy Siegel's "Stocks for the Long Run." The Smith and Cole book that he relied on apparently made clear how they constructed their data, and Siegel seems to have not understood or acknowledged these problems. From the WSJ:

Using data assembled by other scholars, Prof. Siegel extended the history of U.S. stock returns all the way back to 1802. He came to two conclusions that became articles of faith to millions of investors: Ever since Thomas Jefferson was in the White House, stocks have generated a "remarkably constant" average return of nearly 7% a year after inflation. (Adding inflation at 3% yields the commonly cited 10% annual stock return.) And, declared Prof. Siegel, "the risks of holding stocks decrease over time."

There is just one problem with tracing stock performance all the way back to 1802: It isn't really valid. . . .

For the years 1802 through 1820, Profs. Smith and Cole collected prices on three dozen banking, insurance, transportation and other stocks -- but ended up including only seven, all banks, in their stock-market index. Through 1845, they tracked 19 insurance stocks, but rejected 95% of them, adding only one to their index. For 1834 onward, they added a maximum of 27 railroad stocks.

To be a good measure of stock returns, an index should be comprehensive (by including many stocks) and representative (by including the stocks commonly held by investors). The Smith and Cole indexes are neither, as the professors signaled in their 1935 book, "Fluctuations in American Business." They cherry-picked their indexes by throwing out any stock that didn't survive for the whole period, whose share prices were too hard to find or whose returns seemed "inflexible," "erratic," or "non-typical."

The database of early U.S. securities at EH.net has so far identified more than 1,000 stocks that were listed on 10 different exchanges -- including Charleston, S.C., New Orleans, and Norfolk, Va. -- between 1790 and 1860. Thus the indexes relied on by Prof. Siegel exclude 97% of all the stocks that existed in the earliest years of the U.S. market, and include only the bluest of the blue-chip survivors. Never mind all of the canals, wooden turnpikes, rubber-hat companies and the other doomed stocks that investors lost millions on -- and whose returns may never be reconstructed.

There is a second problem with Prof. Siegel's data. . . .

"I made an estimate of the dividend yield," Prof. Siegel told me, "through looking at a smaller set of securities and projecting it out." Money manager Robert Arnott of Research Affiliates LLC has recently estimated the early dividend yield at 5.2%. "Arnott has a much lower estimate, and that's a big difference," said Prof. Siegel. "I mean, I don't know what more to say."

I later called Prof. Siegel to ask him again about the difference between his original research and his book, but he didn't get back to me by press time. . . . .