The more data and back testing that is used to establish correlations and investment factors, the less reliable the research is—or so claims Research Affiliates.
The recent proliferation of data available to analysts and academics has led to the emergence of a huge number of “factors” that the researchers claim can influence investment returns. But the way in which these factors are calculated is highly questionable, according to Jason Hsu, vice-chairman and co-founder of Research Affiliates.
“The newer anomalies [factors] are most likely results of data mining.” —Jason Hsu, Research AffiliatesIn a recent paper, Hsu said he and his colleagues had become “deeply distrustful” of the increasing number of investment factors being proclaimed by new research.
Citing work by Cam Harvey, a professor at Duke University’s Fuqua School of Business in North Carolina, Hsu said 316 investment factors had been identified and published in top journals. However, Research Affiliates’ own “factor robustness” investigations had identified “only a handful of persistent, investable sources of equity returns”.
Traditionally, statistically significant results are indicated by a “t-statistic” of greater than 2. However, Hsu argued that the increasing amount of research—and accelerating pace of new publications—meant that the bar for statistically significant findings should be raised.
“When one runs a back test to assess a signal that is, in fact, uncorrelated with future returns, the probability of observing a t-stat greater than 2 is 2.5%,” Hsu wrote. “However, when thousands upon thousands of such back tests are conducted, the probability of seeing a t-stat greater than 2 starts to approach 100%.”
Professor Harvey concluded that, despite the hundreds of factors identified by dozens of professors and analysts, the main factors achieving a higher t-statistic were still the “old classics” of value, low beta, and momentum.
“The newer anomalies are most likely results of data mining,” Hsu wrote.
“As we add to our research team and thus the number of back tests that we perform in aggregate, we recognize that our ‘false discovery’ rate also increases meaningfully,” he added. In many cases, Hsu said, Research Affiliates only considers results to be significant with a t-statistic of 4 or higher.
In his paper, Hsu also argued that factor investing was insufficient on its own to construct a properly diversified portfolio. Many different combinations of asset classes could be used to gain exposure to a particular factor, if other considerations such as price were ignored, he said.
“In order to create a portfolio with the appropriate exposures at an attractive price, we also need to understand the valuation levels at which the different assets trade,” Hsu said. “Factor-based investing and its complement, asset-class-based investing are, in our mind, incomplete descriptions of the world without each other.”
In a blog post last month, AQR’s Cliff Asness argued that the idea of biased back tests was inaccurate—but his work focused on the factors that Hsu, Harvey, and Research Affiliates also showed to be the most correlated to investment returns.
Jason Hsu’s paper, “The Whole Story: Factors + Asset Classes”, can be downloaded from Research Affiliates’ website.
Related: Active Share, Deconstructed & Asness to Haters: Fama-French Weren’t Wrong. You Are