Why Too Much Research Produces False Factors

Research Affiliates’ Jason Hsu has claimed that the proliferation of investment “factors” is based on “data mining” and is inherently flawed.

The more data and back testing that is used to establish correlations and investment factors, the less reliable the research is—or so claims Research Affiliates.

The recent proliferation of data available to analysts and academics has led to the emergence of a huge number of “factors” that the researchers claim can influence investment returns. But the way in which these factors are calculated is highly questionable, according to Jason Hsu, vice-chairman and co-founder of Research Affiliates.

“The newer anomalies [factors] are most likely results of data mining.” —Jason Hsu, Research AffiliatesIn a recent paper, Hsu said he and his colleagues had become “deeply distrustful” of the increasing number of investment factors being proclaimed by new research.

Citing work by Cam Harvey, a professor at Duke University’s Fuqua School of Business in North Carolina, Hsu said 316 investment factors had been identified and published in top journals. However, Research Affiliates’ own “factor robustness” investigations had identified “only a handful of persistent, investable sources of equity returns”.

For more stories like this, sign up for the CIO Alert newsletter. 

Traditionally, statistically significant results are indicated by a “t-statistic” of greater than 2. However, Hsu argued that the increasing amount of research—and accelerating pace of new publications—meant that the bar for statistically significant findings should be raised.

“When one runs a back test to assess a signal that is, in fact, uncorrelated with future returns, the probability of observing a t-stat greater than 2 is 2.5%,” Hsu wrote. “However, when thousands upon thousands of such back tests are conducted, the probability of seeing a t-stat greater than 2 starts to approach 100%.”

Professor Harvey concluded that, despite the hundreds of factors identified by dozens of professors and analysts, the main factors achieving a higher t-statistic were still the “old classics” of value, low beta, and momentum.

“The newer anomalies are most likely results of data mining,” Hsu wrote.

“As we add to our research team and thus the number of back tests that we perform in aggregate, we recognize that our ‘false discovery’ rate also increases meaningfully,” he added. In many cases, Hsu said, Research Affiliates only considers results to be significant with a t-statistic of 4 or higher.

In his paper, Hsu also argued that factor investing was insufficient on its own to construct a properly diversified portfolio. Many different combinations of asset classes could be used to gain exposure to a particular factor, if other considerations such as price were ignored, he said.

“In order to create a portfolio with the appropriate exposures at an attractive price, we also need to understand the valuation levels at which the different assets trade,” Hsu said. “Factor-based investing and its complement, asset-class-based investing are, in our mind, incomplete descriptions of the world without each other.”

In a blog post last month, AQR’s Cliff Asness argued that the idea of biased back tests was inaccurate—but his work focused on the factors that Hsu, Harvey, and Research Affiliates also showed to be the most correlated to investment returns.

Jason Hsu’s paper, “The Whole Story: Factors + Asset Classes”, can be downloaded from Research Affiliates’ website.

Asset management is one of the most profitable businesses in the world, but some managers will have to step up their game to find growth going forward.

According to the Boston Consulting Group (BCG), the industry’s total assets hit a record high of $74 trillion in 2014 and profits soared 7% to $102 billion.

But, BCG warned, the bull market drove substantial growth, rather than new asset flows.

Net new assets remained low at 1.7%, well below pre-crisis levels.

Want the latest institutional investment industry news and insights? Sign up for CIO newsletters. 

BCG 20151 Asset management’s net revenue growth also faltered in 2014, down to 7% from 9% in 2013, the report said, as both institutional and retail investors put pressure on fees.

“What we’re starting to see is asset managers taking a more sophisticated approach to fees,” Brent Beardsley, senior partner at BCG, told CIO. “The most sophisticated managers are continually assessing fees—it’s active management of pricing.”

Because investors are increasingly reevaluating fee terms—some of which have never been adjusted or revisited in 10 years—managers are also lowering cost growth below asset growth, BCG said.

In addition, investors’ preference for passive products from traditional active options is also putting pressure on fees and pricing, the consulting group argued.

And the industry is likely to become even more competitive and challenging going forward.

“Managers face a future in which growth isn’t a given,” Beardsley said. “Achieving growth will require managers to ramp up their execution and generate more value from their commercial go-to-market capabilities—notably in marketing, sales, and pricing.”

However, the industry’s profitability may be limited to only a number of players rendering these improvements and efforts useless.

According to BCG, the top managers in mutual fund flows also captured the majority of new asset flows—and this “winner-takes-all” phenomenon only increased in the US last year.

Data revealed the top 10 US managers—including Vanguard, BlackRock, and Dimensional Fund Advisors—gained 68% of all flows in 2014, compared with 53% in 2013.

BCG 2015

Additional reporting by Nick Reeve.