Accusations that data mining, or biased sampling, is behind core investment factors’ outperformance are demonstrably false, according to AQR Founder Cliff Asness.
He set about proving it in his latest blog post, “It’s Not Data Mining—Not Even Close.”
“Our most potent weapon in addressing data mining is the out-of-sample test,” wrote Asness, a Ph.D-holder and former head of quantitative research for Goldman Sachs. “If a researcher discovered an empirical result only because she tortured the data until it confessed, one would not expect it to work outside the torture zone.”
Asness selected three factors—momentum, value, and size—identified by his “all-but-dissertation” supervisors Eugene Fama and Ken French in 1992 and much replicated since. Shifting into full quant mode, he backtested the factors for the years following the original sample’s end (1992 to 2015), an extended version of the original sample (1927 to 1991), and Fama-French’s sample itself (1963 to 1991).
SMB: Size HML: Value UMD: Momentum. Source: AQR
Value, momentum, and small-cap factors outperformed over all three sample periods, albeit by less in the extended sample and after Fama and French had publicly identified them. Asness acknowledged that the results provided some “exceptionally minor support” for critics of the research. “However,” he continued, “the cynics are supping on a thin gruel.”
Even in the weakest periods, each factor outperformed to an economically and statistically significant degree. “If at the end of 1991 you invested in these factors and achieved the above results you would be ecstatic without reservation,” the AQR founder pointed out.
Data mining—the statistically flawed backtesting practice of running regressions over multiple time periods and selecting the most favorable result—has drawn increasing ire from academic and industry finance circles.
“If you’re still hawking this story, you have been completely defeated on the field of financial battle, and you must stop.” —Cliff Asness, AQR
Another prominent quant, Guggenheim Partner’s Marcos López de Prado, recently argued it “may invalidate a large portion of the work done over the past 70 years” in financial research.
Data-related criticisms of Fama and French’s landmark study began in the ’90s and continue to emerge. Late last month, for example, financial adviser Michael Edesess published a long screed on “Why ‘Smart Beta’ Is Really Dumb Beta,” calling Fama and French’s work an exercise in “promiscuous data mining.”
Asness agreed that data mining “is a reasonable worry for the overall field now.” Given the results of his out-of-sample testing, however, “it is no longer a reasonable worry for the original research that found these factors.”
“If you’re still hawking this story,” he concluded, addressing Edesess and his fellow critics, “that the original results of Fama and French—and even yours truly and others—were the result of data mining, you have been completely defeated on the field of financial battle, and you must stop.”
Related Content:The Backtesting Crisis & Asness Skewers Hedge Fund Rich List’s ‘Bad Math’