The Backtesting Crisis | Chief Investment Officer

Most backtests published in finance journals are wrong, a quantitative research specialist from Guggenheim Partners has argued.

The tool is as popular with finance academics as is it misused, according to Marcos López de Prado, a senior managing director, which “may invalidate a large portion of the work done over the past 70 years.”

“You never see a bad backtest. Ever. In any strategy.” —Josh Diedesch, CalSTRSLópez de Prado—who holds two doctorates in the subject and a research fellowship with the Lawrence Berkeley National Laboratory—has been sounding the alarm on the “pseudo-mathematics” of backtesting for several years.

In a recent presentation, he singled out a common and critical error he said undermines the majority of research based on historical simulations: multiple testing.

Want the latest institutional investment industry news and insights? Sign up for CIO newsletters. 

This involves running multiple configurations of, for example, a theoretical asset allocation through a set of past market data then picking one to highlight. The practice poisons the selected results with selection bias, he said.

The American Statistical Society has voiced the same concern in its ethical guidelines.

“Selecting the one ‘significant’ result from a multiplicity of parallel tests poses a grave risk of an incorrect conclusion,” the guidelines stated. “Failure to disclose the full extent of tests and their results in such a case would be highly misleading.”

Grave critiques of backtesting are nothing new to institutional investing. Yet asset owners have tended to focus on managers’ use of the technique in marketing products, such as trumpeting phenomenal modeled performance over bizarre time spans.

“Backtesting: I hate it—it’s just optimizing over history,” California State Teachers’ Retirement System Investment Officer Josh Diedesch told CIO last year. “You never see a bad backtest. Ever. In any strategy.”

And according to López de Prado, academics are just as guilty of the practice as asset managers.

Read Marcos López de Prado’s presentation slides and, for a more in-depth discussion, his paper “Quantitative Meta-Strategies.”

Backtesting slide Source: Marcos López de Prado’s 2015 presentation “Backtesting”

The UK government is not yet acting on their stated intention to have the country’s public pensions invest in its infrastructure, according to local authority pension leaders.

“It’s surprisingly difficult to get a hearing in government around infrastructure.” —Mike Jensen, CIO, Lancashire Pension FundSpeaking at a National Association of Pension Funds (NAPF) conference in May, Lancashire County Council CIO Mike Jensen said he found the government’s lack of engagement with UK pensions “surprising” given previous statements from ministers.

“It’s surprisingly difficult to get a hearing in government around infrastructure,” Jensen said. “Given the noises they have made for a considerable time now, that has been quite surprising.”

NAPF Chief Executive Joanne Segars suggested that ministers saw public pensions “as a useful ATM” but had not done much to help pension funds benefit from privatisation of assets.

Never miss a story — sign up for CIO newsletters to stay up-to-date on the latest institutional investment industry news. 

Last month, it emerged that Lancashire was among the bidders for the government’s £400 million ($612 million) stake in Eurostar, which runs trains between London, Paris, and Brussels. However, the stake was sold to Caisse de dépôt et placement du Québec for more than £750 million in March.

“We tried to persuade government that the idea of retaining that asset within the UK tax base had more value than just the price paid,” he said. “We didn’t get a favourable hearing on that. I think there should be mileage in that for a long term investor.”

Jensen said Canadian funds “have more or less had a stranglehold on UK infrastructure” since the government began selling assets. But he emphasised that the combined assets of the Local Government Pension Scheme (LGPS) was sufficiently large to compete with the Canadians and “to present ourselves to government and private infrastructure sellers as a viable alternative to external investors”.

However, Chris Rule, CIO of the London Pension Fund Authority (LPFA), said public pensions needed to present—and represent— themselves better to ministers in order to gain traction in this area.

“When we talk about LGPS we can talk about a collective, but actually if one of those ministers wants to have a conversation, who do they speak to when they want to speak to us?” Rule said. “There are a number of different avenues they could go to. Finding a way we can engage on those levels is a challenge we all need to think about.”

In December, Lancashire and the LPFA announced plans to combine their resources and assets to create a £10 billion entity. While the partnership is still at a very early stage, Jensen stated that “the potential wins are so great that we should bust every gut to make it come to pass”.

Both Jensen and Rule added that, although the arrangement was not yet open to business, other investors would be welcome to join the partnership, both within the LGPS and externally.