How Well Do You Measure Risk? | Chief Investment Officer

Forecasts based on typical techniques are “indistinguishable from random noise” for shorter time horizons, according to a Dutch central bank researcher.

The most common forms of risk measurement may not be the most accurate, according to research from Dutch central bank De Nederlandsche Banke (DNB).

The study—an examination of the accuracy and reliability of risk analysis techniques—focused on the difference between value-at-risk (VaR) and expected shortfall (ES). Of the two methods, VaR forecasts were more accurate, found DNB researcher Chen Zhou and the London School of Economics’ Jon Danielsson.

This counters the common view that VaR is “inherently inferior to ES.”

“Perhaps swayed by the theoretical advantages, ES appears increasingly preferred both by practitioners and regulators,” the study’s authors wrote.

Want the latest institutional investment industry news and insights? Sign up for CIO newsletters. 

In fact, international bank regulator the Basel Committee suggested replacing 99% VaR with 97.5% ES in its latest Basel III market risk proposal.

“This will lead to less accurate risk forecasts,” Danielsson and Zhou argued. “If the regulators are concerned with precision, VaR is preferred.”

To compare the two techniques, the researchers conducted risk forecasts for sample sizes ranging from one year to 50 years. They found that both methods were “highly sensitive” to the sample size, with uncertainty increasing “rapidly” as the sample size decreased. For samples below a few thousand days, the uncertainty became “considerable.”

“At the smallest samples, often the most commonly used in practice, the uncertainty is so large that the risk forecast is essentially indistinguishable from random noise,” the researchers wrote.

ES carried more uncertainty than VaR, when the two techniques were projected at the same probability levels and when using the Basel III combination of 97.5% ES and 99% VaR.

The ES technique was found to have one advantage: It was harder to manipulate than VaR. As manipulated risk forecasts would also lack accuracy, this might be a reason to prefer ES, Danielsson and Zhou noted.

But regardless of the method chosen, the researchers said risk forecasts will remain “virtually indistinguishable from random noise” except when derived from large sample sizes.

“Common practices and trends in risk management are misguided,” Danielsson and Zhou wrote. “It is a concern that vast amounts of resources are allocated based on such flimsy evidence.”

Read the full paper, “Why Risk is so Hard to Measure.”

It’s an investment industry sub-sector that accounts for more than $500 billion in assets, and is only getting more important to investors.

But the rise of smart beta presents a significant challenge to investment consultants’ traditional manager selection strategy, according to the chair of the UK’s Railways Pension Scheme (Railpen).

John Chilman, Railpen/First Group John Chilman, chair, RailpenJohn Chilman, who is also pensions director for train and bus operator First Group, said investors in smart beta funds were likely to hold these for longer periods than fully active strategies, thus reducing the need for regular manager searches.

“Investment consultants make a considerable amount of money from manager searches, and they spend a lot of time doing manager research,” Chilman said. “Smart beta is a more passive style and structure, and as such it is likely to reduce the amount of future searches that will happen.”

For more stories like this, sign up for the CIO Alert newsletter. 

As part of a wide-ranging overhaul of its investment strategy, Railpen’s investment team has made a series of significant changes to its £21 billion ($30 billion) portfolio. The UK’s multiemployer fund for the railways sector has drastically reduced hedge fund exposure, and dispensed with silos in favour of a larger, in-house multi-asset growth fund.

“Active management will probably diminish as a mainstream product and things like smart beta will fill some of that gap.”It has also indicated a desire to reduce investment costs: As well as exiting much of its hedge fund allocation, Railpen has seeded two “multi-factor” funds managed by Unigestion. One takes a long-only approach to allocating to “quality,” “momentum,” “value,” and “size” factors, and the other takes a long/short approach.

Chilman predicted that such funds would continue to become more mainstream as there aren’t significant cost barriers.

“It is a far more cost effective model than active management, and that is where I see things moving,” he said. “Active management will probably diminish as a mainstream product and things like smart beta will fill some of that gap.”

One of the main reasons for this, Chilman asserted, is that active management had been allowed to “masquerade” as pure alpha when many strategies had actually been driven by some of the factors now covered by smart beta.

With more education and engagement with what Railpen calls “alternative risk premia,” Chilman said consultants should still be able to recommend such strategies. “There is still plenty of work for the investment consultants, or the in-house team, to put you into the position of understanding your overall position,” he said.

Railpen has committed time and resources to researching alternative risk premia as the fund’s investment team believes there is “tremendous value compared to both traditional passive and active management,” Chilman added.

Lack of a proven track record, however, can hamper investors’ understanding of smart beta products. While most factors are well-researched and backed up by years of academic research, very few smart beta products have track records covering a full business cycle.

“Back testing is all well and good, but does it really work when you are going through something like the global financial crisis?” Chilman said. “Would something around the model have changed at that point that didn’t flow through in the back-testing environment?”

In addition, the rise in popularity could also prove the downfall of smart beta—at least in theory. If there is a sudden rush toward a particular factor, there is the risk of that factor becoming overpriced. “Potentially this will break the model,” Chilman said.

Hence Railpen’s multi-factor approach. At the time of the Unigestion deal near the start of January, Alexei Jourovski, managing director at the Swiss asset manager, said allocating between factors as a market cycle develops should “significantly improve the risk return profile of the strategy.”

Chilman’s interview featured in a recent report by Clear Path Analysis.