When the California State Teachers’ Retirement System announced that it was swapping its homegrown private-equity benchmark for one created by a private vendor, the move caught CIOs’ attention.

After all, CalSTRS, the nation’s second-largest pension fund, was acknowledging that its internal benchmark was flawed—and also touting that it had found a better measurement tool for an asset class that long has befuddled managers. CalSTRS now uses State Street’s GX Private Equity Index as its main measuring stick for private equity.

CalSTRS CIO Christopher Ailman says State Street’s benchmark is a major step up from the fund’s in-house metric. But he adds the caveat that any private-equity benchmark remains, almost by definition, a work in progress.

“I’ll be the first to say it’s not perfect, but it’s the best out there,” Ailman said. “Compared to a blended market index, it has a much tighter correlation.”

After CalSTRS announced the change in 2014, Ailman received numerous inquiries from other CIOs with questions about honing their private-equity measurement tools.

“We’re all looking for a better mousetrap,” Ailman said. “We’re all looking for better benchmarks, better measurement tools.”

The challenges with finding just the right metric for opaque assets underscores the broader benchmarking battles being fought by CIOs, who face frustration on multiple fronts. There’s the issue of measuring their own performance, a task that grows more daunting with more complex portfolios. There’s also the matter of picking the proper benchmarks to measure outside managers, and policing managers who try to game the metrics.

Finally, CIOs face the tough task of explaining to investors, boards and policymakers why a diversified portfolio isn’t keeping pace with the double-digit returns being served up by one widely watched index commonly used as a benchmark, the Standard &Poor’s 500.

Amidst those challenges, CIOs and consultants say, benchmarks often are misused and misunderstood even by sophisticated investors.

“It’s almost to the point of rarity when benchmarks are used properly, and with any thought,” said Rob Brown, CIO at wealth management firm United Capital and former CIO of the Arizona Public Safety Personnel Retirement System.

CIOs who don’t hit their benchmarks might find themselves out of a job or facing new pressures to meet new benchmarks from their board. Many talented CIOs have left their positions and vacancies are appearing in the field. And Harvard Management Co. has shuffled through CIOs in recent years as the endowment has underperformed its Ivy League peers. Harvard’s $36 billion endowment in January said it would lay off half its staff.

Meanwhile, the recent explosion of ETFs leaves CIOs with a dizzying array of potential benchmarks from which to choose. They can slice and dice the S&P 500 with a smart beta measure that culls the poor performers from the index, or they can rely on data from FactSet, Bloomberg or Thomson Reuters.

They can create their own benchmarks using free datasets from sources such as Dartmouth College finance professor Kenneth French. Or CIOs can outsource their benchmarking to providers such as Northfield Information Services or State Street.

“Yes, there’s information overload, and, yes, there’s an explosion of benchmarks,” said Norton Reamer, founder of Asset Management Finance in New York and co-author of the book “Investment: A History”.

In the case of CalSTRS, which held assets totaling $196 billion at the end of 2016, Ailman is satisfied with the pension plan’s use of benchmarks for its stock and bond portfolios. Those holdings are easily tracked against dynamically weighted benchmarks based on the S&P 500 and the MSCI World Index, Ailman said.

The real challenge comes in measuring illiquid assets such as real estate (which made up 12.6% of CalSTRS’ portfolio as of December 31) and private equity (an 8.4% allocation).

CalSTRS attempted to use real estate investment trusts (REITs) as a proxy for its property holdings, but that approach proved flawed. REITs mirror financial stocks more closely than privately held real estate, Ailman said.

The pension fund also looked at building a private equity benchmark based on the Russell 2000 and on PE firms that had gone public, including The Blackstone Group and Fortress Investment Group, but that, too, proved inaccurately correlated with private equity returns.

CalSTRS’ $16 billion in private equity holdings includes funds managed by Bain Capital, Carlyle Group and Yucaipa, among others.

The frustrating attempts at building benchmarks for those assets led CalSTRS to turn to State Street’s private equity index. CalSTRS has used the GX Private Equity Index since 2007, first as its secondary benchmark and recently as its main measuring stick.

Some argue that pension plans pay too much attention to the relative returns measured by benchmarks and too little heed to absolute returns – after all, real returns are what allow the monthly checks to retirees to clear.

“The most important thing for pension funds is that you meet your liabilities,” said Mike Heale, principal at CEM Benchmarking in Toronto. “Your fund exists for a purpose, and that’s the most important benchmark.”

Dan diBartolomeo, CEO of Northfield Information Services in Boston, agrees that too fine a focus on benchmarks can obscure the overarching need for real returns.

“In the real world, you don’t make pension payments with benchmark-relative returns,” diBartolomeo said.

At CalSTRS, Ailman said he tries to strike a balance between watching overall returns and keeping an eye on benchmarks. They’re an invaluable tool for grading your own decisions, measuring your staff and paying bonuses.

“At the end of the day, what matters is absolute return,” Ailman said. “But as you’re driving down the road, you want a gauge.”

Benchmarking Outside Managers

Managing managers against their benchmarks is another constant challenge for CIOs.

Active managers who are underperforming often use benchmark confusion to mask poor performance.

“You hide behind the fact that it’s so difficult to find the right benchmark,” says Shashi Mehrotra, CIO at The Legend Group in Palm Beach Gardens, Florida.

Hewing too closely to benchmarks sometimes deters investors from taking risk and making smart investment decisions, CIOs say.

“Everybody’s compensation is based on their performance against the benchmark, so you tend to be a little tighter in your decisions,” Wayne Schmidt, chief investment officer at Gradient Investments in Arden Hills, Minnesota. “People primarily get paid with bonuses, and the bonus is tied to how you do against the index.”

Sometimes the pressure to stick to the index can yield costly decisions. Schmidt recalls one fixed-income manager who was urged in early 2008 to take a bigger position in Lehman Brothers to more closely mirror his benchmark index. Six months later, Lehman collapsed.

Ailman decries managers who hug an index too closely as “benchmark aware.” They’re charging a fee for mirroring performance a CIO could get for free.

“We’re paying these people to make money, to outperform the benchmark. I can own the benchmark at no cost,” he said. “We’re paying them ridiculous fees. If it’s an active fund and you’re charging fees, you need to beat the benchmark.”

Even when a manager soundly beats a passive index, the feat frequently is accomplished by luck rather than skill, United Capital’s Brown said.

That means CIOs must use outperformance or underperformance as a reason to ask probing questions. If a manager beats the S&P 500 by 200 basis points for the year, the CIO has a list of queries to ask. How did the manager outperform? What decisions did the manager make?

And the answers can lead to more questions, Brown said. Perhaps the manager runs a highly concentrated portfolio, in which case 200 basis points might not have been sufficient outperformance to outweigh the added risk.

Or, if a manager avoids financials, comparing his portfolio against a benchmark that includes financials won’t give an accurate reading; the portfolio will outperform when banks are down and underperform when they’re up. In that case, Brown said, you’ll want to create a bespoke benchmark that strips out financial stocks.

The notion of using an index as a primary measure of investment success or failure has deep roots in the money management field. No less an eminence than Warren Buffett long has espoused the idea that a manager should be judged on relative return rather than absolute return.

“I would consider a year in which we were down 15% and the Dow declined 25% to be much superior to a year when both the partnership and the Dow advanced 20%,” a young Buffett wrote to early investors, according to the 2016 book “Warren Buffett’s Ground Rules”.

But Northfield Information’s diBartolomeo says that mindset can go too far. He faults many CIOs for giving managers a free ride when it comes to losing money in bad years.

“Say the benchmark fell by 20% and the manager was down 17%,” diBartolomeo said. “Are they a hero because they beat the benchmark by 3%, or are they an idiot because they lost 17%?”

He urges CIOs to build an incentive for absolute returns into their benchmarks, perhaps by blending the performance of an underlying index with cash. Such a move would dampen returns in a bull market, but would mitigate losses in a crash.

“Absolute returns and risk matter, but that’s not being communicated in mandates to managers,” diBartolomeo said.

For diBartolomeo, it’s a mistake to use the words benchmark and index as synonyms. He says an index can be turned into a benchmark only with effort and thought.

Any discussion of whether active managers are earning their keep takes place against a backdrop of investors’ sweeping shift from actively managed funds to passively managed portfolios. Amid a mounting belief that too few active managers beat the market, most money now is moving into passive funds.

Many veteran investors say CIOs need to make sure that they, and not outside managers, choose the measuring stick.

“It’s like the old line that if you don’t know who the patsy is at the poker table, you’re probably the patsy,” says William Bernstein, author of “The Four Pillars of Investing” and manager of a $320 million portfolio. “You should pick the benchmark.”

A final benchmark-related complication comes when a highly visible index outpaces the rest of the market. That’s the case with the S&P 500, which since the market bottom of 2009 has outperformed riskier emerging markets.

The S&P 500 surged again in 2016, rising 10% before counting dividends. The outperformance of U.S. stocks creates an unusual situation where taking on risk doesn’t mean reaping greater rewards.

“It has been a very difficult index to beat, because it has been outperforming every other index,” Legend Group’s Mehrotra said.

Mehrotra tries to patiently explain that a responsible strategy relies on diversification and asset allocation, and that loading up on U.S. stocks wouldn’t be the responsible call.

For pension plans, the strength of the S&P 500 means managers must communicate to board members and policymakers about why the fund didn’t match the headline-grabbing, double-digit returns of U.S. stocks. And for CIOs handling money for individual investors, the questions are similar.

individual investors, the questions are similar. “The outperformance and the S&P being the top dog is not going to last forever,” Gradient’s Schmidt said. “But it’s a challenge, in that we have a lot of explaining to do.”