One of the most important questions you can ask about a risk system is very simply “how do you know it’s correct?” There is no shortage of methods people have tried to answer that question, so let’s take a look at how some of these might apply to risk management.
#1: Appeal to Authority: basically, this is taking someone’s advice on it, presumably from some risk expert. Unfortunately, this is little better than the “trust me” argument.
#2: Safety in Numbers: you do what everyone else does. Pension funds often find themselves in this situation, especially with risk measures that compare them to their peers or to their benchmark. Let’s take tracking error, for example. It measures the volatility of the difference of the fund’s returns to its benchmark’s returns, and the idea is to have as low a tracking error as you can. While this may seem like a way to decrease risk, it really isn’t – it’s just a way to ensure that whatever happens to the benchmark (rise, crash, devaluation, etc.) is sure to happen to the fund. In general, the herd mentality works until it doesn’t, usually in a spectacular way.
#3: You’ve Seen it Before: familiarity breeds acceptance. Behavioural economics and cognitive sciences show us that humans are more likely to accept something just because they’ve heard the words before. Combined with point #2 above, this can pack a mean punch. Let’s take the case of using beta as a measure of risk.
For decades, pension funds have used beta as a relative measure of risk. When your fund is structured to mimic a benchmark – even a blended benchmark like the traditional 60/40 split, then beta is usually a high-quality analytic and it does indeed measure the relative volatility of the fund to that of the benchmark. A beta of 1.1 means the fund is 10% more volatile than the benchmark, while a beta of 0.9 means it’s 10% less volatile. But this use of beta rests on the supposition that the fund’s returns actually do track the benchmark’s returns, the measure of which is usually called “R-squared.” Many people know that if you don’t have a high enough R-squared, then the results can’t be relied on. Unfortunately, many people ignore this fact in practice. What most people don’t know is that “R” is actually the correlation of the fund to the benchmark, so R-squared can also be called “correlation squared.” Today, many pension funds invest in alternatives, which are designed to have a low correlation to other investments. A typical hedge fund correlation to a market index might be 0.2, making the R-squared a meaningless 0.04. In this case, beta is simply not a meaningful concept. Yet, many people continue to use it, oblivious to the fact that it carries no basis in reality. It’s somewhat akin to talking about the results of a medical test when the test is shown to be invalid. It’s worse than wrong – it’s more like voodoo or astrology. It’s downright dangerous.
#4: Test and Reject What Fails. This is the only method that consistently works. Back testing of Value-at-Risk, for example, can show when it works and when it doesn’t. Examining R-squared can tell you when beta is meaningful and when it’s not. As important as it is to verify that your risk analytics are working the way you expect, the critical thing is to stop using a given risk measure while it’s not working. While we’re at it, we should also look into what new risk measures have actually been shown to be more effective.
Dr. Damian Handzy is chair and chief executive officer, Investor Analytics.