An Important Idea

“When an expert provides a calculation of the probability of an outcome, they are really providing the probability of the outcome occurring, given that their argument is watertight.”
An interesting paper making its viral way through the web, which we came across on the Alea blog, presents a very thought provoking assessment of small probability, high stakes risks. From the abstract:

“When an expert provides a calculation of the probability of an outcome, they are really providing the probability of the outcome occurring, given that their argument is watertight.”

The authors, from Oxford University, use as an example the risk that the Large Hadron Collider could destroy the earth, and all of us with it (spoiler alert, they think this would be a bad thing), to lay out a framework based on breaking down risk estimates into three components: theory, model and calculation.

Risk and allocation:

That the authors are from outside the standard finance community is another recommendation. Though there are some fruitful leads they don’t quite connect. One is the relative unimportance of precise probability estimates given high enough stakes. For example, the risk differential between playing Russian Roulette with a gun with one empty chamber and one full chamber versus a gun with 5 empty chambers and one full is mathematically very large but in practice would probably not effect most people’s decision of whether or not to play. Also, the importance of analyzing competing hypotheses would be an important extension of their work, which only gets a passing mention at the end of the conclusion. For example, the crazy guy in Time’s Square holding a sign saying “The End is Near” shirt probably won’t make me support any particular policy, but Neil DeGrasse Tyson on The Daily Show talking about an asteroid hitting the earth may make me more inclined to support spending some tax dollars. The authors mention a tantalizingly elegant approach:

“Finally, we can reduce the possibility of unconscious bias in risk assessment through the simple expedient of splitting the assessment into a ‘blue’ team of experts attempting to make an objective risk assessment and a ‘red’ team of devil’s advocates attempting to demonstrate a risk, followed by repeated turns of mutual criticism and updates of the models and estimates (Calogero 2000). Application of such methods could in many cases reduce the probability of error by several orders of magnitude.”

I have not read the referenced paper by Calogero, but the idea is one I’ve seen work informally but consistently on various Wall Street trading floors--there’s always “that guy” who is a pain-in-the-butt contrarian, sometimes I’ve even been that guy. But I’ve never seen it applied systematically applied by risk management. In fact, Wall Street management structure seems to be the product of an attempt to reduce devil’s advocate behavior to a minimum. Compensation structures have no way to pay the devil’s advocates, if they are wrong they’re just trouble-causers, if they are right, you never know how right unless you ignore them, in which case your firm blows up.

Implicit in all research into risk assessment is the need to make resource (generally, capital) allocation decisions. I’ve seen arguments about whether normal or lognormal models are more appropriate delay the development of risk systems...for years. I’ve been told by the head of risk management at a large firm that my practice of calculating risk according to several different models did him no good, he just needed to know which one I thought was right. (As opposed to him choosing a model he wanted to use to aggregate my risks with the risks of other trading desks, which I could have supported. But he was accepting risk measures being generating by incompatible approaches.) Risk management groups should not be concerned about what the right models are, but but what are the risks associated to using particular models (or by extension, particular traders) and how to allocate firm capital accordingly. As one colleague used to put “Tell me the size of my sandbox and I’ll play in it.” All other concerns are at best distractions and at worst serve to convince management that everything is under control, and we’ve seen how that works out.

bondgeek blog