Quantifying Risk Reduction with an Unknown Denominator

Wednesday, March 07, 2012

Rafal Los


I get so many inspirations from my Twitter conversations it's not even funny.  This time it's Jeremiah Grossman who posted the Facebook statistic on mid-year bug bounties, and the subsequent conversation with Martin Fisher that got me thinking.

The problem I see (and no, I'm confident I'm not alone on this thought process) that exists with all these risk reduction measurements is that they're impossible to quantify.  There is simply no way to say that by doing X, you've reduced risk by Y% - at least not when you don't know the total number of issues that exist.  And therein lies the problem.

This is sort of a Catch 22.  If we knew all the risks in a piece of software we wouldn't need software security people around to point them out to us - but let's assume we live in reality land where that's not possible. 

So now we're left with a need to define how much risk we've reduced by identifying and remediating some number of security-related defects - preferably as a percentage for ease of consumption - except we have one major problem.  That problem is that we don't have the denominator in that fraction.  The denominator is the "total number of security defects that exist"... so we're stuck.

Let's take a more concrete example outside the computer world.  Let's think about a used car.  If you're going to purchase a used car you're going to want to be certain that you don't buy a lemon (something full of defects).  Sound familiar? 

So what you want to do is either find them yourself before you buy, or hire someone to help you do it (a professional) before the purchase is made.  Now if you hire that professional, and he or she finds 5 major issues such as dry-rot in the brake lines, a mis-firing spare plug, and so on... how much risk has that removed from your purchase of that used car? 

The legitimate answer is that you don't know, because you aren't sure of the entirety of the things wrong with that car.  So how much was it worth to you to find those things out before you buy?  That's probably easier because we can say with relative certainty that finding issues with brakes and such are big dollar fixes after the purchase - so that can be quantifiable... especially because you can't drive around a car with failed brakes. 

Sadly this doesn't translate into the software world where we have security issues in software - because quite frankly sometimes companies are willing to live with security defects that we aren't comfortable with as security professionals.

Bringing it back down to my core issue here - any time security professionals start to talk about "risk reduction" we have to acknowledge that if the denominator is unknown (meaning, we don't know the total number of risks present in a system) we can't adequately tell anyone how much we've accomplished by removing 1, 10, or 100 of them. 

So we need alternative ways to quantify risk reduction or else take a different approach to demonstrating business value of security.  This would be a great topic to hear what you think on...

Cross-posted from Following the White Rabbit

Possibly Related Articles:
Enterprise Security
Information Security
Enterprise Security Risk Management Analytics metrics Software Security Assurance Information Security Mitigation Security Solution Remediation
Post Rating I Like this!
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.

Most Liked