Root Cause Analysis (RCA): A Critical Skill

Thursday, May 24, 2012

Rafal Los


Recently at TakeDownCon in Dallas I brought up a term during my offense keynote that I thought everyone in the audience would, and should, be familiar with.

The concept of Root Cause Analysis (RCA) should be a familiar principle to you if you've ever worked the defensive side of information security (or warfare) or if you've ever done any software reverse-engineering or hacking.  

RCA knowledge isn't limited to hackers - anyone doing any sort of incident response should be familiar with performing root cause analysis to identify failures' roots, to unmask the source of a failure and figure out how to keep it from happening again.

If you think about what someone fuzzing an application input format does, the RCA is critical to figure out why something crashed, then how to exploit it in a meaningful way. Unfortunately, when I asked who was familiar with root cause analysis only a few hands out of the whole room went up.  This was a bit distressing.

Root Cause Analysis (RCA) is a critical skill for security professionals, no doubt, so why don't more analysts, penetration testers, and others have that skill or at least baseline knowledge?

There may be several reasons for this condition - one of which is the desperate need for information security professionals in today's workforce.  Whereas in the past I think many of the best minds in security "worked their way up" through the ranks of IT, many today find themselves going straight from college or a non-related profession directly into information security.  

These folks never have the opportunity to accumulate some of those baseline skills that make identifying failures, and creating mitigations a critical role of information security professionals.  It's not about pointing fingers or laying blame, it's about education and making sure the base level of knowledge to be a good security professional is consistent throughout our industry.

I definitely consider the ability to understand and perform an effective root cause analysis part of a security professional's toolkit.  It's baseline knowledge you should have, if you want to be effective in information security.  I further think it is skills like the ability to perform a thorough root cause analysis that separates the "hackers" from the "button-pushers" in the industry.

What makes a good root-cause analysis, anyway?  An RCA is an analysis of a failure to determine the first (or root) failure that cause the ultimate condition in which the system finds itself.  If you're looking at an application crash you should be thinking - "why did it crash this way?"  Your job in performing an RCA is to keep asking the inquisitive "Why?" until you run out of room for questions, and you're faced with the problem at the root of the situation.

Let's take for example an application that had its database pilfered by hackers.  The ultimate failure you may be investigating is the exfiltration of consumer private data, but SQL Injection isn't what caused the failure.  

Why did the SQL Injection happen?  Was the root of the problem that the developer responsible simply didn't follow the corporate policy for building SQL queries?  Or was the issue much more subtle such as a failure to implement something like the OWASP ESAPI in the appropriate manner?  Or maybe the cause was a vulnerable open-source piece of code that was incorporated into the corporate application without passing it through the full source code lifecycle process?  

Your job when you're performing an RCA is to figure this out.

On the offensive side of the coin, you should be able to figure out the root causes of the failures you generate.  "Because the tool  broke the app" doesn't cut it.  I've often heard "I piped garbage into the application inputs until it crashed" from people fuzzing applications.  When asked to repeat the failure they often can't duplicate the condition because they didn't adequately identify the root cause of the initial failure - so the problem often goes unresolved.

Root cause analysis is super-critical in the software security world.  When you're looking at a web application from a dynamic perspective (as many of you do when you test web apps) you often find several parts of the application which are vulnerable to something like cross-site scripting (xss).  

Several, sometimes dozens, of form fields in various parts of the application appear to fail at filtering and handling input appropriately.  If you were to give this condition to a developer to fix - you're going to be in for a fight because all you're doing is telling them where in the application the issues lie, but not in the code.  

Fact is, I recently saw an application with upwards of 50 cross-site scripting vulnerabilities in a web application that all that the same root-cause.  A root-cause analysis (which HP's web application security testing technology can now do in a mostly automated fashion) can link those 50+ XSS issues to a single line of code in the application input handler.  

This sort of efficiency, and pin-point accuracy  allows for not only a more collaborative conversation with development, but for a faster time-to-fix - which is the grail of software security.

Think about that the next time you're testing an application and you find dozens of SQL Injection or Cross-Site Scripting vulnerabilities ... are they all just a single vulnerability which manifests itself in many parts of the application?  Wouldn't you want to know that?

Cross-posted from Following the White Rabbit

Possibly Related Articles:
Information Security
Application Security Methodologies Forensics Incident Response Attacks Network Security Investigation Mitigation Analysis Root Cause Analysis
Post Rating I Like this!
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.