Log Consolidation, SIEM or Both?

Sunday, October 17, 2010

John Verry


In the "old days" (pre-2008) there was a fundamental decision to make when implementing log management technology. 

Do I go log consolidation (LogLogic, LogRythm, Splunk) or do I go Security Information Event Management (Arcsight, NetForensics, Sentinel)? 

It really boiled down to whether or not you needed the increased capabilities of SIEM such as  real-time correlation, dynamic real time data normalization, and advanced integration with other core systems including Identity Management, Network Monitoring, Ticketing, CMDB, & Vulnerability/Configuration Management.

The world has changed. We are of the strong opinion that for most F500 class log management implementations it’s no longer SIEM OR Log Consolidator -- its now SIEM AND Log Consolidator.

The reason is simple; a growing, insane number of events that need to be captured. 

As compliance requirements relating to PCI, PII, PHI, et al drive the need for a greater and greater number of events to be captured, one of SIEM’s advantages (having a RDBMS as its back-end) can quickly become a disadvantage. 

When data rates start to approach 5,000 events per second (which is definitely reachable in many environments) the challenges/limitations associated with getting data into and out of an RDBMS can become problematic: 

  • Few DBAs have experience optimizing databases with these characteristics.
  • Database storage requirements (often on a very expensive SAN) can approach a Terabyte per day.
  • Recovering from a database issue (e.g., rebuilding a corrupted index) becomes very challenging and time consuming.

Leading vendors such as Arcsight and Novell (Sentinel) recognized this issue and have recently developed Log Consolidators that are intended to work seamlessly with their SIEMs so that you can enjoy a best of breed approach to log management/forensics/compliance.

Leveraging Log Consolidators at the “edge” (e.g., at various business units) simplifies the process of deploying a solution, provides a local search-able event repository, and reduces storage requirements by a factor of 10 (or more). 

Only those events that require SIEM capabilities (say 20%) are forwarded to the SIEM in real-time to ensure that full SIEM functionality is retained while eliminating the pain associated with a 5,000 EPS RDBMS).

This approach really allows an organization to end up with the “best of both worlds”.

Possibly Related Articles:
Log Management SIEM
Post Rating I Like this!
Robb Reck My experience has been that companies on both sides of the aisle have been moving to provide both kinds of services. The LogLogics and Splunks are adding a lot more SIEM-type activity, while the SIEMs have integrated log aggregation to their products. I'd like to see the distinction between the two markets fade and see companies evaluated more on their ability to serve both focuses, rather than one of the other.
John Verry Robb,

Good point - I agree. As my experience is more SIEM than Log Consolidation the blog emphasizes that direction.
Anton Chuvakin Another point often missed is that many organization are nowhere near ready for anything near-real-time (SIEM), for monitoring, etc. For example, if yo don't have an IR/IH plan, it is clear that SIEM won't do you any good....

They might all "need" both, but few are ready for it.
John Verry Your comment is so true. Several years ago we were in the early stages of standing up a significant SIEM solution when it became obvious that a public facing B2B application was under attack. Their CISO turned to our technical lead and said "Now what?"

Standing up a SIEM is one thing ... operationalizing it is another.
Robb Reck I bet that company came up with a IR plan pretty darn fast. While it's a tough to do that kind of learning on the fly, it's a lot better than sitting there ignorant. Yay for more knowledge.
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.