This post kicks off a series of posts titled "Opening Pandora's Box", that will cover the untapped wealth that is your corporate logs.
After talking about logging with some people in the customer space, and our engineering and research groups back here at HQ, it's clear to me that logging is more than just something that everyone should be doing... it's like a Pandora's Box that many organizations are almost afraid to tap into.
I thought it would be a good idea to explore this more, so kicking off this series is phase 1 you'll probably go through when you start thinking about logging - anxiety.
Right around now you may be asking yourself... why would anxiety be the first feeling you get when you start to think about logging in your organization?
Elementary my dear Watson - it's because of what you're probably going to find, and you're afraid of it. Think about how well you log right now, and think about how much it tells you about your environment.
Logging can tell you everything from how efficient your servers are, to low often specific IP address ranges belonging to certain countries visit your websites, and even whether you've provisioned enough capacity for your applications in that cloud. Now let's imagine for a minute that you're getting about 1/10th the value from your logs that you can possibly get.
That's right, somewhere around 1/10th the value of what you could be getting - that's around what you're getting today. The other reason that you're probably starting to feel some anxiety is also the white elephant in the room - logs can give you lots and lots of hints on whether you've been hacked at some point, and how bad it is.
Logs are a powerful thing, but most places I've had the pleasure of working with leave the vast majority of their logging treasure untapped. Let me give you an example from real-life, from my past life. In an organization I used to work at we generated around a quarter-terabyte of logs a day... from just the security devices.
I'm not kidding. This only includes things like our firewalls, IPS network, and other security-based devices... I can't even tell you how many lines of logs that was. The maddening thing was that there was absolutely zero chance that we were ever going to look at any of that in anything other than historical search... if that.
The Perl scripts one of my colleagues had written would chug through a days' worth of data on our fastest server with the fastest disk array in just under 5 hours searching for pre-defined patterns in the normalized logs. Again, maddening.
The truly insane part of this whole story is that while we were generating just under 250Gb of log data from our security devices per day - this was only from the security devices and didn't really include anything else. The anything else should have included applications servers, applications, systems, workstations, authentication mechanisms, databases, physical access and so much more.
Here's why I am confident in saying that my team would have gotten anxious any time someone started talking about digging into the logging capabilities and data in our organization - we had no idea what sorts of information was in there.
An interesting story comes from a security engineer I recently talked to at a conference who didn't tell me where he worked, but relayed this story. I think it perfectly illustrates how anxiety is the perfect place to start when you start thinking about logging, and analyzing all those logs you have access to.
I'm paraphrasing now...
"We started by realizing that our syslogd server wasn't enough anymore and that even if we stared at it all day the screen scrolled by way too fast for us to see any of the stuff in red that meant that we were supposed to go investigate it. We upgraded our platform with some trepidation and bought a much bigger machine, adding storage - then we started pushing all of our security systems' logs to the device."
"We noticed weird and abnormal stuff, but none of it was really indicative of anything because we didn't have time to investigate since we were all too busy trying to keep the company from drowning itself in bad IT decisions. When someone suggested we start logging application data per some requirement somewhere in a compliance report - we were all a bit nervous. What would advanced logging capabilities coupled with analytics find that we had missed all this time? "
"The answer quickly smacked us in the face. Once we started getting visibility across many different business units, and many different non-security systems we realized we had so many compromised machines, so many trojaned or hacked systems that we were going to be cleaning up malware and hacks until we were retired... that was a terrifying realization."
Look, logging is simple. You tell the system or application to send log data to device X on port Y, and done.
Wait... you're worried about the analysis aren't you. You're worried that once someone, or some system points out where the issues are and where there are wonky things going on that you're going to have to do something to figure out what the issue really is, and then go on to remediate the situation. You'll then be challenged to continue tweaking, refining queries and search parameters, and gather more and more data to get better and better analytical capabilities.
You're probably worried that this is going to consume all of your time, or that you'll find out that your organization has been pwn3d for years and all your data has already been exfiltrated in real-time, and that as you read this your latest database is being copied off to China... aren't you?
Are you anxious yet? Soon we'll talk about the next stage so that you can get over this anxiety...
Cross-posted from Following the White Rabbit