Podcast: OSSEC, SIEM and Log-based Intrusion Detection Systems - Atomicorp - Unified Security Built on OSSEC

Podcast: OSSEC, SIEM and Log-based Intrusion Detection Systems

Atomicorp’s CEO Mike Shinn walks through his experience with logging, SIEM and OSSEC approaches. He breaks down what is important and how the logging space has evolved over the past 20 years from a security perspective, including the introduction of security automation.

Log-based Intrusion Detection System – LIDS

Log-based intrusion detection (LIDS) was one of the earliest tools used to identify cybersecurity breaches. The software segment has evolved tremendously since the 1990’s and ultimately spawned the Security Information and Event Management (SIEM) category.

Security Information and Event Management – SIEM

SIEM has become popular for log aggregation and visualization but there are other open source tools such as OSSEC that provide similar functionality.

Atomicorp provides unified workload security for cloud, data center or hybrid platforms. Built on OSSEC, the World’s Leading Open Source Server Protection Platform. See our products.


Podcast Transcript: OSSEC, SIEM and Log-based Intrusion Detection Systems

Bret Kinsella: [00:00:00]   This is Episode 4 of the Linux Security Podcast. Today’s topic: Logging, SIEM and OSSEC.

Bret Kinsella: [00:00:16]   All right Mike today we want to talk about OSSEC, logging and SIEM functionality. So why don’t we first talk about logging and why that’s important in the enterprise.

Mike Shinn: [00:00:29]   There’s really at least two reasons that logging tends to be important. There is the reason that tends to fall more in the favor of the engineers themselves which is they need to know what’s going on in the system so that they can fix it or figure out if something bad is happening. There is also for larger enterprises external factors that require them to log things and those tend to be regulatory requirements like PCI DSS, Sarbanes-Oxley for government contractors governance agencies, fisma for European companies. Apparently companies that do business with Europeans, the General Data Protection Regulations (GDPR) that the EU has come out with. So there is both a practical need for logging which is really nothing more than the computer telling you what it’s doing. And then giving you the human being the ability to go back and look at that which may be necessary to figure out if something is broken or why something happened or it could be for something more business related. You might be analyzing the data to figure out who your customers are. So logging has a lot of utilities. From a security perspective, logging gives you the ability to see what’s going on with your system without having to build a very sophisticated piece of software and why that’s important is that there’s an awful lot of software out there doing an awful lot of stuff that can be challenging to instrument. It may be not as simple as just saying “Well, we log everything that our web server does”. But the attack is happening up at an abstraction level kind of above that. The person is maybe interacting with two or three databases behind it through the web application. The web application isn’t actually broken. It can be very, very challenging to figure out what’s going on.

Mike Shinn: [00:02:25]   If the applications are generating telemetry sometimes it’s what people call logs, you can create rules or put some sort of machine learning around that to look at things that are going on in the system to alert you to something that’s suspicious or maybe an attack is going on and so forth. That’s the best use case right try to actually get utility out of the logs but you really can’t diminish the fact that there are regulatory frameworks that absolutely require this that you have to collect data that you have to know what’s going on to some degree of assurance that you’re supposed to look through this data to figure out if something nefarious is or has happened and in some cases you’re supposed to notify certain regulators if you’ve failed to meet the requirements. So there are both practical uses of logging and then I’ll say maybe not so practical. But let’s just say I don’t have a choice with. Right. You have to do it. You’re required to do it. So once this caught on this idea of not just the engineering side of it “Hey I have data and I can look through it” but once the compliance frameworks came in and said “Oh you have all this data great so you need to know what’s going on and you need to know when bad things happen. You need to stop it”, we needed tools to automate that process because there’s entirely too much data. Even in a small organization, it is not only more information than is reasonable for a human being to look through, by the time the human being has discovered something bad has occurred it is already over and ways that really belie the fact that human beings will always lose the race against a computer based attack.

Bret Kinsella: [00:04:16]   So is this where SIEM comes in?

Mike Shinn: [00:04:18]  Well, SIEM came a little later but that’s right. So the first… it’s kind of interesting because you know nothing is really new in cybersecurity it just tends to be either derivative of something before it or really just the same thing we’ve been in decades before… the old term used to be log based intrusion detection because it’s effectively what a SIEM is. Right. So you’re just looking through the logs you’re trying to find things that are bad and you’re alerting on it. So OSSEC was not the first tool out to do this. There was actually some other stuff that came much earlier. There was a tool actually written by one of my colleagues back at Wheel Group… 20 years ago called log check that basically you had a series of regular expressions… rules that would look through logs for various things and would alert you to them. And at the same time it would also… and this is really important for log analysis… you have to do data reduction. It’s not useful to just tell a human being a whole bunch of stuff is happening if the human being then has to then analyze all of that to figure out if something is going on that creates what the human factors people call alarm avalanche. Think the person’s just overwhelmed and either gives up or just simply can’t keep up with what’s going on. So these initial tools probably spent as much time trying to prevent alarm avalanches they did trying to detect things that were bad. And that is a challenging thing to do because people are afraid they might miss something.

Bret Kinsella: [00:05:55]   Well it’s almost the focus of many of them today. We can come back to that. But I think that evolution was security information management. And then there is security event in information management. So that was sort of the evolution. And a SIEM is supposed to help you identify the bad things or the potentially bad things in the box.

Mike Shinn: [00:06:18]>   Yes, either in real time or historically if you go back and you look to see if something occurred and SIEMs and any of these tools can be used certainly for other things that can be used to identify it may even be something bad maybe you just want to see how a particular marketing campaign went. If the logs that you have contain information about who is accessing your websites and your documents and so on and so forth, you could certainly use these technologies to do that. Although to be fair, that’s not really what these tools were initially built for. They were really built as security tools. Not really business analytical tools although that’s certainly the way…

Bret Kinsella: [00:06:58]>   You mean the SIEMs as we know them today?

Mike Shinn: [00:06:59]>   Although they are certainly position that way right. I mean they can be used that way but that is not at all just by the nature of their name. Right. Security event information manager. Not Information and Event managers. Right. Drop the “S” and maybe then you be a little more honest. But it’s data analytical tool based on logs. That’s really what all this stuff is. OSSEC is 13 years old… if I’ve got my math right… and that was one of the first things that it did was log analysis. And there was tools before that did it. This was just taking it to the next level analyzing logs. It does it very well. It’s got a very conservative architecture for doing that. Very paranoid architecture. An interesting thing about logs too is that logs can be used to attack systems. So if you have a tool that passes logs, you have the ability to feed that tool something. So if you’re a bad guy and you can generate an event that is logged, you control the input that goes into the security tools. So some of the first interesting attacks in cybersecurity were against cybersecurity tools themselves. And these were one of the types was to feed malicious data into these applications that would then log it. They would say I got some strange input here. I’m going to make a record of it. So the engineer can come along and look at it and maybe see why this thing failed to function properly. But the attack wasn’t really targeted at the application it was targeted at the thing that was analyzing logs which then typically gave the adversary the ability to really get to the crown jewels of the organization. If they could get to the system that was in charge of notifying people that something bad was happening then they could cover all of their tracks.

Bret Kinsella: [00:08:52]>   Well yeah. The log aggregators connected to all of your system. Once you connect to the log aggregator, you can get to everyone.

Mike Shinn: [00:08:58]>   And it’s trusted. Right. So it’s the thing that tells you that something bad is happening. So it’s sort of akin to to immediately brainwashing your physical security force right. If you could snap your fingers and all your guards were suddenly bad guys, that’s effectively what was occurring here. So there’s been some necessary evolution to the technology to address the fact that to some degree… it’s debatable how much of the data potentially an adversary could control that’s being sent into these tools but some amount of it or maybe a lot of it depending….

Bret Kinsella: [00:09:38]>   Well it could clearly eliminate signal that’s going to go up to the analyst and that’s one of the ways they cover their tracks.

Mike Shinn: [00:09:44]>   It could end up hacking the analyst. I don’t want to be alarmist or whatever. Most of the vendors out there understand this on some level that they know there’s a principle in security… the design of products that we call untrusted data which basically means what it sounds like it’s the data that we don’t trust. Anything that somebody else sends us shouldn’t be trusted. It’s often not easy to know what that is. If you’re sucking in all the data in your enterprise you’re probably not filtering out the stuff that’s untrusted before it gets to your log analyzer. You’re dependent upon whatever’s analyzing those logs to not trust that data. So, it initially was a fantastic tool and then like all things in cybersecurity it’s an arms race. The bad guys figured out a way to use the tool to penetrate the organization and then people had to redesign it so OSSEC really came about around the time when that was occurring. And so there were a number of design choices in the product that were built around allowing it to defend itself and the user of the data from potentially malicious logs that were being sent to the system.

Bret Kinsella: [00:11:03]>   Yeah that makes sense. So we have this. well-known mature space technology space called SIEM. And companies like ArcSight and Splunk and Q Radar are used all over the world by large enterprises… some small but mostly larger enterprises… and then you have OSSEC. So is OSSEC a replacement for those? Is it a compliment to those? Is it both? How are people using it? Vis-à-vis the existing SIEM investment.

Mike Shinn: [00:11:32]>   So you absolutely could use OSSEC to replace any of the SIEM tools that are out there. You could also use it to complement those tools. But it is in some cases older than some of those tools. Certainly its initial purpose was to do the same things that SIEMs do. So it it absolutely can do what a SIEM can do it does a lot more than what a SIEM does because it is also a host based intrusion detection platform and a File Integrity Monitoring System and an active response system. So it’s more than just something that sits in the back of the house and ingests logs. It was initially intended to be a suite of technologies that you would deploy on one or more systems not just to tell you that something bad was happening but to take some immediate action against that malicious behavior to prevent the compromise of the system or the degradation of the system. So it is certainly capable of doing what SIEMs to do. It is also highly extendable. It has a very easy to add to, modify rule language. So it’s not a particularly complicated platform in that sense. And there are a number of GUIs that have been designed to plug into it. But I have seen plenty of organizations that use OSSEC to do the analytics and then to send the post analyzed data on to their SIEMs. So it can complement the existing investments that people have and it could also replace some of those investments.

Bret Kinsella: [00:13:12]>   That makes sense. Thanks a lot Mike for talking about that today.

Mike Shinn: [00:13:15]>   My pleasure.


Atomicorp provides unified workload security for cloud, data center or hybrid platforms. Built on OSSEC, the World’s Leading Open Source Server Protection Platform. See our products.