- Email: [email protected]
Advanced Analysis of Intrusion Detection Logs
ids log analysis operations in multiple regions of the world, hardware and software may be from the same companies but have different configurations in different regions. In January 2002, certain Sony Vaio computers in parts of Asia and certain Middle Eastern and African countries reportedly had flawed software with a security hole. For many companies that might have considered buying large quantities of the same model of Vaio or another computer at the time, the possibility of different levels of vulnerability configurations in different regions would have seemed remote. Information security experts for larger enterprises need to be aware of potential differences in configurations. In addition, the normal vagaries of human behavior can interfere with the best efforts of systems administrators to maintain a consistent level of security for all computers on the same network. “Pushing” antivirus or other security updates to networked computers may be an efficient strategy in general, but may be ineffective where certain users fail to shut down their computers each night.
A sixth rule of thumb is that the most vulnerable part of the enterprise’s defenses against computer or Internet fraud – the enterprise’s employees and supervisors – cannot be easily “patched.” Human foibles, from misplaced trust to carelessness and even temptation, often provide online criminals with a vulnerability that no software can cure. The employee who reads an email purporting to offer intimate photographs of Catherine Zeta-Jones or other female celebrities, and mindlessly clicks on a link that unleashes a virus into the enterprise’s networks, can undermine expensive and carefully maintained security software and expose valuable data to criminals. Managers should therefore factor human vulnerabilities into their Internet fraud risk calculus, and budget for thorough training, creative prevention and education messaging tailored to the needs of their enterprise, and where necessary build in technological measures to minimize the risks to information assets that momentary misjudgments could trigger.
Advanced Analysis of Intrusion Detection Logs Johan Beckers & Jean Paul Ballerini , Technology Consultants, Internet Security Systems EMEA (www.iss.net)
It is important to know that there are various sources of IDS data, i.e. network, server and desktop, and to have a common understanding of what they detect, and how they can be configured. In this article we will look at ways of analysing this data, focusing on the area of the Security Information Management (SIM) systems where consolidation, aggregation and correlation play a key role.
Introduction There has probably never been a time in human history when anybody felt totally secure. Recently though, we have
seen dramatic changes take place, in terms of the types of threats that need to be uncovered and stopped. Before the emergence of the Internet, the nature of security threats was mostly
This brief overview is no substitute for full-blown planning and implementation of a complete Internet security risk assessment. A truly comprehensive risk assessment needs to take into account a broader range of online threats, from denial-of-service attacks to “social engineering” exploits to highly sophisticated penetrations of corporate systems for economic espionage purposes. It should indicate, however, that computer and Internet fraud deserves more attention from government and private-sector enterprises as a source of potential risk than it has received to date.
About the author Jonathan J. Rusch is Special Counsel for Fraud Prevention, Fraud Section, Criminal Division, US Department of Justice, Washington, D.C. The views expressed in this article are solely those of the author. Mr. Rusch may be contacted at 202-514-0631, 202-514-7021 (fax), or [email protected].
physical, for example break-ins. But a burglar would almost always be exposed to CCTV cameras, or other security devices that are able to record movement. Today though, threats can be 'invisible' and because of the Internet's anonymity and flexibility, the risk of being recognized is much reduced. Today, it is very easy to deploy spyware so that information gathering can run over weeks; slowly but thoroughly. The natural question that comes to mind is: what can be done? Intrusion Detection Systems (IDS) are monitoring tools that are used to detect suspicious, unwanted or illegal activities. The challenges that security specialists are facing today are not just about detecting a key event, but also involve recognizing 9
ids log analysis this event within a very large amount of data produced by IDS systems.
Security information management systems Apart from the various IDS platforms (network, servers and desktops), Security Information Management Systems can collect data from vulnerability assessment scanners, firewalls and antivirus etc. As there is a growing need to simplify the process for companies to find their way through this large amount of information, Security Information Management (SIM) systems have recently become more popular. The first step towards an easier detection/analysis process is to have one console in place where all this data is kept. Many companies wish to monitor everything possible in order to have data for forensic purposes. They would rather have false positives than false negatives. On the other hand, creating a lot of noise and false positives overwhelms the security operator monitoring the events and is a common way to hide a real attack. So where is the proper balance?
There is no simple answer to this problem, but consolidation, aggregation and correlation are three technologies introduced by SIM systems for the advanced analysis of IDS logs that deserve some attention.
Most companies would rather have false positives than false negatives but false positives are a common way to hide a real attack Consolidation, aggregation and correlation Consolidation Have you ever sat in front of a screen where lines scroll constantly? This is exactly what would happen to an operator monitoring an IDS console recording millions of events every day. For how long do you think a specialist
Figure 1: Snapshot of how SQL Slammer looks in IDS logs 10
could cope? This is where consolidation and aggregation come into play. First of all, the data coming from all the various sources is normalised and stored in one unique database. The consolidation process is fundamental as vendors have not yet come to an agreement on standardising the naming of security events. Consequently the majority of events, if not all, receive different names from each vendor. All companies who deploy security software from a range of vendors face the same problem. Thus, without the consolidation process, they are unable, to compare and analyse the data gathered by each product effectively. The Common Vulnerabilities and Exposures (CVE®) Standard is a first step in the right direction. The goal of this initiative is to standardise the names for all publicly known vulnerabilities and security exposures.
Aggregation Once the information has been consolidated, the aggregation process follows. Aggregation will group similar events together and will give the simple answer
ids log analysis to how many times that attack happened over a certain time period. Let's take, for example, an event that happens quite frequently: udp-scan. This event can be easily detected several thousand times a day. Security experts want to be able to group and count these events quickly by port, by source IP, by target IP, by 'business asset' (e.g. Web servers or mail servers), or by whichever set of parameters they like best. In general, the monitoring process should make it simple to change the way you look at events in order to identify anomalous behaviour. A lot of SIMs stop at this stage in order to minimise the amount of data and prioritise the importance of security incidents. But this is not enough. You need to take it one step further with correlation.
Correlation Even after consolidation and aggregation, security experts still have to spend a lot of time figuring out what is really happening in their organization from a security point of view. The events that appear as high-risk still need to be investigated. How many are there? How many can a security expert deal with in a day? How many really require further investigation? Wouldn't it be nice if you could know upfront, without having to manually intervene, which high-risk event or any event is harmless? Most companies today have to investigate all security events. Comparing events on the attacked hosts manually is an expensive process. Not only is this process expensive, but it is not as precise, scalable or as fast as an automated approach. This is where correlation makes a true difference. Being able to do realtime impact analysis of an attack launched against your organization would really help a security expert in prioritising the IDS logs into true security incidents. In order to perform impact analysis effectively, the SIM
needs to know about the operating system and vulnerability information of each device that is being monitored, i.e. every asset belonging to the company and whether any countermeasures are automatically enabled. Impact analysis can be accomplished using the following correlation techniques: • Local correlation Host-based intrusion detection verifies the consequences of the attack locally. It is extremely helpful if the local agent logs whether the attack is blocked or not. On the operator's console it should be possible to have additional information describing the success/failure of the attack and, therefore, the necessity for escalation (or not). • OS correlation When attacks are directed to hosts with the wrong operating systems they cannot be successful. If the
security operator sees an event on the console and at the same time is told that it is harmless, this event can immediately be ignored. It might still be a good idea to take note of the source IP and aggregate all the events coming from that IP. This way it is much easier to monitor and re-enact the activity of suspicious IPs. On the other hand, if it is the correct operating system, correlation has to go one step further. • Attack versus vulnerability correlation Generally, for an attack to be successful it needs to find vulnerabilities. There is a well-defined association between what vulnerability(ies) a system has to have for a specific attack to be successful. By collecting and keeping up-to-date the vulnerability information of the company's assets, the operator can see the event and tell at the same time whether the host is vulnerable or not. In the former case there is a need for further investigation; in the latter the event can be ignored.
Figure 2: Advanced analysis of intrusion detection logs 11
ids log analysis List of possible attack statuses using these correlation techniques: • Success likely (target vulnerable) • Failure likely (blocked some packets) • Failure likely (no vulnerability) • Failure likely (wrong OS) • Failed attack (blocked at host) • Unknown impact (not scanned recently) • Unknown impact (vuln check indeterminate) • Unknown impact (OS check indeterminate) • Unknown impact (no correlation) Another correlation technique is called attack pattern analysis, which can be accomplished using the following correlation techniques: • Directional correlation Correlating incidents and events based on the direction of the attack is critically important during the initial stages of worm propagation. This analysis determines if the events are inbound from sources outside the organization; outbound from sources within the organization; or contained within the organization (to and from internal systems). • Attack pattern recognition It is possible to recognize certain patterns in the way attacks are performed, mainly thanks to automated tools. Script kiddies will download and just run them. By being automated, these tools always trigger the same sequence of events and, therefore, a pattern can be recognized. Consequently only one event need be shown to the operator, instead of a list of correlated events. • Multi-step attack pattern recognition Sometimes the attack pattern spans a long period of time. It is a method used by the more expert hackers to avoid being noticed before they are ready to try to gain control of a target. Correlating events into meaningful incidents highlights important activity and reduces the amount of remaining 12
data requiring manual correlation and investigation. Here are a couple of examples: • Attack from compromised host This correlation event tells a security operator that a certain machine has been attacked, successfully compromised and now this machine is performing attacks on the network itself.
It is possible to recognize certain patterns in the way attacks are performed, mainly thanks to automated tool • Worm compromised host This correlation event (similar to the above) tells a security operator that a worm has infected a certain machine and that this machine is now trying to infect other devices on the network. These attacks are launched in such an automated way that there is no doubt that the host is infected with a worm. Figure 1 contains a snapshot of how SQL Slammer appears in IDS logs. Other possible automatic correlations are: • Logon from compromised host. • Logon to compromised host. • Logon failures against multiple hosts. • Logon from spoofed source. Correlated attacks targeted at different geographical locations of your organization from a single source or intruder would result in one of the following: • Targeted break-in attempt. • Targeted break-in attempt and evasion. • Targeted break-in then program shutdown. • Targeted break-in then program startup. • Targeted DoS. • Targeted DoS successful. • Targeted probing.
• Targeted probing and evasion. • Targeted probing then break-in attempt.
Conclusion Figure 2 helps us visualise easily how Security Information Management systems can automate the advanced analysis of intrusion detection logs. The processes described in this article — consolidation, aggregation and correlation — correspond with today's available technology that enables an automated, advanced analysis of the intrusion detection logs. Thanks to these mechanisms, it is possible for an operator to reduce the number of events that need to be investigated from several thousands or even millions, down to less than 10. All events are detected and yet there is no need for the companies to lighten their policies just to be able to cope with the number of events. It is not necessary to collect the information manually in a time consuming, non-scalable process. All the crucial information is collected in one single place and the really serious events stand out and are easily brought to the attention of the security administrator.
About the authors Johan Beckers is Director of Technology Solutions, EMEA, having joined ISS as a consultant from Cap Gemini in 1997. He has an MBA and a degree in Industrial Engineering, with specialisation in telecommunications. Jean Paul Ballerini studied Computer Science at the University of Bologna where he got his PhD. In 2000 he joined ISS where he has been working as a Pre-Sales Engineer first for the Swiss-Austrian area and now at European level.
Copyright 2003 Internet Security Systems NV
A sixth rule of thumb is that the most vulnerable part of the enterprise’s defenses against computer or Internet fraud – the enterprise’s employees and supervisors – cannot be easily “patched.” Human foibles, from misplaced trust to carelessness and even temptation, often provide online criminals with a vulnerability that no software can cure. The employee who reads an email purporting to offer intimate photographs of Catherine Zeta-Jones or other female celebrities, and mindlessly clicks on a link that unleashes a virus into the enterprise’s networks, can undermine expensive and carefully maintained security software and expose valuable data to criminals. Managers should therefore factor human vulnerabilities into their Internet fraud risk calculus, and budget for thorough training, creative prevention and education messaging tailored to the needs of their enterprise, and where necessary build in technological measures to minimize the risks to information assets that momentary misjudgments could trigger.
Advanced Analysis of Intrusion Detection Logs Johan Beckers & Jean Paul Ballerini , Technology Consultants, Internet Security Systems EMEA (www.iss.net)
It is important to know that there are various sources of IDS data, i.e. network, server and desktop, and to have a common understanding of what they detect, and how they can be configured. In this article we will look at ways of analysing this data, focusing on the area of the Security Information Management (SIM) systems where consolidation, aggregation and correlation play a key role.
Introduction There has probably never been a time in human history when anybody felt totally secure. Recently though, we have
seen dramatic changes take place, in terms of the types of threats that need to be uncovered and stopped. Before the emergence of the Internet, the nature of security threats was mostly
This brief overview is no substitute for full-blown planning and implementation of a complete Internet security risk assessment. A truly comprehensive risk assessment needs to take into account a broader range of online threats, from denial-of-service attacks to “social engineering” exploits to highly sophisticated penetrations of corporate systems for economic espionage purposes. It should indicate, however, that computer and Internet fraud deserves more attention from government and private-sector enterprises as a source of potential risk than it has received to date.
About the author Jonathan J. Rusch is Special Counsel for Fraud Prevention, Fraud Section, Criminal Division, US Department of Justice, Washington, D.C. The views expressed in this article are solely those of the author. Mr. Rusch may be contacted at 202-514-0631, 202-514-7021 (fax), or [email protected].
physical, for example break-ins. But a burglar would almost always be exposed to CCTV cameras, or other security devices that are able to record movement. Today though, threats can be 'invisible' and because of the Internet's anonymity and flexibility, the risk of being recognized is much reduced. Today, it is very easy to deploy spyware so that information gathering can run over weeks; slowly but thoroughly. The natural question that comes to mind is: what can be done? Intrusion Detection Systems (IDS) are monitoring tools that are used to detect suspicious, unwanted or illegal activities. The challenges that security specialists are facing today are not just about detecting a key event, but also involve recognizing 9
ids log analysis this event within a very large amount of data produced by IDS systems.
Security information management systems Apart from the various IDS platforms (network, servers and desktops), Security Information Management Systems can collect data from vulnerability assessment scanners, firewalls and antivirus etc. As there is a growing need to simplify the process for companies to find their way through this large amount of information, Security Information Management (SIM) systems have recently become more popular. The first step towards an easier detection/analysis process is to have one console in place where all this data is kept. Many companies wish to monitor everything possible in order to have data for forensic purposes. They would rather have false positives than false negatives. On the other hand, creating a lot of noise and false positives overwhelms the security operator monitoring the events and is a common way to hide a real attack. So where is the proper balance?
There is no simple answer to this problem, but consolidation, aggregation and correlation are three technologies introduced by SIM systems for the advanced analysis of IDS logs that deserve some attention.
Most companies would rather have false positives than false negatives but false positives are a common way to hide a real attack Consolidation, aggregation and correlation Consolidation Have you ever sat in front of a screen where lines scroll constantly? This is exactly what would happen to an operator monitoring an IDS console recording millions of events every day. For how long do you think a specialist
Figure 1: Snapshot of how SQL Slammer looks in IDS logs 10
could cope? This is where consolidation and aggregation come into play. First of all, the data coming from all the various sources is normalised and stored in one unique database. The consolidation process is fundamental as vendors have not yet come to an agreement on standardising the naming of security events. Consequently the majority of events, if not all, receive different names from each vendor. All companies who deploy security software from a range of vendors face the same problem. Thus, without the consolidation process, they are unable, to compare and analyse the data gathered by each product effectively. The Common Vulnerabilities and Exposures (CVE®) Standard is a first step in the right direction. The goal of this initiative is to standardise the names for all publicly known vulnerabilities and security exposures.
Aggregation Once the information has been consolidated, the aggregation process follows. Aggregation will group similar events together and will give the simple answer
ids log analysis to how many times that attack happened over a certain time period. Let's take, for example, an event that happens quite frequently: udp-scan. This event can be easily detected several thousand times a day. Security experts want to be able to group and count these events quickly by port, by source IP, by target IP, by 'business asset' (e.g. Web servers or mail servers), or by whichever set of parameters they like best. In general, the monitoring process should make it simple to change the way you look at events in order to identify anomalous behaviour. A lot of SIMs stop at this stage in order to minimise the amount of data and prioritise the importance of security incidents. But this is not enough. You need to take it one step further with correlation.
Correlation Even after consolidation and aggregation, security experts still have to spend a lot of time figuring out what is really happening in their organization from a security point of view. The events that appear as high-risk still need to be investigated. How many are there? How many can a security expert deal with in a day? How many really require further investigation? Wouldn't it be nice if you could know upfront, without having to manually intervene, which high-risk event or any event is harmless? Most companies today have to investigate all security events. Comparing events on the attacked hosts manually is an expensive process. Not only is this process expensive, but it is not as precise, scalable or as fast as an automated approach. This is where correlation makes a true difference. Being able to do realtime impact analysis of an attack launched against your organization would really help a security expert in prioritising the IDS logs into true security incidents. In order to perform impact analysis effectively, the SIM
needs to know about the operating system and vulnerability information of each device that is being monitored, i.e. every asset belonging to the company and whether any countermeasures are automatically enabled. Impact analysis can be accomplished using the following correlation techniques: • Local correlation Host-based intrusion detection verifies the consequences of the attack locally. It is extremely helpful if the local agent logs whether the attack is blocked or not. On the operator's console it should be possible to have additional information describing the success/failure of the attack and, therefore, the necessity for escalation (or not). • OS correlation When attacks are directed to hosts with the wrong operating systems they cannot be successful. If the
security operator sees an event on the console and at the same time is told that it is harmless, this event can immediately be ignored. It might still be a good idea to take note of the source IP and aggregate all the events coming from that IP. This way it is much easier to monitor and re-enact the activity of suspicious IPs. On the other hand, if it is the correct operating system, correlation has to go one step further. • Attack versus vulnerability correlation Generally, for an attack to be successful it needs to find vulnerabilities. There is a well-defined association between what vulnerability(ies) a system has to have for a specific attack to be successful. By collecting and keeping up-to-date the vulnerability information of the company's assets, the operator can see the event and tell at the same time whether the host is vulnerable or not. In the former case there is a need for further investigation; in the latter the event can be ignored.
Figure 2: Advanced analysis of intrusion detection logs 11
ids log analysis List of possible attack statuses using these correlation techniques: • Success likely (target vulnerable) • Failure likely (blocked some packets) • Failure likely (no vulnerability) • Failure likely (wrong OS) • Failed attack (blocked at host) • Unknown impact (not scanned recently) • Unknown impact (vuln check indeterminate) • Unknown impact (OS check indeterminate) • Unknown impact (no correlation) Another correlation technique is called attack pattern analysis, which can be accomplished using the following correlation techniques: • Directional correlation Correlating incidents and events based on the direction of the attack is critically important during the initial stages of worm propagation. This analysis determines if the events are inbound from sources outside the organization; outbound from sources within the organization; or contained within the organization (to and from internal systems). • Attack pattern recognition It is possible to recognize certain patterns in the way attacks are performed, mainly thanks to automated tools. Script kiddies will download and just run them. By being automated, these tools always trigger the same sequence of events and, therefore, a pattern can be recognized. Consequently only one event need be shown to the operator, instead of a list of correlated events. • Multi-step attack pattern recognition Sometimes the attack pattern spans a long period of time. It is a method used by the more expert hackers to avoid being noticed before they are ready to try to gain control of a target. Correlating events into meaningful incidents highlights important activity and reduces the amount of remaining 12
data requiring manual correlation and investigation. Here are a couple of examples: • Attack from compromised host This correlation event tells a security operator that a certain machine has been attacked, successfully compromised and now this machine is performing attacks on the network itself.
It is possible to recognize certain patterns in the way attacks are performed, mainly thanks to automated tool • Worm compromised host This correlation event (similar to the above) tells a security operator that a worm has infected a certain machine and that this machine is now trying to infect other devices on the network. These attacks are launched in such an automated way that there is no doubt that the host is infected with a worm. Figure 1 contains a snapshot of how SQL Slammer appears in IDS logs. Other possible automatic correlations are: • Logon from compromised host. • Logon to compromised host. • Logon failures against multiple hosts. • Logon from spoofed source. Correlated attacks targeted at different geographical locations of your organization from a single source or intruder would result in one of the following: • Targeted break-in attempt. • Targeted break-in attempt and evasion. • Targeted break-in then program shutdown. • Targeted break-in then program startup. • Targeted DoS. • Targeted DoS successful. • Targeted probing.
• Targeted probing and evasion. • Targeted probing then break-in attempt.
Conclusion Figure 2 helps us visualise easily how Security Information Management systems can automate the advanced analysis of intrusion detection logs. The processes described in this article — consolidation, aggregation and correlation — correspond with today's available technology that enables an automated, advanced analysis of the intrusion detection logs. Thanks to these mechanisms, it is possible for an operator to reduce the number of events that need to be investigated from several thousands or even millions, down to less than 10. All events are detected and yet there is no need for the companies to lighten their policies just to be able to cope with the number of events. It is not necessary to collect the information manually in a time consuming, non-scalable process. All the crucial information is collected in one single place and the really serious events stand out and are easily brought to the attention of the security administrator.
About the authors Johan Beckers is Director of Technology Solutions, EMEA, having joined ISS as a consultant from Cap Gemini in 1997. He has an MBA and a degree in Industrial Engineering, with specialisation in telecommunications. Jean Paul Ballerini studied Computer Science at the University of Bologna where he got his PhD. In 2000 he joined ISS where he has been working as a Pre-Sales Engineer first for the Swiss-Austrian area and now at European level.
Copyright 2003 Internet Security Systems NV