- Email: [email protected]
Incident Response
incident response this relies on a company sending spam to behave in an honest fashion after all, what is to stop them from ignoring a request to opt out? These two different legislative models will result in “chaos” said Wyatt. “I think what is lacking is effective enforcement of legislation and one of the main reasons why there is a lack of effective enforcement is that it is often quite difficult to trace spammers,” said Hendrie-Liano.
ISPA would also like the UK Department of Trade and Industry to consider how individuals breaking the new EU directive will be inves-tigated and the action they will face and clarify whether sending spam will constitute a civil or criminal offence. If it is deemed to be a criminal offence then it thinks the processes set out in the Regulation of Investigatory Powers Act (RIPA) should apply. Hopefully Microsoft’s legal actions will serve as a deterrent to spammers,
Incident Response Jon David Extensive logging of events is regularly done by security devices such as firewalls and routers. Significant time is spent reviewing these logs to determine if systems are under attack, and funds are spent in purchasing event correlation software to readily combine the output of separate sources in the hope of detecting attacks missed by the separate reviewing of individual logs. In addition, intrusion detection systems are put in place to alert of intrusions, and results from such systems are both individually reviewed and coordinated with other information. Detecting the occurrence of incidents is clearly of great importance to enterprises. Little attention, though, has been paid to the ways to properly respond to incidents of an actually or even potentially damaging nature. This writing discusses what is involved in proper incident response.
Begin at the beginning Most people look at incident response as stopping a penetration attempt or successful penetration. While stopping such things is certainly part of incident response, it is neither the beginning or the end of such efforts. Incident response should be in accordance with formal security policy in general, and a formal incident response plan in particular. This incident response plan should itself be based on the security policy. For example, a company might want to prosecute intruders if practical, and be in an industry where certain internal information is sacred. Sometimes intruders must be allowed to continue theirs attacks, while the victim organization monitors their actions to collate enough forensic evidence for a prosecution. The incident response staff must know not to allow this if sacred
information is involved, but to allow it up to a point if non-critical information is being compromised. Which information is sacred, and the point to which other less information may be deliberately allowed to be compromised, must be exactly spelled out as a preliminary to incident response actions. An incident response plan will encompass much, much more than activities relating to possible prosecution. What is important to realize is that there cannot be proper incident response without a formal incident response plan because this plan provides the “right” and “wrong” contexts in which incident response is performed.
And end at the end It happens. The beepers beep, the cell phones ring, an attack has been detected! All appropriate personnel take the actions
and will make them realise that the industry is committed to fighting them and that if caught, action will be taken. Although the Summit is a step in the right direction it would seem that there remains much to be resolved but with a concerted effort from all parties involved and international collaboration the problem of spam could be under control sooner rather than later.
prescribed in the incident response plan and, with any luck, can stem the attack. Everybody is satisfied, their jobs have been done. Not really, though. While the primary aim of incident response is to stop or at least deflect an ongoing incident, secondary aims are often to discover how the incident occurred (so it can better be prevented from happening again) and to determine what damages it may have done (so these may be rectified as best possible). These are after the fact operations, so eliminating the dangers of an ongoing attack is not the end of incident response. Another major consideration is recovery, i.e. getting systems back to their pre-attack status. There might also be the analyses necessary for prosecution, and these can differ significantly from merely discovering what happened and how it happened – This involves discovering the “who” involved, and determining if sufficient admissible evidence exists to make prosecution feasible. All activities of this sort may be viewed as post-incident actions, but are still part of the incident response activities and should be detailed in the incident response plan.
But do not mess up in the middle The area of incident response which is traditionally of most concern is the actual immediate reaction to an attack. Telling somebody how to best react to an incident, though, is comparable to telling them what to do if something is wrong with their car. With a car, the treatments will be drastically different if you feel the tires wobbling as you drive, if the car slows
17
incident response much too slowly when you apply the breaks, if there is a grinding sound whenever the car shifts gears, and so on. Similarly, the appropriate incident response will depend on the incident – or, more likely, the symptoms of the incident and the sources of information about it. Does the nature and source of information about the incident suggest an impending (or actual) distributed denialof-service attack? Are websites being defaced? Are files being deleted or improperly altered? Does the information alerting you to the incident come from your perimeter defenses (which might suggest the places to look for successful penetrations) or from internal integrity checkers (which are telling you damages have been done and where they have been done)? Things like these must be considered before diving into full force incident response activities. (I say “full force” because incidents such as website defacements may have automatic website restoration processes available to deal with such symptoms of what may be broader attacks.) As before, the ways to act under the varying combination of information types, information sources and incident symptoms should be set forth in an incident response plan.
Tools of the trade Just as tools are a prerequisite to making auto repairs and preliminary diagnoses, so, too, are tools necessary in determining the nature of incidents and appropriate responses, and to the actual performance of many response activities. If, for example, your incident response activities are brought on by firewall reports of a large number of improper access attempts, you would logically want to look at the firewall activity logs. There could easily be tens of thousands of entries in such logs, though. There are tools that make extraction of information of interest relatively fast, easy and meaningful. Further, such tools will usually provide “drill down” capabilities so that if you find a set of rejected entries to a series of ports from a single source, you can immediately determine if any attempts from that source were not rejected (and would implicitly demand immediate further attention).
18
If there is a chance you may have been compromised, all servers in the realm of the compromise should really be rebuilt if they may have been compromised. Such rebuilding is a very time consuming and operationally intrusive activity, however, and is strongly resisted by internal business units that rely on those servers for their continued operation. There are tools that can quickly tell you if a system has changed in any critical area. (By far the most popular one is Tripwire, a Unix product, but there are also Tripwire for Windows versions.)
The basic need The common factor in the above sections has been the necessity to be prepared in order to be able to perform effective incident response. While a few examples of the need for such preparedness were given, it is much more widespread than suggested. Up front, you need an incident response plan based on security policy which, in turn, is based on risk analysis. If your operations are such that they fall under any governmental regulatory requirements, awareness of those requirements must be reflected in your incident response plan. If your operations are multinational, they must satisfy appropriate governmental requirements in the countries in which they are put into effect, and such requirements differ widely from country to country. Even if your operations are within a single country, regulations are often different from region to region within that country. In addition to the governmental requirements being important in prioritizing incident response activities, requirements for the admissibility of evidence for prosecution can similarly differ. These differences due to geographic location may well necessitate the creation of many different incident response plans, or at least significantly different sections within otherwise similar plans. Your priorities must be spelled out up front. If two new hires are to be involved in your incident response activities, and one comes from an environment where staying online took precedence over improper data access and the other came from a background where data had to be protected at all costs, they will intuitively
react differently to many incidents. The incident response plan must be such that, after reading and understanding it, both will react in the same way to the same set of events. You must have capable people in all appropriate locations properly trained in incident response activities, and familiar with their particular incident response plan. You must have the appropriate tools on hand, have staff available that knows how to use them and properly interpret their results. (TCPDUMP is a helpful tool in many instances, but it is of virtually no help if there is nobody available capable of getting important information from its results.) Some tools require pre-incident setup. For example, Tripwire compares a “snapshot” of critical areas of a current system to one of the same systems when it was known to be clean. To be useful, the reference (i.e. clean system) “snapshots” must be available for comparison, and this is something that has to be done before any incidents may have compromised the systems. Incident response teams should be formalized prior to the occurrence of incidents. While most would view such groups as consisting of various technical experts, many decisions that have to be made when alternatives are available suggest the need for representation of the Risk Department. Similarly, whether for prosecution or just the possible dismissal of any internal personnel who may be involved, Legal Department representation should be present, and maybe even Human Resources personnel. Many venues today have legal requirements that incidents must be publicly disclosed but, even without such requirements, word might well leak out. This means that Public Relations people should also be involved. While not everybody need be physically present, they should all be available via predetermined telecommunications channels. Unless this is all set up ahead of time, the likelihood that it will come together at the time of an incident is rather low... Special requirements may require the availability of special hardware or software. If prosecution is a goal of incident response, it may be necessary to get unalterable
vulnerability analysis storage devices (e.g. WORM drives) on which to store images of system to be investigated and possibly altered. If remote checks of systems are necessary, software to allow such checks to be made must be presently both locally and at the remote locations. It may be necessary to involve varieties of outside vendors. If, for example, you are worried about the impact of distributed denial-of-service attacks, you might arrange with your ISP (or ISPs) for traffic threatening to overwhelm you to be effectively choked at the edge before impacting you. (And, if one of these attacks does hit you, you had better have established out of band channels of communications – telephone numbers, pagers, whatever – so you can communicate with appropriate parties.) Another reasonable example is contracting for incident response support in areas you cannot yourself service. (This may be either ongoing or on demand support, depending on both your situation and budget.)
Another problem While it is nice to think that we are all equal, is it more reasonable to recognize
that we at best come into the world potentially equal and then evolve differently. This means that not everybody involved in your incident response team will be as good as everybody else on it. If you look at incident response as a 24 x7 operation and if you have incident response teams in many locations, there may be a significant variance in the capabilities of the personnel at different locations at different times. When you consider normal employee turnover, vacations, illnesses and the like which keep “regular” staff from responding, the variance in the capabilities is greater still. Unless we perfect and accept human cloning, this is something we will have to live with. As before, you should anticipate this natural situation and be prepared for it. Your incident response plan might be structured to start with initial actions within the province of anybody reasonably assigned to an incident response team, and move from there to activities requiring more knowledge and experience. If the individuals on a given incident response team do not have these advanced capabilities (which in practice is more often the case than not), specific channels of communication with
The Big Picture on Big Flaws RPC DCOM Vulnerability — What went wrong? Thomas Kristensen, CTO, Secunia Today we can look back on a month where too many IT managers were caught by surprise by the RPC DCOM vulnerability and the subsequent exploits and worms. If we look at how the vulnerability could be exploited, it is clear that an attacker or a worm needs to be able to establish network connections to ports 135/udp, 135/tcp, 139/tcp, 445/tcp, 593/tcp or another port where a DCOM enabled service is listening.
These ports are related to services like RPC (Remote Procedure Call), NetBIOS or CIFS (file sharing). They all provide access to sensitive services and carry sensitive unencrypted data. These ports have been the weakest point in Windows for a long time.
appropriate personnel should have been specified ahead of time, and these auxiliary personnel should have been alerted when the incident was originally detected so they will be ready to respond when and if called on.
Summary and conclusions The “secret” to effective incident response is preparedness. Although the main element of this preparedness is certainly a proper incident response plan, other things, such as the procurement of necessary hardware and software and the training of personnel, are also important. While stopping or mitigating the impact of any incident or attack is of prime importance, it must be done in accordance with the explicit dictates of an incident response plan to be truly effective. Secondary goals such as evaluating the extent of damages done, restoring systems to pre-incident status, determining the causes of the incidents and taking steps to minimize the likelihood that incidents of those types will occur in the future are also important, albeit not of as immediate a nature. Preparation is the key.
For years they have been the target for attacks: • Null sessions. • User enumeration. • Access using default account names, simple passwords or null passwords. • SPAM using Windows Messaging pop-up's. • And many more. In other words these services were never intended to be used on the Internet or other “insecure” networks. These services are intended to be run on internal networks only, any firewall and router in a company should filter this traffic inbound and outbound. It is the first lesson taught at any basic IT security course.
Why did big companies get hit? One problem is portable systems which aren't properly configured with antivirus,
19
ISPA would also like the UK Department of Trade and Industry to consider how individuals breaking the new EU directive will be inves-tigated and the action they will face and clarify whether sending spam will constitute a civil or criminal offence. If it is deemed to be a criminal offence then it thinks the processes set out in the Regulation of Investigatory Powers Act (RIPA) should apply. Hopefully Microsoft’s legal actions will serve as a deterrent to spammers,
Incident Response Jon David Extensive logging of events is regularly done by security devices such as firewalls and routers. Significant time is spent reviewing these logs to determine if systems are under attack, and funds are spent in purchasing event correlation software to readily combine the output of separate sources in the hope of detecting attacks missed by the separate reviewing of individual logs. In addition, intrusion detection systems are put in place to alert of intrusions, and results from such systems are both individually reviewed and coordinated with other information. Detecting the occurrence of incidents is clearly of great importance to enterprises. Little attention, though, has been paid to the ways to properly respond to incidents of an actually or even potentially damaging nature. This writing discusses what is involved in proper incident response.
Begin at the beginning Most people look at incident response as stopping a penetration attempt or successful penetration. While stopping such things is certainly part of incident response, it is neither the beginning or the end of such efforts. Incident response should be in accordance with formal security policy in general, and a formal incident response plan in particular. This incident response plan should itself be based on the security policy. For example, a company might want to prosecute intruders if practical, and be in an industry where certain internal information is sacred. Sometimes intruders must be allowed to continue theirs attacks, while the victim organization monitors their actions to collate enough forensic evidence for a prosecution. The incident response staff must know not to allow this if sacred
information is involved, but to allow it up to a point if non-critical information is being compromised. Which information is sacred, and the point to which other less information may be deliberately allowed to be compromised, must be exactly spelled out as a preliminary to incident response actions. An incident response plan will encompass much, much more than activities relating to possible prosecution. What is important to realize is that there cannot be proper incident response without a formal incident response plan because this plan provides the “right” and “wrong” contexts in which incident response is performed.
And end at the end It happens. The beepers beep, the cell phones ring, an attack has been detected! All appropriate personnel take the actions
and will make them realise that the industry is committed to fighting them and that if caught, action will be taken. Although the Summit is a step in the right direction it would seem that there remains much to be resolved but with a concerted effort from all parties involved and international collaboration the problem of spam could be under control sooner rather than later.
prescribed in the incident response plan and, with any luck, can stem the attack. Everybody is satisfied, their jobs have been done. Not really, though. While the primary aim of incident response is to stop or at least deflect an ongoing incident, secondary aims are often to discover how the incident occurred (so it can better be prevented from happening again) and to determine what damages it may have done (so these may be rectified as best possible). These are after the fact operations, so eliminating the dangers of an ongoing attack is not the end of incident response. Another major consideration is recovery, i.e. getting systems back to their pre-attack status. There might also be the analyses necessary for prosecution, and these can differ significantly from merely discovering what happened and how it happened – This involves discovering the “who” involved, and determining if sufficient admissible evidence exists to make prosecution feasible. All activities of this sort may be viewed as post-incident actions, but are still part of the incident response activities and should be detailed in the incident response plan.
But do not mess up in the middle The area of incident response which is traditionally of most concern is the actual immediate reaction to an attack. Telling somebody how to best react to an incident, though, is comparable to telling them what to do if something is wrong with their car. With a car, the treatments will be drastically different if you feel the tires wobbling as you drive, if the car slows
17
incident response much too slowly when you apply the breaks, if there is a grinding sound whenever the car shifts gears, and so on. Similarly, the appropriate incident response will depend on the incident – or, more likely, the symptoms of the incident and the sources of information about it. Does the nature and source of information about the incident suggest an impending (or actual) distributed denialof-service attack? Are websites being defaced? Are files being deleted or improperly altered? Does the information alerting you to the incident come from your perimeter defenses (which might suggest the places to look for successful penetrations) or from internal integrity checkers (which are telling you damages have been done and where they have been done)? Things like these must be considered before diving into full force incident response activities. (I say “full force” because incidents such as website defacements may have automatic website restoration processes available to deal with such symptoms of what may be broader attacks.) As before, the ways to act under the varying combination of information types, information sources and incident symptoms should be set forth in an incident response plan.
Tools of the trade Just as tools are a prerequisite to making auto repairs and preliminary diagnoses, so, too, are tools necessary in determining the nature of incidents and appropriate responses, and to the actual performance of many response activities. If, for example, your incident response activities are brought on by firewall reports of a large number of improper access attempts, you would logically want to look at the firewall activity logs. There could easily be tens of thousands of entries in such logs, though. There are tools that make extraction of information of interest relatively fast, easy and meaningful. Further, such tools will usually provide “drill down” capabilities so that if you find a set of rejected entries to a series of ports from a single source, you can immediately determine if any attempts from that source were not rejected (and would implicitly demand immediate further attention).
18
If there is a chance you may have been compromised, all servers in the realm of the compromise should really be rebuilt if they may have been compromised. Such rebuilding is a very time consuming and operationally intrusive activity, however, and is strongly resisted by internal business units that rely on those servers for their continued operation. There are tools that can quickly tell you if a system has changed in any critical area. (By far the most popular one is Tripwire, a Unix product, but there are also Tripwire for Windows versions.)
The basic need The common factor in the above sections has been the necessity to be prepared in order to be able to perform effective incident response. While a few examples of the need for such preparedness were given, it is much more widespread than suggested. Up front, you need an incident response plan based on security policy which, in turn, is based on risk analysis. If your operations are such that they fall under any governmental regulatory requirements, awareness of those requirements must be reflected in your incident response plan. If your operations are multinational, they must satisfy appropriate governmental requirements in the countries in which they are put into effect, and such requirements differ widely from country to country. Even if your operations are within a single country, regulations are often different from region to region within that country. In addition to the governmental requirements being important in prioritizing incident response activities, requirements for the admissibility of evidence for prosecution can similarly differ. These differences due to geographic location may well necessitate the creation of many different incident response plans, or at least significantly different sections within otherwise similar plans. Your priorities must be spelled out up front. If two new hires are to be involved in your incident response activities, and one comes from an environment where staying online took precedence over improper data access and the other came from a background where data had to be protected at all costs, they will intuitively
react differently to many incidents. The incident response plan must be such that, after reading and understanding it, both will react in the same way to the same set of events. You must have capable people in all appropriate locations properly trained in incident response activities, and familiar with their particular incident response plan. You must have the appropriate tools on hand, have staff available that knows how to use them and properly interpret their results. (TCPDUMP is a helpful tool in many instances, but it is of virtually no help if there is nobody available capable of getting important information from its results.) Some tools require pre-incident setup. For example, Tripwire compares a “snapshot” of critical areas of a current system to one of the same systems when it was known to be clean. To be useful, the reference (i.e. clean system) “snapshots” must be available for comparison, and this is something that has to be done before any incidents may have compromised the systems. Incident response teams should be formalized prior to the occurrence of incidents. While most would view such groups as consisting of various technical experts, many decisions that have to be made when alternatives are available suggest the need for representation of the Risk Department. Similarly, whether for prosecution or just the possible dismissal of any internal personnel who may be involved, Legal Department representation should be present, and maybe even Human Resources personnel. Many venues today have legal requirements that incidents must be publicly disclosed but, even without such requirements, word might well leak out. This means that Public Relations people should also be involved. While not everybody need be physically present, they should all be available via predetermined telecommunications channels. Unless this is all set up ahead of time, the likelihood that it will come together at the time of an incident is rather low... Special requirements may require the availability of special hardware or software. If prosecution is a goal of incident response, it may be necessary to get unalterable
vulnerability analysis storage devices (e.g. WORM drives) on which to store images of system to be investigated and possibly altered. If remote checks of systems are necessary, software to allow such checks to be made must be presently both locally and at the remote locations. It may be necessary to involve varieties of outside vendors. If, for example, you are worried about the impact of distributed denial-of-service attacks, you might arrange with your ISP (or ISPs) for traffic threatening to overwhelm you to be effectively choked at the edge before impacting you. (And, if one of these attacks does hit you, you had better have established out of band channels of communications – telephone numbers, pagers, whatever – so you can communicate with appropriate parties.) Another reasonable example is contracting for incident response support in areas you cannot yourself service. (This may be either ongoing or on demand support, depending on both your situation and budget.)
Another problem While it is nice to think that we are all equal, is it more reasonable to recognize
that we at best come into the world potentially equal and then evolve differently. This means that not everybody involved in your incident response team will be as good as everybody else on it. If you look at incident response as a 24 x7 operation and if you have incident response teams in many locations, there may be a significant variance in the capabilities of the personnel at different locations at different times. When you consider normal employee turnover, vacations, illnesses and the like which keep “regular” staff from responding, the variance in the capabilities is greater still. Unless we perfect and accept human cloning, this is something we will have to live with. As before, you should anticipate this natural situation and be prepared for it. Your incident response plan might be structured to start with initial actions within the province of anybody reasonably assigned to an incident response team, and move from there to activities requiring more knowledge and experience. If the individuals on a given incident response team do not have these advanced capabilities (which in practice is more often the case than not), specific channels of communication with
The Big Picture on Big Flaws RPC DCOM Vulnerability — What went wrong? Thomas Kristensen, CTO, Secunia Today we can look back on a month where too many IT managers were caught by surprise by the RPC DCOM vulnerability and the subsequent exploits and worms. If we look at how the vulnerability could be exploited, it is clear that an attacker or a worm needs to be able to establish network connections to ports 135/udp, 135/tcp, 139/tcp, 445/tcp, 593/tcp or another port where a DCOM enabled service is listening.
These ports are related to services like RPC (Remote Procedure Call), NetBIOS or CIFS (file sharing). They all provide access to sensitive services and carry sensitive unencrypted data. These ports have been the weakest point in Windows for a long time.
appropriate personnel should have been specified ahead of time, and these auxiliary personnel should have been alerted when the incident was originally detected so they will be ready to respond when and if called on.
Summary and conclusions The “secret” to effective incident response is preparedness. Although the main element of this preparedness is certainly a proper incident response plan, other things, such as the procurement of necessary hardware and software and the training of personnel, are also important. While stopping or mitigating the impact of any incident or attack is of prime importance, it must be done in accordance with the explicit dictates of an incident response plan to be truly effective. Secondary goals such as evaluating the extent of damages done, restoring systems to pre-incident status, determining the causes of the incidents and taking steps to minimize the likelihood that incidents of those types will occur in the future are also important, albeit not of as immediate a nature. Preparation is the key.
For years they have been the target for attacks: • Null sessions. • User enumeration. • Access using default account names, simple passwords or null passwords. • SPAM using Windows Messaging pop-up's. • And many more. In other words these services were never intended to be used on the Internet or other “insecure” networks. These services are intended to be run on internal networks only, any firewall and router in a company should filter this traffic inbound and outbound. It is the first lesson taught at any basic IT security course.
Why did big companies get hit? One problem is portable systems which aren't properly configured with antivirus,
19