- Email: [email protected]
Testing Your Security Defence by Breaking In?
managing network security It is important to point out that dealing with a virus which has been allowed to enter a company will be several orders of magnitude more expensive than the cost of any anti-virus software. The main expense will be time, since it will probably be necessary to physically visit every infected workstation to perform the disinfection and its restoration to the pre-infection state. Having company-standard software installations, possibly supplemented
by disk imaging software, can be very helpful to restore infected workstations.
About the author Dr. Jan Hruska is the technical director of Sophos Plc. A graduate of Downing College, Cambridge, he gained his doctorate at Magdalen College, Oxford. He regularly speaks at computer security conferences. He is a co-author (with Dr. Keith Jackson) of
The PC Security Guide, published by Elsevier (1988, 1989), Computer Security Solutions, published by Blackwells (1990) and Computer Security Reference Book published by Butterworth-Heinemann (1992). He is the author of Computer Viruses and Anti-Virus Warfare published by Ellis-Horwood (1990, 1992). His extracurricular interests are flying, skiing, subaqua diving and piano-playing. E-mail him at [email protected] or visit www.sophos.com.
MANAGING NETWORK SECURITY
Testing Your Security Defence by Breaking In? Maybe Not. Fred Cohen Networks dominate today’s computing landscape and commercial technical protection is lagging behind attack technology. As a result, protection programme success depends more on prudent management decisions than on the selection of technical safeguards. Managing Network Security takes a management view of protection and seeks to reconcile the need for security with the limitations of technology. The theory is that I should be able to test the effectiveness of my protection by learning how to break into my systems and seeing if I can do it. It’s a great theory, and it seems sensible enough on the surface. After all, if I don’t do some sort of testing of my defences, how do I know they work at all? Now I agree completely with the last statement. If you don’t do some sort of testing, you will not be able to tell whether your systems do what they are supposed to do. The question is whether trying to break into your systems is the best way to do that testing. And therein lies the rub. In this month’s article, I will be testing the notion that you can effectively test
16
your defences by trying to break into them. I am in a more formal mood than usual, owing at least in part to the fact that I have been working for the last 48 hours or so (over a weekend of course) reconfiguring a network to get some of the defences really right — and testing these configurations...
Let’s try to break in — just to be sure Trying to break in never makes you sure that someone else won’t be able to break in. The reasoning seems to go: “I am as good as they are, so if I can’t do it, they can’t do it.” This could not be further from the truth — for anyone.
It represents a great leap of ego to believe that there is any individual or small group of individuals who are capable of figuring out and trying every attack that some other individual or group of individuals will come up with. And there is at least one very good reason to believe that such an approach will never succeed. The reason is that the number of attacks that can be tried are far larger than the number that you will be able to try. So the best you can do is to cover some subset of the possible attacks with your practice attacks.
“the best you can do is to cover some subset of the possible attacks with your practice attacks” How big a portion of the attacks can you cover? It turns out that there are a potentially infinite number of attacks. So if you try 1 or 100 or 100 000 or 10^100 attacks, you still haven’t made a dent in all of the possible attacks that can be used against your system. And that is at the heart of the problem. You can try till you turn blue in the face, and all you will have to show for it is that you didn’t succeed yet. You won’t show that an attacker won’t succeed on the very next try. For those of you who
managing network security have studied philosophy of science, you may recognize Karl Popper’s work on confirmations and refutations in this. Essentially, Popper proved that you cannot prove a negative about an infinite set by pure experiment because (to misquote) the proposition that all pigeons are white cannot be proven correct until you look at all pigeons, but even a single black pigeon will prove the proposition wrong.
If we find anything, we can fix it True enough, but if you find attacks, this also means that you need to fix the process you used to devise defences. Because, it is not just a case of missing something, it is a syptom of a failed protection process. But there is a notable exception to this. The exception is an attack that you believed would work and decided not to defend against. Now I have vulnerabilities in my systems that I choose not to defend against for one reason or another, and so do you, and so does everyone, everywhere, who runs a system of any sort. It is the nature of things.
more or less always inviting people to attack you for money if you have any significant business at stake. And they succeed, at times, against most operations. But the problem is that most ‘skilled attackers’ are not ‘skilled defenders’ and they don’t understand the tradeoffs involved in your operation. If they succeed, all they have done is focus you in on one attack mechanism. You will be forced to spend money on it at the expense of something else, and it may be that nobody else would have come across this vulnerability and it was thus a double waste of money. One waste to hire the attacker and another waste to defend against the attack that they used when others might not use it. It would seem far better to hire someone who is skilled at defending systems against attack than to hire someone who knows how to attack systems. Now I admit that defending systems well involves understanding how to attack them, but that’s not to say that knowing how to break into systems involves knowing how to defend them from breakins. To explain this, I look to the physical world.
“I already know that the The physical analogy system is vulnerable, and If you wanted to protect your house from the test will only show that breaking and entering, would you hire a burglar to tell you how? This is an interI was right” esting question, and one that has been But, if I knew of the attack and decided not to defend against it, what good is it to do a test? I already know that the system is vulnerable, and the test will only show that I was right or show that I am not as good at attacking systems as I need to be to exploit the vulnerabilities that I know exist. So it is a waste of time to even try.
Let’s hire someone who is skilled at it Fair enough. The theory is that a skilled attacker will help identify more attacks that might work against us. It is a good theory, and it works in practice. When you invite people to attack you for money, they do it. Of course you are
answered with both yes and no by different folks at different times. The better question, in my view, is whether you should hire a professional security firm or a professional burglary firm to help you understand your defence options. The burglar will tell you that it’s easy to throw a brick through the window, to pick your door lock, to enter through a second storey window, and that they like operating in the dark. They may also ‘case the joint’ along the way, but that’s another issue — the issue of trust. They may tell you that the thing they most fear is a big dog — but maybe the next burglar has an easy way around big dogs.
If you pay them, maybe they will throw a brick through your window or pick a lock or go in through the second storey — does that help you? Now that you know these things, do you know how to protect your home? I don’t think so. Does this mean you should get a big dog, bullet proof windows, a lock that is very hard to pick, and always keep your upstairs windows locked? You could do that, but let’s look at some other options.
“The professional security company will tell you all of the things the burglar told you” The professional security company will tell you all of the things the burglar told you. They probably will not throw a brick through your window. Even if they have a locksmith, they will be offended at the notion of having to pick a lock as proof that a bad person can do it. They may show you how to get to the second floor, by climbing up the trellis on your porch — assuming it doesn’t collapse on top of them. But then they tell you about other options. For example, they will tell you about things like response times with different services and different sorts of alarm systems. Most burglars don’t know much about these things because they avoid houses with these protections or enter them when the alarms are off. But even for the really professional burglar who specializes in high-end robberies, is it really a help for you to have them plan out a way to break in and do so? Is this more of a risk than it’s worth?
Oh yeah... That network I was working on... As I mentioned earlier, I have spent the last 48 hours reconfiguring a network to improve on some of the defences. The network was actually doing just fine and was operating relatively securely for a period of several years the way it was. I didn’t make any security patches at all.
17
managing network security There weren’t any insecure services to patch. All I did was some reconfiguration, the addition of another layer of protection, performance enhancements, and some tightening and tuning. I am not yet fully satisfied with it, but I’m getting ever closer. It had been tested by numerous realworld attackers on an ongoing basis, and none had gotten past the first layer of defences. So why did I go and alter the protections? The answer is simple. I do a lot of tests on my network. None of them are oriented toward breaking in, per se, they are oriented toward determining whether or not it does exactly what I want it to do. For example, I scan to see if machines I don’t want detected are detectable from remote sites. I check to see whether unauthorized services are available. I check to see that the authorized services operate as they are expected to. I check to verify that audit information is as it should be, and I review the audits of various efforts on a regular basis. I also check the systems from sites that have less restricted access than any real users, because I want to be certain that if a protection mechanism breaks down there will be a redundant mechanism that takes over and reports the breakdown long before the overall protection scheme is bypassed. All of these are tests, but none of them are tests that involve breaking in. These tests go well beyond what ‘hackers’ do or could do against this network, because I don’t want to be caught short by only doing what I think someone else might do. I want to cover the full range of possibilities, not just the attack of the day, and not just the things that the limited skills of one person with minimal training and expertise can come up with. The things that I spent a lot of time on this weekend were things that almost certainly won’t prevent an attacker from breaking into my sites, but then that’s why the many attackers that try to break into my sites fail. And when I look at the tradeoffs, it costs a lot less for me to spend the weekend making sure than it does to spend a month making up for it if someone gets in.
18
Conclusions I believe strongly in testing protection, but this is not the same as ‘breaking in’. It is a much more thorough process. We identify the different methods of break-in by large classes, and try to test the defences that we have designed to assure that they operate as intended. This is very different than testing by attacking in that it can be designed to provide very good coverage of the defences we have used and recognizes the limits of those defences without introducing unnecessary risk or cost. Having said this, you might go to my website and look at some of the Strategic Services we provide under Security Assessments. They include: • Dumpster Diving • Web-based Intelligence • Deception Susceptibility • Telecommunications Sweep • Project Data Aggregation • A Bugging Exercise. All of these might be called examples of Testing Your Security by Breaking In. So where do I get off telling you that this is a foolish thing to do when I sell it myself? The difference between testing and breaking in is not always so clear. In the case of good tests, they don’t really break in at all. Customers ask for these services for two reasons: (1) to test defences that are in place to verify their effectiveness, and, (2) to demonstrate the need for defences that are not in place in areas where vulnerabilities are known or suspected. It may seem like a distinction without a difference, but the difference is very real. Like the alarm company that identifies weaknesses and solutions, we identify weaknesses and solutions in your information protection programme. Unlike the burglar hired to show they can break in, testers are not burglars and do not break in. The difference really comes in two ways: (1) To do the job right requires close cooperation with the client. This is necessary in order to provide meaningful tests and demonstrations that tell them more than they knew
before, without introducing new risk. The effort is generally combined with observation by the defending personnel so they can get a ‘simulated experience’ of what an attack is like, and how to handle it. These tests are generally designed to provide a high degree of coverage of possible things that can happen. The tester isn’t looking for one vulnerability that can be exploited, they are looking to find what you have missed and they want to find as many of the vulnerability classes as they can. (2) The level of control over activities must be very tight when the job is done well. For example, every step that is taken, is cleared with the client, with anything questionable addressed, before it is done. If there are possible negative consequences, these are identified and discussed ahead of time, and there are briefings for all parties concerned so that there are no misunderstandings or bad surprises. This is a test — only a test! I believe that these controls are necessary and appropriate to doing a reasonable test of protection. If the test doesn’t cover a wide class of issues, it is not a very useful test. If the proper controls are not in place to make it safe and effective, it is not a very safe test. The goal is safe and effective testing, and that is a far cry from ‘testing a system by breaking into it’.
About the author Fred Cohen is exploring the minimum raise as a principal member of technical staff at Sandia National Laboratories, helping clients meet their information protection needs as the managing director of Fred Cohen and Associates in Livermore California, and educating defenders overthe-Internet on all aspects of information protection as a practitioner in residence in the University of New Haven’s Forensic Sciences Programme. He can be reached by sending E-mail to [email protected] or visiting http://all.net/
by disk imaging software, can be very helpful to restore infected workstations.
About the author Dr. Jan Hruska is the technical director of Sophos Plc. A graduate of Downing College, Cambridge, he gained his doctorate at Magdalen College, Oxford. He regularly speaks at computer security conferences. He is a co-author (with Dr. Keith Jackson) of
The PC Security Guide, published by Elsevier (1988, 1989), Computer Security Solutions, published by Blackwells (1990) and Computer Security Reference Book published by Butterworth-Heinemann (1992). He is the author of Computer Viruses and Anti-Virus Warfare published by Ellis-Horwood (1990, 1992). His extracurricular interests are flying, skiing, subaqua diving and piano-playing. E-mail him at [email protected] or visit www.sophos.com.
MANAGING NETWORK SECURITY
Testing Your Security Defence by Breaking In? Maybe Not. Fred Cohen Networks dominate today’s computing landscape and commercial technical protection is lagging behind attack technology. As a result, protection programme success depends more on prudent management decisions than on the selection of technical safeguards. Managing Network Security takes a management view of protection and seeks to reconcile the need for security with the limitations of technology. The theory is that I should be able to test the effectiveness of my protection by learning how to break into my systems and seeing if I can do it. It’s a great theory, and it seems sensible enough on the surface. After all, if I don’t do some sort of testing of my defences, how do I know they work at all? Now I agree completely with the last statement. If you don’t do some sort of testing, you will not be able to tell whether your systems do what they are supposed to do. The question is whether trying to break into your systems is the best way to do that testing. And therein lies the rub. In this month’s article, I will be testing the notion that you can effectively test
16
your defences by trying to break into them. I am in a more formal mood than usual, owing at least in part to the fact that I have been working for the last 48 hours or so (over a weekend of course) reconfiguring a network to get some of the defences really right — and testing these configurations...
Let’s try to break in — just to be sure Trying to break in never makes you sure that someone else won’t be able to break in. The reasoning seems to go: “I am as good as they are, so if I can’t do it, they can’t do it.” This could not be further from the truth — for anyone.
It represents a great leap of ego to believe that there is any individual or small group of individuals who are capable of figuring out and trying every attack that some other individual or group of individuals will come up with. And there is at least one very good reason to believe that such an approach will never succeed. The reason is that the number of attacks that can be tried are far larger than the number that you will be able to try. So the best you can do is to cover some subset of the possible attacks with your practice attacks.
“the best you can do is to cover some subset of the possible attacks with your practice attacks” How big a portion of the attacks can you cover? It turns out that there are a potentially infinite number of attacks. So if you try 1 or 100 or 100 000 or 10^100 attacks, you still haven’t made a dent in all of the possible attacks that can be used against your system. And that is at the heart of the problem. You can try till you turn blue in the face, and all you will have to show for it is that you didn’t succeed yet. You won’t show that an attacker won’t succeed on the very next try. For those of you who
managing network security have studied philosophy of science, you may recognize Karl Popper’s work on confirmations and refutations in this. Essentially, Popper proved that you cannot prove a negative about an infinite set by pure experiment because (to misquote) the proposition that all pigeons are white cannot be proven correct until you look at all pigeons, but even a single black pigeon will prove the proposition wrong.
If we find anything, we can fix it True enough, but if you find attacks, this also means that you need to fix the process you used to devise defences. Because, it is not just a case of missing something, it is a syptom of a failed protection process. But there is a notable exception to this. The exception is an attack that you believed would work and decided not to defend against. Now I have vulnerabilities in my systems that I choose not to defend against for one reason or another, and so do you, and so does everyone, everywhere, who runs a system of any sort. It is the nature of things.
more or less always inviting people to attack you for money if you have any significant business at stake. And they succeed, at times, against most operations. But the problem is that most ‘skilled attackers’ are not ‘skilled defenders’ and they don’t understand the tradeoffs involved in your operation. If they succeed, all they have done is focus you in on one attack mechanism. You will be forced to spend money on it at the expense of something else, and it may be that nobody else would have come across this vulnerability and it was thus a double waste of money. One waste to hire the attacker and another waste to defend against the attack that they used when others might not use it. It would seem far better to hire someone who is skilled at defending systems against attack than to hire someone who knows how to attack systems. Now I admit that defending systems well involves understanding how to attack them, but that’s not to say that knowing how to break into systems involves knowing how to defend them from breakins. To explain this, I look to the physical world.
“I already know that the The physical analogy system is vulnerable, and If you wanted to protect your house from the test will only show that breaking and entering, would you hire a burglar to tell you how? This is an interI was right” esting question, and one that has been But, if I knew of the attack and decided not to defend against it, what good is it to do a test? I already know that the system is vulnerable, and the test will only show that I was right or show that I am not as good at attacking systems as I need to be to exploit the vulnerabilities that I know exist. So it is a waste of time to even try.
Let’s hire someone who is skilled at it Fair enough. The theory is that a skilled attacker will help identify more attacks that might work against us. It is a good theory, and it works in practice. When you invite people to attack you for money, they do it. Of course you are
answered with both yes and no by different folks at different times. The better question, in my view, is whether you should hire a professional security firm or a professional burglary firm to help you understand your defence options. The burglar will tell you that it’s easy to throw a brick through the window, to pick your door lock, to enter through a second storey window, and that they like operating in the dark. They may also ‘case the joint’ along the way, but that’s another issue — the issue of trust. They may tell you that the thing they most fear is a big dog — but maybe the next burglar has an easy way around big dogs.
If you pay them, maybe they will throw a brick through your window or pick a lock or go in through the second storey — does that help you? Now that you know these things, do you know how to protect your home? I don’t think so. Does this mean you should get a big dog, bullet proof windows, a lock that is very hard to pick, and always keep your upstairs windows locked? You could do that, but let’s look at some other options.
“The professional security company will tell you all of the things the burglar told you” The professional security company will tell you all of the things the burglar told you. They probably will not throw a brick through your window. Even if they have a locksmith, they will be offended at the notion of having to pick a lock as proof that a bad person can do it. They may show you how to get to the second floor, by climbing up the trellis on your porch — assuming it doesn’t collapse on top of them. But then they tell you about other options. For example, they will tell you about things like response times with different services and different sorts of alarm systems. Most burglars don’t know much about these things because they avoid houses with these protections or enter them when the alarms are off. But even for the really professional burglar who specializes in high-end robberies, is it really a help for you to have them plan out a way to break in and do so? Is this more of a risk than it’s worth?
Oh yeah... That network I was working on... As I mentioned earlier, I have spent the last 48 hours reconfiguring a network to improve on some of the defences. The network was actually doing just fine and was operating relatively securely for a period of several years the way it was. I didn’t make any security patches at all.
17
managing network security There weren’t any insecure services to patch. All I did was some reconfiguration, the addition of another layer of protection, performance enhancements, and some tightening and tuning. I am not yet fully satisfied with it, but I’m getting ever closer. It had been tested by numerous realworld attackers on an ongoing basis, and none had gotten past the first layer of defences. So why did I go and alter the protections? The answer is simple. I do a lot of tests on my network. None of them are oriented toward breaking in, per se, they are oriented toward determining whether or not it does exactly what I want it to do. For example, I scan to see if machines I don’t want detected are detectable from remote sites. I check to see whether unauthorized services are available. I check to see that the authorized services operate as they are expected to. I check to verify that audit information is as it should be, and I review the audits of various efforts on a regular basis. I also check the systems from sites that have less restricted access than any real users, because I want to be certain that if a protection mechanism breaks down there will be a redundant mechanism that takes over and reports the breakdown long before the overall protection scheme is bypassed. All of these are tests, but none of them are tests that involve breaking in. These tests go well beyond what ‘hackers’ do or could do against this network, because I don’t want to be caught short by only doing what I think someone else might do. I want to cover the full range of possibilities, not just the attack of the day, and not just the things that the limited skills of one person with minimal training and expertise can come up with. The things that I spent a lot of time on this weekend were things that almost certainly won’t prevent an attacker from breaking into my sites, but then that’s why the many attackers that try to break into my sites fail. And when I look at the tradeoffs, it costs a lot less for me to spend the weekend making sure than it does to spend a month making up for it if someone gets in.
18
Conclusions I believe strongly in testing protection, but this is not the same as ‘breaking in’. It is a much more thorough process. We identify the different methods of break-in by large classes, and try to test the defences that we have designed to assure that they operate as intended. This is very different than testing by attacking in that it can be designed to provide very good coverage of the defences we have used and recognizes the limits of those defences without introducing unnecessary risk or cost. Having said this, you might go to my website and look at some of the Strategic Services we provide under Security Assessments. They include: • Dumpster Diving • Web-based Intelligence • Deception Susceptibility • Telecommunications Sweep • Project Data Aggregation • A Bugging Exercise. All of these might be called examples of Testing Your Security by Breaking In. So where do I get off telling you that this is a foolish thing to do when I sell it myself? The difference between testing and breaking in is not always so clear. In the case of good tests, they don’t really break in at all. Customers ask for these services for two reasons: (1) to test defences that are in place to verify their effectiveness, and, (2) to demonstrate the need for defences that are not in place in areas where vulnerabilities are known or suspected. It may seem like a distinction without a difference, but the difference is very real. Like the alarm company that identifies weaknesses and solutions, we identify weaknesses and solutions in your information protection programme. Unlike the burglar hired to show they can break in, testers are not burglars and do not break in. The difference really comes in two ways: (1) To do the job right requires close cooperation with the client. This is necessary in order to provide meaningful tests and demonstrations that tell them more than they knew
before, without introducing new risk. The effort is generally combined with observation by the defending personnel so they can get a ‘simulated experience’ of what an attack is like, and how to handle it. These tests are generally designed to provide a high degree of coverage of possible things that can happen. The tester isn’t looking for one vulnerability that can be exploited, they are looking to find what you have missed and they want to find as many of the vulnerability classes as they can. (2) The level of control over activities must be very tight when the job is done well. For example, every step that is taken, is cleared with the client, with anything questionable addressed, before it is done. If there are possible negative consequences, these are identified and discussed ahead of time, and there are briefings for all parties concerned so that there are no misunderstandings or bad surprises. This is a test — only a test! I believe that these controls are necessary and appropriate to doing a reasonable test of protection. If the test doesn’t cover a wide class of issues, it is not a very useful test. If the proper controls are not in place to make it safe and effective, it is not a very safe test. The goal is safe and effective testing, and that is a far cry from ‘testing a system by breaking into it’.
About the author Fred Cohen is exploring the minimum raise as a principal member of technical staff at Sandia National Laboratories, helping clients meet their information protection needs as the managing director of Fred Cohen and Associates in Livermore California, and educating defenders overthe-Internet on all aspects of information protection as a practitioner in residence in the University of New Haven’s Forensic Sciences Programme. He can be reached by sending E-mail to [email protected] or visiting http://all.net/