- Email: [email protected]
Linux intrusion discovery: when security fails
linux hacking
Linux intrusion discovery: when security fails
The above shows the use of the 'uptime' command to check the system usage. The system load factor shows how many processes compete for the CPU, thus indicating system speed. In the shown case, the system is heavily overloaded.
Anton Chuvakin, Ph.D., GCIA, GCIH
Excessive memory use is another common indicator:
When security fails, common operating systems react in different ways and the consequences may be more or less severe. In this article, we will cover what happens on a Linux system when it is successfully hacked. This paper aims at giving you or colleagues tools and methods to tell that a Linux system is likely compromised. They should not require any advanced security knowledge while still be effective against many attackers, threatening the organizations. It focuses on performing simple actions and looking at their results. Of course, they are likely not effective against advanced attackers. That is not a major problem since addressing the more common attackers will go a long way towards creating a more secure environment. We will first cover some of the common attacks and intruder behavior. Then we will go to the first signs and suspicions of the intrusion. The main part of the paper then deals with confirming the intrusion. Before we embark on a detailed discussion of the subject we will depict the context of the attack. What are the commonly used (or, rather, abused) network services? Further, what are the common attacks and intruder behavior: what they will hit you with? For a Linux/UNIX system the SANS/FBI Top 20 list for UNIX is a great start. Here are some of the “popular” vulnerable services of interest to attackers: RPC, FTP, HTTP/HTTPS, SMTP and many others. As in the old times, bruteforcing passwords (such as over HTTP, FTP, Telnet, POP3 and SSH) is still a threat. Web application and CGI attacks are common and on the rise. After the initial compromise or for internal attackers, sniffing and local console abuse are popular as well. 10
What do attackers do after they use the above holes? They close the holes, deploy backdoors or Trojans, run IRC (Internet Relay Chat) servers to communicate with their buddies. Subsequent activities often include massive scale scanning and exploitation as well as DoS attacks. The compromised machines are also used for storing “warez” (pirated software) and pirated content or searching for credit cards. First, we are going to summarize hack omens summary, a list of possible intrusion indicators to look for. They include: • Resource waste: such as disk or memory over-utilization. • System failures: system as crashing programs and other failures. • Unusual objects and traces: new files, accounts, processes not started by legitimate users. • Unusual networking: new connection initiations and reception. Many of them fit within the “something just doesn’t feel right!” category that experienced system admins feel when sometimes confronted with an intrusion. How would a system owner be alerted about resource waste? One obvious sign is slow system performance: [anton@bmw anton]$ uptime 11:53pm up 41 days, 8:54, 1 user, load average: 12.14, 9.12, 7.09
[anton@bmw anton]$ free total used free shared buffers cached Mem: 127820 118856 8964 38636 13860 21684 Swap: 336504 336404 100 The above shows that almost all of memory space and swap space are used up. Often, the attackers will use the compromised systems to store their content and data, thus causing 'missing' disk space. [anton@bmw anton]$ df Filesystem 1k-blocks Available Use% Mounted on /dev/hda1 2016016 2016000 1193 99% /
Used
Here we see that 99% of the /dev/hda1 disk on a Linux machine is used up. Slow network connectivity is another important indicator of a possible network resource abuse. A “ping” command can be used to check for that. In addition to the above, miscellaneous failures are also often indicative of covert hacking attempts, that have gone wrong. Also, some attack types such as buffer overflow, have a tendency to generate system failures, no matter whether failed or successful. Here are some of the common failures observed on UNIX/Linux systems: Spontaneous reboots are often an indication that an attacker had to reboot the system to reconfigure something or that a major system malfunction had occurred (possibly as a result of an attacker action). The “Uptime” command can be used to check for reboots:
linux hacking [anton@bmw anton]$ uptime 10:05pm up 3 hours, 1:54, 2 user, load average: 0.14, 0.12, 0.09
tions), new log entries are all considered to be possible intrusion omens, and need to be investigated.
The above indicates that the system was up three hours since the last reboot.
Look out for resource waste and system failures
Application crashes and errors are sometimes generated when hacker exploits fail to gain access but still bring the application down. For example: VM: killing process spamassassin Similarly, application restart messages are often indicative of attackers tampering with the application. However, they are also often generated in the normal course of system operation. Mar 14 05:22:32 ns1 syslogd 1.3-3: restart. The above messages indicates that a system logging daemon was restarted (either by a legitimate user, a scheduled process or an attacker). Authentication failures in small numbers likely indicate that legitimate users have forgotten their passwords. Mar 14 19:02:04 ns1 PAM_unix[29426]: authentication failure; evil(uid=500) -> root for system-auth service Now let’s look at various suspicious new objects appearing on systems. Also, if new files appear on a system, it is often an indication of a successful hacker attack. Here are some of the examples: Files/directories [root@ns1 /tmp]# ls -la total 35 drwxrwxrwt 5 root root 15360 Mar 16 00:22 . drwx------ 2 root root Mar 16 00:22 ...
1024
The above shows a directory called “…” (three dots) that is sometimes used by amateur attackers to hide their tools. New processes, accounts, connections (from server, to client, too many connec-
Now that we have looked at common suspicious activities, we are ready to formulate a plan for the intrusion discovery process. “The Plan” should answer the following questions 'What do the above signs indicate?' In many cases 'Nothing' is the right answer. To get to the answer using default system tools or open source programs is the focus of out effort. Of course, some built-in intelligence is needed (that refers to the user’s own brain!) Here are the areas we will look at in our investigation • • • • • • •
Look for suspicious files. Look for suspicious accounts. Look for system corruption. Look for suspicious networking. Look for suspicious processes. Look for weird log entries. Look for miscellaneous other “weirdness.”
Now we will look at the above in detail. To look for suspicious files we will focus on the following four types of files: 1. Large recently modified files may indicate that a sniffer program that records all network traffic of interest to attackers (such as usernames and passwords) is running on your network: # find / -size +10000k –mtime +7 –print 2. Nobody’s files may indicate that an attacker has created, used and then removed a new user account:
# find / -nouser –print 3. SUID root files in an unusual location (outside of /sbin and /bin, for example) sometimes indicate planted backdoors and hacker tools aimed at regaining privileged access: # find / -uid 0 –perm -4000 –print 4. Weird file names (“. “,” “,”…”, etc) are a reliable indication of amateur hacker activity: # find / -name “...“ –print Another important area is to look for suspicious accounts, such as new accounts with admin privileges or reenabled system accounts. For example, one can run a command 'grep :0: /etc/passwd' which will show all privileged accounts on a system Attempts to install malicious software and hacker tools often lead to various system software failures. Library, system file and even kernel corruption is not uncommon. That often happens since attacker tools might be optimized for different operating system version or other software. Here are some common signs of system corruption. On the RedHat family of Linux one can use the built-in RPM (RedHat Package Manager) to look for corrupted packaged. For example, this command verified installed software integrity: # rpm –qa | sort # rpm –Va | sort In case the user (that means, you!) had thought about installing a system integrity check such as Tripwire or AIDE, now (in the possible intrusion aftermath) is a good time to run it. For AIDE the command is aide –check
while Tripwire requires
11
tcg hardware tripwire –check.
# netstat –nap
Another critical program of a similar kind that does not require any preliminary effort is “Chkrootkit”. Get it at www.chkrootkit.org, install and run as follows:
Looking for suspicious processes is easy. While advanced attackers will hide from such means, some traces will likely be left anyway. Process list can be shown using a standard “ps” command:
# chkrootkit
# ps –aux
It can detect a large set of malicious changes to a system as well as common attacker software, such as rootkits.
Look for daemons with a “.” (dot) in the path (likely started by attackers), processes with strange names, etc. To look up the process details one can look directly into the /proc directory, housing the direct link to system resources. For example, the command “cat /proc/13555” will show details on the process with ID equal to 13555. On RedHat-like Linux system one can query the configured services via “chkconfig –list”.
Looking for suspicious networking can be accomplished easily with system built-in tools as well. For example, promiscuous mode detection (sniffer detection) can be done via one of several means: # ip link | grep PROMISC or # /sbin/ifconfig or # dmesg | grep promisc To detect listening services one can use either the “lsof ” or “netstat” command. There two commands will help:
# lsof –i
To conclude, we will look at what have we learned. We can quickly look for known signs of intrusions. We have a plan for doing that, that doesn’t require any expensive “security tools” or advanced skills. Many regular computer users can be trained to do that and so can the system administrators. Similar methods are available for Windows!
Trusted computing — path to security or road to servitude? Sarah Hilley PC hardware as we know it is about to change forever — and all in the name of security. But watchful bystanders fear the new hardware architecture could have a more sinister payload. Sarah Hilley reports on the trusted computing movement. One day a hacker will sink his teeth into the soft underbelly of a commercial company, only to have all his teeth broken from biting into an iron vault. This is the promise of the trusted computing move12
ment. "We can only go so far in protecting against threats with software," says Tony Redmond, VP and CTO, HP Services. This is the rational core of the activities of the Trusted Computing Group (TCG).
Unusual objects and traces and unusual networking are also signs of a Linux hack
About the author Anton Chuvakin, Ph.D., GCIA, GCIH http://www.chuvakin.org is a Senior Security Analyst with netForensics, a security information management company, where he is involved with designing the product and researching potential new security features. His areas of infosec expertise include intrusion detection, UNIX security, forensics, honeypots, etc. He is the author of a book "Security Warrior" (O'Reilly, January 2004) and a contributor to "Know Your Enemy II" by the Honeynet Project (AWL, June 2004) and "Information Security Management Handbook" (CRC, April 2004). In his spare time he maintains his security portal http://www.info-secure.org
Steered by a body that has arisen from the ranks of the IT giants, the TCG is releasing new hardware that will come embedded with a futuristic intelligent smartcard that is aware of what programs are running. The TCG will begin by sowing its seeds in large corporations, but it won't stop there. The group also plans to plant routers, mobile phones, PDAs, servers and ultimately even home PCs with the new embedded chip. Many spectators believe the ultimate goal of the new hardware is digital rights management (DRM). Others believe that it will sound the death knell for open source software in the enterprise. The TCG group have also had to pacify governments with privacy anxieties — HP's Redmond confirms that the TCG has
Linux intrusion discovery: when security fails
The above shows the use of the 'uptime' command to check the system usage. The system load factor shows how many processes compete for the CPU, thus indicating system speed. In the shown case, the system is heavily overloaded.
Anton Chuvakin, Ph.D., GCIA, GCIH
Excessive memory use is another common indicator:
When security fails, common operating systems react in different ways and the consequences may be more or less severe. In this article, we will cover what happens on a Linux system when it is successfully hacked. This paper aims at giving you or colleagues tools and methods to tell that a Linux system is likely compromised. They should not require any advanced security knowledge while still be effective against many attackers, threatening the organizations. It focuses on performing simple actions and looking at their results. Of course, they are likely not effective against advanced attackers. That is not a major problem since addressing the more common attackers will go a long way towards creating a more secure environment. We will first cover some of the common attacks and intruder behavior. Then we will go to the first signs and suspicions of the intrusion. The main part of the paper then deals with confirming the intrusion. Before we embark on a detailed discussion of the subject we will depict the context of the attack. What are the commonly used (or, rather, abused) network services? Further, what are the common attacks and intruder behavior: what they will hit you with? For a Linux/UNIX system the SANS/FBI Top 20 list for UNIX is a great start. Here are some of the “popular” vulnerable services of interest to attackers: RPC, FTP, HTTP/HTTPS, SMTP and many others. As in the old times, bruteforcing passwords (such as over HTTP, FTP, Telnet, POP3 and SSH) is still a threat. Web application and CGI attacks are common and on the rise. After the initial compromise or for internal attackers, sniffing and local console abuse are popular as well. 10
What do attackers do after they use the above holes? They close the holes, deploy backdoors or Trojans, run IRC (Internet Relay Chat) servers to communicate with their buddies. Subsequent activities often include massive scale scanning and exploitation as well as DoS attacks. The compromised machines are also used for storing “warez” (pirated software) and pirated content or searching for credit cards. First, we are going to summarize hack omens summary, a list of possible intrusion indicators to look for. They include: • Resource waste: such as disk or memory over-utilization. • System failures: system as crashing programs and other failures. • Unusual objects and traces: new files, accounts, processes not started by legitimate users. • Unusual networking: new connection initiations and reception. Many of them fit within the “something just doesn’t feel right!” category that experienced system admins feel when sometimes confronted with an intrusion. How would a system owner be alerted about resource waste? One obvious sign is slow system performance: [anton@bmw anton]$ uptime 11:53pm up 41 days, 8:54, 1 user, load average: 12.14, 9.12, 7.09
[anton@bmw anton]$ free total used free shared buffers cached Mem: 127820 118856 8964 38636 13860 21684 Swap: 336504 336404 100 The above shows that almost all of memory space and swap space are used up. Often, the attackers will use the compromised systems to store their content and data, thus causing 'missing' disk space. [anton@bmw anton]$ df Filesystem 1k-blocks Available Use% Mounted on /dev/hda1 2016016 2016000 1193 99% /
Used
Here we see that 99% of the /dev/hda1 disk on a Linux machine is used up. Slow network connectivity is another important indicator of a possible network resource abuse. A “ping” command can be used to check for that. In addition to the above, miscellaneous failures are also often indicative of covert hacking attempts, that have gone wrong. Also, some attack types such as buffer overflow, have a tendency to generate system failures, no matter whether failed or successful. Here are some of the common failures observed on UNIX/Linux systems: Spontaneous reboots are often an indication that an attacker had to reboot the system to reconfigure something or that a major system malfunction had occurred (possibly as a result of an attacker action). The “Uptime” command can be used to check for reboots:
linux hacking [anton@bmw anton]$ uptime 10:05pm up 3 hours, 1:54, 2 user, load average: 0.14, 0.12, 0.09
tions), new log entries are all considered to be possible intrusion omens, and need to be investigated.
The above indicates that the system was up three hours since the last reboot.
Look out for resource waste and system failures
Application crashes and errors are sometimes generated when hacker exploits fail to gain access but still bring the application down. For example: VM: killing process spamassassin Similarly, application restart messages are often indicative of attackers tampering with the application. However, they are also often generated in the normal course of system operation. Mar 14 05:22:32 ns1 syslogd 1.3-3: restart. The above messages indicates that a system logging daemon was restarted (either by a legitimate user, a scheduled process or an attacker). Authentication failures in small numbers likely indicate that legitimate users have forgotten their passwords. Mar 14 19:02:04 ns1 PAM_unix[29426]: authentication failure; evil(uid=500) -> root for system-auth service Now let’s look at various suspicious new objects appearing on systems. Also, if new files appear on a system, it is often an indication of a successful hacker attack. Here are some of the examples: Files/directories [root@ns1 /tmp]# ls -la total 35 drwxrwxrwt 5 root root 15360 Mar 16 00:22 . drwx------ 2 root root Mar 16 00:22 ...
1024
The above shows a directory called “…” (three dots) that is sometimes used by amateur attackers to hide their tools. New processes, accounts, connections (from server, to client, too many connec-
Now that we have looked at common suspicious activities, we are ready to formulate a plan for the intrusion discovery process. “The Plan” should answer the following questions 'What do the above signs indicate?' In many cases 'Nothing' is the right answer. To get to the answer using default system tools or open source programs is the focus of out effort. Of course, some built-in intelligence is needed (that refers to the user’s own brain!) Here are the areas we will look at in our investigation • • • • • • •
Look for suspicious files. Look for suspicious accounts. Look for system corruption. Look for suspicious networking. Look for suspicious processes. Look for weird log entries. Look for miscellaneous other “weirdness.”
Now we will look at the above in detail. To look for suspicious files we will focus on the following four types of files: 1. Large recently modified files may indicate that a sniffer program that records all network traffic of interest to attackers (such as usernames and passwords) is running on your network: # find / -size +10000k –mtime +7 –print 2. Nobody’s files may indicate that an attacker has created, used and then removed a new user account:
# find / -nouser –print 3. SUID root files in an unusual location (outside of /sbin and /bin, for example) sometimes indicate planted backdoors and hacker tools aimed at regaining privileged access: # find / -uid 0 –perm -4000 –print 4. Weird file names (“. “,” “,”…”, etc) are a reliable indication of amateur hacker activity: # find / -name “...“ –print Another important area is to look for suspicious accounts, such as new accounts with admin privileges or reenabled system accounts. For example, one can run a command 'grep :0: /etc/passwd' which will show all privileged accounts on a system Attempts to install malicious software and hacker tools often lead to various system software failures. Library, system file and even kernel corruption is not uncommon. That often happens since attacker tools might be optimized for different operating system version or other software. Here are some common signs of system corruption. On the RedHat family of Linux one can use the built-in RPM (RedHat Package Manager) to look for corrupted packaged. For example, this command verified installed software integrity: # rpm –qa | sort # rpm –Va | sort In case the user (that means, you!) had thought about installing a system integrity check such as Tripwire or AIDE, now (in the possible intrusion aftermath) is a good time to run it. For AIDE the command is aide –check
while Tripwire requires
11
tcg hardware tripwire –check.
# netstat –nap
Another critical program of a similar kind that does not require any preliminary effort is “Chkrootkit”. Get it at www.chkrootkit.org, install and run as follows:
Looking for suspicious processes is easy. While advanced attackers will hide from such means, some traces will likely be left anyway. Process list can be shown using a standard “ps” command:
# chkrootkit
# ps –aux
It can detect a large set of malicious changes to a system as well as common attacker software, such as rootkits.
Look for daemons with a “.” (dot) in the path (likely started by attackers), processes with strange names, etc. To look up the process details one can look directly into the /proc directory, housing the direct link to system resources. For example, the command “cat /proc/13555” will show details on the process with ID equal to 13555. On RedHat-like Linux system one can query the configured services via “chkconfig –list”.
Looking for suspicious networking can be accomplished easily with system built-in tools as well. For example, promiscuous mode detection (sniffer detection) can be done via one of several means: # ip link | grep PROMISC or # /sbin/ifconfig or # dmesg | grep promisc To detect listening services one can use either the “lsof ” or “netstat” command. There two commands will help:
# lsof –i
To conclude, we will look at what have we learned. We can quickly look for known signs of intrusions. We have a plan for doing that, that doesn’t require any expensive “security tools” or advanced skills. Many regular computer users can be trained to do that and so can the system administrators. Similar methods are available for Windows!
Trusted computing — path to security or road to servitude? Sarah Hilley PC hardware as we know it is about to change forever — and all in the name of security. But watchful bystanders fear the new hardware architecture could have a more sinister payload. Sarah Hilley reports on the trusted computing movement. One day a hacker will sink his teeth into the soft underbelly of a commercial company, only to have all his teeth broken from biting into an iron vault. This is the promise of the trusted computing move12
ment. "We can only go so far in protecting against threats with software," says Tony Redmond, VP and CTO, HP Services. This is the rational core of the activities of the Trusted Computing Group (TCG).
Unusual objects and traces and unusual networking are also signs of a Linux hack
About the author Anton Chuvakin, Ph.D., GCIA, GCIH http://www.chuvakin.org is a Senior Security Analyst with netForensics, a security information management company, where he is involved with designing the product and researching potential new security features. His areas of infosec expertise include intrusion detection, UNIX security, forensics, honeypots, etc. He is the author of a book "Security Warrior" (O'Reilly, January 2004) and a contributor to "Know Your Enemy II" by the Honeynet Project (AWL, June 2004) and "Information Security Management Handbook" (CRC, April 2004). In his spare time he maintains his security portal http://www.info-secure.org
Steered by a body that has arisen from the ranks of the IT giants, the TCG is releasing new hardware that will come embedded with a futuristic intelligent smartcard that is aware of what programs are running. The TCG will begin by sowing its seeds in large corporations, but it won't stop there. The group also plans to plant routers, mobile phones, PDAs, servers and ultimately even home PCs with the new embedded chip. Many spectators believe the ultimate goal of the new hardware is digital rights management (DRM). Others believe that it will sound the death knell for open source software in the enterprise. The TCG group have also had to pacify governments with privacy anxieties — HP's Redmond confirms that the TCG has