- Email: [email protected]
Rogue Servers
rogue servers
Rogue Servers Piers Wilson, Insight Consulting Rogue servers are a constant threat and pose a variety of problems for the IT security team. This article summarises the threats that come from not managing rogue servers and gives prevention and detection tips.
Unauthorized installations Within this discussion we will be considering “rogue servers”. This term will be used to refer to a number of different situations, but all represent some form of unauthorized installation of either physical hardware or software to create a local (usually) IT facility within the existing, managed, supported network/host environment. Examples of this will typically include: • Server platforms installed locally, e.g. by departments, to provide dedicated facilities such as file storage or Intranet (Web) application servers. • Server software applications installed on existing physical platforms to extend their functionality, such as installing/enabling IIS on a W2K server or putting a Web or FTP server application on someone’s desktop PC. • Server software applications which have been installed by default (but are not required) on existing/newly deployed servers that are within the “management framework”.
Figure 1 – A typical example of a server in an office environment, totally outside the control of the IT department.
16
• Third party application servers that are not installed by the IT department (e.g. systems deployed to support phone switches or E-banking applications). There are a couple of other types to consider: • Servers that have been “formally” deployed but have subsequently been converted to RAS servers by the addition of a modem. This sometimes also applies to the third party deployed systems which often have modems configured to allow dial-up support/diagnostics, but outside of the “official” remote access facility. • Existing servers, particularly external facing ones, that have been compromised and have had server software, Trojan control software, unauthorized content or distributed attack tools installed.
“Unauthorized” We have used that word a lot in the sections above. It is a word that has to be defined specifically for each particular organization. Most people would not argue that the above scenarios are undesirable, but unless there are explicit policy statements and supporting standards and procedures in place it may be difficult to categorise any such installation as “unauthorized”. Starting at the top down then your policy should include some set of statements as follows: • Installation of physical hardware and software on the corporate network must
only be performed by the IT department and according to defined configuration standards. • Deployment of additional hardware and software on existing server or workstations must be subject to approval of [IT Manager/Head of Security] and must be undertaken by the relevant IT support team. • Where the modifications are judged to have a potential impact on other network systems, applications or data (at the discretion of the [IT Manager/Head of Security]) the installation must first be successfully tested. • All changes to the hardware, software and configuration of the network or the connected systems must be conducted under a managed Change Control process. Obviously, following on from this you may have some work to do to actually define and establish the standards and processes. You will at the very least need the development of a set of standard configurations for each server types (e.g. W2K, Unix), you must also have in place a testing and approvals mechanism for new developments or procurements. Finally this must be governed by a change control requesting and management process to enable the impacts of deployments, their scheduling/planning and regression steps to be approved prior to implementation.
Problems with “Rogue servers” The exact nature of the problem that unofficially deployed systems present will inevitably vary depending on your perspective. For an outsourcing agency or Facilities Management (FM) company they represent not only lost revenue from the support of IT systems but also they are potential points of an attack whose effect could be wider than the
rogue servers system itself and encroach on formally managed systems where service levels exist. For corporate procurement and IT departments they are often untraceable/ unregistered/unmanaged assets in various states/locations. From an information security perspective, not only do they present a risk through providing a jumping off point for external or internal attackers, but with any systems that have been deployed to fill a business function there is a clear risk to ongoing operations (albeit of the department concerned) and more seriously, a risk that the system is used as a data storage facility for operational data or contact details, documentation etc. which has improper access controls, weak authentication and exists outside of the normal backup and security management process. In general however the following list of issues may be faced:
Data storage/sensitivity The system may have been deployed to host a database used for some business function such as invoice tracking, payroll, contact management or document management. This data storage may not have appropriate security controls applied to its access or use. As it will probably be a separate platform, the authentication may exist on the local system (under the management of the business users, a third party or within the application) and security access control settings (e.g. to restrict access to data, particularly personal data covered by the data protection act) may not be appropriate.
Backup There is risk associated with centralised backup facilities that exist in many environments which operate over the network using either a SAN or tape based system. Although the department, business users or local IT “expert” may be happy to live with the risk, this “head in the sand” approach does not sit well in any environment where responsible IT security and business continuity practices are supposed to be followed.
Lower security configuration The systems deployed at a corporate level by the “official” IT department or FM company should have a secure configuration which includes a structured build process where insecure default settings are turned off, security features are enabled and access rights are tightened. Out of the box these settings are typically at their most insecure Servers deployed by a local IT-knowledgeable person or business manager will, in the vast majority of cases, not have been configured according to either best practice or the local standard. This means the systems will have a number of weaknesses which would otherwise have been addressed.
Versions/support/patching Any server or workstation will need constant support in terms of the roll out of security patches to fix new vulnerabilities. In a corporate environment this can be very hard to keep on top of, but there are solutions – patches can be downloaded centrally and then deployed or a server/platform list can be used to make sure relevant patches can be applied (similar to an asset inventory). Of course a server that has been deployed by a department or business unit independently of this will most probably be passed over when new patches or updates are deployed. The department, not being “IT professionals” may not even realise the importance of this – and if they do, as business users, they may not have the time/resource to undertake the kind of preventative maintenance that all modern operating systems requires. The end result is therefore a single point at which an attack could be successful – however once compromised such a system might then allow other systems to be compromised.
External connectivity Many “unofficially deployed” servers are used to address a specific business application or non-IT requirement. In many cases this solution, server included, has been procured from a third party application provider and is often delivered
and installed by them on behalf of the procuring department. Ironically, as this system is not an officially supported IT server the department will buy support from the provider. In my experience I have come across a number of these types of systems and almost inevitably the support is for the application, OS and hardware. How does this support operate? You guessed it, the third party company dial up into the server using an attached modem and fix things, upload changes etc. This is of course an unauthorized network connection and may stomp all over a carefully designed secure remote access policy. In one case recently the remote support function relied on an ISDN router that was provided by the support firm and effectively (when connected) allowed a routed connection onto the client network. In that particular instance the method of support was chiefly using anonymous FTP to upload patches, binaries and any modifications direct to the program directory.
Domain/address conflicts (master browser) The domain structure and IP addressing is almost always under the control of the IT department. As such when an “unofficial” or rogue server is deployed the hostname, domain membership and IP addressing tends to be largely left to chance. I have seen at least one instance where an IP address was used that had not been removed from the DHCP scope (all the main servers were in a specific address range) and caused problems for workstation users with address conflicts. The fact that systems may not be within the existing domains often tends to lead to the problem of separate sets of credentials (described more fully below), but also, in a Windows environment we have seen problems with the browsing service (available in network neighbourhood – not what you do with Internet explorer) where a new system has repeatedly caused a browser election to decide on a network
17
rogue servers master browser. This caused a substantial amount of additional network traffic and affected all the users in the office, and additional fault finding work for the IT department.
Unmanaged credentials Where systems are deployed outside of an established domain the username/passwords will be managed separately, with policies on password lifetime, length, history etc. unlikely to be applied as stringently or updated as regularly as required by the corporate security standards, or even best practice. There is a risk that users on the server will set the passwords to be the same as their main network login – this potentially exposes these sensitive credentials on an unmanaged, unsecured system.
Asset tracking (software licensing) Although the costs of licences may have been included in the procurement cost, auxiliary software required may not have been. Also where a corporate licence for Windows platforms exists the organisation may have inadvertently paid for unnecessary licences. The migration to a newer flavour of operating system also typically will not happen, thus an environment may upgrade all supported servers to a newer version of Windows with the extraneous systems being left on an older version.
Anti-virus This final issue hardly needs explanation. The anti-virus software, if installed at all, may not be properly configured to collect updates from a corporate central point or even from an external source. As the server is not managed by IT they are unlikely to even notice if the signatures are never updated. Clearly the results of this could be disastrous.
Detection, audit and prevention As with most IT security issues, the policy level controls described right at the start of this article are vital. You can define what you mean by “unauthorized” and prohibit the use of such systems.
18
Of course, you may need to then retrospectively or periodically audit your networks to try to find these systems. A good way to identify rogue servers is to establish a naming convention for the authorized systems. This often will not be followed by a standalone departmental server so if you adopt a standard including an incremental number you can easily tally the expected servers against any new arrivals like “mktg-intranet”. There are of course ways to enforce this, many of the network scanning tools are capable of very quickly auditing a network to discover systems. These include nmap (which is freeware), GFI’s LANGuard (which is excellent value for money and able to audit a network range very quickly) right up to many of the more expensive network mapping/management tools (including the likes of HP Openview, which has an auto-discovery mode). Many of these tools can generate output that can be subsequently used to compare future scans against reference scans thus highlighting any of these changes to the landscape. Ideally of course you would reconcile this against the change control record or asset inventory. The allocation of IP addresses (and IP addressing scheme) can also be used to control assets. One organization used a specific scheme using a 16-bit mask internally and then using the third octet to denote the system type. In addition, all servers were kept in defined subnets, separate from printers or DHCP pool addresses. For example, configure all server IP addresses as x.y.100.z/16, where z was an index number and also included in the hostname (e.g. 172.16.100.35/16, hostname SVR035). This makes it easy to identify server systems that are not in this range but have been configured to get a DHCP address or on some other free IP address by some departmental user. In some cases the extent of the problem has all but escaped the IT department’s ability to catch up, at that point consider embarking on a finite network mapping exercise — you may be surprised what
systems you find and what devices or software they have installed.
Conclusions The issue of unofficial IT equipment and devices being attached to the corporate network is a very real one. The potential problems and costs are frighteningly real and within a properly managed IT/Information security environment they must not be allowed to proliferate. Steps can be taken to identify these sorts of systems when they are present and also to prevent, or make it exceedingly difficult to achieve their establishment – thus forcing people down the proper procurement and support route. However, to be fair to the users and departments, the mechanisms by which IT systems are procured and supported must be made sensible enough that the overhead in “buying a server through IT” is not so great that the unofficial routes are taken. For third party deployments, the IT service provided internally must be allowed to provide and configure a basic, standardized server onto which the application provider can perform the install. Where remote support mechanisms must be established steps must be taken to secure these — as with any third party connection. As with so many issues, it is always best to treat the causes rather than the symptoms.
Useful links: • Nmap (network mapping, OS checking and port scanner), freeware - www.nmap.org • LANGuard (very good network mapping/reconnaissance tool - can also be used to deploy hotfixes), $295-$895 - www.gfi.com • NetworkView (excellent value graphical mapping and system monitoring tool), $59 - www.networkview.com • HP Openview - www.openview.hp. com
Rogue Servers Piers Wilson, Insight Consulting Rogue servers are a constant threat and pose a variety of problems for the IT security team. This article summarises the threats that come from not managing rogue servers and gives prevention and detection tips.
Unauthorized installations Within this discussion we will be considering “rogue servers”. This term will be used to refer to a number of different situations, but all represent some form of unauthorized installation of either physical hardware or software to create a local (usually) IT facility within the existing, managed, supported network/host environment. Examples of this will typically include: • Server platforms installed locally, e.g. by departments, to provide dedicated facilities such as file storage or Intranet (Web) application servers. • Server software applications installed on existing physical platforms to extend their functionality, such as installing/enabling IIS on a W2K server or putting a Web or FTP server application on someone’s desktop PC. • Server software applications which have been installed by default (but are not required) on existing/newly deployed servers that are within the “management framework”.
Figure 1 – A typical example of a server in an office environment, totally outside the control of the IT department.
16
• Third party application servers that are not installed by the IT department (e.g. systems deployed to support phone switches or E-banking applications). There are a couple of other types to consider: • Servers that have been “formally” deployed but have subsequently been converted to RAS servers by the addition of a modem. This sometimes also applies to the third party deployed systems which often have modems configured to allow dial-up support/diagnostics, but outside of the “official” remote access facility. • Existing servers, particularly external facing ones, that have been compromised and have had server software, Trojan control software, unauthorized content or distributed attack tools installed.
“Unauthorized” We have used that word a lot in the sections above. It is a word that has to be defined specifically for each particular organization. Most people would not argue that the above scenarios are undesirable, but unless there are explicit policy statements and supporting standards and procedures in place it may be difficult to categorise any such installation as “unauthorized”. Starting at the top down then your policy should include some set of statements as follows: • Installation of physical hardware and software on the corporate network must
only be performed by the IT department and according to defined configuration standards. • Deployment of additional hardware and software on existing server or workstations must be subject to approval of [IT Manager/Head of Security] and must be undertaken by the relevant IT support team. • Where the modifications are judged to have a potential impact on other network systems, applications or data (at the discretion of the [IT Manager/Head of Security]) the installation must first be successfully tested. • All changes to the hardware, software and configuration of the network or the connected systems must be conducted under a managed Change Control process. Obviously, following on from this you may have some work to do to actually define and establish the standards and processes. You will at the very least need the development of a set of standard configurations for each server types (e.g. W2K, Unix), you must also have in place a testing and approvals mechanism for new developments or procurements. Finally this must be governed by a change control requesting and management process to enable the impacts of deployments, their scheduling/planning and regression steps to be approved prior to implementation.
Problems with “Rogue servers” The exact nature of the problem that unofficially deployed systems present will inevitably vary depending on your perspective. For an outsourcing agency or Facilities Management (FM) company they represent not only lost revenue from the support of IT systems but also they are potential points of an attack whose effect could be wider than the
rogue servers system itself and encroach on formally managed systems where service levels exist. For corporate procurement and IT departments they are often untraceable/ unregistered/unmanaged assets in various states/locations. From an information security perspective, not only do they present a risk through providing a jumping off point for external or internal attackers, but with any systems that have been deployed to fill a business function there is a clear risk to ongoing operations (albeit of the department concerned) and more seriously, a risk that the system is used as a data storage facility for operational data or contact details, documentation etc. which has improper access controls, weak authentication and exists outside of the normal backup and security management process. In general however the following list of issues may be faced:
Data storage/sensitivity The system may have been deployed to host a database used for some business function such as invoice tracking, payroll, contact management or document management. This data storage may not have appropriate security controls applied to its access or use. As it will probably be a separate platform, the authentication may exist on the local system (under the management of the business users, a third party or within the application) and security access control settings (e.g. to restrict access to data, particularly personal data covered by the data protection act) may not be appropriate.
Backup There is risk associated with centralised backup facilities that exist in many environments which operate over the network using either a SAN or tape based system. Although the department, business users or local IT “expert” may be happy to live with the risk, this “head in the sand” approach does not sit well in any environment where responsible IT security and business continuity practices are supposed to be followed.
Lower security configuration The systems deployed at a corporate level by the “official” IT department or FM company should have a secure configuration which includes a structured build process where insecure default settings are turned off, security features are enabled and access rights are tightened. Out of the box these settings are typically at their most insecure Servers deployed by a local IT-knowledgeable person or business manager will, in the vast majority of cases, not have been configured according to either best practice or the local standard. This means the systems will have a number of weaknesses which would otherwise have been addressed.
Versions/support/patching Any server or workstation will need constant support in terms of the roll out of security patches to fix new vulnerabilities. In a corporate environment this can be very hard to keep on top of, but there are solutions – patches can be downloaded centrally and then deployed or a server/platform list can be used to make sure relevant patches can be applied (similar to an asset inventory). Of course a server that has been deployed by a department or business unit independently of this will most probably be passed over when new patches or updates are deployed. The department, not being “IT professionals” may not even realise the importance of this – and if they do, as business users, they may not have the time/resource to undertake the kind of preventative maintenance that all modern operating systems requires. The end result is therefore a single point at which an attack could be successful – however once compromised such a system might then allow other systems to be compromised.
External connectivity Many “unofficially deployed” servers are used to address a specific business application or non-IT requirement. In many cases this solution, server included, has been procured from a third party application provider and is often delivered
and installed by them on behalf of the procuring department. Ironically, as this system is not an officially supported IT server the department will buy support from the provider. In my experience I have come across a number of these types of systems and almost inevitably the support is for the application, OS and hardware. How does this support operate? You guessed it, the third party company dial up into the server using an attached modem and fix things, upload changes etc. This is of course an unauthorized network connection and may stomp all over a carefully designed secure remote access policy. In one case recently the remote support function relied on an ISDN router that was provided by the support firm and effectively (when connected) allowed a routed connection onto the client network. In that particular instance the method of support was chiefly using anonymous FTP to upload patches, binaries and any modifications direct to the program directory.
Domain/address conflicts (master browser) The domain structure and IP addressing is almost always under the control of the IT department. As such when an “unofficial” or rogue server is deployed the hostname, domain membership and IP addressing tends to be largely left to chance. I have seen at least one instance where an IP address was used that had not been removed from the DHCP scope (all the main servers were in a specific address range) and caused problems for workstation users with address conflicts. The fact that systems may not be within the existing domains often tends to lead to the problem of separate sets of credentials (described more fully below), but also, in a Windows environment we have seen problems with the browsing service (available in network neighbourhood – not what you do with Internet explorer) where a new system has repeatedly caused a browser election to decide on a network
17
rogue servers master browser. This caused a substantial amount of additional network traffic and affected all the users in the office, and additional fault finding work for the IT department.
Unmanaged credentials Where systems are deployed outside of an established domain the username/passwords will be managed separately, with policies on password lifetime, length, history etc. unlikely to be applied as stringently or updated as regularly as required by the corporate security standards, or even best practice. There is a risk that users on the server will set the passwords to be the same as their main network login – this potentially exposes these sensitive credentials on an unmanaged, unsecured system.
Asset tracking (software licensing) Although the costs of licences may have been included in the procurement cost, auxiliary software required may not have been. Also where a corporate licence for Windows platforms exists the organisation may have inadvertently paid for unnecessary licences. The migration to a newer flavour of operating system also typically will not happen, thus an environment may upgrade all supported servers to a newer version of Windows with the extraneous systems being left on an older version.
Anti-virus This final issue hardly needs explanation. The anti-virus software, if installed at all, may not be properly configured to collect updates from a corporate central point or even from an external source. As the server is not managed by IT they are unlikely to even notice if the signatures are never updated. Clearly the results of this could be disastrous.
Detection, audit and prevention As with most IT security issues, the policy level controls described right at the start of this article are vital. You can define what you mean by “unauthorized” and prohibit the use of such systems.
18
Of course, you may need to then retrospectively or periodically audit your networks to try to find these systems. A good way to identify rogue servers is to establish a naming convention for the authorized systems. This often will not be followed by a standalone departmental server so if you adopt a standard including an incremental number you can easily tally the expected servers against any new arrivals like “mktg-intranet”. There are of course ways to enforce this, many of the network scanning tools are capable of very quickly auditing a network to discover systems. These include nmap (which is freeware), GFI’s LANGuard (which is excellent value for money and able to audit a network range very quickly) right up to many of the more expensive network mapping/management tools (including the likes of HP Openview, which has an auto-discovery mode). Many of these tools can generate output that can be subsequently used to compare future scans against reference scans thus highlighting any of these changes to the landscape. Ideally of course you would reconcile this against the change control record or asset inventory. The allocation of IP addresses (and IP addressing scheme) can also be used to control assets. One organization used a specific scheme using a 16-bit mask internally and then using the third octet to denote the system type. In addition, all servers were kept in defined subnets, separate from printers or DHCP pool addresses. For example, configure all server IP addresses as x.y.100.z/16, where z was an index number and also included in the hostname (e.g. 172.16.100.35/16, hostname SVR035). This makes it easy to identify server systems that are not in this range but have been configured to get a DHCP address or on some other free IP address by some departmental user. In some cases the extent of the problem has all but escaped the IT department’s ability to catch up, at that point consider embarking on a finite network mapping exercise — you may be surprised what
systems you find and what devices or software they have installed.
Conclusions The issue of unofficial IT equipment and devices being attached to the corporate network is a very real one. The potential problems and costs are frighteningly real and within a properly managed IT/Information security environment they must not be allowed to proliferate. Steps can be taken to identify these sorts of systems when they are present and also to prevent, or make it exceedingly difficult to achieve their establishment – thus forcing people down the proper procurement and support route. However, to be fair to the users and departments, the mechanisms by which IT systems are procured and supported must be made sensible enough that the overhead in “buying a server through IT” is not so great that the unofficial routes are taken. For third party deployments, the IT service provided internally must be allowed to provide and configure a basic, standardized server onto which the application provider can perform the install. Where remote support mechanisms must be established steps must be taken to secure these — as with any third party connection. As with so many issues, it is always best to treat the causes rather than the symptoms.
Useful links: • Nmap (network mapping, OS checking and port scanner), freeware - www.nmap.org