Prior to addressing each of the scanning techniques specifically, we should address a few underlying principles. The Open Systems Interconnection (OSI) model is an International Organization for Standardization (ISO) standard that defines how networked systems communicate. This model is divided into seven layers that define how application content can be sent by one system and/or received by another. The upper layers (5-7) of the OSI model primarily function to interact with the user, whereas the lower layers (1-4) deal with encoding, formatting, and transmission. These layers consist of the following:

The lower layers of the OSI model are largely used to ensure that network traffic successfully arrives at its intended destination. Many of the commonly used protocols at these lower layers necessitate a response from the destination system and, as such, can be leveraged by potential attackers to identify live systems. Techniques discussed in the remainder of this section will leverage protocols used in layers 2, 3, and 4 to discover live network systems. Prior to addressing each of the specific recipes, we will briefly discuss the protocols used and how they can be leveraged for discovery.

The pros and cons of layer 2 discovery with ARP are as follows:

• Pros:
  • Very fast
  • Highly reliable
• Cons:
  • Cannot discover remote systems (non-routable protocol)

Layer 2 discovery scanning is performed using Address Resolution Protocol (ARP) traffic. ARP is a layer 2 protocol that primarily serves the function of translating logical layer 3 IP addresses to physical layer 2 MAC addresses. When a system needs to locate the physical address that corresponds to a destination IP address, it will broadcast an ARP request packet on the local network segment. This ARP request simply asks the entire network, "Who has this IP address?" The system with the specified IP address will then directly respond to the inquiring system with an ARP reply that contains its layer 2 MAC address. The inquiring system will update its ARP cache, which is a temporary record of IP address and MAC address associations, and will then initiate its communications with the host. ARP can be useful in discovering live hosts on a network, because it does not employ any form of identification or authorization prior to responding to requests.

As a result of this, it is possible and even trivial for an intruder to connect to a local network and enumerate live hosts. This can be performed by sending a series of ARP requests for a comprehensive list of IP addresses and then recording a list of queried IP addresses for which responses were received. ARP discovery has both advantages and disadvantages. It is useful in discovery scanning because it is the fastest and most reliable discovery protocol. Unfortunately, it is also a non-routable protocol and can only be used to discover hosts on the local subnet.

The pros and cons of layer 3 discovery with ICMP are as follows:

• Pros:
  • Can discover remote systems (routable protocol)
  • Still relatively fast
• Cons:
  • Slower than ARP discovery
  • Often filtered by firewalls

Layer 3 discovery is probably the most commonly known and used discovery technique among network administrators and technicians. The famous ping command-line utility, which is found natively on both Windows and *nix systems, uses layer 3 discovery. This form of discovery makes use of Internet Control Message Protocol (ICMP). While ICMP has several functions, one that can be particularly useful to identify live systems is the use of echo request and echo response messages. An ICMP echo request is the technical equivalent of one system asking another system, "Are you there?" An ICMP echo response is how the receiving system can answer, "Yes I am." To determine whether a host exists at a particular IP address, a system can send an ICMP echo request to that address. If there is a host with that IP address and everything works as desired, the host will then return an ICMP echo reply. This protocol can be leveraged in the host discovery by performing this sequence in a loop for a comprehensive list of IP addresses.

The output would consist of a list of only the IP addresses for which a reply was received. Layer 3 discovery is effective because it uses a routable protocol to identify live hosts. However, there are also certain disadvantages associated with its use. ICMP discovery is not as fast as ARP discovery. Also, ICMP discovery is not as reliable as ARP discovery, as some hosts are intentionally configured to not respond to ICMP traffic, and firewalls are frequently configured to drop ICMP traffic. Nonetheless, it is a fast and commonly used approach to discovering potential targets on a remote address range.

Layer 4 discovery is highly effective because publicly routable systems are usually only in the public IP space, as they host networked services that are available over Transmission Control Protocol (TCP) or User Datagram Protocol (UDP). In poorly secured environments, a reply can often be solicited from a remote server by sending nearly any UDP or TCP request to its IP address. However, if stateful filtering is employed, it may be possible to only solicit a response from a remote service with a SYN request directed to a port address associated with a live service. Even in highly secure environments with advanced filtering, discovery is possible in most cases if the right request is supplied. However, with 65,536 possible port addresses for both UDP and TCP services, a fully comprehensive discovery process can be very time consuming. The best approach to layer 4 discovery with both TCP and UDP techniques is to find the right balance between thoroughness and expediency.

The pros and cons of layer 4 discovery with TCP are as follows:

• Pros:
  • Can discover remote systems (routable protocol)
  • More reliable than ICMP (filters are less common or selectively implemented)
• Cons:
  • Stateful firewall filters can produce unreliable results
  • Thorough discovery can be time consuming

Layer 4 discovery with TCP consists of sending TCP packets to potential destination addresses with various TCP flag bits activated. Different flag configurations can trigger various responses that can be used to identify live hosts. Unsolicited TCP finish (FIN) or acknowledge (ACK) packets can often trigger reset (RST) responses from a remote server. The synchronize (SYN) packets sent to a remote server can commonly trigger SYN+ACK or RST responses, depending on the status of the service. The intention is not to solicit a particular response, but instead to solicit any response. Any response from a given IP address is a confirmation that a live system is present.

The pros and cons of layer 4 discovery with UDP are as follows:

• Pros:
  • Can discover remote systems (routable protocol)
  • Can even discover remote hosts with all TCP services filtered
• Cons:
  • Inconsistent use and filtering of ICMP port-unreachable responses makes indiscriminate discovery unreliable
  • Service-specific probe techniques limit thoroughness and increase the required scan time

UDP discovery involves sending UDP probe packets to various destination ports in an attempt to solicit a response from live hosts. UDP discovery can sometimes be effective in identifying live hosts that have all TCP services filtered. However, UDP discovery can be tricky because while some UDP services will reply to UDP packets with ICMP port-unreachable responses, others will only reply to unique requests that specifically correspond to a running service. Additionally, ICMP traffic is commonly filtered by egress restrictions on firewalls, making it difficult to perform indiscriminate UDP discovery. As such, effective UDP discovery scanning often requires unique techniques that vary from service to service.