Commands in Use

In this chapter, we introduce the crontab and schtasks commands.

Step 1: Creating a Port Scanner

The first step in the process is to create a port scanner. To do this, you simply need the ability to create a TCP connection to a given host on a given port. This can be accomplished using the bash file descriptor named /dev/tcp.

To create the port scanner, you first need to read in a list of IP addresses or hostnames from a file. For each host in the file, you will attempt to connect to a range of ports on the host. If the connection succeeds, you know the port is open. If the connection times out or you receive a connection reset, you know the port is closed. For this project, we will scan each host from TCP port 1 through 1023.

#!/bin/bash -
#
# Cybersecurity Ops with bash
# scan.sh
#
# Description:
# Perform a port scan of a specified host
#
# Usage: ./scan.sh <output file>
#   <output file> File to save results in
#

function scan ()
{
  host=$1
  printf '%s' "$host"                                       
  for ((port=1;port<1024;port++))
  do
    # order of redirects is important for 2 reasons
    echo >/dev/null 2>&1  < /dev/tcp/${host}/${port}        
    if (($? == 0)) ; then printf ' %d' "${port}" ; fi       
  done
  echo # or printf '
'
}

#
# main loop
#    read in each host name (from stdin)
#     and scan for open ports
#    save the results in a file
#    whose name is supplied as an argument
#     or default to one based on today's date
#

printf -v TODAY 'scan_%(%F)T' -1   # e.g., scan_2017-11-27  
OUTFILE=${1:-$TODAY}                                        

while read HOSTNAME
do
    scan $HOSTNAME
done > $OUTFILE                                             

Take note of this printf and the other one in this function. Neither has a newline, to keep the code all on one (long) line.

This is the critical step in the script—actually making the network connection to a specified port. This is accomplished through the following code:

echo >/dev/null 2>&1  < /dev/tcp/${host}/${port}

The echo command here has no real arguments, only redirections. The redirections are handled by the shell; the echo command never sees them but it does know that they have happened. With no arguments, echo will just print a newline ( ) character to stdout. Both stdout and stderr have been redirected to /dev/null—effectively thrown away—since for our purposes, we don’t care about the output.

The key here is the redirecting of stdin (via the <). We are redirecting stdin to come from the special bash filename, /dev/tcp/… and some host and port number. Since echo is just doing output, it won’t be reading any input from this special network file; rather, we just want to attempt to open it (read-only) to see if it is there.

This is the other printf in the function. If echo succeeds, a connection was made successfully to that port on the specified host. Therefore, we print out that port number.

The printf function (in newer versions of bash) supports this special format for printing date and time values. The %()T is the printf format specifier that indicates this will be a date/time format. The string inside the parentheses provides the specifics about which pieces of date and/or time you want shown. It uses the specifiers you would use in the strftime system library call. (Type man strftime for more specifics.) In this case, the %F means a year-month-day format (ISO 8601 date format). The date/time used for the printing is specified as -1, which just means “now.”

The -v option to printf says to save the output to a variable rather than print the output. In this case, we use TODAY as the variable.

If the user specifies an output file on the command line as the first argument to this script, we’ll use it. If that first argument is null, we’ll use the string we just created in TODAY with today’s date to be the output filename.

By redirecting output on done, we redirect the output for all the code inside the while loop. If we did the redirect on the scan command itself, we would have to use the >> to append to the file. Otherwise, each iteration through the loop would save only one command’s output, clobbering the previous output. If each command is appending to the file, then before the loop starts, we would need to truncate the file. So you can see how much simpler it is to just redirect on the while loop.

The scan output file will be formatted by using a space as a separator. Each line will begin with the IP address or hostname, and then any open TCP ports will follow. Example 9-2 is a sample of the output format that shows ports 80 and 443 open on host 192.168.0.1, and port 25 open on host 10.0.0.5.

192.168.0.1 80 443
10.0.0.5 25

Step 2: Comparing to Previous Output

The ultimate goal of this tool is to detect host changes on a network. To accomplish that, you must be able to save the results of each scan to a file. You can then compare the latest scan to a previous result and output any difference. Specifically, you are looking for any device that has had a TCP port opened or closed. Once you have determined that a new port has been opened or closed, you can evaluate it to determine whether it was an authorized change or may be a sign of malicious activity.

Example 9-3 compares the latest scan with a previous scan and outputs any changes.

#!/bin/bash -
#
# Cybersecurity Ops with bash
# fd2.sh
#
# Description:
# Compares two port scans to find changes
# MAJOR ASSUMPTION: both files have the same # of lines,
# each line with the same host address
# though with possibly different listed ports
#
# Usage: ./fd2.sh <file1> <file2>
#

# look for "$LOOKFOR" in the list of args to this function
# returns true (0) if it is not in the list
function NotInList ()                                            
{
    for port in "$@"
    do
        if [[ $port == $LOOKFOR ]]
        then
            return 1
        fi
    done
    return 0
}

while true
do
    read aline <&4 || break         # at EOF                  
    read bline <&5 || break         # at EOF, for symmetry    

    # if [[ $aline == $bline ]] ; then continue; fi
    [[ $aline == $bline ]] && continue;                       

    # there's a difference, so we
    # subdivide into host and ports
    HOSTA=${aline%% *}                                        
    PORTSA=( ${aline#* } )                                    

    HOSTB=${bline%% *}
    PORTSB=( ${bline#* } )

    echo $HOSTA                 # identify the host which changed

    for porta in ${PORTSA[@]}
    do         
          LOOKFOR=$porta NotInList ${PORTSB[@]} && echo "  closed: $porta"
    done

    for portb in ${PORTSB[@]}
    do
          LOOKFOR=$portb NotInList ${PORTSA[@]} && echo "     new: $portb"
    done

done 4< ${1:-day1.data} 5< ${2:-day2.data}                   
# day1.data and day2.data are default names to make it easier to test

The NotInList function is written to return what amounts to a value of true or false. Remember that in the shell (except inside double parentheses), the value of 0 is considered “true.” (Zero is returned from commands when no error occurs, so that is considered “true”; nonzero return values typically indicate an error, so that is considered “false.”)

A “trick” in this script is being able to read from two different streams of input. We use file descriptors 4 and 5 for that purpose in this script. Here the variable aline is being filled in by reading from file descriptor 4. We will see shortly where 4 and 5 get their data. The ampersand is necessary in front of the 4 to make it clear that this is file descriptor 4. Without the ampersand, bash would try to read from a file named 4. After the last line of input data is read, when we reach the end of file, the read returns an error; in that case, the break will be executed, ending the loop.

Similarly for bline, it will read its data from file descriptor 5. Since the two files are supposed to have the same number of lines (i.e., the same hosts), the break here shouldn’t be necessary, as it will have happened on the previous line. However, the symmetry makes it more readable.

If the two lines are identical, there’s no need to parse them into individual port numbers, so we take a shortcut and move on to the next iteration of the loop.

We isolate the hostname by removing all the characters after (and including) the first space.

Conversely, we can pull out all the port numbers by removing the hostname—removing all the characters from the front of the string, up to and including the first space. Notice that we don’t just assign this list to a variable. We use the parentheses to initialize this variable as an array, with each of the port numbers as one of the entries in the array.

Look at the statement immediately below this number. This variable assignment is followed immediately by a command on the same line. For the shell, this means that the variable’s value is in effect only for the duration of the command. Once the command is complete, the variable returns to its previous value. That’s why we don’t echo $LOOKFOR later in that line; it won’t be a valid value. We could have done this as two separate commands—the variable assignment and the call to the function, but then you wouldn’t have learned about this feature in bash.

Here is where the novel use of file descriptors gets set up. File descriptor 4 gets “redirected” to read its input from the file named in the first argument to the script. Similarly, 5 gets its input from the second argument. If one or both aren’t set, the script will use the default names specified.

Step 3: Automation and Notification

Although you can execute the script manually, it would be much more useful if it ran every day or every few days and notified you of any changes that were detected. Autoscan.sh, shown in Example 9-4, is a single script that uses scan.sh and fd2.sh to scan the network and output any changes.

#!/bin/bash -
#
# Cybersecurity Ops with bash
# autoscan.sh
#
# Description:
# Automatically performs a port scan (using scan.sh),
# compares output to previous results, and emails user
# Assumes that scan.sh is in the current directory.
#
# Usage: ./autoscan.sh
#

./scan.sh < hostlist                                      

FILELIST=$(ls scan_* | tail -2)                           
FILES=( $FILELIST )

TMPFILE=$(tempfile)                                       

./fd2.sh ${FILES[0]} ${FILES[1]}  > $TMPFILE

if [[ -s $TMPFILE ]]   # non-empty                        
then
    echo "mailing today's port differences to $USER"
    mail -s "today's port differences" $USER < $TMPFILE   
fi
# clean up
rm -f $TMPFILE                                            

Running the scan.sh script will scan all the hosts in a file called hostlist. Since we don’t supply a filename as an argument to the scan.sh script, it will generate a name for us by using the year-month-day numerical format.

The default names for output from scan.sh will sort nicely. The ls command will return them in date order without us having to specify any special options on the ls. Using tail, we get the last two names in the list—the two most recent files. In the next line, we put those names into an array, for easy parsing into two pieces.

Creating a temporary filename with the tempfile command is the most reliable way to make sure that the file isn’t otherwise in use or unwritable.

The -s option tests whether the file size is greater than zero (that the file is not empty). The temporary file will be nonempty when there is a difference between the two files compared with fd2.sh.

The $USER variable is automatically set to your user ID, though you may want to put something else here if your email address is different from your user ID.

There are better ways to be sure that the file gets removed no matter where/when the script exits, but this is a minimum, so we don’t get these scratch files accumulating. See some later scripts for the use of the trap built-in.

The autoscan.sh script can be set to run at a specified interval by using crontab in Linux or schtasks in Windows.

$ crontab -l

no crontab for paul

$ crontab -e

no crontab for paul - using an empty one

Select an editor.  To change later, run 'select-editor'.
  1. /bin/ed
  2. /bin/nano        <---- easiest
  3. /usr/bin/vim.basic
  4. /usr/bin/vim.tiny

Choose 1-4 [2]:

0 8 * * * /home/paul/autoscan.sh

schtasks //Create //TN "Network Scanner" //SC DAILY //ST 08:00
//TR "C:UsersPaulAppDataLocalProgramsGitgit-bash.exe
C:UsersPaulautoscan."

Summary

The ability to detect deviations from an established baseline is one of the most powerful ways to detect anomalous activity. A system unexpectedly opening a server port could indicate the presence of a network backdoor.

In the next chapter, we look at how baselining can be used to detect suspicious activity on a local filesystem.

Workshop

Try expanding and customizing the features of the network monitoring tool by adding the following functionality:

1. When comparing two scan files, account for files of different lengths or with a different set of IP addresses/hostnames.

2. Use /dev/tcp to create a rudimentary Simple Mail Transfer Protocol (SMTP) client so the script does not need the mail command.

Visit the Cybersecurity Ops website for additional resources and the answers to these questions.