Chapter 12. Formatting and Reporting

To maximize usefulness, the data collected and analyzed previously must be presented in a clear format that is easy to understand. Standard command-line output is not often well formatted to present large amounts of information, but some techniques can be used to improve readability.

Commands in Use

In this chapter, we introduce tput to control formatting in the terminal.

tput

The tput command can be used to control formatting in the terminal such as cursor location and behavior. Note that tput is actually an extraction. The command looks up the terminal formatting codes in the terminfo database.

Common command parameters

clear

Clear the screen

cols

Print the number of terminal columns

cup <x> <y>

Move the cursor to position <x> and <y>

lines

Print the number of terminal lines

rmcup

Restore the previously saved terminal layout

setab

Set the terminal background color

setaf

Set the terminal foreground color

smcup

Save the current terminal layout and clear the screen

Formatting for Display and Print with HTML

Converting information to HTML is a great way to provide clean and clear formatting if you do not need to view it directly on the command line. This is also a good option if you ultimately want to print the information, as you can use the web browser’s built-in print capabilities.

The full syntax of HTML is beyond the scope of this book, but we will cover some of the basics. HTML is a computer language that is defined by a series of tags that control the way data is formatted and behaves in a web browser. HTML typically uses start tags such as <head> and a corresponding end tag that contains a forward slash such as </head>. Table 12-1 lists several of the most common tags and their purposes.

Table 12-1. Basic HTML tags
Tag Purpose

<HTML>

Outermost tag in an HTML document

<body>

Tag that surrounds the main content of an HTML document

<h1>

Title

<b>

Bold text

<ol>

Numbered list

<ul>

Bulleted list

Example 12-1 shows a sample HTML document.

Example 12-1. Raw HTML document
<html>  1
   <body>  2
      <h1>This is a header</h1>
      <b>this is bold text</b>
      <a href="http://www.oreilly.com">this is a link</a>

      <ol>  3
         <li>This is list item 1</li>  4
         <li>This is list item 2</li>
      </ol>

      <table border=1>  5
         <tr>  6
            <td>Row 1, Column 1</td>  7
            <td>Row 1, Column 2</td>
         </tr>
         <tr>
            <td>Row 2, Column 1</td>
            <td>Row 2, Column 2</td>
         </tr>
      </table>
   </body>
</html>
1

HTML documents must begin and end with the <html> tag.

2

The main content of a web page is contained inside the <body> tag.

3

Lists use the <ol> tag for a numbered list, or the <ul> tag for bulleted lists.

4

The <li> tag defines a list item.

5

The <table> tag is used to define a table.

6

The <tr> tag is used to define a table row.

7

The <td> tag is used to define a table cell.

Tip

For more information on HTML, see the World Wide Web Consortium HTML5 reference.

Figure 12-1 shows how Example 12-1 looks when rendered in a web browser.

An image an HTML web page
Figure 12-1. Rendered HTML web page

To make outputting to HTML easier, you can create a simple script to wrap items in tags. Example 12-2 takes in a string and a tag and outputs that string surrounded by the tag and then a newline.

Example 12-2. tagit.sh
#!/bin/bash -
#
# Cybersecurity Ops with bash
# tagit.sh
#
# Description:
# Place open and close tags around a string
#
# Usage:
# tagit.sh <tag> <string>
#   <tag> Tag to use
#   <string> String to tag
#

printf '<%s>%s</%s>
' "${1}" "${2}" "${1}"

This could also be made into a simple function that can be included in other scripts:

function tagit ()
{
    printf '<%s>%s</%s>
' "${1}" "${2}" "${1}"
}

You can use HTML tags to reformat almost any type of data and make it easier to read. Example 12-3 is a script that reads in the Apache access.log file from Example 7-2 and uses the tagit function to reformat and output the log file as HTML.

Example 12-3. weblogfmt.sh
#!/bin/bash -
#
# Cybersecurity Ops with bash
# weblogfmt.sh
#
# Description:
# Read in Apache web log and output as HTML
#
# Usage:
# weblogfmt.sh < input.file > output.file
#

function tagit()
{
	printf '<%s>%s</%s>
' "${1}" "${2}" "${1}"
}

#basic header tags
echo "<html>"                                            1
echo "<body>"
echo "<h1>$1</h1>"   #title

echo "<table border=1>"   #table with border
echo "<tr>"   #new table row
echo "<th>IP Address</th>"  #column header
echo "<th>Date</th>"
echo "<th>URL Requested</th>"
echo "<th>Status Code</th>"
echo "<th>Size</th>"
echo "<th>Referrer</th>"
echo "<th>User Agent</th>"
echo "</tr>"

while read f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 f12plus    2
do
	echo "<tr>"
	tagit "td" "${f1}"
	tagit "td" "${f4} ${f5}"                        3
	tagit "td" "${f6} ${f7}"
	tagit "td" "${f9}"
	tagit "td" "${f10}"
	tagit "td" "${f11}"
	tagit "td" "${f12plus}"
	echo "</tr>"
done < $1

#close tags
echo "</table>"
echo "</body>"
echo "</html>"
1

There are several ways to print out a bunch of text. We could have used a here document along with the cat program, something like this:

cat <<EOF
<html>
<body>
<h1>$1</h1>
...
EOF

This has the advantage of not needing to repeat all the echo commands. Notice that the $1 substitution will still take place—unless you quote the EOF in some form when invoked. One disadvantage, though, is that we can’t intersperse comments with our input.

2

The logfile is a rather fixed format file, at least for the first several fields. We can read each line from the log file and parse it this way into fields. We also could have used read -a RAOFTXT to read all the fields into an array, one field for each index. The difficulty in that approach comes in printing out all the remaining fields after field 12. With the approach we’ve taken in this script, all the remaining words are all included in the last field—which is why we named it f12plus.

3

Notice that on this line and the next are two arguments enclosed in a single pair of double quotes. On this line, it is both f4 and f5. Putting them both together inside the single pair of quotes makes them a single argument ($2) to the tagit script. Similar reasoning tells us that f12plus needs to be in quotes so that the several words in that field are all treated as a single argument to tagit.

Figure 12-2 shows the sample output from Example 12-3.

Output from weblogfmt.sh
Figure 12-2. Rendered output from weblogfmt.sh

You can use the techniques presented in Chapter 7 to filter and sort the data before piping it into a script such as weblogfmt.sh for formatting.

Creating a Dashboard

Dashboards are useful if you want to display several pieces of information that change over time. The following dashboard will display output from three scripts and update them at a regular interval.

It makes use of the graphical features of the terminal window. Rather than just scrolling the data, page after page, this script will repaint the screen from the same starting position each time so you can see it update in place.

To keep it portable across different terminal window programs, it uses the tput command to ask for the sequence of characters that do graphical things for the type of terminal window in which it is running.

Since the screen is “repainting” over itself, you can’t simply move to the top of the screen and regenerate the output. Why? Because the next iteration may have shorter or fewer lines than the previous output, and you don’t want to leave old data on the screen.

You could begin by clearing the screen, but that visual effect is more jarring if the screen flashes blank before being filled (should there be any delays in the commands that provide the output for display). Instead, you can send all output through a function (of our own making) that will print each line of output but add to the end of each line the character sequence that will clear to the end of the line, thereby removing any previous output. This also allows you to add a little finesse by creating a line of dashes at the end of each command’s output.

Example 12-4 illustrates how to create an on-screen dashboard that contains three distinct output sections.

Example 12-4. webdash.sh
#!/bin/bash -
#
# Rapid Cybersecurity Ops
# webdash.sh
#
# Description:
# Create an information dashboard
# Heading
# --------------
# 1-line of output
# --------------
# 5 lines of output
# ...
# --------------
# column labels and then
# 8 lines of histograms
# ...
# --------------
#

# some important constant strings
UPTOP=$(tput cup 0 0)                               1
ERAS2EOL=$(tput el)
REV=$(tput rev)		# reverse video
OFF=$(tput sgr0)	# general reset
SMUL=$(tput smul)	# underline mode on (start)
RMUL=$(tput rmul)	# underline mode off (reset)
COLUMNS=$(tput cols)	# how wide is our window
# DASHES='------------------------------------'
printf -v DASHES '%*s' $COLUMNS '-'                 2
DASHES=${DASHES// /-}

#
# prSection - print a section of the screen
#       print $1-many lines from stdin
#       each line is a full line of text
#       followed by erase-to-end-of-line
#       sections end with a line of dashes
#
function prSection ()
{
    local -i i					    3
    for((i=0; i < ${1:-5}; i++))
    do
        read aline
        printf '%s%s
' "$aline" "${ERAS2EOL}"	    4
    done
    printf '%s%s
%s' "$DASHES" "${ERAS2EOL}" "${ERAS2EOL}"
}

function cleanup()				    5
{
    if [[ -n $BGPID ]]
    then
      kill %1					    6
      rm -f $TMPFILE
    fi
} &> /dev/null					    7

trap cleanup EXIT

# launch the bg process
TMPFILE=$(tempfile)                                 8
{ bash tailcount.sh $1 | 
  bash livebar.sh > $TMPFILE ; } &                  9
BGPID=$!

clear
while true
do
    printf '%s' "$UPTOP"
    # heading:
    echo "${REV}Rapid Cyber Ops Ch. 12 -- Security Dashboard${OFF}" 
    | prSection 1
    #----------------------------------------
    {                                               10
      printf 'connections:%4d        %s
' 
            $(netstat -an | grep 'ESTAB' | wc -l) "$(date)"
    } | prSection 1
    #----------------------------------------
    tail -5 /var/log/syslog | cut -c 1-16,45-105 | prSection 5
    #----------------------------------------
    { echo "${SMUL}yymmdd${RMUL}"    
            "${SMUL}hhmmss${RMUL}"  
            "${SMUL}count of events${RMUL}"
      tail -8 $TMPFILE
    } | prSection 9
    sleep 3
done
1

The tput command gives us the terminal-independent character sequence for moving to the upper-left corner of the screen. Rather than call this each time through the loop, we call it once and save the output for reuse on each iteration. This is followed by other calls for special sequences also saved for repeated reuse.

2

There are several ways to create a line of dashes; we chose an interesting, though somewhat cryptic, one here. This two-step process makes use of the fact that the printf will blank-fill the resulting string. The * tells printf to use the first variable for the width of the formatted field. The result is a string of 49 blanks and a single minus sign. It saves the printed string into the variable specified by the -v option. The second part of making the line of dashes is then to substitute each and every space with a minus sign. (The double slash tells bash to replace all occurrences, not just the first.)

3

Declaring the variable i as a local is good practice, though not crucial in our script. Still, it is a good habit to follow. It means that our for loop won’t alter any other index or counter.

4

We add the erase-to-end-of-line to every line that is sent through this function, both here and on the next printf. After printing the dashes, that second printf also prints the erase for the following line, where the cursor will be resting until the next iteration.

5

The cleanup function will be called when the dashboard script exits—which is most likely when the user presses Ctrl-C to interrupt and exit. Like our cleanup function in tailcount.sh from Chapter 8, this function will close down functions that we’ve put in the background.

6

Unlike that previous version, which used kill to send a signal to a specific process, here we use the %1 notation to tell kill to signal any and all processes that resulted from a process we put in the background. They are all considered part of the same “job.” Their job numbers (%1, %2, %3, etc.) are determined by the order in which they are put in the background. In this script, we have only one.

7

We are redirecting the output on the cleanup function so that any and all output coming from stdout or stderr will be thrown away. We’re not expecting any, but this makes sure we won’t get any unexpected text. (It’s not good for debugging, but much cleaner on the screen.)

8

The tempfile command generates a unique name and makes sure it isn’t in use so that we know we have a scratch file available for this script, no matter how many instances of this script are running or what other files might be lying around. There is code in the cleanup function to remove this file when the script exits so as not to leave these lying around after each run.

9

This line starts up two scripts from Chapter 8 that do an ongoing count of lines added to the end of a file. The braces group all the processes of this pipeline of commands together and put them in the “background,” disconnecting them all from keyboard input. These processes, and any they spawn, are all part of job 1 (%1), which is the job that the cleanup function will kill off.

10

Each section of the output is sent separately to the prSection function. The commands for a section don’t have to be grouped inside the braces if a single command is generating the output for that section. That is the case for the first three sections, but the fourth section does need the braces to group the two statements (echo and tail) that write output. The braces on this second section, while not necessary, are there in case we ever want to expand this section and have more or different output. The same could be done for all sections, just as a precaution for future expansion. Note the subtle difference in syntax between this use of the braces and the use in the previous note. We don’t need the semicolon because we put the closing brace on a new line.

Figure 12-3 shows the example output of the dashboard script.

rcso 1203
Figure 12-3. Dashboard script output

Summary

Data and information are useful only if they can be easily digested by the end user. HTML provides an easy way to format data for display to the screen or for printing. Creating dashboards can be particularly useful when you need to monitor information in real time.

In the next chapter, we switch gears and start to explore how the command line and bash can help you perform penetration testing.

Workshop

  1. Modify webdash.sh to take two command-line arguments that specify the log entries to be monitored. For example:

    ./webdash.sh /var/log/apache2/error.log /var/log/apache2/access.log

  2. Write a script similar to Example 12-3 that converts an Apache error log into HTML.

Visit the Cybersecurity Ops website for additional resources and the answers to these questions.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset