11
GENERATING ASSET AND VULNERABILITY REPORTS

Now that you have some asset and vulnerability data to work with, you’ll use that data to generate reports about the asset information for each device in your database. When your boss asks, “How many Linux servers do we have?” or “How many of our desktops are susceptible to this new zero-day vulnerability I heard about on the news this morning?” you can use these reports to provide a confident answer. Before diving into this chapter, be sure to work through the database maintenance steps in Chapter 10.

Asset Reports

An asset report is an overview of all the different systems in your environment. It includes information about which OS and services each is running and how many vulnerabilities each has. Asset reports are useful when you want an overview of your organization’s vulnerability environment on a host-by-host basis. Questions you can answer with this report include:

• How many hosts are in my environment?
• How many Linux servers do I have?
• How many vulnerabilities are on my production servers?
• Which of my workstations is most in need of patching?

Planning Your Report

When planning your reports, take the time to determine the information your report should contain. You already have an extensive amount of data, including a list of hosts with OS information, open ports, services, and vulnerabilities. You could just dump all of this data into an enormous spreadsheet, but then you’d have to sift through that data. Instead, we’ll make a CSV file—a smaller and more readable spreadsheet—with only the most important data.

You can use Microsoft Excel or any other spreadsheet program to view and sort the data in a CSV file, create more detailed reports, or do further data analysis. For example, you might create a pivot table showing vulnerability counts per OS or summarize the asset list by OS or by open ports.

The script we’ll use will collect the following information about each host in the database:

• IP address
• Hostname (if available)
• OS
• Open ports (TCP and UDP)
• Services detected
• Number of vulnerabilities found
• List of vulnerabilities by CVE

Table 11-1 shows examples of output for each column, with open ports, detected services, and the list of CVEs collapsed into lists of ­semicolon-separated values.

Collapsing fields with multiple values preserves the data in a searchable format at the cost of some readability. But it’s a necessary compromise to represent all the data in a single CSV-formatted record.

Table 11-1: Asset Report Output Format

Getting the Data

Because we primarily want to focus on hosts, our first task will be to find a list of hosts. In this Mongo database, you’re using IP addresses to uniquely identify each host, so you’re guaranteed to have only one unique document per IP address.

To get a list of distinct IP addresses in the hosts collection, enter this command in the Mongo shell:

> db.hosts.distinct("ip")

Once you have that list, run the query in Listing 11-1, which uses find to get detailed information about a host with a given IP.

➊ > db.hosts.find({ip:"10.0.1.18"})
  {
    "_id": ObjectId("57d734bf1d41c8b71daaee13"),
    "mac": {
      "vendor": "Raspberry Pi Foundation",
      "addr": "B8:27:EB:59:A8:E1"
  },
  "ip": "10.0.1.18",
➋ "ports": [
    {
       "state": "open",
    "port": "22",
    "proto": "tcp",
    "service": "ssh"
    },
    {
      "state": "open",
      "port": "80",
      "proto": "tcp",
      "service": "http"
    },
--snip--
  "updated": ISODate("2020-01-05T02:19:11.974Z"),
➌ "hostnames": [],
  "os": [
    {
      "cpe": [
        "cpe:/o:linux:linux_kernel:3",
        "cpe:/o:linux:linux_kernel:4"
      ],
   ➍ "osname": "Linux 3.2 - 4.0",
      "accuracy": "100"
    }
  ],
  "oids": [
    {
      "proto": "tcp",
    ➎ "oid": "1.3.6.1.4.1.25623.1.0.80091",
      "port": "general"
    }
  ]

Listing 11-1: Mongo output for one ip, reformatted for clarity

The output of the db.hosts.find query ➊ gives you open ports ➋ with port numbers, protocols proto, and service names; a list of detected hostnames (if any) ➌; the detected OS name (if any) ➍; and the OIDs of any vulnerabilities the scanner found on this host ➎.

You can look up each oid from the host document to get the associated CVE or CVEs using the script in Listing 11-2.

> db.vulnerabilities.find({oid:"1.3.6.1.4.1.25623.1.0.80091"})
{
  "_id": ObjectId("57fc2c891d41c805cf22111f"),
  "oid": "1.3.6.1.4.1.25623.1.0.80091",
  "summary": "The remote host implements TCP timestamps and therefore allows
  to compute
the uptime.",
  "cvss": 2.6,
  "vuldetect": "Special IP packets are forged and sent with a little delay in
  between to the
target IP. The responses are searched for a timestamps. If
  found, the
timestamps are reported.",
  "bid": "NOBID",
  "affected": "TCP/IPv4 implementations that implement RFC1323.",
  "threat": "Low",
  "description": "It was detected that the host implements RFC1323.

The
  following timestamps were retrieved with a delay of 1 seconds in-between:
Paket 1: 1
  nPaket 2: 1",
  "proto": "tcp",
  "insight": "The remote host implements TCP timestamps, as defined by
  RFC1323.",
--snip--
  "impact": "A side effect of this feature is that the uptime of the remote
  nhost can sometimes be computed.",
➊ "cve": [
     "NOCVE"
  ],
  "name": "TCP timestamps",
  "updated": ISODate("2020-10-11T00:04:25.601Z"),
  "cvss_base_vector": "AV:N/AC:H/Au:N/C:P/I:N/A:N"
}

Listing 11-2: Excerpted Mongo output for one oid, reformatted for clarity

Right now, we’re only interested in the CVE or CVEs associated with this vulnerability ➊. But the output from Listing 11-2 provides a good deal of additional information that you can mine later. These general queries will obtain more information than you need, which is why we’ll use the script to parse out the information we want to focus on.

There is one more wrinkle: as you saw in “Getting OpenVAS into the Database” on page 91, each OID might be associated with more than one CVE. Do you treat each associated CVE as a separate vulnerability in your vulnerability totals? Or, do you go by the OID count, which might be different from the count of specific CVEs? The script in the next section uses the OID count because it better captures the results the scanner returned. In general, multiple CVEs associated with a single OID are very similar. The alternative is to base the count on the total number of NOCVE or CVE-XXX-XXXX results discovered. This approach makes sense if you’re only interested in unique results that are severe enough to have their own CVE identifier (not the numerous low-severity results OpenVAS will also return).

Script Listing

Listing 11-3 uses the queries we built in the previous section to generate the asset report.

  #!/usr/bin/env python3
  from pymongo import MongoClient
➊ import datetime, sys, csv
  client = MongoClient('mongodb://localhost:27017')
  db = client['vulnmgt']
  outputFile = "asset-report.csv"
➋ header = ['IP Address', 'Hostname', 'OS', 'Open TCP Ports', 
  'Open UDP Ports', 'Detected Services', 'Vulnerabilities Found', 
  'List of CVEs']
  def main():
      with open(outputFile, 'w') as csvfile:
          linewriter = csv.writer(csvfile)
       ➌ linewriter.writerow(header)
          iplist = db.hosts.distinct("ip")
       ➍ for ip in iplist:
              details = db.hosts.find_one({'ip':ip})
              openTCPPorts = ""
              openUDPPorts = ""
              detectedServices = ""
              serviceList = []
            ➎ for portService in details['ports']:
                   if portService['proto'] == "tcp":
                       openTCPPorts += portService['port'] + "; "
                   elif portService['proto'] == "udp":
                       openUDPPorts += portService['port'] + "; "
                   serviceList.append(portService['service'])
            ➏ serviceList = set(serviceList)
               for service in serviceList:
                   detectedServices += service + "; "
               cveList = ""
            ➐ if 'oids' in details.keys():
                  vulnCount = len(details['oids'])
                  for oidItem in details['oids']:
                      oidCves = db.vulnerabilities.find_one({'oid': 
                      oidItem['oid']})['cve']
                      for cve in oidCves:
                          cveList += cve + "; "
              else:
                  vulnCount = 0
            ➑ if details['os'] != []:
                  os = details['os'][0]['osname']
              else:
                  os = "Unknown"
              if details['hostnames'] != []:
                  hostname = details['hostnames'][0]
              else:
                  hostname = ""
            ➒ record = [ details['ip'], hostname, os, openTCPPorts, 
              openUDPPorts, detectedServices, vulnCount, cveList]
              linewriter.writerow(record)
  ➓ csvfile.close()
main()

Listing 11-3: Script listing for asset-report.py

The script has two primary parts: the headers and declarations, and the main loop inside main(). To output to a CSV file, import the Python csv library ➊. The overall format of your CSV file is set with the header array ➋, which will be the first line to write to the output file ➌.

The main loop in the script ➍ goes through a list of unique IP addresses in the Mongo database and retrieves each IP’s associated document. From the ports structure ➎, we collect all open ports (TCP and UDP) and service names, deduplicate the list of service names (because multiple ports might report the same service name) by casting it to a set ➏, and then write the ports and names to semicolon-separated strings. To get the vulnerability count and list, we check the host details for the oids key, count the OIDs found, and then query the vulnerabilities collection to get the corresponding CVE identifiers ➐. Next, we collect the OS name or, if none is included, report the detected OS as Unknown ➑. At the end, we collate all this information into a single line of the output CSV ➒ and write it to the output file. Then we move on to the next IP in the list. When the loop finishes with all the IP addresses, we close the CSV file, which writes the output to disk ➓.

Customize It

You can output the asset report as a nicely formatted HTML, PDF, or Word document. Modules for all three exist for Python. But reports in those formats aren’t easily modified and sorted in another program the way that CSVs are. So you might sort the assets by IP, OS, or number of vulnerabilities in the script before writing the information to a file.

You can collect different details about your assets, depending on your needs, or manipulate certain fields further. For example, the detected OS names are often too granular (or not granular enough) to be useful for aggregation. So you can create a combined OS type field like “Windows” or “Linux” using string matching or regular expressions on the osname field to categorize each host under a more general OS type.

If you want to report on a subset of hosts, you can add logic either in the MongoDB query or in the Python script to select a subset of returned records. See Chapter 13 for more information on how to do this.

Vulnerability Reports

A vulnerability report is an overview of the specific vulnerabilities in your environment. This report is useful for addressing urgent vulnerabilities. For instance, if you can show that a particular vulnerability is widespread in your organization, you can build a case to do an emergency patch.

To generate the report, you’ll write a script that looks for relevant data in your Mongo database and outputs it to a CSV file that is ready for further analysis in a spreadsheet program.

Planning Your Report

As with the previous script, you should first decide what you want to know about the vulnerabilities in your environment. Many fields in the vulnerability database (vulnmgt) that you imported from the scan results and in cvedb (this database is created by cve-search and is the source of most vulnerability details) aren’t presently relevant. Which do we want to focus on? Items in the following list should be plenty for now:

• CVE ID
• Title (from cvedb)
• Description (from cvedb)
• CVSS score (from cvedb)
• CVSS details (from cvedb)
• Number of hosts with this vulnerability
• List of hosts with this vulnerability, by IP

The “number of hosts” field plus the CVSS field will let you prioritize vulnerabilities by sorting the results in a spreadsheet program.

Table 11-2 shows examples of output for each column, again with multiple ­values for one field collapsed as a semicolon-separated list.

Table 11-2: Vulnerability Report Output Format

Getting the Data

The vulnerability information is stored on a per-host basis and referenced by the OpenVAS OID. So you must first collect a list of OIDs from each host in your gathered scan results, cross-reference your vulnerabilities collection to determine the CVE (or lack thereof) for each vulnerability, and then get CVE details from the cvedb collection. In pseudocode, Listing 11-4 shows how the logic works.

For each host in hosts:
    Get all OIDs
    For each OID:
        Get CVE
        Associate CVE with host (CVE => (list of affected hosts) map)
For each CVE:
    Get cvedb fields
    Get and count hosts that are associated with this CVE
    Build output CSV line

Listing 11-4: Pseudocode for finding relevant CVEs

Building reverse correlations (CVE to host) on the fly avoids the overhead of a separate database collection that just has host and CVE pairs.

To handle the fact that some OIDs might have more than one CVE, you can separate OIDs that contain multiple CVEs into separate CSV lines. Or, you can choose only the first associated CVE, ignoring any remaining CVEs on the grounds that they’re likely to be almost identical. The script in the following section separates OIDs into individual CVEs.

Script Listing

Listing 11-5 contains the code we’ll use to generate a vulnerability report.

#!/usr/bin/env python3
from pymongo import MongoClient
import datetime, sys, csv
client = MongoClient('mongodb://localhost:27017')
db = client['vulnmgt']
cvedb = client['cvedb']
outputFile = "vuln-report.csv"
header = ['CVE ID', 'Description', 'CVSS', 'Confidentiality Impact', 
'Integrity Impact', 'Availability Impact', 'Access Vector', 
'Access Complexity', 'Authentication Required', 'Hosts Affected', 
'List of Hosts']
def main():
       with open(outputFile, 'w') as csvfile:
        linewriter = csv.writer(csvfile)
          linewriter.writerow(header)
            hostCveMap = {}
          hostList = db.hosts.find({'oids': {'$exists' : 'true'}})
     ➊ for host in hostList:
               ip = host['ip']
         ➋ for oidItem in host['oids']:
                cveList = db.vulnerabilities.find_one({'oid': 
                oidItem['oid']})['cve']
                 for cve in cveList:
                        if cve == "NOCVE":
                        continue
                     ➌ if cve in hostCveMap.keys():
                         if ip not in hostCveMap[cve]:
                            hostCveMap[cve].append(ip)
                      else:
                        hostCveMap[cve] = [ ip ]
         ➍ for cve in hostCveMap.keys():
               cvedetails = cvedb.cves.find_one({'id': cve})
              affectedHosts = len(hostCveMap[cve])
            listOfHosts = ""
            for host in hostCveMap[cve]:
                listOfHosts += host + "; "
                      if (cvedetails): 
                if "impact" not in cvedetails:
                    cvedetails["impact"] = {"availability": None, 
                    "confidentiality": None, "integrity": None }
                if "access" not in cvedetails:
                    cvedetails["access"] = {"authentication": None, 
                    "complexity": None, "vector": None }
                record = [ cve, cvedetails['summary'], cvedetails['cvss'],
                        cvedetails['impact']['confidentiality'], 
                        cvedetails['impact']['integrity'],
                        cvedetails['impact']['availability'], 
                        cvedetails['access']['vector'],
                        cvedetails['access']['complexity'], 
                        cvedetails['access']['authentication'],
                        affectedHosts, listOfHosts ]
            else:
                record = [ cve, "", "", "", "", "", "", "", "", 
                affectedHosts, listOfHosts ]
             ➎ linewriter.writerow(record)
    csvfile.close()
main()

Listing 11-5: Script listing for vuln-report.py

Because this script’s structure is so similar to asset-report.py (Listing 11-3), let’s just look at the tricky parts. There are two main loops: one to go through each host and build the CVE-to-hosts map ➊ and the other to loop through the resulting map and output a line for each relevant CVE ➍.

The first loop goes through each host document to collect a list of OIDs ➋, resolving them to CVE IDs (if any) via the vulnerabilities collection. Then it builds a dictionary of CVE IDs mapped to a list of vulnerable hosts (identified by IP address). For each CVE, we check whether it’s in the ­hostCveMap dictionary ➌. If it is, we then check whether the current IP is already mapped to the relevant CVE; if it’s not, we add it to the list of IPs associated with that key. If the CVE isn’t in our map and thus has no IPs associated with it, we create a new CVE key in the dictionary and create a list containing the current IP to use as the value. Once this map is complete, a second loop ➍ collects details from the cvedb database for each CVE key in ­hostCveMap. The vulnerability details, including the list of affected hosts collapsed into a single semicolon-separated list, are then written to a single CSV output line ➎.

Customize It

The method of vulnerability analysis I used in Listing 11-5 requires that you load the entire collection of hosts with vulnerabilities into RAM, which might be infeasible for large datasets of thousands of hosts with multiple vulnerabilities per host. To speed up the script, you can prebuild a collection with a 1:1 relationship between hosts and vulnerabilities and then query it for all hosts affected by a given CVE. You can do this either with a dedicated script or by modifying the openvas-insert.py script (Listing 9-8) to build this collection while you’re parsing the OpenVAS output file. This spares your computer from having to load the entire collection of hosts with vulnerabilities into RAM. But you’ll need to add some additional code to your other scripts, delete stale data, and ensure that relevant indexes are created correctly (“Defining Database Indexes” on page 98). Because a separate document collection will provide this mapping, you’ll need to update your other scripts that insert and delete data to make them aware of this mapping.

As mentioned before, this database uses only CVSSv2 scores, because neither OpenVAS nor cve-search provides CVSSv3 scores. If CVSSv3 scores are important in your environment, use other data sources to fill in that gap.

In this script, I ignore all OpenVAS results that report a CVE of NOCVE. Generally, they’re low-severity issues. If you want to include these in the report, you’ll have to pull most of the fields for these results from the OpenVAS data rather than from the CVE database.

Summary

In this chapter, you built your first reports using the data in your vulnerability database. Because the two most important aspects of this database are the hosts (assets) and the vulnerabilities your scans discovered, it’s only natural to report according to these two parameters.

In Chapter 12, we’ll take a short side trip to fully automate the vulnerability scanning program. Then, in Chapter 13, you’ll learn how to produce more complex reports on the data your scans collect.