8
USING THE DATA COLLECTION TOOLS
The goal for your vulnerability management system is to have usable vulnerability data in a database to make it easy to search, analyze, and generate automated reports. You’ve installed all the basic tools, but your database is empty.
In this chapter, you’ll look at the tools you’ll use to collect the raw data for your vulnerability management system: they include Nmap, OpenVAS, and cve-search. To gain some basic familiarity with each tool, you’ll run each manually to explore its configuration options and see what kind of data you can collect with it. If you’re already familiar with these tools, you can move on to the next chapter. There you’ll collect the information you need and store it in the database.
An Introduction to the Tools
Although the information you can gather with the Nmap, OpenVAS, and cve-search tools overlaps, the purpose of each tool is very different. Before we get into the nitty-gritty of command line options and XML outputs, let’s situate the three of them in the overall program.
Nmap
The Nmap network discovery scanning tool was originally developed in 1997 by Gordon Lyon, a programmer known in the security community as Fyodor. In its more than 20-year life, Nmap has had regular development and improvement, and it remains a central tool in the security specialist’s tool belt.
How Does It Work?
The Nmap tool sends various packets to an IP address or a range of IP addresses to gather networking-related information about the hosts at those addresses. You can configure the specific packets it sends and which ports it sends them to: Nmap can be slow and stealthy or very fast and aggressive, depending on how you configure it.
What Is It Good For?
You can use Nmap to do quick scans of network ranges to discover information about the live hosts: their addresses, which services they’re providing on which network ports, and which OS they’re running.
What Is It Not Good For?
Although Nmap will tell you which ports are open on a host, it isn’t a vulnerability scanner. In some cases, Nmap can determine the specific version of a server running on a port, but it won’t match that information with known vulnerabilities or perform additional testing to determine whether certain vulnerabilities exist. The Nmap tool isn’t sufficient for determining how at-risk any of the hosts it finds might be. Instead, it’s best to use the information gleaned from Nmap and as the basis for further analysis, either on its own or with additional data from vulnerability scanners.
OpenVAS
OpenVAS is a vulnerability scanner. Its roots lie in an earlier scanner, Nessus, first released in 1998. In 2005, Tenable Network Security, founded by the Nessus core developers, converted Nessus into a commercial product. In response, a number of open source developers forked the Nessus codebase to continue providing a free and open source scanner project. The current result is OpenVAS, which only barely resembles the modern incarnation of the commercial Nessus product. OpenVAS is divided into a number of separate components: the scanner, a management and scheduling daemon, and the Greenbone Security Assistant, which is a web-based frontend to make configuring and running scans easy.
How Does It Work?
Like Nmap, OpenVAS sends a series of network packets to one or more IP addresses. But unlike Nmap, OpenVAS sends targeted packets to determine specific versions of services and whether or not they’re vulnerable to known attacks. It also contains numerous plug-ins that do more aggressive testing for specific vulnerabilities beyond simple version checking. (For more details about the capabilities of vulnerability scanners, see Chapter 3.)
What Is It Good For?
OpenVAS is ideal for finding specific vulnerabilities on hosts in the targeted network. But, like any network vulnerability scanner, it will only discover network vulnerabilities, not those that can only be exploited from the local system. Its internal database includes significant background detail about the vulnerabilities it can find, including CVSS scores, exploitation consequences, and a comprehensive list of external references containing more vulnerability details. You can also use OpenVAS for host discovery.
What Is It Less Good For?
Although OpenVAS provides host discovery, its OS fingerprinting functionality is less complete than Nmap’s, so it might not identify the specific OS the target is running. By running the two in concert, you’ll get a more accurate view of your network environment.
Differences Between OpenVAS and Commercial Scanners
To be frank, the OpenVAS scanner is free and it shows. Its interfaces, both command line and web based, are functional and complete, but their usability leaves a lot to be desired. Without a commercially supported budget to develop new vulnerability tests, development of OpenVAS is largely left to the open source community. Older vulnerabilities are well supported, but new vulnerability coverage is significantly less comprehensive. However, in an environment where there is no budget for a vulnerability management program, this tool provides significant value. That said, this is the first component you should replace if you have a budget of $1,000 or more and can purchase a commercial scanning product, such as Nessus or Qualys.
cve-search
Once you have asset and vulnerability information from Nmap and OpenVAS, the next step is to bring in additional data sources. The cve-search tool suite will give you a comprehensive local repository of CVE data.
CVE/NVD
The CVE list, provided by the Mitre Corporation, is the authoritative source of CVE information. Its content is synchronized to the National Vulnerability Database (NVD) which is run by NIST; NVD adds additional vulnerability information, such as vulnerability rating, patch information, and more searching options. If you’re looking for information on vulnerabilities, check these two lists first. They provide online search functionality, vulnerability feeds, and the option to mirror their databases locally. But if you want to programmatically grab and parse CVE information, the cve-search tools will save you a lot of time.
cve-search
The cve-search suite is a collection of Python scripts that gathers vulnerability information from several online sources (primarily the CVE/NVD repositories). It then puts that data into a Mongo database for easy querying and analysis. This suite is a gold mine for those interested in doing automated vulnerability data analysis. We’ll use the cve-search suite to build and maintain our local CVE database. Then we’ll query that database directly for analysis instead of using the cve-search frontend tools. But keep in mind that those tools are useful for manual vulnerability searching.
Getting Started with Nmap Scanning
The goal of this exercise is to gain a basic familiarity with Nmap scans. By using this tool, you can determine:
- The number of hosts in a given network segment
- The MAC address of each host, which you can leverage to figure out the underlying hardware
- Which ports are open on each host and which services are running on those ports
- Which OS might be running on the host
For all scripts that involve live scanning, you need a network range or series of hosts that you have permission to scan. Scanning with Nmap, although generally not intrusive, can cause problems with the scanned systems. The risks range from temporary slowness due to resource exhaustion to more serious issues that might require a system reboot or other manual intervention. Penetration testers regularly run into issues scanning networked printers: the probes may cause the printers to output page after page of garbage output, wasting reams of paper if the testers haven’t excluded the printers or removed the paper in advance. Only scan systems you own or have permission to scan
Although you’ll look at a few scanning and output options, you’ll only scratch the surface of Nmap’s capabilities. I highly recommend reading the Nmap manual (run man nmap at the command prompt) to learn about all its features.
Running a Basic Scan
The best way to understand how Nmap works and what kind of information you can gather is to do a basic scan of your network range. I used the 10.0.1.0/24 range, which is my local test network. Running Nmap without any arguments except for the scan range makes it do a basic scan:
# nmap 10.0.1.0/24
This command scans the 1,000 most common ports on each address in the target network range. The output will look something like Listing 8-1.
NOTE
Although an unprivileged user can run this scan, you can gather more information when you run it as root. For example, the MAC address output in Listing 8-1 won’t be available if you run an unprivileged scan.
--snip-- ➊ Nmap scan report for 10.0.1.4 ➋ Host is up (0.0051s latency). Not shown: 997 filtered ports ➌ PORT STATE SERVICE 22/tcp open ssh 88/tcp open kerberos-sec 5900/tcp open vnc ➍ MAC Address: B8:E8:56:15:68:20 (Apple) Nmap scan report for 10.0.1.5 Host is up (0.0032s latency). Not shown: 996 filtered ports PORT STATE SERVICE 135/tcp open msrpc 139/tcp open netbios-ssn 445/tcp open microsoft-ds 5357/tcp open wsdapi MAC Address: 70:85:C2:4A:A9:90 (ASRock Incorporation) --snip-- ➎ Nmap done: 256 IP addresses (7 hosts up) scanned in 663.38 seconds --snip--
Listing 8-1: Default nmap output when run as the root user
The output is separated by host ➊. For each host, you’ll see whether the host is up and how long it takes to contact it ➋, the open ports and the service running on each port ➌, and the MAC (network hardware) address ➍ if available. If you run Nmap as an unprivileged user or if the targeted device is on another subnet, the MAC information won’t be available. Additionally, there’s a summary of scanned hosts and statistics for how long it took Nmap to complete its scan ➎.
Even just using the default options, you can learn a good deal about the systems you scan. However, Nmap can provide even more data.
Using Nmap Flags
You can use the nmap command’s flags to learn more about your network or adjust your scans’ output format.
Getting More Information with -v
The -v flag adds verbosity to the scan; this means that Nmap becomes chattier about what it’s doing:
# nmap -v 10.0.1.0/24
You can add up to three v flags in a row (or -v3) to get more information that you can use to monitor the scan’s progress. For example, adding -vvv to the same scan will result in Listing 8-2.
# nmap -vvv 10.0.1.0/24 Starting Nmap 7.01 ( https://nmap.org ) at 2020-03-04 10:42 PST Initiating ARP Ping Scan at 10:42 Scanning 255 hosts [1 port/host] adjust_timeouts2: packet supposedly had rtt of -137778 microseconds. Ignoring time. adjust_timeouts2: packet supposedly had rtt of -132660 microseconds. Ignoring time. adjust_timeouts2: packet supposedly had rtt of -54309 microseconds. Ignoring time. adjust_timeouts2: packet supposedly had rtt of -59003 microseconds. Ignoring time. adjust_timeouts2: packet supposedly had rtt of -59050 microseconds. Ignoring time. Completed ARP Ping Scan at 10:43, 3.26s elapsed (255 total hosts) Initiating Parallel DNS resolution of 255 hosts. at 10:43 Completed Parallel DNS resolution of 255 hosts. at 10:43, 0.02s elapsed DNS resolution of 16 IPs took 0.02s. Mode: Async [#: 1, OK: 11, NX: 5, DR: 0, SF: 0, TR: 16, CN: 0] Nmap scan report for 10.0.1.0 [host down, received no-response] Nmap scan report for 10.0.1.2 [host down, received no-response] Nmap scan report for 10.0.1.3 [host down, received no-response] Nmap scan report for 10.0.1.4 [host down, received no-response] --snip-- SYN Stealth Scan Timing: About 10.98% done; ETC: 13:37 (0:04:11 remaining) Increasing send delay for 10.0.1.7 from 40 to 80 due to 11 out of 27 dropped probes since last increase. Increasing send delay for 10.0.1.18 from 40 to 80 due to 11 out of 23 dropped probes since last increase. --snip--
Listing 8-2: Verbose nmap output showing debugging and progress-related information
Under normal circumstances, this resulting information might not matter to you. But if a scan isn’t succeeding, more verbose output can help diagnose any issues.
Getting the OS Fingerprint with -O
Another very useful flag is -O, which instructs Nmap to look at the network traffic’s OS fingerprint to determine which OS is running on the systems being scanned:
# nmap -O 10.0.1.5
Just as a human has a unique set of fingerprints, an operating system has characteristics that, taken in combination, are unique to the OS and often point to a specific version or even an OS patch level. But this information isn’t guaranteed to be correct. For instance, custom network stacks can throw off OS detection by changing the fingerprint, and even marginally skilled coders could deliberately shape the network traffic from a system to make it look like it has a different OS. Regardless, OS fingerprinting is a highly useful data point in your asset database.
Listing 8-3 shows example output of the –O flag.
Nmap scan report for 10.0.1.5 Host is up (0.0035s latency). Not shown: 996 filtered ports PORT STATE SERVICE 135/tcp open msrpc 139/tcp open netbios-ssn 445/tcp open microsoft-ds 5357/tcp open wsdapi MAC Address: 70:85:C2:4A:A9:90 (ASRock Incorporation) ➊ Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port Device type: general pupose|phone|specialized ➋ Running (JUST GUESSING): Microsoft Windows Vista|2008|7|Phone|2012 (93%), FreeBSD 6.X (86%) OS CPE: cpe:/o:microsoft:windows_vista::- cpe:/o:microsoft:windows_vista::sp1 cpe:/o:microsoft:windows_server_2008::sp1 cpe:/o:microsoft:windows_7 cpe:/o:microsoft:windows cpe:/o:microsoft:windows_8 cpe:/o:microsoft:windows_server_2012 cpe:/o:freebsd:freebsd:6.2 ➌ Aggressive OS guesses: Microsoft Windows Vista SP0 or SP1, Windows Server 2008 SP1, or Windows 7 (93%), Microsoft Windows Vista SP2, Windows 7 SP1, or Windows Server 2008 (93%), Microsoft Windows Phone 7.5 or 8.0 (92%), Windows Server 2008 R2 (92%), Microsoft Windows 7 Professional or Windows 8 (92%), Microsoft Windows Embedded Standard 7 (91%), Microsoft Windows Server 2008 SP1 (91%), Microsoft Windows Server 2008 R2 (90%), Microsoft Windows 7 (89%), Microsoft Windows 8 Enterprise (89%) No exact OS matches for host (test conditions non-ideal). Network Distance: 1 hop
Listing 8-3: nmap output with OS fingerprinting
If Nmap is uncertain about its fingerprinting, it will clearly report that ➊. But Nmap tells you its best guesses for the running OS ➋, and it provides common platform enumeration (CPE; a standardized reference to specific OS and software packages) references for those guesses. It will even tell you it’s making a more aggressive guess ➌ where it does its best to figure out exactly which version of Windows is on that host. In this case, this system is running Windows 10.
Making Nmap “Aggressive”
The “aggressive” flag -A combines the OS fingerprinting option with version detection and script scanning (equivalent to the flag combination -O -sV --script=default --traceroute). It provides even more information about the host. This scan can be intrusive, and system owners are likely to consider it hostile if they are the target of an aggressive scan. Listing 8-4 shows example output of the -A flag on the same host (10.0.1.5).
# nmap -A 10.0.1.5 --snip-- Nmap scan report for 10.0.1.5 Host is up (0.0035s latency). Not shown: 996 filtered ports PORT STATE SERVICE VERSION 135/tcp open msrpc Microsoft Windows RPC 139/tcp open netbios-ssn Microsoft Windows 98 netbios-ssn 445/tcp open microsoft-ds Microsoft Windows 7 or 10 microsoft-ds 5357/tcp open http Microsoft HTTPAPI httpd 2.0 (SSDP/UPnP) ➊ |_http-server-header: Microsoft-HTTPAPI/2.0 |_http-title: Service Unavailable MAC Address: 70:85:C2:4A:A9:90 (Unknown) —-snip-- Host script results: |_nbstat: NetBIOS name: GAMING-PC, NetBIOS user: <unknown>, NetBIOS MAC: d8:cb:8a:17:99:80 (Micro-star Intl) | smb-os-discovery: ➋ | OS: Windows 10 Home 10586 (Windows 10 Home 6.3) | OS CPE: cpe:/o:microsoft:windows_10::- | NetBIOS computer name: GAMING-PC | Workgroup: WORKGROUP |_ System time: 2020-05-01T16:44:35-04:00 | smb-security-mode: | account_used: guest | authentication_level: user | challenge_response: supported |_ message_signing: disabled (dangerous, but default) |_smbv2-enabled: Server supports SMBv2 protocol TRACEROUTE HOP RTT ADDRESS 1 3.53 ms 10.0.1.5
Listing 8-4: Aggressive scanning nmap output
In this case, the more aggressive scanning (NetBIOS checks) determined the actual version of Windows running on the host ➋. The aggressive scan can also determine additional information about the HTTP server running on port 5357 ➊.
Modifying the Output Format with -o
Of particular importance for this guide is the -o flag, which lets you output in default format (-oN), XML (-oX), or greppable (-oG) text. You can also use the -oS flag to output in the “script kiddie” format, but that is a novelty format and unlikely to be useful to you in this book!
By using the XML flag, you can output your scan results in a format that is easily parseable by XML-aware Python scripts. Let’s scan the same host we did previously but output in XML format using this command:
# nmap -oX output.xml 10.0.1.5
The Nmap tool generates some basic output to the screen. But the actual XML output is in output.xml, so look inside that file as shown in Listing 8-5.
# cat output.xml➊ | xmllint --format -➋ ➌ <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE nmaprun> <?xml-stylesheet href="file:///usr/bin/../share/nmap/nmap.xsl" type="text/ xsl"?> <!-- Nmap 7.01 scan initiated Sat Apr 4 09:26:56 2020 as: nmap -oX output.xml 10.0.1.5 --> ➍ <nmaprun scanner="nmap" args="nmap -oX output.xml 10.0.1.48" start="1523118416" startstr="Sat Apr 4 09:26:56 2020" version="7.01" xmloutputversion="1.04"> ➎ <scaninfo type="syn" protocol="tcp" numservices="1000" services="1,3-4,6-7,9,13,17,19-26, --snip-- 64623,64680,65000,65129,65389"/> <verbose level="0"/> <debugging level="0"/> <host starttime="1523118416" endtime="1523118436"> <status state="up" reason="arp-response" reason_ttl="0"/> ➏ <address addr="10.0.1.48" addrtype="ipv4"/> ➐ <address addr="70:85:C2:4A:A9:90" addrtype="mac"/> ➑ <hostnames> </hostnames> <ports> <extraports state="filtered" count="996"> <extrareasons reason="no-responses" count="996"/> </extraports> ➒ <port protocol="tcp" portid="135"> <state state="open" reason="syn-ack" reason_ttl="128"/> <service name="msrpc" method="table" conf="3"/> </port> --snip-- </ports> <times srtt="1867" rttvar="254" to="100000"/> </host> ➓ <runstats> <finished time="1523118436" timestr="Sat Apr 4 09:27:16 2020" elapsed="20.14" summary="Nmap done at Sat Apr 4 09:27:16 2020; 1 IP address (1 host up) scanned in 20.14 seconds" exit="success"/> <hosts up="1" down="0" total="1"/> </runstats> </nmaprun>
Listing 8-5: An Nmap scan output in XML format
Take a look inside the file using the cat command ➊. The xmllint command ➋ formats the output to indent it properly and make it more readable. (The trailing - in the xmllint command instructs it to take its input from STDIN, allowing us to pipe the output of the previous command into it.)
The first few lines ➌ make up the header, which you don’t need to worry about. Next, you see some basic information about the scan, including the command line that generates this specific output and when it ran ➍. Then there are more scan parameters, including the specific ports that Nmap checked ➎. You see the IP ➏ and MAC ➐ addresses, as well as any hostnames that Nmap detected for this host ➑. In this scan, there are no hostnames, but if there were, the output would look something like this:
<hostnames>
<hostname name="scanme.nmap.org" type="user"/>
<hostname name="scanme.nmap.org" type="PTR"/>
</hostnames>
Next, you see some very interesting output—the details of each open port that Nmap found. TCP port 135 ➒ is open. You know it’s open because you received a SYN-ACK packet after probing it, the time-to-live (TTL) of packets coming from it is 128 hops, and it’s running the Microsoft Remote Procedure Call (MSRPC) protocol. The output also contains the overall run statistics, including the number of hosts found and how long it took to run the scan ➓.
Customize It
I highly recommend experimenting with the Nmap options for your scan to find a set that work best in your environment: that means producing useful information while not overloading the network or causing issues on the systems you’re scanning. Here are a few places to start:
- Different scan types: Instead of the default SYN scan, try others.
- Scan speed: Limit your scan activity to keep your network unclogged, especially over a low-bandwidth connection.
- Different OS-fingerprinting options: Some options are more effective than others, depending on the devices you scan and your overall network configuration.
Zenmap, a graphical frontend for Nmap, is useful if you plan to do a lot of different Nmap scans and want a simpler method than repeatedly creating long shell commands. Zenmap provides output in a browsable format that can make digesting scan results easier.
Getting Started with OpenVAS
In this section, you’ll become comfortable running scans with OpenVAS from the web GUI (Greenbone Security Assistant) and the command line. OpenVAS isn’t a very user-friendly tool, so it’s important you’re familiar with its options before you start running scans and analyzing the resulting data.
This discussion won’t be a comprehensive OpenVAS tutorial. But it will give you enough familiarity with the web and command line interfaces to generate XML scan results you can use in your vulnerability management system.
Running a Basic OpenVAS Scan with the Web GUI
Here you’ll learn how to run a basic OpenVAS scan from the Greenbone web GUI, and in the next section, you’ll run a scan from your command line.
Log into Greenbone at https://localhost:4000/. The default username is admin, and the password is admin, but you’ve changed those credentials already, haven’t you? Because Greenbone uses a self-signed transport layer security (TLS) certificate, your browser will probably warn you that the site isn’t trusted. This is expected; just click through and continue to the login page. When you successfully log in, you should see an empty Greenbone dashboard: you’ll fill it with information as you run scans and discover information about your environment (see Figure 8-1).
Figure 8-1: A Greenbone dashboard after you run a scan
You’ll spend most of your time in the Scans tab where you can run scans and look at the results.
Setting Up Targets and Configurations
You can set up the scan tasks in the Tasks tab, but first you need to set up targets and scan configurations on the Configuration tab.
At the upper left of the Targets page on the Configuration tab is a small star icon; this is Greenbone’s indicator for creating a new item (see Figure 8-2). Click the star icon to create a new target.
Figure 8-2: Creating a new target
Once you click the star, the New Target configuration page will appear (see Figure 8-3).
Figure 8-3: The New Target configuration page
The configuration window contains lots of options, but for now, let’s focus on the basics. Name the target and, if you like, add a description. The most important option is the Hosts field: enter the hosts you want to scan, separating each host with a comma. You can specify hosts using any of the following methods, alone or in combination:
- IP address
- CIDR IP range
- Dash range (for example, 10.0.1.1-10.0.1.3 refers to 10.0.1.1, 10.0.1.2, and 10.0.1.3)
Just below this field is the option to exclude one or more hosts; using the same format as for the Hosts field, exclude any IPs that you don’t want in the scan. For now, we’ll leave the rest of the page at the defaults. Click the Create button.
Under Configuration▸Scan Configs, you’ll see a list of different built-in scan configurations (see Figure 8-4).
You’ll likely want to customize one of these scan types for your own environment. But we’ll stick to using Full and fast for now; it’s a good compromise between speed and thoroughness.
Creating a Task
Now that you have a target set and a scan config to use, you can create the scan task. Return to Scans▸Tasks and click the star icon in the upper left to create a new task. This will open a configuration menu, like the one shown in Figure 8-5.
Much like the target configuration, I recommend you stick to minimal customizations for now, as in Figure 8-5. Name the task and choose the scan target you just created. Scroll down the configuration window, select the Full and fast scan config, and click Create.
You now have a shiny new scanning task. You can prompt OpenVAS to start the task by clicking the play icon on the right of the task list (see Figure 8-6).
The Scans▸Tasks window will list all your tasks and provide some information about each task, such as the severity of the vulnerabilities discovered by the task and the last time the task was completed (see Figure 8-7).
Figure 8-7: The Scans▸Tasks window after you add a few tasks
Now that your scan has started, you can watch its progress in the Scans▸Reports page. Even though the scan has just begun, there’s a good chance there’s already some information there. The scan might take several hours to complete, depending on the number of hosts you’re scanning.
Exporting Your Scan Reports
Once the report status indicates Done, the scan is complete, and you can export the report. We’ll ingest the XML-formatted report into our database in the next chapter, but first we need to generate the report.
The easiest way to do this with Greenbone is to export the report from the Scans▸Reports page. Click the report you want to export (in the Date column), ensure that the drop-down box next to this icon is set to XML or Anonymous XML, and then click the down-pointing arrow icon to download the report (see Figure 8-8).
Figure 8-8: Exporting a report
You can look at the resulting file in a text or XML editor.
Running a Basic Scan from the Command Line
Now you know how to create OpenVAS targets, tasks, and scan reports through the Greenbone web GUI. But to integrate automated scanning into the rest of your vulnerability management program, you’ll need to use the omp command line tool, which was installed with OpenVAS. Then you can schedule your scans with the same scripts you use to ingest the scan outputs into your vulnerability database (see Chapter 9).
To get omp to work, you need to modify the init scripts for openvas-manager and openvas-gsd. The reason is that the packaged version of OpenVAS is configured to listen for commands on a Unix socket (not a network port). Unfortunately, omp can only communicate on a network port, so you must change this configuration. To do so, find the DAEMON_ARGS variable in /etc/init.d/openvas-manager and modify it to read DAEMON_ARGS="-a 127.0.0.1 -p 9390".
Next, in /etc/init.d/openvas-gsa, modify the DAEMON_ARGS line to read DAEMON_ARGS="--mlisten 127.0.0.1 -m 9390".
Then reload the init scripts (systemctl daemon-reload) and restart openvas-manager (service openvas-manager restart) and openvas-gsd (service openvas-gsa restart).
Specify the username and password in ~/omp.config (it’s the same as for the web GUI, so use whatever you’ve changed it to from admin/admin). When you run scheduled jobs, you’ll use the -c command line option to point at this config file, as shown in the following code. This ensures that omp finds the correct configuration file when you run it from cron, whether or not you run it as the scanuser user.
$ omp -c /home/scanuser/omp.config -X "<help></help>"
Now that your OpenVAS command line tool is up and ready to go, let’s test it by setting up some scans.
Testing the Command Line Tool
OpenVAS uses XML for input and output, meaning you’ll send commands as XML blocks to omp using the --xml or -x flag.
Test omp by sending a simple command and verifying that the result looks like Listing 8-6.
$ omp -X "<help></help>" <help_response status_text="OK" status="200"> AUTHENTICATE Authenticate with the manager. COMMANDS Run a list of commands. CREATE_AGENT Create an agent. CREATE_ALERT Create an alert. CREATE_ASSET Create an asset. CREATE_CONFIG Create a config. CREATE_CREDENTIAL Create a credential. --snip-- VERIFY_REPORT_FORMAT Verify a report format. VERIFY_SCANNER Verify a scanner. </help_response>
If you get a result other than status="200" in the first response line, it’s likely that your credentials are incorrect. Verify those, and if you still experience issues, check back to the OpenVAS setup steps in Chapter 7 and ensure you’ve followed the steps in “Updating OpenVAS.”
Creating a Scan Task with omp
Because you’ve already created a scan target and scan configuration via Greenbone, you can use them to initiate scans from the command line. It’s a lot easier to build and test scan targets and configurations using the web GUI and then schedule scans and export reports at the command line, using XML as little as possible.
NOTE
You can find a full XML command reference at http://docs.greenbone.net/API/OMP/omp-7.0.html.
To check our list of available targets and configs, use omp with the –T and -g flags, as in Listing 8-7.
$ omp -T ➊ c8f84568-94ea-4528-b049-56f4029c1368 Target for immediate scan of IP 10.0.1.0/24 7fc8000a-28f7-45ea-bd62-9dec89a1f679 Target for immediate scan of IP 10.0.1.1 a6f26bd5-f1e3-4fd3-88fc-8aa65dd487bc Target for immediate scan of IP 10.0.1.7 206a5a14-ab30-462c-b191-440a30daeb17 Target for immediate scan of IP 10.0.1.8 6539fd3c-871c-43ff-be9c-9768e6bebddd test target $ omp -g ➋ 8715c877-47a0-438d-98a3-27c7a6ab2196 Discovery 085569ce-73ed-11df-83c3-002264764cea empty daba56c8-73ec-11df-a475-002264764cea Full and fast 698f691e-7489-11df-9d8c-002264764cea Full and fast ultimate 708f25c4-7489-11df-8094-002264764cea Full and very deep 74db13d6-7489-11df-91b9-002264764cea Full and very deep ultimate 2d3f051c-55ba-11e3-bf43-406186ea4fc5 Host Discovery bbca7412-a950-11e3-9109-406186ea4fc5 System Discovery
Listing 8-7: Getting target and scan config IDs
The omp –T flag returns a list of configured scan targets ➊ with universally unique identifiers (UUIDs). The omp -g flag returns a list of configured scan types ➋, again with matching UUIDs. With these target and config IDs, create a scan task with the -C flag, as shown in Listing 8-8.
$ omp -C --target=c8f84568-94ea-4528-b049-56f4029c1368 --config=698f691e-7489- 11df-9d8c-002264764cea --name="Scan Task" ➊ dd3617ce-868f-457a-a2f8-bfb7bdb1b8ff
Listing 8-8: Creating a task and getting its ID
You can also do this with an XML command, but the -C format is much simpler! The response you’ll get is a UUID that refers to an OpenVAS task. Think of the task as a combination of a scan configuration and a target; you can kick off that task on demand by referring to its UUID ➊, as shown in Listing 8-9.
$ omp --start-task dd3617ce-868f-457a-a2f8-bfb7bdb1b8ff
<start_task_response status_text="OK, request submitted" status="202"><report
_id>4dc106f1-cf3a-47a1-8a71-06b25d8d2c70</report_id></start_task_response>
Listing 8-9: Starting the task
When the task is complete, you’ll need the report_id to look at the results. The easiest way to tell if the scan is running is to look at the Tasks page in Greenbone. You should see a task in progress in the table at the bottom of the page (see Figure 8-9).
Figure 8-9: A running scan task
You can also check on the scan task’s progress by using the --get_tasks command, which will give you an exhaustive rundown of the scan’s current status. Listing 8-10 shows a (truncated!) example.
$ omp --get-tasks dd3617ce-868f-457a-a2f8-bfb7bdb1b8ff|xmllint --format - <?xml version="1.0"?> <get_tasks_response status_text="OK" status="200"> <apply_overrides>0</apply_overrides> <task id="dd3617ce-868f-457a-a2f8-bfb7bdb1b8ff"> --snip-- ➊ <status>Running</status> --snip-- </get_tasks_response>
Listing 8-10: omp get_tasks output
When the status XML value ➊ changes from Running to Done, the scan is finished. As in Listing 8-5, xmllint formats the output XML into a more readable layout.
Once the scan is complete, you can get the report in XML format using omp --get-report, as shown in Listing 8-11.
$ omp --get-report 4dc106f1-cf3a-47a1-8a71-06b25d8d2c70|xmllint --format - > output.xml
Listing 8-11: Exporting the report
In Listing 8-11, we redirect the output to a file using >. You can run this command at any time during the scan, but the report will be incomplete. There’s a field in the output XML file called scan_run_status; you’ll know you have a full report when the value of this field changes from Running to Done. We’ll look at this XML output format in more detail in Chapter 9.
Customize It
If there’s more than one security analyst in your organization, you can set up additional accounts or integrate with LDAP/RADIUS authentication in the Greenbone web interface to allow others to set up scan targets, run scans, and look at results.
You can do more in-depth scans by configuring SSH and other credentials: this permits OpenVAS to log into some services and conduct deeper scans. You can save these credentials most easily via the Greenbone web GUI under the Configuration tab.
Getting Started with cve-search
Although we’ll use cve-search primarily for its comprehensive database of CVE information, it’s very useful on its own for finding detailed information about any vulnerability with a CVE ID.
Searching for CVE IDs
We’ll use search.py, which is in cve-search/bin/ (see the section “Setting Up cve-search” in Chapter 7), to search for CVEs that affect a product (identified by its CPE) and search for a specific CVE.
In Listing 8-12, we use the -p (for product) flag to find all CVEs that affect Windows 10.
$ ./search.py -p o:microsoft:windows_10
Listing 8-12: Searching for CVEs that affect Windows 10
This command outputs a huge result set, including full details of all CVEs found. In Listing 8-13, I’ve restricted the results returned to CVE ID by passing the value cveid to the –o flag.
$ ./search.py -p o:microsoft:windows_10 -o cveid CVE-2015-6184 CVE-2015-6051 CVE-2015-6048 CVE-2015-6042 CVE-2016-1002 CVE-2016-1005 CVE-2016-1001 CVE-2016-1000 CVE-2016-0999 CVE-2016-0998 CVE-2016-0997 CVE-2016-0996 --snip--
Listing 8-13: CVE IDs that affect Windows 10
The vulnerability management system you’re building will handle tasks like this entirely with the database. But the commands like the one in Listing 8-12 can be useful for filtering a list of vulnerabilities to just those that affect the operating systems your organization has deployed.
Finding Out More About a CVE
To find out more about a specific CVE, specify a CVE ID using the –c flag. Searching for CVE-2016-0996 from Listing 8-13 gives us the results in Listing 8-14.
$ ./search.py -c CVE-2016-0996 -o json | python -m json.tool➊ { "Modified": "2016-03-16T13:53:26.727-04:00", "Published": "2016-03-12T10:59:16.853-05:00", ➋ "access": { "authentication": "NONE", "complexity": "MEDIUM", "vector": "NETWORK" }, ➌ "cvss": 9.3, "cvss-time": "2016-03-16T09:46:38.087-04:00", "id": "CVE-2016-0996", ➍ "impact": { "availability": "COMPLETE", "confidentiality": "COMPLETE", "integrity": "COMPLETE" }, ➎ "references": [ "https://helpx.adobe.com/security/products/flash-player/apsb16-08 .html", "http://www.zerodayinitiative.com/advisories/ZDI-16-193/" ], ➏ "summary": "Use-after-free vulnerability in the setInterval method in Adobe Flash Player before 18.0.0.333 and 19.x through 21.x before 21.0.0.182 on Windows and OS X and before 11.2.202.577 on Linux, Adobe AIR before 21.0.0.176, Adobe AIR SDK before 21.0.0.176, and Adobe AIR SDK & Compiler before 21.0.0.176 allows attackers to execute arbitrary code via crafted arguments, a different vulnerability than CVE-2016-0987, CVE-2016-0988, CVE-2016-0990, CVE-2016-0991, CVE-2016-0994, CVE-2016-0995, CVE-2016-0997, CVE-2016-0998, CVE-2016-0999, and CVE-2016-1000.", ➐ "vulnerable_configuration": [ "cpe:2.3:a:adobe:flash_player_esr:18.0.0.329", "cpe:2.3:o:microsoft:windows", "cpe:2.3:o:apple:mac_os_x", --snip--
Listing 8-14: Details of CVE-2016-0996
The output contains the access vectors for this vulnerability ➋ and all the relevant CVSS and common weakness enumeration (CWE) information. (Note that, like OpenVAS, cve-search doesn’t provide CVSSv3 scores, but only a CVSSv2 score ➌ and details, via its access ➋ and impact ➍ sections.) Another section lists all external references ➎ for this vulnerability: they include a US-CERT alert, the Microsoft patch details, and a number of third-party notifications as well. There’s a human-readable summary of the vulnerability ➏ and CPE information for all the known systems (OS or application) that are subject to this vulnerability ➐.
By using the -o json flag, the command returns a block of JSON without newlines or indentation, so you’ll need to pipe the output through the Python tool json.tool ➊ for readable formatting.
Text Searching the CVE Database
We can also use the -f flag to do an arbitrary text search across the CVE database summary fields. Listing 8-15 shows the first (of many) results of a search for “buffer overflow.”
$ ./search.py -f "buffer overflow" { "Modified": "2008-09-05T16:40:25.38-04:00", "Published": "2005-01-10T00:00:00.000-05:00", "_id": { "$oid": "5706df571d41c81f2d58a882" }, "access": { "authentication": "NONE", "complexity": "HIGH", "vector": "NETWORK" }, "cvss": 5.1, "cvss-time": "2004-01-01T00:00:00.000-05:00", "id": "CVE-2004-1112", --snip--
Listing 8-15: Searching for CVEs related to buffer overflows
With this search tool, you can find complete information for any vulnerabilities matching your specific criteria. By combining flags, you can compile lists of, for instance, buffer overflow vulnerabilities from the last year that affect Linux.
Customize It
Although search.py doesn’t currently support output formatting for CVE searches (only for product searches), you can use Linux command line utilities, like jq, to parse JSON strings and even pull out specific fields. Use these tools to write bash scripts to do whatever data manipulation you please.
Here are a couple of examples:
- Generate a weekly list of relevant (for instance, high severity and affecting Windows 10) vulnerabilities that have been added to the database and then email it to security staff.
- Regularly look at “interesting” CVEs (perhaps one that is affecting your organization that you’ve not yet fully addressed) and note whether new reference URLs have been added to their records.
Your full vulnerability management system and some custom scripting can handle the preceding two use cases, but sometimes it’s easier and more straightforward to perform the same task with command line tools.
Summary
In this chapter, you learned about the main data sources—Nmap, OpenVAS, and cve-search—for your vulnerability management program. You tried each tool to understand the types of information you can gather and started thinking about how to use the three tools to provide a more complete vulnerability picture of your organization.
In Chapter 9, you’ll take the first steps toward painting that picture by writing scripts that will parse and import scanner outputs into your vulnerability database.