7
SETTING UP YOUR ENVIRONMENT

To start constructing a complete vulnerability management system, you must first build the foundation. In this chapter, you’ll set up the base Linux environment for your ­system, install the tools you’ll use in the following chapters, and write a script to ensure all the components are regularly updated.

Setting Up the System

Your first step, of course, is to set up a Linux-based environment for the basic tools you’ll use to build your system: those tools include Nmap, OpenVAS, cve-search, and Metasploit.

Installing the OS and Packages

We’ll use the following software in this guide. Many of these packages will be pre-installed in most Linux distributions. Install the rest using the package manager that comes with your system. (Ubuntu and other Debian-based systems use apt, whereas Redhat-based distributions typically use yum.)

• Linux (this guide uses Ubuntu 18.04 LTS)
• Python 3.3 or later (cve-search requires this Python version)
• MongoDB 2.2 or later, and MongoDB development headers (the package names will vary by distribution; Ubuntu uses mongodb and mongodb-dev)
• SQLite 3 (OpenVAS requires this version)
• Nmap
• pip3 (to install additional Python packages; package is python3-pip in Ubuntu)
• Git (for cve-search, Metasploit, Exploit-db)
• libxml2-dev, libxslt1-dev, zlib1g-dev (for cve-search)
• jq (for JSON parsing)
• cURL (to download files and scripts)
• psql (the PostgreSQL client; to manually look at the Metasploit Framework database)

The Linux distribution you use is up to you. Using a prebuilt virtual machine image can save some time, although I recommend working through the Linux install process manually to customize the installation to your needs.

Although it’s not a requirement, I recommend setting up a dedicated user (I used vmadmin) with sudo privileges that you’ll use to run the scripts we will create. You’ll increase your system security by not using the root user.

Customize It

To tailor this basic setup to your own environment, you could add more scanners (on separate physical or virtual hardware) and configure them to send their reports to a centralized location. Or you could locate the MongoDB instance on a separate machine or a shared server. Using databases other than MongoDB is also an option, although doing so will require modifying the scripts you write.

By deploying these packages on a compact single-board system, such as the Raspberry Pi, you can make a palm-sized vulnerability-scanning system that you could use at different locations as a portable vulnerability management tool.

You can also use a different Unix-type operating system, such as BSD or a commercial Unix like Solaris, or even Windows with the appropriate tools (Python, MongoDB) in place. But collecting all the prerequisites might be a chore. Recent versions of Windows 10 with the Linux subsystem installed can use standard Ubuntu packages.

Installing the Tools

After setting up your base Linux system, the next step is to install the main tools—OpenVAS, cve-search, and Metasploit—which you’ll use to build your vulnerability management system.

Setting Up OpenVAS

OpenVAS is an open source vulnerability scanner that derived from Nessus when Nessus became closed source. The OpenVAS community and Greenbone Networks GmbH currently maintain it.

Installing the Packages

OpenVAS is not part of the standard Ubuntu repository, so we need to add a custom repo to the trusted Ubuntu list. We’ll use the OpenVAS repository built by Mohammad Razavi, which is available at https://launchpad.net/~mrazavi/+archive/ubuntu/openvas/.

To get these packages, add this repository information to your apt ­software sources:

# add-apt-repository ppa:mrazavi/openvas

Next, update apt to make apt aware that the OpenVAS software is in this custom repository:

# apt-get update

Download and install OpenVAS. The download consists of about 100MB of data and in total will take up approximately 500MB of space:

# apt-get install openvas9

Updating OpenVAS

Once the package is installed, run the setup scripts in Listing 7-1 to sync data that OpenVAS uses in its scans. You must run all these scripts as root.

  # greenbone-nvt-sync
  # greenbone-scapdata-sync
  # greenbone-certdata-sync
  # service openvas-scanner restart
  # service openvas-manager restart 
➊ # openvasmd --rebuild --progress

Listing 7-1: Getting OpenVAS ready to go

The rebuilding tool ➊ might take a long time to complete. This is ­normal and is nothing to worry about.

Edit your Redis configuration, which the OpenVAS scanner daemon uses for temporary result storage. Edit /etc/redis/redis.conf and comment out any lines with the format save xx yy (for example, save 900 1). Then restart Redis (# service redis-server restart) and the OpenVAS scanner (# service openvas-scanner restart).

Test the Deployment

Go to https://<your-ip-address>:4000. If everything is up and running, you should see a Greenbone Security Assistant login page. The default login to Greenbone is admin/admin. Once you’re logged in, click around to familiarize yourself with the interface—and don’t forget to change the default credentials to something more secure. Setting up and running scans is a bit complicated, but you’ll delve into more of the OpenVAS and Greenbone scan options in Chapter 8.

Setting Up cve-search

The tool cve-search is a set of Python scripts backed by a MongoDB database that contains a vast amount of publicly available vulnerability information from the official CVE database at https://cve.mitre.org/. You’ll mostly use the cve-search database, not the frontend tools, but those utilities might be ­useful for doing manual vulnerability searches.

Downloading cve-search

You can download cve-search as a .zip or .tarball file from the cve-search site (https://adulau.github.io/cve-search/) and extract it or get it directly from the developer’s online repository using Git. Unlike many of the commands in these chapters, you can do this as a normal (unprivileged) user. The following command installs the cve-search tool into the ./cve-search directory:

$ git clone https://github.com/cve-search/cve-search.git

Installing Dependencies via pip

To ensure that all of cve-search’s prerequisites are in place, use the pip3 tool to set up the requirements in requirements.txt (a file that comes with cve-search). Before running this command, ensure that you’ve installed the libxml2-dev, libxslt1-dev, and zlib1g-dev packages or the equivalent for your Linux distribution:

$ cd cve-search; sudo pip3 install -r requirements.txt

If this command fails, don’t despair. Be patient, look at where errors are occurring in the installation process, make any necessary changes, and retry. You might need to add additional packages via your distribution’s package manager.

Populating the Database

Finally, you’ll build and populate the MongoDB database that serves as the datastore for the cve-search tools using the commands in Listing 7-2. The second and third scripts might take quite a long time to complete.

$ ./sbin/db_mgmt_json.py -p
Database population started
Importing CVEs for year 2002
Importing CVEs for year 2003
Importing CVEs for year 2004
Importing CVEs for year 2005
--snip--
$ ./sbin/db_mgmt_cpe_dictionary.py
Preparing [##################################################] 194571/194571
$ ./sbin/db_updater.py -c
INFO:root:Starting cve
Preparing [##################################################] 630/630
INFO:root:cve has 120714 elements (0 update)
INFO:root:Starting cpe
Not modified
--snip--
INFO:root:
[-] No plugin loader file!

Listing 7-2: Building and updating the CVE database

Testing cve-search

Once the tools are installed and the database is populated, try a simple search to see what kind of information is available in the CVE database. The command ./bin/search.py -c CVE-2010-3333 -o json|jq will find information about CVE-2010-3333, a stack buffer overflow vulnerability in Microsoft Office. Piping the command through jq will format the JSON blob into something much more readable. We’ll look at CVE information in more detail in Chapter 8.

Setting Up Metasploit

The Metasploit Framework tool contains numerous working exploits as well as a scriptable Ruby environment for automating repetitive or complex exploitation tasks. This step is optional because you can build nearly the entire system without using Metasploit.

Installing the Metasploit Framework

The easiest way to deploy the Metasploit Framework on a Linux system is to use the installer script detailed at https://github.com/rapid7/metasploit-framework/wiki/Nightly-Installers/. This script sets up the proper Metasploit repositories and integrates with package management systems, such as yum and apt. Another option is to clone the Git repository and build directly from there, but I follow the installer method in this deployment.

Execute this long chain of commands as root to download and run the latest version of the installer script (msfinstall), which will add the Metasploit repository to apt and then install the Metasploit Framework:

# curl https://raw.githubusercontent.com/rapid7/metasploit-omnibus/master/config/templates
/metasploit-framework-wrappers/msfupdate.erb > msfinstall && chmod 755 msfinstall && 
./msfinstall

If you want more control over the individual steps, follow the instructions at the Github URL above.

Completing and Testing the Installation

Run msfconsole as root to complete the setup.

# msfconsole

This step sets up the Postgres database and eventually brings you to an msf> prompt. From there, you can explore the Metasploit Framework. For example, if you’re interested in whether Metasploit has exploits for the 2014 “Heartbleed” vulnerability, search for CVE-2014-0160, as shown in Listing 7-3.

msf > search cve-2014-0160
[!] Module database cache not built yet, using slow search
Matching Modules
================

   #  Name  Disclosure Date  Rank  Check  Description
   -  ----  ---------------  ----  -----  -----------
➊ 0  auxiliary/scanner/ssl/openssl_heartbleed          2014-04-07       
   normal  Yes    OpenSSL Heartbeat (Heartbleed) Information Leak
➋ 1  auxiliary/server/openssl_heartbeat_client_memory  2014-04-07       
   normal  No     OpenSSL Heartbeat (Heartbleed) Client Memory Exposure

Listing 7-3: Sample Metasploit search output

Either of the two exploits ➊➋ matching this vulnerability could exploit a system vulnerable to CVE-2014-0160. Because I’m just covering installing the Metasploit Framework, I’ll leave further exploitation for you as an exercise. Keep in mind that Metasploit includes working exploit code and it can cause serious issues, including but not limited to crashes, on any systems you try attacking with Metasploit. Be careful and try it only on systems you’re authorized to attack.

Customize It

If you prefer or if your Linux distribution doesn’t have prebuilt packages for OpenVAS available, you can install OpenVAS from source. Doing so will give you some opportunities to customize it for your environment.

Although I installed OpenVAS 9 in this chapter, earlier versions are functional (and might be easier to find for your architecture). Those versions might have a different XML output format, which means you’ll need to modify your scripts.

You can install the cve-search set of scripts in a different location (by cloning the Git repository into a different location) if you want multiple users to use the tool.

By installing Metasploit manually, you’ll have more control over the additional packages it uses—Ruby and PostgreSQL in particular—and can customize your deployment more fully.

Keeping the System Updated

Once all the software is installed, you’ll need to keep it up-to-date. The script in this section shows you how to create an updater script that is suitable for running on a schedule.

Writing a Script for Automatic Updates

First, create a bash script that will run a number of update scripts for the tools you’ve deployed. This way, instead of having to run numerous separate update scripts every time you want to update your toolchain, you can run a single script. Even better, you can add this script to the system scheduler (cron) so it will run automatically on a regular basis.

Save Listing 7-4 as update-vm-tools.sh.

➊ #!/bin/bash
➋ CVE_SEARCH_DIR=/path/to/cve-search

➌ LOG=/path/to/output.log

  # this clears the log file by overwriting it with a single
  # line containing the date and time to an empty file
➍ date > ${LOG}

➎ greenbone-nvt-sync >> ${LOG}
  greenbone-scapdata-sync >> ${LOG}
  greenbone-certdata-sync >> ${LOG}
  service openvas-scanner restart >> ${LOG}
  service openvas-manager restart >> ${LOG}
  openvasmd --rebuild >> ${LOG}

➏ ${CVE_SEARCH_DIR}/sbin/db_updater.py -v >> ${LOG}

➐ apt-get -y update >> ${LOG}

➑ msfupdate >> ${LOG}

➒ echo Update process done. >> ${LOG}

Listing 7-4: A simple system updater script

The shebang identifies that this is a bash shell script ➊, so when you run it, the system knows which interpreter to use. The CVE_SEARCH_DIR variable ➋ points to the cve-search path on your system. The LOG variable ➌ points to a log file, which starts with the current date ➍. The output of all the update commands will be written to the log file.

The same commands you used in Listing 7-1 to sync OpenVAS are used to update it ➎. Check the specific version of OpenVAS that you’ve installed to ensure that these executables and paths are correct. Then ­execute the cve-search updater using the value stored in CVE_SEARCH_DIR to refer to the actual cve-search path on your system ➏. To update the underlying Linux system as well as the OpenVAS packages, run a full system update using the -y flag so the update doesn’t ask for confirmation and runs without human interaction ➐. (If you’re running a Linux system that isn’t based on Debian, your update command might be different; for instance, RPM-based systems like Redhat use yum to update.) Next, we update Metasploit using its own msfupdate script ➑ and write Update process done to the log file ➒.

Set the script as an executable using the following chmod command:

# chmod +x update-vm-tools.sh

Now you can run the script at any time to update your vulnerability management toolchain.

Running the Script Automatically

Now that you have a single update script, you can add it to crontab to run regularly. I set it up to run weekly on Sundays at 4 AM, but you can change that to a different time.

Edit /etc/crontab (as root) and add the following line at the bottom of the file:

0 4 * * 7 root /path/to/update-vm-tools.sh

Many Linux distributions have directories that will automatically run anything in them on a regular basis using cron. For example, Ubuntu runs any script placed in the /etc/cron.weekly/ directory at 6 AM on Sundays. If you want to use this method instead, simply save your update script in that directory or create a symlink to it.

To ensure that the script ran successfully, you can look at the log file generated in the update script (/path/to/update.log in Listing 7-4 ➌) and look at the output of each update script.

Customize It

I placed the data update and system update components in a single script for simplicity. But you might prefer to separate data and application updates either by making the apt update manual or by running it on a different schedule from the other update scripts. You might also change the apt-get command to update only Metasploit and OpenVAS, saving the full system updates for a manual process.

In later chapters (particularly Chapter 12), you’ll schedule other scripts for data collection and analysis. When setting the specific timeframes for these other scripts to run, keep in mind that they might conflict with the system updates in this script, so schedule accordingly.

Summary

In this chapter, you took the first steps to building a complete vulnerability management system: you set up the OS and the underlying tools that you’ll be writing script controls for in the following chapters. It might not seem like much, but from small beginnings you’ll build great things.

In the next chapter, you’ll look at Nmap, cve-search, and OpenVAS in more detail so you can become familiar with their features before you start controlling them via shell scripts and Python.