It is not just the motivated attacker that we are looking for. With virtualization, there is also the legitimate administrator of the virtual infrastructure who makes his life easier by bending some rules. Additionally, an attacker may use the power of virtualization to reshape the topology of the infrastructure according to his needs. In the following sections, we will show some scenarios and detection methods.
Network virtualization allows operations to create almost arbitrary network infrastructures in a static, physical network. This capability is sometimes referred to as Data center as a Service (DCaaS). DCaaS allows the customers to utilize a defined portion of a physical data center to define virtual data centers in software.
Due to malicious access to this capability or human error, the resulting network configuration may expose internal resources to the internet, bypass firewalls, or allow access to malicious services.
Therefore, we will show a simple way to programmatically get the network configuration of a vSphere environment using Python.
Tip
Visualize virtual networks
Most virtualization environments have built-in capabilities to visualize the virtual network setup. For example, VMware vSphere can create an image of the network topology. In a forensic analysis, this may serve as the starting point and support focusing the next step on the most promising assets.
This image was generated with the Windows client software for VMware vCenter Server and it depicts our test setup. Obviously, EvesMachine is not connected properly, that is, it can bypass the Firewall.
The community sample scripts for pyVmomi already provide a script for iterating over all network interfaces, https://github.com/vmware/pyvmomi-community-samples/blob/master/samples/getvnicinfo.py, and displaying the connections of virtual machines. Therefore, we modified this script to display only those virtual machines that have multiple network connections, as follows:
#!/usr/bin/env python
from pyVim import connect
from pyVmomi import vmodl
from pyVmomi import vim
import sys
def generate_portgroup_info(content):
"""Enumerates all hypervisors to get
network infrastructure information"""
host_view = content.viewManager.CreateContainerView(content.rootFolder,
[vim.HostSystem],
True)
hostlist = [host for host in host_view.view]
host_view.Destroy()
hostPgDict = {}
for host in hostlist:
pgs = host.config.network.portgroup
hostPgDict[host] = pgs
return (hostlist, hostPgDict)
def get_vms(content, min_nics=1):
vm_view = content.viewManager.CreateContainerView(content.rootFolder,
[vim.VirtualMachine],
True)
vms = [vm for vm in vm_view.view]
vm_view.Destroy()
vm_with_nics = []
for vm in vms:
num_nics = 0
for dev in vm.config.hardware.device:
# ignore non-network devices
if not isinstance(dev, vim.vm.device.VirtualEthernetCard):
continue
num_nics = num_nics + 1
if num_nics >= min_nics:
vm_with_nics.append(vm)
break
return vm_with_nics
def print_vm_info(vm, hosts, host2portgroup, content):
print "
=== %s ===" % vm.name
for dev in vm.config.hardware.device:
if not isinstance(dev, vim.vm.device.VirtualEthernetCard):
continue
dev_backing = dev.backing
if hasattr(dev_backing, 'port'):
# NIC is connected to distributed vSwitch
portGroupKey = dev.backing.port.portgroupKey
dvsUuid = dev.backing.port.switchUuid
try:
dvs = content.dvSwitchManager.QueryDvsByUuid(dvsUuid)
except:
portGroup = 'ERROR: DVS not found!'
vlanId = 'N/A'
vSwitch = 'N/A'
else:
pgObj = dvs.LookupDvPortGroup(portGroupKey)
portGroup = pgObj.config.name
vlObj = pgObj.config.defaultPortConfig.vlan
if hasattr(vlObj, 'pvlanId'):
vlanId = str(pgObj.config.defaultPortConfig.vlan.pvlanId)
else:
vlanId = str(pgObj.config.defaultPortConfig.vlan.vlanId)
vSwitch = str(dvs.name)
else:
# NIC is connected to simple vSwitch
portGroup = dev.backing.network.name
vmHost = vm.runtime.host
# look up the port group from the
# matching host
host_pos = hosts.index(vmHost)
viewHost = hosts[host_pos]
pgs = host2portgroup[viewHost]
for p in pgs:
if portgroup in p.key:
vlanId = str(p.spec.vlanId)
vSwitch = str(p.spec.vswitchName)
if portGroup is None:
portGroup = 'N/A'
print '%s -> %s @ %s -> %s (VLAN %s)' % (dev.deviceInfo.label,
dev.macAddress,
vSwitch,
portGroup,
vlanId)
def print_dual_homed_vms(service):
"""Lists all virtual machines with multiple
NICs to different networks"""
content = service.RetrieveContent()
hosts, host2portgroup = generate_portgroup_info(content)
vms = get_vms(content, min_nics=2)
for vm in vms:
print_vm_info(vm, hosts, host2portgroup, content)
if __name__ == '__main__':
if len(sys.argv) < 5:
print 'Usage: %s host user password port' % sys.argv[0]
sys.exit(1)
service = connect.SmartConnect(host=sys.argv[1],
user=sys.argv[2],
pwd=sys.argv[3],
port=int(sys.argv[4]))
print_dual_homed_vms(service)First, this script iterates over all (hypervisor) hosts to collect information about the virtual switches that are present on each ESXi system. Then, it iterates over all virtual machines to collect those with more than one network card. Then the information about virtual network cards is combined with the information about virtual switches to derive the information about the connectivity.
Here is the sample output from our lab environment as depicted previously:
user@lab:~$ python listDualHomed.py 192.168.167.26 readonly 'mypwd' 443 === EvesMachine === Network adapter 1 -> 00:50:56:ab:04:38 @ dvSwitch -> dvInternalNetwork (VLAN 8) Network adapter 2 -> 00:50:56:ab:23:50 @ dvSwitch -> dvDMZ (VLAN 0) === Firewall === Network adapter 1 -> 00:50:56:ab:12:e6 @ dvSwitch -> dvInternalNetwork (VLAN 8) Network adapter 2 -> 00:50:56:ab:4b:62 @ dvSwitch -> dvDMZ (VLAN 0)
Our script correctly identified the two systems, EvesMachine and Firewall, being simultaneously connected to different networks. In this particular case, both the systems can be used to connect VLAN 0 with VLAN 8 on the same virtual switch, dvSwitch.
It may sound like an oxymoron, but most virtualization techniques allow direct hardware access. The legitimate reasons to allow virtual systems to directly access a piece of hardware without having to use the services of the hypervisor are as follows:
- Special hardware supposed to be connected to a virtual machine: Special hardware such as radio clocks for virtual time servers or dongles being part of a copy protection mechanism.
- Temporary use of physical media on a virtual system: Sometimes, this capability is used to access media from physical systems from a virtual environment, for example, to restore backups from a physical media to a virtual system. In general, the network attached storage systems should be preferred over attaching physical media to a virtual system.
- Permanent use of drives of a hypervisor from a virtual machine: This can be useful if the virtual system uses software that is provided on physical media and therefore, needs access to a real physical drive for installation and updates of the software. However, one should consider using downloaded versions or ISO images instead of granting direct access to the hardware of the hypervisor.
As you may guess, according to this list, direct hardware access is more the exception than the rule in a modern virtualized data center. Furthermore, direct access to the hypervisor hardware breaks one fundamental principle of virtualization.
Note
Direct hardware access bypasses the security mechanism of the virtualization environment, that is, all the virtual hardware is controlled by the hypervisor. Consequently, direct hardware access always poses the risk of manipulation of hypervisor resources, data leakage, and system instabilities.
The following are some examples of the directly attached hardware that are most likely malicious:
- Network devices (create network connections that are invisible to the hypervisor)
- Keyboard, mouse, and so on (create console access that are invisible to the hypervisor)
- Hypervisor disk partitions
The latter is especially dangerous. If a virtual machine manages to get the raw disk access to the hypervisor, it can manipulate the virtualization environment. The consequences include the complete control over the virtualization environment along with the access to all virtual machines, all virtual networks, the capability to create new rogue resources and reshape the overall network topology.
The following script connects to a VMware vSphere instance and lists all virtual machines with direct hardware access:
#!/usr/bin/env python
from pyVim import connect
from pyVmomi import vmodl
from pyVmomi import vim
import re
import sys
def get_vms(content):
"""Returns a list of all virtual machines."""
vm_view = content.viewManager.CreateContainerView(content.rootFolder,
[vim.VirtualMachine],
True)
vms = [vm for vm in vm_view.view]
vm_view.Destroy()
return vms
def print_vm_hardware_access(vm):
findings = []
for dev in vm.config.hardware.device:
if isinstance(dev, vim.vm.device.VirtualUSB):
findings.append('USB access to host device ' + dev.backing.deviceName)
elif isinstance(dev, vim.vm.device.VirtualSerialPort):
findings.append('Serial port access')
elif isinstance(dev, vim.vm.device.VirtualCdrom):
if not dev.backing is None:
if 'vmfs/devices/cdrom' in dev.backing.deviceName:
findings.append('Access to CD/DVD drive')
elif isinstance(dev, vim.vm.device.VirtualDisk):
if dev.backing is None or
dev.backing.fileName is None or
re.match(r'.*.vmdk', dev.backing.fileName) is None:
findings.append('Suspicious HDD configuration')
if len(findings) > 0:
print '=== %s hardware configuration findings ===' % vm.name
for l in findings:
print l
print "
"
def print_direct_hardware_access(content):
vms = get_vms(content)
for vm in vms:
print_vm_hardware_access(vm)
if __name__ == '__main__':
if len(sys.argv) < 5:
print 'Usage: %s host user password port' % sys.argv[0]
sys.exit(1)
service = connect.SmartConnect(host=sys.argv[1],
user=sys.argv[2],
pwd=sys.argv[3],
port=int(sys.argv[4]))
# access the inventory
content = service.RetrieveContent()
print_direct_hardware_access(content)This script is very eager, that is, it does not check whether the device is actually in a connected state or whether there is media accessible through the device. Nevertheless, an output similar to the following calls for deeper inspection:
user@lab:~$ python listHardwareAccess.py 192.168.167.26 readonly pwd 443 === EvesMachine hardware configuration findings === Access to CD/DVD drive Serial port access USB access to host device path:2/0 version:2 === DeveloperBox hardware configuration findings === Access to CD/DVD drive === dmzBox hardware configuration findings === Access to CD/DVD drive
EvesMachine appears to have direct access to a USB device attached to its hypervisor system. Moreover, there seems to be a direct link to the serial port of the hypervisor. Access to CD/DVD drive of the hypervisor should not be granted in general. However, for a lot of installations, people tend to use the optical drive of the hypervisor to install or update a software.
Tip
Extract hardware configuration from the VMX file
Using a script such as the previous one requires access to the virtual environment. Therefore, the main purpose of such scripts is to narrow the focus of the forensic investigation. For permanent evidence and record, the directory of the virtual machines should be copied from the datastore. There, the VMX file contains all VM specific configuration settings including the hardware access.
In this and the previous sections, virtualization is considered as an additional attack surface. In the following section, we will outline how virtualization techniques can actually support a forensic investigation.