12
AUTOMATING ARACHNI

Arachni is a powerful web application black-box security scanner written in Ruby. It features support for many types of web application vulnerabilities, including many of the OWASP Top 10 vulnerabilities (such as XSS and SQL injection); a highly scalable distributed architecture that allows you to spin up scanners in a cluster dynamically; and full automation through both a remote procedure call (RPC) interface and a representational state transfer (REST) interface. In this chapter, you’ll learn how to drive Arachni with its REST API and then with its RPC interface to scan a given URL for web application vulnerabilities.

Installing Arachni

The Arachni website (http://www.arachni-scanner.com/) gives you the current download package for Arachni across multiple operating systems. You can use these installers to set up Arachni on your own system. Once you’ve downloaded it, you can test it by running Arachni against a server designed to test for web vulnerabilities, as shown in Listing 12-1. Although this command isn’t using the RPC to drive Arachni just yet, you can see what kind of output we will get when scanning for potential XSS or SQL injection vulnerabilities.

$ arachni --checks xss*,sql* --scope-auto-redundant 2
  "http://demo.testfire.net/default.aspx"

Listing 12-1: Running Arachni against an intentionally vulnerable website

This command uses Arachni to check for XSS and SQL vulnerabilities in the website http://demo.testfire.net/default.aspx. We limit the scope of the pages it will follow by setting --scope-auto-redundant to 2. Doing so makes Arachni follow URLs with the same parameters but with different parameter values up to twice before moving on to a new URL. Arachni can scan more quickly when a lot of links with the same parameters are available but all go to the same page.

Within just a few minutes (depending on your internet speed), Arachni should report back a handful of XSS and SQL injection vulnerabilities in the website. Don’t worry—they’re supposed to be there! This website was built to be vulnerable. Later in the chapter, when testing our custom C# automation, you can use this list of XSS, SQL injection, and other vulnerabilities to ensure your automation is returning the correct results.

But let’s say you want to automatically run Arachni against an arbitrary build of your web application as part of a secure software development life cycle (SDLC). Running it by hand isn’t very efficient, but we can easily automate Arachni to kick off scan jobs so it can work with any continuous integration system to pass or fail builds depending on the results of the scans. That’s where the REST API comes in.

The Arachni REST API

Recently, a REST API was introduced so that simple HTTP requests can be used to drive Arachni. Listing 12-2 shows how to start this API.

$ arachni_rest_server
Arachni - Web Application Security Scanner Framework v2.0dev
   Author: Tasos "Zapotek" Laskos <[email protected]>

           (With the support of the community and the Arachni Team.)

   Website:       http://arachni-scanner.com
   Documentation: http://arachni-scanner.com/wiki

  ➊[*] Listening on http://127.0.0.1:7331

Listing 12-2: Running the Arachni REST server

When you start the server, Arachni will output some information about itself, including the IP address and port it is listening on ➊. Once you know the server is working, you can start using the API.

With the REST API, you can start a simple scan using any common HTTP utility such as curl or even netcat. In this book, we’ll use curl as we have in previous chapters. Our first scan is shown in Listing 12-3.

$ curl -X POST --data '{"url":"http://demo.testfire.net/default.aspx"}'➊
  http://127.0.0.1:7331/scans
{"id":"b139f787f2d59800fc97c34c48863bed"}➋
$ curl http://127.0.0.1:7331/scans/b139f787f2d59800fc97c34c48863bed➌
{"status":"done","busy":false,"seed":"676fc9ded9dc44b8a32154d1458e20de",
--snip--

Listing 12-3: Testing the REST API with curl

To kick off a scan, all we need to do is make a POST request with some JSON in the request body ➊. We start a new Arachni scan by passing JSON with the URL to scan using the --data argument from curl and send that to the /scans endpoint. The ID of the new scan is returned in the HTTP response ➋. After creating the scan, we can also retrieve the current scan status and results with a simple HTTP GET request (the default request type for curl) ➌. We do this by calling on the IP address and port Arachni is listening on and appending the ID we obtained when creating the scan for the scans request to the /scans/ URL endpoint. After the scan finishes, the scan log will contain any vulnerabilities found during scanning, such as XSS, SQL injection, and other common web application vulnerabilities.

Once this is done and we have an idea of how the REST API works, we can start writing the code that will allow us to use the API to scan any site we have an address for.

Creating the ArachniHTTPSession Class

As in previous chapters, we will implement both a session and a manager class to interact with the Arachni API. Currently, these classes are relatively simple, but breaking them out now allows greater flexibility should the API require authentication or extra steps in the future. Listing 12-4 details the ArachniHTTPSession class.

public class ArachniHTTPSession
{
  public ➊ArachniHTTPSession(string host, int port)
  {
    this.Host = host;
    this.Port = port;
  }
  public string Host { get; set; }
  public int Port { get; set; }

  public JObject ➋ExecuteRequest(string method, string uri, JObject data = null)
  {
    string url = "http://" + this.Host + ":" + this.Port.ToString() + uri;
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
    request.Method = method;

    if (data != null)
    {
      string dataString = data.ToString();
      byte[] dataBytes = System.Text.Encoding.UTF8.GetBytes(dataString);

      request.ContentType = "application/json";
      request.ContentLength = dataBytes.Length;

      request.GetRequestStream().Write(dataBytes, 0, dataBytes.Length);
    }

    string resp = string.Empty;
    using (StreamReader reader = new StreamReader(request.GetResponse().GetResponseStream()))
      resp = reader.ReadToEnd();

    return JObject.Parse(resp);
  }
}

Listing 12-4: The ArachniHTTPSession class

At this point in the book, the ArachniHTTPSession class should be fairly simple to read and understand, so we won’t go too deep into the code. We create a constructor ➊ that accepts two arguments—the host and port to connect to—and assigns the values to the corresponding properties. We then create a method to execute a generic HTTP request ➋ based on the parameters passed to the method. The ExecuteRequest() method should return a JObject with any data that will be returned by a given API endpoint. Because the ExecuteRequest() method can be used to make any API call against Arachni, the only thing we can expect is that the response will be JSON that can be parsed from the server’s response into a JObject.

Creating the ArachniHTTPManager Class

The ArachniHTTPManager class should also seem simple at this point, as Listing 12-5 shows.

public class ArachniHTTPManager
{
  ArachniHTTPSession _session;
  public ➊ArachniHTTPManager(ArachniHTTPSession session)
  {
    _session = session;
  }
  public JObject ➋StartScan(string url, JObject options = ➌null)
  {
    JObject data = new JObject();
    data["url"] = url;
    data.Merge(options);

    return _session.ExecuteRequest("POST", "/scans", data);
  }

  public JObject ➍GetScanStatus(Guid id)
  {
    return _session.ExecuteRequest("GET", "/scans/" + id.ToString ("N"));
  }
}

Listing 12-5: The ArachniHTTPManager class

Our ArachniHTTPManager constructor ➊ accepts a single argument—the session to use for executing requests—and then assigns the session to a local private variable for use later. We then create two methods: StartScan() ➋ and GetScanStatus() ➍. These methods are all we need to create a small tool to scan and report on a URL.

The StartScan() method accepts two arguments, one of which is optional with a default value of null ➌. By default, you can just specify a URL with no scan options to StartScan(), and Arachni will simply spider the site without checking for vulnerabilities—a feature that could give you an idea of how much surface area the web application has (that is, how many pages and forms there are to test). However, we actually want to specify extra arguments to tune the Arachni scan, so we’ll go ahead and merge those options into our data JObject, and then we’ll POST the scan details to the Arachni API and return the JSON sent back. The GetScanStatus() method makes a simple GET request, using the ID of the scan passed into the method in the URL of the API, and then returns the JSON response to the caller.

Putting the Session and Manager Classes Together

With both of the classes implemented, we can start scanning, as Listing 12-6 shows.

public static void Main(string[] args)
{
  ArachniHTTPSession session = new ArachniHTTPSession("127.0.0.1", 7331);
  ArachniHTTPManager manager = new ArachniHTTPManager(session);

 ➊JObject scanOptions = new JObject();
  scanOptions["checks"] = new JArray() { "xss*", "sql*" } ;
  scanOptions["audit"] = new JObject();
  scanOptions["audit"]["elements"] = new JArray() { "links", "forms" };

  string url = "http://demo.testfire.net/default.aspx";
  JObject scanId = manager.➋StartScan(url, scanOptions);
  Guid id = Guid.Parse(scanId["id"].ToString());
  JObject scan = manager.➌GetScanStatus(id);

  while (scan["status"].ToString() != "done")
  {
    Console.WriteLine("Sleeping a bit until scan is finished");
    System.Threading.Thread.Sleep(10000);
    scan = manager.GetScanStatus(id);
  }

 ➍Console.WriteLine(scan.ToString());
}

Listing 12-6: Driving Arachni with the ArachniHTTPSession and ArachniHTTPManager classes

After instantiating our session and manager classes, we create a new JObject ➊ to store our scan options in. These options directly correlate with the command line options you see from the Arachni tool when running arachni –help (there’s a lot). By storing a JArray with the values xss* and sql* in the checks option key, we tell Arachni to run XSS and SQL injection tests against the website, rather than simply spidering the application and finding all possible pages and forms. The audit option key just below that tells Arachni to audit links it finds and any HTML forms for checks we tell it to run.

After setting up the scan options, we start the scan by calling the StartScan() method ➋ and passing our test URL as the argument. Using the ID returned by StartScan(), we retrieve the current scan status with GetScanStatus() ➌ and then loop until the scan is finished, checking every second for a new scan status. Once this is finished, we print the JSON scan results to the screen ➍.

The Arachni REST API is simple and easily accessible to most security engineers or hobbyists since it can be used with basic command line tools. It is also highly automatable using the most common C# libraries, and it should be an easy introduction for an SDLC or for general automatic use on your own websites for weekly or monthly scans. For some extra fun, try running Arachni with your automation against previous web applications from the book with known vulnerabilities, such as BadStore. Now that we’ve looked at the Arachni API, we can discuss how to automate its RPC.

The Arachni RPC

The Arachni RPC protocol is a bit more advanced than the API, but it’s also more powerful. Although also powered by MSGPACK, just like Metasploit’s RPC, Arachni’s protocol has a twist. The data is sometimes Gzip compressed and is only communicated over a regular TCP socket, not HTTP. This complexity has its benefits: the RPC is blazingly fast without the HTTP overhead, and it gives you more scanner management power than the API, including the abilities to spin scanners up and down at will and create distributed scanning clusters, thus allowing clusters of Arachni to balance scanning across multiple instances. Long story short, the RPC is very powerful, but expect more development focus and support for the REST API because it is more accessible to most developers.

Manually Running the RPC

To start an RPC listener, we use the simple script arachni_rpcd, as shown in Listing 12-7.

$ arachni_rpcd
Arachni - Web Application Security Scanner Framework v2.0dev
   Author: Tasos "Zapotek" Laskos <[email protected]>

           (With the support of the community and the Arachni Team.)

   Website:       http://arachni-scanner.com
   Documentation: http://arachni-scanner.com/wiki

I,[2016-01-16T18:23:29.000746 #18862] INFO - System: RPC Server started.
I,[2016-01-16T18:23:29.000834 #18862] INFO - System: Listening on ➊127.0.0.1:7331

Listing 12-7: Running the Arachni RPC server

Now we can test the listener using another script shipped with Arachni called arachni_rpc. Note the dispatcher URL ➊ in the output of the listening RPC server. We’ll need it next. The arachni_rpc script that ships with Arachni allows you to interface with the RPC listener from the command line. After starting the arachni_rpcd listener, open another terminal and change to the Arachni project root directory; then kick off a scan using the arachni_rpc script, as shown in Listing 12-8.

$ arachni_rpc --dispatcher-url 127.0.0.1:7331
  "http://demo.testfire.net/default.aspx"

Listing 12-8: Running an Arachni scan of the same intentionally vulnerable website via the RPC

This command will drive Arachni to use the MSGPACK RPC, just as our C# code will do soon. If this is successful, you should see a nice text-based UI updating you on the status of the current scan with a nice report at the end, as Listing 12-9 shows.

Arachni - Web Application Security Scanner Framework v2.0dev
   Author: Tasos "Zapotek" Laskos <[email protected]>

           (With the support of the community and the Arachni Team.)

   Website:       http://arachni-scanner.com
   Documentation: http://arachni-scanner.com/wiki
[~] 10 issues have been detected.

 [+]  1 | Cross-Site Scripting (XSS) in script context at
http://demo.testfire.net/search.aspx in form input `txtSearch` using GET.
 [+]  2 | Cross-Site Scripting (XSS) at http://demo.testfire.net/search.aspx
in form input `txtSearch` using GET.
 [+]  3 | Common directory at http://demo.testfire.net/PR/ in server.
 [+]  4 | Backup file at http://demo.testfire.net/default.exe in server.
 [+]  5 | Missing 'X-Frame-Options' header at http://demo.testfire.net/default.aspx in server.
 [+]  6 | Common administration interface at http://demo.testfire.net/admin.aspx in server.
 [+]  7 | Common administration interface at http://demo.testfire.net/admin.htm in server.
 [+]  8 | Interesting response at http://demo.testfire.net/default.aspx in server.
 [+]  9 | HttpOnly cookie at http://demo.testfire.net/default.aspx in cookie with inputs
`amSessionId`.
 [+] 10 | Allowed HTTP methods at http://demo.testfire.net/default.aspx in server.


 [~] Status: Scanning
 [~] Discovered 3 pages thus far.

 [~] Sent 1251 requests.
 [~] Received and analyzed 1248 responses.
 [~] In 00:00:45
 [~] Average: 39.3732270014467 requests/second.

 [~] Currently auditing           http://demo.testfire.net/default.aspx
 [~] Burst response time sum      72.511066 seconds
 [~] Burst response count total   97
 [~] Burst average response time  0.747536762886598 seconds
 [~] Burst average                20.086991167522193 requests/second
 [~] Timed-out requests           0
 [~] Original max concurrency     20
 [~] Throttled max concurrency    20

 [~] ('Ctrl+C' aborts the scan and retrieves the report)

Listing 12-9: The arachni_rpc command line scanning UI

The ArachniRPCSession Class

To run a scan using the RPC framework and C#, we’ll implement the session/manager pattern again, starting with the Arachni RPC session class. With the RPC framework, you get a little bit more intimate with the actual Arachni architecture because you need to deal with dispatchers and instances at a granular level. When you connect to the RPC framework for the first time, you are connected to a dispatcher. You can interact with this dispatcher to create and manage instances, which do the actual scanning and work, but these scanning instances end up dynamically listening on a different port than the dispatcher. In order to provide an easy-to-use interface for both dispatchers and instances, we can create a session constructor that allows us to gloss over these distinctions a little bit, as shown in Listing 12-10.

public class ArachniRPCSession : IDisposable
{
  SslStream _stream = null;
  public ArachniRPCSession(➊string host, int port,
                             bool ➋initiateInstance = false)
  {
    this.Host = host;
    this.Port = port;
   ➌GetStream(host, port);
    this.IsInstanceStream = false;

    if (initiateInstance)
    {
      this.InstanceName = ➍Guid.NewGuid().ToString();
      MessagePackObjectDictionary resp =
                      this.ExecuteCommand("dispatcher.dispatch"➎,
                      new object[] { this.InstanceName }).AsDictionary();

Listing 12-10: The first half of the ArachniRPCSession constructor

The constructor accepts three arguments ➊. The first two—the host to connect to and the port on the host—are required. The third one, which is optional ➋ (with a default value of false), allows the programmer to automatically create a new scanning instance and connect to it, instead of having to create the new instance manually via the dispatcher.

After assigning the Host and Port properties the values of the first two arguments passed to the constructor, respectively, we connect to the dispatcher using GetStream() ➌. If a true value is passed in as the third argument, instantiateInstance (which is false by default), we create a unique name for the instance we want to dispatch using a new Guid ➍ and then run the dispatcher.dispatch ➎ RPC command to create a new scanner instance that returns a new port (and potentially new host if you have a cluster of scanner instances). Listing 12-11 shows the rest of the constructor.

      string[] url = ➊resp["url"].AsString().Split(':');

      this.InstanceHost = url[0];
      this.InstancePort = int.Parse(url[1]);
      this.Token = ➋resp["token"].AsString();

     ➌GetStream(this.InstanceHost, this.InstancePort);

      bool aliveResp = this.➍ExecuteCommand("service.alive?", new object[] { },
                       this.Token).AsBoolean();

      this.IsInstanceStream = aliveResp;
    }
  }

 ➎public string Host { get; set; }
  public int Port { get; set; }
  public string Token { get; set; }
  public bool IsInstanceStream { get; set; }
  public string InstanceHost { get; set; }
  public int InstancePort { get; set; }
  public string InstanceName { get; set; }

Listing 12-11: The second half of the ArachniRPCSession constructor and its properties

At ➊, we split the scanner instance URL (for example, 127.0.0.1:7331) into the IP address and the port (127.0.01 and 7331, respectively). Once we have the instance host and port we will use to drive the actual scan, we assign the values to our InstanceHost and InstancePort properties, respectively. We also save the authentication token ➋ returned by the dispatcher so we can make authenticated RPC calls later on the scanner instance. This authentication token is automatically generated by the Arachni RPC when we dispatch a new instance so that only we can use the new scanner with the token.

We connect to the scanner instance using GetStream() ➌, which provides direct access to the scanning instance. If the connection is successful and the scanning instance is alive ➍, we assign the IsInstanceStream property to true so that we know whether we are driving a dispatcher or a scanning instance (which determines the RPC calls we can make to Arachni, such as creating a scanner or performing a scan) later when we implement the ArachniRPCManager class. After the constructor, we define the properties ➎ for the session class, all of which are used in the constructor.

The Supporting Methods for ExecuteCommand()

Before we implement ExecuteCommand(), we need to implement the supporting methods for ExecuteCommand(). We’re almost there! Listing 12-12 shows the methods we need in order to finish up the ArachniRPCSession class.

public byte[] DecompressData(byte[] inData)
{
  using (MemoryStream outMemoryStream = new MemoryStream())
  {
    using (➊ZOutputStream outZStream = new ZOutputStream(outMemoryStream))
    {
      outZStream.Write(inData, 0, inData.Length);
      return outMemoryStream.ToArray();
    }
  }
}

private byte[] ➋ReadMessage(SslStream sslStream)
{
  byte[] sizeBytes = new byte[4];
  sslStream.Read(sizeBytes, 0, sizeBytes.Length);

  if (BitConverter.IsLittleEndian)
    Array.Reverse(sizeBytes);

  uint size = BitConverter.➌ToUInt32(sizeBytes, 0);
  byte[] buffer = new byte[size];
  sslStream.Read(buffer, 0, buffer.Length);

  return buffer;
}

private void ➍GetStream(string host, int port)
{
  TcpClient client = new TcpClient(host, port);

  _stream = new SslStream(client.GetStream(), false,
                          new RemoteCertificateValidationCallback(➎ValidateServerCertificate),
                          (sender, targetHost, localCertificates,
                          remoteCertificate, acceptableIssuers)
                          => null);

  _stream.AuthenticateAsClient("arachni", null, SslProtocols.Tls, false);
}

private bool ValidateServerCertificate(object sender, X509Certificate certificate,
                               X509Chain chain, SslPolicyErrors sslPolicyErrors)
{
  return true;
}

public void ➏Dispose()
{
  if (this.IsInstanceStream && _stream != null)
    this.ExecuteCommand(➐"service.shutdown", new object[] { }, this.Token);

  if (_stream != null)
    _stream.Dispose();

  _stream = null;
}

Listing 12-12: The supporting methods for the ArachniRPCSession class

Most of the support methods for the RPC session class are relatively simple. The DecompressData() method creates a new output stream from the zlib library available in NuGet, called ZOutputStream ➊. This returns the decompressed data as a byte array. In the ReadMessage() method ➋, we read the first 4 bytes from the stream and then convert the bytes into a 32-bit unsigned integer ➌ that represents the length of the rest of the data. Once we have the length, we read the rest of the data from the stream and return the data as a byte array.

The GetStream() method ➍ is also very similar to the code we used to create a network stream in the OpenVAS library. We create a new TcpClient and wrap the stream in an SslStream. We use the ValidateServerCertificate() method ➎ to trust all SSL certificates by returning true all the time. This allows us to connect to the RPC instances with self-signed certificates. Finally, Dispose() ➏ is required by the IDisposable interface that the ArachniRPCSession class implements. If we’re driving a scanning instance instead of a dispatcher (set in the constructor when the ArachniRPCSession was created), we send the instance a shutdown command ➐ to clean up the scanning instance but leave the dispatcher running.

The ExecuteCommand() Method

The ExecuteCommand() method shown in Listing 12-13 wraps all the functionality required to send commands and receive responses from the Arachni RPC.

public MessagePackObject ➊ExecuteCommand(string command, object[] args,
                                         string token = null)
{
➋Dictionary<string, object> = new Dictionary<string, object>();
➌message["message"] = command;
 message["args"] = args;

 if (token != null)
 ➍message["token"] = token;

 byte[] packed;
 using (MemoryStream stream = new ➎MemoryStream())
 {
   Packer packer = Packer.Create(stream);
   packer.PackMap(message);
     packed = stream.ToArray();
 }

Listing 12-13: The first half of the ExecuteCommand() method in the ArachniRPCSession class

The ExecuteCommand() method ➊ accepts three arguments: the command to execute, an object of the arguments to use with the command, and an optional argument for a token if an authentication token was provided. The method will mostly be used by the ArachniRPCManager class later. We start the method by creating a new dictionary called request to hold our command data (the command to run and the arguments for the RPC command) ➋. We then assign the message key ➌ in the dictionary the first argument passed to the ExecuteCommand() method, which is the command to run. We also assign the args key in the dictionary with the second argument passed to the method, which are the options for the command to be run. Arachni will look at these keys when we send our message, run the RPC command with the given arguments, and then return a response. If the third argument, which is optional, is not null, we assign the token key ➍ the authentication token passed to the method. These three dictionary keys (message, args, and token) are all that Arachni will look at when you send the serialized data to it.

Once we have set up the request dictionary with the information we want to send to Arachni, we create a new MemoryStream() ➎ and use the same Packer class from the Metasploit bindings in Chapter 11 to serialize the request dictionary into a byte array. Now that we have prepared the data to send to Arachni to run an RPC command, we need to send the data and read the response from Arachni. That takes place in the second half of the ExecuteCommand() method, shown in Listing 12-14.

  byte[] packedLength = ➊BitConverter.GetBytes(packed.Length);

  if (BitConverter.IsLittleEndian)
    Array.Reverse(packedLength);

 ➋_stream.Write(packedLength);
 ➌_stream.Write(packed);

  byte[] respBytes = ➍ReadMessage(_stream);

  MessagePackObjectDictionary resp = null;
  try
  {
    resp = Unpacking.UnpackObject(respBytes).Value.AsDictionary();
  }
 ➎catch
  {
    byte[] decompressed = DecompressData(respBytes);
    resp = Unpacking.UnpackObject(decompressed).Value.AsDictionary();
  }

  return resp.ContainsKey("obj") ? resp["obj"] : resp["exception"];
}

Listing 12-14: The second half of the ExecuteCommand() method in the ArachniRPCSession class

Since the Arachni RPC stream uses a simple protocol to communicate, we can easily send our MSGPACK data to Arachni, but we need to send Arachni two pieces of information, not just the MSGPACK data. We first need to send Arachni the size of the MSGPACK data as a 4-byte integer in front of the MSGPACK data. This integer is the length of the serialized data in each message and tells the receiving host (in this case, Arachni) how much of the stream needs to be read in as part of the message segment. We need to get the bytes for the length of the data, so we use BitConverter.GetBytes() ➊ to get the 4-byte array. The length of the data and the data itself need to be written to the Arachni stream in a certain order. We first write the 4 bytes representing the data’s length to the stream ➋ and then write the full serialized message to the stream ➌.

Next, we need to read the response from Arachni and return the response to the caller. Using the ReadMessage() method ➍, we take the raw bytes of the message from the response and attempt to unpack them into a MessagePackObjectDictionary in a try/catch block. If the first attempt is unsuccessful, that means the data is compressed using Gzip, so the catch block ➎ takes over. We decompress the data and then unpack the decompressed bytes into a MessagePackObjectDictionary. Finally, we return either the full response from the server or an exception if an error has occurred.

The ArachniRPCManager Class

The ArachniRPCManager class is considerably simpler than the ArachniRPCSession class, as shown in Listing 12-15.

public class ArachniRPCManager : IDisposable
{
  ArachniRPCSession _session;
  public ArachniRPCManager(➊ArachniRPCSession session)
  {
    if (!session.IsInstanceStream)
      throw new Exception("Session must be using an instance stream");

    _session = session;
  }

  public MessagePackObject ➋StartScan(string url, string checks = "*")
  {
    Dictionary<string, object>args = new Dictionary<string, object>();
    args["url"] = url;
    args["checks"] = checks;
    args["audit"] = new Dictionary<string, object>();
    ((Dictionary<string, object>)args["audit"])["elements"] = new object[] { "links", "forms" };

    return _session.ExecuteCommand(➌"service.scan", new object[] { args }, _session.Token);
  }

  public MessagePackObject ➍GetProgress(List<uint> digests = null)
  {
    Dictionary<string, object>args = new Dictionary<string, object>();
    args["with"] = "issues";
    if (digests != null)
    {
      args["without"] = new Dictionary<string, object>();
      ((Dictionary<string, object>)args["without"])["issues"] = digests.ToArray();
    }
    return _session.➎ExecuteCommand("service.progress", new object[] { args }, _session.Token);
  }

  public MessagePackObject ➏IsBusy()
  {
    return _session.ExecuteCommand("service.busy?", new object[] { }, _session.Token);
}

  public void Dispose()
  {
  ➐_session.Dispose();
  }
}

Listing 12-15: The ArachniRPCManager class

First, the ArachniRPCManager constructor accepts an ArachniRPCSession ➊ as its only argument. Our manager class will only implement methods for a scanning instance, not a dispatcher, so if the session passed in is not a scanning instance, we throw an exception. Otherwise, we assign the session to a local class variable for use in the rest of the methods.

The first method we create in the ArachniRPCManager class is the StartScan() method ➋, which accepts two arguments. The first argument, which is required, is a string of the URL Arachni will scan. The second argument, which is optional, defaults to running all checks (such as XSS, SQL injection, and path traversal, for example), but it can be changed if the user wants to specify different checks in the options passed to StartScan(). To determine which checks are run, we build a new message to send to Arachni by instantiating a new dictionary using the url and checks arguments passed to the StartScan() method and audit, which Arachni will look at to determine what kind of scan to perform when we send the message. Finally, we send the message using the service.scan command ➌ and return the response to the caller.

The GetProgress() method ➍ accepts a single optional argument: a list of integers that Arachni uses to identify reported issues. We’ll talk more about how Arachni reports issues in the next section. Using this argument, we build a small dictionary and pass it to the service.progress command ➎, which will return the current progress and status of the scan. We send the command to Arachni and then return the result to the caller.

The last important method, IsBusy() ➏, simply tells us whether the current scanner is performing a scan. Finally, we clean it all up with Dispose() ➐.

Putting It All Together

Now we have the building blocks to drive Arachni’s RPC to scan a URL and report the results in real time. Listing 12-16 shows how we glue all the parts together to scan a URL with the RPC.

public static void Main(string[] args)
{
  using (ArachniRPCSession session = new ➊ArachniRPCSession("127.0.0.1",
                                     7331, true))
  {
    using (ArachniRPCManager manager = new ArachniRPCManager(session))
    {
      Console.➋WriteLine("Using instance: " + session.InstanceName);
      manager.StartScan("http://demo.testfire.net/default.aspx");
      bool isRunning = manager.IsBusy().AsBoolean();
      List<uint> issues = new List<uint>();
      DateTime start = DateTime.Now;
      Console.WriteLine("Starting scan at " + start.ToLongTimeString());
     ➌while (isRunning)
      {
        Thread.Sleep(10000);
        var progress = manager.GetProgress(issues);
        foreach (MessagePackObject p in
                      progress.AsDictionary()["issues"].AsEnumerable())
        {
          MessagePackObjectDictionary dict = p.AsDictionary();
          Console.➍WriteLine("Issue found: " + dict["name"].AsString());
          issues.Add(dict["digest"].AsUInt32());
        }

        isRunning = manager.➎IsBusy().AsBoolean();
      }
      DateTime end = DateTime.Now;
     ➏Console.WriteLine("Finishing scan at " + end.ToLongTimeString() +
                   ". Scan took " + ((end - start).ToString()) + ".");
    }
  }
}

Listing 12-16: Driving Arachni with the RPC classes

We start the Main() method by creating a new ArachniRPCSession ➊, passing the host and port for the Arachni dispatcher, as well as true as the third argument to automatically get a new scanning instance. Once we have the session and manager classes and are connected to Arachni, we print our current instance name ➋, which should just be the unique ID we generated when we created the scanning instance to connect to it. We then start the scan by passing the test URL to the StartScan() method.

Once the scan is started, we can watch it until it’s finished and then print the final report. After creating a few variables such as an empty list, which we’ll use to store the issues that Arachni reports back, and the time when the scan started, we begin a while loop ➌, which will loop until isRunning is false. Within the while loop, we call GetProgress() to get the current progress of our scan; then we print ➍ and store any new issues found since we last called GetProgress(). We finally sleep for 10 seconds and then call IsBusy() ➎ again. We then start the process all over again until the scan is finished. When all is said and done, we print a small summary ➏ of how long the scan took. If you look at the vulnerabilities reported by your automation (my truncated results are shown in Listing 12-17) and the original Arachni scans we performed by hand at the beginning of the chapter, they should match up!

$ mono ./ch12_automating_arachni.exe
Using instance: 1892413b-7656-4491-b6c0-05872396b42f
Starting scan at 8:58:12 AM
Issue found: Cross-Site Scripting (XSS)➊
Issue found: Common directory
Issue found: Backup file➋
Issue found: Missing 'X-Frame-Options' header
Issue found: Interesting response
Issue found: Allowed HTTP methods
Issue found: Interesting response
Issue found: Path Traversal➌
--snip--

Listing 12-17: Running the Arachni C# classes to scan and report on a sample URL

Because we are running Arachni with all the checks enabled, this site will report a lot of vulnerabilities! In just the first 10 or so lines, Arachni reported an XSS vulnerability ➊, a backup file with potentially sensitive information ➋, and a path traversal weakness ➌. If you wanted to limit the checks Arachni performs to just an XSS vulnerability scan, you could pass a second argument to StartScan with the string xss* (the default value for the argument is *, which means “all checks”), and Arachni would only check for and report any XSS vulnerabilities found. The command would end up looking like the following line of code:

manager.StartScan("http://demo.testfire.net/default.aspx", "xss*");

Arachni supports a wide variety of checks, including SQL and command injection, so I encourage you to read the documentation on the supported checks.

Conclusion

Arachni is an incredibly powerful and versatile web application scanner that should be a tool in any serious security engineer or pentester’s arsenal. As you have seen in this chapter, you can easily drive it in both simple and complex scenarios. If you only need to scan a single application regularly, the HTTP API might be enough for you. However, if you find yourself constantly scanning new and different applications, the ability to spin up scanners at will may be the best way for you to distribute your scans and prevent bottlenecking.

We first implemented a set of simple classes that interfaced with the Arachni REST API in order to kick off, watch, and report on a scan. Using the base HTTP libraries in our toolset, we were able to easily build modular classes to drive Arachni.

Once we finished the simpler REST API, we took Arachni a step further to drive it via the MSGPACK RPC. Using a couple of open source third-party libraries, we were able to drive Arachni with some of its more powerful features. We used its distributed model to create a new scanning instance with the RPC dispatcher, and then we scanned a URL and reported the results in real time.

Using either of these building blocks, you can incorporate Arachni into any SDLC or continuous integration system to ensure the quality and security of the web applications being used or built by you or your organization.