Splunk is one of the most widely used data mining tools. With its data mining and digging capabilities, engineers can take actions based upon decisions. While it is useful in various aspects, here we will see an example of Splunk being used as a Syslog server, with our test router sending a message (as syslog) to this server, and how from automation we can query results from Splunk for these syslogs and take actions.
This is an important part of automation, since based upon certain events (alerts and syslogs), engineers need to perform automated tasks, like self healing, or even triggering emails or using third-party tools to create tickets for various teams to work on.
Here we will see the basic implementation and configuration of Splunk as a Syslog server:
- After downloading and installing Splunk , it can be accessed from the URL http://localhost:8000/en-US/account/login?return_to=%2Fen-US%2F as we can see in the following screenshot:
- Once we login, we create a listener listed to syslogs (in our case we use the TCP protocol and keep the default port 514 open):
Once the configuration is done for TCP port 514 on Splunk (listening for syslog messages), ensure any local firewall on the server is allowing inbound packets to TCP port 514, and our machine is ready to access syslogs from network devices on TCP port 514).
- Configure the router to send syslogs. We apply the following commands on the router to enable logging (In our case the IP for the Syslog server is 192.168.255.250):
config t
logging host 192.168.255.250 transport tcp port 514
logging buffered informational
exit
This configures the router to send syslogs to the given IP address on TCP protocol over port 514. Additionally, we are also stating to log only information syslog messages on the router.
- Once done, for confirmation we can try to perform a shutdown and no shutdown of any interface (Loopback0 in our case), and see the log using the show logging command on the router:
R2#show logging
Syslog logging: enabled (11 messages dropped, 0 messages rate-limited,
0 flushes, 0 overruns, xml disabled, filtering disabled)
Console logging: level debugging, 26 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level informational, 7 messages logged, xml disabled,
filtering disabled
Logging Exception size (4096 bytes)
Count and timestamp logging messages: disabled
No active filter modules.
Trap logging: level informational, 30 message lines logged
Logging to 192.168.255.250(global) (tcp port 514,audit disabled, link up), 30 message lines logged, xml disabled,
filtering disabled
Log Buffer (4096 bytes):
*Mar 1 01:02:04.223: %SYS-5-CONFIG_I: Configured from console by console
*Mar 1 01:02:10.275: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.255.250 started - reconnection
*Mar 1 01:02:32.179: %LINK-5-CHANGED: Interface Loopback0, changed state to administratively down
*Mar 1 01:02:33.179: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to down
*Mar 1 01:02:39.303: %SYS-5-CONFIG_I: Configured from console by console
*Mar 1 01:02:39.647: %LINK-3-UPDOWN: Interface Loopback0, changed state to up
*Mar 1 01:02:40.647: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up
An important aspect to confirm if the router is sending syslogs is the line tcp port 514, audit disabled, link up, which confirms that the router is sending syslog traffic to the Syslog server.
- Here is the raw output on Splunk for the syslog that is generated:
As we see in the New Search section we can write queries to fetch the exact data that we want. In our case we wanted to see only the log from our router with the Interface Loopback0 down messages, hence we wrote the query:
host="192.168.255.248" "Interface Loopback0, changed state to down"
- Now let us see the code from Python that we can write to fetch the same information using a script:
import requests
import json
from xml.dom import minidom
username="admin"
password="admin"
### For generating the session key ####
url = 'https://localhost:8089/services/auth/login'
headers = {'Content-Type': 'application/json'}
data={"username":username,"password":password}
requests.packages.urllib3.disable_warnings()
r = requests.get(url, auth=(username, password), data=data, headers=headers,verify=False)
sessionkey = minidom.parseString(r.text).getElementsByTagName('sessionKey')[0].childNodes[0].nodeValue
#### For executing the query using the generated sessionkey
headers={"Authorization":"Splunk "+sessionkey}
data={"search":'search host="192.168.255.248" "Interface Loopback0, changed state to down"',"output_mode":"json"}
r=requests.post('https://localhost:8089/servicesNS/admin/search/search/jobs/export',data=data , headers=headers,verify=False);
print (r.text)
In the first section, we query the API of Splunk to fetch the authentication session key (or token) to run our queries and get results. Once we have the session key (extracted from the XML output), we create a header and using requests.post we execute our query. The data variable contains our query in the following format:
{"search":'search host="192.168.255.248" "Interface Loopback0, changed state to down"'}
In other words, if we take this in a variable (named Search) , and provide the result as a value to that variable, it would look like below:
Search='search host="192.168.255.248" "Interface Loopback0, changed state to down"'
Additionally we also send another option of output_mode as JSON , since we want the output in JSON (some other values can be CSV or XML).
Executing the same will get the following output:
As we see in the preceding output, we are now retrieving and displaying the value in JSON format.
We will stop our example here, but to enhance this script, this result can now become a trigger on which we can add additional methods or logic to decide on the trigger for further actions. By this logic, we can have self-healing scripts that find out the data (as a trigger), evaluate the trigger (identify it actionable), and take actions based upon further logic.