A Linux-based system makes it simple to monitor the CPU, memory, and disk. There are system files that get continuously updated with this information and numerous tools to read them. Commands such as ;top will let you follow all the running processes and sort them by RAM or CPU usage.

In Python, the psutil (https://pythonhosted.org/psutil) ;project is a cross-platform library you can use to get all this info programmatically.

Combined with the graypy package, you can write a small script to send system metrics to Graylog continuously.

In the following example, an asyncio loop sends the CPU usage in percent every second to Graylog:

    import psutil 
    import asyncio 
    import signal 
    import graypy 
    import logging 
    import json 
 
    loop = asyncio.get_event_loop() 
    logger = logging.getLogger('sysmetrics') 
  
    def _exit(): 
        loop.stop() 
 
    def _probe(): 
        info = {'cpu_percent': psutil.cpu_percent(interval=None)} 
        logger.info(json.dumps(info)) 
        loop.call_later(1., _probe) 
 
    loop.add_signal_handler(signal.SIGINT, _exit) 
    loop.add_signal_handler(signal.SIGTERM, _exit) 
    handler = graypy.GELFHandler('localhost', 12201) 
    logger.addHandler(handler) 
    logger.setLevel(logging.INFO) 
    loop.call_later(1., _probe) 
 
    try: 
        loop.run_forever() 
    finally: 
        loop.close() 

Running this script as a daemon on your server will let you track its CPU usage.

The system-metrics (https://github.com/tarekziade/system-metrics/) project is roughly the same script but adds info about memory, disks, and network. If you use the pip install command, a command-line script will be available to probe your system.

Once the script runs, you can create a dashboard with a few widgets in the Graylog web app, as described in http://docs.graylog.org/en/latest/pages/dashboards.html and create an alert that will send you an email on specific conditions. Alerts in Graylog are configured on Streams, which are processing incoming messages in real-time.

To send an email when the CPU gets over 70%, you can create a stream that will collect the cpu_percent field sent by our psutil script using a stream rule:

From there, you can manage alerts for the stream and add an email one that will get triggered when the condition is met for some time.

Like we did for sending logs, we can also add custom performance metrics inside our microservice code depending on our needs.