Python comes with the logging package, which lets you stream logs to a variety of places including standard out, rotating log files, syslog, or a TCP or UDP socket.

There's even an SMTP backend. In the following example, the email_errors decorator will send an email every time an exception is happening in the decorated function. Note that the handler is doing a telnet session with the SMTP server to send the email, so if there's any issue during that session, you might get a second exception when the logger.exception() function is called:

    import logging 
    from logging.handlers import SMTPHandler 
 
    host = "smtp.example.com", 25 
    handler = SMTPHandler(mailhost=host, fromaddr="[email protected]", 
                          toaddrs=["[email protected]"], 
                          subject="Service Exception") 
 
    logger = logging.getLogger('theapp') 
    logger.setLevel(logging.INFO) 
    logger.addHandler(handler) 
 
    def email_errors(func): 
        def _email_errors(*args, **kw): 
            try: 
                return func(*args, **kw) 
            except Exception: 
                logger.exception('A problem has occured') 
                raise 
    return _email_errors 
 
    @email_errors 
    def function_that_raises(): 
        print(i_dont_exist) 
 
    function_that_raises() 

If the call works, an email will be received with the full traceback enclosed.

Logging to the standard output or a log file is fine when you are developing your service, but as we said earlier, that won't scale in a distributed system.

Sending emails on errors is an improvement, but with high-traffic microservices, it's common to get the same exception a thousand times an hour. If you are spamming an email box with a lot of emails, your server IP will get blacklisted by the SMTP server and your service will be unresponsive because it will be busy sending out lots of emails.

We need something better for a distributed system. A way to collect logs from all microservices with the least overhead possible, and some user interface to visualize them.

There are several existing systems to centralize logs generated by Python applications. Most of them can receive logs in HTTP or UDP payloads, with a preference for the latter because it reduces the overhead when your app sends them.

Sentry (https://sentry.io/) is a well-known tool in the Python community for centralizing error logs and provides a nice UI to deal with tracebacks. When a problem occurs in a service, Sentry can detect and aggregate the errors. The UI has a little resolution workflow that will let people deal with the problem.

But Sentry is focused on errors and is not well suited for general logging. If you want to get logs other than errors, you need to use something else.

Another open-source solution is Graylog (http://graylog.org), which is a general logging application that comes with a powerful search engine based on Elasticsearch (https://www.elastic.co/) where the logs are stored. MongoDB (https://www.mongodb.com/) is also used to store application data.

Graylog can receive any logs via its custom logging format or alternative formats, such as plain JSON. It has a built-in collector or can be configured to work with collectors such as ;fluentd (http://www.fluentd.org/).