The default behavior of the logging module puts messages into the queue with the Queue.put_nowait() method. The advantage of this is that it allows the producers to run without the delays associated with logging. The disadvantage of this is that messages will get lost if the queue is too small to handle a very large burst of logging messages.

We have the following two choices to gracefully handle a burst of messages:

• We can switch from Queue to SimpleQueue. SimpleQueue has an indefinite size. As it has a slightly different API, we'll need to extend QueueHandler to use Queue.put() instead of Queue.put_nowait().
• We can slow down the producer in the rare case that the queue is full. This is a small change to QueueHandler to use Queue.put() instead of Queue.put_nowait().

Interestingly, the same API change works for both Queue and SimpleQueue. Here's the change:

class WaitQueueHandler(logging.handlers.QueueHandler):

    def enqueue(self, record):
        self.queue.put(record)

We replaced the body of the enqueue() method to use a different method of Queue. Now, we can use SimpleQueue or Queue. If we use Queue, it will wait until the queue is full, preventing the loss of logging messages. If we use SimpleQueue, the queue will silently expand to hold all the messages.

Here's the revised producer class:

class Log_Producer_2(Log_Producer): 
    handler_class = WaitQueueHandler 

This class uses our new WaitQueueHandler. Otherwise, the producer is identical to the previous version.

The rest of the script to create Queue and start the consumer is identical. The producers are instances of Log_Producer_2, but otherwise, the script to start and join remains identical to the first example.

This variation runs more slowly, but never loses a message. We can improve the performance by creating a larger queue capacity. If we create a queue with a capacity of 1,020 messages, the performance is maximized because that's the largest possible size of a burst. Finding an optimal queue capacity requires some experimentation. While the exact size depends on the operating system, a size of 30, for example, doesn't lose too many messages. The relative performance of the producer and consumer is important. To see the effects, change the sleep time in the producer to larger or smaller numbers. Also, it can be helpful to experiment by changing the number of producers from 10 to 100.