QueueLogger and Python JSON Logger

First posted on: 2017/03/01

Last modified: 2019/06/17,1b90ad5

Categories: Python

Using QueueLogger with Python JSON Logger

When logging from multiple processes (via multiprocessing module), using QueueHandler is one approach with Python 2.

QueueHandler however sets exc_info attribute of a LogRecord to None since it is not "pickleable" (more on this later). This becomes a problem when you use python-json-logger to format your logs as JSON since it relies on exc_info being set. The result is you don't get exc_info in your logs. This is no however no longer true since this PR was merged. Sample Code.

Solution #1: Modify Python JSON logger

The first solution is to look for exc_text which is set by the standard Formatter class as per my PR and set that as the value of exc_info which means at least we get the string representation of the traceback. This PR is now merged.

Solution #2: Subclass QueueHandler

The second solution is it to subclass logutils.queue.QueueHandler as follows and add pickling support for exc_info via tblib:

# Bring in support for serializing/deserializing tracebacks
# needed by QueueHandler
from tblib import pickling_support
pickling_support.install()


class QueueHandlerWithTraceback(logutils.queue.QueueHandler):
    """ QueueHandler with support for pickling/unpickling
        Tracebacks via tblib. We only override the prepare()
        method to *not* set `exc_info=None`
    """
    def __init__(self, *args, **kwargs):
        logutils.queue.QueueHandler.__init__(self, *args, **kwargs)

    def prepare(self, record):
        self.format(record)
        record.msg = record.message
        record.args = None
        return record

Instead of logutils.queue.QueueHandler, we will then use QueueHandlerWithTraceback instead above (Sample Code).