Logs to the rescue

I’ve always been partial to a few print statements when debugging my personal coding projects. Now, having worked on code behind an API, I have come to appreciate the benefit of switching print() for logger.info. It only takes a few lines to set the root logger configuration, which adds useful metadata to every debugging message:

import logging
import jsonlogger

logger = logging.getLogger(name)
logger.setLevel(logging.DEBUG)

# StreamHandler to stdout
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)

# Define a simple message format
log_format = "%(asctime)s [%(module)s > %(funcName)] : %(message)s"
formatter = jsonlogger.JsonFormatter(log_format)
handler.setFormatter(formatter)

logger.addHandler(handler)

Of course, there is an excellent tutorial at calmcode.io

This has become very handy when I recently took over a machine learning pipeline that was churning out thousands of logs, with Pytorch printing every update in a progress bar and cmdstanpy insisting on telling me about its chains.

In this same logging configuration file, we can turn these annoying logs off. There could be a more consistent way to do this, but this is what worked for me.

import cmdstanpy
cmdstanpy.disable_logging()

noisy_libs = ["neuralforecast", "pandas", "snowflake.connector"]
for lib in noise_libs:
    logging.getLogger(lib).setLevel(logging.WARNING)

Finally, when looking at the logs in DataDog I could see that any logs not generated by me, but by other packages like pandas, were mising the lovely context data (i.e. environment, market, model). We can fix this by adding a filter to the base logger, which then adds our context to all logs in the stdout output.

class ContextFilter(logging.Filter):
    def __init__(self, context):
        super().__init__()
        self.context = context
    
    def filter(self, record):
        for key, value in self.context.items():
            setattr(record, key, value)
        return True

root_logger = logging.getLogger()
logHandler = logging.StreamHandler(stream = sys.stdout)

logHandler.addFilter(ContextFilter(context))