Page MenuHomePhabricator

Make api_db and webapp compliant with logging format requirements
Open, Needs TriagePublic

Description

When designing a program for use in WMF Production environments, there are certain logging standards that we need to meet, by sending the logs through a supported interface (systemd) and should be of a certain format
The standards can be found here
https://wikitech.wikimedia.org/wiki/Logstash/Interface

For Python program, we should follow the example as shown in the page (code below) -- by using a log structurer (pythonjsonlogger)

This ticket will involve implementing this log structurer functionality in both api_db and webapp (webapp is the proposed name for the Flask app that would serve the frontend)

import logging
import logging.config
from pythonjsonlogger import jsonlogger


class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def add_fields(self, log_record, record, message_dict):
        super(CustomJsonFormatter, self).add_fields(log_record, record, message_dict)
        log_record['level'] = record.levelname.upper()


class StructuredLoggingHandler(logging.StreamHandler):
    def __init__(self, rsyslog=False):
        super(StructuredLoggingHandler, self).__init__()
        if rsyslog:
            prefix = '@cee: '
        else:
            prefix = ''
        self.formatter = CustomJsonFormatter(prefix=prefix)


# Demo code below
if __name__ == '__main__':
    logging.config.dictConfig({
        'version': 1,
        'disable_existing_loggers': False,
        'root': {
            'handlers': ['demo'],
            'level': 'DEBUG'
        },
        'handlers': {
            'demo': {
                'class': 'logger_demo.StructuredLoggingHandler',
                'rsyslog': True
            }
        }

    })
    logging.info('It\'s log!')

Tasks

  • Implement JSON based logger in api_db according to above
  • Implement the same for webapp as well.