Logging: how to propagate 'extra' passed to stdlib's log methods?

from google.cloud import logging                                                                                                                                                                                             
logging_client = logging.Client()  
logging_client.setup_logging() 
import logging 
text = 'Hello, world!' 
logging.warn(text, extra={'dd':1})

I expect it to send structured log, but your client strips all passed additional data.

#5888 (comment)

The Python standard library logging module is responsible for how extra is handled: its docs state:

The third keyword argument is extra which can be used to pass a dictionary which is used to populate the dict of the LogRecord created for the logging event with user-defined attributes. These custom attributes can then be used as you like. For example, they could be incorporated into logged messages.

The last bit is handled by the "formatter" using a format string (configured by you) which knows the attribute names. Becuase we don't know the attribute names, there is no safe / clean way to pick them out of the LogRecord's dictionary, which has more than a dozen "standard" attributes (and they vary across Python versions).

The most straightforward way to send structured logs is to use our logger.log_struct, e.g.:

logger = client.logger(log_name)
logger.log_struct(your_dict)

The most straightforward way to send structured logs is to use our logger.log_struct, e.g.:

It is nice, but I want to configure/hack your logger to have behavior similar to Sentry which collect all extra parameters and show them in user interface.

This is the same issue then as #6354.

One option is to replace the current Stackdriver formatter with monkeypatching or your own configuration methods. Here's what I'm using:

from raven.handlers.logging import extract_extra

def _format_stackdriver_json(record, message):
    subsecond, second = math.modf(record.created)

    payload = {
        "message": message,
        "timestamp": {"seconds": int(second), "nanos": int(subsecond * 1e9)},
        "thread": record.thread,
        "severity": record.levelname,
    }

    data, extra = extract_extra(record)
    if extra:
        payload.update(extra)

    return json.dumps(payload)

The Python standard library logging module is responsible for how extra is handled: its docs state:

The third keyword argument is extra which can be used to pass a dictionary which is used to populate the dict of the LogRecord created for the logging event with user-defined attributes. These custom attributes can then be used as you like. For example, they could be incorporated into logged messages.

The last bit is handled by the "formatter" using a format string (configured by you) which knows the attribute names. Becuase we don't know the attribute names, there is no safe / clean way to pick them out of the LogRecord's dictionary, which has more than a dozen "standard" attributes (and they vary across Python versions).

Sure there is: create a dummy LogRecord with no extra and then use vars to filter out fields in one that aren't in the other. Without having a stdlib logging compatible adapter this library is awkward to use. It's not a big deal if fields are over-included!

Navigation Menu

Repository navigation

Logging: how to propagate 'extra' passed to stdlib's log methods? #6488

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Logging: how to propagate 'extra' passed to stdlib's log methods? #6488

Description

Activity

tseaver commented on Nov 14, 2018

iho commented on Nov 14, 2018

tseaver commented on Dec 7, 2018

ewhauser commented on Dec 23, 2018

nimish commented on Jun 19, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions