You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Similar to #1, but with a non-app-framework in mind. In particular the logging functionality, but possibly others such as config.
I'm starting a new project intended to be deployed as an AWS Lambda function. Would your recommendation be to start with python-dockerflow, or stick with mozlog for now?
@hwine That's a good question, as dockerflow is by design targeted at Docker based deployment workflows I'm not sure if that should be used for AWS Lambda deployments. It'd be best to ask the authors of Dockerflow @mostlygeek (and maybe @whd, @robotblake, @jasonthomas) what they think about it.
I just did a AWS Lambda function. I didn't use this package at all. The configuration in python-dockerflow is Django-specific, so that won't fly for Lambda (unless you're doing crazy things).
The logging bits might be helpful, but logs for AWS Lambda go to Cloudwatch, so for it to be really helpful, we'd have to have something that slurps from Cloudwatch and into our logging infrastructure.
Also, if you're using Datadog, then you print the statsd data to stdout and Datadog picks it up by reading Cloudwatch. You can't have that in mozlog format.
I have some rough infrastructure in Pigeon for configuration, logging, and statsd things if that helps. I'd be game for pulling that into python-dockerflow if that helps:
The only time I would consider dockerflow for AWS lambda is if you could spin up docker containers in lambda. Definitely use mozlog for logging as all the downstream log processing is expecting it.
@mostlygeek Are you saying that logging 2.0 pulls logging events from Cloudwatch? I'm pretty sure Pigeon logs aren't going to the logging infrastructure and I was under the impression that wasn't a thing.
lambda logs go to cloudwatch logs. I don't see why those can't make it into the loggion2.0 pipeline. I don't think this is a currently supported by logging2.0 yet.
I'm new to this group -- would there be value in having a Mozilla (which has MPL) setup for "all the things". If so, adding @willkg logging and other support for Lambda might be worthwhile.
I'm pretty sure Pigeon logs aren't going to the logging infrastructure and I was under the impression that wasn't a thing.
I most certainly should be. All applications logs should eventually go into the logging pipeline. As for the timeline of doing so, that's a question for ops.
Activity
jezdez commentedon Apr 25, 2017
@hwine That's a good question, as dockerflow is by design targeted at Docker based deployment workflows I'm not sure if that should be used for AWS Lambda deployments. It'd be best to ask the authors of Dockerflow @mostlygeek (and maybe @whd, @robotblake, @jasonthomas) what they think about it.
willkg commentedon Apr 25, 2017
I just did a AWS Lambda function. I didn't use this package at all. The configuration in python-dockerflow is Django-specific, so that won't fly for Lambda (unless you're doing crazy things).
The logging bits might be helpful, but logs for AWS Lambda go to Cloudwatch, so for it to be really helpful, we'd have to have something that slurps from Cloudwatch and into our logging infrastructure.
Also, if you're using Datadog, then you print the statsd data to stdout and Datadog picks it up by reading Cloudwatch. You can't have that in mozlog format.
I have some rough infrastructure in Pigeon for configuration, logging, and statsd things if that helps. I'd be game for pulling that into python-dockerflow if that helps:
https://github.com/mozilla/socorro-pigeon/blob/735c183ecc626f1bc31648cc68f8f33870aea6c2/pigeon.py#L34
https://github.com/mozilla/socorro-pigeon/blob/735c183ecc626f1bc31648cc68f8f33870aea6c2/pigeon.py#L93
https://github.com/mozilla/socorro-pigeon/blob/735c183ecc626f1bc31648cc68f8f33870aea6c2/pigeon.py#L153
mostlygeek commentedon Apr 25, 2017
The only time I would consider dockerflow for AWS lambda is if you could spin up docker containers in lambda. Definitely use mozlog for logging as all the downstream log processing is expecting it.
willkg commentedon Apr 25, 2017
@mostlygeek Are you saying that logging 2.0 pulls logging events from Cloudwatch? I'm pretty sure Pigeon logs aren't going to the logging infrastructure and I was under the impression that wasn't a thing.
mostlygeek commentedon Apr 25, 2017
@willkg @relud is best to answer that question.
lambda logs go to cloudwatch logs. I don't see why those can't make it into the loggion2.0 pipeline. I don't think this is a currently supported by logging2.0 yet.
hwine commentedon Apr 25, 2017
I'm new to this group -- would there be value in having a Mozilla (which has MPL) setup for "all the things". If so, adding @willkg logging and other support for Lambda might be worthwhile.
jvehent commentedon Apr 26, 2017
I most certainly should be. All applications logs should eventually go into the logging pipeline. As for the timeline of doing so, that's a question for ops.
willkg commentedon Apr 27, 2017
@jvehent @mostlygeek: I had no idea. Good to know. I updated Pigeon to use python-dockerflow for mozlog-style logging.