Testing ergonomics: stubbing [solved]; reliance on env vars to set runtime behavior [wtf]

[Context / level-setting: I've written mostly Ruby for 20 years, and am several months into a lambda project using Python. I haven't yet built a deep understanding of how to use unittest.mock.patch or unittest.mock.MagicMock and am still mostly copy/pasting stuff that seems to have worked so far.]

I've been blithely calling statsd.increment from a concurrent AWS lambda and wondering why the reported numbers are so dang low, until a coworker clued me in to the existence of datadog_lambda.metric.lambda_metric.

So I set about replacing calls where that seemed appropriate, and immediately ran into a problem: I had some unit tests that used statsd.increment = MagicMock() to assert that metrics were sent. I didn't know how to get that working with lambda_metric, so I just commented out the tests to see what else I might have broken...

...which is when I ran into the second problem: the test suite complains failed to send, dropping 8 traces to intake at http://localhost:8126/v0.5/traces: Network error: client error (Connect)... and takes more than twice as long to exit, due to what I presume is a ~5 second timeout.

Apparently datadog.statsd is perfectly content to drop method calls on the floor during tests, but lambda_metric isn't?

Things I've done before coming here to ask for help:

found an old request for a testing mode (issue Would be great to have a test mode that stub out the library #17), but couldn't really tell what was going on in the referenced PR. This stuff might be obvious to a more experienced Pythonista, but see above re: "mostly Ruby."
looked for documentation and didn't see any flashing neon signs saying "hey, noob, here's how to stub this out so your test suite doesn't get bogged down trying to report metrics to nobody."
cloned the repo so I could use ripgrep to try to find examples to follow, but the tests are using subclasses of unittest.TestCase and my projects are using pytest with just bare functions, and those setUp methods are... uh... doing a lot? And it's extremely not obvious what any of this is doing or which bits (if any) are relevant.

Throw a nerd a clue? :)

@geeksam

Hey @geeksam, I'm more than happy to point you in the right direction. Can you share an example of one of the tests you're trying to mock?

lambda_metric is a function, so patch the name your handler looks up. For an application module containing from datadog_lambda.metric import lambda_metric, a bare pytest test can use:

from unittest.mock import patch
import app_direct

with patch.object(app_direct, "lambda_metric", autospec=True) as metric:
    assert app_direct.handler({"records": ["alpha", "beta"]}) == {"processed": 2}
    metric.assert_called_once_with("records.processed", 2, tags=["queue:example"])

If the application instead uses import datadog_lambda.metric as metric, patch app_module.metric.lambda_metric. Patching the original library function after a direct import leaves the application's local binding unchanged; Python documents this under where to patch.

The /v0.5/traces error is a separate tracer-export issue. For local unit tests, set DD_TRACE_ENABLED=false, DD_INSTRUMENTATION_TELEMETRY_ENABLED=false and DD_REMOTE_CONFIGURATION_ENABLED=false before starting pytest, and use -p no:ddtrace if you are not using its pytest integration. A fixture can run after tracing has already been imported.

This runnable example covers both import styles, exact metric assertions and real process-exit checks with a local receiver. Its 12 tests passed with datadog-lambda 8.130.0 / ddtrace 4.16.1, including delayed and unavailable local endpoints. One qualification: that ddtrace release still performs native GET /info discovery with tracing disabled; the normal/unavailable checks exited in about 0.2s, while intentionally stalled discovery still took 3.2s. This is not a fully offline or zero-latency mode. The exact patch target for your project depends on its import line.

@purple4reina

@purple4reina oh, hey, I recognize your name from some other o11y company I worked at briefly!

I have a couple of tests that nail down the "we send metrics" part of the code that look like this (irrelevant details elided):

def test_sends_stats_when_passed_stat_base(some_fixture):
    statsd.increment = MagicMock()
    # run the code under test
    statsd.increment.assert_called_with("wibble.exception", tags=["exception:Wibble"])

[EDIT: I do not know whether the statsd.increment attribute gets reset after that test runs; I'm just sort of hoping for the best so far, because that's the behavior I expect from RSpec's built-in test double helpers and I haven't yet taken the time to dig into this aspect of Python and pytest.]

@tevinch

@tevinch holy cats, that worked example is amazing. Thank you! I'll go study it for at least an hour before I come back with questions. 🫶

Okay! "patch the name your handler looks up" was really useful guidance. Again, thank you. Also, I was able to use the @patch (technically @mock.patch decorator to avoid the visual noise / cognitive overhead of with patch.object(...), which brought a little calm to my Ruby-addled brain. 😉

Now I just need to figure out how to avoid having to specify a bunch of environment variables on the command line—either directly or via an export that I have to remember to run in each new shell.

I know it's possible to use dotenv in tests/conftest.py to "set it and forget it," because that worked nicely on one of my sub-projects—but not another. (I'm working in a project that has several different lambda projects in their own directories, so naturally they've each accumulated their own individual setup quirks.) I'll keep digging and see if I can figure that out.

Changed the title to indicate design feedback to maintainers about the additional stumbling block.

I realize that controlling the app via env vars facilitates ops-level deployment work, and that tools like these necessarily have to do a lot of weird language-level gyrations in order to perform their basic functions—but I've come to believe that all of a library's behavior MUST be configurable from code. (And, obviously, that the configuration API be discoverable. Paradoxically, DD has so much documentation that I often have a hard time finding find the bits I need.)

Navigation Menu

Repository navigation

Testing ergonomics: stubbing [solved]; reliance on env vars to set runtime behavior [wtf] #879

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Testing ergonomics: stubbing [solved]; reliance on env vars to set runtime behavior [wtf] #879

Description

Activity

purple4reina commented on Oct 29, 2026

tevinch commented on Oct 29, 2026

geeksam commented on Oct 29, 2026

geeksam commented on Oct 29, 2026

geeksam commented on Oct 29, 2026

geeksam commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions