Repository navigation
gauge.set_function() doesn't work in multiprocess mode #504
Description
Multiprocess mode's collect() reads the registry files and aggregates metrics that have been written to prometheus_multiproc_dir.
This doesn't work with gauge.set_function() which does not record its value. The provided function is just called during collection.
That means with the following code:
registry = CollectorRegistry()
Gauge("test", "test", registry=self._registry).set_function(lambda: 100)
multiprocess.MultiProcessCollector(registry)
The output will be:
# HELP test test
# TYPE test gauge
test 100.0
# HELP test Multiprocess metric
# TYPE test gauge
test{pid="10705"} 0.0
Current side effects:
- If any mode other than
allorliveallis used, thepidtag won't be included. This results in duplicate metrics being reported to Prometheus. Prometheus currently only uses the first metric it reads, which is non-deterministic due to iteration over the registry's dictionary. - If
registry=Noneto avoid double reporting, only the default value of0.0is reported. - Current way to work around it is to use the mode
alland to ignore gauges in Prometheus and with the tagpid.
Proposal:
I'm not sure how you could incorporate set_function into the multiprocess registry and I'm not convinced how useful of a feature it would be. Is it reasonable to add a new multiprocess_mode: exclude which would prevent the incorrect 0.0 value being reported? Or would it be better to just add documentation to recommend using two independent registries?
Activity
brian-brazil commentedon Jan 29, 2020
This classifies as a custom collector, which it's not possible to make work with multiprocess mode.
MatthewMaclean commentedon Jan 30, 2020
Makes sense, I'll look/ask about ways to work around that in the mailing list.
Update README.md
gauge.set_function is a custom collector
hdost commentedon Dec 6, 2022
Given that multiprocess is getting some traction is it worth re-opening this?
csmarchbanks commentedon Dec 14, 2022
I'm still not sure how this could be done in multiprocess mode. You would need to run the function for each process which is not a pattern we have today and would need to integrate with whatever is starting the process I believe. If there is a simpler way I am all ears, but otherwise I think it should probably be kept as closed.
hdost commentedon Dec 18, 2022
Perhaps just documentation or a failure logged. Right now you just have to wonder
csmarchbanks commentedon Dec 19, 2022
👍 Documentation on the function is a good idea to avoid confusion.
Add note on gauge.set_function not working with mp, see prometheus#504
Add note on gauge.set_function not working with mp, see prometheus#504
Add note on gauge.set_function not working with mp, see #504 (#1098)