Skip to content

Micrometer metrics enhancement requests #2974

New issue
New issue
Closed
#3154
Closed
Micrometer metrics enhancement requests#2974
#3154
Labels
metrics-epic
Milestone
5.3

Description

@tapwaterbuffalo
tapwaterbuffalo
opened on Sep 29, 2025

A few enhancement requests for the micrometer metrics:

  1. Currently, reconciler metrics use operator.sdk.reconciliations.executions.<reconciler name> as the meter name, however when building a generic dashboard for java operator metrics with static panel expressions the metric names must be well known, I think I would suggest using the reconciler name as another tag instead. (Similar comment for operator.sdk.<map name>.size). Then you could perhaps provide a generic Grafana dashboard to evaluate operator health for a few different flavors, like prometheus.
  2. The reconciliations_retries_last and reconciliations_retries_number tags on the operator_sdk_reconciliations_* metric would be better characterized as another two gauges as they are metrics in their own right; When working with metrics series, each change in reconciliations_retries_last and reconciliations_retries_number would appear as a new series as the tags/labels have changed. When detecting problems with a reconciler, I would prefer to trigger on the metric value for the retries and last rather than the label value for the retries and last.
  3. Histogram generation between Micrometer and Prometheus produces Prometheus Native histogram output, it would be nice if we could control whether it uses Native or Classic histograms. The buckets for native histograms are automatically generated and make it difficult to dashboard. Being able to configure buckets like 0.1, 0.5, 1, 2, 5, 10, 20, 30, Inf seconds would be nice as generally we want to tell whether we are within a particular SLA

See the default output below:

# HELP operator_sdk_controllers_execution_reconcile_seconds  
# TYPE operator_sdk_controllers_execution_reconcile_seconds histogram
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.001"} 0
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.001048576"} 0
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.001398101"} 0
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.001747626"} 0
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.002097151"} 0
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.002446676"} 0
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.002796201"} 0
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.003145726"} 0
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.003495251"} 0
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.003844776"} 0
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.004194304"} 0
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.005592405"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.006990506"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.008388607"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.009786708"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.011184809"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.01258291"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.013981011"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.015379112"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.016777216"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.022369621"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.027962026"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.033554431"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.039146836"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.044739241"} 1
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.050331646"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.055924051"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.061516456"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.067108864"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.089478485"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.111848106"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.134217727"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.156587348"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.178956969"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.20132659"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.223696211"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.246065832"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.268435456"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.357913941"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.447392426"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.536870911"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.626349396"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.715827881"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.805306366"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.894784851"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="0.984263336"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="1.073741824"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="1.431655765"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="1.789569706"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="2.147483647"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="2.505397588"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="2.863311529"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="3.22122547"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="3.579139411"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="3.937053352"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="4.294967296"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="5.726623061"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="7.158278826"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="8.589934591"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="10.021590356"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="11.453246121"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="12.884901886"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="14.316557651"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="15.748213416"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="17.179869184"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="22.906492245"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="28.633115306"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="30.0"} 2
operator_sdk_controllers_execution_reconcile_seconds_bucket{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1",le="+Inf"} 2
operator_sdk_controllers_execution_reconcile_seconds_count{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1"} 2
operator_sdk_controllers_execution_reconcile_seconds_sum{controller="tomcatreconciler",resource_group="tomcatoperator.io",resource_kind="Tomcat",resource_name="test-tomcat1",resource_namespace="default",resource_scope="namespace",resource_version="v1"} 0.051458916

Activity

  1. csviri commented on Sep 30, 2025

    @csviri
    csviri
    on Sep 30, 2025
    Collaborator

    Hi @tapwaterbuffalo thank you for your input.

    Then you could perhaps provide a generic Grafana dashboard to evaluate operator health for a few different flavors, like prometheus.

    Yes, this I'm having long time on my mind. Will create a separate issue for this. #2975

    Do you plan to prepare a PR for the other propositions?

  2. csviri
    added this to the 5.3 milestone on Sep 30, 2025
  3. csviri
    added
    metrics-epic
    on Sep 30, 2025

  4. csviri commented on Sep 30, 2025

    @csviri
    csviri
    on Sep 30, 2025
    Collaborator

    Histogram generation between Micrometer and Prometheus produces Prometheus Native histogram output, it would be nice if we could control whether it uses Native or Classic histograms. The buckets for native histograms are automatically generated and make it difficult to dashboard. Being able to configure buckets like 0.1, 0.5, 1, 2, 5, 10, 20, 30, Inf seconds would be nice as generally we want to tell whether we are within a particular SLA

    I agree that we should make this customizable

  5. csviri commented on Feb 9, 2026

    @csviri
    csviri
    on Feb 9, 2026
    Collaborator

    While working on the dashboards / otel integration basically came to the same conclusion:
    #3154

    Will fix those for the v5.3

  6. csviri
    linked a pull request that will close this issue[WIP] improve: micrometer metrics improvements #3164on Feb 9, 2026
  7. csviri
    linked a pull request that will close this issuefeat: metricsV2 + oTel + prometheus sample and Grafana dashboard #3154on Feb 10, 2026

  8. csviri commented on Feb 11, 2026

    @csviri
    csviri
    on Feb 11, 2026
    Collaborator

    @tapwaterbuffalo at the end I ended up writing a v2 version of the metrics, if you have the bandwith pls check it in the related PR, thank you!
    #3154

    see MicrometerMetricsV2

  9. csviri
    closed this as completedon Mar 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    metrics-epic

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions