Conversation
New ansible role runs Alloy + node_exporter as a separate compose project. Collects host metrics, all container logs, and the two Ponder /metrics endpoints (mainnet 42069, testnet 42070), pushing to the Foundation observability VM over the tailnet with cluster=dashtec labels. Requires the tailnet ACL change in foundation-iac (dashtec -> observability 9090/3100) to be applied first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three fixes from an external audit of this PR: - flush_handlers at the end of the alloy role: handlers otherwise run at end of play (hours later behind aztec_node's ready-wait, or never if a later role fails), leaving a re-rendered Alloy config silently unapplied with no re-notify on the next run. - Ponder scrape addresses move to group_vars with a comment tying them to the compose PONDER_PORT/PONDER_TESTNET_PORT defaults, so a port override has one place to update instead of silently killing the scrape. - Comment on the docker.sock mount stating plainly that :ro does not reduce API privilege — socket access is root-equivalent; accepted for log collection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Running set, restarts (CPU-counter resets), CPU and memory per container, trimmed to the four metrics the Foundation Systems dashboard uses. Alloy gets read-only cgroup/Docker mounts and the containerd socket, which current Docker needs to resolve container names. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Drop deploy-time one-offs (`compose run`, e.g. Prisma migrations) and plain `docker run` containers (no Compose project) from the container metrics, and the unused container_label_* labels. Otherwise they inflate the "expected containers" baseline on the Foundation Systems dashboard and its container-missing alert for hours after each deploy. Notes that the containerd socket mount is root-level access, like docker.sock. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an
alloyAnsible role: Grafana Alloy + node_exporter as a separate compose project, pushing to the Foundation observability VM over the tailnet withcluster=dashteclabels. It collects:/metricsendpoints (mainnet :42069, testnet :42070). The addresses live ingroup_varsnext to a note tying them to the composePONDER_PORT/PONDER_TESTNET_PORTdefaults.compose run, e.g. the Prisma migrations) and plaindocker runcontainers are filtered out, so they do not distort the "expected containers" baseline on the dashboard and its alert.The role flushes its handlers at the end, so a config change restarts Alloy before the long-running roles that follow.
Security note: the docker and containerd sockets are mounted read-only, but socket access is root-level access to the host either way;
:rodoes not limit the API. This is the accepted cost of container metrics and logs.Prerequisite: AztecProtocol/foundation-iac#28 merged and applied (the observability VM, and the tailnet ACL that lets dashtec push to it on 9090/3100).
Deploy: the normal
ansible-playbook site.ymlrun; the role is idempotent.Tested end to end on a throwaway VM with a real Ponder 0.15.11 (the version this repo pins): the Ponder metrics the dashboard uses (
ponder_sync_block,ponder_sync_block_timestamp,ponder_sync_is_realtime,ponder_rpc_request_error_total) exist and arrive with the expected labels; container metrics resolve names on current Docker (which needs the containerd socket); one-off containers produce no series.🤖 Generated with Claude Code