Skip to content

make validate still does not run five checks CI runs #175

Closed

Description

@Gerrrt

Follow-up to #68, which fixed the lint list and the SNMP asymmetry. The same
claim in scripts/validate.sh:3 — "Everything CI runs, runnable locally" — is
still not true, for five checks that live as inline shell inside ci.yml and
have no local equivalent at all.

Line numbers are as of #174.

CI-only, no local equivalent

  • ci.yml:92 — Verify image versions are not duplicated outside compose.yaml
  • ci.yml:106 — Verify no image uses a floating tag
  • ci.yml:120 — Verify every image is pinned by digest
  • ci.yml:252 — Parse every dashboard PromQL expression
  • ci.yml:311 — Assert SOPS files are actually encrypted

Confirmed absent from validate.sh by grepping for each one's distinguishing
token (not duplicated outside, floating tag, pinned by digest,
emit-promql, sops:) — all five miss.

The last one is the sharpest. A SOPS file that is not actually encrypted is a
committed plaintext secret, and the only thing that checks is a job you cannot
run before pushing.

The same check written twice, already diverging

"No decrypted artefact is tracked" exists in both places, as two independent
implementations, and they do not agree:

ci.yml:300 loops five patterns through git ls-files | grep -E "(^|/)…".
validate.sh:354 hand-writes five conditions instead:

if git ls-files --error-unmatch "${STACK}/.env" >/dev/null 2>&1 \
   || git ls-files "${STACK}/snmp-exporter/.rendered" | grep -q . \
   || git ls-files | grep -q '\.purge-secrets\.txt' \
   ...

So they cover different things:

tracked file CI make validate
stacks/observability/.env caught caught
stacks/other-stack/.env caught missed — only ${STACK}/.env
some/other/.rendered/x caught missed — only ${STACK}/snmp-exporter/.rendered
nested/certificates/key.pem caught missed — anchored ^certificates/

Every miss is a decrypted secret that passes locally and fails on push, which is
the better direction to fail in — but it is still two copies drifting, and the
second stack this repo grows makes the first row of misses real.

Why this was split out of #68

#68's fix was a shared tool list. These are inline CI shell, so the fix is
different in kind: extract each into a script both callers run, the way
seed-validation-env.sh, snmp-targets.sh --check and now scripts/lint.sh
already are. Doing it in #174 would have tripled that diff and mixed two
refactors.

Suggested shape

  • The three image greps arguably belong inside scripts/check_image_pins.py,
    which already runs in both places and already owns this argument — its
    docstring explains exactly why a grep cannot see a missing pin (make backup uses an unpinned alpine image that evades both CI pinning checks #65). Folding
    them in retires three inline greps rather than relocating them.
  • The dashboard PromQL parse needs promtool, so it wants the same
    binary-or-docker fallback validate.sh already has for promtool/amtool/alloy,
    and a counted skip when neither is available.
  • The SOPS-encrypted assert and the tracked-artefact check are pure git/grep
    with no dependencies — one script each, callable from both, deleting the
    second copy rather than fixing it.

Anything that can skip locally should take the --skips-file contract
scripts/lint.sh and scripts/check_loki_rules.sh now use, so SKIPPED stays
honest. #174 showed why that matters: with docker hidden, main printed five
SKIP lines and reported "4 skipped".

Activity

  1. added a commit that references this issue on Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions