From 4141ecf290b019f706c84788859655308c2ddea6 Mon Sep 17 00:00:00 2001
From: Flotapponnier <160007691+Flotapponnier@users.noreply.github.com>
Date: Thu, 8 Oct 2026 20:25:39 +0200
Subject: [PATCH] keyed harness: break a stale build cache that shipped six
 identical images
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Two merges to main (#2895, #2896) each triggered a build that reported SUCCESS,
and the container really was replaced: /proc/1 showed a boot at 18:21:35 and
pid 1 was /app/monitor. But grepping the running binary for symbols from both
commits returned zero on all four of them. The builder was serving a cached
image whose `COPY . .` layer predated the changes, so six deployments shipped
the same binary.

A BUILD_REF arg between the source copy and the go build gives the layers below
something that changes. The echo is there so the arg is actually consumed; a
bare ARG is not a cache key.

Also recording the diagnostic that cost an hour, because it will recur: the
binary's mtime is NOT a way to tell whether a deploy landed. It comes from the
image layer, so it reads old even in a container that booted a minute ago. I
read 16:35 six times and concluded nothing was deploying, which was wrong — the
containers were fine, the image was stale. Grep the running binary for a string
you just added. That is the test that works.
---
 harnesses/rpc-keyed-latency/Dockerfile | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/harnesses/rpc-keyed-latency/Dockerfile b/harnesses/rpc-keyed-latency/Dockerfile
index 63108cfc1..185cbecac 100644
--- a/harnesses/rpc-keyed-latency/Dockerfile
+++ b/harnesses/rpc-keyed-latency/Dockerfile
@@ -8,6 +8,20 @@ RUN go mod download
 
 COPY . .
 
+# BUILD_REF exists to make the layers below depend on something that changes.
+#
+# 2026-10-08: two merges to main (#2895, #2896) each triggered a build that
+# reported SUCCESS, and the container was genuinely replaced — /proc/1 showed a
+# fresh boot. But `grep` on the running /app/monitor found zero occurrences of
+# any symbol from either commit. The builder was serving a cached image whose
+# `COPY . .` layer predated them, so six deploys shipped the same binary.
+#
+# The file mtime is NOT a way to detect this: it comes from the image layer, so
+# it reads as old even in a freshly started container. Checking the binary for
+# a string you just added is the test that actually works.
+ARG BUILD_REF=unset
+RUN echo "build ref: $BUILD_REF" > /app/.build-ref
+
 RUN CGO_ENABLED=0 GOOS=linux go build -o /app/monitor ./cmd/script
 
 FROM debian:bookworm-slim