Repository navigation
keyed harness: break a stale build cache that shipped six identical images - #2897
Merged
Merged
Conversation
…mages Two merges to main (#2895, #2896) each triggered a build that reported SUCCESS, and the container really was replaced: /proc/1 showed a boot at 18:21:35 and pid 1 was /app/monitor. But grepping the running binary for symbols from both commits returned zero on all four of them. The builder was serving a cached image whose `COPY . .` layer predated the changes, so six deployments shipped the same binary. A BUILD_REF arg between the source copy and the go build gives the layers below something that changes. The echo is there so the arg is actually consumed; a bare ARG is not a cache key. Also recording the diagnostic that cost an hour, because it will recur: the binary's mtime is NOT a way to tell whether a deploy landed. It comes from the image layer, so it reads old even in a container that booted a minute ago. I read 16:35 six times and concluded nothing was deploying, which was wrong — the containers were fine, the image was stale. Grep the running binary for a string you just added. That is the test that works.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
keyed harness: break a stale build cache that shipped six identical images
Two merges to main (#2895, #2896) each triggered a build that reported SUCCESS,
and the container really was replaced: /proc/1 showed a boot at 18:21:35 and
pid 1 was /app/monitor. But grepping the running binary for symbols from both
commits returned zero on all four of them. The builder was serving a cached
image whose
COPY . .layer predated the changes, so six deployments shippedthe same binary.
A BUILD_REF arg between the source copy and the go build gives the layers below
something that changes. The echo is there so the arg is actually consumed; a
bare ARG is not a cache key.
Also recording the diagnostic that cost an hour, because it will recur: the
binary's mtime is NOT a way to tell whether a deploy landed. It comes from the
image layer, so it reads old even in a container that booted a minute ago. I
read 16:35 six times and concluded nothing was deploying, which was wrong — the
containers were fine, the image was stale. Grep the running binary for a string
you just added. That is the test that works.