Repository navigation
port the keyed connection trace to main, which is the branch Railway builds - #2895
Merged
Merged
Conversation
…builds #2892 put this on dev. The probe services deploy from the GitHub repo with no branch pinned, so they build main; the instrumentation was never compiled and four deploys produced an identical image. Nine minutes spent waiting for a log line that could not exist, and a binary whose mtime never moved off 16:35 was the clue I should have read sooner. The diagnostic itself is unchanged and off by default (PROBE_TRACE_CONN=1). What it answers: three of the four keyed providers are published 4 to 6x slower than they answer a direct, pooled, 25-sample measurement taken from inside the probe container, while GetBlock's published number matches its measured one. The residual is about 25 ms, which is the cost of a TLS handshake from these containers. Logging Reused and WasIdle per call settles whether that is connection reuse, and therefore whether the keyed cohort's published ranking is wrong on nine pages. Caching and the rotating id were ruled out separately and the false non-cacheable claim is already corrected in #2894. 65 insertions, zero deletions. go build and go vet clean.
Flotapponnier
added a commit
that referenced
this pull request
Oct 8, 2026
…mages (#2897) Two merges to main (#2895, #2896) each triggered a build that reported SUCCESS, and the container really was replaced: /proc/1 showed a boot at 18:21:35 and pid 1 was /app/monitor. But grepping the running binary for symbols from both commits returned zero on all four of them. The builder was serving a cached image whose `COPY . .` layer predated the changes, so six deployments shipped the same binary. A BUILD_REF arg between the source copy and the go build gives the layers below something that changes. The echo is there so the arg is actually consumed; a bare ARG is not a cache key. Also recording the diagnostic that cost an hour, because it will recur: the binary's mtime is NOT a way to tell whether a deploy landed. It comes from the image layer, so it reads old even in a container that booted a minute ago. I read 16:35 six times and concluded nothing was deploying, which was wrong — the containers were fine, the image was stale. Grep the running binary for a string you just added. That is the test that works.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
port the keyed connection trace to main, which is the branch Railway builds
#2892 put this on dev. The probe services deploy from the GitHub repo with no
branch pinned, so they build main; the instrumentation was never compiled and
four deploys produced an identical image. Nine minutes spent waiting for a log
line that could not exist, and a binary whose mtime never moved off 16:35 was
the clue I should have read sooner.
The diagnostic itself is unchanged and off by default (PROBE_TRACE_CONN=1).
What it answers: three of the four keyed providers are published 4 to 6x
slower than they answer a direct, pooled, 25-sample measurement taken from
inside the probe container, while GetBlock's published number matches its
measured one. The residual is about 25 ms, which is the cost of a TLS
handshake from these containers. Logging Reused and WasIdle per call settles
whether that is connection reuse, and therefore whether the keyed cohort's
published ranking is wrong on nine pages.
Caching and the rotating id were ruled out separately and the false
non-cacheable claim is already corrected in #2894.
65 insertions, zero deletions. go build and go vet clean.