Skip to content

PRG bytes depend on the build locale (link order from an unsorted $(wildcard)) #301

Closed

Description

@JC-000

Observed

The same commit and the same make arguments give different PRGs depending on the build environment's locale.

  • Commit: ded2247 (the v0.5.1 release, https://github.com/JC-000/c64-https/releases/tag/v0.5.1).
  • Arguments: make BACKEND=uci USE_NISTCURVES_ONCHIP=1 TRUST_STORE=1 TRUST_RELEASE=1, the shipped uci-onchip arguments.
  • Machine: macOS 27.0.1. GNU Make 3.81 (/usr/bin/make). ca65 V2.18 and ld65 V2.18 from /opt/homebrew/bin.
environment PRG sha256
interactive shell, LANG=en_US.UTF-8 (what make package used) ffa47fc32e43a16b995faee0d85d3d9312e1791a568820fedf7c0446bd27dfa6
env -i PATH="$PATH" HOME="$HOME" make … (no LANG, so the C locale) 98a3d882ddb7008f6f3aa646d1cf984279840fd22e22b3dd0f80967b53cb730d
env -i PATH="$PATH" HOME="$HOME" LANG=en_US.UTF-8 make … ffa47fc3…, the same as the first row

Independent reproduction (adv-296, reported to the lane, not re-run by me): under env -i, the other two wildcard-built products also move. ip65-onchip gives 1c377d34… (released 810f1158…) and uci-comb gives 6b95c329… (released 67cb0048…).

The two PRGs differ in 4,600 bytes, the first at byte 402 (1-based, as cmp -l counts). build/flags.stamp and build/https_host.inc are identical between the two builds, so the stamp does not capture the difference.

The only difference in the ld65 map's module list is the order of two objects:

$ diff <(modules in C-locale map) <(modules in en_US map)
17d16
< tls13.o:
23d21
< tls_record.o:
24a23
> tls_record.o:
25a25
> tls13.o:

That matches how the two locales order the source file names (ls src/tls*.s):

  • en_US.UTF-8: tls_cert … tls_record_io tls_record tls_transcript tls13. The underscore is ignored when collating.
  • C: tls13 tls_cert … tls_record tls_record_io tls_transcript, in byte order.

Where the order comes from (from the Makefile)

  • Makefile:271 TOP_SRCS := $(wildcard src/*.s), with no $(sort …).
  • Makefile:478 TOP_OBJS := $(patsubst src/%.s,build/%.o,$(TOP_SRCS)).
  • Makefile:483 ALL_OBJS := $(TOP_OBJS) ….
  • Makefile:522 $(LD65) $(LD65FLAGS) -o $@ $(ALL_OBJS) …. The link order is therefore the wildcard's order.
  • Makefile:282 (src/crypto/*.s) and Makefile:286 (src/crypto/shared/*.s) use the same unsorted pattern. They are not implicated by this build's diff.
  • uci-m3 is not affected. Its TOP_SRCS and NET_SRCS are explicit lists (Makefile:466-470), and the M3 demo PRG 916d4162… was identical under both environments.

Suspected, not verified: GNU Make 3.81's $(wildcard) returns names in an order that depends on the locale's collation (glob sorting with strcoll). The map diff and the ls orders are consistent with that. I have not read make's source to confirm it, and other make versions (4.x sorts wildcard results again) may behave differently.

Impact

  • "The PRG is deterministic; its sha256 is the check" (CLAUDE.md, Build) holds only for a fixed locale, and possibly a fixed make version.
  • build_m3_demo.sh builds in a scrubbed environment (env -i, so the C locale). build_prgs.sh (make package) inherits the caller's locale.
  • A reviewer who rebuilds a release with a different LANG gets different bytes for every image whose sources come from a wildcard (ip65, uci-onchip, uci-comb).

Possible fix (for discussion)

  • Wrap the three wildcards in $(sort …). make's sort compares bytes, independent of locale. This changes the link order, and so the bytes, of every wildcard-built image relative to v0.5.1, so it belongs in a release that expects new hashes.
  • Alternatively, pin LC_ALL=C in the Makefile, or in the package scripts.
  • Either way, add a test: build one profile under two locales and require equal PRG hashes.

Found while staging v0.5.1. Its notes promise this issue. The v0.5.1 assets were built under LANG=en_US.UTF-8, and their hashes reproduce under that locale.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions