From 04fe885e1020d4e82740547d0f47048d13f0c364 Mon Sep 17 00:00:00 2001 From: Daniel Rossier Date: Tue, 15 Sep 2026 18:24:27 +0200 Subject: [PATCH 1/2] build: three fixes to the attach/updiff machinery Found while bringing up rpi4_64 in the opencn-ng tree; all three are generic to this build system, so they come here unchanged. 1. patch_identifier() now normalises a leading "./". do_updiff matches a staged patch to the series by the path in the patch header. Staged patches carry git-style "+++ b/"; patches generated years ago by a plain `diff` run from the target directory carry "+++ ./". The two never compared equal, so every existing patch looked unmatched and updiff appended a duplicate for every file it saw -- 25190 patches onto a series of 33, reported as success. This is not hypothetical here: 3872 patches in this tree still carry the "./" form. Any updiff on one of those recipes would have done the same. 2. do_updiff refuses a runaway. do_diffcompose compares the pristine snapshot against IB_TARGET, which is a working tree the user may have run `make` in, and the artefact exclusion list is a blacklist. A single `build.sh uboot` was measured leaving 49 files it does not name. Above IB_UPDIFF_MAX_NEW (20) new files the task now fails and prints the two commands that fix it, instead of writing the series. 3. do_check_attach refuses a tree attached by another recipe. IB_TARGET is a path in the source tree and more than one recipe can claim it: uboot_2022.04 and uboot_2024.07 both attach to u-boot/, selected by PREFERRED_VERSION per platform. The attach is stamped per recipe, so after building platform A and switching to B, B finds its own stamp from an earlier session, skips the attach, and configures A's sources. It failed loudly once (rpi4_64_defconfig missing in a 2024.07 tree) and silently once: a virt64 image built with U-Boot 2022.04 instead of 2024.07, differing in CONFIG_POSITION_INDEPENDENT, so ATF loaded BL33 at 0x60000000 and jumped into a binary linked for 0x0 -- no output, no error, a whole secure boot chain to bisect. do_attach_infrabase now records the owning PF in ${IB_TARGET}.attach.pf and a nostamp do_check_attach refuses on mismatch. It refuses rather than re-attaching: that deletes a tree the user may have edited outside the patch series, which is their call, as with IB_FORCE_ATTACH. A tree with no marker predates this and is adopted silently. A parse-time variant that dropped the stale stamp was tried first and removed: the message printed but bitbake had already settled the runqueue and built the wrong tree anyway. The check has to be a task. Verified: all recipes parse, and the guard was exercised both ways in the originating tree. Co-Authored-By: Claude Opus 5 (1M context) --- .gitignore | 1 + build/meta/classes/base.bbclass | 63 ++++++++++++++++++++++++++++++++ build/meta/classes/patch.bbclass | 44 +++++++++++++++++++++- 3 files changed, 106 insertions(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index 24d2434eae..e5cb6d9273 100755 --- a/.gitignore +++ b/.gitignore @@ -149,5 +149,6 @@ so3/so3/scripts/kconfig/zconf.tab.c # do_attach_infrabase dirty-tree guard manifests (sibling of each attached # component tree, e.g. avz.attach.sha256, u-boot.attach.sha256) *.attach.sha256 +*.attach.pf __pycache__/ /scripts.old/ diff --git a/build/meta/classes/base.bbclass b/build/meta/classes/base.bbclass index b580471b0e..6881ea5d04 100644 --- a/build/meta/classes/base.bbclass +++ b/build/meta/classes/base.bbclass @@ -224,8 +224,71 @@ do_attach_infrabase () { # Record what we just wrote so the next attach can detect local edits. ( cd "${IB_TARGET}" && find . -type f -print0 | LC_ALL=C sort -z | xargs -0 sha256sum ) > "$ib_manifest" 2>/dev/null || rm -f "$ib_manifest" + + # Record WHICH recipe the tree now belongs to. Several recipes can share + # one IB_TARGET -- uboot_2022.04 and uboot_2024.07 both attach to + # u-boot/ -- and each has its own do_attach_infrabase stamp. Without + # this marker, attaching one and then building the other finds a valid + # stamp, skips the attach, and compiles the wrong source tree. See + # do_check_attach() in this class. + echo "${PF}" > "${IB_TARGET}.attach.pf" +} + +# Refuse to build against a tree that belongs to another recipe. +# +# IB_TARGET is a path in the source tree, not under tmp/, and more than one +# recipe can claim it: uboot_2022.04 and uboot_2024.07 both attach to +# u-boot/, selected by PREFERRED_VERSION per platform. do_attach_infrabase +# is stamped per recipe, so after building platform A and switching to +# platform B, B's recipe finds its own stamp from an earlier session, skips +# the attach, and runs do_configure against A's sources. +# +# That fails loudly when the defconfig is missing -- rpi4_64_defconfig in a +# 2024.07 tree -- and SILENTLY when it is not. A virt64 image was built with +# U-Boot 2022.04 instead of 2024.07; the two differ in +# CONFIG_POSITION_INDEPENDENT, so ATF loaded BL33 at 0x60000000 and jumped +# into a binary linked for 0x0. No output, no error, and a whole secure boot +# chain to bisect before the cause turned out to be the build system. +# +# The marker written by do_attach_infrabase names the recipe that owns the +# tree. This task is nostamp, so it is the one thing that always runs, and it +# refuses rather than repairing: re-attaching means deleting a tree the user +# may have edited, and that decision is theirs. A tree with no marker was +# attached before markers existed and is adopted silently. + +do_check_attach[nostamp] = "1" +python do_check_attach() { + import os + + target = d.getVar('IB_TARGET') + pf = d.getVar('PF') + if not target or not pf or not os.path.isdir(target): + return + + try: + with open(target + '.attach.pf') as f: + owner = f.read().strip() + except OSError: + return + + if owner == pf: + return + + bb.fatal( + "%s is attached to %s, not to %s.\n" + "Both recipes share this directory, and each has its own attach " + "stamp, so building now would compile the other recipe's sources.\n" + "Re-attach before continuing:\n" + " rm -f %s.do_attach_infrabase\n" + " IB_FORCE_ATTACH=1 bitbake %s -c attach_infrabase\n" + "Anything you edited in %s that is not in the patch series will be " + "lost; the previous tree is kept as %s.back." + % (target, owner, pf, d.getVar('STAMP'), d.getVar('PN'), target, target) + ) } +addtask do_check_attach after do_attach_infrabase before do_configure + addtask cleansstate after do_clean python do_cleansstate() { sstate_clean_cachefiles(d) diff --git a/build/meta/classes/patch.bbclass b/build/meta/classes/patch.bbclass index a250ea5a89..2a87df00b4 100644 --- a/build/meta/classes/patch.bbclass +++ b/build/meta/classes/patch.bbclass @@ -414,7 +414,15 @@ def patch_identifier(path): "+++ b/" header line. Patches produced by do_diffcompose always carry git-style a/ b/ labels, so this identifier uniquely designates the source file the patch targets, independent of the - patch's own filename or sequence number.""" + patch's own filename or sequence number. + + Older patches in these series were produced by a plain `diff` run from + the target directory and carry "+++ ./" instead, with an absolute + path on the "---" side. Both spellings have to normalise to the same + identifier: when they do not, EVERY existing patch looks unmatched to + do_updiff, which then appends a duplicate for every file in the tree. + Seen on a u-boot series: 25190 patches appended onto 33. So the "./" + strip below is load-bearing, not cosmetic.""" src = None dst = None try: @@ -433,7 +441,9 @@ def patch_identifier(path): if not s or s == '/dev/null': continue if s.startswith('a/') or s.startswith('b/'): - return s[2:] + s = s[2:] + while s.startswith('./'): + s = s[2:] return s return None @@ -544,6 +554,36 @@ python do_updiff() { pass next_number = max(used_numbers, default=0) + 1 + # Sanity gate before touching the series. + # + # do_diffcompose compares the pristine snapshot against IB_TARGET, and + # IB_TARGET is a working tree the user may well have run `make` in. The + # exclude list above catches the build artefacts we know about, but it + # is a blacklist: a tree built with a configuration nobody anticipated, + # or an attach stale enough to predate half the series, produces + # thousands of "new file" patches, appended one per artefact. + # + # A legitimate updiff touches a handful of files, so the threshold is + # deliberately low: a single `build.sh uboot` was measured leaving 49 + # artefacts the blacklist does not name. Refuse and say what to do, + # rather than writing the series and leaving the user to `git checkout` + # their way out. + limit = int(d.getVar('IB_UPDIFF_MAX_NEW') or 20) + new_count = sum(1 for pid in staging_idx if pid not in target_idx) + if new_count > limit: + bb.fatal( + f"do_updiff: {new_count} files in {d.getVar('IB_TARGET')} have no " + f"patch in the series yet (limit {limit}).\n" + f"That is what a BUILT tree looks like to do_diffcompose, not a " + f"set of edits: build output the exclude list does not name is " + f"being mistaken for new source.\n" + f"Clean the tree and re-attach it before folding edits back in:\n" + f" rm -f ${{BUILDDIR}}/tmp/stamps/{pf}.do_attach_infrabase\n" + f" IB_FORCE_ATTACH=1 bitbake {d.getVar('PN')} -c attach_infrabase\n" + f"then redo the edit and re-run updiff. If the count is genuinely " + f"right -- a large import, say -- raise IB_UPDIFF_MAX_NEW." + ) + kept = consolidated = appended = skipped_chain = 0 # Per-file: CONSOLIDATE (overwrite the single existing patch), From 1baa6af190a959150197f8989789220baa20eee7 Mon Sep 17 00:00:00 2001 From: Daniel Rossier Date: Tue, 15 Sep 2026 18:29:42 +0200 Subject: [PATCH 2/2] base: keep quilt's staging out of the attach manifest do_patch drives quilt, which leaves its working state in ${S}: patches/ + patches/series (the stack) and .pc/ (a pristine copy of every file each patch touches, so `quilt pop` can undo it). do_attach_infrabase copies ${S} wholesale into IB_TARGET, so that staging lands there too, and then records a sha256 of every file it wrote. Recording build products in that manifest turns the dirty-guard against itself: quilt regenerates them on the next do_patch, the checksums no longer match, and the guard refuses to re-attach a perfectly clean tree. This tree's u-boot manifest currently carries 55 such entries. Pruned on write, and also filtered on the check so manifests recorded before this fix stop blocking. `|| true` because grep exits 1 when it filters everything away and bitbake runs shell tasks under `set -e`. Already present in infrabase, pos_sol and opencn-ng; this brings the tree in line. Co-Authored-By: Claude Opus 5 (1M context) --- build/meta/classes/base.bbclass | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/build/meta/classes/base.bbclass b/build/meta/classes/base.bbclass index 6881ea5d04..58296f17a1 100644 --- a/build/meta/classes/base.bbclass +++ b/build/meta/classes/base.bbclass @@ -204,7 +204,16 @@ do_attach_infrabase () { # *added* source files are not tracked by the manifest — the ${IB_TARGET}.back # copy remains the last-resort net for that case. if [ -d "${IB_TARGET}" ] && [ -f "$ib_manifest" ] && [ "${IB_FORCE_ATTACH}" != "1" ]; then - ib_dirty=$(cd "${IB_TARGET}" && LC_ALL=C sha256sum -c --quiet "$ib_manifest" 2>/dev/null | sed -n 's/: FAILED.*$//p') + # The quilt staging left by do_patch (patches/, series, .pc) is a + # build product, not source -- it is regenerated on every patch run + # and is gitignored. Recorded in the manifest it made the guard fire + # on its own artefacts and refuse a perfectly clean tree. Filtered + # here as well as pruned below, so manifests recorded before this + # fix stop blocking too. `|| true`: grep exits 1 when it filters + # everything away, and bitbake runs shell tasks under `set -e`. + ib_dirty=$(cd "${IB_TARGET}" && LC_ALL=C sha256sum -c --quiet "$ib_manifest" 2>/dev/null \ + | sed -n 's/: FAILED.*$//p' \ + | grep -vE '(^|/)(\.pc|patches)/' || true) if [ -n "$ib_dirty" ]; then bbwarn "Local modifications detected in ${IB_TARGET}, not captured in the ${PN} patch set:" echo "$ib_dirty" | sed 's|^\./| |' >&2 @@ -223,7 +232,8 @@ do_attach_infrabase () { cp -r ${S}/. ${IB_TARGET} # Record what we just wrote so the next attach can detect local edits. - ( cd "${IB_TARGET}" && find . -type f -print0 | LC_ALL=C sort -z | xargs -0 sha256sum ) > "$ib_manifest" 2>/dev/null || rm -f "$ib_manifest" + ( cd "${IB_TARGET}" && find . \( -name .pc -o -name patches \) -prune -o -type f -print0 \ + | LC_ALL=C sort -z | xargs -0 sha256sum ) > "$ib_manifest" 2>/dev/null || rm -f "$ib_manifest" # Record WHICH recipe the tree now belongs to. Several recipes can share # one IB_TARGET -- uboot_2022.04 and uboot_2024.07 both attach to