Repository navigation
jit.c may be built with an incomplete jit_stencils.h #118943
Description
Activity
I've test-built the package in the Fedora build system.
https://koji.fedoraproject.org/koji/taskinfo?taskID=117554410 (this will eventually be garbage collected)
Some architectures succeeded and some failed with:
python3.13 /builddir/build/BUILD/Python-3.13.0b1/Tools/jit/build.py i686-redhat-linux-gnu --debug usage: build.py [-h] [-d] [-f] [-v] target build.py: error: argument target: invalid get_target value: 'i686-redhat-linux-gnu'python3.13 /builddir/build/BUILD/Python-3.13.0b1/Tools/jit/build.py powerpc64le-redhat-linux-gnu --debug usage: build.py [-h] [-d] [-f] [-v] target build.py: error: argument target: invalid get_target value: 'powerpc64le-redhat-linux-gnu'python3.13 /builddir/build/BUILD/Python-3.13.0b1/Tools/jit/build.py s390x-redhat-linux-gnu --debug usage: build.py [-h] [-d] [-f] [-v] target build.py: error: argument target: invalid get_target value: 's390x-redhat-linux-gnu'I suppose this is expected.
But the problem reported here did not happen. Perhaps it only happens sometimes... ? I will trigger a couple of x86_64 builds to see if I can reproduce this again.
Theory:
I noticed we currently don't do an atomic rename of
jit_stencils.h, and just write it incrementally instead. I think ifmakesees the partially-written file, it'll start compilingjit.cand the bottom of the included file will be missing?Reacted by Miro HrončokOut of 10 builds, I did not reproduce this again.
I suppose this is expected.
Yeah, we're not attempting to support anything that isn't at least in PEP 11 right now.
But the problem reported here did not happen. Perhaps it only happens sometimes... ?
Well, yeah, these ones are intentionally failing early since we won't even attempt to build the JIT. So no
jit_stencils.his generated, andjit.cis never compiled, and the build stops there.By the way, it appears my hunch is correct. With careful use of parallel build and manually inserted
time.sleepcalls, I've been able to coax out similar errors from the compiler (from including a half-generatedjit_stencils.h). It happens ifjit.cis scheduled to build after the file has been opened for writing but before it's finished writing.So writing to a different file
jit_stencils.h.newand doing a rename after is probably the fix here.- changed the title
[-]JIT compiler in Fedora python3.13 package: Python/jit.c:395:1: error: expected expression before ‘int’[/-][+]`jit.c` may be built with an incomplete `jit_stencils.h`[/+]on May 11, 2024 - added a commit that references this issue
on May 16, 2024 FWIW I just seen:
python3.13 /builddir/build/BUILD/python3.13-3.13.0_b2-build/Python-3.13.0b2/Tools/jit/build.py aarch64-redhat-linux-gnu --debug =============================================================== JIT support for aarch64-redhat-linux-gnu is still experimental! Please report any issues you encounter. =============================================================== python3.13 /builddir/build/BUILD/python3.13-3.13.0_b2-build/Python-3.13.0b2/Tools/jit/build.py aarch64-redhat-linux-gnu --debug =============================================================== JIT support for aarch64-redhat-linux-gnu is still experimental! Please report any issues you encounter. =============================================================== Traceback (most recent call last): File "/builddir/build/BUILD/python3.13-3.13.0_b2-build/Python-3.13.0b2/Tools/jit/build.py", line 28, in <module> args.target.build(pathlib.Path.cwd(), comment=comment, force=args.force) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/builddir/build/BUILD/python3.13-3.13.0_b2-build/Python-3.13.0b2/Tools/jit/_targets.py", line 224, in build jit_stencils_new.replace(jit_stencils) ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^ File "/usr/lib64/python3.13/pathlib/_local.py", line 780, in replace os.replace(self, target) ~~~~~~~~~~^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/builddir/build/BUILD/python3.13-3.13.0_b2-build/Python-3.13.0b2/build/debug/jit_stencils.h.new' -> '/builddir/build/BUILD/python3.13-3.13.0_b2-build/Python-3.13.0b2/build/debug/jit_stencils.h' make: *** [Makefile:3026: regen-jit] Error 1 make: *** Waiting for unfinished jobs....This is slightly better than before, but still not entirely OK. I guess when this happens, it means another process created and moved the new file at the same time.
Perhaps the file needs to have a uniq path, and/or be in tmp?
I got it on x86_64 as well, twice in a row. Both builders had 48 CPUs, so the
makecall was heavily parallelized.Should I open a new issue?
6 remaining items
Thanks again, @hroncok!
Bug report
Bug description:
As discussed in https://discuss.python.org/t/pep-744-jit-compilation/50756/42 and further with @brandtbucher, I've tried to change the Fedora's Python 3.13 package to utilize
--enable-experimental-jit=yes-off.As a starting point, I took the 3.13.0b1-1 package from:
https://src.fedoraproject.org/rpms/python3.13/blob/1c24374362/f/python3.13.spec
And I changed:
That means I added
--enable-experimental-jit=yes-offto the debug and optimized non-free-threaded builds and some extra build dependencies.Running the build in Fedora Rawhide (41) chroot resulted in this error during the debug build (which happens first):
Full logs:
@brandtbucher suggested that "perhaps several builds are happening at once for some reason. If so, they could be stepping on each other and maybe truncating the generated file."
Anyway, here is the generated jit_stencils.h file as it was present after the failure.
CPython versions tested on:
3.13
Operating systems tested on:
Linux
Linked PRs
jit_stencils.h#118957