Skip to content

JIT & macOS fat builds #114809

Closed
Closed
JIT & macOS fat builds#114809
Assignees
brandtbucher
Labels
OS-mactopic-JIT

Description

@ronaldoussoren
ronaldoussoren
opened on Jan 31, 2024

We support fat builds of Python on macOS, in particular used for the "Universal 2" installers on macOS (configure --enable-universalsdk --with-universal-archs=universal2). This uses clang's builtin support for compiling multiple architectures in one go (e.g. clang -arch arm64 -arch x86_64 ...).

As far as I can see now the machinery for JIT (--enable-experimental-jit) does not support this at the moment.

On possible way to get this to work (untested and likely incomplete) is to generate multiple variants for jit_stencils.h when doing a fat build and generate an umbrella header that includes the right one:

// jit_stencils.h

#if defined(__arm64__)
#include "jit_stencils-arm64.h"
#elif defined(__x86_64__)
#include "jit_stencils-x86_64.h"
#endif

Linked PRs

Activity

  1. ronaldoussoren
    added
    OS-mac
    on Jan 31, 2024

  2. brandtbucher commented on Feb 1, 2024

    @brandtbucher
    brandtbucher
    on Feb 1, 2024
    Member

    I agree that we should support this.

    Something that should be fixed (or figured out) first is getting --with-universal-archs with just one platform to work correctly. For example, if I'm on an M2 Mac and ./configure --enable-universalsdk --with-universal-archs=intel-64, the host configure var is set to aarch64-apple-darwin23.2.0 (it even prints out a message that I'm building for a tier two platform).

    host is what's used to build the JIT, so mayhem ensues. Maybe we should use a new var, that can hold several space-separated values?

    Once we have cross-builds on one platform working on macOS, it probably wouldn't be too hard to build two (or more!) JITs from there by passing several platforms to the build script.

    I'm happy to take on the JIT side of this project. Can somebody else figure out the prerequisite configure stuff? :)

  3. brandtbucher
    self-assigned this
    on Feb 1, 2024

  4. ronaldoussoren commented on Feb 19, 2024

    @ronaldoussoren
    ronaldoussoren
    on Feb 19, 2024
    ContributorAuthor

    The patch below teaches configure.ac about mixing --experimental-jit and --with-universal-archs. The Tools/jit/build.py script will get called with multiple target triples (except for intel-64 and intel-32 which describe only 1 architecture).

    I had to move handling --with-universal-archs to below handling --with-universal-archs, other than that the change it fairly mundane. Setting ARCH_TRIPPLES is a bit weird, mostly done this way to save some typing.

    BTW. The patch can result in calling the JIT tooling with a number of ancient Mac architectures (like PPC and i386), feel free to error out on those. IMHO we should remove those from configure entirely, but that's for a different issue.

    diff --git a/configure.ac b/configure.ac
index b39af7422c..9f560e6088 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1621,27 +1621,6 @@ else
   AC_MSG_RESULT([no])
 fi
 
-# Check for --enable-experimental-jit:
-AC_MSG_CHECKING([for --enable-experimental-jit])
-AC_ARG_ENABLE([experimental-jit],
-              [AS_HELP_STRING([--enable-experimental-jit],
-                              [build the experimental just-in-time compiler (default is no)])],
-              [],
-              [enable_experimental_jit=no])
-AS_VAR_IF([enable_experimental_jit],
-          [no],
-          [],
-          [AS_VAR_APPEND([CFLAGS_NODIST], [" -D_Py_JIT"])
-           AS_VAR_SET([REGEN_JIT_COMMAND],
-                      ["\$(PYTHON_FOR_REGEN) \$(srcdir)/Tools/jit/build.py $host"])
-           AS_VAR_SET([JIT_STENCILS_H], ["jit_stencils.h"])
-           AS_VAR_IF([Py_DEBUG],
-                     [true],
-                     [AS_VAR_APPEND([REGEN_JIT_COMMAND], [" --debug"])],
-                     [])])
-AC_SUBST([REGEN_JIT_COMMAND])
-AC_SUBST([JIT_STENCILS_H])
-AC_MSG_RESULT([$enable_experimental_jit])
 
 # Enable optimization flags
 AC_SUBST([DEF_MAKE_ALL_RULE])
@@ -2436,42 +2415,50 @@ yes)
                UNIVERSAL_ARCH_FLAGS="-arch ppc -arch i386"
                LIPO_32BIT_FLAGS=""
                ARCH_RUN_32BIT=""
+              ARCH_TRIPPLES=`echo {ppc,i386}-apple-darwin`
                ;;
             64-bit)
                UNIVERSAL_ARCH_FLAGS="-arch ppc64 -arch x86_64"
                LIPO_32BIT_FLAGS=""
                ARCH_RUN_32BIT="true"
+              ARCH_TRIPPLES=`echo {ppc64,x86_64}-apple-darwin`
                ;;
             all)
                UNIVERSAL_ARCH_FLAGS="-arch i386 -arch ppc -arch ppc64 -arch x86_64"
                LIPO_32BIT_FLAGS="-extract ppc7400 -extract i386"
                ARCH_RUN_32BIT="/usr/bin/arch -i386 -ppc"
+              ARCH_TRIPPLES=`echo {i386,ppc,ppc64,x86_64}-apple-darwin`
                ;;
             universal2)
                UNIVERSAL_ARCH_FLAGS="-arch arm64 -arch x86_64"
                LIPO_32BIT_FLAGS=""
                LIPO_INTEL64_FLAGS="-extract x86_64"
                ARCH_RUN_32BIT="true"
+              ARCH_TRIPPLES=`echo {arm64,x86_64}-apple-darwin`
                ;;
             intel)
                UNIVERSAL_ARCH_FLAGS="-arch i386 -arch x86_64"
                LIPO_32BIT_FLAGS="-extract i386"
                ARCH_RUN_32BIT="/usr/bin/arch -i386"
+              ARCH_TRIPPLES=`echo {i386,x86_64}-apple-darwin`
                ;;
             intel-32)
                UNIVERSAL_ARCH_FLAGS="-arch i386"
                LIPO_32BIT_FLAGS=""
                ARCH_RUN_32BIT=""
+              ARCH_TRIPPLES=i386-apple-darwin
                ;;
             intel-64)
                UNIVERSAL_ARCH_FLAGS="-arch x86_64"
                LIPO_32BIT_FLAGS=""
                ARCH_RUN_32BIT="true"
+              ARCH_TRIPPLES=x86_64-apple-darwin
                ;;
             3-way)
                UNIVERSAL_ARCH_FLAGS="-arch i386 -arch ppc -arch x86_64"
                LIPO_32BIT_FLAGS="-extract ppc7400 -extract i386"
                ARCH_RUN_32BIT="/usr/bin/arch -i386 -ppc"
+              ARCH_TRIPPLES=`echo {i386,ppc,x86_64}-apple-darwin`
                ;;
             *)
                AC_MSG_ERROR([proper usage is --with-universal-arch=universal2|32-bit|64-bit|all|intel|3-way])
@@ -2565,6 +2552,28 @@ yes)
     ;;
 esac
 
+# Check for --enable-experimental-jit:
+AC_MSG_CHECKING([for --enable-experimental-jit])
+AC_ARG_ENABLE([experimental-jit],
+              [AS_HELP_STRING([--enable-experimental-jit],
+                              [build the experimental just-in-time compiler (default is no)])],
+              [],
+              [enable_experimental_jit=no])
+AS_VAR_IF([enable_experimental_jit],
+          [no],
+          [],
+          [AS_VAR_APPEND([CFLAGS_NODIST], [" -D_Py_JIT"])
+           AS_VAR_SET([REGEN_JIT_COMMAND],
+                      ["\$(PYTHON_FOR_REGEN) \$(srcdir)/Tools/jit/build.py ${ARCH_TRIPPLES:-$host}"])
+           AS_VAR_SET([JIT_STENCILS_H], ["jit_stencils.h"])
+           AS_VAR_IF([Py_DEBUG],
+                     [true],
+                     [AS_VAR_APPEND([REGEN_JIT_COMMAND], [" --debug"])],
+                     [])])
+AC_SUBST([REGEN_JIT_COMMAND])
+AC_SUBST([JIT_STENCILS_H])
+AC_MSG_RESULT([$enable_experimental_jit])
+
 case "$CC_BASENAME" in
 *mpicc*)
     CFLAGS_NODIST="$CFLAGS_NODIST"

  5. ronaldoussoren commented on Feb 20, 2024

    @ronaldoussoren
    ronaldoussoren
    on Feb 20, 2024
    ContributorAuthor

    @brandtbucher, I've added a draft PR that seems to work for me (that is, a universal2 build works and won't crash for both light interactive use and a partial test run for both arm64 and x86_64).

    The PR is a bit rough and contains one unrelated change due using the compiler included in Xcode (not command line tools) and installing llvm-16 manually and not through homebrew.

  6. AlexWaygood
    added
    topic-JIT
    on May 30, 2024

  7. savannahostrowski commented on Sep 14, 2024

    @savannahostrowski
    savannahostrowski
    on Sep 14, 2024
    Member

    I was digging into this a bit but wanted to ask a perhaps naive question unrelated to JIT-specific builds: is there a reason we have FAT builds for Mac but not Windows?

    I can see the benefits of a universal installer from an end-user perspective, as some users might not know which architecture they're using. However, universal installers also have downsides, such as increased binary size.

    Has there been any discussion about providing separate installers for Mac x86 and ARM architectures?

  8. savannahostrowski
    added a commit that references this issue on Apr 30, 2025

    GH-114809: Add support for macOS multi-arch builds with the JIT enabl…

    26c0248
  9. savannahostrowski
    closed this as completedon May 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

OS-mactopic-JIT

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions