Skip to content

Using a custom sysimage with PythonCall causes error during juliacall initialization #436

Open
Open
Using a custom sysimage with PythonCall causes error during juliacall initialization#436
Labels
bugSomething isn't working

Description

@brian-dellabetta
brian-dellabetta
opened on Dec 28, 2023

Affects: Both

Describe the bug
I would like to use a custom sysimage of our julia package which has PythonCall.jl as a dependency. The image builds fine, and is useable with julia --sysimage=/path/to/myimage.so, but when I try to use it as the custom sysimage with juliacall, I hit the following initialization error:

PYTHON_JULIACALL_SYSIMAGE=/path/to/myimage.so python
>>> from juliacall import Main as jl
fatal: error thrown and no exception handler available.
InitError(mod=:C, error=AssertionError(msg="CTX.which == :PyCall"))
#35 at /home/xxx/.julia/packages/PythonCall/wXfah/src/cpython/context.jl:144
with_gil at /home/xxx/.julia/packages/PythonCall/wXfah/src/cpython/gil.jl:10 [inlined]
with_gil at /home/xxx/.julia/packages/PythonCall/wXfah/src/cpython/gil.jl:9 [inlined]
init_context at /home/xxx/.julia/packages/PythonCall/wXfah/src/cpython/context.jl:141
__init__ at /home/xxx/.julia/packages/PythonCall/wXfah/src/cpython/CPython.jl:21
jfptr___init___72284 at /home/xxx/.julia/sysimages/myimage.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
jl_module_run_initializer at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/toplevel.c:75
_finish_julia_init at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/init.c:855
julia_init at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/init.c:804
ijl_init_with_image at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/jlapi.c:66 [inlined]
ijl_init_with_image at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/jlapi.c:55
unknown function (ip: 0x7f29a0a18d1c)
unknown function (ip: 0x7f29a0a18288)
unknown function (ip: 0x7f29a0a2bab9)
unknown function (ip: 0x7f29a0a2c1b2)
_PyObject_MakeTpCall at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a13df561)
PyEval_EvalCode at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a1447fbd)
unknown function (ip: 0x7f29a14304f7)
_PyEval_EvalFrameDefault at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a13df561)
unknown function (ip: 0x7f29a13f45bc)
PyObject_CallMethodObjArgs at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
PyImport_ImportModuleLevelObject at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a13df561)
PyEval_EvalCode at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a145f1fc)
unknown function (ip: 0x7f29a145f195)
unknown function (ip: 0x7f29a13c6eb6)
_PyRun_InteractiveLoopObject at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a1311c71)
PyRun_AnyFileExFlags at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a130e572)
Py_BytesMain at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at python (unknown line)

I tried a sysimage that didn't have PythonCall imported or as a dependency, and juliacall initalized as expected. Maybe this isn't possible, but the Assertion error is occuring at this line, and I'm not using PyCall anywhere, so perhaps it is a bug in init_context?

Steps to reproduce

In a fresh directory, create

  1. build.jl:
using Pkg

Pkg.add("PythonCall")

using PythonCall

Pkg.add("PackageCompiler")
using PackageCompiler

create_sysimage(:PythonCall, sysimage_path="/home/xxx/.julia/sysimages/myimage.so", precompile_execution_file="precompile.jl")
  1. precompile.jl:
using PythonCall

call julia --project=. build.jl (preferably in a fresh project) and then PYTHON_JULIACALL_SYSIMAGE=/home/xxx/.julia/sysimages/myimage.so python -c 'from juliacall import Main as jl'

Your system
Please provide detailed information about your system:

  • Debian 11 (Docker container on MacOS 14.2, sysimage builds on host were failing for me)
  • Julia 1.10, Python 3.11, PythonCall 0.9.15, juliacall 0.9.15
    -Pkg.status() after running build.jl:
Status `~/.julia/environments/v1.10/Project.toml`
  [9b87118b] PackageCompiler v2.1.16
  [6099a3de] PythonCall v0.9.15
  • juliapkg.status().
/home/xxx/julia_depot/environments/pyjuliapkg/pyjuliapkg/juliapkg.json
Julia 1.10.0 @ /home/xxx/julia/bin/julia
Packages:
  <ourpackage>: {'uuid': 'xxx', 'dev': True, 'path': '/path/to/ourpackage.jl'}

Additional context
Add any other context about the problem here.

Activity

  1. brian-dellabetta
    added
    bugSomething isn't working
    on Dec 28, 2023
  2. brian-dellabetta
    changed the title [-]Using a sysimage with PythonCall already imported causes error during juliacall initialization[/-] [+]Using a custom sysimage with PythonCall causes error during juliacall initialization[/+] on Dec 28, 2023

  3. cjdoris commented on Jan 25, 2024

    @cjdoris
    cjdoris
    on Jan 25, 2024
    Member

    Thanks for the report. I suspect the issue is that some state is being saved in the sysimage, i.e. __init__ sets some variables somewhere, and these values are what are saved in the sysimage, which is confusing __init__ because they are not their default values. So probably __init__ needs to explicitly set these to the default.

  4. brian-dellabetta commented on Feb 5, 2024

    @brian-dellabetta
    brian-dellabetta
    on Feb 5, 2024
    ContributorAuthor

    Thanks @cjdoris for the reply. I will try setting the default values in __init__ later this week, hopefully it's a quick fix

  5. brian-dellabetta commented on Feb 13, 2024

    @brian-dellabetta
    brian-dellabetta
    on Feb 13, 2024
    ContributorAuthor

    I added some printlns to PythonCall.jl (see diff here) to get some more information. I thought it would be a matter of setting CTX, but the issue is more subtle than that.

    Running from juliacall import Main as jl without sysimage, CTX.is_embedded is true.
    Running the same with PYTHON_JULIACALL_SYSIMAGE set, it is false and continues down the function until it fails on this line.

    This makes sense, the flag gets set to true normally because __PythonCall_libptr is set in the startup script before the using PythonCall line, but it's false when running with a sysimage because the PythonCall __init__ gets called at julia startup, before any other commands are run.

    I'm not sure what the best solution is here. Maybe if the sysimage env var is set, consider it embedded and exit out of the init?

    CTX.is_embedded = haskey(ENV, "JULIA_PYTHONCALL_SYSIMAGE") ? true : hasproperty(Base.Main, :__PythonCall_libptr)

    There might be edge cases where this isn't desired.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions