Skip to content

docker in kubernetes #161

Open
Open

Description

@steviehs

For the syncosync build, we have set up our own gitlab runners on kubernetes and run a docker inside it, which worked from time to time...

now, there is a strange issue:
from CustomPiOS everything looks fine... until the chroot with qemuaarch64 happens: qemu states a wrong exec format...
I have tried now several hours to debug - therefore I forked CustomPiOS to add debug info - but I have no glue...

https://gitlab.com/syncosync/syncosync/-/jobs/2221508446

shows the pipeline constantly failing and the issue

https://gitlab.com/syncosync/syncosync/-/jobs/2221508446#L816 states the architecture of "touch"
https://gitlab.com/syncosync/syncosync/-/jobs/2221508446#L852 states the architecture is wrong...

Any ideas what could get wrong?

Running the docker locally on my debian works fine... so I guess, it has something to do with loop mounts and privileges or so.

Sorry for my vague request, but I am rather glueless... and I hate the idea of setting up bare metal idling around to build a new distro once a week...

Activity

  1. guysoft commented on Mar 18, 2022

    @guysoft
    Owner

    The error is

    /chroot_script: line 81: /usr/bin/touch: cannot execute binary file: Exec format error
    
    usr/bin/touch: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=55caec4d0916cca9fdb58167c5a7a8b78a9d2d2e, for GNU/Linux 3.7.0, stripped
    + ls -la usr/bin/touch
    

    It would seem that this line is not really executing qemu-aarch64-static:

    chroot . usr/bin/qemu-aarch64-static /bin/bash /chroot_script
    

    If anyone has gitlab experience that could help

  2. woodcoder commented on Apr 21, 2023

    @woodcoder

    I ended up here while trying to run CustomPiOS on Ubuntu 22.04 under Parallels on new Apple silicon (ARM64).

    Initially the Exec format error happened when trying to run /bin/bash at line 69 of the custompios script. This happened because the assumption that arm(fh) 32-bit code is supported by all aarch64 chips is not true. [See this discussion for more info, but lack of support can be detected by calls to dpkg --print-foreign-architectures or lscpu | grep op-mode.]

    Hacking the custompios script to use qemu-arm-static in this case allowed me to run bash successfully - but then gave me the exact same error as in this issue - the Exec format error for /usr/bin/touch.

    I believe this is because while the shell runs under emulation, other programs that the shell starts aren't automatically run under qemu.

    In my situation I eventually traced this to binfmts not being setup correctly - which is what normally solves this problem. My workaround is to create the missing /var/lib/binfmts/qemu-arm config with the arm magic numbers and enable it with sudo update-binfmts --enable qemu-arm. This allows the ELF 32-bit code to run on the M1/M2 chips and CustomPiOS to run out of the box (I could even undo my changes to the custompios script).
    __

    For this issue, I therefore think it's likely that qemu-aarch64-static is being executed successfully (otherwise the error would been seen for bash, not touch).

    So instead I wondered if this issue might be solved by running update-binfmts --enable qemu-aarch64 - the way that the custompios script does when .dockerenv is present?

  3. guysoft commented on Nov 7, 2023

    @guysoft
    Owner

    Update binfmts should happen here:

    update-binfmts --enable qemu-aarch64

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions