Skip to content

python:3.9 - Failed run apt update from the last version of the image #837

Closed
Closed
python:3.9 - Failed run apt update from the last version of the image#837

Description

@aviadpriel
aviadpriel
opened on Jun 15, 2023

as titled

today morning we found that our builds failed
we are using python:3.9

when we run apt update the build failed

it looks that it fails because of the upgrade to bookworm

we get this errors

#5 [build 2/4] RUN apt-get update &&     apt-get install -y --no-install-recommends     poppler-utils
#5 sha256:fdb97fe2ebc3fcf79626c6628756f871cc0d6b0fc02ba21a2949fe33d31a2373
#5 0.240 Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
#5 0.247 Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
#5 0.248 Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
#5 0.277 Err:1 http://deb.debian.org/debian bookworm InRelease
#5 0.277   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY F8D2585B8783D481
#5 0.303 Err:2 http://deb.debian.org/debian bookworm-updates InRelease
#5 0.303   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131
#5 0.329 Err:3 http://deb.debian.org/debian-security bookworm-security InRelease
#5 0.329   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 54404762BBB6E853 NO_PUBKEY BDE6D2B9216EC7A8
#5 0.330 Reading package lists...
#5 0.337 W: GPG error: http://deb.debian.org/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY F8D2585B8783D481
#5 0.337 E: The repository 'http://deb.debian.org/debian bookworm InRelease' is not signed.
#5 0.337 W: GPG error: http://deb.debian.org/debian bookworm-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131
#5 0.337 E: The repository 'http://deb.debian.org/debian bookworm-updates InRelease' is not signed.
#5 0.337 W: GPG error: http://deb.debian.org/debian-security bookworm-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 54404762BBB6E853 NO_PUBKEY BDE6D2B9216EC7A8
#5 0.337 E: The repository 'http://deb.debian.org/debian-security bookworm-security InRelease' is not signed.
#5 0.337 E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
#5 0.337 E: Sub-process returned an error cod

Activity

  1. aviadpriel
    changed the title [-]Failed run `apt update` from the last version of the build from image `python:3.9`[/-] [+]`python:3.9` - Failed run `apt update` from the last version of the image[/+] on Jun 15, 2023

  2. Cookiehook commented on Jun 15, 2023

    @Cookiehook
    Cookiehook
    on Jun 15, 2023

    We're seeing the same issues on python:3.10. Worked around by using python:3.10-bullseye for now.

  3. aviadpriel commented on Jun 15, 2023

    @aviadpriel
    aviadpriel
    on Jun 15, 2023
    Author

    yes, same for python:3.9-bullseye

    but we have couple of security fixes on Debian 12 that we want to get

  4. a1723 commented on Jun 15, 2023

    @a1723
    a1723
    on Jun 15, 2023

    The main question is: Will it be fixed in python:xx images or now we must always used images with "bullseye" postfix ?

  5. yosifkit commented on Jun 15, 2023

    @yosifkit
    yosifkit
    on Jun 15, 2023
    Member

    I am unable to reproduce. I'd suggest updating docker and libseccomp on the host. Newer base OS's use newer system calls and an older libseccomp can block them since they are unknown to it. You can verify that it is libseccomp by running the bookworm image with --security-opt seccomp=unconfined.

    This is similar to the update to Ubuntu focal: docker-library/mongo#606 (comment)

    $ docker run -it python:3.9-bookworm bash
Unable to find image 'python:3.9-bookworm' locally
3.9-bookworm: Pulling from library/python
bba7bb10d5ba: Already exists
ec2b820b8e87: Already exists
284f2345db05: Already exists
fea23129f080: Already exists
7c62c924b8a6: Already exists
b2210932934e: Pull complete
ee9c01829d92: Pull complete
d6285f41f1b6: Pull complete
Digest: sha256:98f018a1afd67f2e17a4abd5bfe09b998734ba7c1ee54780e7ed216f8b8095c3
Status: Downloaded newer image for python:3.9-bookworm
root@0b679f272394:/# apt-get update
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [24.2 kB]
Fetched 9176 kB in 1s (7771 kB/s)
Reading package lists... Done
root@0b679f272394:/#

  6. InterferencePattern commented on Jun 15, 2023

    @InterferencePattern
    InterferencePattern
    on Jun 15, 2023

    I see the same for python:3.8-slim and others after the update from buster -> bookworm.
    I think the image needs an update of the GPG public keys for the bookworm apt repos.

    EDIT: Until this is fixed, you can use the -buster versions of images, such as python:3.8-slim-buster

  7. jmmclean commented on Jun 15, 2023

    @jmmclean
    jmmclean
    on Jun 15, 2023

    hitting us as well, using python:3.7

  8. keever commented on Jun 15, 2023

    @keever
    keever
    on Jun 15, 2023

    This broke our ci docker build jobs that used python:3.9-slim We used docker v24.0.2 to do the build.

  9. muasif80 commented on Jun 16, 2023

    @muasif80
    muasif80
    on Jun 16, 2023

    It seems like the pointers for the latest image tags are moved to bookworm. The python:3.9-slim-bullseye might work instead of python:3.9-slim or maybe python:3.9-slim-buster will work too. The official 3.9 image github update shows that the version is updated from buster to bookworm.

  10. jacobeturpin commented on Jun 16, 2023

    @jacobeturpin
    jacobeturpin
    on Jun 16, 2023

    Can confirm that our builds were also impacted using python:3.8-slim. Modifying to python:3.8-slim-bullseye resolved the issue, so we're temporarily using that until the issue is resolved.

  11. mitsuo0114 commented on Jun 19, 2023

    @mitsuo0114
    mitsuo0114
    on Jun 19, 2023

    Same here in out CI tool. I could solved with downgrading.

    I'm assuming following change in Debian may have some affect to CI. (I haven't checked deeply)

    https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#python3-pep-668

    5.2.2. Python Interpreters marked externally-managed
    The Debian provided python3 interpreter packages (python3.11 and pypy3) are now marked as being externally-managed, following PEP-668. The version of python3-pip provided in Debian follows this, and ?will refuse to manually install packages on Debian's python interpreters, unless the --break-system-packages option is specified.

    If you need to install a Python application (or version) that isn't packaged in Debian, we recommend that you install it with pipx (in the pipx Debian package). pipx will set up an environment isolated from other applications and system Python modules, and install the application and its dependencies into that.

  12. aebm commented on Jun 19, 2023

    @aebm
    aebm
    on Jun 19, 2023

    Hi, that is a "problem" with the change of the debian base image used. It seems related to seccomp and it is mentioned here debuerreotype/docker-debian-artifacts#197 (comment)

    Example (PS: Don't run you containers with more privileges that necessary =) )

    docker run --name=dummy --rm -ti python:3.11-slim bash -c 'apt-get update && apt-get install -y less && echo OK'
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [28.3 kB]
Fetched 9180 kB in 2s (4697 kB/s)                         
Reading package lists... Done
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code

    vs

    docker run --name=dummy --rm -ti --privileged python:3.11-slim bash -c 'apt-get update && apt-get install -y less && echo OK'
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [28.3 kB]
Fetched 9180 kB in 2s (4844 kB/s)                          
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  less
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 131 kB of archives.
After this operation, 321 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 less amd64 590-2 [131 kB]
Fetched 131 kB in 0s (3313 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package less.
(Reading database ... 8386 files and directories currently installed.)
Preparing to unpack .../archives/less_590-2_amd64.deb ...
Unpacking less (590-2) ...
Setting up less (590-2) ...
OK

  13. aebm commented on Jun 19, 2023

    @aebm
    aebm
    on Jun 19, 2023

    I can confirm that it "works" when unconfined. Don't try this at home

    docker run --name=dummy --rm -ti --security-opt seccomp=unconfined python:3.11-slim bash -c 'apt-get update && apt-get install -y less && echo OK'
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [28.3 kB]
Fetched 9180 kB in 2s (4821 kB/s)                          
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  less
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 131 kB of archives.
After this operation, 321 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 less amd64 590-2 [131 kB]
Fetched 131 kB in 0s (2989 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package less.
(Reading database ... 8386 files and directories currently installed.)
Preparing to unpack .../archives/less_590-2_amd64.deb ...
Unpacking less (590-2) ...
Setting up less (590-2) ...
OK

  14. tianon commented on Jun 20, 2023

    @tianon
    tianon
    on Jun 20, 2023
    Member

    The proper fix is updating libseccomp2 (and Docker/runc) on your host. 👍

  15. yosifkit commented on Jun 20, 2023

    @yosifkit
    yosifkit
    on Jun 20, 2023
    Member

    (basically the same comment as redis/docker-library-redis#365 (comment))

    Root cause: it is Docker with libseccomp so a newer syscall used in Debian Bookworm packages/libs is being blocked. (Thanks for confirming @aebm)

    libseccomp lets you configure allowed syscalls for a process. Docker sets a default seccomp profile for all containers such that only certain syscalls are allowed and everything else is blocked (so, newer syscalls that are not yet known to libseccomp or docker are blocked).

    • verify that it is libseccomp by running the Bookworm-based image with --security-opt seccomp=unconfined
    • one fix:
      • update libseccomp and docker on the host running the containers
    • one workaround:
      • switch to the *bullseye images (in the python images, these will continue to be maintained/updated until the respective Python end of life or the next Debian release, Debian Trixie)

  16. 5 remaining items

  17. ad-m-ss commented on Jul 20, 2023

    @ad-m-ss
    ad-m-ss
    on Jul 20, 2023

    I'm having this issue on Mac, on python 3.11. I have the latest docker installed, but licseccomp is not used in Mac as far as I know, any suggestions?

    Docker on Mac usually uses Linux VM. Linux VM uses libseccomp.

  18. k0d3r1s commented on Jul 21, 2023

    @k0d3r1s
    k0d3r1s
    on Jul 21, 2023

    which libseccomp version is required for this to be working?

  19. openbrian commented on Jul 27, 2023

    @openbrian
    openbrian
    on Jul 27, 2023

    Just a comment here. If you're running your builds in a dockerized Jenkins, for example, you'll also be using docker:dind (docker in docker). You may need to upgrade docker here. docker pull docker:dind

  20. skabbit commented on Aug 15, 2023

    @skabbit
    skabbit
    on Aug 15, 2023

    I verify, that switching to python:3.7-slim-buster (or python:3.7-buster in my case) resolved this issue with NO_PUBKEY.

    Also, faced this issue only while using docker:18.09.7-dind runner in CI/CD, but cannot reproduce it on docker v20.10.13 manually running docker cmd.

  21. jbothma
    added a commit that references this issue on Aug 31, 2023

    Pin debian release docker-library/python#837

    0417523

  22. mke21 commented on Sep 21, 2023

    @mke21
    mke21
    on Sep 21, 2023

    I've got this problem running in gitlab-ci with the worker running on a private Kubernetes cluster. I'm not sure how to fix it apart from using the older bullseye image which obviously is a stopgap solution. This is the setup for the build:

    variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ""

services:
  - name: docker:stable-dind

image: docker:stable

  23. sergei-lobanov commented on Oct 4, 2023

    @sergei-lobanov
    sergei-lobanov
    on Oct 4, 2023

    Any plans for a normal fix?

  24. tianon commented on Dec 7, 2023

    @tianon
    tianon
    on Dec 7, 2023
    Member

    Any plans for a normal fix?

    #837 (comment)

    The proper fix is updating libseccomp2 (and Docker/runc) on your host. 👍

  25. tianon
    closed this as completedon Dec 7, 2023

  26. fgoura commented on Feb 28, 2024

    @fgoura
    fgoura
    on Feb 28, 2024 · Hidden as resolved

  27. tianon commented on Feb 28, 2024

    @tianon
    tianon
    on Feb 28, 2024 · Hidden as resolved

  28. fgoura commented on Feb 28, 2024

    @fgoura
    fgoura
    on Feb 28, 2024 · Hidden as off-topic

  29. anilrajrimal1 commented on Aug 7, 2025

    @anilrajrimal1
    anilrajrimal1
    on Aug 7, 2025

    All these discussions and i am still having the issue with python3.9-slim

  30. tianon commented on Aug 7, 2025

    @tianon
    tianon
    on Aug 7, 2025
    Member

    @anilrajrimal1 it's likely docker-library/official-images#16829 (or a full disk, which might be the disk of a VM if you're on Docker Desktop or similar)

  31. docker-library
    locked as resolved and limited conversation to collaborators on Aug 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions