Alpine 3.14 images can fail on Docker versions older than 20.10

We run a slightly old Container-Optimized OS from Google (13310.1041.24) on thousands of Google N2D machines, and after the latest updates using Alpine v3.14, we noticed all sorts of strange failures:

cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:3.0.2-alpine3.13 bundle --version
Bundler version 2.2.22
cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:3.0.2-alpine3.14 bundle --version
`/root` is not writable.
Bundler will use `/tmp/bundler20210709-1-4lza5z1' as your home directory temporarily.
Bundler version 2.2.22

cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:2.7
irb(main):001:0>
cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:2.7-alpine
Traceback (most recent call last):
	6: from /usr/local/bin/irb:23:in `<main>'
	5: from /usr/local/bin/irb:23:in `load'
	4: from /usr/local/lib/ruby/gems/2.7.0/gems/irb-1.2.6/exe/irb:11:in `<top (required)>'
	3: from /usr/local/lib/ruby/2.7.0/irb.rb:393:in `start'
	2: from /usr/local/lib/ruby/2.7.0/irb/init.rb:18:in `setup'
	1: from /usr/local/lib/ruby/2.7.0/irb/init.rb:121:in `init_error'
/usr/local/lib/ruby/2.7.0/irb/locale.rb:121:in `load': No such file to load -- irb/error.rb (LoadError)

cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:3.0.2-alpine3.13
irb(main):001:0>
cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:3.0.2-alpine3.14
/usr/local/lib/ruby/3.0.0/irb/locale.rb:121:in `load': No such file to load -- irb/error.rb (LoadError)
	from /usr/local/lib/ruby/3.0.0/irb/init.rb:195:in `init_error'
	from /usr/local/lib/ruby/3.0.0/irb/init.rb:18:in `setup'
	from /usr/local/lib/ruby/3.0.0/irb.rb:402:in `start'
	from /usr/local/lib/ruby/gems/3.0.0/gems/irb-1.3.5/exe/irb:11:in `<top (required)>'
	from /usr/local/bin/irb:23:in `load'
	from /usr/local/bin/irb:23:in `<main>'

https://alpinelinux.org/posts/Alpine-3.14.0-released.html mentions:

The faccessat2 syscall has been enabled in musl. This can result in issues on docker hosts with older versions of docker (<20.10.0) and libseccomp (<2.4.4), which blocks this syscall.

Our version:

cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker --version
Docker version 19.03.9, build 9d98839

Can we rollback the Alpine v3.14 upgrade and make this opt-in instead of the default while we work to upgrade our systems?

I should note this problem did not occur on Google's supported cos-85-13310-1260-2 image:

Related links:

https://bugzilla.redhat.com/show_bug.cgi?id=1900021
Docker fix: Add openat2 and faccessat2 to default seccomp profile. moby/moby#41353
Backport to Docker 19.03.16: [19.03 backport] seccomp profile updates moby/moby#41381

Can we rollback the Alpine v3.14 upgrade and make this opt-in instead of the default while we work to upgrade our systems?

just use the alpine3.13 tag

@stanhu

@stanhu thank you for this, I ran into this issue a few hours before this post and was finally at the part of looking at the alpine change as the cause (issues running containers on ECS Fargate) but am glad to see you having the same issue.

I'll use the 3.13 tag as @J0WI suggests.

Thanks!

just use the alpine3.13 tag

We run a CI service where users can choose any tag, and existing builds that use ruby:2.7-alpine or other builds abruptly have stopped working. We're now trying to upgrade the OS, but this issue seems like it could burn others.

From another alpine-bug issue docker-library/php#1177 (comment)

The more specific aliases exist specifically to give users an escape hatch to stay on a specific underlying distribution release and control their own consumption and upgrade timing. Just like users using :latest, users using a bare alpine tag should expect some degree of breakage over time, and I don't think there's much we can reasonably do about that.

(This is also the common pattern of usage/tagging across all the official images -- PHP is not unique in this regard.)

I think the only thing unique about this particular bump is that the most common breakage is more disruptive than usual, but there's not really much we can do about that, given it's a change in musl/Alpine itself (https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0#faccessat2).

We're running into this same issue with Docker 20.10.2 running on Ubuntu LTS. Docker is installed from the Docker ubuntu repo.

Clearly rolling down to 3.13 "solves" things, but these seem like broken 3.14 images to me.

EDIT: Resolved by upgrading docker-ce to 20.10.7.

@jamesbrooks

The same problem with alpine 3.14 on Fargate.
@jamesbrooks If you find a way to use the 3.14 on Fargate, please let us know.

Use Alpine 3.13

This problem happens to me on AWS Fargate (1.4) with ruby:2.7-alpine (1.14). It works when downgraded to ruby:2.7-alpine3.13.

This problem happens to me on macOS Big Sur (Version 11.5.1) with ruby:2.7.4-alpine. Although, my colleagues do not encounter the same issue in Ubuntu.

Docker version 20.10.7, build f0df35096d
docker-compose version 1.29.2, build unknown
docker-machine version 0.16.2, build bd45ab1
Dinghy 4.6.5

Gemfile

source "https://rubygems.org"
git_source(:github) { |repo| "https://github.com/#{repo}.git" }

ruby "2.7.3"

gem "bootsnap", ">= 1.4.4", require: false
gem "puma", "~> 5.0"
gem "rails", "~> 6.1.4"
gem "rack-cors", "~> 1.1"

gem "active_interaction", "~> 4.0"

group :database, :default do
  gem "discard", "~> 1.2"
  gem "pg", "~> 1.1"
  gem "strong_migrations", "~> 0.7"
end

group :graphql, :default do
  # TODO: manage graphiql through docker specific image
  gem "graphql", "~> 1.12"
  gem "graphql-batch"
  gem "graphql-rails_logger", "~> 1.2"
end

group :development do
  gem "annotate"
  gem "listen", "~> 3.3"
  gem "spring"
end

group :test do
  gem "factory_bot_rails"
  gem "faker"
  gem "rspec-rails"
  gem "rswag-specs"
  gem "simplecov"
  gem "spring-commands-rspec"
end

group :code_quality, :development, :test do
  gem "bullet"
  gem "byebug"
  gem "pry"
  gem "pry-byebug"
  gem "pry-rails"
  gem "reek", require: false
  gem "rubocop", require: false
  gem "rubocop-graphql", require: false
  gem "rubocop-performance", require: false
  gem "rubocop-rails", require: false
  gem "rubocop-rspec", require: false
  gem "stackprof", require: false
end

group :instrumentation, :default do
  gem "ddtrace", require: "ddtrace/auto_instrument"
  gem "sentry-rails", "~> 4.2"
end

I have created this Dockerfile:

FROM ruby:2.7.4-alpine

RUN apk add build-base tzdata postgresql-dev postgresql-client

WORKDIR /usr/src/app
COPY Gemfile* ./
RUN bundle install

# Add a script to be executed every time the container starts.
COPY entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/entrypoint.sh
ENTRYPOINT ["entrypoint.sh"]
EXPOSE 3000

COPY . .

CMD ["rails", "server", "-b", "0.0.0.0"]

The error occurs when running the bundle install:

Building api
Sending build context to Docker daemon  438.8kB
Step 1/11 : FROM ruby:2.7.4-alpine
 ---> edd3dc5af692
Step 2/11 : RUN apk add build-base tzdata postgresql-dev postgresql-client
 ---> Using cache
 ---> 7370fb782090
Step 3/11 : WORKDIR /usr/src/app
 ---> Running in 4f66666b84ac
Removing intermediate container 4f66666b84ac
 ---> 20cc36d2f120
Step 4/11 : COPY Gemfile* ./
 ---> 63eee2b21da9
Step 5/11 : RUN bundle install
 ---> Running in e13ccb87957b
`/root` is not writable.
Bundler will use `/tmp/bundler20210727-1-1x1ft1f1' as your home directory temporarily.
Fetching gem metadata from https://rubygems.org/..........
Fetching rake 13.0.6
Installing rake 13.0.6
Gem::FilePermissionError: You don't have write permissions for the
/usr/local/bundle directory.
An error occurred while installing rake (13.0.6), and Bundler cannot continue.
Make sure that `gem install rake -v '13.0.6' --source 'https://rubygems.org/'`
succeeds before bundling.

In Gemfile:
  annotate was resolved to 3.1.1, which depends on
    rake
The command '/bin/sh -c bundle install' returned a non-zero code: 5
ERROR: Service 'api' failed to build : Build failed

starlight: pin alpine to 3.13

@jamesbrooks

The same problem with alpine 3.14 on Fargate.
@jamesbrooks If you find a way to use the 3.14 on Fargate, please let us know.

This problem happens to me on AWS Fargate (1.4) with ruby:2.7-alpine (1.14). It works when downgraded to ruby:2.7-alpine3.13.

@morgoth @samnang > seems to be working on Fargate 1.4.0 now, tested with ruby:alpine

@alexbrebu

please see the alpine linux 3.14 release notes: https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0#faccessat2. you should always read the release notes when upgrading linux distro, or expect surprises.

if, after reading the release notes, it is still not working for anybody, please post:

docker version (not docker --version) and docker info. if you are using hosted Docker, then your provider name and relevant information (e.g. advertised Linux distro and version).
full docker invocation, input (e.g. Dockerfile or stdin), and output

to avoid cluttering the issue, please use <details> element for big output, e.g. https://gist.github.com/ericclemmons/b146fe5da72ca1f706b2ef72a20ac39d.

@alexbrebu see above, and if you are using docker desktop, what version?

@stanhu, docker 19.03 is officially EOL. however, you can continue using it (at your own risk) if you upgrade to runc v1.0.0-rc93 or apply the documented workaround. or, if you don't care about^W^W^Wdon't need security at the OS level, which is the only decent reason to be running without ten months of security updates, then just use --security-opt=seccomp=unconfined?

although i don't maintain this image, i don't think it is a good idea to downgrade the version. it would be unexpected for users to suddenly get a lower version, as well as a version not matching the latest version advertised on the alpine linux website. additionally, some people may be using the community repo, which is not supported beyond the latest alpine linux release.

Closing because this isn't something we can fix in the images -- the proper solution is to either use the alpine3.13 images or update Docker, runc, and/or libseccomp (with the latter being the better option long term).

This does appear to be an issue on Azure App Service. Downgrading to Alpine 3.13 resolved the issue for me. Just leaving some breadcrumbs for anyone who finds this thread with the same issue.

I solved this issue by replacing dinghy setup and docker (installed with homebrew) with Docker for Mac.

Step 1

dinghy destroy
brew uninstall docker-machine docker-compose dinghy docker docker-machine-nfs

Step 2
Check your configuration file and make sure you clean DOCKER_* envs if any

cat ~/.bash_profile
cat ~/.bashrc

I was running into an issue where installing a gem from a Git repo source failed. I would get the following error when running bundle install:

You need to install git to be able to use gems from git repositories

bundle exec git version would also fail, saying the command was not found.

Updating docker via sudo apt-get install docker-ce docker-ce-cli resolved the issue. Posting this just in case one other person has a similar problem.

Fix bundle gems issue in docker

AAC-329 Setup accessibility testing and linting in Cypress (#874)

Apologies for bumping a phantom thread (Although it's not that old).

The advice I found out through different mediums (Use alpine 3.13), works.... However .....

Ruby 3.1.x series in dockerhub doesn't feature an alpine 3.13 build combination. As such this issue now prohibits people from using Ruby 3.1.x

I personally get this issue when using Docker in Docker. And have raised a few issues (Probably in the wrong places)

Any advice from people on next steps? Asking whoever maintains said images under ruby:xxxx in docker hub to promote / create an alpine 3.13 one would fix the issue for 1year in theory and feels like a reasonably simple set of maintenance steps to give an immediate fix?

The correct solution here is to update Docker and libseccomp on your host.

is libseccomp a package. So something like apt-get libseccomp?

On Debian (and derivatives like Raspbian, Ubuntu, etc) the package name is libseccomp2.

FYI This doesn't fix the issue. But I've raised it elsewhere.

=> CANCELED [internal] load build context 18.1s
=> => transferring context: 494.25MB 18.1s
=> [1/8] FROM docker.io/library/ruby:2.6-alpine@sha256:382ce92de42ef027bf1bfe382c3f3c3878042c41c07da8b8aa41855db0894762 0.0s
=> CACHED [2/8] RUN apk add git && apk add --update --no-cache jq curl firefox-esr xvfb mesa-dev mesa-gl chromium-chromedriver chromium build-ba 0.0s
=> ERROR [3/8] RUN gem install rails -v '5.2.4' 18.1s

[3/8] RUN gem install rails -v '5.2.4':
#6 17.73 ERROR: While executing gem ... (Gem::FilePermissionError)
#6 17.73 You don't have write permissions for the /usr/local/bundle directory.

Try older alpine (docker-library/ruby#351 (comment))

Navigation Menu

Repository navigation

Alpine 3.14 images can fail on Docker versions older than 20.10 #351

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Alpine 3.14 images can fail on Docker versions older than 20.10 #351

Description

Activity

J0WI commented on Jul 9, 2021

jamesbrooks commented on Jul 9, 2021

stanhu commented on Jul 9, 2021

wglambert commented on Jul 9, 2021

simmerz commented on Jul 12, 2021

morgoth commented on Jul 12, 2021

samnang commented on Jul 19, 2021

alexbrebu commented on Jul 27, 2021

rohanmangal commented on Aug 6, 2021

Hello71 commented on Aug 15, 2021

1 remaining item

tianon commented on Aug 20, 2021

dhnaranjo commented on Sep 15, 2021

alexbrebu commented on Oct 15, 2021

KashubaK commented on Nov 15, 2021

luke-hill commented on Mar 1, 2022

tianon commented on Mar 1, 2022

luke-hill commented on Mar 2, 2022

tianon commented on Mar 2, 2022

luke-hill commented on Mar 3, 2022

David-Mysize commented on Jun 12, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions