Not working in Bitbucket pipeline: "authorization denied by plugin pipelines: Command not supported."

It seems to be both docker and testcontainers issue:

Running name.antonsmirnov.apptogether.dao.DeveloperDaoTest
08:08:56.040 [main] INFO  EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
08:08:56.060 [main] INFO  DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=tcp://localhost:2375
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='root'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=tcp://localhost:2375,registryUsername=root,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/root/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

08:08:56.066 [main] INFO  DockerClientFactory - Docker host IP address is localhost
08:08:56.227 [main] INFO  DockerClientFactory - Connected to docker: 
  Server Version: 17.12.0-ce
  API Version: 1.35
  Operating System: Alpine Linux v3.7 (containerized)
  Total Memory: 32171 MB
08:08:57.182 [testcontainers-netty-1-6] ERROR ResultCallbackTemplate - Error during callback
com.github.dockerjava.api.exception.DockerException: {"message":"authorization denied by plugin pipelines: Command not supported."}

	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:111) [testcontainers-1.7.2.jar:na]
	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [testcontainers-1.7.2.jar:na]
	at org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [testcontainers-1.7.2.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.211 sec <<< FAILURE!

Fabric docker plugin had the same issue and they were able to fix it with relatively small changes in plugin. See issues:

This is blocker issue and can be easily fixed.

testcontainers 1.7.2 used

@4ntoine

Hey @4ntoine, thank for the feedback regarding Bitbucket pipeline.
I'm not sure yet it it's easy to fix, our aux container (Ryuk) needs to mount the docker socket and from a quick look at the linked discussion, this is forbidden in Bitbucket pipelines.

@kiview

Hey @kiview test-containers works in pipelines, the issue above (as I described here: https://bitbucket.org/site/master/issues/15844/bug-in-authorization-plugin#comment-45561539) is not an issue in the auth plugin interaction with the custom java docker client nor an issue with how the library talks to docker (its working fine over tcp) its in how the MySqlContainer class by default mounts the mysql.conf file.

Users will (in order for this to work on pipelines), need to reconfigure the default (hostPath) directory for this file as we limit what paths can be mounted into docker containers due to the layout of how the containers and filesystems interact, and than the library works fine :)

@nathanburrell

Hi @nathanburrell, thanks a lot for the clarification.
So how should we ideally tackle this? Change our MySqlContainer implementation, or mention it in the docs as a workaround for pipelines compatibility?

I believe paths should be adjusted to be in allowed by Bitbucket, see debug on local machine:

Issue testcontainers#700 - Not working in Bitbucket pipeline: "author…

@4ntoine

Hey @4ntoine, would you like to submit your fix as a PR? We can look into it together.
Whoops sorry, you've already submitted the PR in #704

sorry i did not have time to fix, but i was able to use https://github.com/palantir/docker-compose-rule instead (it works)

Can you provide a way to disable Ryuk?
I have found workaround:

public abstract class TestBase {
    static {
        try {
            DockerClientFactory.instance().client();
        } catch (DockerException e) {
            if (!e.getMessage().contains("BITBUCKET_CLONE_DIR")) {
                throw new IllegalStateException(e);
            }
            // Ignore exception related to reach outside of BITBUCKET_CLONE_DIR in ResourceReaper
        }
    }
}

It's ignoring first exception caused by Ryuk trying to connect to docker.sock. Is there a way to not use sock interactions?

@dstepanov

Hi @dstepanov,

DIsabling Ryuk doesn't fix the issue, rather delays it. Does it work if you disable the startup checks? See here how to do that: https://www.testcontainers.org/usage/properties.html

@rnorth

Hey @rnorth ,

Any updates on this? Have you guys discussed it internally (cc: @kiview @bsideup)?

Having an opt-out parameter to disable Ryuk would work for all CI/CD use cases (like Bitbucket Pipelines). At the moment we have two sets of customers impacted:

Existing users using testcontainers: they will be impacted when we remove them from the exclusion list in February.
New users: currently being impacted as they can't use testcontainers with Bitbucket Pipelines.

I would appreciate your help here, so that we reduce the impact to our customers and they can keep using testcontainers happily in Bitbucket Pipelines.

Cheers,
Raul

@bsideup

Hi Raul, Nathan,
I’m very sorry for not replying sooner. Yes, we have had internal discussions and agree that disabling the Ryuk component via an environment variable is the best path forward.

@bsideup is working on it and we’d like to get this released very soon and communicated out to users.

Thank you for your help, and I apologise again.

Richard

_{Sent with GitHawk}

@raulgomis

Hi @raulgomis,
we've discussed this together in our call yesterday and we decided to solve this with #1023.

This will be a quite non invasive hack/fix and most probably won't impact the Testcontainers UX on Bitbucket Pipelines negatively, since the containers should be removed by CI environment automatically after the build.

Great news!

Thanks a lot team 🙌, our users will love that you guys are planning to fix it.

Cheers,
Raul

@raulgomis

Hi @raulgomis & @nathanburrell,

I just pushed a PR:
#1181

If you set TESTCONTAINERS_RYUK_DISABLED environment variable to true, we will not start Ryuk:
https://5c4c37a5560ab600080292aa--testcontainers.netlify.com/features/configuration/#disabling-ryuk

Please try it with Jitpack (revision 03a5c368ae) if you have time:
https://www.testcontainers.org/jitpack_dependencies/

@bsideup

@bsideup, I tried to use your PR and now I can start containers with a docker setup providing container isolation with a user namespace mapping (see #712).

@schrieveslaach

@schrieveslaach thanks for trying!

FYI the userns parameter will be supported once we integrate #1007

@bsideup

@bsideup, thanks for pointing out this PR. I was going to ask you exactly that question. 😉

We've now released this in Testcontainers 1.10.6!

Please upgrade as soon as you can and follow the steps here for disabling Ryuk: https://www.testcontainers.org/features/configuration/#disabling-ryuk

I'll close this ticket now, but please keep us posted if there are any new issues with this.

Thanks a lot, this is working like a charm in Bitbucket 😄

Thanks for actioning this quickly :)

We are organising some communicuations to customers in the coming weeks to tell them to update and set that variable, and I can also continue to start using test-containers more in the pipelines codebase itself ;)

@nathanburrell

Hey, @nathanburrell is test-containers fully working on the bitbucket pipeline at this point? I'm, guessing that it is, but I'm having some problems using it with Cassandra. Basically, locally the tests passed but on bitbucket pipeline that seems to be a problem to communicate with the Cassandra container.

Hi,

Thanks a lot. The issues #700 and #1135 helped me, to get testcontainers running in Jenkins with dind-rootless inside Kubernetes.
In hope, this might help someone else, a stripped down example for a Jenkins build job pod definition:

apiVersion: v1
kind: Pod
spec:
  securityContext:
    fsGroup: 1000
    runAsGroup: 1000
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: dind
      image: docker:20.10-dind-rootless
      imagePullPolicy: Always
      env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      securityContext:
        # still needed: https://docs.docker.com/engine/security/rootless/#rootless-docker-in-docker
        privileged: true
        readOnlyRootFilesystem: false
    - name: gradle
      image: openjdk:11
      imagePullPolicy: Always
      env:
        - name: DOCKER_HOST
          value: tcp://localhost:2375
        # needed to get it working with dind-rootless
        # more details: https://www.testcontainers.org/features/configuration/#disabling-ryuk
        - name: TESTCONTAINERS_RYUK_DISABLED
          value: "true"
      command:
        - cat
      tty: true
      volumeMounts:
        - name: docker-auth-cfg
          mountPath: /home/ci/.docker
      securityContext:
        capabilities:
          drop:
            - ALL
        allowPrivilegeEscalation: false
        privileged: false
        readOnlyRootFilesystem: false
  volumes:
    - name: docker-auth-cfg
      secret:
        secretName: docker-auth

It seems upgrading test containers version to 1.16.0 solved the issue.

Navigation Menu

Repository navigation

Not working in Bitbucket pipeline: "authorization denied by plugin pipelines: Command not supported." #700

22 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Not working in Bitbucket pipeline: "authorization denied by plugin pipelines: Command not supported." #700

Description

Activity

4ntoine commented on May 17, 2018

kiview commented on May 17, 2018

nathanburrell commented on May 18, 2018

kiview commented on May 18, 2018

4ntoine commented on May 18, 2018

kiview commented on Jun 8, 2018

4ntoine commented on Jun 8, 2018

dstepanov commented on Aug 6, 2018

bsideup commented on Aug 6, 2018

22 remaining items

raulgomis commented on Jan 24, 2019

rnorth commented on Jan 24, 2019

kiview commented on Jan 24, 2019

raulgomis commented on Jan 24, 2019

bsideup commented on Jan 25, 2019

schrieveslaach commented on Jan 25, 2019

bsideup commented on Jan 25, 2019

schrieveslaach commented on Jan 25, 2019

rnorth commented on Jan 30, 2019

modye commented on Jan 30, 2019

nathanburrell commented on Jan 30, 2019

andreppires commented on May 28, 2019

czunker commented on Aug 19, 2021

Maciag312 commented on Oct 13, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions