Can't connect to Ryuk container when using remote Docker host

Version 1.6.0
Not sure if this came from multiple threads but never seen it with single thread.
Many tests use several types of containers.

java.lang.ExceptionInInitializerError
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:105)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:355)
	at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Can not connect to Ryuk
	at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:158)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:116)
	at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:126)
	at org.testcontainers.containers.JdbcDatabaseContainer.<init>(JdbcDatabaseContainer.java:35)
	at org.testcontainers.containers.PostgreSQLContainer.<init>(PostgreSQLContainer.java:28)
	at com.tests.container.DbContainer.<init>()
	at com.tests.MyTest.<clinit>()
	... 27 more

@alexcase52

Hi @alexcase52,

Could you please try with 1.5.1 + multiple threads? I want to make sure that this is a regression in 1.6.0 :)

@alexcase52

Hi @alexcase52,

I get also the error only on 1.6.0, 1.5.1 is fine.
Testing on Windows 10 with Java9.

Is deamon exposition on tcp://xy:2375 without TLS mandatory ?

@llfbandit

@llfbandit

Is deamon exposition on tcp://xy:2375 without TLS mandatory ?

This is currently still mandatory (until we will be able to support npipes or Docker for Windows will support Unix sockets).

https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/

Well... kind of tiny scope for now...

I'm experiencing a similar issue when running on Jenkins, with a DOCKER_HOST pointing to some other host:

  2018-04-03 17:59:19.719  WARN --- [containers-ryuk[] o.testcontainers.utility.ResourceReaper  : Can not connect to Ryuk at 10.17.48.167:32800
  java.net.ConnectException: Connection refused
  	at java.net.PlainSocketImpl.socketConnect(Native Method)
  	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
  	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  	at java.net.Socket.connect(Socket.java:589)
  	at java.net.Socket.connect(Socket.java:538)
  	at java.net.Socket.<init>(Socket.java:434)
  	at java.net.Socket.<init>(Socket.java:211)
  	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:111)
  	at org.testcontainers.utility.ResourceReaper$$Lambda$41/1921545475.run(Unknown Source)
  	at java.lang.Thread.run(Thread.java:745)

Downgrading to 1.5.1 fixes the issue.

reverting testContainers version to 1.5.1 from 1.6.0 due to issue in …

@alvarosanchez

@alvarosanchez
Thanks for the info, is 10.17.48.167 the ip of your DOCKER_HOST? Should be. Could this be a problem with a firewall?
I assume unrelated, but I've experienced this problem when inside a network with a captive portal, so this can be some DNS hickup.

is 10.17.48.167 the ip of your DOCKER_HOST?

Yes, it is.

Could this be a problem with a firewall?

No, because then version 1.5.1 wouldn't work either. Note that the client machine (a Jenkins instance) is able to run Docker commands on that DOCKER_HOST via docker CLI and also the Gradle plugin.

@alvarosanchez

@alvarosanchez
With version 1.6.0 we've introduced a new implementation for the ResourceReaper component, which will now run outside of the JVM in its own Docker container (the container is called Ryuk and the error appears when trying to connect to this container).
Since we don't have an automated CI build for this scenario (remote DOCKER_HOST), I'm not sure if this implementation broke the compatibility with some remote scenarios.

I've changed the title of the issue to better reflect the current state of knowledge we have, I hope it's okay for everyone involved?

@alexcase52 Were you able to find out if this issue is related to multiple Gradle threads?

@alvarosanchez

@alvarosanchez
Have you tried using 1.7.0? Maybe the container startup ran into a timeout as mentioned in #621.

@kiview

@kiview 1.7.2 :)

Hi, Just encountered this on 1.7.3 running a test on our Jenkins server:

9:26:52.538 [main] INFO  org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
09:26:52.555 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=tcp://NAME_OF_MY_DOCKER_HOST:2375
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='jenkins'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=tcp://NAME_OF_MY_DOCKER_HOST:2375,registryUsername=jenkins,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/var/lib/jenkins/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

09:26:52.557 [main] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is NAME_OF_MY_DOCKER_HOST
09:26:52.921 [main] INFO  org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 17.09.0-ce
  API Version: 1.32
  Operating System: CentOS Linux 7 (Core)
  Total Memory: 7823 MB
09:26:54.359 [testcontainers-ryuk] WARN  org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at NAME_OF_MY_DOCKER_HOST:32774
java.net.NoRouteToHostException: No route to host (Host unreachable)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at java.net.Socket.<init>(Socket.java:434)
	at java.net.Socket.<init>(Socket.java:211)
	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
	at java.lang.Thread.run(Thread.java:748)

Hi, I have similar error on 1.7.3 inside Jenkins image.
In docker processes I see that Ruak image has been started, but it impossible to connect to it for some Reason...

10:00:00.109 [main] INFO  org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
10:00:00.129 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='root'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=root,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/root/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

10:00:00.989 [main] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
10:00:01.134 [main] INFO  org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 17.03.2-ce
  API Version: 1.27
  Operating System: Ubuntu 14.04.5 LTS
  Total Memory: 16047 MB
10:00:02.686 [testcontainers-ryuk] WARN  org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:39972
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at java.net.Socket.<init>(Socket.java:434)
	at java.net.Socket.<init>(Socket.java:211)
	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
	at java.lang.Thread.run(Thread.java:748)
10:00:03.692 [testcontainers-ryuk] WARN  org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:39972
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at java.net.Socket.<init>(Socket.java:434)
	at java.net.Socket.<init>(Socket.java:211)
	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
	at `java.lang.Thread.run(Thread.java:748)

We ran into the same problem as well

java.lang.IllegalStateException: Can not connect to Ryuk
	at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:148)
	at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:64)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:124)
	at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:142)
	at org.testcontainers.containers.JdbcDatabaseContainer.<init>(JdbcDatabaseContainer.java:45)
	at org.testcontainers.containers.MariaDBContainer.<init>(MariaDBContainer.java:26)

We're using Testcontainers inside a custom JUnit-Rule which

creates a Network
creates two containers using this network
starts the containers inside the before-method within independent threads
stops those containers inside the after-method

This might not be the polite way to use Testcontainers but this replaced some legacy code and works much better than before.
We've started the tests several times, this error seems to occur in a non-deterministic way.

Using Testcontainers version 1.10.5.

have tried creating the docker group? it solves permission issues

# create new group
sudo groupadd docker
# add current user to the group
sudo usermod -aG docker $USER
# log in to the group
newgrp - docker

Please see #1274 . It is very likely that the local UNIX socket is not opened on your docker host.
Adding -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375 when running the docker server should solve your issue.

Ryuk seems to be listening on random port(33088 and 33089 and so on). This is unacceptable when docker host is behind firewall. I can't open all the port only for Ryuk.

@skyline75489

@skyline75489
It is not just Ryuk, but every container we start will be listening on a random port.

@bsideup

@bsideup
And is there any way to force Ryuk to bind on the specific port not the random one ? We are having the similar issues with docker-host behind the firewall.

@Wosin

@Wosin no, sorry. As I said - it is not just Ryuk, but any container we start.

I've run into a similar problem but on localhost. After starting a docker container, the program hanged for two minutes trying to connect to it (only info log is saved, but there were requests every second in debug log). After that, while obtaining connection, it raised com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure Caused by: java.net.ConnectException: Connection refused (Connection refused) (stacktrace).

Reboot solved the problem. I think that the reason for such behavior was a nearly full load of the RAM with enabled swap, that had an about quarter load.

If someone comes here having broken bitbucket pipeline with error org.testcontainers.utility.ResourceReaper : Can not connect to Ryuk at localhost:32768 java.net.SocketException: Broken pipe (Write failed) then here is the solution:
https://www.testcontainers.org/supported_docker_environment/continuous_integration/bitbucket_pipelines/

Closing as outdated, since the issue seemed to be caused by network misconfiguration (see #572 (comment) for an advice on how to fix it)

Try testcontainers/testcontainers-java#572 (comment)

What do you think about this variable TESTCONTAINERS_HOST_OVERRIDE=host.docker.internal ? The source is here testcontainers configuration and this can be relevant for some people also https://github.com/testcontainers/testcontainers-java/blob/main/docs/supported_docker_environment/continuous_integration/dind_patterns.md

Navigation Menu

Repository navigation

Can't connect to Ryuk container when using remote Docker host #572

20 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Can't connect to Ryuk container when using remote Docker host #572

Description

Activity

bsideup commented on Feb 2, 2018

llfbandit commented on Feb 7, 2018

kiview commented on Feb 7, 2018

llfbandit commented on Feb 7, 2018

alvarosanchez commented on Apr 3, 2018

kiview commented on May 14, 2018

alvarosanchez commented on May 14, 2018

kiview commented on May 14, 2018

kiview commented on May 15, 2018

bsideup commented on May 15, 2018

lon-pilot-mpf commented on May 23, 2018

deshchenko commented on May 24, 2018

20 remaining items

StefanHufschmidt commented on Feb 6, 2019

ciri-cuervo commented on Apr 15, 2019

mariusneo commented on Apr 15, 2019

skyline75489 commented on Jun 18, 2019

bsideup commented on Jun 18, 2019

Wosin commented on Jul 12, 2019

bsideup commented on Jul 12, 2019

errandir commented on Sep 6, 2019

vkopichenko commented on Sep 9, 2019

bsideup commented on Nov 9, 2020

psadovsky commented on Jan 5, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions