Repository navigation
Codeship builds fail: can not connect to Ryuk #581
Description
TestContainers works locally. Builds on Codeship fail with the following:
2018-02-10_19:22:35.027 WARN o.t.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:32768
java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:134)
at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:136)
at java.lang.Thread.run(Thread.java:748)
2018-02-10_19:22:35.044 WARN o.t.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:32768
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:139)
at java.lang.Thread.run(Thread.java:748)
I do not use volume mapping for this service, but I use volumes to build other services. My understanding of https://www.testcontainers.org/usage/inside_docker.html is that I do not need to mount the Docker socket volume.
I've set add_docker: true in my codeship-services.yml, which does the following:
Add the environment variables DOCKER_HOST, DOCKER_TLS_VERIFY and DOCKER_CERT_PATH from the host.
If DOCKER_CERT_PATH is set, it will mount the certificate directory through to the container. See add_docker for an example using Docker-in-Docker.
https://documentation.codeship.com/pro/builds-and-configuration/services/
Activity
estebangarcia commentedon Mar 30, 2018
I'm having the same issue with CircleCI. Using v1.6.0
kiview commentedon Mar 31, 2018
Hi @estebangarcia, would it be possible for you to migrate to CircleCI 2.0?
We are using CircleCI 2.0 in Testcontainers ourselves, so everything should work fine out of the box (look at our CircleCI config).
Since pre 2.0 CircleCI was using lxc directly themselves (AFAIK?), you have to jump through a lot more hoops to get Testcontainers working there.
bsideup commentedon Apr 11, 2018
Hi @gavinmh,
Could you please try with 1.7.0?
kiview commentedon May 15, 2018
Since this is possibly fixed by the fix for #621 I'll close this for now.
@gavinmh I'll gladly reopen the issue if the problems persists for you on 1.7.0 🙂
kszalontai commentedon Jul 27, 2018
I'm having the same issue with version 1.8.1
Codeship pro,
add_docker=true,ryuk.container.timeout = 120sic2 commentedon Aug 2, 2018
Yes, please. Re-open this issue since it still persists in 1.8.1
kszalontai commentedon Aug 6, 2018
@kiview Could you please reopen this? ty
kiview commentedon Aug 6, 2018
Reopened as requested, bug seems to be still present.
bountin commentedon Aug 13, 2018
We're experiencing the same when the load on the build server (Jenkins) is (very) high.
One thing that eased the pain was upgrading docker where we used to use an early version of 2017. The docker service wasn't responsive during peak testing (eight parallel test runs) which lead me to that direction.
Still, we had to reduce the number of parallel builds because of testcontainer timeouts. I haven't tried to increase
ryuk.container.timeoutbecause IMO thirty seconds should be enough - also in a high load scenario.thammerl commentedon Aug 20, 2018
@bountin I don't think the timeout is related to the issue we are having. I turned on the testcontainers logging and this is what I got:
As you can see, only 1.7 seconds passed between connecting to docker and the socket exception.
thammerl commentedon Aug 20, 2018
After having a closer look at the code, I came up with the following findings/questions:
Obviously there were two different threads trying to initialize the ryuk container.Shouldn't this only be executed once? Is the potential race condition inDockerClientFactoryproblematic? UPDATE: There are two stack traces because theindexvariable isn't incremented and the outermost while(true)-loop will try to connect again. As there aren't any further stack traces, I assume that the thread gets stuck in the while-loop waiting for the "ACK" response.Is it reasonable to just log a warning here? TheI understand now that this retry-logic is intentional to wait until the ryuk server is available.ryukScheduledLatchwill never be decremented to zero and will result in an IllegalStateException once the timeout is reached. Is it possible to recover from anIOExceptionthat is thrown when sending the filters to ryuk?ResourceReaperfrom the stack trace don't match. The same is true for 1.8.2. Haven't checked any other releases. The sources for the releases on GitHub are fine.Sorry if the answers to some of my questions are obvious. I'm new to the project and only guessing how things should play together here.
thammerl commentedon Aug 21, 2018
Now I had a build with numerous stack traces instead of just two. But after some attempts to connect it seems to get stuck in the while-loop waiting for the acknowledgment and after the 30 second timeout the
IllegalStateExceptionkicks in:testcontainers.log
thammerl commentedon Aug 21, 2018
I've created a pull request that should fix the issue. See #843.
vpavic commentedon Aug 27, 2018
We're also occasionally experiencing the same issue with Spring Session builds on Jenkins. Here's the stacktrace from one of those builds:
kiview commentedon Oct 1, 2018
@vpavic Does this still happen for you, or was it resolved with #843 in 1.9.0?
vpavic commentedon Oct 1, 2018
Thanks for the ping - I forgot to follow up on this.
We were still experiencing the problem with
1.9.0-rc1, and have setryuk.container.timeout=120property since. This appears to have fixed it, but we've also had some updated on this infrastructure side and have upgrade to1.9.1since.I'll try to remove
ryuk.container.timeout=120some time this week and see how it behaves, just to tryand isolate the problem.
guss77 commentedon Oct 29, 2018
I'm having the same issue with
1.9.1on Circle CI (2.0). I'm running testcontainers for a JDBC server for Jooq code generation:I can't figure out how to set
ryuk.container.timeout=120- usind-Don the maven command line doesn't seem to do anything.Here's the full debug log when running Maven with
-X:vpavic commentedon Oct 30, 2018
@guss See the docs on custom properties.
As a side note for @kiview and @bsideup, it would be nice to have
ryuk.container.timeout(or any other missing properties) added to that page for a complete reference.try to build on circle ci without test containers as it currently fai…
guss77 commentedon Oct 30, 2018
Sorry about the weird link, Github is sometimes too enthusiastic about referencing issues.
I've tried with setting the timeout to 120, and indeed the build takes 2 more minutes but the Ryuk connection still fails:
rafaelalmeida commentedon Dec 12, 2018
@guss77 Did you manage to solve this? Running into the same issue now with
1.10.2.ysb33r commentedon Feb 6, 2019
I am seeing the same issue with 1.10.6 but only on Mac running Docker Desktop
Similar stack trace:
I am running this side-by-side with a Linux laptop (Docker engine 18.06.0-dev build e68fc7a) that has less memory than the Mac, but I am not seeing the issue there.
kiview commentedon Feb 7, 2019
@ysb33r Did you already try to increase the timeout in
ryuk.container.timeout?https://www.testcontainers.org/features/configuration/
ysb33r commentedon Feb 7, 2019
Yes I have set it to 120, but possibly a complete restart of Docker Desktop seemed to have resolved the problem.
ciri-cuervo commentedon Apr 15, 2019
have tried creating the docker group? it solves permission issues
AlexanderArgyriou commentedon Jun 9, 2023
Hello, any update here ? https://stackoverflow.com/questions/76442027/cannot-connect-to-ryuk