Repository navigation
Latest Docker for Mac - apparent change to port forwarding breaks TCP liveness checks #160
Description
This long-standing part of our container liveness checks seems to be broken on Docker for Mac from at least beta 14 onwards:
new Socket(ipAddress, port).close();
The original behaviour of this was to simply verify that the relevant TCP port was listening, catching an IOException if the container was not listening yet. However, now Docker for Mac seems to be running its own perpetually listening TCP proxy that accepts the connection regardless of whether there is actually a listening process inside the container.
e.g. outside of testcontainers, this demonstrates the behaviour:
$ docker run -p 8080 -d alpine:latest sh -c "yes | nc -l -p 8080" # real listening process inside container
$ docker run -p 8080 -d alpine:latest sh -c "yes > /dev/null" # port mapped, but not listening
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c77d07ba926c alpine:latest "sh -c 'yes | nc -l -" About a minute ago Up About a minute 0.0.0.0:32822->8080/tcp small_mirzakhani
871c2c80ceaf alpine:latest "sh -c 'yes > /dev/nu" 6 minutes ago Up 6 minutes 0.0.0.0:32821->8080/tcp tender_raman
$ lsof | grep LISTEN
[.. snip ..]
com.docke 1481 rnorth 11u IPv4 0x90cfe27e6cde7eb3 0t0 TCP *:32821 (LISTEN)
com.docke 1481 rnorth 13u IPv4 0x90cfe27e6a3ab7ab 0t0 TCP *:32822 (LISTEN)
This seems to make the current wait check too dumb!
I'd be keen to know if others are experiencing this.
I suspect the fix should include opening an inputstream and verifying that the first read doesn't indicate end-of-stream, but I'd like to do some checking to be sure that this doesn't have unintended consequences.
Activity
rnorth commentedon Jun 16, 2016
Scratch that idea as a potential fix - as I suspected, it only works where the container process has something to send and doesn't wait for data from the client first.
rnorth commentedon Jun 18, 2016
Brief update: doing the equivalent of
docker exec <id> cat /proc/net/tcpmight potentially be a solution for checking whether the container is listening on a port, but for test cases at least it's impeded by gliderlabs/docker-alpine#143. It seems that a netcat process in the container (as used for simulating a simple running server in the testcontainers test suite) does not expose listening port data in the TCP socket table for some reason.rnorth commentedon Jun 18, 2016
Should be fixed in a later DfM beta, as per this post: https://forums.docker.com/t/port-forwarding-of-exposed-ports-behaves-unexpectedly/15807/2
Add custom Redis wait strategy as an example and to mitigate #160
rnorth commentedon Jul 6, 2016
Current recommended workaround for this issue is to implement a custom service-specific
WaitStrategythat checks the service is correctly running. e.g. use an actual client library for the service to check, such as in 64d3451.bsideup commentedon Oct 24, 2016
@rnorth got bitten by this issue. Looks like it's not fixed in the latest release of DfM. Do you have any ideas what to do with it?
Update:
I managed to workaround it by using:
It uses BASH's magical device
/dev/tcp/$HOST/$PORT. Not sure if we can use it in TC until Docker is fixed, but at least it helps to workaround the issue.rnorth commentedon Oct 24, 2016
Ah, interesting @bsideup. The workaround (that IIRC @outofcoffee has used) is a service-specific wait strategy. e.g. to change the test from 'is the socket listening?' to 'does the socket talk HTTP?'
It's frustrating. Your workaround is very clever; I'd tried something similar but not as elegant 😆
I think bringing this into TC and making it the default might be a hack too far, but perhaps there's some merit in making it available so people can opt-in..? I like the principle of us doing the hard/dirty work so that users don't have to 😄
bsideup commentedon Oct 25, 2016
@rnorth doing it right now (bash-based port check in Docker for Mac environment), will send PR soon
Use command-based port check in HostPortWaitStrategy in case of Docke…