Authentication for Docker Hub private registry without credential helpers

Following up here on a question about authentication for a private repo hosted at docker hub that I posted on Slack.
For some context, credentials are stored in ~/.docker/config.json in the auth field (this is actually base64(username:password)) eg:

{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "abcdef=",
            "email": "peter.vr@acme.com"
        }
    },
    "HttpHeaders": {
        "User-Agent": "Docker-Client/18.03.1-ce (darwin)"
    }
}

We supply image names such acme/microservice:0.1 to testcontainers. There are also public images from Docker Hub (redis, postgres etc) and implicitly from quay.io (ryuk). Now, looking at the following code:

https://github.com/testcontainers/testcontainers-java/blob/master/core/src/main/java/org/testcontainers/utility/RegistryAuthLocator.java#L80

reposName ends up being "" (empty) and it falls back to defaultAuthConfig which I believe is controlled by ~/.docker-java.properties (ala docker-java project), which does not exist and hence it fails trying to download the image:

2018-08-05 14:22:22,061 ERROR [testcontainers-netty-1-13] c.g.d.c.a.ResultCallbackTemplate: Error during callback
com.github.dockerjava.api.exception.NotFoundException: {"message":"pull access denied for acme/microservice, repository does not exist or may require 'docker login'"}

	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:103)
	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33)
	at org.testcontainers.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)

In contrast, the default behaviour of docker pull in the absence of a registry host, is to pull from docker hub and as such matches the URL https://index.docker.io/v1/ in ~/.docker/config.json - this does not appear to be the default behaviour of testcontainers.

I would like to see testcontainers choosing the auth from ~/.docker/config.json in the https://index.docker.io/v1/ stanza within findExistingAuthConfig(config, ""). To pursue this behaviour I put together this PR: #819 - I ran into a number of dead ends:

Just replacing the reposName with index.docker.io in case it's empty does not work, docker-java explicitly checks for that hostname and errors out (why???? it's completely valid using the docker command line docker pull index.docker.io/acme/microservice:0.1)
Modifying the auth lookup to find the stanza for index.docker.io in config.json (as in this PR: WIP: Add support for docker hub private registry credentials #819) does not work either, from what I can tell, docker-java does not decompose the auth field (into username/password) before creating the JSON authentication header as per https://docs.docker.com/engine/api/v1.37/#section/Authentication so credentials are not being sent properly:

2018-08-05 14:33:08,776 INFO  [pool-6-thread-3] ?.0.112]: effective auth config [AuthConfig[username=<null>,password=<null>,email=<null>,registryAddress=https://index.docker.io/v1/,auth=abcdef=,registrytoken=<null>]]
2018-08-05 14:33:09,917 ERROR [testcontainers-netty-1-15] c.g.d.c.a.ResultCallbackTemplate: Error during callback
com.github.dockerjava.api.exception.NotFoundException: {"message":"pull access denied for acme/microservice, repository does not exist or may require 'docker login'"}

	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:103)
	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33)

The problems with using docker-java.properties are:

Only supports a single registry URL, it seems these credentials are aimed at pushing to a single private registry (not really for pulling from multiple (private) registries)
If credentials are set here, testcontainers fails to pull ryuk:0.2.2 from quay.io - I have not confirmed but I wonder if it's trying to the use the credentials configured in docker-java.properties (for index.docker.io) for quay.io - a bug?

2018-08-05 14:43:34,343 ERROR [testcontainers-netty-1-5] c.g.d.c.a.ResultCallbackTemplate: Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://quay.io/v2/testcontainers/ryuk/manifests/0.2.2: unauthorized: Invalid Username or Password"}

	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:109)
	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33)
	at org.testcontainers.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)

The most confusing thing - why has nobody else reported this? Am I just doing something horribly wrong? :)

I think that #813 and #790 are related to this ticket, but thanks for the detailed write-up. I'll have a think about this.

Also from ffissore in #836:

I'm using testcontainers 1.8.3. I have a test db image on a private repo on AWS ECR.
When I run my tests with
eval $(aws ecr get-login --no-include-email); mvn clean test
testcontainers is unable to pull the image.

Error message is
com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://REPOID.dkr.ecr.eu-west-1.amazonaws.com/v2/IMAGENAME/manifests/IMAGETAG: no basic auth credentials"}

My .docker/config.json is

{
"auths": {
"REPOID.dkr.ecr.eu-west-1.amazonaws.com": {
"auth": "AUTH"
},
"https://index.docker.io/v1/": {
"auth": "AUTH"
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/18.06.0-ce (linux)"
}
}
Attached a snippet of the logs log.txt

@kiptix

And from @kiptix in #835:

It works with version 1.8.0, but with 1.8.1 i got the following exception:
Caused by: com.github.dockerjava.api.exception.DockerClientException: Could not pull image: unauthorized: authentication required

OS is Linux.
Docker version 17.05.0-ce, build 89658be
~/.docker/config.json (with replaced url and token):
{
"auths": {
"registry.url": {
"auth": "token"
}
}
}

@nefilim

@nefilim thanks for the draft PR - it's been a good basis for trying to figure this out while on a long plane journey :) I have some additions worked out.

This is still WIP, but tackles the two underlying issues that you and others have encountered:

basic auth credentials not being parsed out of the config.json file (for all private repos)
registry auth for index.docker.io being rejected. To resolve this I've overridden the effectiveAuthConfig method of DockerClientConfig for now, to try and come up with a solution that doesn't require changing docker-java (yet).

In the long term, this requires a fair amount of modification to docker-java to do it cleanly, but for now we can basically layer this on top and refactor later.

@nefilim

FWIW, I can confirm that I'm experiencing the same problem as @nefilim.

I think I've done the work necessary in #845! However, given that understanding/reproducing the registry setup is potentially the harder thing here, it'd be extremely helpful if people affected by this could give this a try before we call the PR done. Would that be OK?

There is a jitpack build available with the following details:

Repository: maven { url 'https://jitpack.io' }
Dependency: com.github.testcontainers.testcontainers-java:testcontainers:auth-fixes-SNAPSHOT (or use 6d3d4ea86c for a recent non-snapshot version)

Thanks!

Oh, I can now appreciate the problem. Nice solution. I've been trying it out on my workstation (macOS) as on a CI server with my work-around removed. Both are working correctly...
macOS:

DEBUG o.t.utility.RegistryAuthLocator - Looking up auth config for image: foo/stubservice:latest
DEBUG o.t.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /Users/myuser/.docker/config.json (exists) and commandPathPrefix:
DEBUG o.t.utility.RegistryAuthLocator - registryName [index.docker.io] for dockerImageName [foo/stubservice:latest]
DEBUG o.t.utility.RegistryAuthLocator - Executing docker credential helper: docker-credential-osxkeychain to locate auth config for: index.docker.io
DEBUG o.t.s.o.z.exec.ProcessExecutor - Executing [docker-credential-osxkeychain, get].
DEBUG o.t.s.o.z.exec.ProcessExecutor - Started java.lang.UNIXProcess@398dada8
DEBUG o.t.s.o.z.exec.WaitForProcess - java.lang.UNIXProcess@398dada8 stopped with exit code 0
DEBUG o.t.utility.RegistryAuthLocator - Credential helper provided auth config for: index.docker.io
DEBUG o.t.utility.RegistryAuthLocator - found creds store auth config [AuthConfig{username=mydockerhub, password=hidden non-blank value, auth=blank, email=null, registryAddress=index.docker.io, registryToken=blank}]

Centos:

DEBUG o.t.utility.RegistryAuthLocator - Looking up auth config for image: foo/stubservice:latest
DEBUG o.t.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /home/centos/jenkins/workspace/myservice_component-test-TXAW6D7VAHKETGC2PDRAEWPZFPITNCY62VAZZPLUCI2CNW5OCMOQ@tmp/3a0bfcca-b0da-4266-b6f0-0780fe87df9d/config.json (exists) and commandPathPrefix: 
DEBUG o.t.utility.RegistryAuthLocator - registryName [index.docker.io] for dockerImageName [foo/stubservice:latest]
DEBUG o.t.utility.RegistryAuthLocator - found existing auth config [AuthConfig{username=jenkins, password=hidden non-blank value, auth=hidden non-blank value, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]
DEBUG o.t.d.a.AuthDelegatingDockerClientConfig - effective auth config [AuthConfig{username=jenkins, password=hidden non-blank value, auth=hidden non-blank value, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]

Also, also thanks for avoiding the repository credentials in the logs. I was going to ask about that. This project is really looking great.

@reardonm

Thanks @reardonm, it's useful to get feedback that this is working for you.

The fix works for me, too

@rnorth

Thanks so much @rnorth - looking good here too.

Sorry for the delay, was travelling and had some technical issues. Don't know gradle at all (use SBT) - not very trivial/clear to try and get it to publish to a local ivy repo, took some time to find a workaround :)

Navigation Menu

Repository navigation

Authentication for Docker Hub private registry without credential helpers #820

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Authentication for Docker Hub private registry without credential helpers #820

Description

Activity

rnorth commented on Aug 5, 2018

rnorth commented on Aug 16, 2018

rnorth commented on Aug 16, 2018

rnorth commented on Aug 16, 2018

reardonm commented on Aug 20, 2018

rnorth commented on Aug 23, 2018

reardonm commented on Aug 24, 2018

rnorth commented on Aug 26, 2018

kiptix commented on Sep 3, 2018

nefilim commented on Sep 4, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions