How to login to private DockerHub registry in runtime?

Hi,

I did my best to try to avoid asking a FAQ, looking through the existing issues, so I hope I'm not asking the one here.

It looks like the general approach in TC is logging to the Docker Hub using the credentials from config.json. I mean #819 and questions like #813. What about the ability to pass credentials to the test runtime, via system or env property, so that TC would log in automatically, without relying on config.json?

Imagine a Jenkins agent which runs tests for different projects that keep images in different orgs on Docker Hub, with different creds. Having a single config.json for everyone would be hard. IMO, much better approach would be to get username+password from Jenkins credentials and use them right away in the TC to pull the images.

I see that Docker Java accepts a few runtime parameters, but that has no effect when running a TC test.

For instance, I'm passing system variables that Docker Java supports (plaintext):

registry.username=myuser
registry.password=mypwd
registry.email=myemail@mydomain.com

But the result in negative:

2018-11-08 16:30:46.901 [main]  INFO  o.t.d.EnvironmentAndSystemPropertyClientProviderStrategy : Found docker client settings from environment
2018-11-08 16:30:46.913 [main]  INFO  o.t.d.DockerClientProviderStrategy : Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='myuser'
    registryPassword='mypwd'
    registryEmail='myemail@mydomain.com'
    dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=myuser,registryPassword=mypwd,registryEmail=myemail@mydomain.com,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/home/myuser/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'
..............................
2018-11-08 16:30:50.582 [Thread-2]  ERROR 🐳 [docker-compose] : Pulling myimage       ... error
2018-11-08 16:30:51.315 [Thread-2]  ERROR 🐳 [docker-compose] : ERROR: for myimage  pull access denied for myorg/myimage, repository does not exist or may require 'docker login'

Is there anything I'm missing?

I can't really say what might be going wrong with your specific dockerhub example, but I can add that for a private repository (that isn't dockerhub), it was possible to achieve login by setting the registry.username, registry.password, registry.email and registry.url as environment variables. This was using version 1.10.1 of testcontainers-java. I did not try the same for a private dockerhub repo.

However, I might also add that environment variables (or a specifically named properties file containing them) are still far from ideal for unit tests. For example up until this point I have not found a way to use different credentials when separate repositories are used in different junit test classes in the same (maven) project. It seems the docker client config of docker java is only loaded once with the environment variables, but not reloaded in a different test where other values are set for the environment variables. So I am also interested to know what is the proposed way to switch credentials "at runtime" as you put it.

@kshepitko

Hi @kshepitko

Testcontainers generally supports the docker credential store/credential helper mechanisms, and I'd really recommend using these if you can.

However, you might be hitting a case which we don't currently support: docker-compose cannot access these credentials at present, unfortunately. You could try using the local compose binary mode instead of the default compose-in-container model. That would let docker-compose see the docker hub credentials, but comes at the expense of needing a docker-compose binary on build/dev machines (beware #986 if you're installing docker-compose at the moment).

As an alternative, you could pre-pull the images you require before running tests. Depending on the number of images you need to pull and rate of change, that may be an easier workaround in the short term.

I'm logged in to a private registry (for which I used docker login), but testcontainers is unable to get an image from it ("no basic auth credentials"). I'm not sure it supports the docker credential mechanisms that well (at least on Mac) :(

I could probably get away with configuring the AuthConfig of the client, but I can't do that before the image is resolved (it happens in the constructor, and I can't override the docker client factory because it's a singleton). It's really a huge issue for me. Do you have any recommendations?

same here getting "no basic auth credentials" during buildAppDockerImage
if I do docker pull "dependency image" then build is ok

Which version of Testcontainers are you both using?

Please could you post some logs? The particular logs that we should look at are for the RegistryAuthLocator class (I’m afraid I don’t have the package name to hand). INFO+ level logs should be enough, but if you could turn on DEBUG level logs for that class it would probably help.

_{Sent with GitHawk}

I think this might be caused by the already fixed bug in docker-java: docker-java/docker-java#1146

I suggest updating the version of it here and we'll be able to test whether it helped :) I'm pretty positive it will

@rnorth

@rnorth
org.testcontainers:testcontainers:1.10.+ -> 1.10.5
Now I think that the problem is actually related to com.liferay.app.docker gradle plugin as it fails during buildAppDockerImage task. Will have to investigate it a bit deeper there.

@kubukoz

@kubukoz thanks for that link. Updating to the latest docker-java seems worth a shot. Unfortunately there are breaking changes in that version; we have to resolve these in #1007.

@kshepitko

Hi @kshepitko

Testcontainers generally supports the docker credential store/credential helper mechanisms, and I'd really recommend using these if you can.

However, you might be hitting a case which we don't currently support: docker-compose cannot access these credentials at present, unfortunately. You could try using the local compose binary mode instead of the default compose-in-container model. That would let docker-compose see the docker hub credentials, but comes at the expense of needing a docker-compose binary on build/dev machines (beware #986 if you're installing docker-compose at the moment).

As an alternative, you could pre-pull the images you require before running tests. Depending on the number of images you need to pull and rate of change, that may be an easier workaround in the short term.

Hi @rnorth

It's funny, but I'm actually using a slave with a local docker compose binary for test running. It is totally not picking up the credentials passed down. Looks like TC/DockerJava doesn't perform docker login.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

This issue has been automatically closed due to inactivity. We apologise if this is still an active problem for you, and would ask you to re-open the issue if this is the case.

For anyone that struggles with this:
I've managed to get it working by adding a static-block before my Images are resolved by Testcontainers.

    static {
        // needs to be executed, before Docker images are resolved
        System.setProperty("registry.username", "<your-username>");
        System.setProperty("registry.password", "<your-password>");
    }

@JapuDCret

Thanks @JapuDCret !
So in general the docker registry (docker login) is resolved by using the system env properties

        System.setProperty("registry.username", "<your-username>");
        System.setProperty("registry.password", "<your-password>");

But what about the docker registry URL ❓
What are the options to configure it.
Thanks

@dfurmans

@dfurmans We set it directly in the image name:

    public static final DockerImageName IMAGE_NAME =
            DockerImageName.parse("my-repo.example.com:1234/com/example/my-image:0.1");

that has the advantage, that you only download the images from your repo, that are actually needed from there. Everything else will be downloaded normally (meaning the docker image registry configured in your docker runtime will be used).

This has one disadvantage, that the image name is no longer com/example/my-image, but the whole stuff at the top.

Also we moved away from setting the username&password via system properties and moved to every developer needing to perform a docker login to the repo, this will then be cached by the docker runtime:

docker login my-repo.example.com:1234-u username

You can check, if you're able to pull the image, with this command (takes less time than starting the testcontainers):

docker image pull my-repo.example.com:1234/com/example/my-image:0.1

@JapuDCret

Thank you @JapuDCret for a great solution!

@JapuDCret

Yes, @JapuDCret's second solution is the way to do this!

@rnorth

@rnorth could you clarify how this works in the context of CI?
We are trying to migrate from docker-compose to Test Containers. We have exactly the setup described here.
Everything works fine locally, but when we build it in Jenkins we are getting unauthorised from our private repo when we try to pull an image

Could you explain how we could leverage the config.json to work with TestContainers?

@ctron-te

@ctron-te Are you sure you have the setup described in the docs? You said you are migrating away from docker-compose, but the docs talk about the docker-compose setup.

Else, just do the docker login before running your tests, Testcontainers will pick up the auth config automatically then.

I mean we are using the config.json described in the doc and we are using it with docker-compose (not TestContainer docker-compose module, just docker-compose). I suspect if we used the docker-compose module of TestContainer it would just work but that's not what we are trying to achieve.
I will try to run docker login and see what happens

@ctron-te

@ctron-te Testcontainers and the docker-java library should be picking up your ~/.docker/config.json configuration automatically.

If you adjust the log level (via config) so that org.testcontainers.utility.RegistryAuthLocator is logging at DEBUG level, you should see a lot of information about how the configuration is being interpreted.

@rnorth

@rnorth @kiview apologies, there was a bug in path for DOCKER_CONFIG which we figured out by enabling DEBUG as you suggested 😅
Thanks again for the help and the top product 🚀

Navigation Menu

Repository navigation

How to login to private DockerHub registry in runtime? #968

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

How to login to private DockerHub registry in runtime? #968

Description

Activity

kroonprins commented on Nov 21, 2018

rnorth commented on Nov 22, 2018

kubukoz commented on Jan 3, 2019

crawlik commented on Jan 21, 2019

rnorth commented on Jan 21, 2019

kubukoz commented on Jan 21, 2019

alex-vinnik-sp commented on Jan 21, 2019

rnorth commented on Jan 21, 2019

kshepitko commented on Mar 6, 2019

stale commented on Jun 4, 2019

stale commented on Jun 18, 2019

JapuDCret commented on Jul 2, 2021

dfurmans commented on Aug 17, 2021

JapuDCret commented on Aug 18, 2021

dfurmans commented on Nov 26, 2021

rnorth commented on Nov 26, 2021

ctron-te commented on Jan 25, 2022

kiview commented on Jan 25, 2022

ctron-te commented on Jan 25, 2022

rnorth commented on Jan 25, 2022

ctron-te commented on Jan 25, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions