Containers are not stopped on OSX #253

Activity

1. rnorth commented on Dec 9, 2016

   

   rnorth

   on Dec 9, 2016 via email

   Member

   More actions

   Hi @pavelpp Thanks - that sounds odd, so sorry you're getting this. Are we talking fully forked JVMs, or could this be parallel threads in the same JVM? That might be relevant in tracing this. This probably will be a tricky one to debug, but I'll try and help. Is there any chance you could provide DEBUG level logs, and ideally a sample repo that shows the bug? Thanks Richard

   …

   On Fri, 9 Dec 2016 at 20:24, pavelpp ***@***.***> wrote: System: OSX 10.10.5 Tried with latest docker-machine and Docker for Mac TC version 1.1.7 HW: Model Name: MacBook Pro Model Identifier: MacBookPro11,5 Processor Name: Intel Core i7 Processor Speed: 2,5 GHz Number of Processors: 1 Total Number of Cores: 4 L2 Cache (per Core): 256 KB L3 Cache: 6 MB Memory: 16 GB Problem: When running parallel selenium tests with gradle (4 forks), sometimes tests leave containers running even though in shutdown hooks I do stop them. Mostly the problem happens with BrowserWebDriverContainer, but sometimes other GenericContainer's are showing the same behaviour. The problem can't be reproduced when single tests are run, so it's kind of difficult to debug. Only when I run all tests in parallel "ghost" browser containers start piling up after 5 minutes or so and I have to kill them manually from command line (if I don't them my tests will start failing and new containers fail to start). I know this is a very vague way to report a problem, but maybe someone can give me a hint on where to start with debugging. At this point I'm stuck and out of ideas. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#253>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIET7an5e_psCoyEm0xV1JgBSGwgQmvks5rGbkIgaJpZM4LJUtT> .
2. pavelpp commented on Dec 9, 2016

   

   pavelpp

   on Dec 9, 2016

   Last edited by pavelpp

   ContributorAuthor

   More actions

   These are forked JVM's. Tests are started like this:

   test {
       includes = [
               "zt/selenium/junit/*Test.class",
               "Scenario*"
       ]
   
       maxParallelForks 4
   
       testLogging {
           displayGranularity 1
           events "PASSED", "FAILED", "SKIPPED"
           showStandardStreams true
           exceptionFormat = 'full'
       }
   
       afterTest {
           System.out.println()
       }
   

   Can't really share the repo. Logs - will try to get.
3. pavelpp commented on Dec 10, 2016

   

   pavelpp

   on Dec 10, 2016

   ContributorAuthor

   More actions

   I noticed timezone is incorrect in my browser container. Is there a way to set correct timezone via testcontainers?
4. bsideup commented on Dec 10, 2016

   

   bsideup

   on Dec 10, 2016

   Member

   More actions

   @rnorth this is my colleague, I'll try to debug his project since I have an access to the project:)

   Maybe you have some ideas what to check?

   FYI: the project is based on Cucumber, all container management is done manually (not with JUnit rules), I suspect it to be an issue.
5. rnorth commented on Dec 10, 2016

   

   rnorth

   on Dec 10, 2016 via email

   Member

   More actions

   Ah cool, thanks! As it seems to only be a problem in parallel execution, my initial suspicion is a problem with the Reaper. Perhaps there's some kind of concurrency issue with multiple processes calling the docker daemon in parallel - e.g. A mutex on listing and deleting containers. If that's the case the answer might, unfortunately, involve putting retry loops into the reaper. Re the timezones problem, BrowserWebDriverContainer should be setting the timezones correctly from the user.timezone system property, falling back to UTC if that's not set. Please could you provide more info on what you're seeing there @pavelpp? Thanks Richard

   …

   On Sat, 10 Dec 2016 at 08:59 Sergei Egorov ***@***.***> wrote: @rnorth <https://github.com/rnorth> this is my colleague, I'll try to debug his project since I have an access to the project:) Maybe you have some ideas what to check? FYI: the project is based on Cucumber, all container management is done manually (not with JUnit rules), I suspect it to be an issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#253 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIET2YO1xKrvXDUijOdvOieyNbRzNSWks5rGmn9gaJpZM4LJUtT> .
6. pavelpp commented on Dec 10, 2016

   

   pavelpp

   on Dec 10, 2016

   Last edited by pavelpp

   ContributorAuthor

   More actions

   @after("@web")
   cleanup method is invoked after every scenario with tag @web. I suspect that the problem might be in how container is started. Is it possible that start() method somehow starts two copies of the container, thus stop() method will only stop the last copy? I'm not getting any errors when stopping containers.

   Yesterday experimented with this instead of calling stop():

       public void forceKill(String containerId) {
           try {
               DockerClientFactory.instance().client().removeContainerCmd(containerId).withForce(Boolean.TRUE).exec();
           } catch (Exception e) {
               e.printStackTrace();
           }
       }
   

   same result. No change, so my only theory is that sometimes two browser containers are started with one start() method, leaving one of them orphaned, until all tests finish running
7. pavelpp commented on Dec 11, 2016

   

   pavelpp

   on Dec 11, 2016

   Last edited by pavelpp

   ContributorAuthor

   More actions

   Regarding timezones. I did not set any timezone myself, maybe that was the issue, but it's not documented anywhere. I've added browserContainer.withEnv("TZ", "Europe/Helsinki") and now time seems to be correct.
8. pavelpp commented on Dec 12, 2016

   

   pavelpp

   on Dec 12, 2016

   ContributorAuthor

   More actions

   Case closed. Problem with orphaned containers boiled down to me not calling driver.quit() before stopping container. Still don't quite understand why this is necessary though.
9. pavelpp

   closed this as completedon Dec 12, 2016
10. rnorth commented on Dec 12, 2016

    

    rnorth

    on Dec 12, 2016 via email

    Member

    More actions

    Hi @pavelpp Thanks - glad you found a way to work around. If you don't mind I'll reopen the ticket though. What you're seeing isn't the developer experience I want! It's worth us checking where containers could slip through the cracks, as any leftover containers are a pure waste of user resources. Richard

    …

    On Mon, 12 Dec 2016 at 08:16 pavelpp ***@***.***> wrote: Case closed. Problem with orphaned containers boiled down to me not calling driver.quit() before stopping container. Still don't quite understand why this is necessary though. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#253 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIET4krX-ieImKhWugn77vJd32DNINnks5rHQLsgaJpZM4LJUtT> .
11. pavelpp commented on Dec 12, 2016

    

    pavelpp

    on Dec 12, 2016

    Last edited by pavelpp

    ContributorAuthor

    More actions

    After further investigation I still face the issue. Tracked the problem down to simultaneous initialization of browser in several parallel tests. Sometimes (randomly) one of the started instances is not going the usual route (I rename containers at start, adding "browser" prefix). And for some reason this very container it is not getting stopped after test is over. As I have lots of tests and each of them starts new browser container, during an hour or so such orphaned containers keep piling up. I've run the same sub-set of tests multiple times and sometimes they create containers as expected - sometimes not. It's random, but I suspect it has something to do with timing of browser container creation.

    $ docker ps
    CONTAINER ID        IMAGE                                           COMMAND                  CREATED             STATUS              PORTS                                              NAMES
    e1e309c3ee18        selenium/standalone-chrome-debug:3.0.1-barium   "/opt/bin/entry_point"   14 seconds ago      Up 12 seconds           0.0.0.0:33077->4444/tcp, 0.0.0.0:33076->5900/tcp   browser_awesome_babbage
    69b73e2ebf15        selenium/standalone-chrome-debug:3.0.1-barium   "/opt/bin/entry_point"   14 seconds ago      Up 13 seconds           0.0.0.0:33075->4444/tcp, 0.0.0.0:33074->5900/tcp   admiring_poincare
    

    This is how I create container:

    import org.jetbrains.annotations.NotNull;
    import org.openqa.selenium.remote.DesiredCapabilities;
    import org.testcontainers.containers.BrowserWebDriverContainer;
    import org.testcontainers.images.RemoteDockerImage;
    
    import java.time.Duration;
    
    public class BrowserContainerProvider {
    
        @NotNull
        public static BrowserWebDriverContainer getBrowserWebDriverContainer() {
            return new BrowserWebDriverContainer() {
                @Override
                protected void configure() {
                    super.configure();
                    withStartupAttempts(5);
                    withStartupTimeout(Duration.ofMinutes(5));
                    withDesiredCapabilities(DesiredCapabilities.chrome());
                }
    
                @Override
                public String getDockerImageName() {
                    setImage(new RemoteDockerImage("selenium/standalone-chrome-debug:3.0.1-barium"));
                    return super.getDockerImageName();
                }
            };
        }
    }
    

    Am I doing it wrong?