Skip to content

Limit total CI run time to 24-ish hours. #331

Open

Description

@hategan

The CI tests do not currently have a time limit. Some of the tests that involve queing systems do have some explicit timeouts, but that's rather specific.

It is possible for certain tests to be buggy and trigger a deadlock. This can happen in a PR, so a faulty PR that may not even be merged could cause a CI run to never finish. On some machines with NFS mounts, this may lead to some file locking issues that could prevent subsequent CI runs from happening. This, in turn, means that one needs to manually log into test machines and kill the deadlocked process. This is less than ideal.

Since CI runs are technically supposed to run every 24 hours, there is somewhat of an implicit assumption that they should last less than that. Exceeding that can be an issue via the NFS problem mentioned above and also for @at, since a new run is scheduled only after the previous run finishes.

Anyway, we should put a cap of, say, 23 hours on a CI run.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions