Skip to content

[PSST] New test -- check local clock for skew #93

Description

@bbockelm

Worker nodes with extreme skew can cause GSI failures that are difficult to debug. We should add this as a new PSST test.

Brian notes there is a timestamp in the TLS v1.2 handshake and this can be used as a lightweight mechanism to determine clock skews. Need to lookup an example link.

Activity

  1. roksys commented on Jun 29, 2017

    @roksys
    Contributor

    @bbockelm @stlammel
    What about using ntpdate -q <hostname_of_ntp_server>?
    We could use CERN NTP servers.

  2. bbockelm commented on Jul 3, 2017

    @bbockelm
    Author

    I worry that ntp would be blocked by the site, given its propensity to be used in attacks. Worth testing out.

    Alternate ideas:

    • Use something like tlsdate to extract the server date from a remote TLS connection.
    • HTCondor can report the ServerTime as part of the query response. We could perhaps ask upstream if this is sufficiently lightweight to test from the pilots.
  3. roksys commented on Jul 4, 2017

    @roksys
    Contributor

    I added check for clock skew.

    Local clock is compared with server clock (in this case https://google.com), which is taken from HTTP header. It has precision of ~ 1 sec. I guess it should be enough as only extreme skew can cause some problems.

    Currently threshold is set to 5 sec.

  4. roksys commented on Jul 24, 2017

    @roksys
    Contributor

    What could be the maximum acceptable clock skew?
    I was looking at the x509 spec, but did not find anything relevant.

  5. bbockelm commented on Jul 28, 2017

    @bbockelm
    Author

    A lot of implementations allow for a small number of minutes of clock skew - it's pretty inconsistent.

    I would maybe set to critical at 5 minutes and warn at 10s?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions