Skip to content

ARM test timeouts #1343

Closed
Closed
ARM test timeouts#1343
Assignees
silverwind
Labels
armIssues and PRs related to the ARM architecture.testIssues and PRs related to Node.js core tests and test infrastructure.

Description

@silverwind
silverwind
opened on Apr 4, 2015

A few test currently fail on ARM because they have too high expectations for processing time for these rather slow platforms. I'll work in fixing this for v6 and v7 which I can test on both RPi 1 and 2.

@rvagg you mention in #283 (comment) a very-slow flag for tests. Could you point me how I'd add something like that to the test framework? Maybe you have an branch I could base my work off?

Activity

  1. silverwind
    added
    testIssues and PRs related to Node.js core tests and test infrastructure.
    on Apr 4, 2015

  2. silverwind commented on Apr 4, 2015

    @silverwind
    silverwind
    on Apr 4, 2015
    ContributorAuthor

    Failing tests are:

    ARMv6:

    parallel/test-child-process-fork-net2
parallel/test-fs-empty-readStream (may not be timeout related)
parallel/test-debug-signal-cluster
parallel/test-http-end-throw-socket-handling
parallel/test-repl-timeout-throw (may not be timeout related)
parallel/test-stringbytes-external
parallel/test-tls-fast-writing
parallel/test-tls-wrap-timeout
sequential/test-force-repl
sequential/test-net-GH-5504
sequential/test-pipe

    ARMv7

    parallel/test-child-process-fork-net2
parallel/test-tls-wrap-timeout
sequential/test-force-repl
  3. silverwind
    self-assigned this
    on Apr 5, 2015

  4. silverwind commented on Apr 5, 2015

    @silverwind
    silverwind
    on Apr 5, 2015
    ContributorAuthor

    Looks like my overclocked RPi2 already passes all tests as-is. It barely makes the repl test in like 900ms of the 1000ms available. Standard clocked RPi2 still needs like 1300ms though :P

    WIP branch: https://github.com/silverwind/io.js/tree/arm-test-timeout

  5. rvagg commented on Apr 6, 2015

    @rvagg
    rvagg
    on Apr 6, 2015
    Member

    @silverwind tbh I was just imagining something in common.js that did a lazy check of the arch when requested and if it returned armv6 then it would be identified as a "slow" platform and the tests that run too slow would internally adjust their timeout length. I haven't looked in detail at how the armv7 builds are going but we may need to have a semi-slow in there as well, or else adjust the timeouts for all platforms to make them work.

    Currently in the CI mix we have:

    • iojs-armv7-ubuntu140 - an Odroid XU3 which is a bit of a beast and doesn't seem to have trouble with the tests
    • iojs-armv7-wheezy - a couple of armv7 servers provided by Scaleway that run Debian Wheezy armhf (i.e. not a Raspbian variant or anything fancy). I haven't been watching test results for these and I don't know much about the hardware these are running on but I'm betting they aren't as powerful as the XU3 machine.
    • iojs-armv8-ubuntu1404 - armv8 / aarch64 provided by Linaro on their cluster, this is on an APM X-Gene Mustang board, I doubt we're at the stage of looking at timeouts on this
    • iojs-raspbian-wheezy-pi1 - an armv6 rotation of 2 x Pi v1 and 1 x Pi v1plus, this is where we have most of the timeout problems and a lot of it seems to be related to how long it takes to start a process
    • iojs-raspbian-wheezy-pi2 - one Pi v2 which is armv7, I haven't looked in detail at the state of timeouts and how this compares with the two other armv7 configurations.

    /cc @iojs/platform-arm @iojs/build

  6. Fishrock123 commented on Apr 6, 2015

    @Fishrock123
    Fishrock123
    on Apr 6, 2015
    Contributor

    iojs-armv7-ubuntu140 - an Odroid XU3 which is a bit of a beast and doesn't seem to have trouble with the tests

    Samsung Exynos5422 Cortex™-A15 2.0Ghz quad core and Cortex™-A7 quad core CPUs [1]

    Dang. :P

    iojs-armv7-wheezy - a couple of armv7 servers provided by Scaleway that run Debian Wheezy armhf (i.e. not a Raspbian variant or anything fancy). I haven't been watching test results for these and I don't know much about the hardware these are running on but I'm betting they aren't as powerful as the XU3 machine.

    I did some digging about this the other day, best I could find is https://www.scaleway.com/faq/server, which doesn't have too much perf / processor info. I do suspect these have higher perf than an rpi2 however.

  7. rvagg commented on Apr 6, 2015

    @rvagg
    rvagg
    on Apr 6, 2015
    Member

    yeah, the XU3 is a beautiful piece of hardware, even has a tiny little fan on it that spins up occasionally, I'd like a second one for the cluster so I might put out a call some time to get the funds for one.

    The Scaleway machines have 3 of these:

    processor   : 3
model name  : ARMv7 Processor rev 2 (v7l)
Features    : half thumb fastmult vfp edsp thumbee vfpv3 tls idiva idivt vfpd32 lpae
CPU implementer : 0x56
CPU architecture: 7
CPU variant : 0x2
CPU part    : 0x584
CPU revision    : 2

    Reporting:

    Hardware    : Marvell Armada 370/XP (Device Tree)
Revision    : 0000
Serial      : 0000000000000000

  8. Fishrock123 commented on Apr 6, 2015

    @Fishrock123
    Fishrock123
    on Apr 6, 2015
    Contributor

    Only 3 cores eh? the scaleway deployed ones are supposed to be 4-cores. Hmm.

  9. rvagg commented on Apr 6, 2015

    @rvagg
    rvagg
    on Apr 6, 2015
    Member

    sorry, I meant 4

  10. silverwind commented on Apr 6, 2015

    @silverwind
    silverwind
    on Apr 6, 2015
    ContributorAuthor

    Here's what I currently have:

    https://github.com/silverwind/io.js/blob/2774b24050464e031d2824f0cfbaa4b3325e90b3/test/common.js#L182-L200

    I think the best approach would probably be to benchmark process startup time once in common.js and calculate a "slowness factor" from that. Could probably do that for all platforms or just ARM.

  11. rvagg commented on Apr 6, 2015

    @rvagg
    rvagg
    on Apr 6, 2015
    Member

    Maybe make it lazy like opensslCli, inFreeBSDJail and localhostIPv4 so the user of the variable doesn't need to care about how it works and it doesn't need to be cached within tests.

    At one stage I had the impression that some timeouts needs a 10x increase on the Pi1 machines. Perhaps you could just wire up the test cases that need this and tune from there though.

  12. silverwind commented on Apr 6, 2015

    @silverwind
    silverwind
    on Apr 6, 2015
    ContributorAuthor

    Yeah, the armv6 value is arbitrary right now. I only ran armv7 so far, which seems to need something like 1.3-1.4 on a standard RPi2.

  13. jbergstroem commented on Apr 6, 2015

    @jbergstroem
    jbergstroem
    on Apr 6, 2015
    Member

    How about moving test we know are slow to a slow folder? That way we can run them as a separate job (tools/test.py --timeout $foo slow)

  14. silverwind commented on Apr 6, 2015

    @silverwind
    silverwind
    on Apr 6, 2015
    ContributorAuthor

    @jbergstroem Yeah we could bunch them together so we know the problematic ones. Would you suggest creating subfolders like slow/parallel and slow/sequential?

    Also, not sure about that arg. I think with a quick benchmark in common.js in place, we don't need to to fiddle with timeout values.

  15. jbergstroem commented on Apr 6, 2015

    @jbergstroem
    jbergstroem
    on Apr 6, 2015
    Member

    @silverwind my suggestion would be to just have one level depth. Half of the problem with timeouts is that the test runner might be busy doing other stuff. Running them sequentially will reduce time spent in each test (but probably longer overall).

  16. bnoordhuis commented on Apr 6, 2015

    @bnoordhuis
    bnoordhuis
    on Apr 6, 2015
    Member

    We have test/pummel for slow tests. I filed PR #1357 for fudging the per-test timeout factor.

  17. silverwind commented on Apr 7, 2015

    @silverwind
    silverwind
    on Apr 7, 2015
    ContributorAuthor

    Maybe make it lazy like opensslCli, inFreeBSDJail and localhostIPv4

    Those are getters which can't have arguments, so not really possible through a getter at least.

  18. silverwind
    added a commit that references this issue on Apr 9, 2015

    test: increase timeouts on ARM

    7049d7b

  19. silverwind commented on Apr 9, 2015

    @silverwind
    silverwind
    on Apr 9, 2015
    ContributorAuthor

    Fixed by 7049d7b

  20. silverwind
    closed this as completedon Apr 9, 2015

  21. rvagg commented on Apr 10, 2015

    @rvagg
    rvagg
    on Apr 10, 2015
    Member

    whoa, all green (blue), great work everyone!

  22. silverwind commented on Apr 10, 2015

    @silverwind
    silverwind
    on Apr 10, 2015
    ContributorAuthor

    Now we just need to squash the few intermittent failures that come up on CI randomly.

  23. brendanashworth
    added
    armIssues and PRs related to the ARM architecture.
    on Jul 14, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

armIssues and PRs related to the ARM architecture.testIssues and PRs related to Node.js core tests and test infrastructure.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions