Skip to content

Hanging test timeout is ignored by Symbolic execution #1095

Open

Description

@alisevych

Description

When a hanging test is generated by Symbolic execution, it doesn't have Timeout annotation.

To Reproduce

  1. Install UTBot plugin on IntelliJ Idea
  2. Check Symbolic execution is enabled and Hanging test timeout is 1000 (default)
  3. Create a project and add the following class:
public class A {

    public int hangForSeconds(int seconds) {
        for (int i = 1; i < seconds; i++) {
            try {
                Thread.sleep(1001);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }
        }
        return seconds;
    }
}
  1. Open the generated test

Expected behavior

Tests whose generation lasts for more than 1000 millisec should fail with TimeoutException.

Actual behavior

Test generated by Symbolic execution, lasts for , it doesn't have Timeout annotation.
Tests generated by Fuzzing has correct annotation.

Visual proofs (screenshots, logs, images)

public class ATest {
    ///region Test suites for executable org.example.A.hangForSeconds

    ///region SYMBOLIC EXECUTION: SUCCESSFUL EXECUTIONS for method hangForSeconds(int)

    /**
     * <pre>
     * Test iterates the loop {@code for(int i = 1; i < seconds; i++) } once.
     * Test afterwards returns from: {@code return seconds; }
     * </pre>
     */
    @Test
    @DisplayName("hangForSeconds: ThreadSleep -> return seconds")
    public void testHangForSeconds_ThreadSleep() {
        A a = new A();

        int actual = a.hangForSeconds(2);

        assertEquals(2, actual);
    }
    ///endregion

    ///region FUZZER: SUCCESSFUL EXECUTIONS for method hangForSeconds(int)

    @Test
    @DisplayName("hangForSeconds: seconds > 0 -> return 1")
    public void testHangForSecondsReturnsOne() {
        A a = new A();

        int actual = a.hangForSeconds(1);

        assertEquals(1, actual);
    }
    ///endregion

    ///region FUZZER: EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS for method hangForSeconds(int)

    @Test
    @DisplayName("hangForSeconds: seconds = 134217729 (mutated from 1)")
    @Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
    public void testHangForSeconds() {
        A a = new A();
        
        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
        assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> a.hangForSeconds(134217729));
    }
    ///endregion

    ///endregion
}

Environment

Windows 10 Pro
IntelliJ IDEA 2022.2.2 / 2022.1.4

Additional context

Also Fuzzing test generation mode throws exception in Concrete execution logs:

Exception in thread "executor" java.lang.IllegalMonitorStateException
	at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
	at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:406)
	at org.utbot.common.ThreadBasedExecutor$invokeWithTimeout$1.invoke(ThreadUtil.kt:46)
	at org.utbot.common.ThreadBasedExecutor$invokeWithTimeout$1.invoke(ThreadUtil.kt:43)
	at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)

Activity

  1. self-assigned this
    on Oct 5, 2022
  2. moved this to Todo in UTBot Javaon Oct 5, 2022
  3. Damtev commented on Oct 7, 2022

    @Damtev
    Member

    It looks like a corner case for the child process timeout - as such as the generated test takes 1000 + epsilon milliseconds to execute, a child process does not detect such overflow of hanging tests timeout. Tests are generated as expected in case of decreasing the hanging tests timeout setting or increasing sleeping time. @sergeypospelov what do you think?

  4. alisevych commented on Oct 7, 2022

    @alisevych
    MemberAuthor

    @Damtev @sergeypospelov Sorry, corrected description, I've used 1001 milliseconds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

ctg-bugIssue is a bug

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions