Fuzzing is generating a false test for Random

@alisevych

Description

Fuzzing is generating a test for Random without mocks - when Mock everything outside class is checked.
The test asserts the value returned by Random and fails.

To Reproduce

Run a project in IntelliJ Idea 2022.1.3 - 2022.2.2
Install RC plugin build
Set Fuzzing to 100% (in File -> Settings -> Tools -> UnitTestBot)
Generate tests with UnitTestBot for utbot-sample/src/main/java/org/utbot/examples/mock/MockRandomExamples.java
for useSecureRandom() - check Mock everything outside class.

Expected behavior

There should be only valid tests generated.

Possible solutions

Fuzzer can use mock Random(?)
Generated tests should be double-checks that they return same results(?)
other solution(?)

Actual behavior

Fuzzing is generating a test for Random without mocks. It asserts the value returned by Random and fails.

Visual proofs

The following test is generated by Fuzzer:

    ///region FUZZER: SUCCESSFUL EXECUTIONS for method useSecureRandom()

    @Test
    @DisplayName("useSecureRandom: arg_0 = MockRandomExamples() -> return 1")
    public void testUseSecureRandomReturnsOne() {
        MockRandomExamples mockRandomExamples = new MockRandomExamples();

        int actual = mockRandomExamples.useSecureRandom();

        assertEquals(1, actual);
    }
    ///endregion

Environment

Windows 10 Pro
IntelliJ IDEA 2022.2.2

Additional context

Originally posted by @alisevych in #1102 (comment)

@korifey

@korifey @denis-fokin @Markoutte
May be the team can consider a common solution for handling changing data?
That can be Random, Date/DateTime, data from files/other resources, etc.
How to determine them? and how to mock?

@SBOne-Kenobi

We should definitely reproduce every test, traversed by concrete executor. My proposal is to enhance concrete executor. It must automatically fixate behavior of certain methods and return back information to engine information about it.
E.g. if concolic executor find some new Random() it should return value to engine that will generate static mocking for this case.
@SBOne-Kenobi could you try to elaborate this idea.

@korifey

concolic executor find some new Random()

Also need to find Random usages.
And other Random implementations can be used.

@korifey Would it be too long to rerun every test generated by Fuzzing? - to check if received data is changing from run to run

Also, there is the same problem with Random when generating tests for vertexSum method from recursion package.
It is worth to mention that it reproduces not every time.
Steps to reproduce:

Run runIde task
Copy recursion package to the test project inside runIde
Generate a test for vertexSum method with the following settings: JUnit5, not parameterized, any mocking strategy. You have to have two generated tests, and regenerate if there is only one.

One (the test below) of the two generated tests will fail when running because of Random usage inside vertexSum method.

///region FUZZER: SUCCESSFUL EXECUTIONS for method vertexSum(int)

/**
 * @utbot.classUnderTest {@link Recursion}
 * @utbot.methodUnderTest {@link Recursion#vertexSum(int)}
 */
@Test
@DisplayName("vertexSum: n = 134217732 (mutated from 4) -> return 293591018")
public void testVertexSum() {
    Recursion recursion = new Recursion();

    int actual = recursion.vertexSum(134217732);

    assertEquals(293591018, actual);
}
///endregion

Navigation Menu

Repository navigation

Fuzzing is generating a false test for Random #1135

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Fuzzing is generating a false test for Random #1135

Description

Activity

alisevych commented on Oct 11, 2022

korifey commented on Oct 11, 2022

alisevych commented on Oct 17, 2022

sofurihafe commented on Oct 21, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions