Skip to content

Clarify scope of EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS test suite #1133

Closed
#1250
Closed
Clarify scope of EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS test suite#1133
#1250
Assignees
korifeyMarkoutteamandelpie
Labels
comp-summariesSomething related to the method names, code comments and display names generationcomp-uiImprovements of plugin UI appearence and functionalityctg-bugIssue is a bug
Milestone
2022.12 Release

Description

@alisevych
alisevych
opened on Oct 11, 2022

Description

Clarify scope of generated suite with EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS.

Expected behavior

Is it correct that only explicitly thrown exceptions without try-catch should be located in this suite?

Environment

Fuzzing is on.

Context

Example:

Test with String.repeat(-1) throws [java.lang.IllegalArgumentException: count is negative: -1]

Expected result

The test should be put into FUZZING: IMPLICITLY_THROWN_UNCHECKED_EXCEPTIONS region.

Actual result

The test is currently put into FUZZING: EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS region.

Originally posted by @alisevych in #1102 (comment)

Activity

  1. korifey
    added this to UTBot Javaon Oct 11, 2022
  2. korifey
    moved this to Todo in UTBot Javaon Oct 11, 2022
  3. alisevych
    assigned
    Markoutte
    ,
    amandelpie
    and
    korifey
    on Oct 11, 2022
  4. alisevych
    changed the title [-]Clarify scope of EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS region[/-] [+]Clarify scope of EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS test suite[/+] on Oct 11, 2022
  5. alisevych
    added this to the 2022.11 Release milestone on Oct 11, 2022

  6. amandelpie commented on Oct 12, 2022

    @amandelpie
    amandelpie
    on Oct 12, 2022
    Collaborator

    I am happy to discuss about clusters for fuzzing tests bazed on the utExecution.result.

    At this moment all unsuccessful fuzzing execution are grouped to the one cluster with the name "EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS", probably the common name is incorrect

    Also, I could create a separate clusters for all types of utExecution.result

    • UtConcreteExecutionFailure
    • UtExplicitlyThrownException
    • UtImplicitlyThrownException
    • UtOverflowFailure
    • UtSandboxFailure
    • UtTimeoutException
      But @Markoutte looks like not all results are possible for Fuzzing execution, am I right?

  7. Markoutte commented on Oct 12, 2022

    @Markoutte
    Markoutte
    on Oct 12, 2022
    Collaborator

    If it collects exceptions of any kind, I'd prefer name like "EXCEPTIONS_THROWN" without any details about checked/unchecked.

  8. korifey commented on Oct 12, 2022

    @korifey
    korifey
    on Oct 12, 2022
    Contributor

    If it collects exceptions of any kind, I'd prefer name like "EXCEPTIONS_THROWN" without any details about checked/unchecked.

    We should at least differentiate Error suite from regression suite

  9. alisevych
    added
    ctg-bugIssue is a bug
    comp-summariesSomething related to the method names, code comments and display names generation
    on Oct 17, 2022

  10. alisevych commented on Oct 17, 2022

    @alisevych
    alisevych
    on Oct 17, 2022
    MemberAuthor

    As a suggestion:
    EXPECTED_EXCEPTIONS
    UNEXPECTED_EXCEPTIONS

  11. amandelpie commented on Oct 24, 2022

    @amandelpie
    amandelpie
    on Oct 24, 2022
    Collaborator

    @alisevych what do you mean under EXPECTED_EXCEPTIONS and UNEXPECTED_EXCEPTIONS from the following list?
    Is it the same as the proposed by @korifey the splitting of exceptions and errors (failures)?

    Am I right?

    1. UtConcreteExecutionFailure (UNEXPECTED_EXCEPTIONS)
    2. UtExplicitlyThrownException (EXPECTED_EXCEPTIONS)
    3. UtImplicitlyThrownException (EXPECTED_EXCEPTIONS)
    4. UtOverflowFailure (UNEXPECTED_EXCEPTIONS)
    5. UtSandboxFailure (UNEXPECTED_EXCEPTIONS)
    6. UtTimeoutException (EXPECTED_EXCEPTIONS)

  12. amandelpie commented on Oct 24, 2022

    @amandelpie
    amandelpie
    on Oct 24, 2022
    Collaborator

    @korifey is it correct to say that all result types could be grouped in Errors and Exceptions?

    Exceptions:
    UtExplicitlyThrownException
    UtImplicitlyThrownException
    UtTimeoutException

    Failures:
    UtOverflowFailure
    UtSandboxFailure
    UtConcreteExecutionFailure

  13. amandelpie commented on Oct 24, 2022

    @amandelpie
    amandelpie
    on Oct 24, 2022
    Collaborator

    @korifey @Markoutte @alisevych

    Also as a varinat we could unify it with the clusters produced by the Symbolic Engine.
    It has the following logic based on type of exceptions (checked/unchecked)

        is UtExecutionSuccess -> ExecutionGroup.SUCCESSFUL_EXECUTIONS
    is UtImplicitlyThrownException -> if (this.exception.isCheckedException) ExecutionGroup.CHECKED_EXCEPTIONS else ExecutionGroup.ERROR_SUITE
    is UtExplicitlyThrownException -> if (this.exception.isCheckedException) ExecutionGroup.CHECKED_EXCEPTIONS else ExecutionGroup.EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS
    is UtOverflowFailure -> ExecutionGroup.OVERFLOWS
    is UtTimeoutException -> ExecutionGroup.TIMEOUTS
    is UtConcreteExecutionFailure -> ExecutionGroup.CRASH_SUITE
    is UtSandboxFailure -> ExecutionGroup.SECURITY

  14. alisevych commented on Oct 24, 2022

    @alisevych
    alisevych
    on Oct 24, 2022
    MemberAuthor

    @amandelpie @Markoutte @korifey
    Please correct me if needed:

    As for Overflow exceptions - they should be treated as errors or ignored - based on the setting:
    image
    Also , what is meant by "ignored": just skipped?

    From my point of view implicitly thrown (by other classes/packages) and explicitly thrown (by user code) - should be in different suites.
    Implicitly - possible errors. Should be added to error suite (?)
    Explicitly - expected ones. Should be treated as regression suite - add to successful executions (?)

    Sandbox failures should be shown separately - with distinct explanation how to add permissions and regenerate the tests if needed.

    Timeout exceptions should be added to error suite.

    Concrete execution failures - are UnitTestBot issues. Should be shown in error suite or separately.

  15. alisevych
    modified the milestones: 2022.11 Release, 2022.12 Release on Nov 8, 2022
  16. amandelpie
    closed this as completedin #1250on Nov 8, 2022
  17. Repository owner moved this from Todo to Done in UTBot Javaon Nov 8, 2022
  18. alisevych
    added
    comp-uiImprovements of plugin UI appearence and functionality
    on Nov 30, 2022

  19. alisevych commented on Feb 27, 2023

    @alisevych
    alisevych
    on Feb 27, 2023
    MemberAuthor

    @rudolf101 @zishkaz
    Currently generated test cases for Java are divided into the following regions:

    image

    <Source name> : <Suite name> for method <Method signature>

    You can find some details in PR - #1250.

    Also there is Errors report region to collect internal errors from concrete execution (only count, full details should be in logs).

    Let's consider and discuss an appropriate solution for JavaScript test cases division into suites and their naming.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

comp-summariesSomething related to the method names, code comments and display names generationcomp-uiImprovements of plugin UI appearence and functionalityctg-bugIssue is a bug

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions