Skip to content

test(cli): add coverage for adk conformance test command options #6730

Closed

Description

@YASHcode-IIITV

Description

The adk conformance test CLI command supports several options and input combinations, but these CLI behaviors should have explicit automated coverage to prevent regressions.

The command currently supports:

  • Running conformance tests with no path, defaulting to the tests directory.
  • Running tests against one or more explicitly provided directories.
  • Selecting replay or live mode.
  • Generating a Markdown report with --generate_report.
  • Specifying a custom report directory with --report_dir.
  • Selecting a streaming mode with --streaming-mode.

This issue proposes adding focused unit tests for the CLI argument handling and dispatch behavior without requiring real LLM execution.

Proposed Changes

Add pytest coverage for the adk conformance test CLI command covering:

  1. No path provided → defaults to the tests directory.
  2. A single test directory is correctly passed to the conformance runner.
  3. Multiple test directories are accepted.
  4. --mode replay is correctly propagated.
  5. --mode live is correctly propagated.
  6. --generate_report is correctly propagated.
  7. --report_dir <path> is correctly propagated.
  8. --streaming-mode values are parsed and propagated correctly.
  9. Invalid --mode values are rejected by Click.
  10. Invalid --streaming-mode values are rejected by Click.

Where appropriate, the conformance runner should be mocked so these tests remain fast and deterministic.

Expected Outcome

The CLI's argument parsing and dispatch behavior should have regression coverage, making future changes to the conformance command less likely to introduce unnoticed CLI regressions.

Testing

Run the relevant CLI unit tests with pytest and ensure the existing ADK test suite continues to pass.

Scope

This should be limited to automated test coverage for the existing CLI behavior. No changes to the underlying conformance execution logic are required.

Activity

  1. chelsealong commented on Aug 15, 2026

    @chelsealong
    Contributor

    Picking this one up now — opening a PR shortly. Flagging it here so nobody duplicates the work; if someone is already on it, say so and I will drop mine.

  2. YASHcode-IIITV commented on Aug 15, 2026

    @YASHcode-IIITV
    ContributorAuthor

    hi maintainer i do like to work on this issue, as i have looked through the issue and planed to , reproduce the failure when one of the provided toolset fails during tool discovery , and also add a regression test covering the partial failure case.
    if this approach aligns with the expected behavior , can i plz be a part of it and also can you assign me this issue .
    and would be making a pr soon on this as well

  3. YASHcode-IIITV commented on Aug 15, 2026

    @YASHcode-IIITV
    ContributorAuthor

    @chelsealong i would like to work on this issue

  4. sanketpatil06 commented on Aug 18, 2026

    @sanketpatil06

    Hi @YASHcode-IIITV, thanks for raising a feature request. If you already have a solution in mind, please feel free to open a PR. our team would be happy to review it.

  5. YASHcode-IIITV commented on Aug 18, 2026

    @YASHcode-IIITV
    ContributorAuthor

    @sanketpatil06 thx , I will be posting the pr shortly .

  6. YASHcode-IIITV commented on Aug 19, 2026

    @YASHcode-IIITV
    ContributorAuthor

    @sanketpatil06 i have opened the pr and would be grateful to get reviewed as well

  7. self-assigned this
    on Aug 28, 2026
  8. GWeale commented on Aug 28, 2026

    @GWeale
    Collaborator

    landed in #6874 as 7d9f2d0, all ten cases are in tests/unittests/cli/utils/test_cli_tools_click.py now and pytest -k conformance is 11 passed on main. reopen if a specific case is still uncovered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

tools[Component] This issue is related to tools

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions