Skip to content

Self-testing framework: observe UX patterns and identify self-improvement opportunities #23

New issue
New issue
Closed
Closed
Self-testing framework: observe UX patterns and identify self-improvement opportunities#23
Labels
go:noNot pursuingsquadSquad triage inbox — Lead will assign to a membertype:spikeResearch/investigation — produces a plan, not code

Description

@bradygaster
bradygaster
opened on Feb 12, 2026

Problem: Squad can't currently observe its own behavior patterns — rendering issues, UX quirks, output quality, timing — in a structured way. We rely on anecdotal user reports to identify problems.

Vision: A self-testing or self-observation framework where Squad can:

  1. Capture behavioral snapshots — what does the output actually look like to users? Are there rendering issues, malformed tables, broken markdown, noisy tool output leaking through?
  2. Pattern detection — across sessions, identify recurring UX problems (e.g., 'agents go silent for too long', 'tables render wrong in terminal X', 'output hygiene violations')
  3. Self-improvement proposals — when patterns are detected, auto-file issues or suggest prompt/template changes
  4. Regression testing — after changes to squad.agent.md or spawn templates, verify that known-good behaviors still work

Research questions:

  • How would Squad capture its own output for analysis? (session logs? dedicated test harness?)
  • Can we run 'smoke test' sessions that exercise common workflows and validate output format?
  • What metrics matter? (response time, output length, tool call count, rendering correctness, personality consistency)
  • Could a dedicated agent role ('QA Observer') watch sessions and flag issues?
  • How do we test across different terminal renderers / Copilot hosts?

Concrete first step: Design a test session format where Squad runs itself through scripted scenarios and a reviewer agent grades the output quality.

Activity

  1. bradygaster
    added
    enhancement
    on Feb 12, 2026

  2. bradygaster commented on Feb 12, 2026

    @bradygaster
    bradygaster
    on Feb 12, 2026
    CollaboratorAuthor

    Sprint Plan Scope (from Brady)

    Goal: Design and implement self-improvement tests that run in GitHub Actions — NOT locally. The CLI environment stays free for the user to keep working.

    Key Constraints

    1. Tests run in Actions or via @copilot coding agent — offload test execution to CI, not the local CLI session.
    2. Improvement opportunities file as issues in THIS repo only — bradygaster/squad. Never file issues in other repos. (We tried that once. Never again.)
    3. Actionable output — when a test detects a pattern or regression, it files an issue with enough context to act on.

    Deliverables

    • Sprint plan for implementing the self-testing framework
    • GitHub Actions workflow(s) that exercise common Squad scenarios (smoke tests)
    • A reviewer/grader mechanism that evaluates output quality (rendering, output hygiene, personality, timing)
    • Auto-file issues in bradygaster/squad when regressions or improvement opportunities are detected
    • Regression test suite for squad.agent.md and spawn template changes

    What "Good" Looks Like

    • A PR changes squad.agent.md → Actions runs smoke tests → if output quality degrades → issue filed automatically in this repo
    • Periodic scheduled runs catch drift or new patterns → issues filed here
    • Zero local CLI overhead — all observation and testing happens in CI

    cc @copilot

  3. bradygaster
    added
    squadSquad triage inbox — Lead will assign to a member
    go:noNot pursuing
    type:spikeResearch/investigation — produces a plan, not code
    and removed
    enhancement
    on Feb 12, 2026

  4. bradygaster commented on Feb 13, 2026

    @bradygaster
    bradygaster
    on Feb 13, 2026
    CollaboratorAuthor

    Closing as go:no — this is a cool idea but it's out of scope for v0.4.0. The current sprint priority is VS Code client parity (#32-#34). Self-testing framework can be revisited post-v0.4.0 when the core multi-surface story is solid.

    Key concern: this is significant infrastructure work (CI-based smoke tests, auto-filing issues, regression detection) that doesn't contribute to the immediate goal of shipping Squad on VS Code.

    Reopening candidate for v0.5.0 or later.

  5. bradygaster
    closed this as completedon Feb 13, 2026
  6. bradygaster
    added a commit that references this issue on Feb 13, 2026

    docs: add feature showcase prompts to sample-prompts.md

    a684c3c
  7. tamirdresher
    added 5 commits that reference this issue on Mar 5, 2026

    blog: add workstreams horizontal scaling post (github#23)

    f9723a7

    docs: add blog post github#23 — Squad Goes Enterprise (ADO support)

    9584557

    fix: add ADO platform awareness to governance file (squad.agent.md)

    b34c4d7

    docs: add blog post github#23 — Squad Goes Enterprise (ADO support)

    7d0b611

    fix: add ADO platform awareness to governance file (squad.agent.md)

    3f0721a
  8. bradygaster
    added 2 commits that reference this issue on Mar 8, 2026

    feat: Azure DevOps platform adapter — Squad for enterprise (#191)

    ea70504

    feat: CommunicationAdapter — platform-agnostic agent-human communicat…

    2f93944
  9. tamirdresher
    added a commit that references this issue on Mar 8, 2026

    docs: add release notes blog github#26 + remove duplicate ADO blog gi…

    b4779ad
  10. bradygaster
    added a commit that references this issue on Mar 8, 2026

    docs: add release notes blog #26 + remove duplicate ADO blog #23 (#278)

    2092506
  11. bradygaster
    added 2 commits that reference this issue on Aug 21, 2026

    docs(e2e): record E4 agent-binding verification result

    c3fd953

    docs(e2e): record E4 agent-binding verification result (PASS, n=1) (#…

    51ea275
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    go:noNot pursuingsquadSquad triage inbox — Lead will assign to a membertype:spikeResearch/investigation — produces a plan, not code

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions