Skip to content

Claude plugin: semantic rules in hooks, skills and commands #153

Open

Description

@mindsocket

Summary

Semantic rules (#114, PR #151) add paid, slower, opt-in evaluation to sctx validate and validate-file. The Claude plugin (hooks, skills, commands) was not reviewed as part of that work beyond a safety pin. This issue is to decide how the plugin should treat semantic rules.

Current state

  • The UserPromptSubmit and Stop hooks run sctx validate <space> --json to baseline a space and report new issues. A user who sets semantic: true or judge.autoRun in config would have every hook run judge the whole space, against a 60s hook timeout. PR feat(rules): semantic rules with closed verdicts, Jev and Anthropic judges #151 pins --no-semantic in the hook invocation so hooks can never trigger judging.
  • The /validate-space command runs sctx validate $ARGUMENTS, so users can pass --semantic explicitly.
  • The skill documents --semantic, --no-semantic and --rule-filter and has a short authoring note. Nothing guides when to use semantic rules or how to read their results.

Questions

  1. Should any hook ever judge? A Stop hook that judges only the files changed this turn (validate-file --semantic, which now judges only the file and its gating ancestors) could catch framing problems early, but costs money per turn and risks the timeout. Opt-in via plugin setting, or never?
  2. Skill guidance. When should an agent author a semantic rule rather than an expression rule? How to write a good rubric, verdicts and failOn? Does sctx schemas show list a rule's mode and verdicts so the agent sees them before authoring content?
  3. Audit-and-fix loops (Agent skill: audit-and-fix loop over validation output #117). Semantic violations carry evidence, and the evaluation records carry confidence and a verdict. Should an audit loop treat low-confidence verdicts differently (surface rather than auto-fix)? A "fix" that rewrites content to satisfy a judge can overfit the judge.
  4. Agent as judge. The agent running the loop could fulfil a semantic rule itself instead of calling a configured provider (interface-only; the CLI would emit the pending node/rule pairs). That avoids a second API key and bills nothing extra, but needs a protocol for returning verdicts. Related to Add qualitative content assessment to the agent skill #37 (qualitative assessment in the skill).
  5. Privacy. Judging sends node content to a third-party provider. The skill should say so before suggesting --semantic on a space.

Relates

Activity

  1. mindsocket commented on Oct 14, 2026

    @mindsocket
    OwnerAuthor

    Update from PR #151. failOn (mentioned in question 2) is gone; the surface the plugin sees is now:

    • severity as a verdict map, verdicts entries with {description, guidance}, rule-level judge, and when. sctx schemas show lists a rule's mode, per-verdict severities, judge and when, so an agent can see them before authoring.
    • Violations carry verdict and guidance (in --json and the text message), plus evidence. guidance is the natural thing for an agent to relay or act on.
    • --no-semantic is pinned in the hook's sctx validate call, so hooks never judge. --judge <name> overrides every rule's provider and implies --semantic.
    • Judge providers must be explicit in config (never inferred from env keys), so an agent can tell whether judging is configured by reading config, not guessing from the environment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions