Open
Description
Summary
Semantic rules (#114, PR #151) add paid, slower, opt-in evaluation to sctx validate and validate-file. The Claude plugin (hooks, skills, commands) was not reviewed as part of that work beyond a safety pin. This issue is to decide how the plugin should treat semantic rules.
Current state
- The
UserPromptSubmitandStophooks runsctx validate <space> --jsonto baseline a space and report new issues. A user who setssemantic: trueorjudge.autoRunin config would have every hook run judge the whole space, against a 60s hook timeout. PR feat(rules): semantic rules with closed verdicts, Jev and Anthropic judges #151 pins--no-semanticin the hook invocation so hooks can never trigger judging. - The
/validate-spacecommand runssctx validate $ARGUMENTS, so users can pass--semanticexplicitly. - The skill documents
--semantic,--no-semanticand--rule-filterand has a short authoring note. Nothing guides when to use semantic rules or how to read their results.
Questions
- Should any hook ever judge? A
Stophook that judges only the files changed this turn (validate-file --semantic, which now judges only the file and its gating ancestors) could catch framing problems early, but costs money per turn and risks the timeout. Opt-in via plugin setting, or never? - Skill guidance. When should an agent author a semantic rule rather than an expression rule? How to write a good
rubric,verdictsandfailOn? Doessctx schemas showlist a rule's mode and verdicts so the agent sees them before authoring content? - Audit-and-fix loops (Agent skill: audit-and-fix loop over validation output #117). Semantic violations carry
evidence, and the evaluation records carryconfidenceand a verdict. Should an audit loop treat low-confidence verdicts differently (surface rather than auto-fix)? A "fix" that rewrites content to satisfy a judge can overfit the judge. - Agent as judge. The agent running the loop could fulfil a semantic rule itself instead of calling a configured provider (interface-only; the CLI would emit the pending node/rule pairs). That avoids a second API key and bills nothing extra, but needs a protocol for returning verdicts. Related to Add qualitative content assessment to the agent skill #37 (qualitative assessment in the skill).
- Privacy. Judging sends node content to a third-party provider. The skill should say so before suggesting
--semanticon a space.
Relates
- Semantic rule evaluation: make descriptive rules executable (LLM-as-judge) #114, Agent skill: audit-and-fix loop over validation output #117 (audit-and-fix loop), Add qualitative content assessment to the agent skill #37 (qualitative assessment in the agent skill), Create evals for the structured-context agent skill #38 (skill evals)
Metadata
Metadata
Assignees
Labels
No labels
Activity
mindsocket commentedon Oct 14, 2026
Update from PR #151.
failOn(mentioned in question 2) is gone; the surface the plugin sees is now:severityas a verdict map,verdictsentries with{description, guidance}, rule-leveljudge, andwhen.sctx schemas showlists a rule's mode, per-verdict severities,judgeandwhen, so an agent can see them before authoring.verdictandguidance(in--jsonand the text message), plusevidence.guidanceis the natural thing for an agent to relay or act on.--no-semanticis pinned in the hook'ssctx validatecall, so hooks never judge.--judge <name>overrides every rule's provider and implies--semantic.