Repository navigation
feat(rules): semantic rules with closed verdicts, Jev and Anthropic judges - #149
mindsocket wants to merge 1 commit into
Conversation
Semantic rules (mode: semantic) declare a closed verdict set (verdicts, failOn) and are judged per node, all of a node's rules in one provider call. Judges: Anthropic (default claude-haiku-5-5), OpenAI-compatible, Jev via OpenRouter decisions API, fake. Semantic rules run inside cascade gating (#115) after deterministic rules and only with --semantic. --rule-filter narrows rules by mode, category, severity, type or id.
| const JUDGE_CONFIG_SCHEMA = { | ||
| type: 'object', | ||
| properties: { | ||
| provider: { type: 'string', enum: ['anthropic', 'openai', 'fake'] }, |
There was a problem hiding this comment.
jev is missing from the provider enum, so a config file with "provider": "jev" fails loadConfig validation with Invalid config. docs/rules.md, JudgeConfig in src/judge/types.ts and createJudgeProvider all support jev; only SCTX_JUDGE_PROVIDER=jev works today.
| provider: { type: 'string', enum: ['anthropic', 'openai', 'fake'] }, | |
| provider: { type: 'string', enum: ['anthropic', 'openai', 'jev', 'fake'] }, |
| gatesDescendants: { type: 'boolean' }, | ||
| }, | ||
| required: ['id', 'category', 'description', 'check'], | ||
| required: ['id', 'category', 'description'], |
There was a problem hiding this comment.
Dropping check from required with no conditional replacement means an expression rule with no check now passes metadata validation, although docs/rules.md says check is required unless mode: semantic. Without a hierarchy, validateRules silently skips the rule. With one, gating.ts calls evaluateExpression(rule.check as string, ...) with undefined, so the rule likely reports a violation on every node. Please require check when mode isn't semantic (e.g. if/then), and regenerate the metaschema.
| } | ||
|
|
||
| const globalRules = rules.filter((r) => r.scope === 'global'); | ||
| const globalRules = rules.filter((r) => r.scope === 'global' && r.mode !== 'semantic'); |
There was a problem hiding this comment.
A rule with mode: semantic and scope: global is in neither list. globalRules here excludes semantic rules, and perNodeRules (line 150) excludes scope: global. So in the hierarchy path it is never judged and produces no error. validateRules (src/schema/validate-rules.ts) ignores scope for semantic rules and judges them on every node. The metadata contract allows this combination. Either reject it in the contract or handle it the same way in both paths.
Closes #114
Stacked on #148 (cascade gating, #115); base is
issue-115-cascade-gating. Retarget tomainonce #148 merges.What
Semantic rules (
mode: semantic) judged by a model or decision model, running inside cascade gating. Opt-in via--semantic.verdicts({verdict: description}) andfailOn; defaultpass/fail. Judges return a verdict, optional confidence/probabilities, optional evidence.claude-haiku-5-5), OpenAI-compatible, Jev via OpenRouter's alpha decisions API (provider: "jev",OPENROUTER_API_KEY, never auto-detected), fake for tests.gatedRules), and gate their subtree pergatesDescendants. Judge errors are reported but never gate.--rule-filter dim=values(mode, category, severity, type, id) onvalidateandvalidate-file.docs/rules.md.Not in this PR
Jev tested against a mocked fetch only (no live call). No retry/concurrency/cache.
validate-file --semanticjudges the whole space then filters (lazy ancestor judging is a follow-up).includeGatedis still one switch for both rule kinds. Rule scope beyond the node itself (ancestors, siblings) not yet supported.checknot yet required for non-semantic rules.661 tests pass; typecheck and biome clean.