Skip to content

feat(rules): semantic rules with closed verdicts, Jev and Anthropic judges - #149

Closed
mindsocket wants to merge 1 commit into
issue-115-cascade-gatingfrom
spike-114a-builtin-provider
Closed

mindsocket wants to merge 1 commit into
issue-115-cascade-gatingfrom
spike-114a-builtin-provider

Conversation

@mindsocket

Copy link
Copy Markdown
Owner

Closes #114

Stacked on #148 (cascade gating, #115); base is issue-115-cascade-gating. Retarget to main once #148 merges.

What

Semantic rules (mode: semantic) judged by a model or decision model, running inside cascade gating. Opt-in via --semantic.

  • Closed verdicts. A rule declares verdicts ({verdict: description}) and failOn; default pass/fail. Judges return a verdict, optional confidence/probabilities, optional evidence.
  • Per-node batching. All semantic rules on a node go to the judge in one call. Chat LLMs (Anthropic, OpenAI-compatible) make one request per rule; Jev answers them in one request.
  • Providers. Anthropic (default claude-haiku-5-5), OpenAI-compatible, Jev via OpenRouter's alpha decisions API (provider: "jev", OPENROUTER_API_KEY, never auto-detected), fake for tests.
  • Gating. Semantic rules run after a node's deterministic rules, are skipped on gated nodes (gatedRules), and gate their subtree per gatesDescendants. Judge errors are reported but never gate.
  • --rule-filter dim=values (mode, category, severity, type, id) on validate and validate-file.
  • Docs: new "Semantic rules" section in docs/rules.md.

Not in this PR

Jev tested against a mocked fetch only (no live call). No retry/concurrency/cache. validate-file --semantic judges the whole space then filters (lazy ancestor judging is a follow-up). includeGated is still one switch for both rule kinds. Rule scope beyond the node itself (ancestors, siblings) not yet supported. check not yet required for non-semantic rules.

661 tests pass; typecheck and biome clean.

Semantic rules (mode: semantic) declare a closed verdict set (verdicts, failOn)
and are judged per node, all of a node's rules in one provider call. Judges:
Anthropic (default claude-haiku-5-5), OpenAI-compatible, Jev via OpenRouter
decisions API, fake. Semantic rules run inside cascade gating (#115) after
deterministic rules and only with --semantic. --rule-filter narrows rules by
mode, category, severity, type or id.
Comment thread src/config.ts
const JUDGE_CONFIG_SCHEMA = {
type: 'object',
properties: {
provider: { type: 'string', enum: ['anthropic', 'openai', 'fake'] },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jev is missing from the provider enum, so a config file with "provider": "jev" fails loadConfig validation with Invalid config. docs/rules.md, JudgeConfig in src/judge/types.ts and createJudgeProvider all support jev; only SCTX_JUDGE_PROVIDER=jev works today.

Suggested change
provider: { type: 'string', enum: ['anthropic', 'openai', 'fake'] },
provider: { type: 'string', enum: ['anthropic', 'openai', 'jev', 'fake'] },

gatesDescendants: { type: 'boolean' },
},
required: ['id', 'category', 'description', 'check'],
required: ['id', 'category', 'description'],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropping check from required with no conditional replacement means an expression rule with no check now passes metadata validation, although docs/rules.md says check is required unless mode: semantic. Without a hierarchy, validateRules silently skips the rule. With one, gating.ts calls evaluateExpression(rule.check as string, ...) with undefined, so the rule likely reports a violation on every node. Please require check when mode isn't semantic (e.g. if/then), and regenerate the metaschema.

Comment thread src/schema/gating.ts
}

const globalRules = rules.filter((r) => r.scope === 'global');
const globalRules = rules.filter((r) => r.scope === 'global' && r.mode !== 'semantic');

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A rule with mode: semantic and scope: global is in neither list. globalRules here excludes semantic rules, and perNodeRules (line 150) excludes scope: global. So in the hierarchy path it is never judged and produces no error. validateRules (src/schema/validate-rules.ts) ignores scope for semantic rules and judges them on every node. The metadata contract allows this combination. Either reject it in the contract or handle it the same way in both paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant