Skip to content

feat(explain): add evaluation event hooks to the RVM - #898

Open
snehabandla wants to merge 2 commits into
microsoft:mainfrom
snehabandla:explain/vm-hooks
Open

snehabandla wants to merge 2 commits into
microsoft:mainfrom
snehabandla:explain/vm-hooks

Conversation

@snehabandla

Copy link
Copy Markdown
Contributor

Summary

Adds regorus::explain::ExplanationRecorder, a trait that receives the RVM's evaluation events, behind the default-off explanations feature. Install one with RegoVM::set_explanation_recorder and remove it with RegoVM::take_explanation_recorder. The recorder receives:

  • check: each executed comparison, membership, truthiness, negation, every quantifier, and failed existence check (an undefined value that had to be defined, including the collection of some x in xs or xs[_]). Each event has the operator, the operand values and the outcome (Success, Failure or Undefined);
  • body_end: the end of each rule body, with its outcome and, for complete rules and functions, the produced value;
  • emission: each value produced by a partial set or partial object rule;
  • builtin_error: builtin and arithmetic operator errors, with the builtin name, a message without a source location, and whether the error was raised;
  • not_captured: constructs whose contributions are not tracked yet (comprehensions);
  • reset: the start of each evaluation.

Events identify their instruction by program counter, rule, definition, body and call depth (HookContext). They carry no source text: that is decoded from the compiled program when a report is built. Events borrow the VM's values; a recorder copies what it keeps.

This PR also contains a separate fix commit, fix(rvm): clear loop and comprehension stacks before each rule body. In run-to-completion mode, a rule body that failed with an error inside a loop or comprehension left entries on the rule's loop and comprehension stacks. The next body of the same rule then sent its guards to the stale entry. For example, an else body evaluated after an error in a comprehension looped until the instruction limit. Each body now starts with empty stacks.

Details

  • Gating. Every hook site is #[cfg(feature = "explanations")] and checks whether a recorder is installed before doing any work.
  • Semantics. The recorder module docs list where events differ from the policy text:
    • a rule is evaluated once and later references are cached;
    • cached builtin calls report no error again;
    • parameter patterns are not reported as body checks;
    • array destructuring reports a length comparison;
    • iterating an empty collection or a non-collection reports no check of its own.
  • Errors. Operator errors use OPA-style type messages, for example operand 1 must be number but got string or divide by zero. | and & are reported as or and and. Errors that end the evaluation, such as exceeded limits or numeric overflow, are not reported.
  • Comprehensions. Checks inside a comprehension, and in rules or functions evaluated for it, are delivered with HookContext::in_comprehension set. The comprehension itself is reported as not_captured.
  • Recorder behaviour. The recorder runs inside the evaluation, and its time and allocations count toward the VM's limits. If it panics, the panic propagates and the recorder is dropped. ExplanationRecorder requires Send + Sync, so RegoVM stays Send + Sync with the default arc feature. Without arc, Value is not Send, so a recorder that stores values cannot satisfy the bound.
  • Azure Policy. Azure Policy conditions and their allOf/anyOf/not combinations are not reported.
  • Overhead (cachegrind instruction counts per evaluation over loop, call, nested-rule and every workloads, in both execution modes):
    • Feature off: Suspendable mode is identical to main. Run-to-completion adds at most 20 instructions per evaluation (≤ 0.09%), all from the stack-clear fix.
    • Feature on, no recorder: +0.15% to +0.53%.
    • Feature on, with a recorder that ignores every event: +5% to +21%.

Compatibility

New public API, available only with the explanations feature. With the feature off, the only change is the stack-clear fix, which changes run-to-completion results only for rules whose earlier body failed with an error inside a loop or comprehension.

Related

Testing

  • New tests in tests/rvm/explain_hooks.rs cover:
    • every comparison operator, membership, truthiness, negation and every;
    • bindings and parameter patterns, which are not reported as checks;
    • partial-rule emissions and undefined object entries;
    • body, definition, helper and call-depth identification;
    • builtin and arithmetic error messages, set-operator names and strict-mode overflow;
    • comprehensions and errors inside them;
    • iteration over undefined collections;
    • recorder reset, including a panicking reset;
    • Azure Policy programs, and RegoVM staying Send + Sync with arc.
  • New YAML case tests/rvm/rego/cases/rule_body_isolation.yaml. It fails without the stack-clear fix by reaching the instruction limit.
  • cargo fmt --all -- --check (including bindings); cargo clippy --all-targets --all-features -- -Dwarnings; cargo clippy --no-default-features -- -Dwarnings; clippy for the ffi, java, python and wasm bindings.
  • cargo test, cargo test --features rego-extensions, and cargo test --test aci / --test kata with and without rego-extensions; cargo test --doc.
  • cargo test --features explanations,azure_policy --lib --test mod: 839 lib tests and 378 integration tests passed, including the explanation corpus.
  • OPA conformance: cargo test --features opa-testutil,serde_json/arbitrary_precision,rego-extensions --test opa over tests/opa.passing (2875 passed, 0 failed).
  • tests/ensure_no_std builds for thumbv7m-none-eabi; cargo build --all-features.

snehabandla and others added 2 commits November 3, 2026 04:16
In run-to-completion mode, a rule body that failed with an error inside a
loop or comprehension left its entries on the rule's loop and
comprehension stacks. The next body of the same rule then routed its
guards to the stale entry. For example, an `else` branch evaluated after
an error in a comprehension looped until the instruction limit.

Clear both stacks before each body. The stacks are already swapped per
rule call, so this only drops entries left by the previous body.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add `ExplanationRecorder`, a trait that receives the RVM's evaluation
events, behind the default-off `explanations` feature. Install one with
`RegoVM::set_explanation_recorder`. It receives:

- one check per executed condition (comparison, membership, truthiness,
  negation, `every`, undefined values), with its operands and outcome;
- the end of each rule body, with the value of a complete rule or
  function;
- the values emitted by partial set and object rules;
- builtin and arithmetic operator errors;
- the start of comprehensions, whose checks are not attributed to the
  enclosing body.

Events carry the program counter, rule, definition and body indices,
call depth and whether they occur inside a comprehension. Both execution
modes deliver the same events. Without a recorder, each hook site costs
one `is_some()` test; with the feature off, the hooks are compiled out.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The explanation tests fail to compile without arc because their value-capturing recorder cannot satisfy the required Send + Sync bounds.

5 open issues
What changed in this PR

Adds opt-in evaluation events to Regorus’s RVM, building on the existing explanation report types. Recorders receive borrowed values and execution context; source details remain in the compiled program.

Changes:

  • Introduces the recorder trait and public event types.
  • Adds feature-gated hooks across both RVM execution modes.
  • Clears loop and comprehension stacks between rule bodies and adds regression coverage.
File Description
tests/​rvm/​rego/​mod.rs Runs Rego suites with a no-op recorder installed.
tests/​rvm/​rego/​cases/​rule_body_isolation.yaml Adds a regression for stale state after comprehension errors.
tests/​rvm/​mod.rs Registers the hook test module.
tests/​rvm/​explain_hooks.rs Tests event payloads, context, lifecycle, errors, and execution-mode parity.
tests/​explanation_corpus/​mod.rs Runs corpus evaluations with a no-op recorder.
src/​rvm/​vm/​state.rs Resets the recorder for each evaluation.
src/​rvm/​vm/​rules.rs Reports body completion, excludes parameter checks, and clears per-body stacks.
src/​rvm/​vm/​mod.rs Feature-gates the VM hook module.
src/​rvm/​vm/​machine.rs Stores and initializes recorder state.
src/​rvm/​vm/​loops.rs Reports iteration over undefined collections.
src/​rvm/​vm/​functions.rs Reports builtin errors.
src/​rvm/​vm/​explain.rs Implements recorder management, context, event decoding, and delivery.
src/​rvm/​vm/​dispatch.rs Adds hooks for checks, arithmetic errors, emissions, and comprehensions.
src/​explain/​recorder.rs Defines and documents the recorder contract and event types.
src/​explain/​mod.rs Re-exports the public recorder API.

Review effort: Max

//! Events delivered to an `ExplanationRecorder` by the RVM, in both
//! execution modes.

#![cfg(feature = "explanations")]
Comment thread src/rvm/vm/dispatch.rs
};
#[cfg(feature = "explanations")]
if self.explain_enabled() {
self.explain_guard(program, register, mode, passed);
Comment thread src/rvm/vm/dispatch.rs
ObjectSet { obj, key, value } => {
#[cfg(feature = "explanations")]
if self.explain_enabled() {
self.explain_emission(Some(key), value);
Comment thread src/rvm/vm/explain.rs
Comment on lines +182 to +184
if (mode == GuardMode::NotUndefined && passed) || self.explain_in_parameters() {
return;
}
Comment thread src/rvm/vm/state.rs
Comment on lines +109 to +112
#[cfg(feature = "explanations")]
if self.explain_enabled() {
self.explain_reset();
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants