Skip to content

Reference architecture review: extract transparent agent-loop patterns from waku-agent #57

Open

Description

@PhysShell

Classification

Context

waku-agent is a deliberately small teaching implementation of an agent harness:

gateway -> working context -> LLM/tool loop -> reply
                    |                |
                  memory          tracing/eval

Its strongest property is transparency: the loop, memory stores, provider adapters, observer events and deterministic evals are small enough to inspect directly.

007 should not copy the project wholesale. It can, however, use it as a reference implementation for several explicit boundaries that are easy to lose once agent code accumulates framework indirection.

Useful patterns to evaluate

1. Small explicit loop contract

A turn produces explicit data such as:

reply
iterations
ordered tool calls
per-call arguments and result
usage/cost metadata

This aligns with #42, but the runtime contract should remain independent from the persisted evidence schema.

2. Observer/event seam around the loop

The loop emits structured events for:

LLM call started/finished
tool call started/finished
streamed text
iteration limit reached
turn completed/failed

Dashboard, tracing and evidence collection subscribe to these events without being wired into the control flow.

#43 remains the canonical persisted run-event/reducer contract. This issue should only define how an in-process agent loop reports observations into that layer.

3. Deterministic side-effect evals separated from LLM-as-judge

Examples:

  • assert that the expected tool fired;
  • assert exact arguments or normalized intent;
  • assert the expected file/database/diff artifact exists;
  • separately score subjective response quality.

A model should not judge facts that a deterministic verifier can establish directly. Apparently software still needs this written down every few years.

4. Editable memory with explicit provenance

The reference separates semantic, episodic and procedural memory and keeps the source inspectable.

For 007, any future memory/context layer should add stronger contracts:

source artifact/message reference
created_at / valid_from / valid_to
confidence
supersedes / contradicts
retention classification
promotion status: observed -> proposed -> accepted

No model-generated memory should silently become policy or trusted execution context.

Patterns to reject or strengthen

LLM gate before cheap local retrieval

waku-agent performs an extra small-model call to decide whether to query local SQLite FTS5. For cheap local retrieval this can add more latency, cost and failure modes than the lookup it avoids.

Preferred evaluation order:

  1. run cheap deterministic/local retrieval;
  2. inspect rank/score and result count;
  3. inject nothing when confidence is low;
  4. use an LLM router only when retrieval itself is expensive or multi-source.

This should be measured, not adopted because the diagram calls it a hero moment.

Broad exception-based retries

Do not use except Exception to:

  • retry an LLM request with a different parameter name;
  • fall back from streaming after partial output;
  • reinterpret authentication, rate-limit, timeout and protocol errors as the same condition.

Retries must be typed, bounded and recorded. Ambiguous post-send failures must not silently repeat non-idempotent work.

Untyped tool failures

A tool exception must not tear down the entire turn or become arbitrary text. Define a typed result equivalent to:

Success
InvalidArguments
PolicyDenied
TimedOut
Cancelled
TransientFailure
PermanentFailure
ProtocolError

The result should carry retryability, redacted diagnostics and artifact references.

Prompt-text idempotency

A plaintext history marker such as [tools used: ...] is not an idempotency mechanism.

Side-effecting tools need stable invocation/action IDs, deduplication keys and replay-safe semantics linked to #42/#43.

Proposed scope

Produce a short architecture proposal that maps each reference pattern to one of:

ADOPT
ADAPT
REJECT
DEFER

The proposal should define:

  1. the provider-neutral turn/loop interface;
  2. the in-process observer event interface and its mapping to Run semantics: replayable event reducer and fail-closed required-gate verdicts #43;
  3. typed tool outcomes and retry rules;
  4. idempotency requirements for side-effecting tools;
  5. deterministic eval categories versus judge-based evals;
  6. the minimum provenance contract for any future memory/context source;
  7. a benchmark plan for retrieval gating rather than assuming an LLM gate helps.

Suggested document:

docs/agent-loop-reference-review.md

Required controls

  1. A tool timeout is returned as a typed outcome and remains visible in run evidence.
  2. An authentication failure is never retried as a compatibility fallback.
  3. Streaming failure after partial output does not silently duplicate the full request without an explicit policy/event.
  4. Repeated side-effecting tool invocation with the same idempotency key does not duplicate the effect.
  5. Deterministic evals verify artifacts and tool effects without an LLM judge.
  6. Judge-based evals cannot override failed deterministic controls.
  7. Memory/context records retain source and validity metadata.
  8. Cheap local retrieval versus LLM-gated retrieval is benchmarked for latency, token cost and answer quality.

Acceptance

Non-goals

Activity

  1. PhysShell commented on Nov 21, 2026

    @PhysShell
    OwnerAuthor

    Architecture donor: Kiro's unified agent harness (reference only, 21.11.2026)

    Research Page · Primary engineering source.

    Kiro previously had three independently evolving harness implementations (IDE/TS, CLI/Rust, web/Python). Separate session formats, compaction, subagent semantics and incompatible permission matchers caused drift. Their redesign is a single harness behind multiple clients; capability-level permission rules (fs_read, fs_write, shell, web_fetch, mcp, subagent) are evaluated through Cedar, with deny-wins composition.

    For the existing waku-agent reference-review, add a contract comparison, not a framework migration: (1) loop/runtime interface vs UI adapters, (2) consistent typed tool failure and retries, (3) authority decisions enforced across all tools in a capability class, including wrappers, (4) stable session state/compaction across clients, (5) independent verifier output mapped to #42/#43. Negative control: deny fs_read yet read via grep/glob/MCP; if it bypasses, policy is not actually capability-wide.

    Boundaries: 007 parked; keep RESEARCH/NOT ADOPTED. Do not add second event reducer, invent a policy DSL, or replace the current harness based on a vendor blog. Relative benefit and real interfaces are UNKNOWN until a bounded comparison.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions