Repository navigation
Description
Classification
- Change type: architecture / design spike
- Priority: P2
- Expected value: medium-high — clarifies the seam between 007 orchestration, provider adapters, tool execution, memory/context assembly and evals
- Implementation risk: low for the review; medium if adopted without typed failure/idempotency contracts
- Recommendation: YES, after Run semantics: replayable event reducer and fail-closed required-gate verdicts #43 establishes replayable run truth
- Related: Run evidence v2: action-level records linked to findings, costs and verifier outcomes #42 (action-level evidence), Run semantics: replayable event reducer and fail-closed required-gate verdicts #43 (replayable event reducer)
- Reference: https://github.com/ShenSeanChen/waku-agent
Context
waku-agent is a deliberately small teaching implementation of an agent harness:
gateway -> working context -> LLM/tool loop -> reply
| |
memory tracing/eval
Its strongest property is transparency: the loop, memory stores, provider adapters, observer events and deterministic evals are small enough to inspect directly.
007 should not copy the project wholesale. It can, however, use it as a reference implementation for several explicit boundaries that are easy to lose once agent code accumulates framework indirection.
Useful patterns to evaluate
1. Small explicit loop contract
A turn produces explicit data such as:
reply
iterations
ordered tool calls
per-call arguments and result
usage/cost metadata
This aligns with #42, but the runtime contract should remain independent from the persisted evidence schema.
2. Observer/event seam around the loop
The loop emits structured events for:
LLM call started/finished
tool call started/finished
streamed text
iteration limit reached
turn completed/failed
Dashboard, tracing and evidence collection subscribe to these events without being wired into the control flow.
#43 remains the canonical persisted run-event/reducer contract. This issue should only define how an in-process agent loop reports observations into that layer.
3. Deterministic side-effect evals separated from LLM-as-judge
Examples:
- assert that the expected tool fired;
- assert exact arguments or normalized intent;
- assert the expected file/database/diff artifact exists;
- separately score subjective response quality.
A model should not judge facts that a deterministic verifier can establish directly. Apparently software still needs this written down every few years.
4. Editable memory with explicit provenance
The reference separates semantic, episodic and procedural memory and keeps the source inspectable.
For 007, any future memory/context layer should add stronger contracts:
source artifact/message reference
created_at / valid_from / valid_to
confidence
supersedes / contradicts
retention classification
promotion status: observed -> proposed -> accepted
No model-generated memory should silently become policy or trusted execution context.
Patterns to reject or strengthen
LLM gate before cheap local retrieval
waku-agent performs an extra small-model call to decide whether to query local SQLite FTS5. For cheap local retrieval this can add more latency, cost and failure modes than the lookup it avoids.
Preferred evaluation order:
- run cheap deterministic/local retrieval;
- inspect rank/score and result count;
- inject nothing when confidence is low;
- use an LLM router only when retrieval itself is expensive or multi-source.
This should be measured, not adopted because the diagram calls it a hero moment.
Broad exception-based retries
Do not use except Exception to:
- retry an LLM request with a different parameter name;
- fall back from streaming after partial output;
- reinterpret authentication, rate-limit, timeout and protocol errors as the same condition.
Retries must be typed, bounded and recorded. Ambiguous post-send failures must not silently repeat non-idempotent work.
Untyped tool failures
A tool exception must not tear down the entire turn or become arbitrary text. Define a typed result equivalent to:
Success
InvalidArguments
PolicyDenied
TimedOut
Cancelled
TransientFailure
PermanentFailure
ProtocolError
The result should carry retryability, redacted diagnostics and artifact references.
Prompt-text idempotency
A plaintext history marker such as [tools used: ...] is not an idempotency mechanism.
Side-effecting tools need stable invocation/action IDs, deduplication keys and replay-safe semantics linked to #42/#43.
Proposed scope
Produce a short architecture proposal that maps each reference pattern to one of:
ADOPT
ADAPT
REJECT
DEFER
The proposal should define:
- the provider-neutral turn/loop interface;
- the in-process observer event interface and its mapping to Run semantics: replayable event reducer and fail-closed required-gate verdicts #43;
- typed tool outcomes and retry rules;
- idempotency requirements for side-effecting tools;
- deterministic eval categories versus judge-based evals;
- the minimum provenance contract for any future memory/context source;
- a benchmark plan for retrieval gating rather than assuming an LLM gate helps.
Suggested document:
docs/agent-loop-reference-review.md
Required controls
- A tool timeout is returned as a typed outcome and remains visible in run evidence.
- An authentication failure is never retried as a compatibility fallback.
- Streaming failure after partial output does not silently duplicate the full request without an explicit policy/event.
- Repeated side-effecting tool invocation with the same idempotency key does not duplicate the effect.
- Deterministic evals verify artifacts and tool effects without an LLM judge.
- Judge-based evals cannot override failed deterministic controls.
- Memory/context records retain source and validity metadata.
- Cheap local retrieval versus LLM-gated retrieval is benchmarked for latency, token cost and answer quality.
Acceptance
- Architecture review document exists with ADOPT/ADAPT/REJECT/DEFER decisions.
- Proposed loop and typed tool-outcome contracts are provider-neutral.
- Observer events map cleanly into Run semantics: replayable event reducer and fail-closed required-gate verdicts #43 without duplicating the persisted reducer schema.
- Side-effect idempotency links to Run evidence v2: action-level records linked to findings, costs and verifier outcomes #42 action identities/evidence.
- Eval policy explicitly separates deterministic outcomes from subjective judging.
- Future memory/context records require provenance and cannot auto-promote into trusted policy.
- Retrieval-gate recommendation is supported by a benchmark plan.
Non-goals
- No adoption of
waku-agentas a dependency. - No dashboard, Telegram or voice gateway work.
- No general personal-assistant product scope.
- No vector database requirement.
- No new persisted event/reducer schema; that belongs to Run semantics: replayable event reducer and fail-closed required-gate verdicts #43.
- No duplicate action evidence contract; that belongs to Run evidence v2: action-level records linked to findings, costs and verifier outcomes #42.
- No autonomous memory-to-policy promotion.
Activity
PhysShell commentedon Nov 21, 2026
Architecture donor: Kiro's unified agent harness (reference only, 21.11.2026)
Research Page · Primary engineering source.
Kiro previously had three independently evolving harness implementations (IDE/TS, CLI/Rust, web/Python). Separate session formats, compaction, subagent semantics and incompatible permission matchers caused drift. Their redesign is a single harness behind multiple clients; capability-level permission rules (fs_read, fs_write, shell, web_fetch, mcp, subagent) are evaluated through Cedar, with deny-wins composition.
For the existing waku-agent reference-review, add a contract comparison, not a framework migration: (1) loop/runtime interface vs UI adapters, (2) consistent typed tool failure and retries, (3) authority decisions enforced across all tools in a capability class, including wrappers, (4) stable session state/compaction across clients, (5) independent verifier output mapped to #42/#43. Negative control: deny fs_read yet read via grep/glob/MCP; if it bypasses, policy is not actually capability-wide.
Boundaries: 007 parked; keep RESEARCH/NOT ADOPTED. Do not add second event reducer, invent a policy DSL, or replace the current harness based on a vendor blog. Relative benefit and real interfaces are UNKNOWN until a bounded comparison.