Skip to content

test(kb): prove in-loop RAG context injection end-to-end - #599

Merged
delete-cloud merged 1 commit into
mainfrom
feat/kb-in-loop-validation
Jun 14, 2026
Merged

delete-cloud merged 1 commit into
mainfrom
feat/kb-in-loop-validation

Conversation

@delete-cloud

@delete-cloud delete-cloud commented Jun 14, 2026 •

Copy link
Copy Markdown
Owner

Summary

Step ① of the personal-agent RAG roadmap: prove that enabling the kb plugin actually makes RAG run in the agent loop (it never has — kb is wired in core/app.py but absent from the default [agent.plugins].enabled, so retrieval was only ever validated standalone via CLI/evals).

Key fact: KBPlugin is a context-injection plugin (build_context), not a model-callable tool — once enabled it auto-retrieves and injects matched KB chunks into the context on every user message.

What this adds

  • tests/integration/test_kb_in_loop.py — one hermetic test that:
    • enables kb via a temp config override (does NOT touch shipped agent.toml / helm configmap),
    • injects a deterministic fake embedding_fn (KBPlugin supports it) and monkeypatches the KB OpenAI clients to hard-fail if a real endpoint is hit,
    • indexes a sentinel fact into a temp LanceDB KB,
    • runs one real PipelineAdapter turn with a mocked LLM provider,
    • asserts the sentinel chunk appears in the context the provider receives ([Context Pack] / ## Repo references).
  • No production code changes. Purely additive, no network, no external services.

Verification

  • uv run pytest tests/integration/test_kb_in_loop.py tests/coding_agent/plugins/test_kb_plugin.py -q → 14 passed (run outside codex sandbox).

Scope / not in this PR

  • ② o6n cutover (cluster-side ops).
  • ③ helm kb.enabled knob + [kb] render + SiliconFlow embedding secret + indexing Job for SRE/Obsidian corpora.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Tests
    • Added integration test for KB plugin context injection verification.

Add a hermetic integration test that enables the kb plugin via a temp
config override (not the shipped default), injects a deterministic fake
embedder, indexes a sentinel fact, and runs one real PipelineAdapter turn
with a mocked LLM provider — asserting the retrieved chunk lands in the
context the provider receives. No production code changes; no network.

Closes step 1 of the personal-agent RAG roadmap (validate in-loop RAG).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jun 14, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e3999ac2-27a4-45ed-8b81-61c8f13f2f92

📥 Commits

Reviewing files that changed from the base of the PR and between 23bcae4 and fe7d12c.

📒 Files selected for processing (1)
  • coding-agent/tests/integration/test_kb_in_loop.py

📝 Walkthrough

Walkthrough

A new async integration test file is added that verifies the KB plugin injects RAG context into the agent loop. It configures an agent with the kb plugin, indexes a sentinel document using a fake embedder, monkeypatches embedding clients and the plugin class, mocks the LLM provider, runs one turn, and asserts the prompt contains the injected context.

Changes

KB In-Loop RAG Context Injection Integration Test

Layer / File(s) Summary
End-to-end KB RAG injection test
coding-agent/tests/integration/test_kb_in_loop.py
Adds test_kb_plugin_injects_indexed_context_into_agent_loop: writes agent.toml enabling the kb plugin, creates and indexes a sentinel document into a KB instance with a fake embedding function, monkeypatches KB embedding client getters and the module-level KBPlugin with a TestKBPlugin subclass, replaces the LLM provider with an AsyncMock that streams fixed events and captures messages, then asserts plugin state, StopReason.NO_TOOL_CALLS, final message "ack", one provider call, and that the serialized prompt includes context pack markers, repo references, and the sentinel fact.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐇 A sentinel fact tucked in a doc so small,
Fake embeddings dodge the network's call.
The plugin stitches context neat and true,
"ack" comes back — the agent knew!
No real LLM, just mocks in play,
The rabbit checks RAG works today. ✨

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/kb-in-loop-validation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@delete-cloud
delete-cloud marked this pull request as ready for review June 14, 2026 10:58
@delete-cloud
delete-cloud merged commit a46619f into main Jun 14, 2026
2 checks passed
@delete-cloud
delete-cloud deleted the feat/kb-in-loop-validation branch June 14, 2026 10:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant