Repository navigation
Open
Description
Problem
search_wiki (#233/#234) is a single BM25 index over concepts/, entities/, and
summaries/ combined. Two related issues surface as a wiki grows:
- Concepts/entities shouldn't be keyword-searched at all. They are meant to be
discovered by semantic browsing (an LLM scanningindex.md's one-line briefs and
picking the relevant slug by meaning), not by keyword frequency — but they currently
share one BM25 corpus with summaries, so an unrelated but keyword-dense concept page
can outrank the summary the caller actually needed. - One BM25 tier conflates two very different recall needs. A query for a generic
term (e.g. "case") mostly needs a precision filter (which document's one-line brief
is actually about that topic), while a query for a specific fact (e.g. an exact field
name) needs recall into full document bodies — a single combined index tunes for
neither well, and long PageIndex documents (sources/*.json) aren't searchable at all
today, even though details like authorship/creation dates only ever live there, never
in a generated summary.
Vorschlag
- Add
list_taxonomy_items()/get_taxonomy_item()(openkb/agent/tools.py) for semantic
browsing of persistedconcepts//entities/pages (kind-filterable), separate from any
BM25 search. - Add
TieredWikiSearch(openkb/fulltext_index.py) with three independent BM25 tiers,
each over a different granularity so a query only pays the recall/precision trade-off
it actually needs:briefs— one-linedescription/legacybrieffrontmatter persummaries/*.md.summaries— full body ofsummaries/*.md.sources—sources/*.md(whole file) +sources/*.jsonPageIndex docs, indexed
per page (not per document), each hit carrying aLocator(kind="page", value=N)
directly usable withget_page_content(doc_name, pages=str(N)).
- Add
Locator(kind: "line"|"page") onSearchHitso a hit can point at an exact line
(short docs) or page (PageIndex docs), not just "this file".
This issue covers the core engine only — no CLI/MCP/agent wiring (tracked separately as
follow-ups building on this one).
Kontext
- Builds on feat(agent): hybrid BM25 full-text search tool for query/chat agent #233/feat(agent): hybrid BM25 search_wiki tool for query/chat agent #234's
WikiFullTextIndex/search_wiki(kept unchanged, byte-for-byte
backward compatible, for existing callers) —TieredWikiSearchis additive, sharing the
same BM25 math via an extracted_BM25Scorer.
This issue was drafted with the assistance of an AI assistant.
Metadata
Metadata
Assignees
Labels
No labels
Activity