Skip to content

[bug] A failed vector delete is swallowed and unrecoverable — deleted knowledge keeps answering RAG queries #242

Closed

Description

@serge-ivo

What happens

agent-storage/knowledge.ts:34:

async deleteKnowledge(id: string): Promise<KnowledgeDoc | null> {
    const existing = await this.doStorage.get<KnowledgeDoc>(`kb:${id}`);
    if (!existing) return null;
    await this.doStorage.delete(`kb:${id}`);                       // ← record gone first
    await this.vectorDelete("knowledge", id).catch(() => undefined); // ← failure swallowed
    …
}

Two problems compounding:

  1. The vector removal is swallowed. If Vectorize errors (transient 5xx, timeout — vectorDelete
    batches deleteByIds 128 at a time, vectors.ts:267), the chunks stay in the index.
  2. The record is deleted first, so there is nothing left to retry against. The doc id is gone
    from DO storage; the user cannot delete it again, and no sweeper reconciles the two stores.

Result: the document disappears from Knowledge → Documents, but RAG keeps retrieving its chunks
permanently. The agent goes on citing a document the user deleted, with nothing recording why.

The Index tab (vector inventory) will show a source with no corresponding document — visible
evidence of the orphan, but no control that can remove it.

Why this is worth fixing beyond tidiness

Deleting a knowledge document is the action a user takes when a document is wrong or shouldn't
be there
— stale instructions, a mistakenly uploaded personal or client file. "It's deleted but
the agent still knows it" is the one outcome that must not happen silently.

Notable contrast in the same file

updateKnowledge (:44) gets this right and says so explicitly:

await this.vectorDelete("knowledge", id).catch(() => undefined);
// Not swallowed — a failed re-index must surface (the update_knowledge tool reports it).
await this.vectorizeStore("knowledge", id, …);

So the authors already reasoned about surfacing vector failures on the update path. The delete path
just didn't get the same treatment.

(Note the same swallow on the pre-delete inside updateKnowledge has a milder consequence: the doc
still exists, and re-saving retries. Worth tightening, but it is not the same severity.)

Suggested shape

  • Delete the vectors FIRST, then the record. A failure then leaves the document visible and the
    delete retryable, which is the recoverable ordering.
  • Do not swallow. Propagate so delete_knowledge / the console can report "couldn't remove this
    document's indexed content — try again", instead of reporting success.
  • Optionally, a reconcile pass for orphans already stranded by this (the Index tab can enumerate
    vector sources; any knowledge source with no kb:{id} record is an orphan).

Verification

  • Vectorize made to fail: the document is still listed, the delete reports failure, and a retry
    succeeds and removes both.
  • After a successful delete, a /search for text unique to that document returns nothing.
  • Mutation-check: restore the swallow and the failure test must go green again.

Activity

  1. serge-ivo commented on Aug 6, 2026

    @serge-ivo
    ContributorAuthor

    Shipped in fa7b00e.

    Verified: true, and broader than the ticket. knowledge.ts:34 was exactly as quoted. But AgentDO.handleDeleteKnowledge (agent-do.ts:930) — the path the console and DELETE /v1/instances/:id/knowledge/:id actually use — has the same ordering bug: it deleted the record first, and while it did not swallow, the throw arrived after the doc was already gone. So the console path produced an error and an orphan. Both are fixed.

    (Small correction: the 128-at-a-time batching at vectors.ts:267 is clearRepoVectors. vectorDelete issues one unbatched deleteByIds. Doesn't change the analysis.)

    What shipped

    • deleteKnowledge: vectors first, then the record, failure propagated with a message the agent can relay ("…it would keep answering searches. Try again."). executeStorageTool's outer catch turns that into a failed delete_knowledge tool result, so the model reports the failure instead of "Deleted X".
    • handleDeleteKnowledge: same reordering; on vector failure it logs to the durable error store and returns 503 with the document still listed, so the user can just delete it again.

    Tests — agent-storage/knowledge-delete.test.ts, real engine against an in-memory DO store and a Vectorize stub that can be made to fail. Covers your three verification points including the retry-after-outage. Mutation-checked: restoring the swallow turns 3 of the 5 red.

    Left open, deliberately:

    • The pre-delete swallow inside updateKnowledge (:57). You called it milder and you're right — the doc still exists and re-saving retries, and tightening it would make a transient blip fail an edit that would otherwise succeed on the re-index that follows.
    • No reconcile pass for orphans already stranded. Anything the old code orphaned is still in the index. The Index tab shows them (vector_stats / /v1/instances/:id/vectors) but there's still no control that removes a knowledge source with no kb:{id} record. Worth its own issue if there are known orphans in production.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions