Repository navigation
Description
Problem
atomic_write_bytes() in openkb/locks.py (the shared helper behind atomic_write_text/atomic_write_json, used by the compiler, indexer, converter, config, and page_ops) performs a single, unretried os.replace(tmp_path, path) to publish a write. On Windows, a freshly-written temp file can be briefly locked by an external process (real-time antivirus scanning, the search indexer, backup/sync agents) right before the rename — os.replace() then raises PermissionError (WinError 5, Access is denied) even though the lock typically clears within milliseconds to a couple of seconds.
Because this exception surfaces all the way up, _run_compile_with_retry() in cli.py (added in #229) treats it like any other failure: it discards the fully-generated summary/concept/entity content already held in memory and re-runs the entire LLM compile pipeline from scratch (summary, concepts-plan, every concept/entity page, summary-rewrite) just to retry a single file rename — wasting the LLM cost and time of a full document compile for what is, at the OS level, a transient and typically self-clearing lock.
Reproduktion
Observed during a long-running batch openkb add (single KB, debug: true, thousands of documents processed sequentially over several hours):
[ERROR] Compilation failed: [WinError 5] Zugriff verweigert: '<kb>\wiki\summaries\.<doc>-xxxxxxxx.md.xxxxxxxx.tmp' -> '<kb>\wiki\summaries\<doc>-xxxxxxxx.md'
The retry (_run_compile_with_retry, 2 attempts) re-ran the whole compile and hit the same error again on the same file before finally failing the add. A later automatic retry (the calling batch script re-scans and re-adds failed documents) succeeded on the same file without any code change, confirming the lock was transient.
Kontext
- OS: Windows 11, NTFS local disk (not a network/cloud-sync path)
- Python 3.12/3.13,
openkbinstalled editable from a recentmain-based branch - Affects any long-running, high-file-count
addsession, not a specific document type
Vorschlag
Add a small, bounded retry-with-backoff around the os.replace() call inside atomic_write_bytes() itself (e.g. a handful of attempts with short exponential backoff, catching PermissionError specifically), so the transient lock is absorbed at the exact point it occurs. This fixes the issue for every caller of atomic_write_text/atomic_write_json, not just the compiler, and — because it resolves before the exception ever reaches _run_compile_with_retry — avoids the wasteful full-pipeline regeneration in the common case without needing to restructure the compiler's generate/persist flow.
This issue was drafted with the assistance of an AI assistant.
Activity