Repository navigation
Open
Description
Hello,
I have tried now multiple times to get to use my LM Studio runtime for openKB on my M3 Pro/36GB macbook pro. LM Studio running on localhost, and then these examples from the README.md on their github page:
openkb add paper.pdf
openkb add ~/papers/ # Add a whole directory
openkb add https://arxiv.org/pdf/2509.11420 # Or fetch from a URL
With the models qwen/qwen2.5-coder-14b, qwen/qwen3-14b and google/gemma-4-31b. I have had absolutely no luck in ingesting any document(s) with any of these models (atleast no sign of progress in multiple hours).
Let's take LM studio and gemma-4-31b. 2 hours after the start of the session, I see these errors:
$ openkb add https://arxiv.org/pdf/2509.11420
Downloading: https://arxiv.org/pdf/2509.11420
Saved: raw/2509.11420v1_6.pdf (2.1 MB PDF)
Adding: 2509.11420v1_6.pdf
Long document detected — indexing with PageIndex...
start find_toc_pages
no toc found
process_no_toc
start_index: 1
divide page_list to groups 3
start generate_toc_init
************* Retrying *************
root ERROR: Error: litellm.Timeout: APITimeoutError - Request timed out. Error_str: Request timed out.
************* Retrying *************
root ERROR: Error: litellm.Timeout: APITimeoutError - Request timed out. Error_str: Request timed out.
************* Retrying *************
root ERROR: Error: litellm.Timeout: APITimeoutError - Request timed out. Error_str: Request timed out.
************* Retrying *************
root ERROR: Error: litellm.Timeout: APITimeoutError - Request timed out. Error_str: Request timed out.
************* Retrying *************
root ERROR: Error: litellm.Timeout: APITimeoutError - Request timed out. Error_str: Request timed out.
start generate_toc_continue
openkb.indexer WARNING: PageIndex attempt 1/3 failed for 2509.11420v1_6.pdf: Failed to index /Users/sonny/Code/open-kb/raw/2509.11420v1_6.pdf: 'dict' object has no attribute 'extend'
start find_toc_pages
(repeat the steps two more PageIndex attempts and then)
[ERROR] Indexing failed: Failed to index 2509.11420v1_6.pdf after 3 attempts:
...
$
My .openkb/config.yaml:
# .openkb/config.yaml
llm:
provider: "openai"
model: "openai/gemma-4-31b"
pipeline:
long_document_threshold: 60
chunk_size: 2000
and my .env:
# .env
# Force LiteLLM to send all OpenAI-provider traffic to LM Studio
OPENAI_API_BASE=http://localhost:1234/v1
# Use any placeholder string (LM Studio doesn't validate this, but LiteLLM requires its presence)
LLM_API_KEY=local-lm-studio-key
Does anyone have any idea what I might me doing wrong? Has anyone gotten openKB working on local AI?
Metadata
Metadata
Assignees
Labels
No labels
Activity
SonnyRajagopalan commentedon Jun 26, 2026
I figured out atleast one problem. It turns out that I need to change the value of
_JSON_RESPONSE_FORMATin$OPENKB_DIR/.venv/lib/python3.12/site-packages/openkb/agent/compiler.pytotext. It's a quick fix and a rather clunky one to do this, but I changed the line after the import statements incompiler.pyfromto
and that seemed to make openkb work.
However, a simple 15-page PDF (https://arxiv.org/pdf/1706.03762v5) took over 2 hours to complete, and towards the end, I did end up getting a lot of these errors anyway:
docs(examples): openkb usage guides with real generated artifacts (#141)
KylinMountain commentedon Jun 26, 2026
Hi @SonnyRajagopalan — this isn't a bug, it's two local-runtime limits. The APITimeoutError is LiteLLM's default timeout aborting your slow local calls (the 'dict' has no attribute 'extend' is just the downstream symptom). The Context size exceeded is LM Studio's default context window (often 4096) being too small.
Two fixes, no need to edit site-packages:
drop_params: true replaces your _JSON_RESPONSE_FORMAT edit. Note 14B–31B models are slow at PageIndex's long-context work — try a short PDF first, or a cloud key to confirm the pipeline. Full reference:
examples/configuration/.SonnyRajagopalan commentedon Jun 26, 2026
Thank you, @KylinMountain .
I removed the
$OPENKB_DIR/and then reinstalled openkb. I made theconfig.yamlexactly what you show above (kept.envthe same) and still get this error:I wonder if the
drop_paramssettings is not taking for some reason.jeffm commentedon Jul 4, 2026
I had the same problem on Windows. Based on my futzing, it does not appear that drop_params has any effect. In the end, I had CODEX modify the codebase to create an exception for lm_studio, and, combined with a vastly increased timeout and context window, I got it to work on a relatively small document (~21kb of pure text) in a reasonable period of time.
arunov commentedon Jul 21, 2026
My workaround was to use my LM Studio model as
anthropic/gemma-4-31b-qat.