Skip to content

fix(llm): type agenerate_streaming as directly async-iterable - #413

Open
lplalbert wants to merge 1 commit into
apache:mainfrom
lplalbert:fix/base-llm-streaming-contract
Open

lplalbert wants to merge 1 commit into
apache:mainfrom
lplalbert:fix/base-llm-streaming-contract

Conversation

@lplalbert

Copy link
Copy Markdown
Contributor

Refs #107 (Tiny Task: fix TODO/FIXME in LLM code)

What

BaseLLM.agenerate_streaming was declared async def ... -> AsyncGenerator[str, None] with
a docstring-only body, so type checkers resolve the call as
Coroutine[..., AsyncGenerator[...]] — exactly the FIXME in answer_synthesize.py
("Expected type 'AsyncIterable', got 'Coroutine[Any, Any, AsyncGenerator[str, None]]'
instead"). Every implementation (hugegraph-llm-sdk openai/ollama/litellm providers) is a
true async generator function, and every call site uses plain async for. The declaration
was the only thing disagreeing with the runtime contract.

Changes

  • BaseLLM.agenerate_streaming: async def → def, documented as "implement as an async
    generator function; the call result is directly async-iterable and must not be awaited".
  • BaseLLM.num_tokens_from_string: return annotation -> str → -> int (implementations
    return token counts).
  • Removed the now-stale FIXME in answer_synthesize.py.
  • Added src/tests/models/llms/test_base_llm_contract.py pinning both contracts.

Validation (actually run; offline only)

  • red→green (demonstrated via stash): 2 failed on the old declaration → 2 passed after
  • SKIP_EXTERNAL_SERVICES=true uv run pytest src/tests/models/ src/tests/operators/llm_op/
    → 231 passed, 4 skipped
  • ruff format --check / ruff check → clean
  • ty check → the FIXME's Coroutine-vs-AsyncIterable diagnostic is gone (stash-compared;
    remaining answer_synthesize hits are pre-existing str | None prompt noise)
  • Real-provider verification: none (offline tests only)

Refs #107

🤖 Generated with Claude Code

BaseLLM declared 'async def agenerate_streaming(...) -> AsyncGenerator[...]',
which type checkers resolve as a coroutine of a generator. Every
implementation is an async generator function and every call site (e.g.
answer_synthesize) iterates the call result with plain 'async for', so the
declaration contradicted the implemented contract and produced the
"Expected 'AsyncIterable', got 'Coroutine[...]'" FIXME.

- declare agenerate_streaming as 'def ... -> AsyncGenerator[str, None]'
  (implement as an async generator function; do not await the call)
- align num_tokens_from_string with the implementations, which return int
- drop the now-resolved FIXME at the call site
- add contract tests that fail on the old declarations

Co-Authored-By: Claude Code <noreply@anthropic.com>
@github-actions github-actions Bot added the llm label Nov 9, 2026

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 9/10. Summary: The BaseLLM declarations for agenerate_streaming and num_tokens_from_string match every provider and every async for caller, ty drops the override and Coroutine diagnostics with none added, and the focused models/llms and llm_op tests pass.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Score: 9.0/10 · Confidence: high

Blocking: no

Summary

The abstract signatures match the async generator and token-count contracts used by the providers and caller.

Evidence

Exact-head provider and caller inspection; applicable LLM and quality CI succeeded.

Validation notes

No live provider integration or local test run; the change does not alter provider execution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants