Summary

Severity: Medium-High
Area: Reliability / providers
Affected files: agentkit/llm/openai.py:141-143, agentkit/llm/gemini.py:159-197, agentkit/core/agent.py:298-315
Suggested branch: fix/provider-refusal-handling
Suggested PR title: fix(llm): surface refusals, blocks and empty responses as failures

This is finding AUDIT-07 of 18 from a consolidated audit of agentkit/. It is deliberately self-contained so it can be fixed on its own branch with its own PR.

Related findings: Related: AUDIT-08 (status reporting).

AUDIT-07 — Medium-High — Refusals, blocks and empty responses become SUCCEEDED

Files

• agentkit/llm/openai.py:141-143
• agentkit/llm/gemini.py:159-197
• agentkit/core/agent.py:298-315

Current code — agentkit/llm/openai.py:141-143

            choice = raw_response.choices[0]
            choice_message = choice.message
            text = choice_message.content or ""

Current code — agentkit/core/agent.py:298-315

                        else:
                            # Final natural language answer reached
                            duration_ms = max(0, int((time.perf_counter() - start_time) * 1000))
                            logger.info("Agent run succeeded in %d ms", duration_ms)
                            messages.append(Message.assistant(response.text))
                            return await _finalize_and_return(
                                RunResult(
                                    run_id=run_id,
                                    session_id=session_id,
                                    goal=goal,
                                    status=RunStatus.SUCCEEDED,
                                    final_answer=response.text,
                                    ...

Why this is a bug

Nothing in the codebase inspects choice.finish_reason (content_filter, length, …), message.refusal, response.prompt_feedback.block_reason, candidate.finish_reason (SAFETY, RECITATION, MAX_TOKENS) or an empty completion. The agent only branches on "are there tool calls?", so a blocked or empty answer is reported as a successful run.

Reproduced

Gemini: prompt_feedback.block_reason="SAFETY", empty parts
  -> LLMResponse(text="", tool_calls=[])
  -> RunResult(status=SUCCEEDED, final_answer="")

Pass cases

• A Gemini response with prompt_feedback.block_reason set, or a candidate with finish_reason in {SAFETY, RECITATION, PROHIBITED_CONTENT, BLOCKLIST, SPII}, results in an explicit failure (not SUCCEEDED) with a failure_reason naming the block reason.
• An OpenAI response with finish_reason == "content_filter" or a non-empty message.refusal results in an explicit failure with the refusal surfaced.
• An empty completion with no tool calls and no text is a failure (e.g. RunStatus.FAILED with failure_reason="empty provider response"), never SUCCEEDED with an empty answer.
• A successful run always has a non-empty final_answer.
• RunStatus gains (or reuses) an explicit status for provider refusals, and POST /runs returns it in status with a populated failure_reason.
• Regression tests for each provider path, asserting status != "succeeded".

Definition of done (per AGENTS.md)

• New/updated unit and integration tests covering the pass cases above, all passing (pytest).
• ruff check ., ruff format --check . and mypy agentkit clean.
• Docstrings updated where behaviour changed; docs/PROGRESS.md updated (status, PR link, commit count, notes).
• A new ADR in docs/DECISIONS.md if an architectural decision was made.
• Branch pushed and PR opened (never commit directly to main).

Baseline at the time of writing: pytest is green on main (all tests pass; tests/real/* skipped unless AGENTKIT_REAL_INFRA=1).