Skip to content

fix: stop blank exception messages from masking turn failures - #685

Merged
delete-cloud merged 1 commit into
mainfrom
fix/empty-turn-error-masking
Jul 31, 2026
Merged

delete-cloud merged 1 commit into
mainfrom
fix/empty-turn-error-masking

Conversation

@delete-cloud

@delete-cloud delete-cloud commented Jul 31, 2026 •

Copy link
Copy Markdown
Owner

Root cause

Observed live (o6n): a codex turn failed and the session showed failure_details: "Agent turn failed: error must be non-empty", with pod logs showing ValueError: error must be non-empty from runtime_store.py __post_init__ during RuntimeTurnFinalizer.complete → finish_run → update_agent_run. The real failure (codex OAuth refresh 403) was completely masked.

Traced path:

  1. An exception escaping a pipeline stage is wrapped by agentkit as PipelineError(str(exc), stage=...) (src/agentkit/runtime/pipeline.py:535). If the original exception's str() is empty, the wrapper's str() is empty too.
  2. PipelineAdapter.run_turn built the outcome with bare error=str(exc) (src/coding_agent/adapter/pipeline.py:236) — no fallback, so TurnOutcome.error == "". Same pattern existed in runs/lifecycle.py (handle_fatal/handle_generic) and tools/subagent.py.
  3. RuntimeTurnFinalizer.complete → finish_run → update_agent_run, where _require_non_empty("error", ...) raised ValueError("error must be non-empty"), which then became the user-visible failure — the original error was lost entirely.

Provider-agnostic? Yes. Verified empirically that an httpx.HTTPStatusError (the codex 403) survives the full provider → pipeline → adapter path with its message intact; the bug bites for any exception whose str() is empty/whitespace (bare RuntimeError(), wrappers re-raised without a message, etc.), regardless of provider. The empty string the pod wrote must have come from such an intermediate exception; the code had no guard against it anywhere along the path.

Fix

  • Primary — exception_error_message() in src/coding_agent/adapter/types.py: returns str(exc) when non-blank, otherwise falls back to the exception type name, unwrapping a chained __cause__ (PipelineError: RuntimeError). Used at every TurnOutcome.error / run-error construction site: adapter/pipeline.py, runs/lifecycle.py (handle_fatal, handle_generic, incl. last_failure_details), tools/subagent.py. No empty error can reach persistence again, and the user-visible error is always informative.
  • Defense-in-depth — _normalize_optional_error() in src/coding_agent/stores/runtime_store.py: update_agent_run (JSONL/SQLite/PG) normalizes blank error strings to None at the write boundary, so a blank error can never trip the AgentRunRecord non-empty invariant on write or read-back.
  • Postmortem — added PM-0026 pattern with release checks.

Tests

  • test_adapter_types.py: unit tests pinning the fallback (empty, whitespace, verbatim message, chained cause, httpx 403 message preserved).
  • test_pipeline_adapter.py: provider stream raising RuntimeError() (empty str) → outcome.error == "PipelineError: RuntimeError".
  • test_run_lifecycle.py: error handler with empty-message exception records non-empty error; end-to-end repro — empty-message exception outcome → finalizer backed by real SQLiteRuntimeStore → run finishes failed with error == "RuntimeError", no ValueError.
  • test_sqlite_runtime_store.py / test_jsonl_runtime_store.py: blank error normalizes to None, no raise.

Verification

  • uv run pytest tests/coding_agent -q: 1620 passed, 20 xfailed
  • Focused PM release checks (adapter + lifecycle + store tests): all pass
  • uv run pytest tests/agents tests/coding_agent/subagents: pass
  • ruff check / ruff format --check on all changed files: clean

Not merged — review welcome.

Summary by CodeRabbit

  • Bug Fixes

    • Prevented blank exception messages from being saved as run errors.
    • Improved error details by falling back to the exception type or underlying cause when no message is available.
    • Ensured failed runs complete and persist correctly without storage validation errors.
  • Documentation

    • Added a postmortem documenting the issue, affected workflows, resolution, and prevention checks.

An exception with an empty str() (e.g. bare RuntimeError, or a
PipelineError(str(exc)) wrapper around one) produced TurnOutcome.error ==
"". The runtime store's non-empty error invariant then raised
ValueError("error must be non-empty") during update_agent_run, masking
the real provider failure (observed live: codex OAuth refresh 403
surfaced only as the masking ValueError).

- adapter/types.py: add exception_error_message() falling back to the
  exception type name (plus chained cause) when str(exc) is blank
- adapter/pipeline.py, runs/lifecycle.py, tools/subagent.py: use it at
  every TurnOutcome.error / run-error construction site
- stores/runtime_store.py: _normalize_optional_error() maps blank error
  strings to None at the update_agent_run write boundary (JSONL/SQLite/PG)
  as defense-in-depth
- postmortem: add PM-0026 pattern

Provider-agnostic: any exception with empty str() hit this, not just codex.
@coderabbitai

coderabbitai Bot commented Jul 31, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The change adds exception-message normalization, applies it to adapter and lifecycle error handling, normalizes blank errors before runtime-store persistence, adds regression tests, and documents the issue in postmortem PM-0026.

Changes

Exception error normalization

Layer / File(s) Summary
Exception message normalization and error outcomes
coding-agent/src/coding_agent/adapter/types.py, coding-agent/src/coding_agent/adapter/pipeline.py, coding-agent/src/coding_agent/tools/subagent.py, coding-agent/tests/coding_agent/test_adapter_types.py, coding-agent/tests/coding_agent/test_pipeline_adapter.py
exception_error_message preserves non-blank messages, follows explicit causes, and falls back to exception type names. Pipeline and subagent error outcomes use the helper. Tests cover these cases.

Lifecycle and runtime-store persistence

Layer / File(s) Summary
Lifecycle and runtime-store persistence
coding-agent/src/coding_agent/runs/lifecycle.py, coding-agent/src/coding_agent/stores/runtime_store.py, coding-agent/tests/coding_agent/test_run_lifecycle.py, coding-agent/tests/coding_agent/test_jsonl_runtime_store.py, coding-agent/tests/coding_agent/test_sqlite_runtime_store.py
Lifecycle error details use normalized messages. JSONL, SQLite, and PostgreSQL stores convert blank errors to None. Tests cover direct store updates and SQLite-backed run finalization.
Postmortem records
coding-agent/postmortem/index.yaml, coding-agent/postmortem/patterns/PM-0026-never-persist-blank-exception-messages-as-run-errors.md
Postmortem PM-0026 records the blank exception-message persistence failure, affected code paths, fixes, and release checks.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 27.59% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the primary fix for blank exception messages masking turn failures.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/empty-turn-error-masking

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@coding-agent/src/coding_agent/adapter/types.py`:
- Around line 46-48: Update exception_error_message to track visited exception
identities while traversing explicit __cause__ links, stopping or falling back
to a non-recursive message when a cycle is detected, including self-referential
and two-exception cycles. Add regression tests covering both cycle shapes and
ensure normal chained-cause formatting remains unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f6d7e85f-cd3b-48d1-9e07-4ec3a4f2c8cb

📥 Commits

Reviewing files that changed from the base of the PR and between 180d03d and 783cbda.

📒 Files selected for processing (12)
  • coding-agent/postmortem/index.yaml
  • coding-agent/postmortem/patterns/PM-0026-never-persist-blank-exception-messages-as-run-errors.md
  • coding-agent/src/coding_agent/adapter/pipeline.py
  • coding-agent/src/coding_agent/adapter/types.py
  • coding-agent/src/coding_agent/runs/lifecycle.py
  • coding-agent/src/coding_agent/stores/runtime_store.py
  • coding-agent/src/coding_agent/tools/subagent.py
  • coding-agent/tests/coding_agent/test_adapter_types.py
  • coding-agent/tests/coding_agent/test_jsonl_runtime_store.py
  • coding-agent/tests/coding_agent/test_pipeline_adapter.py
  • coding-agent/tests/coding_agent/test_run_lifecycle.py
  • coding-agent/tests/coding_agent/test_sqlite_runtime_store.py

Comment thread coding-agent/src/coding_agent/adapter/types.py
@delete-cloud
delete-cloud merged commit a847e6e into main Jul 31, 2026
2 checks passed
@delete-cloud
delete-cloud deleted the fix/empty-turn-error-masking branch July 31, 2026 18:11
delete-cloud added a commit that referenced this pull request Jul 31, 2026
…686)

* docs(spec): webui codex oauth device-flow login design

* docs(spec): multi-account codex oauth design

* docs(spec): disconnect deletes local record only, no remote revoke

* feat(webui): codex OAuth device flow + multi-account support

- SettingsPanel: new CodexAccountsCard listing connected accounts
  (label/email/plan badge) with confirm+disconnect, add-account device
  flow (label input, verification link, click-to-copy user code, 3s
  polling until authorized/error/expired, retry), in-flight flow
  cancel, pending-flow recovery on reload, and a not-supported note
  when the server lacks the /oauth/* endpoints (404).
- Header: append connected codex:<label> accounts to the provider
  dropdown (static list on fetch failure); codex providers get
  gpt-5.5/5.4/5.2 datalist presets instead of a live model fetch.
- lib/api.ts: 6 oauth client methods with defensive payload parsing;
  lib/types.ts: CodexFlow/CodexFlowStart/OAuthAccount types.

* feat(server): multi-account codex OAuth device flow for webui

- oauth/store.py: default_auth_path() with CODING_AGENT_OAUTH_AUTH_PATH
  env override; OAuthStore() and oauth CLI resolve through it
- server/oauth_flows.py: in-memory flow registry + background poll
  tasks (pending/authorized/error/expired/cancelled, 10min TTL);
  label from explicit request or id_token (email first, account-id
  short code, -2/-3 suffix on conflict); local exception_error_message
  helper (stands in for unmerged #685)
- server/http_server.py: POST /oauth/codex/start, GET
  /oauth/codex/flows{,/{id}}, POST /oauth/codex/flows/{id}/cancel,
  GET /oauth/accounts, DELETE /oauth/accounts/{key} (local delete,
  no remote revoke); 400 at session create for unconnected codex:<label>
- plugins/llm_provider.py: codex:<label> routing — StoreBackedCodex-
  TokenSource parameterized by provider key, refresh writes back same key
- server/schemas.py: CreateSessionRequest.provider accepts ProviderName
  literals plus codex:<label> (^[a-z0-9][a-z0-9-]{0,30}$)
- tests: tests/ui/test_oauth_flows.py (mocked CodexOAuthClient), plugin
  codex:* routing tests, store env-path tests
- docs: web-client-api-contract.md gains the oauth endpoints section

* fix(webui): stop polling lost codex oauth flows

Review P2: the flow poll catch swallowed every error, so a flow the
server forgot (restart cleared the in-memory registry, TTL prune) or a
failing auth path was retried every 3s until page close. Now a 404
marks the flow expired with a server-lost note, and other errors give
up after 5 consecutive failures. Expired rows render flow.error when
present.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant