Skip to content

fix(deps): pin patched LiteLLM and repair integration CI - #460

Merged
Hecate0821 merged 4 commits into
mainfrom
codex/patched-eval-dependencies-20261012
Oct 13, 2026
Merged

Hecate0821 merged 4 commits into
mainfrom
codex/patched-eval-dependencies-20261012

Conversation

@Hecate0821

@Hecate0821 Hecate0821 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

The existing LiteLLM pin blocks consumers that require the security fixes for GHSA-3cv6-jpf6-8222. Pin tested LiteLLM 1.105.0 and require tested MCP 1.30.0 on its supported v1 API; fresh installs otherwise select MCP v2, which removed streamablehttp_client.

Repair the integration failures exposed by the upgrade:

  • Use E2B's v2 Sandbox.create() and public files API so sandbox access tokens reach file/command requests. Preserve older code-interpreter SDK support and sandbox cleanup. Pin the tested dev SDK to 2.34.0; newer releases force OpenTelemetry changes incompatible with the currently locked optional adapters. Update HTTP/2 dependencies to patched releases.
  • Schedule each test file on one xdist worker so retry evaluations and their mock-verification tests share state. Remove the verification's previous silent success when no calls were recorded.
  • Declare missing credentials as an explicit prerequisite for the existing Klavis live tests. Use the model already exercised by tracing CI, configurable with EP_KLAVIS_MODEL. Add always-running judging, authorization, dataset-adapter, and sandbox-compatibility tests. When live Klavis credentials are present, failed rollouts and service errors fail the test rather than returning a passing zero-score evaluation.

Validation:

  • 24 focused tests pass on each of Python 3.10, 3.11, and 3.12 with two xdist workers.
  • All three previously failing E2B live tests pass with SDK 2.34.0: Python, JavaScript, and fractional scoring.
  • Broad Python 3.12 run: shards 3 and 4 pass (421 and 388 tests); shard 1 passes its 146 locally available checks, and the remaining local failures require unavailable OpenAI credentials or Docker access. The three Pydantic metadata tests also pass with a dummy key (no inference requests). Those live-service jobs remain enabled in GitHub CI.
  • Frozen all-extras installation, dependency compatibility, Ruff, lock consistency, shell syntax, diff checks, and wheel build pass.
  • Prior dependency compatibility validation: 143 unit tests per Python 3.10–3.12 and downstream cookbook CPU suite of 5,953 passed, 107 skipped, and 116 expected failures.

Klavis live tests require their existing environment variables when run manually. Core CI runs their offline checks and explicitly skips live Klavis evaluations without credentials. This PR adds no secret mappings or credential examples; the workflow and .env.example match main. E2B and Fireworks tracing live tests remain enabled.

Local validation of credential-wiring removal: 6 Klavis offline tests passed and 2 live tests explicitly skipped with both Klavis variables unset; Ruff and diff checks passed. Final CI: 17/17 checks passed on cleanup commit 4bc87f2ea2a1984d61a2b6ae7c053d27c2954ccd, including all 12 Python 3.10–3.12 core-test shards. Python CI and live Fireworks tracing both passed.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8502738. Configure here.

Comment thread pyproject.toml Outdated
@Hecate0821 Hecate0821 changed the title fix(deps): pin patched litellm and preserve mcp v1 APIs fix(deps): pin patched LiteLLM and repair integration CI Oct 13, 2026
@Hecate0821
Hecate0821 merged commit 9b642d9 into main Oct 13, 2026
17 checks passed
@Hecate0821
Hecate0821 deleted the codex/patched-eval-dependencies-20261012 branch October 13, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants