Skip to content

Proxied file uploads bypass the LiteLLM Proxy and are sent to api.openai.com #6722

Closed

Description

@tranminhquang4421

Describe the bug

When a model is served through a LiteLLM Proxy, ADK uploads file content (PDFs
and the other types in _SUPPORTED_FILE_CONTENT_MIME_TYPES) to an endpoint
resolved independently of the completion call. The completion goes to the
proxy. The upload does not.

For azure this fails locally with a missing credentials error. For openai
it does not fail at all. get_openai_credentials defaults api_base to
https://api.openai.com/v1, so if an OPENAI_API_KEY is present in the
environment the upload succeeds against the public OpenAI API. File content
meant for a self hosted or vendored proxy leaves it, and the developer's own
OpenAI key is what sends it.

That is the case I care about here. A proxy is usually deployed precisely
because the data is not supposed to reach the provider directly.

To Reproduce

.env, which is a plausible developer setup: proxy credentials plus a personal
OpenAI key left over from other work.

LITELLM_PROXY_API_BASE=http://127.0.0.1:8877
LITELLM_PROXY_API_KEY=sk-proxy-secret
OPENAI_API_KEY=sk-personal-developer-key

repro.py. The local server stands in for the proxy so we can see what it
receives, and the httpx patch records anything that leaves localhost instead of
sending it.

import asyncio, json, threading
from http.server import BaseHTTPRequestHandler, HTTPServer

from dotenv import load_dotenv
load_dotenv()

MODEL = "litellm_proxy/openai/gpt-4o"

PROXY_RECEIVED = []


class _Proxy(BaseHTTPRequestHandler):

  def do_POST(self):
    n = int(self.headers.get("Content-Length") or 0)
    self.rfile.read(n)
    PROXY_RECEIVED.append(self.path)
    if "files" in self.path:
      payload = {"id": "file-via-proxy", "object": "file", "bytes": n,
                 "created_at": 0, "filename": "f.pdf",
                 "purpose": "assistants", "status": "processed"}
    else:
      payload = {"id": "chatcmpl-1", "object": "chat.completion", "created": 0,
                 "model": "m", "choices": [{"index": 0, "finish_reason": "stop",
                 "message": {"role": "assistant", "content": "ok"}}],
                 "usage": {"prompt_tokens": 1, "completion_tokens": 1,
                           "total_tokens": 2}}
    body = json.dumps(payload).encode()
    self.send_response(200)
    self.send_header("Content-Type", "application/json")
    self.send_header("Content-Length", str(len(body)))
    self.end_headers()
    self.wfile.write(body)

  def log_message(self, *a):
    pass


HTTPServer.allow_reuse_address = True
threading.Thread(
    target=HTTPServer(("127.0.0.1", 8877), _Proxy).serve_forever,
    daemon=True).start()

import httpx
LEFT_THE_PROXY = []
_orig_send = httpx.AsyncClient.send


async def _spy(self, request, *args, **kwargs):
  if request.url.host not in ("127.0.0.1", "localhost"):
    LEFT_THE_PROXY.append(str(request.url))
    raise RuntimeError(f"blocked egress to {request.url}")
  return await _orig_send(self, request, *args, **kwargs)


httpx.AsyncClient.send = _spy

from google.adk.models.lite_llm import LiteLlm
from google.adk.models.llm_request import LlmRequest
from google.genai import types

llm = LiteLlm(model=MODEL)
request = LlmRequest(model=MODEL, contents=[types.Content(role="user", parts=[
    types.Part(text="summarize this"),
    types.Part.from_bytes(data=b"%PDF-1.4 confidential",
                          mime_type="application/pdf"),
])])


async def main():
  try:
    async for _ in llm.generate_content_async(request, stream=False):
      pass
  except Exception as e:
    print(f"raised         : {type(e).__name__}: {str(e)[:100]}")
  print(f"proxy received : {PROXY_RECEIVED or []}")
  print(f"left the proxy : {LEFT_THE_PROXY or []}")


asyncio.run(main())

Output on main at 370027a:

raised         : APIConnectionError: Connection error.
proxy received : []
left the proxy : ['https://api.openai.com/v1/files', ...]

The proxy receives nothing. The upload goes to OpenAI. It only surfaces as a
connection error because the script blocks the request. Without the block it
succeeds, using OPENAI_API_KEY.

Expected behavior

The upload goes wherever the completion goes. If the proxy endpoint cannot be
determined, the upload fails instead of falling back to a provider default.

Three configurations reach this

All three were run with the script above, changing only .env and MODEL.

# Configuration Model proxy received left the proxy
1 LITELLM_PROXY_API_BASE + LITELLM_PROXY_API_KEY litellm_proxy/openai/gpt-4o [] api.openai.com/v1/files
2 USE_LITELLM_PROXY=true openai/gpt-4o [] api.openai.com/v1/files
3 neither, only OPENAI_API_KEY litellm_proxy/openai/gpt-4o [] api.openai.com/v1/files

Case 2 is worth spelling out. LiteLLM supports routing unprefixed models
through the proxy with USE_LITELLM_PROXY, and its own docstring says that
flag exists for Google ADK (BerriAI/litellm#10559). A model string of
openai/gpt-4o gives no hint that it is proxied, so anything matching on the
litellm_proxy/ prefix alone will miss it.

For reference, LiteLLM resolves the proxy endpoint for the completion call in
litellm/llms/litellm_proxy/chat/transformation.py:

api_base = api_base or get_secret_str("LITELLM_PROXY_API_BASE")
dynamic_api_key = api_key or get_secret_str("LITELLM_PROXY_API_KEY")

litellm.acreate_file does not consult those, so unless ADK forwards an
endpoint the two halves of the request diverge.

Relation to #6538 and #6578

#6578 (merged as 602e58d) fixed the payload shape for proxied models. Nested
litellm_proxy/azure/... identifiers now resolve to the underlying provider,
so the upload path runs instead of emitting an inline file_data block. That
part is correct and nothing here contests it.

The same commit also set custom_llm_provider="openai" for proxied uploads.
That is the line that turns cases 1 and 3 from a local failure into a request
to api.openai.com, because openai is the one provider whose credential
resolution defaults to the public API:

# litellm/llms/openai/common_utils.py
resolved_api_base = (
    api_base or litellm.api_base or os.getenv("OPENAI_BASE_URL")
    or os.getenv("OPENAI_API_BASE") or "https://api.openai.com/v1"
)

On main, _get_content takes only parts, provider and model. There is
no parameter through which a proxy endpoint could arrive, and ADK never sets
litellm.api_base. So there is no configuration in which that override reaches
a proxy. It resolves to the public endpoint every time.

Environment

  • ADK main at 370027a. The behavior starts at 602e58d.
  • litellm 1.85.7, and also reproduced on 1.84.0, the floor of the
    litellm>=1.84 constraint in pyproject.toml. In both,
    OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS is {openai, hosted_vllm} and
    create_file has no litellm_proxy branch, so routing the upload as
    litellm_proxy is not an option either.
  • Python 3.11, macOS.

Additional context

I have a fix and will open a PR referencing this issue. One part of it is a
behavior change I would rather have decided explicitly than slipped into a
diff: when no proxy endpoint can be determined, the upload raises instead of
quietly using the provider default. For anyone who has OPENAI_API_KEY set
that converts a currently "working" upload into an error. I think an explicit
error is better than a silent misroute, but it is a trade off and it is your
call.

Activity

  1. surajksharma07 commented on Aug 14, 2026

    @surajksharma07
    Collaborator

    @tranminhquang4421 The hardcoded openai fallback in the upload path is what's sending it to api.openai.com.
    #6723 lines up with the root cause and passed our own before/after check locally so it looks like the right fix.

    Since you flagged the Azure-backed proxy case as unverified against a live endpoint could you try that PR against your actual setup and confirm it holds before we look at merging?
    Would rather have your real-world config confirm it than assume from our synthetic repro alone.

  2. added
    request clarification[Status] The maintainer need clarification or more information from the author
    models[Component] This issue is related to model support
    on Aug 14, 2026
  3. tranminhquang4421 commented on Aug 14, 2026

    @tranminhquang4421
    Author

    Thanks for checking it locally, glad the before/after lines up on your side too.

    On the live Azure verification: I don't have an Azure backed proxy deployment.
    That's what I meant by flagging it as unverified in #6723, and the same in
    #6578 earlier. It's a gap in what I can test rather than a test I skipped, so
    it isn't something I can close myself.

    Two things that might help you decide how much that blocks merging.

    The reported leak doesn't depend on Azure. Two of the three paths in #6722
    resolve to openai and never enter Azure code at all:

    # Model Provider ADK resolves
    1 litellm_proxy/openai/gpt-4o openai
    2 openai/gpt-4o with USE_LITELLM_PROXY=true openai
    3 litellm_proxy/openai/gpt-4o, no proxy endpoint set openai

    The root cause is get_openai_credentials defaulting api_base to
    https://api.openai.com/v1. Nothing in that chain is Azure specific, which is
    why your before/after run reproduced it without an Azure deployment.

    On main, an Azure backed proxy model doesn't reach Azure either. The
    custom_llm_provider="openai" override replaces the provider before the
    upload, so the azure branch in litellm/files/main.py never runs:

    model                 : litellm_proxy/azure/my-deployment
    provider ADK resolves : azure
    custom_llm_provider   : openai      <- what acreate_file actually receives
    

    Same repro, same .env, only the model swapped:

    proxy received : []
    left the proxy : ['https://api.openai.com/v1/files', ...]
    

    So on current main the Azure case fails in the same place and for the same
    reason as the OpenAI one, and a live Azure endpoint would mostly confirm that
    the upload goes to api.openai.com instead of to Azure.

    That said, I don't want to argue the Azure test is pointless, because after
    #6723 it stops being the same path.
    With the fix the provider is no longer
    overridden, so the azure branch runs for the first time:

    custom_llm_provider : azure
    api_base            : http://127.0.0.1:8877
    

    That branch goes through get_azure_credentials and puts an api-version on
    the URL (/openai/files?api-version=...), which the openai branch never
    does. My stub accepts that happily. A real proxy in front of Azure is where
    you'd find out whether the forwarded api_base and the resolved api-version
    actually line up with what the deployment expects, and whether the returned
    file_id is then accepted by the completion. That is a genuine gap and it is
    exactly the part I can't cover from here.

    So the request is fair, I just can't be the one to satisfy it. Two ways to get
    it if you want it before merging:

    Happy to dig into whatever either turns up, including the api-version
    handling if it doesn't match a real deployment.

    One housekeeping note since you tagged this account: I opened this issue from
    @tranminhquang4421 and #6723 from @vietnamesekid. Same person, two
    accounts, and @vietnamesekid is the one that signed the CLA on #6578. Worth
    knowing in case the CLA check on #6723 looks inconsistent with this thread.

  4. MilkyWay008 commented on Aug 14, 2026

    @MilkyWay008

    That's a nasty one...... completion goes to the proxy but the upload resolves its own endpoint, the kind of thing that slips through testing. If you need a stopgap before the PR lands: get_openai_credentials checks OPENAI_BASE_URL then OPENAI_API_BASE before defaulting to api.openai.com, so pointing either at your proxy routes the upload through it too. And if you'd rather it never touch that key, unset OPENAI_API_KEY and it fails loudly instead of silently exfiltrating. Nice repro btw, the egress-blocking spy makes it obvious.

  5. surajksharma07 commented on Aug 17, 2026

    @surajksharma07
    Collaborator

    Good catch on 205b9f7. ADK's own USE_LITELLM_PROXY check was accepting "1" but LiteLLM's get_secret_bool only reads "true"/"false" so =1 would've had the upload going through the proxy while the completion went direct to the provider. Same class of split this whole PR is closing just the opposite direction.

    On the Azure verification makes sense you can't close that gap yourself given the provider override only stops happening after this fix. Reaching out to @ross-p on #6538 since they've got a live Azure-backed proxy rather than block on it here. Otherwise this is looking close just need varunbiluri's sign-off on 56c2519 and we can move toward merge.

  6. added a commit that references this issue on Oct 5, 2026
    24f2a2e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

models[Component] This issue is related to model supportrequest clarification[Status] The maintainer need clarification or more information from the author

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions