Repository navigation
Description
Describe the bug
When a model is served through a LiteLLM Proxy, ADK uploads file content (PDFs
and the other types in _SUPPORTED_FILE_CONTENT_MIME_TYPES) to an endpoint
resolved independently of the completion call. The completion goes to the
proxy. The upload does not.
For azure this fails locally with a missing credentials error. For openai
it does not fail at all. get_openai_credentials defaults api_base to
https://api.openai.com/v1, so if an OPENAI_API_KEY is present in the
environment the upload succeeds against the public OpenAI API. File content
meant for a self hosted or vendored proxy leaves it, and the developer's own
OpenAI key is what sends it.
That is the case I care about here. A proxy is usually deployed precisely
because the data is not supposed to reach the provider directly.
To Reproduce
.env, which is a plausible developer setup: proxy credentials plus a personal
OpenAI key left over from other work.
LITELLM_PROXY_API_BASE=http://127.0.0.1:8877
LITELLM_PROXY_API_KEY=sk-proxy-secret
OPENAI_API_KEY=sk-personal-developer-key
repro.py. The local server stands in for the proxy so we can see what it
receives, and the httpx patch records anything that leaves localhost instead of
sending it.
import asyncio, json, threading
from http.server import BaseHTTPRequestHandler, HTTPServer
from dotenv import load_dotenv
load_dotenv()
MODEL = "litellm_proxy/openai/gpt-4o"
PROXY_RECEIVED = []
class _Proxy(BaseHTTPRequestHandler):
def do_POST(self):
n = int(self.headers.get("Content-Length") or 0)
self.rfile.read(n)
PROXY_RECEIVED.append(self.path)
if "files" in self.path:
payload = {"id": "file-via-proxy", "object": "file", "bytes": n,
"created_at": 0, "filename": "f.pdf",
"purpose": "assistants", "status": "processed"}
else:
payload = {"id": "chatcmpl-1", "object": "chat.completion", "created": 0,
"model": "m", "choices": [{"index": 0, "finish_reason": "stop",
"message": {"role": "assistant", "content": "ok"}}],
"usage": {"prompt_tokens": 1, "completion_tokens": 1,
"total_tokens": 2}}
body = json.dumps(payload).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *a):
pass
HTTPServer.allow_reuse_address = True
threading.Thread(
target=HTTPServer(("127.0.0.1", 8877), _Proxy).serve_forever,
daemon=True).start()
import httpx
LEFT_THE_PROXY = []
_orig_send = httpx.AsyncClient.send
async def _spy(self, request, *args, **kwargs):
if request.url.host not in ("127.0.0.1", "localhost"):
LEFT_THE_PROXY.append(str(request.url))
raise RuntimeError(f"blocked egress to {request.url}")
return await _orig_send(self, request, *args, **kwargs)
httpx.AsyncClient.send = _spy
from google.adk.models.lite_llm import LiteLlm
from google.adk.models.llm_request import LlmRequest
from google.genai import types
llm = LiteLlm(model=MODEL)
request = LlmRequest(model=MODEL, contents=[types.Content(role="user", parts=[
types.Part(text="summarize this"),
types.Part.from_bytes(data=b"%PDF-1.4 confidential",
mime_type="application/pdf"),
])])
async def main():
try:
async for _ in llm.generate_content_async(request, stream=False):
pass
except Exception as e:
print(f"raised : {type(e).__name__}: {str(e)[:100]}")
print(f"proxy received : {PROXY_RECEIVED or []}")
print(f"left the proxy : {LEFT_THE_PROXY or []}")
asyncio.run(main())Output on main at 370027a:
raised : APIConnectionError: Connection error.
proxy received : []
left the proxy : ['https://api.openai.com/v1/files', ...]
The proxy receives nothing. The upload goes to OpenAI. It only surfaces as a
connection error because the script blocks the request. Without the block it
succeeds, using OPENAI_API_KEY.
Expected behavior
The upload goes wherever the completion goes. If the proxy endpoint cannot be
determined, the upload fails instead of falling back to a provider default.
Three configurations reach this
All three were run with the script above, changing only .env and MODEL.
| # | Configuration | Model | proxy received |
left the proxy |
|---|---|---|---|---|
| 1 | LITELLM_PROXY_API_BASE + LITELLM_PROXY_API_KEY |
litellm_proxy/openai/gpt-4o |
[] |
api.openai.com/v1/files |
| 2 | USE_LITELLM_PROXY=true |
openai/gpt-4o |
[] |
api.openai.com/v1/files |
| 3 | neither, only OPENAI_API_KEY |
litellm_proxy/openai/gpt-4o |
[] |
api.openai.com/v1/files |
Case 2 is worth spelling out. LiteLLM supports routing unprefixed models
through the proxy with USE_LITELLM_PROXY, and its own docstring says that
flag exists for Google ADK (BerriAI/litellm#10559). A model string of
openai/gpt-4o gives no hint that it is proxied, so anything matching on the
litellm_proxy/ prefix alone will miss it.
For reference, LiteLLM resolves the proxy endpoint for the completion call in
litellm/llms/litellm_proxy/chat/transformation.py:
api_base = api_base or get_secret_str("LITELLM_PROXY_API_BASE")
dynamic_api_key = api_key or get_secret_str("LITELLM_PROXY_API_KEY")litellm.acreate_file does not consult those, so unless ADK forwards an
endpoint the two halves of the request diverge.
#6578 (merged as 602e58d) fixed the payload shape for proxied models. Nested
litellm_proxy/azure/... identifiers now resolve to the underlying provider,
so the upload path runs instead of emitting an inline file_data block. That
part is correct and nothing here contests it.
The same commit also set custom_llm_provider="openai" for proxied uploads.
That is the line that turns cases 1 and 3 from a local failure into a request
to api.openai.com, because openai is the one provider whose credential
resolution defaults to the public API:
# litellm/llms/openai/common_utils.py
resolved_api_base = (
api_base or litellm.api_base or os.getenv("OPENAI_BASE_URL")
or os.getenv("OPENAI_API_BASE") or "https://api.openai.com/v1"
)On main, _get_content takes only parts, provider and model. There is
no parameter through which a proxy endpoint could arrive, and ADK never sets
litellm.api_base. So there is no configuration in which that override reaches
a proxy. It resolves to the public endpoint every time.
Environment
- ADK
mainat 370027a. The behavior starts at 602e58d. - litellm 1.85.7, and also reproduced on 1.84.0, the floor of the
litellm>=1.84constraint inpyproject.toml. In both,
OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERSis{openai, hosted_vllm}and
create_filehas nolitellm_proxybranch, so routing the upload as
litellm_proxyis not an option either. - Python 3.11, macOS.
Additional context
I have a fix and will open a PR referencing this issue. One part of it is a
behavior change I would rather have decided explicitly than slipped into a
diff: when no proxy endpoint can be determined, the upload raises instead of
quietly using the provider default. For anyone who has OPENAI_API_KEY set
that converts a currently "working" upload into an error. I think an explicit
error is better than a silent misroute, but it is a trade off and it is your
call.
Activity
surajksharma07 commentedon Aug 14, 2026
@tranminhquang4421 The hardcoded openai fallback in the upload path is what's sending it to api.openai.com.
#6723 lines up with the root cause and passed our own before/after check locally so it looks like the right fix.
Since you flagged the Azure-backed proxy case as unverified against a live endpoint could you try that PR against your actual setup and confirm it holds before we look at merging?
Would rather have your real-world config confirm it than assume from our synthetic repro alone.
tranminhquang4421 commentedon Aug 14, 2026
Thanks for checking it locally, glad the before/after lines up on your side too.
On the live Azure verification: I don't have an Azure backed proxy deployment.
That's what I meant by flagging it as unverified in #6723, and the same in
#6578 earlier. It's a gap in what I can test rather than a test I skipped, so
it isn't something I can close myself.
Two things that might help you decide how much that blocks merging.
The reported leak doesn't depend on Azure. Two of the three paths in #6722
resolve to
openaiand never enter Azure code at all:litellm_proxy/openai/gpt-4oopenaiopenai/gpt-4owithUSE_LITELLM_PROXY=trueopenailitellm_proxy/openai/gpt-4o, no proxy endpoint setopenaiThe root cause is
get_openai_credentialsdefaultingapi_basetohttps://api.openai.com/v1. Nothing in that chain is Azure specific, which iswhy your before/after run reproduced it without an Azure deployment.
On
main, an Azure backed proxy model doesn't reach Azure either. Thecustom_llm_provider="openai"override replaces the provider before theupload, so the
azurebranch inlitellm/files/main.pynever runs:Same repro, same
.env, only the model swapped:So on current
mainthe Azure case fails in the same place and for the samereason as the OpenAI one, and a live Azure endpoint would mostly confirm that
the upload goes to
api.openai.cominstead of to Azure.That said, I don't want to argue the Azure test is pointless, because after
#6723 it stops being the same path. With the fix the provider is no longer
overridden, so the
azurebranch runs for the first time:That branch goes through
get_azure_credentialsand puts anapi-versiononthe URL (
/openai/files?api-version=...), which theopenaibranch neverdoes. My stub accepts that happily. A real proxy in front of Azure is where
you'd find out whether the forwarded
api_baseand the resolvedapi-versionactually line up with what the deployment expects, and whether the returned
file_idis then accepted by the completion. That is a genuine gap and it isexactly the part I can't cover from here.
So the request is fair, I just can't be the one to satisfy it. Two ways to get
it if you want it before merging:
had
AZURE_API_BASE/AZURE_API_KEYset alongside the proxy config anddidn't hear back, but a run of this branch on their deployment would answer
both questions above.
the before/after harness running, so pointing it at that would be a shorter
path than anything I can do.
Happy to dig into whatever either turns up, including the
api-versionhandling if it doesn't match a real deployment.
One housekeeping note since you tagged this account: I opened this issue from
@tranminhquang4421and #6723 from@vietnamesekid. Same person, twoaccounts, and
@vietnamesekidis the one that signed the CLA on #6578. Worthknowing in case the CLA check on #6723 looks inconsistent with this thread.
MilkyWay008 commentedon Aug 14, 2026
That's a nasty one...... completion goes to the proxy but the upload resolves its own endpoint, the kind of thing that slips through testing. If you need a stopgap before the PR lands:
get_openai_credentialschecksOPENAI_BASE_URLthenOPENAI_API_BASEbefore defaulting to api.openai.com, so pointing either at your proxy routes the upload through it too. And if you'd rather it never touch that key, unsetOPENAI_API_KEYand it fails loudly instead of silently exfiltrating. Nice repro btw, the egress-blocking spy makes it obvious.surajksharma07 commentedon Aug 17, 2026
Good catch on 205b9f7. ADK's own USE_LITELLM_PROXY check was accepting "1" but LiteLLM's get_secret_bool only reads "true"/"false" so =1 would've had the upload going through the proxy while the completion went direct to the provider. Same class of split this whole PR is closing just the opposite direction.
On the Azure verification makes sense you can't close that gap yourself given the provider override only stops happening after this fix. Reaching out to @ross-p on #6538 since they've got a live Azure-backed proxy rather than block on it here. Otherwise this is looking close just need varunbiluri's sign-off on 56c2519 and we can move toward merge.
fix: keep proxied file uploads inside the proxy