Skip to content

Ollama support #49

Closed
Closed

Description

@curiousily

I'm trying to use the LiteLLM (LLM?) with Ollama support like this:

from google.adk.agents import Agent
from google.adk.models.lite_llm import LiteLlm

MODEL = "qwen2.5"


def get_weather(city: str) -> str:
    """Retrieves the current weather report for a specified city.

    Args:
        city (str): The name of the city (e.g., "New York", "London", "Tokyo").

    Returns:
        str: A report on the current weather in the specified city.
    """

    return f"It is always sunny in {city}!"


root_agent = Agent(
    name="weather_reporter",
    model=LiteLlm(model=f"ollama_chat/{MODEL}"),
    instruction="You are a helpful assistant. You always answer weather questions.",
    description="A helpful assistant that provides weather information.",
    tools=[get_weather],
)

When I send a message to the agent I get a response, but when tool call is required (see the screenshot) I got:

litellm.exceptions.APIConnectionError: litellm.APIConnectionError: Ollama_chatException - {"error":"json: cannot unmarshal array into Go struct field ChatRequest.messages.content of type string"}

Tried also with Llama3.2, Gemma3, DeepSeekR1-14B - they can't even respond to a normal message.

Environment:

MacOS 15.4
Ollama 0.6.5
adk-python 0.1.0
litellm 1.65.5

Is there a better way to call Ollama models?
Maybe support the OpenAI SDK (also can be used with Ollama) or provide native Ollama support?

Thank you!

Image

Activity

  1. codefromthecrypt commented on Apr 11, 2025

    @codefromthecrypt
    Contributor

    FWIW you can get it to work by using the LiteLlm openai variant until there's an openai one (#27)

    # TODO: native openai https://github.com/google/adk-python/issues/27
    os.environ["OPENAI_API_BASE"] = os.getenv("OPENAI_BASE_URL")
    
    --snip--
        model = os.getenv("CHAT_MODEL", "gpt-4o-mini")
        agent = Agent(name=app_name, model=LiteLlm(model="openai/" + model), tools=[your_tools_here])

    Then, use the normal openai ENV like this:

    # Override default ENV variables for Ollama
    OPENAI_BASE_URL=http://localhost:11434/v1
    OPENAI_API_KEY=unused
    # sizes <3b tend to miss tool calls
    CHAT_MODEL=qwen2.5:3b
    
  2. fesnault commented on Apr 11, 2025

    @fesnault

    @codefromthecrypt you saved my day, I was struggling with the adk web not working at all. When I said hi, it tried to call tools, then called in an infinite loop, that was so strange.
    With your changes, it's fine now, it answers correcly when not about weather, and calls the tool when needed.
    Thank you !

  3. curiousily commented on Apr 11, 2025

    @curiousily
    Author

    @codefromthecrypt thank you! Thank worked (better). I gave you a shout-out in the video: https://www.youtube.com/watch?v=RgLSTGFjpqI

  4. hangfei commented on Apr 12, 2025

    @hangfei
    Collaborator

    @polong-lin please consider incorporate the Oallma setup in our next doc release. Thanks.

  5. selcukgun commented on Apr 12, 2025

    @selcukgun
    Collaborator

    @curiousily You can go back to using ollama_chat provider once the fix above is merged.

    Using ollama_chat provider:

    export OLLAMA_API_BASE="http://localhost:11434" && adk web

    Or you can continue using openai provider name with OPENAI_API_BASE override.

    You can peek into the request sent by litellm to ollama server by adding the followings in your agent.py

    import litellm
    litellm._turn_on_debug()

    payload looks same for both methods. only routes seem different; /api/chat vs /v1 (which seems to have circumvented to parsing issue).

  6. hangfei commented on Apr 13, 2025

    @hangfei
    Collaborator

    the change is merged but it's not released to PyPI yet. For now, you could build a local wheel to unblock yourself.

  7. curiousily commented on Apr 13, 2025

    @curiousily
    Author

    Thank you everyone! Will test once there is a PyPI release.

  8. boyangsvl commented on Apr 19, 2025

    @boyangsvl
    Collaborator

    mark as closed as it's fixed in v0.2

  9. codefromthecrypt commented on Apr 24, 2025

    @codefromthecrypt
    Contributor

    thanks for sorting this and @curiousily thanks for the shout out. humbled!

  10. marreapato commented on Dec 16, 2025

    @marreapato

    I am still getting this issue on adk 0.3.0. Would anyone know why?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions