Skip to content

allow to use locally installed OLLAMA #5

Closed

Description

@pcause

I alrreayd have phi4 installed with ollama. Can you allow this to use ollma installed models? If you do, also support the OLLAMA_HOST spec to find the ollama instance

Activity

  1. codingmoh commented on Apr 22, 2025

    @codingmoh
    Owner

    Alright! - Sure, I'll have a look at it!

  2. ivostoykov commented on Apr 27, 2025

    @ivostoykov

    +1

    It makes no sense to download again. Please add some settings somewhere - ie. ~/.open-codex where the model can be changed. Perhaps such file may be useful for additional tuning

  3. rpCal commented on Apr 28, 2025

    @rpCal

    +1

  4. codingmoh commented on Apr 28, 2025

    @codingmoh
    Owner

    Okay, an agent using chatcompletions API is required for that, which can be achieved now, especially if we only stick to bash completions for the moment.

    I'll draft some changes tomorrow, maybe done by Wednesday. Thanks

  5. ivostoykov commented on Apr 28, 2025

    @ivostoykov

    Just another hint ;-) if using existing api will be inappropriate, perhaps just setting the path to a gguf file will be enough. But this means to handle various LLMs

  6. codingmoh commented on Apr 28, 2025

    @codingmoh
    Owner

    @ivostoykov Thanks for the suggestion! It’s like we had the same thought at the exact same time – I just checked where ollama stores the models to avoid multiple downloads!

    They seem to be stored here on macOS: ~/.ollama/models

    My overall goal was actually to optimize prompts and agents (code implementations) for a specific model, because each model requires a certain level of optimization to ensure the best quality, right?

    In the case of ollama:
    Right now, I don’t know yet what I’ll stick to, so we’d probably have to provide a general-purpose prompt for ollama models.

    One idea could also be to introduce an OPEN_CODEX_PROMPT / OPEN_CODEX_PROMPT_TXT environment variable for models brought by the user - right now, this approach sounds reasonable to me.

  7. ivostoykov commented on Apr 28, 2025

    @ivostoykov

    @codingmoh, Depending on how Ollama is executed, the location may vary. Allowing users to specify the exact path would enhance convenience. Alternatively, searching for *.gguf files and presenting them for user selection is another viable approach.

    cheers

  8. codingmoh commented on Apr 28, 2025

    @codingmoh
    Owner

    Oh, I didn’t know that the path could vary.
    Honestly, I think the idea of searching for GGUF files isn’t bad at all, I have to admit.

    The question, of course, is how this implementation could look like:
    Which folders would you search? For example, would you search the .cache folder, the home directory, or really the entire filesystem?

    But honestly, I really like the idea or the general approach.

    We could also rely on locations that are "known," like .huggingface, .ollama, and whatever else comes along.

    It’s actually not a bad idea at all.

  9. ivostoykov commented on Apr 28, 2025

    @ivostoykov

    Indeed searching for GGUF is a bit tricky because of the access. My preferences go with the config file where the user could place link to the desired models (GGUFs). At the end of the day this is an app for people having some knowledge of the dir architecture, so it shouldn't be a problem. This seems to me the easiest and fastest approach.

  10. self-assigned this
    on Apr 30, 2025
  11. linked a pull request that will close this issueadded ollama support #10on Apr 30, 2025
  12. codingmoh commented on Apr 30, 2025

    @codingmoh
    Owner

    I'll set up the ability to lookup gguf via config file in the next version, as I'd have to check how to "mount" effectively. We could introduce a GGUF_DIR env variable, and DEFAULT_MODEL, couldn't we? - Or is there any additional config you're thinking of needing?

    For now I have added support for ollama now via ollama api (and its python sdk), the pr is attached to the issue, in case you'd like to have a look at the implementation.

    I'll review again more carefully tomorrow, and merge it

    Essentially I've added:

    • Added --ollama flag to enable Ollama mode
    • Added --model flag to specify the model (e.g., llama3)
  13. ivostoykov commented on Apr 30, 2025

    @ivostoykov

    Yes, this is the right direction. About the additional config, I believe it could be ~/.open-codex/open-codex.conf where all needed variables may be placed (sys instruct for instance) then on start check env variables and open-codex.conf merging the result. This will be flexible enough, I believe.

  14. codingmoh commented on Apr 30, 2025

    @codingmoh
    Owner

    Would you suggest to have .env overwriting open-codex.conf?

  15. ivostoykov commented on Apr 30, 2025

    @ivostoykov

    I mean that if you use GGUF_DIR env variable, and DEFAULT_MODE then /open-codex.conf if exists should take precedence

  16. added a commit that references this issue on May 3, 2025
    07d0b40
  17. codingmoh commented on May 4, 2025

    @codingmoh
    Owner

    @ivostoykov thanks again! - I've meanwhile merged and released a version supporting ChatCompletions API - gguf dir with config shall be supported in an upcoming release!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions