Skip to content

vLLM inference on finetuned model #2

Open

Description

@projan-shakya

It seems like you have used your finetuned model for inference in vLLM. When i try my finetuned model, sometimes there is gibberish audio or some of the audio tokens are skipped. Is there any workaround that?

Activity

  1. SebastianBodza commented on Mar 30, 2026

    @SebastianBodza
    Owner

    Yes my model has a different chat template as i accidentally changed it a bit. Overall the standard transformers inference is working correctly?

    As i do not have a GPU present, I just threw GPT-5.4 at it you can give the branch https://github.com/SebastianBodza/Orpheus_Distributed_FastAPI/tree/official-setup a try. LMK if that works.

  2. projan-shakya commented on Mar 31, 2026

    @projan-shakya
    Author

    it is still gibberish or skips some words when the audio is generating. The issue seems to persist in every inference engine with fine tuned models, but your finetuned model seems to work fine.

  3. SebastianBodza commented on Mar 31, 2026

    @SebastianBodza
    Owner

    Then you probably have an issue during training with the chat template setup vs. the inference. You might want to double check your template setup

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions