Skip to content

The model doesn't respond properly #625

Closed

Description

@SudarshanShetty1796

I'm using the following model:
https://www.kaggle.com/models/google/gemma-3/gemmaCpp/3.0-4b-it-sfp

When I ask a question, the model starts generating gibberish text continuously and doesn’t stop at all. I’ve tried rebuilding the project multiple times, but the issue persists right from the first prompt.

Image

Commands used:

make -j4 gemma

./gemma \     
--tokenizer tokenizer.spm \
--weights 4b-it-sfp.sbs \
--weight_type sfp \
--model 4b-it

System Details:
macOS (Apple Silicon - M4 chip)
CMake version: 3.22.0
Compiler: Apple Clang (Xcode CLT)
gemma.cpp repo from GitHub, built via standard cmake and make

Could this issue be related to the model itself, or is it likely caused by something in the build or runtime setup?

Activity

  1. jan-wassenberg commented on Jun 19, 2025

    @jan-wassenberg
    Member

    Thanks for reporting. Tests are currently passing for me with Gemma 3 4B on x86, but there might be a difference with M4.

    Let's first rule out partial/corrupted weights downloads - sha1sum of the weights should be c8af5c86600f9f389ff157fab76b497e5badf3d9 and for the tokenizer 8a8e05deb279bd5bdca094fb7ececabe732a9534.

    Next, are you building from the dev branch? Does an asan build (add -fsanitize=address to CXXFLAGS) report any issues?

  2. SudarshanShetty1796 commented on Jun 19, 2025

    @SudarshanShetty1796
    Author

    sha1sum is the same as yours
    4b-it-sfp.sbs => c8af5c86600f9f389ff157fab76b497e5badf3d9
    tokenizer.spm => 8a8e05deb279bd5bdca094fb7ececabe732a9534

    are you building from the dev branch?

    yes

    Does an asan build (add -fsanitize=address to CXXFLAGS) report any issues?

    I'm very new to this AI environment — in fact, I'm just trying to learn and don’t yet fully understand many of the keywords, as my expertise lies in a different software domain.

    I didn’t completely understand what needed to be done, but after doing some research, I followed the steps below. I ran into an error, so I’d appreciate it if you could confirm whether these steps were what you intended — or if not, could you please explain the correct steps in more detail?

    1. I cleared my build/ folder, keeping only the model weight and tokenizer files inside it.
    2. I went back to the root of the gemma.cpp project and ran this command: cmake -B build -DCMAKE_CXX_FLAGS="-fsanitize=address"
    3. After that completed, I ran: cmake --build build
    4. Then I moved into the build/ directory and executed the model with: ./gemma \ --tokenizer tokenizer.spm \ --weights 4b-it-sfp.sbs \ --weight_type sfp \ --model 4b-it
    5. The model loaded and gave a prompt (Hello), but it threw an error.
      I’ve copied the full logs and error into a text file — let me know if that would help you debug.

    Thanks in advance!

    Error.txt

  3. jan-wassenberg commented on Jun 19, 2025

    @jan-wassenberg
    Member

    Thank you, that is helpful. The asserts point to the problem: Highway had a mismatch between the feature check and usage of bf16 dot products. I expect to land a fix for that tomorrow.

  4. SudarshanShetty1796 commented on Jun 21, 2025

    @SudarshanShetty1796
    Author

    Hello,
    Is this fixed? Can I try?

  5. jan-wassenberg commented on Jun 23, 2025

    @jan-wassenberg
    Member

    The Highway fix has landed and #630 should go in soon as well.
    Please let us know if that fixes it for you.

  6. SudarshanShetty1796 commented on Jun 24, 2025

    @SudarshanShetty1796
    Author

    I tried it, and it's working fine now! Thank you.

    I'm able to chat with the model.

    However, when I run the ./gemma command, it prints the following error. Is this something to be concerned about, or can it be ignored?

    Image
  7. jan-wassenberg commented on Jul 3, 2025

    @jan-wassenberg
    Member

    Glad to hear, thanks for following up :)
    The warnings are to be expected, OS X does not provide the APIs that would be required for pinning. We can update the code to skip these warnings on OS X.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions