Repository navigation
[Python] clip_tokenize: unknown token ' ' #107
Description
I've copy-pasted the example python code from examples/python_bindings/README.md
The tokenizer complains about the spaces in 'cat on a Turtle'.
I've tried both "mys/ggml_CLIP-ViT-B-32-laion2B-s34B-b79K/CLIP-ViT-B-32-laion2B-s34B-b79K_ggml-model-f16.gguf" and the q8_0 variant.
Full log:
(venv)$ python test_clip.py
[File Info] models/CLIP-ViT-B-32-laion2B-s34B-b79K_ggml-model-q8_0.gguf
clip_model_load: description: two-tower CLIP model
clip_model_load: GGUF version: 2
clip_model_load: alignment: 32
clip_model_load: n_tensors: 397
clip_model_load: n_kv: 25
clip_model_load: ftype: q8_0
clip_model_load: text_encoder: 1
clip_model_load: vision_encoder: 1
clip_model_load: model size: 156.10 MB
clip_model_load: metadata size: 0.13 MB
clip_model_load: text model hparams
n_vocab 49408
num_positions 77
t_hidden_size 512
t_n_intermediate 2048
t_projection_dim 512
t_n_head 8
t_n_layer 12
clip_model_load: vision model hparams
image_size 224
patch_size 32
v_hidden_size 768
v_n_intermediate 3072
v_projection_dim 512
v_n_head 12
v_n_layer 12
clip_model_load: 24 MB of memory allocated
clip_tokenize: unknown token ' '
Similarity score: 0.1402660459280014
Activity
monatis commentedon Dec 4, 2024
Hi @h3ndrik, thanks for raising the issue. Did you compile it from source or install from PyPi?
h3ndrik commentedon Dec 4, 2024
PyPi
I did an
pip install clip_cppin a Python 3.11.9 venvmonatis commentedon Dec 4, 2024
Weird. Cannot reproduce on Ubuntu 24.04. But look more into this later. Can you provide what OS and locale are you using?
h3ndrik commentedon Dec 4, 2024
Alright, thanks for confirming this is a bug. Maybe I can find something. I mean it seems to run. Just the tokenization seems to fail.
LANG=en_IE.UTF-8 but I've set a few things to de_DE.UTF-8
OS is: NixOS 24.11 (which might be a problem, since NixOS is very different to other Linux distros and for example doesn't follow the FHS. I thought I set up the environment correctly to mitigate for that, but maybe I made a mistake.)
h3ndrik commentedon Dec 4, 2024
Seems to be the capitalized "T" in "Turtle". If I do the following, it runs without errors:
text_2encode = 'cat on a turtle'But I don't know why it doesn't show 'T' in the warning message then, but ' '
monatis commentedon Dec 4, 2024
Oh missed that. CLIP tokenizers lowercase the input text. See the
"do_lowercase": truesetting heremonatis commentedon Dec 4, 2024
But I should add lowercasing anyway.
MotzWanted commentedon Jan 21, 2025
@monatis, I'm encountering a similar issue with the conversion of the ViT-B-16-SigLIP-i18n-256 model to
ggufusing the scriptmodels/convert_hf_to_gguf.py. Despite the conversion completing successfully, the model's performance does not align with the expected behavior I observe when running it viaopen-clip.When executing the same Python code as @h3ndrik, including lowercase string, the tokenizer fails to recognize spaces, leading to unknown token errors. Could you provide insights into what might be going wrong with the conversion process?
LimitlessDonald commentedon Jun 22, 2025
Hi, were you able to resolve this ?
monatis commentedon Jun 23, 2025
@LimitlessDonald When I have a quick look at this model, it uses
T5Tokenizerastokenizer_classintokenizer_config.json. The original CLIP usesBERTTokenizerand this implementation is aligned for it. AFAIK, T5 uses a sentencepiece (BPE) tokenizer, which regards the space character as an ordinary byte, unlike the wordpiece tokenizer. So this requires to implement sentencepiece algorithm as a tokenizer implemented in.LimitlessDonald commentedon Jun 25, 2025
@monatis Thanks for your feedback and your software, its really helpful.
I just wanted to know if there is any plans to support multi-language models like this https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1
monatis commentedon Jun 26, 2025
Hi @LimitlessDonald, I'm aware that this project remained a little bit outdated, but it is getting continuous attention from the community to my surprise and it's quite satisfying. I'mm 100% sure that it deserves a rework to support multilingual tokenizers, other varients and compute backends. The only reason why I cannot give an exact deadline is, I'm running a newly established startup of my own, but I'll do my best to do these feature developments very soon.
LimitlessDonald commentedon Jun 26, 2025
Thanks @monatis . I am quite an experienced software engineer, I have implemented many models in my work, but not built something like your software, I would like to contribute to it if possible . Do you have any pointers to some crash course , that is condensed to get me up and running quickly to enable me contribute to this repo ?
monatis commentedon Jun 27, 2025
hi @LimitlessDonald, the main tensor library and file format used in this repository is the same as in GGML / GGUF / llama.cpp. This repository was written at a very early stage of llama.cpp when we just coded the GGUF file format, then we made some modifications to the GGUF format itself and, more importantly, compute backends in GGML. So the current implementations in llama.cpp and GGML might differ from that in this repo, that's why I need a modernization in the codebase. If you are interestedin these work, This wiki page might be a good starting point. Sorry but I don't know a comprehensive documentation because GGML and llama.cpp are fast-moving projects, lacking detailed and difinitive documentation. You should embrace the necessity to read the source code from time to time to better understand how things are implemented and wired together etc.