Vision only model memory issue #84

I have created a vision-only model with the convert tool. When im trying to load it and use it with the python binding as shown below my memory just explodes until my system crashes.

model = Clip(
            model_path_or_repo_id="/path/to/model",
        )
# This is where it explodes
model.load_preprocess_encode_image("/path/to/image")

I have tried both with or without quantization and same results. When I run with same with the full model, i.e. not vision-only it works. Didn't investigate further, and this is a decent workaround for now for me, just wanted to let you know :)