Repository navigation
Support 1 * 128 and 128 * 128 block-wise quant? #38
Description
Activity
Hi @zfan2356 , thank you for raising this issue. The lower-level
nvmath.bindingsfor those scaling types should be included in the next release of nvmath-python. I'll also start working on adding support to our higher-level host APIs innvmath.linalg.advanced.matmul.Reacted by Sergey Maydanov@szkarpinski Thanks for your reply! I also recommend supporting the pointer array mode for Group GEMM. Both of these features are commonly used in modern LLM pretrain. nvmath-python is a user-friendly Python library, and I hope it can support these two features as soon as possible. Thank you very much!
In the CUDA 12.9 cuBLASLt documentation, I noticed support for 1×128 and 128×128 block-wise quantization methods. However, I found that nvmath-python currently lacks bindings for this type of quantize approach. I wonder do we have any plan for support this approach?
https://docs.nvidia.com/cuda/cublas/index.html#cublasltmatmulmatrixscale-t