You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After reviewing the code, it appears that our decoding stage relies on flash-attn. For the prefilling stage, the three ops used are based on a Triton-implemented version of dynamic sparse flash attention, which does not depend on the flash-attn library. We plan to support a version that does not rely on flash-attn in the future, although this version will have slightly higher latency compared to the one using flash-attn.
Activity
iofu728 commentedon Jul 8, 2024
Hi yawzhe,
Thank you for your support in MInference.
After reviewing the code, it appears that our decoding stage relies on
flash-attn. For the prefilling stage, the three ops used are based on a Triton-implemented version of dynamic sparse flash attention, which does not depend on theflash-attnlibrary. We plan to support a version that does not rely onflash-attnin the future, although this version will have slightly higher latency compared to the one usingflash-attn.[-][Question]: A6000支持加速推理吗?[/-][+][Question]: Is A6000 supported?[/+]iofu728 commentedon Jul 15, 2024
Please upgrade MInference to version 0.1.4.post3, which does not depend on flash_attn. You can do this by running the following command: