mla

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

Design hardware-friendly model architectures and migrate existing LLMs with minimal performance loss

Light-field imaging application for plenoptic cameras

[ICLR 2025] Palu: Compressing KV-Cache with Low-Rank Projection

xKV: Cross-Layer SVD for KV-Cache Compression [ICML 2026]

Light field geometry estimator for plenoptic cameras

An efficient and scalable attention module designed to reduce memory usage and improve inference speed in large language models. Designed and implemented the Multi-Head Latent Attention (MLA) module as a drop-in replacement for traditional multi-head attention (MHA) in large language models.

Tiny-MoE is a lightweight Mixture-of-Experts language model built entirely from scratch in native PyTorch and trained end-to-end on Kaggle using free 2× NVIDIA T4 GPUs. The project implements modern LLM techniques—including MLA, RoPE, YaRN, streaming pre-training, and efficient inference—without relying on existing model implementations.

Official repo for AMD hybrid models training and inference workflow

A production-grade LLM architecture built from scratch in PyTorch. Features Multi-Head Latent Attention (MLA), Mixture of Experts (MoE), GRPO alignment, and a complete 31-part educational course.

MHA, MQA, GQA, MLA 相关原理及简要实现

从 BibTeX、RIS、CSL JSON、CSV/TSV 或普通参考文献列表生成带 Word REF 交叉引用、正文上标数字引用和自动编号参考文献的文献综述 DOCX。

A Mixture of Experts model with latent attention designed for efficient training and inference.

Faithful from-scratch reimplementation of DeepSeek-V3 (MLA + aux-loss-free MoE + MTP + speculative decoding), ~412M params, Chinchilla-optimal 8.4B-token training on a single A100 80GB.

DeepSeek-style MoE + MLA from scratch in PyTorch, with a router-specialization probe (mutual information) and a dense/+MoE/+MLA ablation. Educational, nano-scale, measured.

A code deep-dive on one of the key innovations from Deepseek - Multihead Latent Attention (MLA)

Denoised Neural Portfolio Optimizer (Deep Markowitz with Spectral Filtering)

Recurrent-depth transformer, fixed. Fork of kyegomez/OpenMythos with scatter-based MoE (2.94x faster), proper ACT halting, DeepSeekMoE load balancing, SDPA kernels, and a working training loop.

学术文献查找与引文生成工具 - Citation Finder Skill for OpenClaw

Pure-PyTorch DeepSeek-V3/K3-style LLM stack where every serving optimization ships with a bit-exactness oracle: disk-tier expert streaming, KV prefix pool, MTP break-even telemetry, oracle-gated config evolution. 60 tests, all CPU-verified.