Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch
-
Updated
Jun 17, 2024 - Python
Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch
[NeurIPS 24] MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Tasks
A self-contained study of mixture-of-experts routing and agent aggregation, with offline Python experiments and synthetic fixtures.
Google DeepMind: Mixture of Depths Unofficial Implementation.
Topological Manifold Control: A multi-paradigm (PyTorch/JAX) geometric morphing engine utilizing differentiable soft-gating, optimized via XLA fused kernels and autograd-isolated non-blocking pipelines.
通过基于“滞后梯度博弈”的内生演化压力,替代传统 MoE 的外在负载均衡约束,实现神经网络拓扑结构的自组织特化与层级化
[NeurIPS 2026] SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization
Adaptive model-capacity orchestration — dynamically select, load, cache, and budget adapters, experts, models, and future model blocks.
Function-preserving neuron growth in PyTorch, exact training-state recovery, and controlled capacity-allocation experiments with scoped negative results.
Research prototype: hierarchical module/expert routing and adaptive depth in a small byte-level language model (preliminary results, under audit)
Official implementation of UniF-MoE: a unified framework for token-adaptive MoE computation
Research code for testing whether causal token surprisal can guide adaptive computation, sparse refinement, and learned compute allocation in byte-level language models.
To associate your repository with the conditional-computation topic, visit your repo's landing page and select "manage topics."