Megablocks on ROCm documentation#
2026-02-26
1 min read time
Use Megablocks on ROCm to build sparse Mixture-of-Experts (MoE) LLMs on AMD Instinct GPUs, enabling high-throughput expert routing and compute efficiency for cost-effective recommendation and dialogue systems.
Megablocks is a lightweight library for MoE training. The core of the system is efficient “dropless-MoE” and standard MoE layers. Megablocks is integrated with stanford-futuredata/Megatron-LM, where data and pipeline parallel MoE training is supported.
Megablocks on ROCm supports distributed pre-training, activation checkpointing and recomputation, distributed optimizer, mixture-of-experts, and dropless mixture-of-experts.
Megablocks is part of the ROCm-LLMExt toolkit.
The Megablocks public repository is located at ROCm/megablocks.
To contribute to the documentation, refer to Contributing to ROCm.
You can find licensing information on the Licensing page.