---
title: "MOEReview Sukhbaatar: Branch-Train-MiX"
source: https://www.jemoka.com/posts/kbhmoereview_sukhbaatar_branch_train_mix/
---

Its MOEReview Li: Branch-Train-Merge but MoEs now. Each layer is combined by standard moe routing with a weight that is tuned.
