---
title: "MOEReview Fedus: Switch Transformers"
source: https://www.jemoka.com/posts/kbhmoereview_fedus_switch_transformers/
---

At scale, with regularization (including dropout), k=1 on expert routing is fine!
