---
title: "MOEReview Krajewski: Scaling Laws for MoE"
source: https://www.jemoka.com/posts/kbhmoereview_krajewski_scaling_laws_for_moe/
---

Define &ldquo;granularity&rdquo; as:
\begin{equation} G = \frac{d_{\text{ff}}}{d_{\text{expert}}} \end{equation}
at \(G=1\), we have a dense model; at \(G&gt;1\), we have some kind of MoE.
Here are thy scaling laws:
notice how its mostly linear! tiny experts yay!
