---
title: "MOEReview Zhang: Mixure of Attention Heads"
source: https://www.jemoka.com/posts/kbhmoereview_zhang_mixure_of_attention_heads/
---

Split \(Q\) projection and attention out projection into experts, with one router coordinating them.
Better than MHA performanec.
