---
title: "ACL2025 Pagoni: Patches Scale Better Than Tokens"
source: https://www.jemoka.com/posts/kbhacl2025_pagoni/
---

One-Liner &ldquo;Patches in groups of tokenization scale better than tokens&rdquo;
Motivation / Novelty typical byte-level LMs don&rsquo;t are very expensive because many tokens its hard to go beyond 4-6 bytes per token: Zipf&rsquo;s Law so, we model them as token patches Notable Methods token patch &ldquo;how do we segment the byte sequence into patches?&rdquo; &mdash; insight: group predicable tokens after every hard choice! i.e., once you train a model, there are &ldquo;obvious&rdquo;
patcher and unpatcher cross attend
Key Figs New Concepts Notes
