---
title: "ICLR2025 Wu: Retrieval Head Explains Long Context"
source: https://www.jemoka.com/posts/kbhiclr2025_wu_retrieval_head_explains_long_context/
---

Motivation Previous works contain &ldquo;heads&rdquo; that perform some specific mechanism from context retrieval.
Retrieval Head Authors shows that Retrieval Heads exist in transformers: using Needle in a Haystack framework.
Key Insight There exists certain heads which performs retrieval, as measured by the retrieval score.
Methods Measuring Retrieval Behavior &ldquo;retrieval score&rdquo;: how often does a head engage in copy-paste behavior.
token inclusion: current generated token \(w\) is in the edle maximal attention: same token gives the maximum attenion score
