---
title: "SU-CS224N APR162024"
source: https://www.jemoka.com/posts/kbhsu_cs224n_apr162024/
date: 2024-04-16
---

Why do Neural Nets Work Suddenly? Regularization see regularization
We want to be able to manipulate our parameters so that our models learn better&mdash;for instance, we want our weights to be low:
\begin{equation} J_{L2}(\theta) = J_{reg}(\theta) + \lambda \sum_{k}^{} \theta^{2}_{k} \end{equation}
or good &lsquo;ol dropout&mdash;&ldquo;fetaure dependent regularization&rdquo;
Motivation classic view: regularization works to prevent overfitting when we have a lot of features NEW view with big models: regularization produces generalizable models when parameter count is big enough Dropout Dropout: prevents feature co-adaptation =&gt; results in good regularization
Language Model See Language Model
