LoRA initialization
Showing that $0\neq 0$
Blog
Paper summary: BERT
Let's get bidirectional
Paper summary: Attention is all you need
The original Transformer paper
Summary of "Scaling Laws for Neural Language Models"
After yesterday's look at BERT[@devlin2019bert] and the benefits of bidirectional attention, we are covering a different direction today. While previous papers focused on _how_ to build the model (arc
Weight tying does not imply embedding tying
A closer look on weight tying and its effects on token embeddings
nanochat's gpt.py
Step-by-step exploration of nanochat's model
DPO is SFT
Close examination of gradients of DPO and SFT