Skip to main content

Series

Building a Transformer from Scratch

I built a GPT-style decoder-only transformer from scratch, tokenizer included, and trained it at two sizes, 110M and 336M parameters, on FineWeb-Edu. This series follows the whole journey one stage per post: tokenization, embeddings, RoPE, attention, LayerNorm, the feed-forward network, training and evaluation, mistakes included.

Posts in this series