Pinned Loading

1. Transformer-Pretraining-and-Interp Transformer-Pretraining-and-Interp Public

   I Pretrained a Transformer of 134M params from Scratch on around 1.2 Billion Tokens. Now, I am Applying Interpretability techniques to Understand Internals.

   Python

2. GPT-Pretraining GPT-Pretraining Public

   A 134M-parameter GPT pretrained from scratch on FineWeb-Edu on one free-tier GPU, then audited: 4 bugs found in its own training run, every reported number shipped with the command that reproduces it.

   Jupyter Notebook 6 3

3. Coding-Transformers-using-pytorch Coding-Transformers-using-pytorch Public

   Breaking down the Attention Is All You Need paper into clean, modular PyTorch code — making the Transformer architecture accessible to learners and researchers. Full Article: https://medium.com/@Um…

   Jupyter Notebook