Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
About this project
LLMs-from-scratch is an open-source learning resource project on GitHub: Implement a ChatGPT-like LLM in PyTorch from scratch, step by step. It is a learning resource rather than a running service: courses, curated lists, tutorials or reference implementations that explain how the field actually works. It is maintained by rasbt and published as a public repository you can read, fork and run yourself.
On GitHub the repository is tagged ai, artificial-intelligence, attention-mechanism, deep-learning, finetuning, from-scratch, generative-ai, gpt, instruction-tuning, language-model, large-language-models and llm, which is a fair summary of what it covers. The value is in the structure and the worked examples, so check how recently it was updated before following it end to end.
It is written mainly in Jupyter Notebook and released under no standard SPDX license identifier — the repository uses a custom or modified license, so read it before commercial use. It is actively maintained: the most recent commit landed on November 9, 2026.
To try it, clone the repository and follow its README; that file is the canonical install guide. Because it runs on your own infrastructure, budget time for environment setup: dependencies, model weights and configuration usually take longer than the first successful run.
Key facts as recorded on GitHub: 108,551 stars; 16,795 forks; primary language Jupyter Notebook; last commit on November 9, 2026; topics ai, artificial-intelligence, attention-mechanism, deep-learning, finetuning, from-scratch.