This video isn’t embeddableWatch on YouTube →

Stanford CS25: V2 I Introduction to Transformers w/ Andrej Karpathy

January 10, 2023 Introduction to Transformers Andrej Karpathy: https://karpathy.ai/ Since their introduction in 2017, transformers have revolutionized Natural Language Processing (NLP). Now, transformers are finding applications all over Deep Learning, be it computer vision (CV), reinforcement learning (RL), Generative Adversarial Networks (GANs), Speech or even Biology. Among other things, transformers have enabled the creation of powerful language models like GPT-3 and were instrumental in DeepMind's recent AlphaFold2, that tackles protein folding. In this speaker series, we examine the details of how transformers work, and dive deep into the different kinds of transformers and how they're applied in different fields. We do this by inviting people at the forefront of transformers research across different domains for guest lectures. More about the course can be found here: https://web.stanford.edu/class/cs25/ View the entire CS25 Transformers United playlist: https://www.youtube.com/playlist?list=PLoROMvodv4rNiJRchCzutFw5ItR_Z27CM 0:00 Introduction 0:47 Introducing the Course 3:19 Basics of Transformers 3:35 The Attention Timeline 5:01 Prehistoric Era 6:10 Where we were in 2021 7:30 The Future 10:15 Transformers - Andrej Karpathy 10:39 Historical context 1:00:30 Thank you - Go forth and transform

Andrej Karpathyguest

May 19, 20231h 11mWatch on YouTube ↗

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

EPISODE INFO

Released: May 19, 2023
Duration: 1h 11m
Channel: Stanford Online
Watch on YouTube: ▶ Open ↗

EPISODE DESCRIPTION

January 10, 2023 Introduction to Transformers Andrej Karpathy: https://karpathy.ai/ Since their introduction in 2017, transformers have revolutionized Natural Language Processing (NLP). Now, transformers are finding applications all over Deep Learning, be it computer vision (CV), reinforcement learning (RL), Generative Adversarial Networks (GANs), Speech or even Biology. Among other things, transformers have enabled the creation of powerful language models like GPT-3 and were instrumental in DeepMind's recent AlphaFold2, that tackles protein folding. In this speaker series, we examine the details of how transformers work, and dive deep into the different kinds of transformers and how they're applied in different fields. We do this by inviting people at the forefront of transformers research across different domains for guest lectures. More about the course can be found here: https://web.stanford.edu/class/cs25/ View the entire CS25 Transformers United playlist: https://www.youtube.com/playlist?list=PLoROMvodv4rNiJRchCzutFw5ItR_Z27CM 0:00 Introduction 0:47 Introducing the Course 3:19 Basics of Transformers 3:35 The Attention Timeline 5:01 Prehistoric Era 6:10 Where we were in 2021 7:30 The Future 10:15 Transformers - Andrej Karpathy 10:39 Historical context 1:00:30 Thank you - Go forth and transform

SPEAKERS

Andrej Karpathy
guest
AI researcher and engineer known for work on deep learning and large language models; creator of NanoGPT.

EPISODE SUMMARY

In this episode of Stanford Online, featuring Andrej Karpathy, Stanford CS25: V2 I Introduction to Transformers w/ Andrej Karpathy explores karpathy Explains Transformers: History, Attention Mechanics, NanoGPT Code, and Why They Dominate The instructors open the course by presenting a timeline from RNNs and LSTMs through the 2017 Attention Is All You Need paper to generative models and ChatGPT, and by listing open challenges such as long-term memory, quadratic attention cost, and controllability.

RELATED EPISODES