Lex Fridman PodcastKempf & Kunhya on Lex Fridman: How FFmpeg runs 90% of video
By inferring codec from content rather than file format or extension; FFmpeg assembly code now underlies YouTube, Netflix, and ninety percent of internet video.
EVERY SPOKEN WORD
150 min read · 30,025 words- 0:00 – 2:17
Episode highlight
- JKJean-Baptiste Kempf
The important is, is your code good? We care about excellent code. We don't care who you are. Like maybe you're a dog. I don't care, right? I don't care where you come from. I need to look at your code. Oh, yeah, but I'm engineer at this very large company in, in, in Italy, in Germany, in the US. We don't care. We care about the quality of your code because this is what defines our community and which means that we have a lot of people who contribute who are some very different backgrounds and, and, and very introverts. Sure. But that's okay, right?
- LFLex Fridman
FFmpeg is probably one of the biggest CPU users in the world. Everything we've just said in the past couple of minutes, every sentence is someone's lifetime's work. There are books about every sentence. So the level of complexity in many cases is inordinate.
- JKJean-Baptiste Kempf
FFmpeg has one hundred thousand lines of assembly for all the codecs.
- LFLex Fridman
For all codecs. Mm-hmm.
- JKJean-Baptiste Kempf
And just this one has two hundred and forty thousand. Every cycle matters. We are talking about probably three billion devices which are going to decode video nonstop because, for example, thirty percent of the video from Netflix are now in AV1, fifty percent of YouTube.
- LFLex Fridman
This is what peak video codecs should look like. Seventy-nine point nine percent assembly, nineteen point six percent C, and, uh, zero point five percent other.
- JKJean-Baptiste Kempf
And what's incredible is with those tweets, which is factual, people get crazy.
- LFLex Fridman
For the last two years, they go crazy. No, intrinsics is fine. The compiler-
- JKJean-Baptiste Kempf
You can optimize your compiler. Auto-vectorization is your fault. You don't understand. And we've tried that forever, right?
- LFLex Fridman
For two years and two years later, showing hundreds of examples of, of handwritten assembly. No, no, no, you're doing it wrong. The compiler can do this. The intelligence agencies tried to like say, "Can you put a backdoor in VLC?"
- JKJean-Baptiste Kempf
Yes. Two of them.
- LFLex Fridman
Well, what do you say?
- JKJean-Baptiste Kempf
No.
- LFLex Fridman
[laughs]
- JKJean-Baptiste Kempf
Well, I was a lot less polite.
- LFLex Fridman
Basically saying, "Hell no."
- JKJean-Baptiste Kempf
Like if we had to compromise our software, we would shut it down. This is clear.
- LFLex Fridman
Any tweets, uh, Kieran, you regret?
- SPSpeaker
Tweets I regret?
- LFLex Fridman
Or is it like that, how does the French song go? Regret nothing.
- JKJean-Baptiste Kempf
Don't regret anything. No, it's because regrets are attacks on your mind.
- 2:17 – 5:35
Introduction
- LFLex Fridman
The following is a conversation all about FFmpeg and VLC with Jean-Baptiste Kempf and Kieran Kunhya. FFmpeg is an open source software system that is the invisible backbone behind YouTube, Netflix, Chrome, VLC, Discord, and basically every platform that touches video or audio on the internet. It can decode, encode, transcode, stream, and play almost any video or audio format ever created. To me, it is one of the most incredible software systems ever developed, and it's all done by volunteers. VLC is also a legendary piece of software. It is an open source media player that plays basically anything you throw at it, any format, any platform, no ads, no tracking. It has been downloaded over six billion times, and again, for me, it has been one of my favorite pieces of software ever with the most legendary logo, which I of course had to honor in this conversation by wearing the VLC traffic cone hat the whole time. So again, above all else, thank you to the incredible volunteer engineers who put their heart and soul into this code that has been used and loved by billions of people. Thank you. And about the two great engineers and human beings I'm, uh, talking to in this episode, Jean-Baptiste is the president of VideoLAN and is a key figure behind VLC and FFmpeg. Kieran is a longtime codec engineer, FFmpeg contributor, and the man behind the now infamous FFmpeg account on Twitter/X that I recommend everybody follow for the memes and for the unapologetic celebration of open source and great low-level software engineering. Let me also say that it's inspiring and humbling that so much of modern civilization rests on software built by people who are not chasing fame or money, but are obsessed with the craft of engineering. We live in a world where billions of people consume video every day without ever thinking about the invisible machinery underneath it. But that machinery matters. Open source infrastructure matters. It is one of the great examples of human beings quietly collaborating across borders to build something useful, durable, and elegant for the rest of us. And so this conversation is not just about codecs and media pipelines. It is also about the deeper spirit of engineering and generosity that makes projects like FFmpeg possible. Again, I can never say it enough. Thank you. This is the Lex Fridman Podcast. To support it, please check out our sponsors in the description where you can also find links to contact me, ask questions, give feedback, and so on. And now, dear friends, here's Jean-Baptiste Kempf and Kieran Kunhya.
- 5:35 – 9:59
Weirdest things VLC opens
- LFLex Fridman
So the legend goes VLC can open everything. What's the weirdest thing?That you know that it can open
- JKJean-Baptiste Kempf
You know, there is a ton of people who are using VLC to record VHS videos, right? Like, it's just like you plug it with a capture card and you can basically record VHS video.
- LFLex Fridman
Well, how does that work?
- JKJean-Baptiste Kempf
Basically, it's, you know, those type of capture card where you can put a Peritel in or, or RGA, and you put that, and actually VLC can play those type of cards, and there is a module which allows to control directly some, those VCR camcorders. We support DVD audios lately, right? We spent the, the summer working on DVD audio support, and like there is no, no one's making any DVD audio support. There is a custom encryption schemes.
- LFLex Fridman
What about Lucasfilm?
- JKJean-Baptiste Kempf
Oh, yeah, and there is of course all the weird codecs support, game codecs supported by, by FFmpeg.
- KKKieran Kunhya
The one Star Wars video game, the first ten-second opening sequence, someone has gone and implemented that and made sure that's bit exact on one disc that existed at one time of one little sequence in the game.
- JKJean-Baptiste Kempf
A- and then funnily, there, there was a, at one VideoLAN conference, we made a competition to me- make the weirdest and most horrible file ever-
- LFLex Fridman
Mm-hmm
- JKJean-Baptiste Kempf
... and see if VLC could play it.
- LFLex Fridman
What did it end up being? What's the file?
- JKJean-Baptiste Kempf
It was an MKV file made by Derek-
- LFLex Fridman
Yeah
- JKJean-Baptiste Kempf
... which each of the frame was changing resolution, aspect ratio [laughs]
- LFLex Fridman
[laughs]
- JKJean-Baptiste Kempf
... rotation, and, and, and it was like-
- LFLex Fridman
Did it work?
- JKJean-Baptiste Kempf
Yes. And there was another one where the whole video was actually animated subtitles, right? SSA, right? So-
- LFLex Fridman
Yeah. I remember that, yeah
- JKJean-Baptiste Kempf
... each, each, this one was-
- LFLex Fridman
[laughs]
- JKJean-Baptiste Kempf
And so each, each frame was a black frame, but on top of that, there was a, a, a subtitle that was animated for each frame.
- KKKieran Kunhya
There was a file that's a valid zip and a valid MP3 at the same time or something like that, so [laughs]
- JKJean-Baptiste Kempf
So yeah, we'd made a competition of stupid files.
- LFLex Fridman
[laughs] And it worked. It opened all of the stupid files.
- JKJean-Baptiste Kempf
Yes.
- LFLex Fridman
By the way, uh, for people who are not familiar, I am wearing a hat. Would it be fair to say this is the best worst logo of all time, the cone?
- JKJean-Baptiste Kempf
Yeah, by far, right? The logo of VLC is so iconic, right? Like we are a team with a small number of people, and, and the icon is known everywhere. I go to middle of nowhere in India or in China, people know the cone, right? And, and twenty-five percent of the website traffic that comes to, uh, our main website is cone player, right? So, so many people don't know VLC, right? They know the cone player.
- LFLex Fridman
That's the thing that Google for is cone player.
- JKJean-Baptiste Kempf
Yeah. They, they go on Google and they, they put cone player and, and they download VLC, right? So that's iconic. A- and once we tried to change it as a joke, right? We said it was going to be a, a type of, um, uh, caterpillar construction, and, and we said that during April first. Uh, and-
- 9:59 – 19:20
How video playback works
- LFLex Fridman
So the thing that we're talking about is everything around video codecs, video encoding, video decoding, video streaming, video player client that I'm wearing on my head, the entire ecosystem enabling free media. Uh, we'll talk about FFmpeg, we'll talk about VideoLAN, VLC, and all the other incredible video technology, uh, that is used probably by billions of people. So JB, you're the lead developer behind the legendary VLC player. Kieran, amongst many other things, you're lead developer behind the legendary FFmpeg handle on Twitter. And, uh, both of you have spicy opinions, I would say. So today we wanna talk about FFmpeg and VLC. Uh, for context for people who are not aware, and I'm sure basically everybody listening to this have used these two technologies probably regularly without knowing it. So FFmpeg underlies basically most video on the internet, including YouTube, Netflix, Chrome, Firefox, of course VLC, and countless other video platforms. It is estimated that over ninety percent of video processing workflows online and offline involve FFmpeg. VLC has been downloaded at least six point five billion times. But likely that number, 'cause it's impossible to really count the number, uh, is much higher than that. Virtually any operating system supports virtually any media format, the limitation being it can't open pancakes. So-
- JKJean-Baptiste Kempf
[chuckles]
- LFLex Fridman
Uh, can we just lay out some of the basics to, to help people understand what's involved in all of this? So when we press playOn a video player like VLC, what happens? What-- How does it go from the, the file or the stream to the pixels on the screen and the sound on the speaker? What are the big stages to be aware of?
- JKJean-Baptiste Kempf
So there are several stages, right? The first stage is to get from an address, right, which is the ca- type of URL, to give you a byte of streams, right? So this would be, for example, HTTP, file, DVD, right? You give the path to the media, and it gives you a, a stream of data.
- KKKieran Kunhya
The stream needs to be cut up by what's known as the container, the demultiplexer or demux. Um, we'll try and keep the jargon light throughout this, but, um, it needs to go and start demarcating video and audio frames. So it just gets data from the operating system blocks at a time and needs to start cutting these frames up into compressed data. It then needs to start doing simple parsing of the video frames-
- LFLex Fridman
Mm-hmm
- KKKieran Kunhya
... mainly to figure out whether that codec is GPU decodable or needs to fall back to software. We're very sort of used to assuming the GPU will play all of these things. There'll be hardware acceleration. I think it's up to forty-five percent of files are not GPU decodable. So these need to be probed. They need to be detected. There can be variants of a given codec, some of which are decodable on the GPU. Different vendors of GPU might have different capabilities, so those need to be detected. So if, if it's GPU capable, you pass it through to the GPU black box. So now if there's a software fallback, that means in the beginning is, is to first do deentropy coding, so removing the mathematical coding of the bit stream. So this uses capabilities such as Huffman coding or arithmetic coding to actually decompress the mathematical layer of the bit stream. We then need to start reading the syntax elements for intra prediction. So intra prediction are like still images of the video, so your iframes.
- LFLex Fridman
Mm-hmm.
- KKKieran Kunhya
So this works and operates in the spatial domain. So you do your intra prediction spatial domain. You, you have a residual because your prediction isn't quite matching that of reality. So you've made a prediction, but then there's a little bit left, and that's what's known as the residual. This is stored in the frequency domain, and these are quantized to decompact their space. We then need to do the inverse transform to bring them back to the spatial domain and apply these residuals.
- LFLex Fridman
So a lot of the process of the decoding is this thing is compressed.
- KKKieran Kunhya
Yes.
- JKJean-Baptiste Kempf
Yes.
- LFLex Fridman
And you have to predict the highest quality thing that's supposed to go there. Iframe-
- KKKieran Kunhya
Yeah
- LFLex Fridman
... is the best representation you have spatially.
- KKKieran Kunhya
Yes.
- JKJean-Baptiste Kempf
Yes.
- LFLex Fridman
And then you, uh, and then there's a lot of temporal compression that can happen depending on the codec, and then you're predicting. You're predicting what the reality that was captured in this rawest form.
- JKJean-Baptiste Kempf
Yeah, because what, what people don't realize is that the compression on video and audio is one hundred times, right? Like, people don't realize how compressed we, we do, right? For audio, you move, you compress by when you go from normal audio to MP3, you compress by ten times, right? When, when you move to video, you need one hundred times, two hundred times, right? So you need to remove all the details, but that you don't care about. Because all the compressions that we do, and that's very important, people forget about that, is to be viewed by humans, right?
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
So all the codecs, either for audio, mimic basically how your ear works, right? And, and a lot of things about, like, the, the, the response on the ear and same for, for your eyes, right? And, and so, for example, on video, we don't work on RGB, right? Everyone expect to work in RGB. We don't, right? We move to YUV, which is basically one is luminance, brightness, and the other are colors. And this matches your eyes, where inside your eyes you have the cones and the buttons, right? With some of them look on brightness and more on, on the other on colors, right? So we need to compress a lot, and so we need to degrade. But in order to degrade, we need to match the human perception, and this is why it's so difficult. And then we need to use the maximum power, mathematical power, uh, very complex technologies. Uh, we move to the frequency domain, as Kieran said. We do a ton of dequantizing and, and in order to get the best compression, but it still looks good.
- LFLex Fridman
You're trying to compress in order to maximize the highest quality thing for human perception.
- JKJean-Baptiste Kempf
That is correct, and that is correct. And this is very important, right? Compression is not like a zip, right? A zip, you have data in, you get data out, right? And you try with all the, the zip compression to arrive with, uh, uh, the limit. Here we are degrading the signal, right? And so we need to degrade both the audio and the video signal in the best way possible. A- and we can do that, but it involves, first, a, a lot of theoretical knowledge about how it works, the eye works, but it, a lot of mathematical, uh, change, a lot of mathematical tricks, right? For example, when you move to RGB and you do go to YUV, for example, what we do very often is that we scale down the resolution of the color compared to the brightness. And most of the time-- And just this, without compression, it divides the size by two. But most people don't see it, right? Um, and so on and so on, right? Uh, and then you go to very complex mathematical, uh, change. So of course, uh, Fourier's transform, which de facto are not Fourier transform. They are like, um, discrete cosinus transform, but that's the same idea. So frequency domain, um, we split the video by blocks, right? So that's why when it's wrongly decoded, you see those blocks and badly encoded, you see those blocks, a- and so on, to arrive to compression states that are insanely high, right? And each generation of the codec is like thirty percent less-
- LFLex Fridman
Mm-hmm
- JKJean-Baptiste Kempf
... for the same quality, right? And this requires amount of power, um, of computational power that are huge.
- KKKieran Kunhya
No, maybe you should, you should elaborate. It's thirty percent better, but an order of magnitude, perhaps, perhaps even two orders of magnitude more compression power. That, that's the big difference.
- LFLex Fridman
What do you mean by compression power?
- KKKieran Kunhya
Sorry, CPU power to achieve that level of compression.
- LFLex Fridman
Oh, yeah. So and you have to be able to leverage the CPU and sometimes GPU, like you mentioned. And then we should mention that a lot of this programmingUh, is done at the lowest possible stack-
- JKJean-Baptiste Kempf
Yeah
- 19:20 – 30:07
Video codecs and containers
- LFLex Fridman
Uh, we mentioned codecs, but I don't think you mentioned, uh, containers. So what, what's the actual containers for some of the stuff we're talking about? So people are familiar with MP4, uh, MOV, MKV. So anyway, what, what are containers versus, uh, the thing that goes inside?
- JKJean-Baptiste Kempf
So the container is what we call also the muxer, right? When I say demuxing, it means decontainerizing, right? So a- actually, if you look, mux, mul- multiplexer, and demultiplexer, right? Mux and demux are those. And same, a codec is actually coder-decoder, right?
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
Um, and, um, so containers are this collection of multiple tracks, right? So it's a, what normal people call the file format, but it's a bit more, um, um, subtle than that. But the most known one, of course, is MP4, but, uh, when I started, it was AVI, right? AVI was the, the video format from-
- LFLex Fridman
Mm-hmm
- JKJean-Baptiste Kempf
... from, uh, Microsoft, and, uh, MOV, M-O-V, which became MP4, was a format from Apple. Um, in the open source community, uh, one of the person that is still active on VideoLAN is called Steve Lhomme and sta- started the, this Matroska format, which is, like, a bit more complex and, and, and more future, uh, proof. Um, and, um, there are so many others.
- LFLex Fridman
So, I mean, there's a, it's a pretty common thing, and m- maybe it'll even happen in this conversation, that people confuse container and the codec, right? So confuse MP4 and H.264, for example. Is that a horrible violation?
- JKJean-Baptiste Kempf
No, it's not, because technically, the name of H.264 is MPEG-4 Part 10. Because MPEG-4 is actually a meta specification which has several things in it, right? There is the Part 2. Uh, so there is, like, audio codecs, right? AAC de facto is MP4 audio-
- LFLex Fridman
Mm-hmm
- JKJean-Baptiste Kempf
... something. There is, uh, actually several video codecs, right, inside the MPEG-4 specification. One of them is MPEG-4 Part 10, called also AVC, called also H.264.
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
Right? So it's completely the fault of the industry to, to, to, to make things difficult to understand. So that's very difficult so that people then don't understand why sometimes you talk about MPEG-4 Part 10, where you mean H.264, and why it's not MP4.
- LFLex Fridman
So you can technically shove in all kinds of different codecs inside containers and horribly so.
- KKKieran Kunhya
But broadly speaking, though, MP4 is understood to generally be H.264 plus AAC audio. 99% of the time, that's that, and that, the, the rest are de minimis, the small effects, you know, edge effects really compared to that. So it's not the end of the world. There, there are people who do get annoyed by that. But also in reality, something like VLC, just to point out, the file may say .MP4, but it may be something completely different, and that's one of the challenges both FFmpeg and VLC have is the real world is a completely different place to a three-letter file format.
- JKJean-Baptiste Kempf
And this is very important to say, right? Like, for example, in VLC and in FFmpeg, we discard the file format, right? We, we look into the file to understand what's in it, because so many people, like, they say, "Oh, it's a video, it should must be MP4," but technically it's an MOV or maybe it's a MKV, right? So we analyze in real time everything that we have, and we don't trust, uh-
- LFLex Fridman
Mm-hmm
- JKJean-Baptiste Kempf
... the, the, the format.
- LFLex Fridman
So what information does the fact that it's .MP4 give you?
- JKJean-Baptiste Kempf
It helps, right? It gives you a hint, right? Just like, oh, it, it's finished by .MP4. I will start first by opening, probing it with the MP4 container demuxer to see, well, it should be that. But I don't trust it, and if I'm lost, I say, "Okay, maybe I'm going to try it." So it bumps the priority of the module.
- LFLex Fridman
So how do you get to, uh... Just to take a bit of a tangent there. You know, the dumb thing is if you try the MP4, but it turns out it's a different codec than you would have expected, uh, most players just break there.
- JKJean-Baptiste Kempf
Yes.
- KKKieran Kunhya
Yes.
- LFLex Fridman
And so how do you not break? There's just, uh, philosophically, I'm sure there's a bunch of stumbling blocks along the way where you, y- it's easier to just break and stop, freak out. That's it. How does VLC not?
- JKJean-Baptiste Kempf
This is why VLC is popular. Um, but the reason is because actually VLC was, is just a client of a streaming solution called VideoLAN from, from, from very long time ago, from the late '90s. And when you're playing video, which are on UDP, right, in network, they might be damaged, right? So you don't trust your inputs, and this is very important into the security, is that you don't trust your inputs. SoEverything in VLC is prepared to, um, work with broken files.
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
And it's a philosophical idea from the beginning, and everything is engineered into that. A-and it's a culture, right? And so for example... And VLC became very popular on that because a long time ago when people were, uh, pirating content, um, which they do a lot less, uh, today, um-
- LFLex Fridman
And none of us ever have done so.
- JKJean-Baptiste Kempf
No, of course not.
- LFLex Fridman
Yeah.
- JKJean-Baptiste Kempf
Um, the metadata to play some files like AVI is at, at the end of the file, right? And when you're downloading, you don't have that, right? So VLC was just like, "Hey, this file is broken, but I'm still going to try to interpret it," and this was very useful.
- 30:07 – 51:07
FFmpeg explained
- LFLex Fridman
we've, uh, kind of danced around it. We talked about VLC, the logo, the hat. Uh, let's talk about FFmpeg. What, what is FFmpeg exactly?
- JKJean-Baptiste Kempf
FFmpeg is basically the low-level libraries for codec, so compressions and decompression, muxers and demuxers, and filters. It's-- The core is this, and then you have a several tool which allow you to create a type of pipeline to process any type of video files. Uh, and it's used as a library absolutely inside everything from VLC to Chrome to your smart TVs to basically any video that you see online, um, you usually use FFmpeg. And FFmpeg in it has all those type of tools and sometimes depend on other libraries like x264, uh, Libvpx, and others, right? So it's really now the de facto, um, tool to process images.
- KKKieran Kunhya
From a philosophical level, I think it's incredible that your home videos, your, your grandmother's home videos and trillion-dollar corporations effectively are on a level playing field using the same technology stack.
- JKJean-Baptiste Kempf
Mm-hmm.
- KKKieran Kunhya
Uh, it's, it wouldn't be a surprise, you know, these big companies just have 3,000-line FFmpeg commands.
- JKJean-Baptiste Kempf
Mm-hmm.
- KKKieran Kunhya
There are some that use the API, but there are some that just have long command lines.
- LFLex Fridman
So yeah, there's a bunch of tools, like literally command line tool, FFmpeg, of course, FFprobe. There's libraries, Libav codec, Libav format, Libav filter. But the FFmpeg on the command line-
- JKJean-Baptiste Kempf
Yeah
- LFLex Fridman
... is, like, legendary because you can cut-- Like there's so many parameters. You can customize everything to hell.
- JKJean-Baptiste Kempf
It's a language. It's an actual language.
- LFLex Fridman
It's an actual om- yeah, you could think of it as a programming language.
- JKJean-Baptiste Kempf
Yeah, of course, I'm sure. Because so most of the people, they're going to take FFmpeg file in, file out, and specify the format, right? But you can-- We've seen thousands of characters, and we've seen also, like, people, like, doing, um, programming, um, uh, generation of command lines to, to, to make FFmpeg. There is a ton of people who are using AI to generate command lines for FFmpeg because you have no idea what it is. But you can do- specify so many filters, right, uh, on command line, right? So, um, FFmpeg is this collection of toolbox for multimedia processing that everyone, everyone uses, and everyone that is watching your videos are also using, right? You, you're on YouTube. Well, it's FFmpeg on, on the client side. Well, the, your server side, on the server side. The client side is probably Chrome. Well, you're using FFmpeg also.
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
And, and you're using OBS to record. Well, it's FFmpeg, right? You're using a ton of important, like, big box, professional boxes. Well, it's very possible that inside some part of FFmpeg is running.
- LFLex Fridman
I mean, there, there's like so many just to give people an idea. Like I, I use FFmpeg a lot on, on everything, just trivial stuff like, uh, take a video, add an intro video and an outro video, and fade one into the other, like, uh, what is it called? Dip to black, like where it, it dips and then shows the next video and does the same thing with audio. Uh, there's like a cross dissolve of the audio. It's a quiet-- It quiets the audio and makes it loud again. And then there's a bunch of stuff like, uh, showing the captions on screen hard, like baking the captions in. You can customize the font. You can do all kinds of layering of audio and video. There's a million things and, of course, all of that works, like, magically with basically any codec. Like anything you, you can shove in on the audio and the video side, it works.
- JKJean-Baptiste Kempf
But it's, it's like if you, if you look at, for example, you can do things that you would do with Adobe After Effects-
- LFLex Fridman
Mm-hmm
- JKJean-Baptiste Kempf
... in command line on FFmpeg, right? It's, and it's very interesting because, for example, for, for, for imagings, there is not such tool. There is a few tools, but not with the breadth of FFmpeg.
- LFLex Fridman
So ImageMagick has a similar kind of-
- JKJean-Baptiste Kempf
Yes, but you will not-
- LFLex Fridman
... spirit, but-
- JKJean-Baptiste Kempf
... do some filters, complex filters
- LFLex Fridman
Yeah.
- JKJean-Baptiste Kempf
You don't have the equivalent of Photoshop-
- LFLex Fridman
No
- JKJean-Baptiste Kempf
... in command line, right? But you, for video, you have FFmpeg in command line.
- LFLex Fridman
Yeah. It's incredible. I mean, it's like an, it's an example of a thing when a bunch of great people get together and they get a vision, and they stick by that vision for many years, which is incredible.
- JKJean-Baptiste Kempf
And the vision behind, and the same for VLC and FFmpeg, is that we make every- everything that is very complex easy to use for the normal people, for everyone.
- LFLex Fridman
Mm-hmm.
- 51:07 – 55:46
Linus Torvalds
- JKJean-Baptiste Kempf
right?
- LFLex Fridman
So one, one of the legends of the community is, uh, of course, Linus Torvalds, who, uh, created Linux and is a longtime maintainer of the Linux kernel. As the legend goes, he can be pretty harsh on this meritocratic process of reviewing the code and saying it's not good enough. Can, can you just speak to, to the legend of Linus Torvalds?
- JKJean-Baptiste Kempf
Linus is one of a kind, right? And, and I would even go and say that what he did on Git is more interesting than what he did on the Linux kernel.
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
He's very harsh, but what people don't see is usually when he's harsh to, it's people who are maintainer of part of the kernel, right? So they know him, right? So he's not very harsh like that to everyone. The thing is, what he created in his room is basically powering every server online, right? Even at Microsoft cloud called Azure, I'm quite sure seventy, eighty percent of the servers are running Linux. All your Android phones are running Linux. What he did, um, with the power of open, of open source, sure, is amazing. Um, and yes, the quality of the Linux kernel is very high, and yes, it's difficult, but we cannot compromise on that. We cannot compromise on quality because in the end, and you have to understand that, is the core community of VLC is five people. The core community of FFmpeg is ten to fifteen, and we are the ones who are going to maintain your code, right?
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
Because one thousand contributors in the timeline and just ten staying, it's one percent chance that someone comes and stays. One percent. So you will have change of job, change of wives, you have children, you have accident in life, you're going to change jobs, whatever. You're not going to come back. It's most likely. So we are the one going to man-maintain your code. It needs to be maintainable. It needs to be excellent. And yes, sometimes that means that you need to rework your work because it was good, but it's not excellent. And we need excellence because we are very few to maintain something that is critical for the whole.
- LFLex Fridman
But we should also mention that there is some, some spiciness, some harshness to the language that's sometimes used when you're, uh, keeping this high bar of excellence. Uh, is there something to say to that?
- JKJean-Baptiste Kempf
It's, it's true, right? Um, it's also the fact that, for example, what we're doing is low level, it's extremely technical. Uh, you get into this community, the tone gets, um, very like a type of-- It's a subculture, right? So people who arrive from the external are basically not known to the subculture. Most of those people around FFmpeg and VLC, we do VideoLAN DevDays, VDD, uh, every, every year. They are so fun in real life, and they love it.
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
But it's true that you're online and sometimes, like, the, the tone, you don't realize how it is. Uh, but that's okay.
- LFLex Fridman
It's a culture. I mean, you, you get this in the gaming culture. There's pretty harsh, intense, the way people communicate, and it's-- everyone understands that the way you show love and respect just looks different in different communities. Sometimes people... [chuckles] It de- it depends. If it's a book club, usually people are going to be much sweeter. If it's an open source [chuckles] project that's very high stakes and used by millions of people-
- JKJean-Baptiste Kempf
But it's very not often insults that you see, for example, in the gaming, right? And so Linus' tone is a bit unusual, even for the open source community. It's more like it's more harsh on the result, saying, "No, this is not good. This is crap." Those type of things that you will see.
- LFLex Fridman
Try not to make it about the person, make it about the code.
- JKJean-Baptiste Kempf
Yeah.
- KKKieran Kunhya
Yes. It's very, very matter of fact, and I think you've got to look at it in terms of, you know, the famous FFmpeg is developed almost entirely by volunteers, and that's true. And you've got to imagine someone's done a hard day's work at their day job. They come home. You know, terseness might be a thing, you know, it... And, and that's not something to take personally.
- LFLex Fridman
you're tired, you're busy, but you still care about this open source stuff.
- JKJean-Baptiste Kempf
Yeah.
- LFLex Fridman
But you may not be able to explain and, and handhold someone on every subtle detail.
- JKJean-Baptiste Kempf
And also you have to realize that most people don't speak English as native language.
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
And, and this is especially for, for open source projects like FFmpeg and VLC, which are mostly centered out of Europe. Sometimes like, um, people who are from the US or, or just like are very not happy about the tone, but most of the time it's also like they don't know better, right? It's difficult. Language is-- English is a difficult language. There is so many subtleties and tones and so on that you don't have, right?
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
So, um, often it's also difficult in those type of community about like different cultures and languages.
- 55:46 – 1:10:04
Turning down millions to keep VLC ad-free
- LFLex Fridman
So as the legend goes, JB, you, uh, repeatedly turned down millions of dollars to, uh, keep VLC open source free for everyone without ads. So take me through the reasoning behind that decision of leaving millions of dollars on the table.
- JKJean-Baptiste Kempf
Yeah, that's like almost a meme, right, on, on Reddit or-
- LFLex Fridman
There literally is a meme on Reddit. [chuckles]
- JKJean-Baptiste Kempf
9GAG and yeah, yeah, I see. There's-
- LFLex Fridman
You [chuckles] looking like a wizard in the, in the VLC hat on Reddit. This is, uh, JB, the creator of VLC media player. He refused tens of millions of dollars in order to keep VLC ad-free. [chuckles] Thanks, Jean-Baptiste Kempf. You can even summon him on Reddit. [chuckles]
- JKJean-Baptiste Kempf
Yeah. And usually if you see, right, it's usually like people tag me, right? And, and, and then there is me, and then like I say, "Good morning."
- LFLex Fridman
Good.
- JKJean-Baptiste Kempf
I got twenty-four K upvotes, which is great, right? My karma on Reddit is amazing. [both laughing] At least on that account. Um, so the question is, needs to be answered firstly, what is a story about VLC, right? Um, because yes, this is true, I refused dozens of millions of dollars, yes, several time. Yes, I could be a multimillionaire and be somewhere on the beach. Um, but I did not do it because I thought it was not moral, and it was not the right thing to do. And this is very important for myself, is to be like, I work for the greater good, I work for people, and I don't want-- It's not just by my, by myself. But the reason is also because I, I did not feel that I'm completely legitimate to do that, and let me explain to you why. VLC story is a very weird story. In France, we have university, and we have a type of top colleges, um, and those top of excellency schools are engineering school, business schools, and basically lawyers a-and medical, right? But they're outside of university, and in order to enter those, you spend two years working like crazy math, physics to enter those best engineering school. One of the school is called the École CentraleParis. It has changed name since, but it was called the École CentraleParis. And because it was Centrale, they had to move it because it was too small after the World War II, and, and they moved it-- They wanted to move it to the central of France in a place called Clermont-Ferrand, and the alumni decided that this was not okay, right? It is the, the school that Eiffel, right, the, the one who did the Eiffel Tower, attended to, right? So they said, "No, no, we are amazing, great school. We cannot do that." And so they bought a piece of land south of Paris, uh, very near Paris. Um, and it was a campus managed by a nonprofit of the alumnis, okay? Because of that, everything on the campus was managed by student. M- The university did nothing, right? So, uh, radio, uh, TV, uh, supermarket, uh, library, um, defining who was going t- into which rooms. Um, everything was managed by the student.
- LFLex Fridman
That's amazing. That's an amazing experiment, that it all, all didn't go to hell quickly. It somehow flourished.
- JKJean-Baptiste Kempf
It worked great, and I learned so much, um, in my life doing those side activities, right? Because you're, you're twenty-two, and you need to run your campus, else you don't have electricity, right?
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
So you care about that, right? But anyway, in the '80s, um, they did a full experiment of deploying a network, uh, mostly sponsored by IBM and 3Com, which was a token ring network. So token ring is something that probably almost no one knows about anymore. It's a networking technology where you don't have routers, right?
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
Everyone is linked. Um, it's type-- like really a ring, and when you want to send a message, you talk to your neighbor, who's going to put the message to the next one, who's going to put the things to the next one in terms of ring. The issue with token ring is, of course, is that it's very slow because every computer on the network needs to open the message, see if it's okay. Is it for me? No, it's not, and then send it back like a token, which is traveling around the ring. In the '80s, you're doing some Telnet and sending mails as university. That's okay, right? But starts the '90s. I-In the '90s, um, and start video games, and when you have li- high latency in video games, basically you die, right? So in nineteen ninety-four, nineteen ninety-five, around Doom and Duke Nukem coming around, they want a faster network. So the students go and see the university and say, "You know what? We want a faster network. We need to work," which, and also play video games. Um, and the university tells them that basically, "Oh, I'm sorry. We cannot help you because you understand the campus is not ours. You manage it, so do something. And you should see some basically partners of this university and basically go away." And they go, and they actually go and see, um, the, the, the CIO of, uh, Bouygues, which is a large, uh, French company and who's doing some TVs, uh, in France. And he says, "Well, you know what? The future of video is satellite." Well, today we know it's not, but at least it was a good idea. In nineteen ninety-five, the first of satellite d-dish, and he says that instead of having like one satellite dish and a big decoder for each of the students, which are one thousand and five hundred, what about you build, like you put a n-Enormous dish and only one decoder, and you send the video directly on the network.
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
And that required a very fast network. Today, it's obvious, but at the time was, like, the first to do video streaming. So they built this project, which was called Network 2000. Of course, right, we are in the '90s, right? Everything is-
- LFLex Fridman
[laughs]
- JKJean-Baptiste Kempf
... futuristic is called 2000, like-
- LFLex Fridman
Yeah, 2000, yeah.
- JKJean-Baptiste Kempf
And so they do the Network 2000 project. It's completely hacked. It crashes after 45 seconds. That's okay. The demo is 40 seconds. It leaks memory. That's okay. They put 64 megabyte of RAM instead of the 8 or 16 you have, and the demo should have stopped there, and that was the Network 2000 project by the students.
- LFLex Fridman
What was the format of the video that they had to work with?
- JKJean-Baptiste Kempf
MPEG-2 because satellite is MPEG-2 TS for transport, MPEG-2 video, and MPEG-2 audio at that time. And the project should have stopped there. Everyone was happy. They had, like, amazing ATM network at 155, 55 megabits per second. They had probably one of the best network in Europe at that time, and they stopped the project. Six months or a year later, two students arrive and say, "Well, you know what? Maybe other people care about video streamed on a local network," and they create the VideoLAN project, VideoLAN. And one of them is called Christophe Massot, that is a good friend of, uh, both Kieran and me, and they start the project. It's not even open source yet, and they spend around three years to get the school to agree to make it open source. Because the, the university wanted to get some-- because of the IP and copyright of the students, wanted to basically monetize these MPEG-2 decoders.
- LFLex Fridman
Just to be clear, so what's the main application? Streaming on a local network?
- JKJean-Baptiste Kempf
It was streaming on a local network.
- LFLex Fridman
By the way, that's just, like, to state the obvious. This is before YouTube. This is before-
- JKJean-Baptiste Kempf
Ten years before YouTube. You have a Pentium 60 or 75, right? Y-y-you, the main machine was 48, 86DX at 33 megahertz, right?
- KKKieran Kunhya
Bear in mind, tele-television was the main form of video at the time. You could get new channels. In the '90s, having even one new channel when you, you grew up with four channels, having a fifth or a sixth-
- JKJean-Baptiste Kempf
Mm-hmm
- KKKieran Kunhya
... was a big deal, and so having this satellite service with, you know, dozens, even hundreds of channels was so groundbreaking.
- JKJean-Baptiste Kempf
Especially because this is university where you had ton of different nationalities, right? So there was a ton of people who wanted-- So the-- in the end, they had, like, several dishes on different type of satellite, right? Because, for example, a lot of people were coming from, uh, the Maghreb or the Middle East and, and they, they, so they went to different type of, uh, uh, of satellites. Anyway, the solution w-worked great, and they started the VideoLAN project. The VideoLAN project has several s- and, and some are completely crazy solutions, like one how to create multicast on an unicast network, but let's not come to that.
- 1:10:04 – 1:29:18
FFmpeg & Google drama
- LFLex Fridman
Let me talk a little bit more about the open source movement, about the fact that, as you say over and over and over and over, FFmpeg, uh, is, uh, and many open source projects are built by volunteers. So, uh, there's a bit of drama recently, uh, Kieran, on the interwebs, on Twitter. Uh, you have a spicy style on Twitter that I think articulates and celebrates all the incredible developers and, uh, development and the, uh, the code, especially Assembly that's involved in, uh, building some of these codecs and building some of this incredible technology. But that brings us to the, a bit of a debacle that happened. Tell me the full saga of what happened with the Google security engineers.
- KKKieran Kunhya
Just to be clear, Google are one of the biggest supporters of open source out there. They have been for a long time. It's just, I think some things kind of went a bit overboard this time. So FFmpeg itself, and, and this is not like a secret, it's on the homepage, you know, the, it processes untrusted data. There can be security issues when you parse untrusted data. That's very normal. But recently what changed was Google started using AI to create security reports on an open source project, FFmpeg. Volunteers had to deal with that. They did-- They provided very limited funding and they even went to the media first announcing how good their AI was before the issues could be fixed.
- LFLex Fridman
And this is in the public forum.
- KKKieran Kunhya
Yeah, this is all public.
- LFLex Fridman
So reporting, reporting an issue using AI to find an issue in the code, which is a security vulnerability, and then reporting that publicly before you're able to fix it.
- KKKieran Kunhya
Yeah. It's announcing how good their I- AI is, that they provided a standard 90-day industry deadline, um, without poten- without really understanding the nature of volunteer-driven development. In addition, this vulnerability was on an obscure 1990s game codec. Um, the way-- And f- and let's look at it from their standpoint to begin with. Let, let's, um, you know-
- LFLex Fridman
Yeah. Can you steer me in their case?
- KKKieran Kunhya
Yeah, sure. They have substantial resources working on the security of open source projects that, you know, are ubiquitous and they've used, you know, a lot of compute to do that and very expensive and very capable security researchers, um, to do that. And that, that's their viewpoint is they are contributing by doing that. But I think that's where opinions differ. Um, it, it, it opened up a lot of interesting fissures, uh, I would say. Um, it does seem that there's a portion of the security community that, um, look at themselves a bit like building architects that never have to go to site. You know, going to site is something that is a little bit beneath them, the actual day-to-day construction. They're there to do their security things and it's someone else's problem. The security industry also kind of has a very aggressive tone towards things. The, the language they use is extremely aggressive. They use very strong language like, "You will get popped." Um, so and to, to Joe Public, get popped, you know, means something quite bad. Um, for them it means to get hacked. The way I would look at it personally is a little bit like the padlock on your home. Um, not everyone-- The padlock on your home or, you know, the lock on your home is there to, to protect against thecapabilities of, of what it's there to protect. It's not there to protect nuclear secrets. It's not there to protect Fort Knox. And it could be looked at that they're using AI at a level of scale to go and pick those locks and then say, "Hey, your locks, your lock's not secure. You need to deal with this." Whereas actually they're the ones with resources to be able to, to fix this. But that seems to not be something either they'll contribute to in terms of patches or in terms of financially. And the scale of AI is kind of the issue. The, the, the bug reports are very wordy. They're very, very... It's almost a denial of service by AI-generated bug reports on very niche codecs. Um, and the other issue the security community has is everything is marked high priority. You're going to, you know, "This is the most important thing in the world, and you need to deal with this. High, high, high, vulnerable, scary, scary, scary," on a game codec used on one disk in 1993.
- LFLex Fridman
Yeah.
- KKKieran Kunhya
And that, that's where the dichotomy lies. Going around telling everyone that their padlock's not safe, well, that's a hobby project of somebody. The, the safety of that codec is consummate to what that person thinks. It's their hobby. It's good that they're security analyzing it, but it doesn't need a big scary warning, "This is a critical vulnerability." Um, may also, may, may recently also, um, see that, um, there was another quote-unquote vulnerability. It wasn't at Google in this case, but, um, a filter could overflow and have an integer overflow, and one of your pixels could be the wrong color, and this was marked high, 7.5 severity in red. And at some point, the security industry needs to realize you can't keep crying wolf like this because this just leads to people, you know, the equivalent thereof of putting password stickers on their PC. You know, you can't just keep crying wolf every day. And, and I appreciate, you know, that's their modus operandi is to create as much scare and fear. But from the Google standpoint, at the end of the day, they need to contribute either financially or with patches. Google uses FFmpeg at, at a scale probably you or I couldn't even contemplate, millions of CPU cores. And yes, they contribute in areas mostly regarding their own products, so VP9, AV1. But in a wider sense, there's a disproportionate level of contribution. Yes, they fund students. Yes, they fund Summer of Code. And I think, uh, so Alex Strange is a former FFmpeg developer, um, I think posting in a personal capacity.
- LFLex Fridman
So he, uh, posted about security engineers on, uh, Hacker News. Uh, his post reads, "The problem with security reports in general is security people are rampant self-promoters," in parentheses, "Linus once called them something worse. Imagine you're a humble volunteer open source developer. If a security researcher finds a bug in your code, they're going to make up a cute name for it, start a website with a logo. Google is going to give them a million-dollar bounty. They're going to go to DEF CON and get a prize, and I assume go to some kind of secret security people orgy where everyone is dressed like they're in The Matrix. Nobody [chuckles] is going to do any of this for you when you fix it." [both laughing] So basically commenting on the sort of the incentives for the different, uh, people involved and misaligned.
- JKJean-Baptiste Kempf
The problem here is, is a disproportion of means on discovery compared to patching it, right? And this is the biggest issue, right? Um, a- and after that debacle, Google did some changes-
- KKKieran Kunhya
They are now starting to send patches, which is, yeah
- JKJean-Baptiste Kempf
... And they also now have reward tools for fixing issues. So it, it has changed a bit because of that, uh, debacle. So it's good, right? But we've seen, and we talk about Google, but we have seen like some, some other large companies saying, "Oh, you need to fix this bug because it's critical in our product."
- LFLex Fridman
Can you explain the XZ fiasco? The FFmpeg tweet reads, "The XZ fiasco has shown how a dependence on unpaid volunteers can cause major problems. Trillion-dollar corporations expect free and urgent support from volunteers. Microsoft, Microsoft Teams posted on a bug tracker full of volunteers that their issue is high priority. After politely requesting a support contract from Microsoft for long-term maintenance, they offered a one-time payment of a few thousand dollars instead. This is unacceptable. We didn't make it up. This is what Microsoft, Microsoft Teams actually did." And then they, uh, you give the image and the details and all that kind of stuff, showing that these trillion-dollar companies are not giving much money, not giving much support.
- KKKieran Kunhya
They think, they think an open source project is a traditional vendor that they have an SLA. They think a public bug tracker is actually, you know, a, a third-party vendor's Jira where you can do all of these things. It, it, it's not. It is there to report bugs. I think the thing that made this particularly heinous was the name-dropping of Microsoft, the name-dropping that this is a visible product. If this was, if this was a, just a general bug report, I think that would have made it a lot better.
- LFLex Fridman
Yeah, so they literally said, like, "This is a big deal because a lot of people are using it in Microsoft." I wonder what happens psychologically. So I think what happens in these companies, maybe you can correct me, is they... Y- you're, you're right. They just think of FFmpeg as like a vendor that Microsoft surely is paying a huge amount of money to. They kind of assume that in their interaction, and nobody anywhere on the stack is going like, "Wait a minute. Shouldn't we be giving like millions of dollars to FFmpeg?"
- JKJean-Baptiste Kempf
And this is a very big problem in, in large... Like we're talking about some companies, but it's the same everywhere, right? Um, a lot of those companiesLike, the, the-- when we talk to that person, right? He was just like a, a manager on one project in Microsoft Teams, right? He had, like, never really, mm, discussed with open source community. He had no idea, right? It was like... A-and but the problem is that usually there is what we call OSPOs, right, open source program offices, in those type of companies, and they are the ones who are supposed to discuss with open source vendors-
- LFLex Fridman
Mm-hmm
- JKJean-Baptiste Kempf
...um, or, or v- open source communities. But, like, they often don't explain that correctly internally, right? And here it's just like, "We are not your supplier. If you want me to be your supplier, I'm very happy, right? I will send you a contract and SLAs." Like, I created f- five companies who are doing that around open source projects, so that's okay.
- LFLex Fridman
We should say that some, some of the spicy tweets that Kieran, you're behind, and some of the debacle produced results.
- KKKieran Kunhya
Yes.
- LFLex Fridman
Positive results.
- KKKieran Kunhya
Donations have increased substantially. They're still not enough to cover even a single full-time developer, but on both a, you know, awareness level and a technical level, there's substantially more technical awareness and sort of awareness of the importance of FFmpeg as a result, as a result of X and, and what's happened. I can say, you know, it, it, it solved its purpose. People realize the level of importance FFmpeg has.
- JKJean-Baptiste Kempf
And on VideoLAN, it's the same, right? Like, for example, a, a very simple example. For more than a year, we couldn't update VLC on Android because of a bug on the Play Store, on Android Play Store, right? The only way we got someone to answer was to put an very spicy, as you say, um, tweet saying that we are going to stop distributing VLC for Android, right? And we have around one hundred million people using that. Um, and now then someone from Android actually came and discussed to us, right? Um, we had the same issue with, with, uh, Microsoft or, or like saying that we were going to stop distributing VLC on the Windows Store. And unfortunately, we are so small that the only very strong power we have to solve those issues is blaming on social network because it snowballs and now they listen to us. But so as large companies often have difficulty talking to us. Like, for example, VLC, right, is probably one of the top 10 software used on Windows. I am not part of Microsoft ISV programs, right? I don't have a point of contact at Microsoft, right? Well, I'm sure any other software, Adobe, Spotify, has a co- point of contact. I don't have that, right? So raising awareness works. It's sometimes very spicy, lot of drama. Well, X and Twitter are okay for that, but it's efficient.
- LFLex Fridman
Uh, so everybody listening to this should go, uh, follow FFmpeg on Twitter, on X, follow VideoLAN on Twitter, on X. Uh, go donate. Donate-
- KKKieran Kunhya
Thank you
- LFLex Fridman
...to FFmpeg.
- KKKieran Kunhya
And thank you, Lex. Over the years, several years, you've been a supporter of, you know, FFmpeg and VideoLAN on X, you know, sh- giving us shout-outs, appreciating, you know, what we do.
- LFLex Fridman
FFmpeg for life.
- 1:29:18 – 1:35:55
FFmpeg developers
- KKKieran Kunhya
Uh, what do you think motivates all the incredible contributors that we've been talking about? Like, what's the, what's the engine? It's so interesting to see.
- JKJean-Baptiste Kempf
So-
- KKKieran Kunhya
Like you said, they're sitting in the basement. What's the driver? What's the engine there?
- JKJean-Baptiste Kempf
There are many, uh, drivers, but weirdly the main one is that what we do in multimedia plays videos, and video is cool, right? And, and for example, we have so many people in the community who arrive because they loved watching anime, right? A- and this is like the advice when people ask me, "What should I work on in open source? How do I start?" And my quest- my answer is always the same. Work on something you love.
- KKKieran Kunhya
Mm-hmm.
- JKJean-Baptiste Kempf
I am working on VLC because I love movies, right? And I love watching the same movies over and over, even if my wife hates me when I do that, right? But because it's interesting, right? Because it's a topic that you like, right? The first, that's the first thing where people come to usually to VLC and FFmpeg. The second thing is that technically we, because we search for excellent, this is the best school ever, right? This is the best school ever of programming. If you're good in C, in FFmpeg, if you know how to write Assembly, I assure you you're going to be one of the best programmers ever, even if you're working on writing TypeScript, because this is the most amazing thing to do. And you will, like, have to get reviews by some of the most seasoned programmer ever who are going to look at every part of your code and tell you why it's not great. It's like we are the best teachers that you've ever had in programming, right?
- KKKieran Kunhya
Andrew Kelley started Zig. He was an FFmpeg developer and started Zig after-
- JKJean-Baptiste Kempf
Mm-hmm
- KKKieran Kunhya
... his FFmpeg school. I mean, it, it's the place to learn so many aspects of programming in the real world, in a thing used by billions of people. You have nowhere to hide. You have to be open and honest about your flaws and, and how you can learn and be better.
- JKJean-Baptiste Kempf
And what is also interesting in multimedia is that you have 16 milliseconds to display a frame. It's not like a game engine where you can basically slow down and wait a frame. Like, so it's, you need to be good, right? There is no choice, else you don't have your video. It, and, and because of how codecs, if you miss a frame, you're going to destroy the look of the video, right? So you need to be good. You need to be perfect to have the right thing. But also is that it's not just pure programming in the mathematical fence, right? A lot of people don't understand, but, um-In order to program correctly on the open source multimedia community, you need to understand how computer works. And, and when you write assembly, you need to understand about, uh, CPU pipelining, right? You need to understand how SIMD works, how the ALU works, right? You need to understand what, how IO works, right? And this is what I think that is missing to a lot of engineers and software engineers today, is understanding what we call computer architecture. Um, and, like, seriously, like, some of the debates is like, "Should we use this assembly call or this one?" And people say, "Well, no, it's going to be, like, three cycle on this, uh, type of CPU and this one," and has massive impact on the output, right?
- KKKieran Kunhya
We should expand, FFmpeg is probably one of the biggest CPU users in the world. There's proba- it's probably running-
- JKJean-Baptiste Kempf
Mm-hmm
- KKKieran Kunhya
... as we speak easily 100 mil- order of magnitude 100 million, maybe even a billion CPUs as we speak. So every instruction matters. There's not... The, the, the impact, at least in terms of CPU, is massive for everything that we do.
- JKJean-Baptiste Kempf
So first you come because it's an interesting subject, then you stay because it's excellent, and in the end, you're very proud of it because it's on the end of everyone. Like, so many people like, "Oh, I'm working for whatever consulting company, and I'm doing some portal to download invoices for your PG&E." Wow, great. Like, so many w- jobs are like that. You're not going to, to tell that to your grandma. But if you go to see your grandma and say, "I do this so that you can play video on your laptop," they understand. And this is very important, right? Because you're working on VLC, FFmpeg, x264. It's in the end of hundreds of million of people, and you have an impact. Um, and so you can be proud of yourself. And so I think that in addition to great, doing a great resume, all those things are why people contribute.
- KKKieran Kunhya
Yeah, th- those are side effects. My favorite quote on this topic is, uh, John Collinson. He said, "The world is a museum of passion projects." You know, everything out there is, is a passion project. And open source multimedia and open source in general, you can just do that so much faster. There's such a faster network effect, you know?
- JKJean-Baptiste Kempf
Mm-hmm.
- KKKieran Kunhya
I can open a cafe, and that can be my passion project, but I have to get building codes, I have to build a building, I have to find a th- a, find a location, I have to do all the, you know, all sorts of things. Well, in the software world, that passion project can be, can move quickly, it can be amplified by the network effect, and that amplification can be more than the, the, the sum of, the sum of the parts. You know, you can be, you can find people interested in extremely obscure things and have a network effect and make something that is truly amazing.
- LFLex Fridman
And, uh, on that topic of passion projects, uh, Tim Sweeney actually said in a reply to a tweet that was, uh, complimenting JB, he said, quote, "Many things in the world only happen because an awesome person decides to do it. This is the case with VLC." And that speaks to something interesting to me, that it does seem that a small number of people, sometimes one person, can create something incredible in the software world. Like you said this over and over and over. I think JavaScript is an incredible thing created by, by, uh, initially a single person. The, some of the programming languages like Python and C and Java, like just one person has this vision, has this design, and brings it sometimes over a weekend is the initial spark.
- JKJean-Baptiste Kempf
Yes, Linus built Git in two weeks.
- KKKieran Kunhya
Yeah.
- JKJean-Baptiste Kempf
Wow.
- KKKieran Kunhya
Yeah.
- LFLex Fridman
Yeah, it changed the world, Git. I mean, it really changed the world.
- KKKieran Kunhya
Linus' passion project. "Hey, I- uploading this tarball to an FTP, like, deal with it." [chuckles]
- JKJean-Baptiste Kempf
But for me, it's not just in software, right? And I believe in, in, in individuals that are going to change the world, right? And it's with a good, as you said, vision, right? I want to do that. Um, it is useful, it will be useful. And whether it's going to, like, build train or cars or rockets or something, like I, I believe people who believe in themself and have a vision can have a huge impact for humanity.
- 1:35:55 – 1:40:29
VLC and FFmpeg
- LFLex Fridman
Let's actually s- uh, zoom out before we zoom back in. We'll just keep going up and down the, the stack. So, uh, you know, we've been talking back and forth VLC and FFmpeg. Kieran, you said that FFmpeg and, uh, VideoLAN, VLC coexist, and there's no central point of importance. It's a kind of what you call the binary star system.
- KKKieran Kunhya
Yeah.
- LFLex Fridman
Uh, they succeed because of each other. Can you explain the difference, how they interact? What is the-
- KKKieran Kunhya
Sure
- LFLex Fridman
... Are they competitors?
- KKKieran Kunhya
I don't, I don't think they're competitors. I think, I think the simple answer is, the short answer before I go into detail is VLC is to FFmpeg as Android is to Linux. So they depend on each other, but they, they coexist because of each other. So they are a binary star system is the analogy I used.
- LFLex Fridman
By the way, I feel horrible that I just recently learned that Alpha Centauri, the closest star system to us, is a triple star system.
- JKJean-Baptiste Kempf
And, and when you start doing the physics, it's a nightmare, right?
- LFLex Fridman
Yeah.
- KKKieran Kunhya
Oh, yeah.
- JKJean-Baptiste Kempf
But, but, but like-
- KKKieran Kunhya
Hence the three-body problem. But anyway. So a lot of FFmpeg pipelines involve the x264 project, which is a VideoLAN project. I w- I would put a finger in the air that's, and say 80-plus percent of those pipelines are dependent on a VideoLAN project. VLC, obviously, as we've discussed, a VideoLAN project, uses FFmpeg, gives it reach, exposure to weird files, uh, historically used some donation money to fund FFmpe- FFmpeg development, and we'll talk a bit maybe about some of the reverse engineering later. So it's a binary star system. They, they work and feed off each other. Many of the developers are shared. There's no central location. It's a virtuous cycle working together.
- LFLex Fridman
And we should mention that x264 is the encoder for H264 video standard. So H264 is the standard.
- KKKieran Kunhya
Yes.
- LFLex Fridman
x264-
- KKKieran Kunhya
Is the implementation, open source implementation of the standard
- LFLex Fridman
... that's used by basically everybody-
- KKKieran Kunhya
Yeah
- LFLex Fridman
... for everything.
- KKKieran Kunhya
Yeah
- LFLex Fridman
It's that, that is the main driver of this. When you think of an MP4 file that has H264 codec in it
- KKKieran Kunhya
If it, if it came from a software environment like a data center or somewhere, the chances are it was created with x264.
- LFLex Fridman
And that's under the flag of, uh, VideoLAN.
- KKKieran Kunhya
That's a VideoLAN project. So in the VideoLAN graphic, it sits in the VideoLAN world.
- LFLex Fridman
And VideoLAN has a, says a bunch of stuff in it. Go to the VideoLAN website, there's a bunch of icons.
- JKJean-Baptiste Kempf
Like if you look, there is so many libraries, right? Um,
- LFLex Fridman
Lib DVD CSS.
- JKJean-Baptiste Kempf
Yes.
- LFLex Fridman
Lib DVD Nav, Lib DVD PSI, Lib VLC of course, VLC Unity, Lib blu- [laughs] Lib Blu-ray.
- JKJean-Baptiste Kempf
Lib Blu-ray.
- 1:40:29 – 1:43:46
History of FFmpeg
- LFLex Fridman
it. Uh, can we give a shout-out to some of the people along the way? We th- didn't really quite talk about the history of FFmpeg, so, uh, m- maybe can, can you tell me about Fabrice? Can you tell me about Michael Niedermayer? Can, can you tell me about some of the key figures here?
- KKKieran Kunhya
Let's just talk about the, the eras of FFmpeg, 'cause there, there's key eras that, and key people that made this possible. Um, Fabrice Bellard, as you mentioned, creating the concept, and then probably in the 2000 era, I would call the era, eras tour of FFmpeg is the 2000 era was Michael Niedermayer. So key things he got done was exhaustive support for DivX and Xvid at the time, and all sorts of weird variants of what's known as MPEG-4 Part 2. So this predates the MPEG-4 Part 10 that we're used to. So this was 2000 era video codecs where there were, oh, flavor after flavor of weird, weird decoders.
- LFLex Fridman
Mm.
- KKKieran Kunhya
At the time in the 2000s, you needed a new player to play every different type of file format. So there was Windows Media Player to play Windows Media formats. There was RealPlayer to play RealMedia formats, and those were the other, the other key thing in FFmpeg at the time were native decoders for those. I actually do remember being a teenager, I must have been, figuring out there was this one player that could play, could decode these files without having separate bloated players. 'Cause at the time when you downloaded RealPlayer, there was a ton of other stuff in there, a ton of ads, a ton of other things, and just having a simple library that was fast led to that. And then I think 2008 was a, 2008 onwards was a big change because that's when H264 got its maturity and, and I think something hopefully we'll talk about a bit more. This was the beginning of high definition video. So H264 was the key decoder of that. So I'd call that the late 2000s and 2010s, and that's when the big reverse engineers came along and really did astonishing work. The beginning was a single player that could play Xvid, DivX, Windows Media, and RealPlayer was already in a massive achievement in itself without codec packs, without weird stuff you had to download that had weird ads and weird spyware.
- JKJean-Baptiste Kempf
VLC 1.0 was out on those times, 2000, 2009, 2010.
- KKKieran Kunhya
Yeah.
- JKJean-Baptiste Kempf
And this is, like, where it exploded.
- LFLex Fridman
Yeah, without codec packs, it just works-
- KKKieran Kunhya
Yeah
- LFLex Fridman
... across all these different-
- JKJean-Baptiste Kempf
It, the fact though, it's just like all the codecs packs are FFmpeg inside VLC, plus we have other modules for all the type of codecs.
- KKKieran Kunhya
But at, at the time that wasn't, is there were weird... In the 2000s, there were weird codec packs with DLLs coming from this place, DLLs coming that-
- JKJean-Baptiste Kempf
With a lot of spyware
- KKKieran Kunhya
... with spyware, with you know what. You, it wasn't reliable, you didn't know, and having a single player that was open source or single playback module/player that could do this, that was open source. But I think the thing to emphasize is this task in the 2000s that Michael did was, was Sisyphus, and it was really the number of edge cases are, were beyond comprehension in terms of you could have a Chinese CCTV system that did one weird variant of MPEG-4 Part 2, mem- what's known as MPEG-4 ASP, and that was a weird variant, and you had to fix that without breaking everybody else-
- JKJean-Baptiste Kempf
Mm-hmm
- KKKieran Kunhya
... times a million.
- LFLex Fridman
So that's, uh,
- 1:43:46 – 1:57:01
Reverse engineering codecs
- LFLex Fridman
so you said that- that's where a lot of the reverse engineering was happening.
- KKKieran Kunhya
It start, it started in the 2000s with the Windows Media stuff-
- LFLex Fridman
Yeah
- KKKieran Kunhya
... 'cause that was proprietary. It started with the RealMedia, started with Benjamin Larsson.
- JKJean-Baptiste Kempf
Kostya Shishkov.
- KKKieran Kunhya
Kostya Shishkov, that era. Those were the key, that was the key groundwork.And then in, in the 2010s was kind of the Paul Maheral, Kostya era building, doing, doing some of the most difficult codecs. Um, JB maybe can talk about GoToMeeting 4 and GoToMeeting 5, and-
- LFLex Fridman
What? What's the, what's the GoToMeeting?
- JKJean-Baptiste Kempf
So, so, so, so, so, like, let- let's talk about this amazing Ukrainian guy called Kostya, who was, uh, at that time living in Germany and who was in love with Sweden, right? He-- And the guy was the most... He, he's like, like, like a lot of the people in the community are very clever. He's one of those who are like borderline geniuses, right?
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
He was able to reverse engineering extremely complex codecs. Um, and he, he does that, and we do a to- a bit of engineers with Kieran, but clearly not at this level. [chuckles]
- LFLex Fridman
No, no, no.
- JKJean-Baptiste Kempf
Um, he reverse engineered binary blobs, which are 20 megabytes?
- KKKieran Kunhya
Yeah, so just for reference, uh, um, one megabyte binary blob to reverse engineer is probably order of magnitude a month of work, and this guy is doing 20, 30 megabyte blobs. Maybe we'll talk about that in a minute, about the subtleties of how you do that. But this guy is doing it for very difficult and very obscure codecs.
- JKJean-Baptiste Kempf
And, and did that for fun, right?
- KKKieran Kunhya
Yeah.
- JKJean-Baptiste Kempf
And, and, um, so GoToMeeting, uh, was a big problem with VLC because that was like the number f- number one, uh, feature request for a long time, so I put a bounty. A-and the guy at some point said, "Okay, JB, I'm going to do it." And in a matter of two months, and then he explained how he did it. He was just like, "Oh, I looked at the code, like this looked like a, a, a DCT that I used to see on WMV and so on." He did that, and the funniest part is that the code he's wr-written is a ton of jokes. And there is, there is a ton of JB, right, my name, and, and Kempf, um, and Kempf and Kostya, uh, jokes inside the code. The code is beautiful, right?
- LFLex Fridman
So one of the things I, I, I wanna comment is, uh, I've gotten a chance to speak to some of the developers, some of the as- assembly language level, uh, uh, people, and they all always make everything sound like it's kinda easy. There's a, a kind of humility because, I mean, maybe [chuckles] just the level of what's required to do this stuff is so high that everything else seems easy, I guess is the lesson to take away from that.
- JKJean-Baptiste Kempf
So in the community, like some of the most impressive people are the ones doing reverse engineering-
- LFLex Fridman
Yeah
- JKJean-Baptiste Kempf
... and the other ones doing the assembly folds, right?
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
Um, and both of those type of people are amazing. Um, x264, for example, became amazing because, uh, of a, a guy called, uh, Lorraine Merritt-
- KKKieran Kunhya
Yeah
- JKJean-Baptiste Kempf
... who is, was from University of Washington, I think.
- KKKieran Kunhya
At the time, yeah.
- JKJean-Baptiste Kempf
And who was like, who made everything great and fast doing a ton of assembly. Um, uh, yeah. So this is like the, the, the, the golden era, I guess, where so many things got done.
- KKKieran Kunhya
Yeah, if you look at Kostya, for example, he looked at the world as a binary specification. He, he didn't need documentation or anything. It's, "I have a binary and I can figure all of this out." And, and he, he regularly used the phrase binary specification. Ah, you know, it's not a problem. And he went and he would go away, and he would come back, and he, he would do interesting stuff.
- LFLex Fridman
Can you, uh, actually speak to the details or any, any add color and texture to what it takes to reverse engineer a blob?
- KKKieran Kunhya
Yeah. So, so let's look at GoToMeeting, for example, is a good one because, um, I record a meeting, uh, on GoToMeeting, for example. How do I play it back without needing this GoToMeeting player? There may not even be a player. I may, I may need to send a recording of a meeting to someone that doesn't have a player or whatever. So fi-first of all, there's a ton of other stuff there. There's a, a actual video conferencing client. You need to go and find... It may be easy, it may not be easy to find the actual module doing the decompression. You need a way to actually dump the YUV data from the module. So often it rever- it involves opening in a disassembler, trying to guess where the hooks are to incorporate that module and run that module natively to decode a sample file. So figure out where this module is doing the decoding process and, and find a way to hook in and output the raw YUV data, 'cause you will need that-
- LFLex Fridman
Mm-hmm
- 1:57:01 – 2:01:08
FFmpeg testing
- LFLex Fridman
That reminds me of a thing that you, uh, mentioned to me about FATE testing and, like, the insanely rigorous process that's used to test everything that's incorporated into, um, into FFmpeg. Can you take me through the testing process?
- KKKieran Kunhya
Yeah. So FFmpeg has a system called FATE, FFmpeg Automated Testing Environment.
- LFLex Fridman
Mm-hmm.
- KKKieran Kunhya
Because FFmpeg runs on so many different OSs and can be compiled with so many different compilers, there's been a crazy number of configurations. So you can see the absurd combination of, um, compiler variants, operating system variants, instruction sets. You can see at the top macOS has tons of different variants because it has iOS, it has tvOS.
- LFLex Fridman
Well, I'm looking at a page fate.ffmpeg.org 81 minutes ago, 76 minutes ago, looking at the different architectures, the operating systems, the different compilers, Apple Clang version, uh-
- KKKieran Kunhya
Combinations are crazy
- LFLex Fridman
... the combination is insane. RISC-
- KKKieran Kunhya
So these are all run by volunteers, so these are all volunteer systems. The, the ones at the top, for example, the Macs I host in my office, for example, uh, host all sorts of different stuff. Other people host other things. So it's really there to make sure, um, c- because FFmpeg does quite complex C code, for example, you do have miscompilations. So the compiler will sometimes compile C code incorrectly. For example, this happens once in a while.
- LFLex Fridman
Oh, there's like, there's a log of all the compilations.
- KKKieran Kunhya
Yeah, log of all the compilations, all the tests. I think one of the other ones will show all the tests passing.
- JKJean-Baptiste Kempf
If you click, you can see all the tests-
- KKKieran Kunhya
Yep
- JKJean-Baptiste Kempf
... back. All tests successful.
- KKKieran Kunhya
In logs test, yeah. So you see all those tests are passing of all the different codecs, all the different filter transformations, all the-
- LFLex Fridman
[laughs]
- KKKieran Kunhya
The, the level of scale is, is quite crazy.
- LFLex Fridman
Oh, that's nuts.
- KKKieran Kunhya
On all the combinations. It's, it's not just a matrix at this point. It's like a pivot table of different combinations.
- LFLex Fridman
That's nuts.
- KKKieran Kunhya
And it, it's a key part of what we do because you, you may be able to test something locally, you make a change, but actually that breaks GCC version 11 on Mac or something like that, and you're able to then fix that. We also have miscompilation, so the C code, sometimes the compiler can have a bug in it where it creates the wrong output, and that can have quite a big effect sometimes on a vid- on a video because of the way frames have dependencies. Even a small change in the output can, can cascade to actually quite big glitches.
- JKJean-Baptiste Kempf
You see PowerPC, you see RISC, you see ARM.
- KKKieran Kunhya
There was PowerPC, there was RISC, there was weird stuff in the past like DEC Alpha. There was-
- JKJean-Baptiste Kempf
You see Visual Studio, different versions of Clang or GCC
- KKKieran Kunhya
... Visual Studio, Intel compiler, Apple Clang, you name it.
- LFLex Fridman
What are some of the pain points? Like maybe do you have emotional triggers, uh, maybe nightmares about a particular operating system, a particular container, codec combination of-
- KKKieran Kunhya
I mean, for, for me it's really easy because, uh... So I, I have a day job. Um, my company builds, the company I started builds, um, equipment for broadcasting sports matches between TV stadiums and studios, for example. We have to work with 10-bit video, and 10-bit video has, um, a set of challenges that you can't process 10-bit data natively on a CPU. So that means you have to stick it in, um, 16 bits. So that means you have six wasted bits. So there's different packing formats to actually pack the data more efficiently because when you send that over a network, you lose... 'Cause you need to, you need to save that 40%. For example, on PCI Express, you may only have bus bandwidth to do that. And so I think internally we have about... Some, some are, some are industry ones and some are internal to our own hardware that we build. We have a, I think a five by five or six by six matrix of every single format to every single other format conversion. In fact, one, one of them I sent you, and they're all written in handwritten assembly, and they're all written, and they all support different CPU generations. So this is really traumatic, handling all these different combinations times a million.
- LFLex Fridman
By the way, the company you're talking about is Open Broadcast Systems.
- KKKieran Kunhya
Yeah, so no, no relation to the free OBS streaming service.
- LFLex Fridman
Yeah.
- 2:01:08 – 2:25:26
Assembly code (handwritten)
- LFLex Fridman
Yeah.
- KKKieran Kunhya
Um, but JB and I have started companies broadly speaking around the FFmpeg VLC ethos, so that's really low-level work. So, so in most companies, this, this wouldn't be written in assembly. C- It would be accepted that C's, C is fast. Um, as you can see from that, C is not fast. Um-
- LFLex Fridman
Uh, so here it says 62 times faster than C.
- KKKieran Kunhya
Yeah. So it's taking those, the ethos of doing low-level programming, real-time programming, and take- using that for commercial applications, and JB and I have started companies around that, in many cases hiring developers from the open source community to use that ethos. And, and so that, that's a great example of, of some of the things we're doing. In most companies it would be, say, "Oh, I'll write this in C and it's fast and we're done." But actually you can get a lot better.
- JKJean-Baptiste Kempf
For me, like some of the headaches we have is around some OS that are difficult to support, right? Because, um, uh, if you look at VLC and thanks to FATE and FFmpeg, we run on, the last version of VLC runs on Windows XP and still run there and runs on Windows 11. Um, we work on macOS 10.7 to the la-latest macOS, whatever it is, right, uh, 26. Um, we work on iOS since iOS 9, well, we are actually iOS 26, right? We support, um, we support, uh, many types of Linuxes, BSD, Solaris. The last version still runs on OS2, right? Like there is maybe 10 users of OS2 in the world, and one of them is maintaining VLC. Then you realize that this very small team around VLC and using FFmpeg codecs and, and all the other ones support more OSs thanMicrosoft or Google or Apple, and they have infinite amount of, uh, of, uh, power and resources. But for example, the worst is iOS. Um, for, in order to build on iOS 9, we need to do some very clever mixing of several version of the Xcode IDE and SDK from Apple, from several version, and do a type of Frankenstein version of that so that we can still support iOS 9, which is not supported at all by the, the compiler of, uh, uh, of, uh, Apple in order to still run on Arm32, um, on iOS 9. And you've seen on Fate that it was still supporting iOS 9, right? So, so, um, my headaches are mostly related to the, the support of so many OSs. And it's important because, like, we receive so many people saying, "Hey, thank you, I still have my iPad 2 to watch movies," and it still works on iOS 9, right? And it's also an impact of, like, not forcing people to buy new hardware when it works fine, if you optimize it correctly, which brings us to what we were saying about Assembly. It's also fighting, like, the fact that you need to buy something new nonstop, uh, while you could optimize more, which is a lost art.
- LFLex Fridman
You gotta tell me about this lost art or this, uh-
- JKJean-Baptiste Kempf
Yeah
- LFLex Fridman
... the, the carriers of the flame of Assembly. What, uh, what is, what, what is Assembly? Why is it beautiful?
- JKJean-Baptiste Kempf
Yeah.
- LFLex Fridman
Why is it challenging? How does it work?
- KKKieran Kunhya
So when you write Assembly code, you write this using the instructions the actual processor is using directly. So most of the time you would write in a language, let's take C as a good example. The compiler would use that to create, um, Assembly language and machine code instructions for you based off your C code. And there's a specific flavor of Assembly that we use in FFmpeg that's called SIMD, SIMD, single instruction, multiple data. So this means, for example, say, say I want to add five to a number in scalar Assembly, so this is what's known as you work on an individual element. So I wanna have a number of-- I have the number 10 and I want to add five. I use the add instruction, and I add five to 10, and I get 15. With SIMD, with SIMD, I can have a whole vector of 16 different numbers. They could all be different. If I want to add five to that, I can, I can run one instruction, and that one instruction sums all 16 elements. And that, as you can imagine, lends itself very well to video. Video, uh, video is, you know, pixel grid, so I can perform operations on multiple pixels at the same time. The key thing that we do differently in FFmpeg is we don't use any abstractions or any major abstractions on top of that. So there's a part of the world that uses what's known as intrinsics. So these are C functions that behave very similarly but not quite the same to writing Assembly by hand. So the registers that data i- uh, is stored in on the CPU, the compiler allocates those for you. And so the key thing to understand was when we write SIMD is we have a 10x, and not percentage, 10x to 50x speed improvement. That, that function is 62x, um-
- LFLex Fridman
That's nuts
- KKKieran Kunhya
... on the FFmpeg account, as you know, posts and tweets a lot about that to try and say, "Hey, we are doing this stuff."
- LFLex Fridman
You are a person who sees the beauty in Assembly, but it's also extremely useful for these kinds of application to actually-
- KKKieran Kunhya
Yes
- LFLex Fridman
... significantly outperform even C, which is crazy.
- JKJean-Baptiste Kempf
It is necessary.
- KKKieran Kunhya
Yes.
- JKJean-Baptiste Kempf
Right? Because, like, one of the projects, um, that we need to talk about is called David, right? So David is a decoder for the format, uh, that was done by Alliance for OpenMedia, uh, which is an, a video decoder called AV1.
- LFLex Fridman
So it-- for people who don't know, we've been talking about H.264. AV1 is another hugely popular standard and codec that is increasingly taking over, uh, the internet.
- JKJean-Baptiste Kempf
And when this format was launched, uh, many people said, especially even from the Alliance for OpenMedia, right, which is Google, Netflix, Amazon, uh, Mozilla, say, "Well, this format is so complex, it must be done in hardware to do decoding," right? And well, I arrived, uh, with a few other people, uh, mostly, um, Ronald, uh, Henrik, and Martin, and we said, "We need to have an extremely good software decoder, uh, because it's going to take time to have hardware." And so we wrote this project, which is beyond insane. Um, we are talking about 30,000 line of C, but 240,000 lines of handwritten Assembly, right?
- LFLex Fridman
Handwritten Assembly, 240,000 lines. That's incredible. That mean-- I mean, some of the stuff we're talking about is probably the biggest Assembly, uh, code bases.
- JKJean-Baptiste Kempf
To give you an idea, and Kieran can correct me, but I think the FFmpeg has 100,000 lines of Assembly for all the codecs.
- KKKieran Kunhya
For all codecs. Mm-hmm.
- JKJean-Baptiste Kempf
And just this one has 240,000. Um, it's a VideoLAN project, of course. Um, and it is optimized at the maximum because the motto when we're starting the project is every cycle matters, right? Every cycle matters because David is used in VLC and in some software AV1 playback stacks. We are talking about probably three billion devices which are going to decode video nonstop because, for example, 30% of the video from Netflix are now in AV1, 50% of YouTube, right? So, and you often don't have a hardware decoder because not many devices have a hardware decoder. And with David, we realized that with one or two cores, you were able to decode 720p correctly. So it is, like, literally-
- LFLex Fridman
[laughs] Yeah, that's David
- JKJean-Baptiste Kempf
... incredible, right?
- KKKieran Kunhya
That's David. Look at that Lex.
- LFLex Fridman
Yeah. So this is, uh, another spicy tweet from you. "This is what peak video codecs should look like."
- KKKieran Kunhya
79.9% assembly, 19.6% C, and, uh, 0.5% other
- 2:25:26 – 2:34:42
Rust programming language
- JKJean-Baptiste Kempf
a better programmer.
- LFLex Fridman
Uh, what do you think about the Rust programming language? 'Cause that's a bit of a meme.
- JKJean-Baptiste Kempf
We have very different opinions with Kieran.
- KKKieran Kunhya
I, I think it's valuable what they're doing in terms of memory safety as a concept.
- LFLex Fridman
Can it achieve the s- some of the speed up that assembly achieves?
- KKKieran Kunhya
Oh, not, not assembly by hand, no. I think that that's a given. C potentially, but I see it very... It has a very big Esperanto vibe, uh, about it. It's like we're gonna solve this, and we're doing this in a particular way.
- LFLex Fridman
Meaning it's a bit too utopian?
- KKKieran Kunhya
There's a lot of focus on the self-importance rather sol- than solving real world problems. It reminds me of the Sinclair C5. Sir Clive Sinclair of Sinclair Computers built a car, and he said, "Oh, everyone will be traveling around in, in, uh, one of these electric cars." And it was, um... Rust reminds me of that where I think the community doesn't, the community doesn't quite understand that in order to get people to move, you have to build something that's as good as, if not better than what you have now. Um, yes, people are doing Rust rewrites, but if they're, if they only, if they only do eighty-five, ninety percent of the feature set of what, what we need, like things like core utils, that last one percent takes ninety-nine percent of the time. Um, to, to use Elon's famous quote, "Prototypes are easy." Like this kind of stuff is easy, but this... To get a real electric car, you have to make a car as good as, if not better than what we have now, and Rust isn't in that stage yet. I think we-- I don't think anyone would object to seeing Rust code in FFmpeg, but it needs to work as well and support the same unit testing as everything else. It needs to be flawless. It, it can't just randomly break. They can't just randomly break ABI when they want to. It needs to, it needs to have, I think, more... I think it still has only one compiler implementation.
- JKJean-Baptiste Kempf
Yes.
- KKKieran Kunhya
So i-it's got to be as good as, if not better. And saying, "Hey, here's my utopia of memory safety," isn't enough, even though we all, we probably all agree that that's the goal.
- JKJean-Baptiste Kempf
So, um, I've done a ton of Rust, and the two major topics I had was adding Rust modules inside VLC. One of the reasons VLC got popular, um, and which was one of the main architectural decision, is that VLC is a very small core and a ton of modules, right? And so you can write modules in C, in C++, in Objective-C, and anything that is basically interoperable with C. And, and, and so we t- we, we did some Rust modules, and so I have experience on that, and, and I wrote some of it. And also, like my, my new startup, uh, called Kyber, is an open source project mainly done in Rust. Um, what Rust is extremely good in, in the sense that, um-It's a better C++ that cares about memory and allows you to do things about memory ownership that no one else can do so far. However, it's great when you start a new project from scratch and you do everything in Rust, but it's very not good when you interop with ex-existing part. And some part of the Rust community believes that they need to rewrite everything, and everything will be better with Rust. And, and the answer is like, no. Like, I'm almost always in all my years of being engineer, manager, CTO of startup and so on, don't rewrite, right?
- LFLex Fridman
Is that-- That's the, that's the initial instinct for a lot of people when they show up to a code base, uh, probably before LLMs, is like b- probably because they don't understand the, the, the wisdom of the way things have been done in the past. They say, "Well, we need to rewrite it." Hence why there's a thousand JavaScript frameworks.
- JKJean-Baptiste Kempf
But the reason is the following, and this is very important to understand. It is an order of magnitude easier to write code than read code.
- LFLex Fridman
Yeah.
- JKJean-Baptiste Kempf
And you see that also with LLM. They can write code, but analyzing is a lot-
- LFLex Fridman
Mm
- JKJean-Baptiste Kempf
... more difficult. And so when you arrive a-and when you arrive to a very complex piece of code, right? You, you don't understand it, right? Um, because it's so much more effort to understand the code from someone else because you don't have the thought process. Um, a-and often I joke about, um, some, some languages, uh, mostly, uh, Perl, for example, um, which has very complex syntax. And imagine I am at my maximum intellectual efficiency in programming, right? And I write the best code ever. I will not be able to understanding myself six months later, right?
- LFLex Fridman
Mm.
- JKJean-Baptiste Kempf
Because reading code is more difficult. So very often you arrive, you don't understand all the wisdom, all the business logic, the reasons that were done that is maybe not documented. And you say, "Well, I'm going to rewrite it." And the thing is, no, you don't, right? Because that's, as Kieran said, right? I'm going to rewrite core utils in Rust, and then of course you arrive very quickly at 80%, um, then 90% takes a bit more time, and then you got the last ones, right? On the other side, right? So for new projects, it's great. Everything related to parsing files, uh, networking, uh, because of the memory, uh, checker, memory checker, it's amazing, and there is nothing else. To answer a bit differently for us, imagine I take a, a piece of software like, uh, DaVinci or x264, right? Which has a ton of runtime in assembly, right? Um, I rewrite the C part in Rust, right? So it's more secure. Yes. But then you arrive into the assembly, and you can jump anywhere in the memory because we are doing handwritten assembly. So even if I rewrite the C part in Rust, for security reason, you, you break all the security when you rewri- you write handwritten assembly because we can jump anywhere. So in my opinion, we need to do something that is, um, secure assembly, right? So which is compile time, check the assembly, which is similar to the check assembly projects that we're doing on, on DaVinci and x264 with VideoLAN, is to start instrumenting your assembly at compile time to check that it's not jumping anywhere in the memory. Because else you might rewrite a part of C in Rust, but if you want to have the same performances, you're going to have inline assembly, and so you destroy your whole security model. Um, so that's a bit what I think about Rust.
- KKKieran Kunhya
No, I, I just wanna... I, I would say on a personal level, I'm so in awe about the assembly. I actually-- Once in a... It never gets old, the speed improvements to show 62X. Uh, so there are months p- on a personal level, I, I run our internal test suite at work and just see I'm still in awe at the gains we have.
- LFLex Fridman
Well, there, there's a source of joy and happiness for programming for different reasons. But I think one of the gr-greatest happiness is, is in the optimization of code.
- KKKieran Kunhya
Yeah.
- LFLex Fridman
And it sounds like y-you're, like, at the cutting edge of that.
- KKKieran Kunhya
I was like, "Whoa, that was cool."
- JKJean-Baptiste Kempf
And in the community, I want to speak about two people who are wizards of assembly, right? The two of them are actually working, uh, living i-in north of Europe, uh, Sweden a-and, uh, and Finland. And, um, Henrik Gramner knows so much about Intel x86 assembly that when we ask questions at Intel about things, they tell, like, "Why are you asking us Intel? You have Henrik. Henrik knows better." He knows all the cycles of almost all the SIMD instruction by all the CPU generation. "Oh, yes, this is a P4, this is a Nehalem, this is a Core 2," et cetera. That person is, like, the best person on assembly in the world.
- KKKieran Kunhya
Yes.
- JKJean-Baptiste Kempf
And he's the nicest person that you've seen, like very... He, he arrive, you don't see, uh, he's amazing. And the other one is, is called Martin, Martin Storsjo, and he's... They're doing mostly the same on Arm, right? So Neon, right? And iPhones and Androids and so on. And he codes in assembly on his phone, editing it with the crappy, uh, uh, keyboard, like virtual keyboard you have while watching his kids play in the playground, right? Like, like this is just like wizard level. So those two people are like-
- KKKieran Kunhya
Yes.
- LFLex Fridman
So a part when you're programming assembly at that high level, a part of that is knowing the architecture that you're programming on.
- JKJean-Baptiste Kempf
Yes.
- 2:34:42 – 2:43:04
FFmpeg and Libav fork
- KKKieran Kunhya
ball game.
- LFLex Fridman
We're, uh, taking a non-linear journey through history here, but we're talking about, uh, Michael Niedermayer. Uh, and I, I wanted to ask about this. Uh, for a time there was a split in FFmpeg and, uh, Libav.
- JKJean-Baptiste Kempf
Yes. So, um, in open source projects, um, sometimes you disagree, right? Um-
- LFLex Fridman
[laughs] You have such a nice way of putting it, yeah.
- JKJean-Baptiste Kempf
And the good thing is, because of the license, you're allowed to basically do your own, right?
- LFLex Fridman
Yeah.
- JKJean-Baptiste Kempf
Um, and this is normal, and this has happened all the time, right? For at the point there was a, a GCC at the time of GCC2 and EGCS, uh, which became then GCC3, right? There is what we told KHTML with WebKit, with Blink. Um, it is a same process. And also, like, when I want to do a new feature today in VLC, I fork, I do my thing on my own, and then I merge back to the community. So there was a, a, a split in the open source community on FFmpeg, which become Libav and FFmpeg, and after a few years, well, the community merged back and people moved on. It's a bit, um, drama that is normal in open source community, but forks are even... They're important because they, they, they change the, the status quo of a community. Um, um, not talking about FFmpeg a-a-and, and Libav here, but the, or the GCC fork made GCC a ton better because the, some people wanted to change the architecture fundamentally to make it faster. And of course, it's always question of people and so on, but in the end you realize that FFmpeg today is better than it was, uh, before the fork. Um, and now, well, we're back all together, right? Uh, and I spent a lot of time, and, and Kieran can say, uh, in the, in, in the community. Um, it's not often, to be honest, uh, very well explained, uh, because a ton of the reasons are not very public. Uh, but I think that's, that's normal, and that's good.
- LFLex Fridman
Yeah, I mean, you're making it sound really nice, but there, there is battl- there's pretty heated battles inside open source projects. I mean, it is a very passionate community and you're kind of in a distributed way have to define the direction of things.
- JKJean-Baptiste Kempf
Yeah.
- LFLex Fridman
So here looking at, uh, Perplexity, "FFmpeg and Libav split in 2011 mainly over project governance, leadership style, and development processes, not because of a fundamental technical disagreement. Uh, FFmpeg effectively absorbed Libav's work [chuckles] while Libav withered and most distributions and developers moved back to FFmpeg." Yeah, that was a, that was a weird, that was a we- from a user's perspective, that was a weird experience 'cause, you know, I'm a Linux user, so, you know, whether it's Ubuntu and so on, all of a sudden, I think for, for a c- for, for a little bit, Ubuntu, I feel like, am I remembering correctly, switched to Libav and-
- JKJean-Baptiste Kempf
12, 14, something like that.
- LFLex Fridman
Yeah.
- JKJean-Baptiste Kempf
Something like that.
- LFLex Fridman
And then they switched back to FFmpeg.
- JKJean-Baptiste Kempf
They switched back, yeah.
- LFLex Fridman
I was like, "What is happening?" [chuckles] So on the sort of, um, you f- you get to feel the ripple effects of the different internal debates that are happening.
- KKKieran Kunhya
To be fair, on Apple, when you type GCC, you get Clang. Like they, they did something like that as well, so.
- JKJean-Baptiste Kempf
Yeah. So, so to me it's like the fork was like heated drama, but most of the development from Libav was merged back into FFmpeg, right?
- LFLex Fridman
Yeah.
- JKJean-Baptiste Kempf
So de facto FFmpeg got a s- a, a superset around Libav, and so that gave the user, because in the end we work the user- for the users, a, a larger set of features and a ton of things that were, um, discussed.
- LFLex Fridman
Mm-hmm.
- JKJean-Baptiste Kempf
For example, the debate on reviews, on, on how we push are something that now is completely settled in FFmpeg and is following what mostly what everyone in the community agrees, right? So de facto, everyone who was active on Libav came back i, in work on FFmpeg because l- the disagreements were fixed, and in the end, FFmpeg is stronger, uh, than it, it was before, right? And-
- LFLex Fridman
Mm-hmm
- JKJean-Baptiste Kempf
... I know people love drama, but, um-
- LFLex Fridman
Well, my, my main concern, I, I understand, and I think looking at the, the long history, it's all for the good. But I do... I am concerned because there's so few humans that are critical to the success of open source projects that I have seen it, uh, be a psychological toll on folks and, you know, sometimes leads to burnout. So you have these incredible people that are at the core of open source projects. There is a moment that happens 'cause, like, what is the motivation of doing it? Ul- ultimately it's because you're passionate about it and it makes you happy. Then a certain point you wake up and it's like, "This's been a bit too much heat from the drama." So, like, at the, at the project level, the project continues and often flourishes, but sometimes there's these individual humans that are just like-
- JKJean-Baptiste Kempf
But-
- LFLex Fridman
"I've had enough." [chuckles]
- JKJean-Baptiste Kempf
Yeah, but it's not just about forks, right? So there's a g- very-
- LFLex Fridman
Yes
- JKJean-Baptiste Kempf
... what, what you, what you are, um, referring to is, uh, one of the most challenging and most interesting part of, uh, open source today is maintainers burnout, right?
- 2:43:04 – 2:50:51
Open source burnout
- JKJean-Baptiste Kempf
the mental health of the open source maintainers is something that large corporations don't care or don't see, right? It's just like, oh, yeah, I'm just doing an open source, uh, report and so on.
- JKJean-Baptiste Kempf
Some of it is financial, but some of it-- And b- people should definitely support open source financially-
- JKJean-Baptiste Kempf
Yeah
- JKJean-Baptiste Kempf
... all across the board.
- JKJean-Baptiste Kempf
Yeah.
- JKJean-Baptiste Kempf
But some of it is also, like, spiritual on a basic human level. There's something that happens, like, with this image of F- of FFmpeg and so much of the internet depending on it, where people almost, like, talk down to the folks who are carrying these projects forward and maintaining it. In the security community, they certainly did. That was one of, that was one of the things I think that argument came out is there was, there was a portion of the security community who's like, "No, these guys write crap code. They need to fix their crap code." I'm like, "No, no, no, no. This is a guy's hobby project. You've, you've have a security bot that's gone and found some AI-generated stuff. That guy didn't write crap code. It's just an edge case to the 99th point nine nine nine nine nine percentile he didn't think about because it's ho- his hobby project decoding Star, Star Wars games."
- JKJean-Baptiste Kempf
Forget the hobby project aspect of it. It's, it's just hard work and it's b- it's beautiful, and it's like the, the right approach there is to celebrate people-
- JKJean-Baptiste Kempf
Sure
- JKJean-Baptiste Kempf
... for doing incredible, incredible work. It's, it's just incredible that humans step up-
- JKJean-Baptiste Kempf
Sure
- JKJean-Baptiste Kempf
... not getting really paid at, at first or maybe ever, and then they're doing it out of the love of it, and we need to... Like, human civilization runs on people like that. We need to celebrate them.
- JKJean-Baptiste Kempf
To, to, to give you an idea, I received death threats o- on VideoLAN, right? And, um-
- JKJean-Baptiste Kempf
You mentioned that to me. Like, what, what is, what is behind that?
- JKJean-Baptiste Kempf
So that must be, what, 2009, 2010, right? Um, uh, Apple is moving from PowerPC to Core Duo, s- um, that's probably in 2006. And by 2009 or 2010, I decide that we are not going to do new versions of VLC for PowerPC.
- JKJean-Baptiste Kempf
Mm-hmm.
- JKJean-Baptiste Kempf
At that time, like VLC, we were close to the number 1.0 release. We were four of us, right? Like, just like, no, this is not possible. So I receive a death threat with some powder in it, right? It-- Remember there was some-
Episode duration: 4:18:22
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode nepKKz-MzFM
Get more out of YouTube videos.
High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.
Add to Chrome