Tim Sweeney: Fortnite, Unreal Engine, and the Future of Gaming | Lex Fridman Podcast #467

Humans are by far the hardest part of computer graphics because millions of years of evolution have given us dedicated brain systems to detect patterns in faces and infer emotions and intent. Because cavemen had to, uh, when they see a stranger, determine whether they were likely friendly or they were- might be trying to kill them. Um, and so people in the world have extraordinarily detailed expectations of a, a face, and we can notice imperfections, especially imperfections arising from computer graphics limitations. Okay, one part is capturing humans, and so we've built really advanced, dedicated hardware that puts a human in a capture sphere with dozens of cameras and then taking high-resolution, high-frame-rate video of them as they go through a range of motions. And then capturing the human face is complicated because the nuanced detail of our faces and how all of the muscles and sinews and fat work together to give us different expressions. So it's not only about the shape of a person's face, but it's also about the entire range of motion that they might go through. So that's the data problem. There's a lot of other problems with computer graphics, you know. There's technology for rendering hair w- which is really hard 'cause you can't render every... A- again, we know the laws of physics. It would be easy to just render every hair, it would just be a billion times too slow. Um, so you need approximations that capture the net effect of hair on rendering and on pixels without calculating every single interaction of every light with every strand of hair. Um, that's one part of it. There's detailed features for different parts of faces. There's subsurface scattering because we think of humans as opaque, but really our- our skin is... Light travels through it. It's not completely opaque, and the way in which light travels through skin has a huge impact on our appearance. You know, this is why you... There's no way you can paint a mannequin to look realistic for a human, uh-

Mm-hmm.

... you know? It's just a solid surface, um, and will never have the sort of detail you- you see.

That kind of blew my mind, like, thinking through that. Hmm, I think I heard that sort of the oiliness of the skin creates very specific nuanced, complex reflections, and then some light is absorbed and travels through the skin and that creates textures that our human eye is able to perceive, and it creates the thing that we consider human, whatever that is. All of that while considering all the muscles involved in making a nuanced expression, just the subtle squinting of the eyes or the subtle formation of a smile. It's the subtlety of human faces that you have to capture (laughs) . Like, the difference between a real smile and a fake smile. But the way they show, like, beginning of a formation of a smile that actually reveals a deep sadness. All of that, like when I watch a human face, I can, like, read that. I could see that. You have to have the tools that in real time can render something like that, and that's incredibly difficult.