Skip to content

By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

The Pragmatic EngineerThe Pragmatic Engineer

Software correctness and programming languages with Sean Allen

We do not mention "AI" a single time - even though this happens to be a deep conversation on the biggest challenges of developing with AI.

Gergely OroszhostSean Allenguest
Nov 10, 20261h 25mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

Great conversations run long. Their best ideas don't have to.

uListen is free on iPhone, iPad, Android, Chrome, and Claude.

See how

Transcript

0:00 – 1:11

Intro

Gergely Orosz 0:00 You wrote a very interesting blog post, which I'm gonna quote, "Tests are a means to an end. The means are unimportant."

Sean Allen 0:06 There is absolutely amazing stuff that you can do with that testing. But in the end, you are still doing something at the late stages in the game. It's true that everybody in the end, in some way, whether they admit it or not, is testing in prod. None of the tools that we have are good enough to guarantee whatever it is-

Gergely Orosz 0:24 Technical debt only hurts if you have to pay it back.

Sean Allen 0:27 Yes, you can very much get yourself into a position where, unfortunately, before you've even found product market fit, you might have debted yourself into a corner. But the idea that exists in a lot of stuff where it's like, "Oh, no, every single bit of debt is bad, it always has to be paid off," it's not borne out in reality.

Gergely Orosz 0:44 Sean, great to have you.

Sean Allen 0:45 Thank you.

Gergely Orosz 0:46 We need to start with this crazy fact that we discovered. The first company I ever worked for that did professional software, it was called SenseNet. S-E-N-S-E-N-E-T. What was the first company that you worked for?

Sean Allen 0:59 SenseNet. S-E-N-S-E-N-E-T.

Gergely Orosz 1:02 [laughs] And they're different companies.

Sean Allen 1:04 Yeah.

Gergely Orosz 1:04 Yours was in New York.

Sean Allen 1:06 Yep.

Gergely Orosz 1:07 Mine was in Budapest, Hungary. [laughs]

Sean Allen 1:08 Oh, I didn't, I didn't know yours was in Budapest. Hey.

1:11 – 4:45

Sean’s path into tech

Gergely Orosz 1:11 Yeah.

Sean Allen 1:11 Yeah.

Gergely Orosz 1:11 And rewinding the clock, uh, a lot before your first job, how did you get into tech? In fact, did you even wanna get into tech, or was, uh, was music more your thing?

Sean Allen 1:21 Music was definitely more my thing. But, uh, I graduated from college, and I realized I had no idea what I was gonna do, and I just started going through, uh, my address book. And the first person I found who said I could come stay with them, uh, lived in New York City. So I went, and a good friend of mine was also living there. And, uh, he eventually... He was like... I couldn't, I couldn't find a job. I got close to getting one, which was as a speech writer for a US senator, uh, but I did not get that. And a friend got me a job fixing computers at Scholastic. And then another friend told me, "Hey, if you learn Perl, you can move to Boston, and you can..." You know, there was a bunch of people I'd gone to college with who started a startup. So I learned Perl. And right about the time that I finished being good enough learning Perl over the course of three to four months to, you know, do that, uh, that startup had collapsed. And then it started to just be like, "Ah."

Gergely Orosz 2:15 This was the dot-com bust, right, around that time?

Sean Allen 2:17 It, well, n- no, no, that was, that was even before the dot-com bust. That was a combustible set of human beings, uh, working together. But yeah, at, at that point, it started being like, "I, I like this. I enjoy it." I'd done it a bit, like, earlier on, uh, when I was, like, a teenager 'cause I... There was a video game, Lode Runner, and I, I loved Lode Runner. And I really wanted to, to create things that were like Lode Runner. So I asked my grandmother to get me a book about computer programming. But I didn't know anything about computer programming, and my grandmother didn't know anything about computer programming. So she went to a computer store in, uh, Fort Worth, Texas. And the guy gave her a book, which was, uh, Machine Language for the Commodore 64. So I was like, "Wow, programming is really hard." But yet, that's, I kinda got started that way. And it was the first time I was like, "Oh, there's a thing that people will pay me lots of money for." And I was like, "I have no other thing that people would pay me lots of money for. I should get really good at this."

Gergely Orosz 3:18 And then so how did you get really good at, at programming? Like, you, you then worked at different companies, right?

Sean Allen 3:24 I spent an awful lot of my free time, um, just trying to get better at it. I think I, I, leaps and bounds, I got interested in, uh, BeOS, uh, in the '90s.

Gergely Orosz 3:38 BeOS?

Sean Allen 3:38 Yes. I was really into BeOS. And, um, there were some very interesting and helpful people in that community. There was a guy. Uh, all the compiler tools for BeOS were made by this company called Metrowerks that also made, uh, compilers for macOS at that time. And for some reason, the guy who was the head of their, uh, their compiler group took an interest on a mailing list in helping me learn C++. And there was, uh, there was somebody who was, uh, knew C++ and was really nice on the mailing list as well, who, um... I know, I know basically nothing about sci-fi at all. So it didn't mean anything to me that that person turned out to be Neal Stephenson, who-

Gergely Orosz 4:20 No way.

Sean Allen 4:20 Yes. Who, um, who-

Gergely Orosz 4:22 World-famous sci-fi author.

Sean Allen 4:24 Yes. Who basically was just, like, would exchange, like, emails with me and advice on, on learning, like, C++ programming, which was really early on for me. And I mean, there were other people as well, but those were two who really, like, stood out to me. Like, I've still never read a Neal Stephenson thing. I think I read a couple pages of Snow Crash. But it's, it's always a fun story to tell people.

4:45 – 6:49

Joining Antithesis

Gergely Orosz 4:45 Yeah. And how did you end up at Antithesis?

Sean Allen 4:47 Well, the, the short version would be is that Will Wilson, the CEO, sent me a message while I was looking for a new job in December, which was just a link to a thing that they'd posted on the Who's Hiring for, uh, on Hacker News, and said, "Would you be interested in this?" And it was basically like, "Do you want to do Jepson-style things?" And my response was, "That sounds like it could be fun."

Gergely Orosz 5:10 And what do you do at Antithesis now?

Sean Allen 5:12 I'm on a team called the Red Team. And I think if you asked everybody at the company what the Red Team does, you'd get probably different answers from almost everyone. Um, I would, I would put it as Will collected a bunch of really smart people together and was like, "I want you to use our product the way that customers would use our product." One of the things I've done as part of that is I've intentionally never used the tools which are available to us internally to work with the, the product and just done it so that I can only work with it with the exact same tools that, uh, the customers would have for themselves.

Gergely Orosz 5:48 That's kind of a neat concept-

Sean Allen 5:49 Yeah

Gergely Orosz 5:49 ... ha- having an internal team who is a customer team, except, you know-

Sean Allen 5:52 Yes

Gergely Orosz 5:53 ... that you can talk to them. They will reason with you.

Sean Allen 5:55 Yeah.

Gergely Orosz 5:55 They'll, they'll explain how the customer thinks.

Sean Allen 5:57 Yeah. And w- so we're, we all have a lot of experience, and we've been given a lot of free rein to do anything as long as it, as long as people are looking at it going, "Oh, that's valuable to the company." So I worked a lot on, on skills for helping agents work better with Antithesis. But I mean, I think the stuff that I like the most, um, is just doing stuff to teach customers and teach people internally as well. A lot of the stuff that I've learned over the years for How to, how to be sure that your software works well, um, and that, you know, for me, that starts way down the stack, way before anything that anybody calls testing or whatnot. Yeah, and just talking to people about that. I love meeting people. I love talking to people, and it's a subject that I, that I really love. I don't know. It's kind of a dream job.

6:49 – 12:25

OCaml and type systems

Gergely Orosz 6:49 And speaking of dream job, one thing that you've been really going deep into is just different languages, which I would call a little bit of exotic languages, not... I'm not talking about the mainstream TypeScript or maybe Rust or Go, et cetera, but a few other languages. Can we talk about OCaml specifically? What is OCaml, and why are all the hedge funds and finance companies in love with it?

Sean Allen 7:15 So OCaml is a language that came from an academic background, primarily, uh, in France originally. Um, and it was, it came out at the time when people were trying to combine functional programming with object-oriented stuff. This was a really big deal that people, like, tried to do, particularly in the '90s. At that point, it was bolting objects onto everything. Objects were very much seen as the future, et cetera. And it's interesting because the version of OCaml that's actually caught on at places like Jane Street and, and other places where it get used a lot is one where none of the O features in OCaml really get used.

Gergely Orosz 7:56 What is the O?

Sean Allen 7:57 The O is the object part.

Gergely Orosz 7:59 Oh.

Sean Allen 7:59 And, and there's, there is a whole series of features which are very object-oriented about it, but I can't think of anyone I know who does OCaml who actually really uses any of the object-oriented features. But it's, it's a language, um, which is descended from ML, which is a entirely different languages of families from like Algo languages where, where like Python and Perl and, for the most part, Ruby, C, C++. These are all, these are all Algo languages. And then there's, uh, an, a- the ML family of which, uh, OCaml is, is one. There are, there are a number of others in there. Uh, depending on who you ask, some people will say Haskell's kind of in there, kind of not. And, um, all the ML languages generally provide a, a different set of, uh, uh, functionality and capabilities to people. The most famous, I think, at this point ML, uh, influenced language would be Rust. There's an awful lot of ML-type stuff in Rust. The, the sum types, the pattern matching, et cetera, all of this is stuff that came from-

Gergely Orosz 9:02 M- M-ML just sounds like a bit like more of a declarative language? Or w-what is the ML s... H-How would you describe it?

Sean Allen 9:10 The ML languages are generally, uh, they're built around strong type systems, right? In the same way that, that Rust is built around a, a strong type system, where you take some types, which is basically saying it could be one of these three things. They- they're kind of like enumerations, uh-

Gergely Orosz 9:27 Yeah

Sean Allen 9:27 ... from something like Java that, that people might be familiar with. Or, hey, Rust is a lot more popular now, so some people might be familiar with it from there. And you can combine that together with, uh, pattern matching. And, and if you can then also combine that together with exhaustive matching, um, where what it means is that I have this sum type, which could be, say, five different things.

Gergely Orosz 9:48 Mm-hmm.

Sean Allen 9:49 Right? And the way that, the way that I will work on it is instead of an if statement, I do pattern matching, but I'm matching on the type that the thing is.

Gergely Orosz 9:57 Mm-hmm.

Sean Allen 9:58 Um, and then you can combine that with exhaustive matching. What you can end up with is a code base where you can make one change and say, "Actually, now I have six different things of this instead of five." And the compiler will tell you every single place in your code base where you're not handling-

Gergely Orosz 10:16 Hmm

Sean Allen 10:16 ... that sixth type, and it, it is a gloriously wonderful, uh, experience.

Gergely Orosz 10:24 It, it, it sounds like the compiler gives you additional testing that you might not get from a compiler in, let's say, a language like C# or, or TypeScript, whatnot.

Sean Allen 10:36 Yes.

Gergely Orosz 10:36 The compiler there will tell you the types, but if I understand correctly, the, the exhaustive matching is the key here. And the fact that there's matching, it goes and checks kind of maybe conditions or, or assumptions that are in these languages, like TypeScript. I'll just use it as an example. It's kind of, you know, you'll figure it out runtime-

Sean Allen 10:53 Yeah

Gergely Orosz 10:54 ... if something breaks, and here in compile time, there's a lot of structure rules, and there's, of course, like logic. It, it goes through there.

Sean Allen 11:01 Yeah.

Gergely Orosz 11:01 So you kind of front load a lot of that validation, if you will.

Sean Allen 11:05 Yeah. A-all, all of the languages which are ML-like in some type of way or are in that general family are very much languages where it's like I want to put as much as possible into static checking in the compiler and have the compiler do work for me.

Gergely Orosz 11:21 Okay, so now I understand why some people say, who are into Rust, that Rust helps you ship fewer bugs because it's an ML, it's type language, which the compiler will do a lot more work, and it will have a lot stricter rules that you need to become familiar with. But then in return-

Sean Allen 11:38 Yeah

Gergely Orosz 11:38 ... the compiler just, like, iterates through a lot of things, and it'll give you a compile error.

Sean Allen 11:42 Yeah. Ear-early on, when I was learning to program, um, I was really into, like, different programming languages and just learning, like, different things that were possible. And, uh, fairly early on, I decided I was going to pick one thing, and I was gonna dedicate myself to that for a while. And, uh, the two languages I ended up settling on were, uh, C and Ada, and it was mostly it was C. The, the choice of C came because I was doing a lot of programming for, I think it was macOS. It was either 6 or 7. I think it was 6 at that point, and Apple was converting an awful lot of their stuff from Pascal to C at that time. So I was like, "Oh, okay, well, C could be one that could be interesting." And then Ada was just entirely

12:25 – 14:28

Ada and compiler guarantees

Sean Allen 12:25 different and foreign.

Gergely Orosz 12:25 What, what, what is, what is Ada like for those of us like myself who don't know?

Sean Allen 12:29 Ada is much more in that very, very strict static checking thing.

Gergely Orosz 12:34 Mm-hmm.

Sean Allen 12:34 Um, Ada is a language that in a lot of ways was driven by the needs and desires of, like, what the Defense Department wanted to have. For, um, for knowing that there were not going to be bugs of, of a language which is very deeply specified and will not let you write incorrect programs.

Gergely Orosz 12:55 Yeah, we're talking like literal mission-critical stuff.

Sean Allen 12:57 Yes, yes, mission-critical stuff. NASA, the Defense Department, et cetera, these, these were types of people who Ada was, was aimed at. Ada still exists. It has a small but robust market for people, and the number one thing that the Ada companies really go after is, "Oh, so you like Rust. How would you like to have some more guarantees?"

Gergely Orosz 13:18 [laughs]

Sean Allen 13:18 Like, that's, that's, that's where, that's where the Ada stuff is. And this is, this is one of my great regrets, was that, um, I didn't really know much about programming, and I didn't really know much about programming languages, and I could not get programs to compile in Ada.

Gergely Orosz 13:35 In Ada because it's so damn strict, right?

Sean Allen 13:36 It, and it was so frustrating. And, and I could get C programs to compile. It didn't really dawn on me, the fact that my C programs were segfaulting all the time and didn't really run for very long, but it was like, thank you for the cookie that it compiled, you know? Now I can move on to the other thing. But I spent years developing an intuition of how I could do stuff in C so that my programs didn't die and collapse and, and now all I want is I like, I recognize, like I had all these rules for myself for writing concurrent C and C++ and to keep myself from, uh, segfaulting programs. And even, even though I became really rigorous with that, I'm a human, and I screw stuff up all the time, and I would much rather have a compiler help me with that. And so there's a lot of stuff in like OCaml and other languages like it, uh, F#. Rust.

14:28 – 18:26

F# and .NET

Gergely Orosz 14:28 Can, can, c- can we talk, can we talk a little bit about F#?

Sean Allen 14:31 Yes.

Gergely Orosz 14:32 For folks who have not used F# but they might have used C# or TypeScript, how would you describe F# and why it's different? And it, right, it's, it's a language that Microsoft created or-

Sean Allen 14:42 Y-

Gergely Orosz 14:42 ... or they champion. I'm not sure if they created it.

Sean Allen 14:44 Uh, it, so it came out of Microsoft Research. Um, uh, there's a fellow, Don Syme, who's really responsible for everything F#. And it basically started as, uh, OCaml for .NET, where .NET at the time was just C# and the .NET runtime.

Gergely Orosz 15:02 And, and .NET is Microsoft's framework where it was really easy to access. I remember Windows APIs. It had a lot of utility functions. It was their everything framework.

Sean Allen 15:11 It, it, basically their version of the JVM.

Gergely Orosz 15:14 Yeah.

Sean Allen 15:15 Right. Um-

Gergely Orosz 15:15 The Java virtual machine, yeah

Sean Allen 15:16 ... we, and I have no special insight into it, but from somebody who was around when all those things happened, there was a whole bunch of legal kerfuffle with Microsoft adding Windows-specific APIs to their version of Java and Sun threatening to sue them or something.

Gergely Orosz 15:32 There was a lawsuit, and so I think-

Sean Allen 15:33 Yeah

Gergely Orosz 15:33 ... uh, uh, Anders Hejlsberg on this podcast, uh, shared the detail that they needed to develop C# and I guess .NET as well because of a pending or, or active Sun lawsuit.

Sean Allen 15:44 Yeah.

Gergely Orosz 15:44 And so they could no longer use Java even though they wanted to use it, and that kinda settled it.

Sean Allen 15:48 Yeah.

Gergely Orosz 15:48 So they built .NET.

Sean Allen 15:49 Th- it did, and, and .NET was, was great because they took like pretty much everything they'd learned from the JVM and things where ideas had been baked into the JVM that would be kinda hard to change at that point, and they did different things which at the time seemed like much better ideas. People argue about whether they are or not at this point.

Gergely Orosz 16:08 Mm.

Sean Allen 16:08 But it doesn't really matter. But, you know, .NET was a very nice platform, um, et cetera, for everything, and they had one language basically that was on it, which was C#, which is an awful lot like Java. It has differences, but fundamentally, the basic semantics of the language are the same. You can-

Gergely Orosz 16:24 Very similar.

Sean Allen 16:24 Yeah. Um, and so F# was a, "Hey, let's bring some different semantics. Let's bring some different stuff to this," which was originally like, "Let's try to bring OCaml." And I think the thing that's amazing about F# is, is not really specifically any language features from it, um, because it, it has an awful lot of the things that OCaml has, but then it doesn't have some. And some people really like to complain about the stuff that it doesn't have who are big OCaml fans or whatever. But the thing that F# brought in a way that I've never really seen for anything else is it, is it fits so well onto .NET as a platform, and it fits so well. It's really the most seamless experience I've ever seen for something. Like I, I did a lot of stuff with Scala and other languages on the JVM, and there are lots of rough edges with, with interacting-

Gergely Orosz 17:19 Mm-hmm

Sean Allen 17:19 ... with, um, Java where F# doesn't have that. And F# developed, in a lot of ways, a lot more slowly, and you can see how Don did this beautiful job of not running ahead with F# and, and making a language which would be hard on the integration and having lots of rough edges, and it would be like, "We're gonna push this a little bit forward, and then we're gonna wait for the .NET runtime to catch up to us so that everything continues to work. And oh, now the .NET runtime has added some stuff." And it, it, you can see this thing where it's like, ah, F# pushes forward a little, and then, oh, clearly this inspired like the .NET team some, and there's like a symbiotic relationship that was there. And over the years, an awful lot of the really interesting and cool features from F# have made their way into, uh, C#. And about the only thing that's really missing at a semantic level is actually the, uh, the sum types with exhaustive matching. And based on the changes that have happened in the last few years with C#, it really looks like eventually even that would probably be incorporated

18:26 – 27:16

Pony and safe concurrency

Sean Allen 18:26 into C#.

Gergely Orosz 18:27 And then one language I wanted to ask you about is Pony.

Sean Allen 18:29 Yes.

Gergely Orosz 18:29 What is Pony?

Sean Allen 18:31 Pony is a, uh, is an open source, uh, language that I work on. It was originally created by, uh, a friend of mine who, uh, uh, we both went to a, uh, a college for people who had not done particularly well in high school-

Gergely Orosz 18:48 Yep

Sean Allen 18:48 ... and were in danger of, say, failing out and being high school dropouts, where it was a, it was an early college that would take people in like that who were not good in high school.

Gergely Orosz 18:58 Yeah.

Sean Allen 18:58 Of which both my friend Sylvan and I were like that. And Sylvan was actually the one who was like, "Hey, if you learn Perl, you can come to Boston and we can work on all of this stuff."

Gergely Orosz 19:08 Oh, yeah.

Sean Allen 19:08 Yeah. So I had gone off to work at a, uh, a place that was, uh, we were trying to build, uh, high-speed data processing stuff for banks. And we started off d- by doing that for doing consulting with one of the big banks in New York City. And we were just doing different consulting jobs, and we were looking at stuff. And around the time we were doing that, I was thinking about what a system like this would look like and how I'd wanna build it, et cetera, and everything. And a big part of it was, it was like, it needs to be really fast, right? So this rules out a whole class of languages, and it needed to be fast and have latency guarantees in a way that, uh, the JVM itself made really difficult and was for... And for inside of banks was reserved for specialized things of either using C++ or, um, the Azul on JVM.

Gergely Orosz 19:59 When, when you're saying the JVM made it difficult, was it because of garbage collection? Was it because of-

Sean Allen 20:04 Garbage collection, latency issues, et cetera. There's been a lot of work on the JVM since then. Yeah.

Gergely Orosz 20:08 Yeah, 'cause, 'cause, 'cause with garbage collection, it's like sometimes your latency will spike like a normal function that should, I don't know, take this many milliseconds. It could take way longer-

Sean Allen 20:16 Yes

Gergely Orosz 20:16 ... 'cause, oh, garbage collection just kicked off, but-

Sean Allen 20:19 Yeah

Gergely Orosz 20:19 ... not randomly.

Sean Allen 20:20 P- particularly at that time, on the standard of, uh, available to everyone, uh, JVM versions, uh, all the garbage collection would stop the world, which literally means in order to collect garbage, everything has to stop. And so you have a request that's just come in, and your latency's gonna go from, yeah, milliseconds to huge spikes. This would be very bad in, like, trading systems, et cetera.

Gergely Orosz 20:41 Oh, yeah.

Sean Allen 20:41 So, like, the JVM was used primarily for systems that ran alongside trading systems, which were, you know, where... But e- even then, it could still cause problems. You have, you have a risk system which is supposed to be evaluating, uh, whether you should shut down trading on particular things or not. Um, and it has a spike like that that can still be hugely expensive and whatnot. So there was a company, uh, they made a, a... They still do, um, uh, Azul Systems. They, and they made a JVM which was pauseless, um, but it cost a lot of money, so most projects didn't actually use it, et cetera, and whatnot. And so we, we're looking at, okay, we're trying to replace the JVM-type systems for stuff where people don't wanna pay for Azul, and we need something which will be really fast. And we n- also need something where the, the guarantee that we had to make for people was, "We're not gonna lose any of your data, and we're not gonna mess any of it up," which is concurrency in, for speed, close to the metal for whatever that means in terms of having some level of control over data locality, et cetera, and all of the things that go into being fast. Um, if folks are interested in it, looking, looking into, uh, Mechanical Sympathy, a lot of stuff that Martin Thompson has done there, um, can be really interesting. But we were looking for this collection of characteristics where we need to know it's working, and we, and, and, like, our guarantees around not messing up your data were incredibly important to us. And I was like, I know C. I'm good at C. Um, I write C++ like it's still 1998, so I don't know if I would call myself good at C++. I still write C++ like it's 1998 for the most part. And I was like, I don't think that we can meet all of these guarantees, and I certainly don't think that I can hire people to work on a startup salary who would be able to pull this off with something like that. And I was like, I don't know what we're gonna do for this, but we continued to do consulting in this. And for one of the few useful things that's ever happened to me with LinkedIn, um, Sylvan, who I hadn't talked to in a few years, sent me a message and said, "Hey, look at this thing that I made." And it was a, it was an early version of Pony where he'd been working at, uh, Morgan Stanley.

Gergely Orosz 23:01 Yep.

Sean Allen 23:01 And he had built a thing that looked like a... It was, it was basically C, right? Um, and it was an actor model-type thing, which we could go into actor model in a bit. But, uh, it was all written in, people had to write stuff in C-

Gergely Orosz 23:16 Yep

Sean Allen 23:17 ... uh, on top of it. And it came with all of the problems that C still had in terms of concurrency and seg faulting and everything that it had way back when I picked it instead of Ada. And the runtime worked really well, but the people who were writing the C on top of it had to be really expert and had to do things just right, or they would mess up all of the great things that the runtime could give them. And so Sylvan then decided, "I'm going to go off. I'm going to learn a whole bunch of math so I can get really good at type system stuff, and I'm gonna go off, and I'm going to make my own programming language." And he did a PhD at Imperial in the UK to do all of that, and the thing that popped out of the end of that was Pony. And he sent me the thing of, "Hey, look what I made," and it, it was Pony. And I looked at it for what we were trying to do at Wallaroo, and I was like, "This is actually kind of great for us."

Gergely Orosz 24:09 And Wall, Wallaroo was a startup-

Sean Allen 24:09 Yes

Gergely Orosz 24:09 ... where he worked at an AI company-

Sean Allen 24:11 Yes

Gergely Orosz 24:11 ... right?

Sean Allen 24:11 Yeah. Wallaroo eventually became an AI company. At, at that point in time, it was a we're trying to sell high-speed data processing to banks, et cetera, and whatnot. Pony, nobody, nobody had used it at all-

Gergely Orosz 24:22 Yeah

Sean Allen 24:22 ... whatsoever. Um, there were, like, the, the best that we could say when we decided to adopt it, and people would be like, "What's that?" We'd be like, "Oh, well, it started as an idea at Morgan Stanley," which can actually buy you an awful lot of, uh, awful lot of room when you're trying to sell to banks. And the fact that Jane Street was basically the primary user of, uh, of OCaml anywhere in the world, and OCaml had become completely associated with Jane Street at that point.

Gergely Orosz 24:52 Mm.

Sean Allen 24:52 People were like, "Oh, so are you gonna, are you gonna be like to Pony what Jane Street is to OCaml?" And we're like, "Well, if we're successful enough, yes. That, that's kind of the idea." So, and that's a lot of backstory to, to answer the question. Pony is a, a language which is Absolutely in some way descended from the ML family of stuff. It has some, it has a lot of stuff in common with OCaml. Um, it has some stuff in common with F#. Um, it has some stuff in common with Rust because of that, um, where it's very much aimed at making it easy to write fast concurrent programs which are highly asynchronous and make it so that all of the rules that I learned, uh, to keep stuff from segfaulting are things that the compiler will enforce for you instead of, well, instead of, instead of you doing which what I, what I would do, which is and invariably screw something up.

Gergely Orosz 25:51 Yeah.

Sean Allen 25:51 Um, and, and have it crash on you.

Gergely Orosz 25:54 Yeah. I think you described Pony makes illegal states unrepre- unrepresentable.

Sean Allen 25:58 A large majority of them, yes, which, and in that way, it has a lot in common with Rust. Like one of the, but the, uh, the model for how it does that in terms of concurrency is very different. I find the actor model, which where, um, which is probably, at this point, most popular from Elixir, and maybe that used to be you could say that it was, it was Erlang, but now it's probably really Elixir on that. That the actor model makes it, uh, much easier to reason about, uh, concurrency and, um, uh, Pony gives you a built-in concurrency model the same way that Elixir and Erlang do in a way that you don't get with, uh, something like Rust and is very much the type system and everything is very much built around making it really easy to do asynchronous programming. Um, and I think it has just one of the best experiences for doing that combination of things, where there are other languages like Rust, which are very good at providing you with, um, uh, guarantees and whatnot, but aren't necessarily this full kind of, uh, uh, omakase-

Gergely Orosz 27:04 Mm-hmm

Sean Allen 27:04 ... um, sort of thing for no, actually, here's how you do fearless concurrency. You're, you're not picking and choosing a bunch of things. We've picked a bunch of things, and we put them together for, for you.

27:16 – 31:48

Rethinking garbage collection

Gergely Orosz 27:16 And can we talk a little bit of garbage collection?

Sean Allen 27:18 Yes.

Gergely Orosz 27:18 And we just mentioned how it's slow, but I'll quote you in a blog post you previously wrote, or this, I think this was a talk. It says, "Most engineers think garbage collection is slow. They're not wrong about their experience. Stop-the-world collectors really do destroy all your tail latencies, but garbage collection is slow confuses the symptom with the cause." What is the cause? And I mean, the symptom is that sometimes the world stops, right?

Sean Allen 27:41 Yeah. So, well, the first thing I would say is that most people, when they think of garbage collection, they think of like what the JVM does. Um, I think of garbage collection as a different thing, which is that garbage collection is a mechanism by which your program frees memory that it no longer needs. Because, um, there is, there is definitely this thing where sometimes people think about you on one side, you're doing like malloc and free and whatnot, and on the other side, you have garbage collection. But if you wanna go fast, you, you don't malloc and free all the time because this is really slow. These are system calls. The first step that people do is they, they allocate a whole bunch of memory at once, and then they start like slicing off little bits to themselves using, like, there are things called like slab allocators. They might do arena allocation. There's a whole, whole set of memory allocators. And then it becomes, well, I am now managing this memory. How do I know when it's free and when I can u- when I can, when I can free it, et cetera, and whatnot? You're already halfway to what most people think of as garbage collection at that point, and it becomes, it becomes a, a story of how am I doing these things, and what is the, what are the trade-offs that these things come with? And in terms of the JVM, uh, all of the memory is basically reachable from anywhere, from any thread, and the, and so it's very hard to be able to free memory in a way that doesn't cause you to have to do a stop-the-world.

Gergely Orosz 29:11 Mm-hmm.

Sean Allen 29:11 And the stuff that concurrent collectors like, uh, Java has the Shenandoah collector, Azul has theirs too, is actually really impressive engineering for how they manage to do what's called concurrent garbage collection. Pony has a garbage collector, which is instead of having a stop-the-world, what you have instead is that this is, this is a bit of a lie in order to get to the truth, uh, without going too deep. But each actor has its own heap, is the way that somebody could think about it, which is also how people describe stuff for the Erlang VM BEAM, which is also not technically true there, but it makes it easier to think about, and that each actor is responsible for freeing memory that it's used. So you do not, you do not garbage collect and stop the entire thing. What you do is instead is that when an actor is done doing some work, it can check to see through some heuristic, do I have memory that I can, that I can free?

Gergely Orosz 30:11 And with an actor, we can think of like as a, something like a thread, maybe a bit more lightweight than that.

Sean Allen 30:16 Yes.

Gergely Orosz 30:16 But it's kind of your own kind of like world that's kind of separate from other actors or-

Sean Allen 30:21 Yep, and so instead of, instead of I'm going to stop my entire program in order to free memory, it's like, oh, hey, well, I did 10 application operations, and now I'm gonna do one memory-freeing operation. And what you, what you end up with, with, with like Pony and with other concurrent memory-freeing things that are like that, where the things can operate independent of one another, is you end up with these things where instead of having these really spiky latency things like people are used to seeing if they run things like Cassandra or whatnot on a standard JVM setup where it's just like, whoop, my tail latencies are really bad, is that you just get basically flat latencies for the most part, or, or a tiny little bridge, and the thing that you give up on that is that at your fastest moment, you have a little bit less throughput than you might otherwise.

Gergely Orosz 31:14 Yeah.

Sean Allen 31:14 And this is still garbage collection, but it's just, it's, it's an entirely different, like, take on garbage collection. And And it's, it's a... Really the difference between that and what a lot of people end up doing in sophisticated, you know, memory allocators that they write for themselves and whatnot, is that at this point, they've completely, basically taken themselves as a programmer out of the equation. They've come up with an algorithm or something for deciding when memory should be freed instead of them having to try and code in what the particular rules for

31:48 – 32:51

Software correctness

Sean Allen 31:48 it would be.

Gergely Orosz 31:48 So let's talk about software correctness. How do you define software correctness?

Sean Allen 31:53 I would define software correctness as my program does what I expected it to do at, at a, at a really loose level. I tend to talk about things in terms of invariance, right, which is a more formal specification term. But in, in the end, invariant or it's, it's a guarantee of which even more loosely is my, my software is doing what I expect it to do. But in a lot of cases, people don't necessarily know what they expect their software to do. They know when it's not working, but they don't necessarily know when it's working. But software correctness is really about, like, it is a way of approaching software and a way of thinking about it, of being, of, of knowing that it is, it is upholding what it is supposed to do. It is software which is, you know, which is going to do its business purpose or whatever it was written for, and it's not going to wake anybody up at two o'clock in the morning. That, that is correct

32:51 – 35:41

Testing as a means to an end

Sean Allen 32:51 software.

Gergely Orosz 32:52 Now, for correct software, clearly you wanna, you know, do some testing. But you wrote a very interesting blog post, which I'm gonna quote you. "No one cares about testing. That's going to hurt the feelings of the people who do care about testing, but I'm going to stand by the statement. That might sound odd coming from someone who has a reputation for talking about testing, but I don't value tests. Tests are a means to an end. The means are unimportant; the end is all that is important."

Sean Allen 33:19 I wrote that because I was thinking about stuff where I, I'm at Antithesis, and people think of Antithesis as a testing company, and I don't think of it, it that way. I think of it as a software verification company and a correctness company, um, which is very different to me. Tests are things that you put in, uh, in order to verify things are working the way that you can for things that you had no better way to do it. Like, the worst case for a test is manual testing that people do a lot of times. And if you walk backwards from that, you can, maybe I can automate this. And you can get more and more sophisticated with the testing that you're doing. And there is absolutely amazing stuff that you can do with that testing. But in the end, you are still, you are still doing something at the late stages in the game, right? Whereas if you go back again to the, the C, C and Ada thing that, like, for a decision that I made really early on, C gave me almost nothing in terms of correctness. It was just a big, giant gun that I would point it at my own foot and shot my foot off with over and over again. And Ada would give me an awful lot of support for actually constructing correct software from the beginning, of not allowing me to do things that were gonna be incorrect. The whole reason why my Ada programs wouldn't compile was because they were fundamentally memory unsafe with what I was trying to do, and they were going to lead to some sort of memory corruption, a crash, a data, data race, any-anything like that. And, uh, you know, I, I believe very much that you should be, you should be doing as much as possible before you ever get to testing. That you should, you should make it so that these things where people are testing to make sure my thing is only, is only ever in one of these three states. If you have a, if you have a language, and ML languages are particularly good, um, at this sort of thing, you can usually represent within the type system, like, what the states are, what the allowable transitions are, et cetera, and make sure that things can't ever transition to a state that you weren't expecting. And this is just... It's huge because when you're trying to think about, "Does my system work?" you're not thinking about the entire thing all the way down. You're just thinking about the layer of stuff that your type system won't allow you to express.

35:41 – 40:52

TDD and property-based testing

Gergely Orosz 35:41 So I understand this with ML languages, that they allow you to do a lot more of this. But could we not play devil's advocate that if you do test different, test-driven development, you write the tests first and then you develop, you might do something similar?

Sean Allen 35:56 So I, I love TDD. Um, I think TDD is one of the best mechanisms for writing great APIs that people are going to work with because you're actively looking at, how do I actually work with this particular thing? The TDD ecosystem, um, is v-very much in love with these things that are called katas, and these are little, uh, programming exercises that you do. And there's-

Gergely Orosz 36:22 Oh, the, the daily ones.

Sean Allen 36:23 Yes. And w-well, there's, and there's, there's, there's one kata that usually comes in different forms, uh, which is, which is always a pair programming one. And it is, it is one person's job to write the test, and it's the other person's job to implement the thing which will make the test pass. And, and the thing that you're supposed to do as the person who's doing the implementing is you're supposed to do the simplest thing possible in order to make it pass. And the thing that people-- when people would do this with me, they quickly discover is I took that very literally. And I would, I would just hard code things over and over and over again into the implementation to make their particular example-based test, which is what most people do with, with TDD and unit tests, pass, right? So if it's like, "Hey, what's two plus two?" And that's the only test they have, I'm just gonna have it return four, right? And if they do two plus three, I can do a little thing that says, "Oh, okay, one of them was two, one of them was three. I'm not even gonna do this. I'm just gonna return five." The kata is not supposed to stop until the person who has the test is happy with the implementation, the person who's writing the test. And The only way to defeat somebody who's being very, very, very little at it is that you have to start randomizing the inputs. And the moment that you start randomizing the inputs, great, you're now into the land of what normally v- what people call property-based testing, right?

Gergely Orosz 37:53 Absolutely.

Sean Allen 37:53 Because you're generating results in order to get it to cover more space than you might come up with for examples.

Gergely Orosz 37:59 I mean, if, if I was on the other side of that, I'd, I'd, I'd probably be thinking of like, "All right, Sean's being a bit of a jerk here." [laughs]

Sean Allen 38:06 Uh, if you take that and, and you make it a bit more profane, th- that was what people would say to me on a regular basis, yes.

Gergely Orosz 38:13 Be- because, like, basically you're saying that you would go to the efforts of like let's say if I, you know, like a, a- addition, right? Like there's a question of like all right, like a sum. Like I, you, you put in numbers and it spits it out, very simple to implement, right?

Sean Allen 38:25 Yeah.

Gergely Orosz 38:25 If I start writing the tests and you write the implementation, you, you, you might just literally like start like write edge cases for like oh, if this, return that. If that, return this. And just, just the more tests you write you'll be like, "I, I can keep doing this. I'll just add more if statements in there."

Sean Allen 38:40 Yes.

Gergely Orosz 38:40 And you're saying the only way to beat you was to either have your test do random things or invent or come across the, the area of property-based testing where you can say for this property just run every single one of them. [laughs]

Sean Allen 38:54 Y- yes, some- something like that. And, and yeah, people would get frustrated with me with that. But by the time we were done with the exercise, you, we, we, we, the one place I worked we, we periodically would do this. We would like all the engineers would do this, and whoever it was who got paired with me would come out and be like, "I'm the only one who didn't finish the exercise." But they'd be like, "Also, oh my God, I think I'm gonna be much better at writing tests now."

Gergely Orosz 39:18 Mm-hmm. Yeah.

Sean Allen 39:19 And, and it's great, and this is, this is, this is a thing that can lead to much better tests. But in the end, like a awful lot of the things that were being tested were things that you wouldn't necessarily have to test if you had a stronger compiler.

Gergely Orosz 39:32 Mm-hmm. Mm-hmm. Yep.

Sean Allen 39:33 And you know, that particular place was a Java place, and there, there is a type system for Java which can prevent a number of things, but still allows an awful lot of freedom in the same way that, uh, the same way that C allows an awful lot of freedom. For years, um, I was, I was, I was a person who loved like, uh, dynamic programming languages. Smalltalk is still one of my favorite programming languages. I absolutely love it. I wrote a lot of Ruby, I wrote a lot of Perl, I wrote a lot of Python. And I was always like, "I don't really need the compiler's help for this," right? And I eventually realized it was well, yeah, but that's 'cause I primarily worked on small systems-

Gergely Orosz 40:13 Mm-hmm

Sean Allen 40:14 ... where I could hold everything in my head.

Gergely Orosz 40:16 Mm-hmm.

Sean Allen 40:17 And I only worked with one system, and it was just always fresh of mind. And when I started working on bigger systems and at bigger companies where there were more things that I had to work with and there were more people who I was working with-

Gergely Orosz 40:31 Mm-hmm

Sean Allen 40:31 ... more things entered the system where I didn't really know what it was, I started to really value like things ev- even with, with Java. You could just be like, "Okay, well, I'm supposed to be replacing this class. I'm just going to delete this class, and the compiler will at least tell me every single place that breaks, and I can start to have an idea of where do I need to explore the code

40:52 – 49:30

Designing testing workloads

Sean Allen 40:52 base."

Gergely Orosz 40:52 One thing you wrote about is this thing called workloads, and you actually did a talk on this, uh, and you wrote, "You're likely missing an important thing in your testing strategy, workloads." What are workloads?

Sean Allen 41:07 So, um, with things like property-based testing or-

Gergely Orosz 41:12 And property-based testing meaning that you, you say my function here is a property that can be passed. Let's say this is an integer.

Sean Allen 41:20 Yep.

Gergely Orosz 41:20 And then your test harness can like just-

Sean Allen 41:22 And-

Gergely Orosz 41:23 If it's an int 32 it could literally go through all the integers.

Sean Allen 41:26 It's, it's, yes.

Gergely Orosz 41:26 Or it can do random or it can do a thousand of them, et cetera.

Sean Allen 41:29 Yes. And but o- once you get past the, the simple cases that most people use for property-based testing, which is like here's a property for showing that, that, you know, you can reverse a list and then you can reverse it again and you have the same thing you started with, which is really simplistic. But if you wanna start applying this to I want to do property-based testing on my distributed system or on my database, for example-

Gergely Orosz 41:49 Mm-hmm

Sean Allen 41:49 ... the, the workload is whatever it is that you're going to use to validate that the thing is actually working, combined with the inputs that you're going to create in order to drive your code, in order to, in, in order to test it, right? As, as a simple example, if, if you wanted to test a key value store, right?

Gergely Orosz 42:11 Yep.

Sean Allen 42:11 Then-

Gergely Orosz 42:11 Like, like Redis or your own key value store.

Sean Allen 42:13 If you have a key value store, it has three operations, get, put, and delete. Um-

Gergely Orosz 42:18 Simple

Sean Allen 42:18 ... those are, those are all things which are going to be input into the system and are part of your workload. But as an additional part of your workload you're gonna have to decide how many keys am I going to have? What is the range of keys that I'm gonna look like? How big are the keys? How big are the values that I'm going to put in there? And your workload is this combination of all of this stuff. It's your system, it's your things that you're gonna use to decide whether it's working, and it's the inputs into the system as well. And it's very easy for people when they read about property-based testing and they see these simple examples of like oh, reverse a list, to be like, "Oh, okay great, it's randomness. Randomness is gonna get this for me." And so a, a simplified version of what somebody might do is be like, "Okay, I have three, I have three operations, a get, put, and a delete, and I'm gonna pick one of those operations to do." Talking about it in the simplified manner where we're ignoring the fact that you could have different keys and whatnot. If you do that, you're not actually going to trigger too many interesting places in your code because you have an equal chance of you do, you do, you do a put, and then you're probably gonna do a delete at some point-

Gergely Orosz 43:27 Yeah

Sean Allen 43:27 ... nearby, and the odds of you doing, say- 10,000 puts in a row and potentially pushing into areas where your thing might need to start swapping stuff out to disk or-

Gergely Orosz 43:39 Mm-hmm

Sean Allen 43:39 ... or do some type of garbage collection or whatnot.

Gergely Orosz 43:42 Or, or, or resharding, et cetera.

Sean Allen 43:43 Yeah. Or resharding any of those. They're simply never gonna happen because you're just moving right around this little-

Gergely Orosz 43:49 Yeah, just up, up and down for like-

Sean Allen 43:50 Yes

Gergely Orosz 43:50 ... one or two elements, which is-

Sean Allen 43:52 Yes

Gergely Orosz 43:52 ... trivial.

Sean Allen 43:52 It, it's, it's, it's a, it's a drunken walk, and drunken walks never get very far from where they start. So yeah, the, the, the workload is very much in, in part about, like, a big part of it is just what are the inputs that can, that can drive interesting behavior here? And sometimes for something like the key value one, it might mean, well, I, I need to drive it to really interesting places, but also other, other times, uh, workloads can be deceptively simple and, and validate really, really, really powerful things. For the simple side, um, because we need a whiteboard if we're gonna do the, uh, the input side. Um, for, for the simple side, there's, there's, there's a, there's a workload which is commonly used for testing, uh, isolation guarantees in asset databases, which-

Gergely Orosz 44:40 Mm-hmm

Sean Allen 44:40 ... I don't know, I think is probably the single most important thing that you want to guarantee with a, with an asset database. And it's called the bank test. And it is just about the simplest thing you could possibly... And it, it's amazing. So you set up some number of accounts that all exist in a table, right? And you say, "In this system, there's $10,000. And we're gonna div- divide this up amongst these... what- however many accounts we have." So we say 10 accounts, each one of them starts with $1,000, and there's $10,000 in the system. And all we're gonna do for this workload is we're going to just inside of a transaction, we're gonna transfer money from one account to another, which just means moving it from one column to another. And we're gonna have another thing that runs at the same time and is just constantly querying what is the total amount of money that's in the system. And if the isolation guarantees hold, then the only value it should ever see is 10,000.

Gergely Orosz 45:36 Mm-hmm.

Sean Allen 45:36 And this is a very simple test.

Gergely Orosz 45:38 'Cause g- you, you keep transferring between one another.

Sean Allen 45:41 Yep. And, and you, and you run many different transactions.

Gergely Orosz 45:44 Yeah.

Sean Allen 45:44 And the odds are if, if, I mean, if the database is good, then in a normal situation where you just run this on your laptop, you should never see any, any, any weird failures. However, you take this as, like, a distributed database, and you put it into an environment where you start in- introducing faults into it, like say, introducing network faults in between nodes, killing nodes off, et cetera.

Gergely Orosz 46:09 Mm-hmm.

Sean Allen 46:09 Um, Kyle Kingsbury with his Jepsen stuff made this hugely famous, regularly going to, you know, like the top of Hacker News and everything. It's in the same vein of what we do at Antithesis as well, and all sorts of interesting bugs start popping out from this really simple workload. And it's because it's exercising, like, a really core constraint there of, of the system, an invariant, which is that I should never be able to read something that's happening in inside another transaction.

Gergely Orosz 46:38 But the workload itself, is, is the workload you describing the system, here's the $10,000, here's the accounts, here's how you can move between them, or is the workload like one, one of these things, like moving, moving money between accounts and you're like, "Let's repeat this like many, many times"?

Sean Allen 46:54 Depending on, on how maximalist of a view you wanna take, right? You could, you could, you could be talking about it in a sense where the workload is also the particular database that you're testing in this entire thing. Because Postgres has a slightly different way of doing those things perhaps than like MySQL-

Gergely Orosz 47:11 Mm-hmm

Sean Allen 47:11 ... than some other, than some other database, and therefore, you could talk about that as a different workload. Or you could just talk about at a conceptual level, the bank test as a workload, where you're not even considering the database that's in it or any of the particular implementation details. And people regularly talk about workloads from both of those ends, from just the, the, the bank test i- in a, in a, in a platonic sense, right, is a workload, right? But then there's also the, you know, the non-idealized version of it where there's a particular implementation, you know-

Gergely Orosz 47:43 Mm. So-

Sean Allen 47:43 ... that ex- that exists here.

Gergely Orosz 47:44 So could I substitute in my mental model workload as a complex test case or a complex series of test cases and like this thing that we're, we're using to test, right? Like a workload is, is, is meant to test something or, or, or-

Sean Allen 47:59 Yes

Gergely Orosz 47:59 ... or play- or replay a scenario or something like that, right?

Sean Allen 48:02 Usually not replay a scenario. That, that, that, that would be, that would be a different thing.

Gergely Orosz 48:05 Not replay yet, but like construct a scenario.

Sean Allen 48:08 So I would avoid the word construct because now when we're, we're talking about construct, we're, we're getting back in towards a thing that looks like the example-based stuff that we're doing with unit tests.

Gergely Orosz 48:18 When, when, when... Yeah, when you write the specific thing.

Sean Allen 48:20 Yeah.

Gergely Orosz 48:20 So, so it's more like the point of the workload is that you set the rules, and then it kind of works itself out. Like in, in this case with the bank transfer stuff, we never say exactly how we should transfer. We just kind of want transfers back and forth randomly, non-randomly-

Sean Allen 48:35 Yeah

Gergely Orosz 48:35 ... ideally hit interesting edge cases, you know. Just-

Sean Allen 48:37 And-

Gergely Orosz 48:37 Just get creative with it, right? And that is the workload.

Sean Allen 48:40 Yeah. And, and if it's, and if it's like, if it's like most software that's been around for a while and has, and has gone through some sort of rigorous testing, then no bugs should pop out of a workload like the bank test. They, they shouldn't, right? Perhaps more complicated workload stuff might pop out, but it really depends. But for something like the bank test, it shouldn't. But bugs will start popping out like crazy from something like the bank test when you start subjecting it to like a Jepsen or an Antithesis-type experience, which is also the type of experience that happens all the time in production environments and that like, uh-

Gergely Orosz 49:16 Yeah

Sean Allen 49:16 ... Lauren Hochstein does a incredible job all the time writing up for, "Hey, here's what happened in this incident. Here's what happened in this incident."

Gergely Orosz 49:24 Yeah.

Sean Allen 49:24 "Let me, let me, as somebody who's like been versed in this stuff for a long time, try to break it down for you," and

49:30 – 57:44

Why quality isn’t a gate

Sean Allen 49:30 yeah.

Gergely Orosz 49:30 One interesting thing you said about quality, and I'll quote you again, is, is, uh, on how quality is not a gate. And so you wrote, "Quality is not a gate. The whole gate idea rests on a belief we're dragging into the light. The belief is that quality is something you hold by keeping the bad out. Catch the bug right at the door. It's a comforting picture, but it isn't how s- good s- good teams stay good."

Sean Allen 49:53 So this is, this is, this is really in a lot of ways why I don't like talking about q- Testing as a thing, and I like talking about correctness.

Gergely Orosz 50:01 Mm-hmm.

Sean Allen 50:01 Because, like, for as much as possible, your system for whatever it is should not allow bad things to happen, period, from just how you can write the software and static checks that you should be able to get. Like in an ideal world, you should be able to write a specification in like a, a language like TLA+ or-

Gergely Orosz 50:23 Mm.

Sean Allen 50:23 Or Alloy or something like that and have a perfect representation of that as an implementation. Or ideally just take that and, and put it... Like be able to magically go from that into an implementation which somehow also has all of your performance stuff, et cetera. We're nowhere near close to that world, but that would be an ideal world and you just simply know that no bad thing can happen because you've, you've used a model checker to prove this. And they can do incredible things for you, and we don't really have that. Um, and so another, uh, level that we can go to is just static checking inside of type systems for things, for, for the stuff that like Pony does, for the stuff that like Rust does, for the stuff that OCaml does. Um, in an interesting way, Erlang does a lot of this type of stuff as well, but it does it without a type system, but it does it with the actor model for we're going to limit what you can actually do in terms of stuff where you have to send everything by messages, and every single message, the data that one actor has is copied, uh, and sent to another actor such that there can't be any data races with the sharing of this memory because it's never actually shared, it's just copied. And in this way, we've also done a correct by construction sort of thing that you could also get from a type system or whatnot. And we want to have this foundation to build upon, and the, the testing aspect should be for the things that whatever our static means that we have available to us just can't enforce. Because in the end, we only have, we have programming languages, and the programming languages all come with trade-offs. And like, um, the programming language that maybe has the perfect type system to represent everything that you need to do, well, one, it probably doesn't exist. And then two, even if it did exist, it probably would not have some characteristic that you also needed as well in terms of performance or something else. So you'd need to settle some and pick a language that has better performance guarantees or, or what- whatever it is. And then at that point, what you're left with is testing and v- and verification.

Gergely Orosz 52:34 Yeah, but I kind of wanna push back on this, like-

Sean Allen 52:37 Yeah

Gergely Orosz 52:37 ... the quality gates, 'cause the idea of quality gates is like, okay, you build the code, you have a compiler, it does some stuff. The languages we use for practical reasons will, uh, they'll, there'll be a bun- bunch of bugs for application level bugs. Even if the compiler was perfect, it couldn't test for the, uh, the business logic being correct. And then the, the way we used to think about it, or, or I think a lot of people still think about it, is okay, you write this thing and you have quality gates. You have like code review, CI/CD, static analysis. I guess in production you have observability, and there's these gates where like, it's a little bit like sometimes you'll see the Swiss cheese model, like each layer will catch some stuff. Like code review, hmm, sometimes it might catch bugs, although we know it's not a great place to do it. But especially now we're not sure what's happening with code review, but it could catch bugs or, or readability. Inside of Google they have readability, uh, review, like it'll catch readability issues in, in their case. And then CI/CD will catch some, some bugs with the tests or some, some regressions that we've seen and we've coded things for them and so on. You're saying that quality gates is, is that quality is not a gate. So does that mean that you don't believe in having these, I guess let's just call them gates, in your release process? Or you're just saying just think about it differently, like think about it from the beginning. You're, it's, it feels like you're taking back the idea of quality to the, to the very beginning, right?

Sean Allen 53:51 Uh, very much there, and also I would say I, I disagree with your statement that, uh, business logic can't be statically, uh, enforced and whatnot.

Gergely Orosz 53:59 Okay. That's fair.

Sean Allen 53:59 Um, um, but let, let's, let's leave that, let's leave that rabbit hole aside for now.

Gergely Orosz 54:03 Yeah.

Sean Allen 54:03 But, uh, additionally, like testing, particularly in the way that it's, it's normally done outside of the Jepson and, uh, Antithesis type style, um, is-

Gergely Orosz 54:13 The, the, the unit testing, end-to-end testing when they're wrong. CI/CD, yeah.

Sean Allen 54:16 Is, is, is the, um, Honeycomb has awesome shirts, right? I test in prod. I mean-

Gergely Orosz 54:24 Yeah

Sean Allen 54:24 ... which is, which is-

Gergely Orosz 54:24 They're an observability platform and-

Sean Allen 54:26 Yes.

Gergely Orosz 54:26 Yeah.

Sean Allen 54:26 Yeah. And, and, and it, it, it's true that like everybody in the end in some way, whether they admit it or not, is testing in prod.

Gergely Orosz 54:34 Yeah.

Sean Allen 54:34 None of the tools that we have are good enough to guarantee whatever it is. Like even if you're doing an incredible job with like an Antithesis Jepson type tool and you're thinking up all these great and interesting inputs, there's gonna be some data which got corrupted at some point in time which exists-

Gergely Orosz 54:52 Yeah

Sean Allen 54:52 ... in your production system-

Gergely Orosz 54:53 Yeah

Sean Allen 54:53 ... which you haven't randomly hit, your generator simply can't generate-

Gergely Orosz 54:58 Yeah

Sean Allen 54:58 ... whatever it might be that is going to happen in production, and we do not know. It's out there and we are, we are testing it. And the idea that somehow we have this q- this gate where it's like, "Oh, if it's gone through this, it's quality, it can go out," is just, it's, it's kind of a lie, right? Like we have hopefully increasing levels of assurance as we go along, but we don't actually have a gate where we know it's quality.

Gergely Orosz 55:23 So but you're, you're saying like these things can help, it's just like know that there will be holes. Like there, there-

Sean Allen 55:29 Yes, there, there are going to be holes. We have, we have no-

Gergely Orosz 55:31 Things, things will sneak through and adding another gate, you're saying there's no one gate that you could add that will be like, "All right, this now seals off all the remaining holes." Uh-

Sean Allen 55:39 Yeah

Gergely Orosz 55:39 ... as lo- as long as we're looking at these kind of gates of like one more, you know, like-

Sean Allen 55:43 And-

Gergely Orosz 55:43 The same, the same as if, if you add one more test or two more tests or three more tests when you like hard code the values, those will, they'll make it better, sure, but they will, they will not-

Sean Allen 55:53 And-

Gergely Orosz 55:53 ... guarantee quality.

Sean Allen 55:55 So most of the places I've been where there are teams which I think are what I would call low functioning, uh, which are ones wh- where they tend to move Slowly, they don't deliver stuff a lot, or they deliver stuff quickly, but, like, they regularly have to do emergency releases to fix stuff, et cetera.

Gergely Orosz 56:11 Mm-hmm.

Sean Allen 56:11 Are people who think about, like, some sort of thing around traditional CI/CD stuff now, w- at least traditional now that's built around, like, pull requests or, or, or a staging environment where they run some tests before it goes out to prod. That's generally how they think about it. It's like, "Oh, this is, this is where I have these quality gates." And the, and the people who move quickly and with confidence, and then it actually holds up and they, you know, they're not getting alerted all the time, are people who think about quality from the very beginning of sitting down and being like, "Oh, hey, I should validate, for example, uh, if I'm doing a thing which is a distributed database," which is a particularly exempt one. I should be using something like a TLA+ to make sure that my consensus algorithm actually works, et cetera and everything, and to make sure that I don't have bugs in that. Or to take a Jepsen Antithesis type tool and put the very first version of your stuff before you ever ship it to production in there and go, "Does it behave like I expect it to?" Because it's really easy to fix stuff early on when you first create the thing, and it's really hard if you go, like, which I've seen many places do, where it's like, "Oh, here's our new system," and they don't really do any testing on it until, like, somewhere way down the line and nobody has any idea whether it's, like, actually working or not, or even necessarily any idea to know what does working really mean. Like, they haven't asked themselves from the beginning, "How will I know this thing is working?"

57:44 – 1:00:50

Testing complex systems

Gergely Orosz 57:44 But, uh, I wanna push back on, like, you know, that you can just write a test for this if it's working. I'll give you an ex- example where it might be really hard to write test that it's working. A Go- Google search. I type in, "The best testing tool in the world," right?

Sean Allen 57:59 Yep.

Gergely Orosz 57:59 And if I ask 10 people, and five of those work at vendors, they all say something different should be the number one, right?

Sean Allen 58:06 Yes.

Gergely Orosz 58:06 The people who work at vendors, they will say, "My vendor should be the number one. Obviously, I'm working here. We're doing amazing stuff in, uh, in testing." Some people will think, oh, like, they'd like the Wikipedia article. Some people will say, like, "I think this, this, this should happen." And, and of course, when I run the same Google search day after day, those things will change a little bit, like the, the results will change.

Sean Allen 58:27 Yep.

Gergely Orosz 58:27 And we can assume that Google is, is, knows what they're doing and, and is correct. But now if I wanted to just even write a test for, for a complex system, and of course, we know the reason it changes because Google calculates the page rank, and they take a bunch of signals, and they also take things like real time, so, like, news will be fresher. But now we have this really complex system. And if I'm... I will be given the simple task of, like, well, write some tests so that we can tell if this small segment of Google search works fine, let's say inside of Google. I would now struggle 'cause I'm like, well, you know, if we went like, all right, like you can use any, any formal language or, or TLA+

Sean Allen 59:02 Yeah

Gergely Orosz 59:02 ... I would struggle to code. Like, what, what am I expecting? E- even though when you look at it, you can kind of squint and say, like, "This looks good enough."

Sean Allen 59:09 Well-

Gergely Orosz 59:09 Or if you have enough data.

Sean Allen 59:10 But, but there are some things that we, that we, we, we can test. For example, there's some invariants we can do.

Gergely Orosz 59:16 Mm-hmm.

Sean Allen 59:16 Which, like, for example, I put a thing in the box that I submitted. I expect to get results back.

Gergely Orosz 59:20 That's one in-

Sean Allen 59:20 Right

Gergely Orosz 59:20 ... so that's one, one invariant, for example.

Sean Allen 59:22 Like, there is, there is an invariant there, and then we start getting into things and, you know, I did the machine learning stuff for a while, which is actually in recommendation systems.

Gergely Orosz 59:30 Mm-hmm.

Sean Allen 59:30 And that gets, that gets into the interesting thing of you, like, oh, well, what should the number one ranking be, right? And that is, that is a, that is a very different thing than is it working, right? Like, that's very different. Is it working is did I put a thing into the box and I got something back? It... the, the is it relevant is an entirely different thing. Like-

Gergely Orosz 59:49 Oh, and then with the is it working, we can now ask questions like if, if, if I submit like a, a thousand requests in a second, like is it still working? If, if, if we have concurrent requests coming in, is, is it working? If in the middle of submitting a request I submit a second one or navigate away and so on.

Sean Allen 1:00:04 Yeah.

Gergely Orosz 1:00:04 So like we-

Sean Allen 1:00:04 And, and-

Gergely Orosz 1:00:05 So now you're saying we can decompose this into different parts of a problem.

Sean Allen 1:00:08 And, and are the results that I'm getting results that are for my actual query?

Gergely Orosz 1:00:12 Mm-hmm.

Sean Allen 1:00:13 Where, where what's in it might not matter, but we, we can, we can set up tests so that each query gets some type of marker that goes along with it, and we make sure that those actually came back and that the server didn't get confused and give me your results, which, okay, maybe for search, I don't know, I don't know how anyone would ever know if anything happened. But I think everybody's heard about the incidents where, uh, like, you know, the X service started showing, you know, somebody else's account to people and whatnot. That's a really bad one there, you know.

Gergely Orosz 1:00:41 Yeah.

Sean Allen 1:00:42 As somebody who's had their identity stolen three times, I really hope that, that, you know, nobody's showing my PII to, uh, somebody else accidentally.

1:00:50 – 1:09:57

Supply chain security

Gergely Orosz 1:00:50 So I, I want to talk about some stuff that are interesting learnings in, in the industry, and I know you're passionate about them. One is supply chain security.

Sean Allen 1:00:58 Yes.

Gergely Orosz 1:00:59 What, what is supply chain security? I mean, I, I think we're hearing, hearing it a lot more-

Sean Allen 1:01:03 Yeah

Gergely Orosz 1:01:03 ... unfortunately, especially in open source, right?

Sean Allen 1:01:06 Yep. So basically, supply chain security is talking about if, if you have something which is popular, right, then the easiest way to compromise that thing is-

Gergely Orosz 1:01:15 We're talking software, open source software.

Sean Allen 1:01:17 Yeah, software, software, is to find the weakest link for it and to attack that, right? Which, and this, this applies to software, but this also applies to hardware, for example.

Gergely Orosz 1:01:29 Oh.

Sean Allen 1:01:29 Right?

Gergely Orosz 1:01:30 Right.

Sean Allen 1:01:30 Um, where there can be all sorts of security guarantees that, like, um, AMD makes where they have, uh, trusted, uh, computing modules which are in there, which Intel makes. But if you, if you really want to attack those systems, what you wanna have is somebody who when they're putting this together can put a malicious firmware onto those machines.

Gergely Orosz 1:01:53 Yeah.

Sean Allen 1:01:53 How do you know, how do you know that that is actually- Like the firmware that it's supposed to be. And yeah, I, I have a, I have a security background as well, and this, this gets talked about as, like, the trusting computing base. But people usually talk about this, um, when they talk about supply chain security at this point now, usually about software and things like... Wasn't it Left-- Was Left-Pad a supply chain security one years ago where somebody was like, it was like this little module that was on npm that did something, and somebody put, like, a crypto miner or something in it?

Gergely Orosz 1:02:25 It, it-

Sean Allen 1:02:25 Yeah

Gergely Orosz 1:02:25 ... it, it was typically, like, I'm not sure how trivial it is, but, like, it's replacing a package-

Sean Allen 1:02:30 Yeah

Gergely Orosz 1:02:30 ... that is widely used. There, there was a, some very simple package that was doing some trivial stuff.

Sean Allen 1:02:36 But, like, so a, a great way that this could happen, for example, if you can go after something which people use and they don't think about too much, particularly, say, in, like, a CI/CD system, and you can get your code running inside of the CI/CD system, you now have access to all of the secrets that are running inside that CI/CD system, which means that you can start publishing software which might have malware in it, who knows what else, and whatnot. Great. That's, that's one. But even more so, like, almost every single programming language that exists right now, just it's basically, it's basically like, here you go. There's something. I downloaded it off the internet. Maybe I, maybe I checked its hash or whatnot, and it can do literally anything. I downloaded a thing which is supposed to pad out some text, but it's actually allowed to, I don't know, run a crypto miner, make, make network connections, et cetera, and everything. And, and all of these, these are, these are supply chain attacks, and they can have really devastating effects.

Gergely Orosz 1:03:39 And, and they're happening. We saw it in npm. I think we're, I think we're seeing it in the Python, uh, community as well.

Sean Allen 1:03:46 Yes.

Gergely Orosz 1:03:46 There's a quite good question of what are efficient defenses? Because, you know, like, there's a couple of naive defenses. One is, "Oh, let's just have a cool-down period. Let's not get any new packages for at least a week, and we're hoping that if there's a supply chain attack, it will be big enough that it'll be published, and it'll be resolved." Of course, the downside is that if there's a hot, hot, hotfix, there's literally a security vulnerability, we're now leaving ourselves open for a week. But that's kind of one naive. The other very naive is, "Let's get rid of our dependencies. Let's just bring it in-house." Or let's... And then the third one is, "Let's fork it. Let's have this, this fork, and maybe we can have a security team who can decide to, like, update or merge or keep it in-house." But these are the kinda, like, very... All of them are very naive and-

Sean Allen 1:04:33 Yeah

Gergely Orosz 1:04:33 ... and I'm, I'm wondering, what are you seeing work and being practical?

Sean Allen 1:04:37 Yeah.

Gergely Orosz 1:04:37 'Cause y- it seems you're trading off kind of latency of being able to get fresh packages which often have either critical fixes or really good functionality for being afraid that what if that thing is poisoned?

Sean Allen 1:04:49 And, and, and if everybody decides to do the cool-down period, who's going to notice?

Gergely Orosz 1:04:54 Exactly.

Sean Allen 1:04:55 Right.

Gergely Orosz 1:04:55 We, we haven't even talked about that, right?

Sean Allen 1:04:56 Yes. Yes. Um, so I think part of this is, is that there is no good answer for almost every single programming language that exists, and any of the ones where they might have a good, uh, answer to this, um, are ones that nobody uses. Because it, it, it really has not been a concern for a long time.

Gergely Orosz 1:05:14 Yeah.

Sean Allen 1:05:14 The way that pretty much almost all package management and almost all open source does, uh, dates back to, like, the way that Perl did with CPAN, and which was a, which is an incredibly open and trusting thing in the same way that almost all early internet stuff was, where it's really easy to spoof somebody sending an email, all of this stuff. It was, it was a network that just assumed everybody was a good actor.

Gergely Orosz 1:05:38 Yeah, and I guess back then there was a higher bar for entry as well. Like, if you're a malicious actor, you at least needed to learn to code, [chuckles] or you needed to hire someone, which again, is now a little bit trickier, 'cause once they know what we're doing, it's a higher b-barrier to entry. But, but that's changed, right? Even the barrier to entry has changed.

Sean Allen 1:05:55 Yeah. But e-even, even back then, it wasn't true. One of the, one of the people who was a high school mess-up who I went to college with was somebody who'd been arrested by the US government for unleashing, when he was a teenager, a worm, like, across a whole bunch of government computers that he'd gotten into.

Gergely Orosz 1:06:10 Hmm.

Sean Allen 1:06:11 And, and, and that actually got further out and was one of the, one of the big incidents at, at that time. So, like, I mean, it was still happening then. It was just, it was at a smaller scale, and not everybody was depending on these things at th- this particular point in time. And, and the fundamental thing is that for any dependency, pretty much for any language like JavaScript or Ruby or, or, or Rust or anything that, that comes in for the most part, basically the code that you wrote has the exact same capabilities for what it can do as the code that you happen to, you bring in from the internet. And your code can do absolutely anything, probably, and that other code could do absolutely anything as well. And so how are you possibly going to vet all of this code if you're going to have all of these dependencies, which gets back to the, to that problem. And in the end, nobody is going to solve this using the existing stuff we have. People might try to do sandboxing or whatnot, but that's just, in the end, like, every sandbox can be escaped from. It's usually not that hard and whatnot. And th- there's a whole area of, uh, study which, which is in these areas. Uh, uh, there's a guy named Mark Miller who developed a, a language called E, which has some stuff called object capabilities, um, where there's a lot of interesting research. We're, we're here right now as, as part of Bug Bash in, in Copenhagen, and the first talk of the day was, uh, from Sophia. She's a professor at, uh, Imperial, which was actually talking about, like, some object capability work, et cetera, they've done. But the basic idea of object capability is that in order to take any action within the system, you have to be given the capability in some fashion to do it so that the system would not allow you, for example, to open, uh, a network connection unless you have the capability to do that. And the originating program, your code that is written, uh, has to give that capability to other stuff. And there, there, you know, there's no popular languages that do this, um, because, you know, because Pony is written by, like, Sylvan initially and myself now, the object capabilities are built in. But this is, this is a m- this is one means to, like, to be able to have it to be like, "I can trust this code more." But, but you, you need to go, you need to go further than that because if your, if your programming language allows for some type of foreign function interface, which almost all of them do, so that you can utilize, like, C code or, or Rust code or anything using a C ABI, um, if you don't have some way to control that completely, then it doesn't matter if you have object capabilities 'cause somebody can just go out the, uh, the FFI door. And all of these are things which need to be addressed, and there's no way that these things will be addressed by just incrementally doing stuff within the system that we currently have.

Gergely Orosz 1:09:07 Well, it sounds like there's a reason that software security is getting more important, and it's a complex topic, I think.

Sean Allen 1:09:12 Yes.

Gergely Orosz 1:09:12 We, we, we did a podcast episode on software security, but it felt we were just scratching the surface 'cause as with anything, right? It's, it's a nuance. It's, it's a detail where it gets complicated. And, and I'm sure there are so many strategies every, every company and team decides on what strategies to do, from the very simple when you're a small startup and... Or, well, you're just YOLO at first, and then-

Sean Allen 1:09:30 Yeah

Gergely Orosz 1:09:30 ... and then you start to do, again, a simple thing. It's just a cool down, hoping that not everyone cools down and it'll be okay. And then it gets a lot more nuanced. Scanning for dependencies, evaluating them, reducing unnecessary ones, scanning dependency chains, and so on. Like, you can go really deep. You can go to go all to point-

Sean Allen 1:09:47 Yeah

Gergely Orosz 1:09:47 ... to have, have a security team who will vet them or try to verify them or, or even maybe have contact with the, know who's building it and so on and so forth.

1:09:57 – 1:11:54

Side-channel attacks

Sean Allen 1:09:57 Yeah. And the, the really horrifying part of all of this is we could do all of these, all of these things, the ones I've mentioned and even more, and none of it, in the end, would be a really good defense because we made a, we made a trade-off years ago in order to go faster with our CPUs. Which is, we introduced speculative execution, and speculative execution works 'cause it's like, "Hey, I have extra cycles, and this thing could go down this branch, or it could go down this branch, and I'm gonna execute-"

Gergely Orosz 1:10:27 Keep both of them

Sean Allen 1:10:28 ... both at the same time, which has given us the vast majority of the speed-ups that we've gotten for, like, the last, uh, over 20 years at this point. Um, and, and people have probably seen stuff for, like, side-channel attacks, is generally what they're called, and there's a whole variety of these. Because the thing is, is that those speculative branches have to actually exist in memory. And so if I am good enough as an attacker, I might... You, you executed a branch for what happens if I'm authorized and what happens if I'm not authorized.

Gergely Orosz 1:11:01 Mm-hmm.

Sean Allen 1:11:01 And it turns out for a sufficiently, like, knowledgeable person, which doesn't actually take that much knowledge to get, even if they don't actually have the access to the system, can use a side-channel attack to read the memory that was created for this is what the access actually is. Or even better, they can find, like, they can, they can get the, the keys out of, out of memory that are used for security, et cetera and whatnot. So we could do everything else right, and as long as, like... I worked at, I worked at Microsoft Research on the confidential computing group.

Gergely Orosz 1:11:33 Mm-hmm.

Sean Allen 1:11:33 And we built, uh, a, a confidential container offering, which is, which is still there at Microsoft. And the thing that we always said about our security guarantees, which were great, was we tacked on at the tail end, "modulo side-channel attacks." Which is, we can't protect you against side-channel attacks. Every single guarantee that we have is violated

1:11:54 – 1:14:38

Confidential computing

Sean Allen 1:11:54 by side channel.

Gergely Orosz 1:11:54 And what is confidential computing?

Sean Allen 1:11:56 Confidential computing is computing where basically the end goal is that the software can be run in an environment where the operators of the environment are not trusted.

Gergely Orosz 1:12:08 Mm-hmm.

Sean Allen 1:12:09 Because in insecure-

Gergely Orosz 1:12:10 So, so, so, like-

Sean Allen 1:12:11 ... in security-

Gergely Orosz 1:12:11 ... you deploy something on the cloud, and you're not sure who's running that cloud.

Sean Allen 1:12:15 Years ago, there was a thing that, that happened where it turned out, uh, some Twitter employees were actually, uh, working as agents for the Saudi Arabian government and were spying on the activities through DMs, et cetera, and stuff that, uh, that Saudi dissidents were doing.

Gergely Orosz 1:12:30 Yeah.

Sean Allen 1:12:30 It, it, it's... It could be this sort of thing, or it could be the thing where, you know, oh, you might trust Microsoft or Amazon, and you don't think they're gonna steal your data. But actually, you're trusting everybody who has their access, which means that you're trusting every employee not to get phished.

Gergely Orosz 1:12:47 Yeah.

Sean Allen 1:12:47 Or you're entrusting every employee not to have a money problem where they could be bribed-

Gergely Orosz 1:12:52 Yes

Sean Allen 1:12:52 ... et cetera, whatnot. And so confidential computing is an attempt to make it so that you don't have to trust those people. Because in security, the more things you trust, the less secure that your system is.

Gergely Orosz 1:13:04 Yeah.

Sean Allen 1:13:04 Right? And so, yeah, in the confidential computing stuff, we could provide these great guarantees for people, except for side-channel attacks, which could, which can violate all of the guarantees. Like, the only, the only way to possibly to fully lock it down at that point would be is to say that speculative execution is going to be removed from the system. And, and when the first speculative execution attacks started coming out, like, they patched the Linux kernel to handle just one individual case, and I don't remember the exact numbers. But it was like, oh, like, here's a... I think the first one they did was, like, a 50% performance drop, and they tried to, like, claw back performance from that, and they... But it was like, oh, and no, here's another attack, and here's another attack, and here's another attack. And at a certain point, I think people just kind of realized, well, we're just gonna play Whac-A-Mole because we don't, we don't wanna give up on the, um, on the performance we can get from that. And I'm very concerned about security, but I also, in open source stuff I write, I absolutely do things where I'm taking advantage of speculative execution. And the side of me that wants to make stuff go fast loves it 'cause I can just be like, "Oh, yeah-" The branch predictor, which works off of this and knows that normally it goes down this path, will make these, this branch, which used to be really expensive, really cheap to do, and this means I can do things that would be really hard to make go fast now and make go fast, and we've all kind of made that, like, that deal of we're willing to go fast, but, oh, it's just fundamentally insecure.

1:14:38 – 1:20:14

Rethinking technical debt

Gergely Orosz 1:14:39 I wanted to ask you about tech depth. Uh, 10 years ago, you wrote something, and I'll read it back to you, and I, I wanna b- I'm curious how your thinking might have changed. You, you wrote this about a horror story at a, at a company or horror stories that you've heard. "Here's the thing, though. We were making money even though these kind of horror stories were happening. We were growing and supporting ourselves. We were continually creating value for the company. Yeah, we had a lot of technical debt, but we kept finding inventive ways to sidestep having to pay it. The debt metaphor we've adopted is misleading. You don't always have to pay off the debt you created. You can work around it and avoid ever having to pay it off. Technical debt only hurts if you have to pay it back."

Sean Allen 1:15:15 Yes. I, I, I com-- One hundred percent. Yes.

Gergely Orosz 1:15:19 Still stand by it.

Sean Allen 1:15:20 Still, still, still stand by that.

Gergely Orosz 1:15:21 This, this was 2015.

Sean Allen 1:15:22 I mean, I am very much a person who wants to build malleable software, et cetera, and whatnot.

Gergely Orosz 1:15:28 Malleable as in?

Sean Allen 1:15:29 M- malleable as in stuff which will, can adapt to, like, the new situation-

Gergely Orosz 1:15:33 Mm-hmm

Sean Allen 1:15:33 ... because the situation within which software was built is not going to be the situation that's going to be in a year, in two years, in three years, et cetera. And so it needs to be stuff which can be made to change, and that doesn't necessarily mean change the code, but can be made to fit, like, the new environment that it needs to be in, where one of the answers to that is oftentimes just throw the thing away. That particular one, I believe, was motivated by one where, um, there was an awful lot of debt that was stacked around how the database was set up and a limit to sharding that could happen from MySQL and it getting harder and harder, and that ended up being completely sidestepped by throwing money at a, uh, a distributed, uh, a distributed database, which still made it look like a single database. The trade-off there was, one, you pay, we paid them a decent amount of money, and two, there were weird edge cases in terms of, like, the actual, like, guarantees that you would expect from, like, a single-node MySQL system-

Gergely Orosz 1:16:37 Mm-hmm

Sean Allen 1:16:37 ... you wouldn't actually always get, but they were so very rare that it's like this is not a company which is using Ada, um, et cetera, so this incredibly rare thing which I would occasionally track, like, things down to, like, is, is not that important. You could just completely sidestep it. In a lot of cases, you know, it could be, "Hey, the business has changed, and that system is no longer needed." Like, it ran for three years. It's not needed anymore. It's gone. You never had to pay off any debt that you, you had for that thing.

Gergely Orosz 1:17:08 So, so you're, you're, you're saying that maybe a lot of us are not thinking about tech depth the way that actually happens, which is you often create it. Like, you just do a bunch of hacks or, or just w- like, stuff that you know would not make sense, let's say, at a massive production company, but maybe you're a startup or early stage, and then later on, you might never have to pay it 'cause you might just either write new software or just introduce a new piece of technology that just invalidates it.

Sean Allen 1:17:33 Or not find product market fit.

Gergely Orosz 1:17:35 And then-

Sean Allen 1:17:36 Run out of runway.

Gergely Orosz 1:17:37 And then why does it matter?

Sean Allen 1:17:38 Yeah. Yes.

Gergely Orosz 1:17:39 Yeah, I, I still remember when I worked at Uber, I was on a rides team or, like, our-- We were building the rides product, uh, which was Uber's mature product. It's been around for years. It was making serious money, and the Uber Eats team was just starting out. And one of the engineers came into our code base and, like, committed a change 'cause they needed the, the shared functionality. They needed to use some of that platform. And they came in, uh, they deleted a test that was failing after they made the change, and they went out, and they made the change. And we're like, "What are you doing?" Like, you cannot delete the test. And we got into this standoff. They were like, "Well, look, it just n-need to n- just needed to get it done," or it wasn't even needs. It was, it was jump, jump bikes. And at some point, I started to understand that actually, like, these people, like, they, they really are thinking only about tomorrow 'cause they're not sure if they're gonna be around the week after. And sure enough, Uber sold off that whole division six months later, and it didn't find, I guess, product market fit inside of Uber. It became its own company. But I was like, "Oh, okay." I guess when you're thinking like that, you actually just wanna delete tests. Like, you're like, "Works. Let's roll it out. Let's see if..." They didn't even care if it worked for 100% of customers. They cared if it kinda worked for their purposes for, for, at that point.

Sean Allen 1:18:50 Yeah, and, and, you know, sometimes you really do pay the cost, right? You do. And usually you're paying the cost because you were successful. However, you could also... I mean, it could also happen that people are not paying attention to-- They're, they're not paying attention to debt, but they're also not thinking about it in a way of, of what is the most malleable way to do this particular thing, which I can-- So that I can do it quickly in order to validate things. Because it's very different between I'm just going to hack this thing in here versus a I'm going to come up with a thing where I'm gonna do it quickly as possible, and it's not the full thing. It fits some tiny little sliver perhaps, and there's all sorts of edge cases that it won't handle. But it's easy for me to remove-

Gergely Orosz 1:19:42 Mm-hmm

Sean Allen 1:19:42 ... and if we build five more things on it, like, in the next three months, it's not gonna slow us down because ultimately we are a, we are a startup, for example, that is just trying to figure out what we're doing as quickly as possible. And yes, you can very much get yourself into a position where unfortunately before you've even found product market fit, you might have debted yourself into a corner. But the idea that, that exists in a lot of stuff where it's like, "Oh, no, every single bit of debt is bad, it always has to be paid off," is just... I mean, it, it's not borne

1:20:14 – 1:25:17

Going slow to go fast

Sean Allen 1:20:14 out in reality.

Gergely Orosz 1:20:14 So one, one theme of both what we're talking about, what, a-as we were talking earlier about your career and how you build software is how you love to go fast.

Sean Allen 1:20:23 Yes.

Gergely Orosz 1:20:24 And that you've learned to go fast, sometimes you need to go slow. And you, you write about this as well, so I'll quote you on this. "Go slow to go fast. It means writing a planning document and coming to agreement on what we're doing before anyone touches code. It means knowing how we'll test a thing before we start building it. It means taking a few hours or a couple of days to throw away spike on the parts we don't understand yet. It means stopping when production is on fire at 2:00 AM." Some of these are, are so counterintuitive, right?

Sean Allen 1:20:56 Not to me.

Gergely Orosz 1:20:57 Not to you.

Sean Allen 1:20:58 Not to me. I mean, I think they absolutely probably were counterintuitive, like, earlier in my career and everything, where it was just like, "Oh, no, we need to get that thing done." But it, it's kind of like the thing where, where eventually you realize actually working 80 hours every week is not resulting in me going faster. It's just resulting in me making more mistakes because I'm tired all the time, and I'm constantly now spending my time tracking down bugs instead of working on new functionality, and that really when I get tired, I should just step away from the computer. That is a, "I'm going to go slow to go fast. I'm going to make sure that I'm rested, that I'm healthy, that my, my brain, which is a key part of all of this, is actually functioning correctly." One of the worst time-wasters that I've, I've seen in my career is people not spending time to make sure that when they're talking to each other and using words, that they actually mean the same thing.

Gergely Orosz 1:21:55 Mm.

Sean Allen 1:21:56 And like, three weeks later, the thing pops out the other end and somebody looks at it and goes, "That's not what I wanted." Like, this is, this is why, like, with Agile, th-this id- the idea of sprints was supposed to bake in this, we do not go very long without looking at it to make sure that we are in agreement for the direction that we're going in. If you're going to build a big, complicated thing, it behooves you to spend a few days where everybody just... The first part of development is not writing code, but it's just writing down how the thing is going to work, what it's supposed to do, and making sure everybody actually agrees on all of these things. Because it's very easy to change things there. You haven't invested a lot of time in coding the thing, et cetera. You're just working to make sure, are we all actually talking about the same thing? Years ago, there was a, a wonderful argument from people on the internet in the distributed systems community where the two sides were just going back and forth at each other, and they were going back and forth at each other because they were talking about what exactly once means in terms of, like, delivering messages.

Gergely Orosz 1:23:05 Oh, exactly once delivery.

Sean Allen 1:23:06 Right.

Gergely Orosz 1:23:06 Yeah.

Sean Allen 1:23:06 Yes. And, and, like, one group of people meant exactly once processing, and they were talking about idempotence and a thing only actually having an impact one time, and the other group was talking about the message literally arriving at a computer only once. And these people over here are like, "You're liars. You can't do exactly once." Like, and the... And they just spent all of this time yelling at each other because nobody actually bothered to sit down and go, "This is what I mean by exactly once."

Gergely Orosz 1:23:38 Yeah.

Sean Allen 1:23:38 And the same sort of thing, like, can come up, like, all the time. You send somebody off, it's like, "Oh, hey, I jotted down this little thing on a card." And I guess people use like Linear, they probably still use Jira, whatever it is now. "And I jotted down this little thing. You go pick up the thing, you do it," and then, like, this person spent, I don't know, an hour, a day, five days building this thing, and it's like, "Oh, yeah, no, I... D- that's not what I... That's not what we were expecting." And you went fast and you got nowhere, right? Like, you spend the time to establish the foundation for whatever that is, make sure everybody's pointed in the same direction. And as part of it, like, for example, if you wanna go fast later on, you should be thinking when you're designing a thing for how you're gonna know that it's actually working right from the beginning. Because there are ways you can build a system where it's almost impossible to know whether it's working, or you can build it where it's easy to know that it's working, which can include some of the stuff I've talked about from static stuff. But even just making it so that you have good observability into the system, right?

Gergely Orosz 1:24:42 Mm.

Sean Allen 1:24:42 Like, we're building a system and we're gonna put it out there in production. How are we gonna know whether it's down? How are we, how are we gonna know whether it's got gray failures, where a whole bunch of people are getting a really bad experience, but it's still up and pinging in the load balancer or whatnot? Just sitting down and thinking about these things and making sure that they're things that will be easy to do and are not going to drag you down later on so that you spend a little bit of time up front, get everybody on the same page, figure out what you're gonna do, and then you can just go really fast after that.

Gergely Orosz 1:25:12 Awesome. Well, well, Sean, this was awesome. Thanks for having this conversation.

Sean Allen 1:25:15 Yes. This, this was wonderful.

Episode duration: 1:25:19

In the uListen app, the transcript lights up word by word as you listen. Get uListen

Transcript of episode OrXKvOjR-vI

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.