269 - Good vs. bad science: how to read and understand scientific studies

1. 0:00 – 0:18

   Intro

   1. PAPeter Attia0:00

      [upbeat music] Hey, everyone. Welcome to "The Drive" podcast. I'm your host, Peter Attia. Hey, Bob. How are you, man? Looking pretty studious there in the library today.

   2. BKBob Kaplan0:13

      Hey, Peter. Thanks very much. Yeah, just getting some reading in before the-

   3. PAPeter Attia0:16

      [chuckles]

   4. BKBob Kaplan0:17

      ... before the podcast.

2. 0:18 – 3:19

   The ever-changing landscape of scientific literature

   1. PAPeter Attia0:18

      [chuckles] Well, this is, uh, this is gonna be a pretty good one because, um, as you may recall about, um, I don't know, four or five months ago, maybe longer, I was on a podcast with Tim Ferriss, and [lip smacks] uh, I don't know how it came up, but I do remember somehow it came up that, um, you know, we had spent a lot of time writing this series, "Studying Studies," and, uh, God, that's been four years ago, I think. Um-

   2. BKBob Kaplan0:45

      Yeah

   3. PAPeter Attia0:45

      ... but [clears throat] we didn't really have something more digestible for folks on how to make sense of the ever-changing landscape of scientific literature and, uh, you know, in some ways how to, how to kinda distinguish between the signal and the noise of, of the research news cycle. Um, and I remember after that, Tim and I went out for dinner, and he, he kept sort of pressing me on, "Well, what can I do to get better at this process?" Um, you know, are there newsletters I sh- I should be subscribing to and things like that. And, and while I'm sure that there are, I didn't know what they were off the top of my head. Um, and so I think what we've done here, w- when I say we, I mean you, um, w- what, what you have done here is kind of aggregate all the questions that have come in over the past year, basically, um, that pertain to understanding the structure of science. And, uh, I really ... I don't know. I looked through the questions, uh, last week, and I was pretty excited. I think it's gonna be a s- sweet discussion, and I hope this serves as an amazing primer for people, um, to, to really understand the process of scientific experiments a- and, and everything from how, how, how studies are published and, and obviously what some of the limitations are. So anything else you wanna add to that, Bob, before we jump in?

   4. BKBob Kaplan2:02

      I agree. I think it's a fun topic. We get so many, we get so many of these questions that we end up, or at least I do, um, or to the website, where we'll point readers to one of the parts of the "Studying Studies," but I think sometimes just talking about it and explaining it can help a lot. So I think this will be really useful, um, as far as quest- like a question and answer session rather than just a, just reading a blog.

   5. PAPeter Attia2:28

      Yeah. And, and, and again, I don't think this displaces that other stuff. I think we go into probably more detail on some things there, but I also think we're gonna cover things here that aren't covered there. So, um, depending on how you like to get your info, this, uh, this could be fun. So where, where do you wanna start?

   6. BKBob Kaplan2:44

      So I think the ... We have, again, a lot of questions, but I think the, th- this question gets to the core of, of I think what we're trying to do here, which is how can a user or a person who has no scientific background better understand studies that they read in the news or in the publications, uh, to know if the findings are solid, solid or not, especially in today's age, where you can easily see two studies that contradictory, uh, contradict each other? Coffee's good. Coffee's bad. Eggs are good. Eggs are bad. [chuckles] Um, so I thought we could run through, uh, a bunch of questions,

3. 3:19 – 6:39

   The process for a study to progress from idea to design to execution

   1. BKBob Kaplan3:19

      with the f- the first one, um, that we got here is, "What is the process for a study to go from an idea to design and execution?"

   2. PAPeter Attia3:28

      [lip smacks] Yeah. Uh, and it is a great st- this is a great question. Um, I mean, in theory, [chuckles] it should start with a hypothesis. Uh, good science is generally hypothesis driven. Um, so [clears throat] I think the cleanest way to, to think about that is to take the position that there is, um, no relationship between, um, two phenomena. So we, we would call this sort of a null hypothesis. So, um, uh, my hypothesis might be that drinking coffee makes your eyes turn darker. Um, so I would have to state that hypothesis, and then I would have to frame it in a way that says, um, my null hypothesis is that when you drink coffee, your eyes do not change in color in any way, shape, or form, and that would imply that the alternative hypothesis is that when you drink coffee, your eyes do change color. You can already see, by the way, that there's nuance to this because am I specifying what color it changes to? Does it get darker? Does it get lighter? Does it change to blue, green? Does it just get the darker shade of whatever it is? But let's put that aside for a moment and just say that you will have this null hypothesis and you will have this alternative hypothesis. And to be able to formulate that cleanly is sort of the first step here. The second thing, of course, is to conduct an experimental design. How are you going to test that hypothesis? Um, uh, as we're gonna talk about, a really, really elegant way to test this is using a randomized controlled experiment. Um, if it's possible to blind it, we'll talk about what that means. Uh, but, you know, and you'll have to decide, well, how long should we make people drink coffee? How frequently should they drink coffee? How are we going to measure eye color? You ... These are the questions that come down to experimental design. You then have to determine a very important variable, which is how many subjects will you have, and of course that will depend on a number of things, including how many arms you will have in this study. But it comes down to doing something, uh, that's called a power analysis, and this is so important that we're gonna spend some time talking about it today, although I won't talk about it right now. Uh, if this study, uh, involves human subjects or animal subjects, you will have to get something called, um, an institutional review board to approve the ethics of the study. So you'll have to get that IRB approval. Um- You'll have to determine what your primary and secondary outcomes are, get the protocol approved, uh, develop a plan for statistics, and then pre-register the study. Uh, all of these things happen before you do the study, and of course, in parallel to this, you have to have funding. So, uh, those are kind of the steps that go into doing an experimental study. And what we're gonna talk about, I think, in a minute, is that there are some studies that are not experimental, uh, where some of these steps are obviously skipped.

4. 6:39 – 19:14

   Various types of studies and how they differ

   1. BKBob Kaplan6:39

      Yeah. One of the, one of the questions we got was, what are the different types of studies out there, and what do they mean? For example, you know, an observational study versus a, a randomized controlled study. What are the different types of studies?

   2. PAPeter Attia6:51

      Yeah. So I, I mean, I, I think broadly speaking, you can break studies into three categories. Um, one would be observational studies, um, and we'll, we'll bifurcate those or trifurcate those in a minute. Uh, then you can have experimental studies, and then you can have basically summations of and/or reviews of and/or analyses of studies of any type. So, um, let's kind of start at the bottom of that pyramid. Um, and I, I think you actually have a figure that I don't like very much, but I know it's-

   3. BKBob Kaplan7:36

      I was gonna say.

   4. PAPeter Attia7:36

      Yeah.

   5. BKBob Kaplan7:36

      I thought it was one of your favorites.

   6. PAPeter Attia7:37

      Yeah, I can't stand it. I, I'll tell you what I like about the figure. I like the color schema because my boys are so obsessed with rainbows that if I show them this figure, they're going to, they're gonna be really happy. So let's, let's pull up said rainbow figure.

   7. BKBob Kaplan7:54

      Okay, got it.

   8. PAPeter Attia7:57

      Okay. So, uh, you can sort of see these, these, these buckets here, and, and again, at the, at the, at the level of kind of talking about them, I think this makes sense. What I don't agree with the pyramid for, Bob, is that it, it puts a hierarchy in place that suggests a meta-analysis is better than a randomized control trial, uh, which is not necessarily true. But let, let's just kind of go through what each of these things mean. So, so looking at the observational studies, an individual case report is, uh, you know, look, I, the, I think w- w- the se- first or second paper I ever wrote in my life when I was in medical school was an individual case report. Um, it was a patient who had come into, um, clinic. This was at, when I was at the NIH, and this was a patient with metastatic melanoma, and their calcium was sky high, dangerously high, in fact. And obviously, our first assumption was that this patient had, uh, metastatic disease to their bone and that they were lysing bone and calcium was, was, you know, leaching into their bloodstream. Turned out that wasn't the case at all. Um, it turned out they had something that had not been previously reported in patients with melanoma, which was they had developed this P-- uh, uh, parathyroid hormone-related like hormone, um, in response to their melanoma. So, um, this is a hormone that exists normally, but it doesn't exist in this format. Um, and so their cancer was causing them to have more of this hormone that was causing them to, uh, raise their calcium level. It was interesting because it had never been reported before in the literature, and so I wrote this up. This was an individual case report. Um, is there any value in that? Sure, there's some value in that. The next time a patient with melanoma shows up to clinic and their calcium is sky high and someone goes to the literature to search for it, they'll see that report and, uh, you know, it will hopefully save them time in getting to the diagnosis.

   9. BKBob Kaplan9:57

      So your mentor and friend, Steve Rosenberg, I think of him when I think of individual case reports where, I think if you listen to the podcast, he talks about this, but a lot of what motivated him early on, I think, were, were just a couple of, a couple of cases, and it, I think it gets back to that first question too about the process for a study to go to an idea to design to execution, which is to, to have a hypothesis, you need to make an observation.

   10. PAPeter Attia10:21

       Right.

   11. BKBob Kaplan10:22

       So you make an observation, you say, "Hmm, that's strange." And I think that that, that's what individual case reports can represent sometimes is, is this is a really, this is an interesting observation. It's hypothesis generating for the most part, but it really, it might kickstart a, a larger trial or it might, might kickstart a career. You never know.

   12. PAPeter Attia10:40

       Yeah, e- exactly. Um, now of course, it's not gonna be generalizable, right? It's, uh, I can't make any statement about, um, you know, the frequency of this in the, the broader subset of patients, and obviously I can't make any comment about any intervention that may or may not change the outcome of this. So that gets us to kind of our next thing, which is like a, a case series, uh, or, or set of studies. So here you're basically doing the same sort of thing, um, but, but in, in, in plural effectively. Um, so, so you, you wouldn't just look at one patient. You would say, "Well, um, you know, I've now been looking back at my clinical practice and I've had, you know, 27 patients over the last 40 years that have demonstrated this very unusual finding." And, uh, uh, another example of this going back to the Steve Rosenberg case would be, uh, one could write a paper that looks at all spontaneous regressions of cancer. Well, there's obviously been a number of them. They're infrequent. Um, in fact, they're exceedingly rare, but there are certainly enough of them that one could write, uh, a case series. So, s- so now let's consider cohort studies. So, so cohort studies are larger studies, um, and they can be retrospective or they can be prospective. So I'll give you an example of both. So a, a retrospective observational cohort study would be Like, let's go back and look at all the people who have used saunas for the last 10 years and, um, look at how they're doing today relative to people who didn't use saunas over the last 10 years. So it's retrospective. We're looking backwards. It's observational. We're not d- we're not doing anything, right? We're not telling these people to do this or telling those people to do that. [lips smack] Um, and the hope when you do this is that you're going to see some sort of pattern. Undoubtedly, you will see a pattern. Of course, the question is, will you be able to establish causality in that pattern? Um, cohort studies can just as easily, although more time consumingly, be prospective. So you could say, "I want to follow people over the next five years, 10 years, who use saunas and compare them to a similar number of people who don't." And now, in a forward-looking fashion, [clears throat] we're going to be examining the other behaviors of these people and ultimately what their outcomes are. Do they have different rates of death, heart disease, cancer, Alzheimer’s disease, other metrics of health that we might be interested in? Again, we're not intervening. There's not an experiment per se. We're just observing, but now we're doing it as mar- as we march forward through time. Okay, so this brings us to the kind of the next layer of this pyramid, which are the experimental studies. Uh, and again, we sort of divide these into randomized versus non-randomized. And of course, this idea of randomization is going to, um, be a very important one as we go through this. Uh, so a non-randomized trial, uh, sometimes gets referred to as an open-label trial where, uh, you take two groups of people and you give one of them a treatment, and you give the other one, uh, either a placebo or a different treatment, but you don't randomize them. There's a reason that they're in that group. So you might say, um, you know, "We want to study the effect of a certain antibiotic on a person that comes in the ER, and we're going to take all the people that come in who look a certain way," maybe they have a certain, a fever of a certain level or a white blood cell count of a certain level, "and we're gonna give them the antibiotic. And the people who come in, but they don't have those exact signs or symptoms, we're going to not give an antibiotic to, and we're gonna follow them." Uh, that's kind of a lame example. Um, you could do the same sort of thing with surgical interventions. We're gonna try to ask the question, is surgery better than antibiotics for appendicitis or suspected a-appendicitis? But we don't randomize the people to the choice. There's some other factor that is going to determine whether or not we do that. And as you can see, that's going to have a lot of limitations because presumably there's a reason you're making that decision, and that reason will undoubtedly introduce bias. So of course, the gold standard that we always talk about is a randomized control trial where whatever question you want to study, you study it, but you attempt to take all bias out of it by randomly assigning people into the treatment groups, the two or more treatment groups. Um, we'll talk about things like blinding later 'cause you can obviously get into more and more rigor when you do this. But before we leave the kind of experimental side, anything you want to add to that, Bob?

   13. BKBob Kaplan15:56

       Uh, I would add, so non-randomized controlled trials, maybe another example. I'm trying to think of another, uh, example, which is you have patients maybe, maybe making a decision beforehand, which we'll get into selection bias. But it, you-- people might wanna go, they might wanna go on a statin, let's say, and then other, uh, you give them a choice, and the other ones might wanna go on some other drug like ezetimibe.

   14. PAPeter Attia16:15

       Mm-hmm.

   15. BKBob Kaplan16:16

       And they're, they're, they're basically selecting themselves into two groups, but you could, you could compare those two groups and see how they do. Um, but you haven't, uh, it hasn't been randomized. There's a, there's a lot of bias that can go into that. There could be a lot of reasons why one group is selecting a, a particular treatment over the other. And so that's why I think when we get to randomized trials that it shows the power of randomization.

   16. PAPeter Attia16:39

       [lips smack] Yeah. Yeah. Exactly. Um, uh, let's-- we don't need to go back to the figure, but, uh, people might recall that at the top of that pyramid was, you know, systemic reviews and meta-analyses. Let's just talk about meta-analyses since they, um, are, are probably the most powerful. So, so this is a statistical technique where you can combine data from multiple studies, um, that are attempting to look at the same question, basically. So, um, uh, each study gets a relative weighting, um, and the weighting of a study is sort of a function of, um, its precision. It depends a little bit on sample size, uh, other, other events in the study. Larger studies which have smaller standard errors are given more weight than smaller studies with larger standard errors, for example. Um, a very common way that these-- you'll, you'll, you'll know you're looking at a, a meta-analysis. We should have had a figure for this, but, uh, I'll describe it the best I can. They usually have a figure somewhere in there that will show kind of across rows all of the studies. So let's say there's 10 studies included in the meta-analysis, and then, um, they'll have the hazard ratios for each of the studies. So they'll represent them usually as little triangles. So the, the triangle will represent the ninety-five percent confidence interval of what the hazard ratio is, which we'll, we'll talk about a hazard ratio, but it's, it's, it's a basically a marker of the risk. [lips smack] And you'll see all 10 studies, and then they'll show you the final summation of them at the bottom, which of course you wouldn't be able to deduce looking at the figure, um, but it takes into account that mathematical weighting. So on the surface, meta-analyses seem really, really great, right? Because if- One trial, one randomized trial is good, ten must be better. Um, but again, I, and I, I know I've said this before probably three or four times over the past few years on the podcast, um, but as, as James Yang, one of the, uh, smartest people I ever met when, when I was, uh, uh, both a student and fellow at NCI, uh, once said during a journal club about a meta-analysis that was being presented, he said something to the effect of, "A thousand sows' ears makes not a pearl necklace." And you know, that, that's just an eloquent way to say that garbage in, garbage out, right? So if you do a meta-analysis of a bunch of garbage studies, you get a garbage meta-analysis. So it, it can't, uh, it can't clean garbage. Um, it, it simply can aggregate it. So a meta-analysis of great randomized control trials will produce a great meta-analysis.

5. 19:14 – 28:32

   The different phases of clinical trials

   1. BKBob Kaplan19:14

      So I'm sure they, they try to control for garbage, you know, the, the researchers and the investigators. But I think to your point, with the pearl necklace, imagine if you had, say, ten, ten trials and nine of them are garbage. One of them is really good, really rigorous randomized controlled trial, and you're looking at the top of the pyramid and you're saying, "Well, meta-analysis is the best. We should be looking at this meta-analysis." Meanwhile, you've got that one randomized controlled trial that actually is worth its salt. It's, it's rigorous, et cetera, that, that w- I would say if you had the option, I think you probably would rely more on that one randomized controlled trial, which is lower on the pyramid. So I think that's probably, I think, uh, you've told me one of your hang-ups with a, with a pyramid, 'cause it's not necessarily top of the pyramid is gonna be some meta-analysis of, of randomized controlled trials.

   2. PAPeter Attia20:00

      [lip smack] That's right, yeah. A- and, and so I don't want to suggest ra- meta-analyses are not great. What I want to suggest is you can't just take a meta-analysis as gospel without actually looking at each study. You don't get a pass at examining each of the constitutive studies within a meta-analysis, is, is really the, the point I think we want to make here.

   3. BKBob Kaplan20:22

      Yeah. There's, there's one thing in here that, that isn't represented, but we had a few questions about it, I think a couple. People are asking about, you know, you know, what's the difference between a, a phase three and a phase two or a phase one clinical trial? Do you know what's going on there?

   4. PAPeter Attia20:38

      Yes. Uh, and the, these are great questions. So, so here we're talking about human clinical trials, and, and this, this phraseology is, is, is used by the FDA, um, here in the United States. And, and typically, um, the, the world does tend to follow in lockstep, but not always with, with kind of the FDA's process. So, um, [lip smack] if you go way, way, way back, uh, you, you have an interesting idea. You, you have a, a drug that you think is-- or a molecule that you think will have some, you know, benefit. L- let's think of it as a, a cancer therapeutic. Um, and you've, uh, you've done some interesting experiments in animals and, you know, you maybe started with some mice and you went up to some rats, and maybe even you've done something in primates, and now you're really sort of committed to this as, uh, y- the success of this and the safety of this in animals looks good, so it's both safe and efficacious in animals, and you now decide you want to foray into the human space. Well, the first thing you have to do is, um, file for something called an IND, an investigational new drug application. So after you do all of this preclinical work, you have to file this IND with the FDA, and that basically sets your intention of testing this as a drug in humans. And the first phase of that, which is called phase one, is geared specifically to dose escalate this drug from a very, very low level to determine what the toxicity is across a range of doses that will hopefully have efficacy. So these are typically very small studies, uh, usually less than a hundred people, and they're typically done in cohorts. So you might say, well, the first, you know, twelve people are going to be at zero point one milligrams per kilogram, and assuming we see no adverse effects there, we'll go up to point one five milligrams per kilogram for the next twelve people, and if we have no issues there, we'll escalate it to point two five, da, da, da, da, da, da. Now, you, you'll notice, Bob, I said nothing in there about does the drug work, right?

   5. BKBob Kaplan22:54

      Right.

   6. PAPeter Attia22:54

      So these are going to be patients with cancer. If this is a drug that's being, um, sought as a treatment for colon cancer, these are gonna be patients that all have colon cancer. They're often going to be patients who have metastatic colon cancer, right? So these are gonna be patients who, um, have progressed through all other standard treatments, uh, and who are basically saying, "Look, sign me up for this clinical trial. I realize that this first phase is, um, not going to be necessarily giving me a high enough dose that I could experience a benefit, um, and that you're really only looking to make sure that this drug doesn't hurt me. But nevertheless, I want to participate in this trial." So if the, if the drug gets through phase one safely, then it goes to phase two, and the goal of phase two is to continue to evaluate for safety, but also to start to look for efficacy. But this is done in an open label fashion. What that means is they're not randomizing patients to one drug versus the other typically. They can, but usually it's now we think we know one or two doses that are gonna produce efficacy. They were deemed safe in the phase one. We're now going to, um, take patients and give them this drug and look for an effect. And a lot of times, if there's no control arm in the study, you're going to compare to the natural history. So let's assume that we know that patients with metastatic colon cancer have, um, on standard of care, have, uh, you know, a, a median survival of X months. Well, we're gonna give these patients this drug and see if that extends it anymore. And of course, you could do this with a control arm, but now it adds the number, number of patients to the study. So again, typically very small studies, uh, can be, you know, in the 20, 30, 40, 50 range, maybe up to a few hundred people.

   7. BKBob Kaplan24:53

      And that one, Peter, I think is a probably a good example of if you have the non-randomization, this might be a case where, say it's an immunotherapy and, and people know about the immunotherapy, and it's been really effective for a particular, gets approved for a particular cancer, let's say. And there are a lot of people that, that know about it, and there are cancer patients that know about it, and they, they wanna get that treatment, but it's not approved. If they're talking to their doctor, they, they, or maybe they're online, they, they might enroll in one of these trials because they really wanna try the, the drug and maybe they, you know, they might believe in it more than, than some other treatment.

   8. PAPeter Attia25:24

      Yep. Yeah. Th- there are lots of things that can introduce bias to a phase two if it does not have randomization. Again, the goal would be to still randomize in phase two, um, because you really do want to tease out efficacy. [clears throat] So if a study, uh, if a, if a compound succeeds in phase two, which means it continues to show no significant adverse safety effects, which by the way, doesn't mean it doesn't have side effects, right? It, the, every treatment has side effects. It's just that it doesn't have, um, side effects that are deemed unacceptable for the risk profile of the patient, and it shows efficacy. So really, you have to have these two things. You then proceed to phase three. And here, a phase three is a really rigorous trial. This is a huge step up. It's typically a log step up in the number of patients. So you'd, you know, you're talking potentially thousands of patients here. Um, and this is absolutely a placebo-controlled trial, or not necessarily placebo, uh, but it, it can be sort of standard of care versus standard of care plus this new agent, but it is randomized. Whenever possible, it is blinded, and with drugs, that's always possible. Um, and these are typically longer studies. Uh, so, so now, because you have so much more sample size, you're going to potentially pick up side effects that weren't there in the first place. And of course, now you really have that gold standard for measuring efficacy. And it's on the basis of the phase one, phase two, and mostly phase three data that a drug will get approved or not approved for broad use, which leads to a fourth phase, which is a post-marketing study. So phase four studies take place after the drug has been approved, and they're used to basically get additional information because once a drug is approved, you now have more people taking it, um, and they may also be using this to look at other indications for the drug. So, uh, we talked about this recently, right? A phase four trial, um, with semaglutide being used to look at obesity, uh, versus its original phase three trials, which were looking at diabetes. So kind of that's a, the drug's already been approved. This study isn't being done to ask the question, sh- should semaglutide be on the market? No, it's on the market. It's basically expanding the indication for semaglutide in, in this case, so that insurance companies would actually pay for it for a new indication. But given the size and the number of these studies, you're also looking for, hey, is there another side effect here that we missed in the phase three?

   9. BKBob Kaplan28:08

      Right. And it might be the, the particular population that might have a different risk profile so that, uh, you might, you might have a different threshold.

   10. PAPeter Attia28:15

       That's right, because you're not doing this in patients with type 2 diabetes. You're doing this in patients who explicitly don't have diabetes but have obesity. Ah, you know, different patients. K- are we gonna see something different here? So yeah, so, um, anyway, that's, that's the, that's the long and short of phases one, two, three, and four.

6. 28:32 – 49:27

   Observational studies and the potential for bias

   1. BKBob Kaplan28:32

      Okay. So going back to observational studies, are there any things that you look for in particular that, that will increase or decrease your confidence in it, whether the, the, it's a pearl necklace or, or garbage?

   2. PAPeter Attia28:44

      Yeah. I mean, I, and you've already kind of touched on this, but I, I think that, um, you know, selection bias is, is, is a big one. Uh, so, so I think when, when I think about observational studies, whether they be prospective or, um, retrospective, the healthy user bias, I think, is one of the more common ones we see in the epidemiology as it pertains to health. So I wouldn't even know where to begin talking about these studies 'cause the examples are so myriad. But, you know, is bacon bad for you? Well, if you look at observational epidemiology, bacon is almost always bad for you. I don't know what the hazard ratios are, Bob, but it's, it's probably in the neighborhood of 1.3 or something like that, meaning it has about a 30% increase in the risk of basically anything you look at, right? Whether it be cancer, heart disease, death. Is that directionally right?

   3. BKBob Kaplan29:38

      I, I think that's right. I mean, there's probably more nuance 'cause they're, you know, it's, the WHO is looking at, I think they said over 700 epidemiological studies for, uh, red meat consumption, and I think they also had processed meat consumption.

   4. PAPeter Attia29:50

      Mm-hmm.

   5. BKBob Kaplan29:51

      When you look at those, you know, well, we can get into it, but how are they measuring bacon consumption? You know, they're, they're using these food frequency questionnaires, probably get into this recall bias. Um, but yeah, generally, I think, uh, I think with the WHO stuff, I think it was about 20% to 30%, um, associated increase.

   6. PAPeter Attia30:07

      Yeah. And so you look at that on the surface, of course you'd be concerned. You'd be like, "Oh my God, like, I shouldn't be eating fill in the blank. I shouldn't be drinking coffee. I shouldn't be eating bacon. I shouldn't be eating meat at all. Um, the problem with these studies is that you can't ever, no matter how much you try to statistically reconcile it, you can't strip out the fact that people make choices not in isolation. So is there any difference between a person who makes a lifelong decision to not eat meat and a person who doesn't? Of course there is, and it's going to come down to many things that go beyond their diet, including things that can't be controlled for. Now, obviously, you can control for some things. Smoking. A person who doesn't eat meat is far less likely to smoke than a person who does. A person who doesn't eat meat is probably far more likely to exercise or pay attention to their sleep habits or be more compliant with their medications or things like that. Um, again, not that people who don't eat meat don't care about those things, but people who don't eat meat... Pardon me, not the people who eat meat don't care about those things. People who don't eat meat, basically that is a proxy. That is a really good marker for someone who is very, very health conscious. So this healthy user bias permeates everywhere, and by the way, it permeates in both directions. So, um, if you look at the epidemiology that started to become very popular about 10 years ago, that was suggesting that diet soda was more fattening than soda, right? So drinking a Diet Coke is worse than drinking a Coke. Well, on the surface, that doesn't seem to make a lot of sense, right? I mean, Diet Coke has no calories in it. Coke is full of just liquid sugar. And of course, it gets you thinking, "Oh, is it the aspartame or whatever else?" Well, I mean, a far simpler explanation is look at people who are drinking diet soda versus people who are drinking soda. You could make an argument, and I, I think this is the argument, that as a person is becoming more metabolically ill and they're being informed that they really need to stop drinking soda, they're gonna be drinking diet soda. And so it's very difficult to look at just people drink this, people drink that. They're otherwise identical, and simply the only difference between them is what they drink. Um, it just, it just doesn't really hold up. So anyway, you're always gonna look for that, that sort of healthy user bias. You talked about another bias a second ago, which is information or recall bias, and I think many people are just shocked to learn how clunky and kludgy nutritional epidemiology is. I mean, it's... Like, when you think about all of the amazing technology we have in the world and y- you know, you look at... [chuckles] I mean, we, we just recently, uh, did a podcast and, you know, talking about some of the most cutting edge, uh, tools of neuroscience that allow you to examine the behavior of a single neuron using channelopsins and all these things. Like, that's at one end of science. And at the other end of science, we have this thing called a food frequency questionnaire-

   7. BKBob Kaplan33:21

      [laughs]

   8. PAPeter Attia33:22

      ... where you, you get a call from, from Billy, and he asks you, "Hey, um, do you remember how many times a week you ate oatmeal for the past year?" I mean, I, I pay quite a bit of attention to what I eat.

   9. BKBob Kaplan33:38

      [laughs]

   10. PAPeter Attia33:38

       I don't, I don't know how I'd answer that question.

   11. BKBob Kaplan33:40

       You just go to your spreadsheet of all your, your oatmeal consumption, right?

   12. PAPeter Attia33:44

       Yeah. Now, you've dug into this a bit, Bob. C- can you... I- I'm being a bit tongue-in-cheek and facetious. Can you try to make the case that, that recall bias isn't really that bad and I'm just exaggerating?

   13. BKBob Kaplan33:54

       Uh, I can't make that case. I, I guess it might depend on, you know, w- what are you recalling?

   14. PAPeter Attia34:01

       Okay. What's the best-

   15. BKBob Kaplan34:02

       So are you recalling your smoking history, let's say-

   16. PAPeter Attia34:04

       What's the best case? Yeah. What's the, what's the best case scenario?

   17. BKBob Kaplan34:05

       Right? Compared to food... Best case scenario. Well, maybe so... I would... If... I would probably get out of the food category. It might have to do with something, say, like smoking history, and you might, you might even have receipts of the last time that you, you paid for cigarettes or something like that, that if you ask people how much did you smoke in the last year, I think you can get a more accurate answer. But with the food frequency questionnaires, I think, I mean, there's so many analyses, but even just the number of foods that are out there compared to the number of foods that are, that are encapsulated in the food frequency questionnaire, vastly different. It only covers a very small portion of it. But... And it's actually the foods that, that I think the epidemiologists often look at, like the, the red meat consumption and things like that, that people will, um, vastly under... Uh, the, is that they, they will underestimate. It's when they do, like, these validity studies and actually follow them, or they do, like, a food log compared to their food frequency, compared to the food frequency questionnaires, it's... The correlation is so low that it's, it's so underestimated that you're not really getting an accurate picture. So, um, I don't know about, like, a best case scenario with food frequency questionnaires for food. I guess it would depend on... It would be on frequency. Imagine if you got a food frequency questionnaire that was, uh, maybe more technologically advanced, where you got, like, a... It's, it's an app where you literally recall at the end of each day, "What did I eat today?" Or even after your each meal. But the problem is, I think is the frequency of this, that oftentimes it's, it's, it... You're, you're doing one questionnaire for what did you eat over the course of, say, uh, one year or two years, or even they'll just do one fr- food frequency questionnaire at your, at the beginning of the study at baseline. They'll follow up with these people for, say, say, 10 years, 20 years, and the assumption is they don't change their eating habits. They never ask them again, you know, "What happened here?" But if they look at... You know, they might compare two groups. One group has higher bacon consumption than the other. The assumption is, is that they're gonna continue with those dietary habits for, you know, in perpetuity. Um, so again, that's not a best case by, uh, scenario. But the... I guess the best case scenario is that, that you could have more rigor, I think, if you did it more frequently, because obviously, I think, you know, I... If I asked you what you had for breakfast this morning, I think y- you'd probably have more confidence in the answer than what did you have for breakfast on January, uh, 3rd of, of last year.

   18. PAPeter Attia36:22

       Yeah. I mean, I, I think with nutrition, I just Because I spend so much time i- doing this type of stuff with patients, uh, it's, it's, it's metaphysically impossible and I, and I re- I mean, I really feel strongly that we should abandon food frequency questionnaires and no study should ever be published that includes them. I, I... That's just, that's gonna... I'm gonna anger a number of epidemiologists listening to this, but I, I really think we need to put a stop to that. I think where recall is reasonable is, as you said, on things that are more profound. I mean, uh, if we wanted to do a study on the, the, uh... Think of something really that you would never forget, like, oh, childbirth. Like, a- re- asking women to recall how many times have you been pregnant, how many times did you either have an abortion or miscarry, and how many times did you deliver at term? Like, something that profound, yeah, I would feel confident that if you asked a woman that question over the past 10 years of her life, you would get very accurate answers. But by the way, it still doesn't tell me that I would be able to infer causality. If I was trying to look at women who have, you know, never had a, a miscarriage versus women who have had miscarriages, just because I look back and ask them to tell me those things doesn't mean that embedded within those differences are other biological or social or economic factors. So y- you kinda get where we're going here, which is, you know, I think epidemiology has a place, but I think it's... The, the pendulum has sort of swung a little too far and it's, it's, it's, its place has been asserted as being more valuable than I think it probably is. Um, you, you kind of talked about something that's, um, I think a very important bias that exists in any study, but it... I think this is actually a big problem in, in experimental studies, in, in, in prospective studies if they're done incorrectly, which is something called performance bias. So the Hawthorne effect is, is basically an effect that says if a person is watching you, you will change your behavior. So anybody who has tried to fastidiously log what they eat every day, which I've done, um, many people have done this, there's no question you change your behavior. Just by logging what you eat, you will change what you eat. How much more will you do it when you know someone is going to look at it? Unbelievably so. In fact, you could make a case that one of the most efficacious dietary interventions known to man is having somebody watch what you eat every meal. Uh, not just every meal, every moment. Like, and whether you have somebody virtually or literally watching you at every moment eat, um, especially someone who, you know, you're not entirely comfortable with-

   19. BKBob Kaplan39:14

       [laughs]

   20. PAPeter Attia39:15

       ... that's going to have, uh, an enormous impact. I think, I think, uh, isn't this, isn't there a name for this that's like a car re- It's like, is it the Hertz effect or what?

   21. BKBob Kaplan39:24

       Yes. There, there's a... It's the Avis effect because actually I think Hertz might... In the '70s and '80s, Avis was always behind-

   22. PAPeter Attia39:31

       Oh, they were behind Hertz

   23. BKBob Kaplan39:32

       ... it was a budget.

   24. PAPeter Attia39:33

       Yeah, yeah, yeah.

   25. BKBob Kaplan39:33

       It was either Hertz or Budget, Budget Rent a Car. So though, so their slogan I thought was great, you know, kinda co- interesting. Basically, "We're number two," and then, then they would say, "We try harder," because, like-

   26. PAPeter Attia39:43

       Because we're number two

   27. BKBob Kaplan39:43

       ... we've got this inferiority complex. We're number two. And that, um, trying to think of an example 'cause that, the Hawthorne effect is, you know, it's almost like a experimenter bias that the experimenter is watching the, observing the, the people under the lamp. That was a, the, where this came from, um, and looked at work productivity with different, different lighting. Um, and they got, you know, they got the clipboard and it's could be your boss that's out there and looking and watching you. Um, and so that experimenter's having the effect. With the Avis, with the Avis effect, I think it's more, um... Trying to think of an example. So say that you are, say you Peter, competitive Peter, were enrolled in a, in a, like a cycling trial, and you realize, say, say it's open label, and you get a, you know, you get a placebo or you, you get nothing. And you know that there's another group out there that, say, I think we, we looked at this a little bit. Say it's like this lotion that you put on and supposedly it's supposed to improve your performance. You know, there could be a part of you that says, like, "I'm gonna beat those guys." You know, the, the control group. They're gonna say like, "I... You know, we're number two. Like, we're not getting the special treatment, um, so we're gonna win this thing." You know, it sounds like that wouldn't happen, but I think that people, if they enroll in a trial, sometimes there's that competitive nature and it's also to your point about, um, if you have somebody watching you, that that could also, uh, adjust, adjust your performance. And now you have this whe- whether it's not physically somebody watching you, but you know that there's a trial that's following you and that you know that they're gonna be looking, you know, a year down the road, two years down the road, or even, you know, three weeks down the road is probably more common, and they're gonna, they're gonna test you again and see how you do and see if you improve or see if you don't. Um, those things can play a role. And just to, to the, to the point of somebody that, you know, might be not pleasant that's watching you as far as, uh, food, there's a, there's a great Saturday, Saturday Night Live clip with, it's The Rock in the clip, and it's a commercial for, it's called, uh, Nicotrel, and it's a smoking cessation, uh-

   28. PAPeter Attia41:34

       [laughs]

   29. BKBob Kaplan41:35

       ... treatment. And it's actually The Rock is named Nick Cottrel, and the guy, there's a guy on a couch and he's about to smoke a cigarette and this, you know, The Rock comes out jacked, smack, you know, smacks him, smacks the cigarette out of his hand and, and it's one of the most effective smoking cessation programs I've seen.

   30. PAPeter Attia41:51

       [laughs] I, I bet. I think there's a more, um, sinister form of performance bias that, that creeps up in clinical trials, uh, especially in randomized control trials where you think at the surface, "Wow, this is a really well done study." So you'll take two groups and, uh, let's say it's a weight loss trial, and we're gonna test, um, calorie restriction versus, uh, the... Pick your diet, your, your, your favorite. The pota- the all potato diet. Um, so the-
7. 49:27 – 1:03:49

   Experimental studies: randomization, blinding, and other factors that make or break a study

   1. BKBob Kaplan49:27

      Absolutely. Um, so if we look at experiments, experiments or experimental studies compared to observational studies, are there things you look for specifically or in particular for experimental studies to increase or decrease your confidence in them?

   2. PAPeter Attia49:40

      Yeah. Well, first and foremost, randomization, right? So if an experiment isn't randomized, again, it doesn't mean that it's useless, but it just means it's gonna be a lot harder to really make sense of this. And, and randomization needs to be, um, a rigorous randomization. Uh, y- you can, you can randomize incorrectly, believe it or not. I think there's a very famous example with PREDIMED, which was a study that when it was published was kind of a remarkable finding, a very large study, something like 7,500 people randomized into three groups, 2,500 per group, uh, given basically two different dietary patterns, a Mediterranean diet in two versions and a low-fat diet. Um, this was a primary prevention study, so it was looking at people who are high risk but who haven't had heart attacks or anything yet, and it was looking at mortality. And the study was actually stopped early, again, something we're gonna talk about in a second, uh, because it had such a positive effect. So the Mediterranean diet had such a favorable, um, effect relative to the low-fat diet that people were dying at a rate far less, uh, such that it would have been unethical to continue the study for the, I think, the seven and a half years it was planned to run, and I think they stopped it in the four-year mark, um, and, and sort of declared victory. But then something happened, Bob. What, what happened?

   3. BKBob Kaplan51:01

      Actually, so they went, they went back and reanalyzed the, the, this PREDIMED group. Um, the first paper was published in New England Journal of Medicine 2013, and they had a basically a correction, but they, they almost like a brand-new article, um, addressing some issues. So they did a, they did a reanalysis that was published in 2018. I think it came from this, this fellow named John Carlisle, who had this way, this, this way of looking, and I think this was -- We have a, we've got a email on this with David Allison, who's a great statistician. He talks about this in his article too, um, as well, where he, he looked at this. But, um, this fellow named John Carlisle did this analysis where he looked at thousands of studies, and he could flag the studies and see does this, does this truly look like randomization based on some particular statistics. And the PREDIMED study was flagged looking like this doesn't look like proper randomization. There might be something going on here. And I think according to the, you know, the, the media outlets, I think I read in The New York Times, they talked to the, the lead or the senior investigator and, and he said that it turns out that, um, some of the villages or the, the clinics, I forget how many clinics in total there, there were in the study. But at 11 of the clinics, they, uh, one of the, uh, investigators were randomizing the entire clinics, um, to one group. And this is what I think is really interesting about just looking at, like if you really wanna dig into a study, sometimes you really have to get the story, which is, and, and, and why you think it-- I think oftentimes you look at randomization, oh, that's really simple. You just randomize people to different groups and, you know, blinded or unblinded. It's very hard, it's very hard to blind, uh, the fact that you, you see your, you see your neighbor get a delivery every week of, of ol- of a jug of olive oil or a, or a sack of, uh, mixed nuts, which were the two Mediterranean groups. And I think what happened was people started complaining in the, in the, in the villages. They're like, "What do I get?" And they're like, "You got your, you got your low-fat diet pamphlet, remember? Like, we give it to you every year." Um, so that happened in that study, and that, that's typically referred to as a, a-- So you can do that in a study, and that's typically referred to as a, as a cluster, a cluster, um, randomization, where you might, you might randomize one classroom to another classroom, which might be convenient, but it, it requires different statistical methods. So that was one of the things that happened there. And then they also, another way-

   4. PAPeter Attia53:24

      Let's, let's use that example-

   5. BKBob Kaplan53:25

      Yeah

   6. PAPeter Attia53:25

      ... 'cause that's actually a really good one, right? If you wanna study the effects of meditation on, you know, attention span of kids, it's very different to say we're gonna just take 100 kids and randomize 50 into one group, 50 into another, and separate them versus saying we've got a class, two classes over here, two classes over here, we're gonna split those two and two into the effect. That's a totally different, um, type of randomization. One is a true randomization, one's a cluster randomization. And while you can do the latter, it requires a different statistical adjustment. So yeah, I think, um, PREDIMED basically had to reanalyze all of their data in light of that. It, it turned out in the case of PREDIMED, the results still held, um, but it will always kind of be a cloud that hangs over it.

   7. BKBob Kaplan54:19

      Yeah. And Ioann- I think Ioannidis to, you know, just to make the, his point, when he found it, he, I think he, he was a, a huge fan of the PREDIMED study. And something that he said, which I think might be intuitive, is he said if they're, if they're randomizing entire villages, you know, to, to a group and they're not, they're not accounting for it, he thinks like, "I'm not sure that's gonna be the only problem in that study," and that every, you know, everything was uncovered. But I-

   8. PAPeter Attia54:42

      Yeah

   9. BKBob Kaplan54:42

      ... but on the flip side, it's really, really hard to do everything, everything, everything right in a study. You're going to make mistakes

   10. PAPeter Attia54:49

       Yeah. A- and now imagine randomizing a household where you say, "Okay-

   11. BKBob Kaplan54:53

       Yep

   12. PAPeter Attia54:53

       ... you know, y- Dad, you're on a Mediterranean diet for the next seven years. Mom, you're on a low-fat diet for the next seven years." I mean, it, it, it starts to get very difficult. Um, okay, so a- again, uh, that's one important thing. You also wanna make sure, is there a control group? Um, not all random assign... Not all, um, prospective trials have control groups. Sometimes it's a single group where a person serves as their own control, and there's typically a crossover. So you'll take a group, you'll randomize them into two, but there's, um, it's not that one group is getting treatment A and the other group is getting p- placebo or treatment B. It's both groups get both treatments, plus or minus a placebo, in different orders. And, uh, this is a great statistical tool, provided the treatment doesn't interfere with the, the washout or the, the, the sort of the treatment doesn't interfere with the control session. Um, the reason this is powerful is you, you need far fewer subjects when everybody gets to serve as their own control, so it greatly reduces the, you know, basically the, the cost and logistics of a study. Uh, but you run into challenges, right? So if you take 20 people are going to take this drug that is supposed to, you know, help them exercise better for eight weeks, and another group is gonna take a placebo for eight weeks and exercise, and then everybody switches, 'cause that's the right way you would do it. You had some people start first on the treatment, some people start first on the placebo. You have to figure out, is there, do you need a gap between the treatments? Because will the effects of that drug linger into the placebo period for one group, which is not what's happening to the other group, and even if it is, um, even if you're only doing it with one group, are you confounding the effect of that treatment? I hope that makes sense, Bob. I don't know if I'm making sense. I know you know what I'm saying.

   13. BKBob Kaplan56:51

       Yeah.

   14. PAPeter Attia56:51

       But is there a better way to explain that?

   15. BKBob Kaplan56:53

       Yeah. I think that makes sense. Uh, the one other point I was gonna make about that, too, are the crossover groups. I was gonna ask, ask you about that because I've seen the, the, the statistical power, I guess you would call it, of the crossover groups is you can see relatively small studies.

   16. PAPeter Attia57:06

       Yeah.

   17. BKBob Kaplan57:06

       Not a lot of people, p- pretty short, and you look at the p-values, and we'll get into that, but they're, you know, .000, you know, something. And I, I guess, uh, the, like, the assumption when I was thinking about it when you were talking about it and, you know, they serve as their own controls, it's almost as if they're treating them like if you, if you could get identical twins and randomize those, those identical twins to one group or the other, you would think that's great 'cause you're controlling for so many things, um, about, you know, the physiology or the, the, the, the genotype, e- et cetera, about those people, and it's almost like they treat these crossover groups as that you're, you're, you're, you're almost cloning these people. You're com- comparing them to themselves. But I think you... It's a, it's a good point. And, and there might be something about the order of, of the, the treatments that they receive. If they get, you know, treatment A and then treatment B, maybe, you know, one might have an effect on the other and-

   18. PAPeter Attia57:53

       The really good ones go AB and then BA. They divide them into two groups and go AB and BA.

   19. BKBob Kaplan57:57

       Right.

   20. PAPeter Attia57:58

       And, and, yeah, it, it really comes down to the fact that you can use a what's called a paired t-test. Um, it, and, and the, the st- the simplicity of the statistic of the paired t-test is, is part of its elegance here in that it, it, uh, it, it basically eliminates a lot of variance. Um, okay, so then we talked earlier about this. Blinding. What does that mean? So in an ideal world, both the subjects and the investigators should not know who is getting the treatment and who is getting the placebo. At a minimum, the subjects should not know. That would be single blinding. But again, double blinding is always preferred if possible because the, um, investigators can be biased. They can have hidden biases, um, if they know the outcome. So for example, um, if patients are being given a drug for weight loss, you could say, well, it's pretty easy to, you know, blind the patients from that. But if the investigators know that, they might behave differently towards the patients for whom they expect greater weight loss if they believe that this drug is effective. So again, very important and sometimes very challenging. You know, I think we talked about this in the podcast with Rick Doblin. One of the huge challenges of studying psychedelics is it's very difficult to blind anybody.

   21. BKBob Kaplan59:20

       Mm.

   22. PAPeter Attia59:21

       Uh, uh, most of all, the user, right? The subject. You know, when you... One group is getting psilocybin and the other group is getting, even if it's niacin, which causes some flushing, you know, it's, it's, it's not hard to know which group you're in, and that may affect the results. Um, size matters. Duration matters. Um, and, and, and basically the, uh, the generalizable, generalizability of the study. So is it in a population that replicates or looks like what I'm interested in studying, whether it's me or my patient or whomever I, I care about? And there are strengths and weaknesses to mass heterogeneity of studies. So the more heterogeneous a study in terms of its patient population, well, the more generalizable the results are, but the higher and, uh, the higher the bar for finding it. So one of the big, uh, I think this is, I think this has got a lot of attention lately, but I think for a while it was a relatively unknown kinda dirty little secret of medicine, was how many clinical trials involved men only, and how many drugs were approved for both men and women, but on the basis of only being studied in men. Uh, and the rationale for this was that it was more complicated to study women, right? So women-

   23. BKBob Kaplan1:00:38

       Mm-hmm

   24. PAPeter Attia1:00:38

       ... especially premenopausal women, because they have a menstrual cycle, that really changes things hormonally, and therefore it's more complicated to do studies and look at drug kinetics and all sorts of things in women. And so the easier way to do that was to just study it in a hetere-- in a homogeneous population of men. Well, of course, that poses an enormous problem if you're now trying to extrapolate the utility of that drug in women. So, uh, it's an extreme example, but a very important one. Um, for large studies, you, you tend to want to know, is this, is this a multi-site or a single site? Again, Predimed's a great example, right? So you had a multi-site study, and there were probably significant differences between how the sites were run. So there's an advantage to multi-sites because in theory it brings more heterogeneity. It should cancel out the effect of any one study over another, but it's harder to control, and therefore you can have e- whether it be deliberately or non-deliberately rogue studies, uh, or sites rather, introducing more, more bias. Um, I think another thing I really look at here is [clears throat] how big is the association of the effect? Um, and we'll talk about this with power, but you can have something that is statistically significant, so in that sense the study is, quote-unquote, "A success," but it's clinically irrelevant. The effect is not that big. So we, we've tested this new drug for blood pressure, and it lowers systolic blood pressure by one millimeter of mercury after a year of use. And it's like, okay, that might be statistically significant if the study was large enough. Is it clinically significant? Almost assuredly not. Um, you want to pay attention to what the adverse events were, um, both in frequency, severity, and distribution. Um, you want to pay very close attention to who funded the trial. Uh, trials don't fund themselves, and, uh, a lot of trials are funded by drug companies. Now, again, they're usually done with, um, very clear data monitoring and data analytics. And despite all of the sort of fear-mongering out there, it's not like pharma really gets to put their hand on the scale of these pharma studies. Um, but where I think things can get a little dicey is in terms of, you know, things getting buried in supplemental journals and things like that. So, so you do want to pay a bit of attention to who's, who's funding a trial. A- a- and I think e- even more important than that is kind of understanding what the conflicts of interest are of the authors. And, and nowadays those have to be declared, but there's been a huge amount of hoopla over that, and there have been some very famous examples of, um, people who are on editorial boards of journals or publishing like crazy and not declaring that, "Hey, I'm a paid consultant of these 10 pharma companies and I'm writing, uh, or doing experiments on drugs by these people," or, "I'm an editor on journals that are commenting on, on, on this." And then finally, you, you really want to understand if the study was adequately powered, and this becomes very important if the study has a null outcome. So

8. 1:03:49 – 1:16:21

   Power, p-values, and statistical significance

   1. PAPeter Attia1:03:50

      you want to just spend a minute and we'll talk about power?

   2. BKBob Kaplan1:03:53

      Yeah, I think that, that makes sense.

   3. PAPeter Attia1:03:55

      So power is, um, uh, defined as one minus beta, where beta is defined as the probability of a false negative. Um, now let's contrast that for a moment by talking about what a false positive is. A false positive is defined as alpha, and that's also known as the p-value. Okay, so, so let me kind of restate this. So, so, um, a, a p-value is basically trying to answer the question, what's the probability of rejecting the null hypothesis when it is in fact true? So if the p-value is zero, it means it's impossible, and if it's one, it means you are absolutely going to do it. So obviously we want p-values that are as small as possible. Uh, it can never be zero, but you want them to be as close to zero as possible, and basically we say 5% is our minimum threshold, uh, or really our maximum threshold. That's the ceiling that we'll put on this idea. So, um, uh, you go back to what we talked about at the outset. So the default position is, uh, that the, that the null hypothesis is correct, that there is no difference between the groups. So, um, this term statistical significance basically means that the null hypothesis is rejected if the p-value is less than that pre-stated level. Um, I don't know if I'm explaining this really, really well, Bob. Is there anything you would add to this? 'Cause I, I think this is an important idea, even though its p-values are so ubiquitous, but I, I, I think it's maybe worth spending one more minute on it before we go back to power.

   4. BKBob Kaplan1:05:40

      Yeah, I, I mean, to me it, it sounds like it makes sense. I'm trying to think of a, of somebody who might not understand it as well. But I thi- I think, I think that the, those examples that you gave are good, that, that it... And this is-- So you see on most papers, I think you'll, you'll see this p-value of 0.05, and we can get into the confidence intervals, but you'll see 95% confidence interval, um, and p-value of 0.05. And like you said, that, that's your false positive rate. So I, I guess maybe imagine... And it's an arbitrary threshold, so you could try to submit a paper. And I, I've seen sometimes it's-- I usually catch it by the confidence interval. I'll see 90% confi- confidence interval on some table and I'll-- or some figure or table, and I'll look at it and it'll, they'll, they'll use a p-value of, of less than 0.1 for... And maybe they have some justification for it or, or, or not, but it really is this arbitrary threshold. So I think like imagine if your p-value was, uh, you're gonna-- If it's less than, you know, 0.95, we're gonna, we're gonna reject the null hypothesis. So you could have a, you know, in, in theory if, if this thing, if ni- if, if, if the, you know-- It's not exact, but if it's the chance of this being a false positive is about 90% based on your, based on your analysis, um, that's still, you, you would, you would reject the null hypothesis that there's no difference between these groups, which sounds sort of insane. So it's, it's kind of this, I don't know, I think it was this guy Fisher who maybe es- established this 0.05, but this has been the threshold that they're, they're more or less they're willing to accept, at least for the purposes of, of a single trial, that the p-value of 0.05, that they're, they're willing to accept a level of, of a A false positive in their results and still, um, make that claim that they, they rejected that hypothesis

   5. PAPeter Attia1:07:26

      R-right. Because if you make the p-value so low, if you say, "No, my threshold is .0000001," well, th-then you really run the risk of discarding a lot of information that turns out to be kind of relevant. So, um, y-y-you know, it is a fine balance between those two. So, so now it's-

   6. BKBob Kaplan1:07:46

      That would be, that would be a, a false negative, right? So if it-

   7. PAPeter Attia1:07:48

      Exactly

   8. BKBob Kaplan1:07:49

      ... the lower there. Yeah. So you would-

   9. PAPeter Attia1:07:50

      Yep

   10. BKBob Kaplan1:07:50

       ... there might be an effect, but you're not gonna see it.

   11. PAPeter Attia1:07:52

       So, so this false negative rate, um, w-we typically allow to be a larger number. It's typically between 10 and 20%. And what that means is the f- the, the, the, the flip side of that is we have 80 to 90% power, because one minus that fa- accepted false negative rate is called your power. And I, you know, I, I think this is one of the most important concepts to understand in designing any sort of clinical trial, whether it's humans, animals, any sort of intervention. So, um, there's a, there's a table, uh, they're all over the place, but this is the one I've always liked. It's old. It's probably 10 years old, um, probably longer than that, actually. But it's out of, uh, an old great, uh, textbook, uh, cancer textbook on clinical trials. So, so pull up, pull up this table, Bob, and we'll kind of walk through.

   12. BKBob Kaplan1:08:45

       Okay. Got it. Power table.

   13. PAPeter Attia1:08:47

       Okay. All right. These look a little intimidating at the outset, so l-let's kind of walk through, um, how to interpret this. So th- what this table is saying is you want to presuppose you know what the difference is between the treatment groups. You, you, you have to say, "I believe that the difference between the success rate in the treatment between group A and group B is going to be X percent, and the smaller of the two is Y percent." So let's come up with a real number. So I think that we are going to look at how this drug, um, you know, impacts your rate of surviving a urinary tract infection or being, you know, cured of this infection. And I think that the placebo group is going to have a success rate of 25%, and I think that the treatment group is going to have a success of 35%. So I think there's a 10% gap, and I think the lower of those two is 25%. So you go to .25 on the horizontal axis, um, and you go to .1 over on the, the, the column, and you'll see there's two numbers there, 459 and 358. And the upper of those two is if you want 90% power, i.e. 10% false negative, and the lower of those two is for 80% power or 20% false negative rate. And those numbers basically tell you how many people you need in each of the two treatment groups if you want to be significant at a level of .05%. So what do you notice when you look at this? You notice that the bigger the gap, the bigger the effect size between the two groups, the fewer subjects you need. So if you march from right, or pardon me, from left to right in this table, holding that effect size at .25, if you say, well, the difference is 15%, it goes to you only need 216 or 165. If the difference is 30%, so one group is gonna have a 25% success rate, one group's gonna have a 55% success rate, you're down to 60 and 47. And if you go out to a 50% difference, so one group is gonna have a 25% response rate, the other group's a 75% response rate, you're now down to, to needing somewhere between 18 and 23 people per arm. And by the way, if you go down to 5%, one group responds at 25%, the other at fi- at 30%, you're at 1,700 or nearly 1,300 depending on your level of power. So I, I appreciate everybody kind of bearing with me as I went through this power table. It seems like one of the driest things in the world, but, but a- as my mentor once told me, it's the single most important table you should ever familiarize yourself with if you want to be in the business of designing clinical trials or basically any sort of experiment, because it is just so easy to get this wrong and over or underpower an experiment. So what does that mean? So to underpower experiment I think is the more common mistake here. So you simply don't have enough people in the study to, um, appreciate a difference if it is there. And so the study ends up being null. The p-value does not exceed the threshold of .05, and you say, "Look, there is no difference between treatment A and treatment B," when in reality there may well have been, but you didn't have the power to determine it, and therefore you don't actually know if you should have rejected the null hypothesis or accepted it. Does that-- I think that makes sense, right, Bob? Does that-

   14. BKBob Kaplan1:13:05

       Yeah

   15. PAPeter Attia1:13:05

       ... that-

   16. BKBob Kaplan1:13:06

       Yeah

   17. PAPeter Attia1:13:07

       ... uh, I think the other problem, equally sinister, perhaps not as common, is when a study is overpowered. And now you have more people in the study than you should have had for the effect size, and you start to find things that are statistically significant But are probably irrelevant clinically. So you-- that's when you start to pick up an effect size of 1% when you're dealing with something clinically that should never be thought of as being relevant below 10% detection threshold. So notwithstanding the fact that you also probably, you know, had more people in a study than you needed to, it could have cost more. A-a-and you typically don't see this as much with clinical trials, but you'll see this more with kind of data dump trials. Um, so, so sort of data mining studies where they're, you know, grossly overpowered. Um, okay, I kind of got way off on a tangent there. I don't know why I went down that path of power-

   18. BKBob Kaplan1:14:02

       [laughs]

   19. PAPeter Attia1:14:02

       ... but, um, I know it's important.

   20. BKBob Kaplan1:14:05

       So I think we got on the subject, uh, because we were looking at things you look for in an experimental study that increase or decrease your confidence in it, and I think it's something that's, if, if people have this list, it's, it's often left off. I think it's important.

   21. PAPeter Attia1:14:18

       Yep. Okay, good. So yeah, power, power matters. And, and when you look at a study and it's not significant, you should ask the question: Was this study powered correctly? And I mean, I can't tell just by looking. I actually have to pull out that table we just went over and, and go through the matrix and go, "Okay, well, you know-

   22. BKBob Kaplan1:14:41

       Yeah

   23. PAPeter Attia1:14:41

       ... this is how many people were in it. Therefore, at 80% power, they were detected to tell a difference between the two groups of this much with an effect size here." And, and, and then a lot of times I'll go, "Oh, wow, this actually study, this, this study wasn't powered appropriately anyway, so I, I've learned nothing new here unfortunately."

   24. BKBob Kaplan1:14:57

       Is it true that you have a laminate of this in your wallet, this power table?

   25. PAPeter Attia1:15:00

       [laughs] I don't anymore, but I used to have a laminated copy at my desk, yes. Uh, yeah, I, I ma- I made placemats out of it for the kids. They, they love it.

   26. BKBob Kaplan1:15:09

       [laughs] Very nice.

   27. PAPeter Attia1:15:10

       Yeah.

   28. BKBob Kaplan1:15:11

       Oh, hours of enjoyment.

   29. PAPeter Attia1:15:12

       [laughs]

   30. BKBob Kaplan1:15:12

       Um, so related to this, I, I think, you probably, without looking at their, you know, their, their power analysis, which often I think this is like a may- maybe a tip is, often you won't see anything in their paper, but you might see it in the, in the protocol if they include that, where they'll, they'll, they'll talk about how they powered the study, what was their justification, what was the effect size that they were looking for, and how many, how many participants did they, did they need, and, and then you can look at how many they actually got in the trial that actually completed the trial or, or enrolled in the trial. And to your, I think to your point of overpowering a study, sometimes you could, you might be able to discern it if you're looking at that example or I think you said there's a drug that, that lowers systolic blood pressure by, uh, by one millimeters of mercury, and the results are statistically significant. I think that, that, that might put up your feelers and say, you know, how many thousands of patients were, were in this study? And, and, um, I think that that, that gets to another question, which is looking at how, how these differences are actually determined, whether when you're looking at, when you're looking at the, say, the, the effect in one group versus the effect in other, in another group. So

9. 1:16:21 – 1:26:39

   Measuring effect size: relative risk vs. absolute risk, hazard ratios, and “number needed to treat”

   1. BKBob Kaplan1:16:21

      what are some of the ways in which researchers measure the association or the ef- the quote unquote effect size in these studies?

   2. PAPeter Attia1:16:27

      Well, a lot of times it's only reported as a relative risk, but I, I think you and I have harped on this in the past, which is y- you can't really talk about relative risk without knowing absolute risk, and sometimes they don't give you enough data in the paper to do that, and it's infuriating actually.

   3. BKBob Kaplan1:16:40

      Mm-hmm.

   4. PAPeter Attia1:16:40

      But absolute risk is, um, let's use the example, right? It's sort of like, you know, group one had, at the end of this study, a, a, a 5% risk of, uh, dying, and the other group had a 3% risk of dying. So what's the absolute risk? It's 5% in one group, 3% in another. So therefore, we have what's called the ARR, or the absolute risk reduction, is the delta between those two. So the ARR is 5% minus 3% is 2%. There was a 2% absolute risk reduction. Um, and that's important to know because often what's only reported is the relative risk reduction, which is the absolute risk reduction over the non-exposure absolute risk. So in this case, the relative risk reduction would be whatever the absolute one was, I think 2% divided by the non-exposure risk, which is 5%, so that's 40%. So they had a 40% relative risk reduction going from 5% to 3%. So both of those things are important, but again, it's really critical that you know both. One of my favorite examples of this, of course, is the famous Women's Health Initiative, which was looking at the increase in the risk of breast cancer for the women who were receiving the, um, estrogen and, uh, synthetic progesterone treatment. Now notwithstanding the fact that I, and we've talked about this 100 times, why I don't think that that study was a good study in any way, shape, or form, and I don't think that that study demonstrated there was any difference in risk, statistically, here's what got reported. It got reported that the women receiving the hormone replacement therapy had a 25% increase in breast cancer, and that was true at a relative risk level. But the absolute risk difference was a difference of five women per thousand to four women per thousand. So as you went from four cases of breast cancer per thousand women to five cases of breast cancer per thousand women, that is indeed an increase of 25%. Five minus four is one, divided by four is 0.25. But what's the absolute risk reduction, or in this case, risk increase? It's one over 1,000, or 0.1%. So, you know, what I usually say to women when we're talking about hormone replacement therapy is you can kind of use that as your ceiling for the true risk increase of this therapy, even if you discount the 12 mistakes in that study that make it hard to believe that, that that effect size would hold. So another way that we tend to measure effect size or association is using something called a hazard ratio. Um, a hazard ratio actually involves some really complicated math that we're not gonna get into, uh, something called a Cox proportional hazard, which I'm embarrassed to say I don't actually know the math anymore. There was a day when I did, and I remember it was not easy for me to learn. Um, I had to go out and buy a bunch of, um, books on statistics because even though my background's in math, I did not have a huge background in stats. And, um, it wasn't like rocket science, but I remember really having [chuckles] to, to sort of understand the mathematics behind the Cox proportional hazard. The magic of the hazard ratio is it is temporal, so it captures the risk of something, i.e., the hazard over time, and that differentiates it from something called an odds ratio, which can't do that, which only can measure kind of over the entire period of time what is the risk. So, um, uh, at, at the risk of oversimplifying this a little bit, let's talk about the hazard ratio over a given period of time, but acknowledging that it's, it's real magic is its ability to tell you what's happening at any point in time. So, um, let's just pretend we're talking about a, a cancer drug trial and, um, the hazard rates, i.e., the rates of disease progression, were, uh, 20% in one group and 30% in another group. Um, so the people getting the drug progressed 20% of the time. The people not getting the drug or getting the placebo progressed 30% of the time. So the hazard ratio is the ratio of 0.2 to 0.3, which is 0.667. So in other words, their, um, the, the treatment group was 67% as likely to experience disease progression as the control group. Um, so now again, you could flip the math and say, well, what if you saw the exact same rates but in something that was desirable? So then it would be the 0.3 over the 0.2 would be 1.5, so your hazard ratio would be 0.15, which means there's a 50% increase in the benefit or the harm if it's something that's harmful. So you can look at the single number of a hazard ratio and you can determine what the increase or decrease in risk was. So, so Bob, I'll, I'll quiz you and you, you tell, tell me and the, and the listener how you're figuring this out. [clears throat] The hazard ratio is 0.82.

   5. BKBob Kaplan1:22:18

      0.82. So let's say that the... it's point you're comparing the experiment to the control or the experimental group to the control. The experimental group is, uh, has a... I would probably flip it. I would say they have a 18% reduced risk of whatever the event is you're talking about-

   6. PAPeter Attia1:22:38

      Very good

   7. BKBob Kaplan1:22:38

      ... of progression.

   8. PAPeter Attia1:22:39

      All right.

   9. BKBob Kaplan1:22:40

      Yeah.

   10. PAPeter Attia1:22:40

       All right.

   11. BKBob Kaplan1:22:40

       Thank you.

   12. PAPeter Attia1:22:40

       So gi-give me one, Bob.

   13. BKBob Kaplan1:22:42

       Nice. Uh, how about we'll go the other way, other side of the ledger, the other side of one, a hazard ratio of 2.2.

   14. PAPeter Attia1:22:50

       Okay, well, that's a, that's a tricky one. Let's -- if we said-

   15. BKBob Kaplan1:22:53

       Yeah

   16. PAPeter Attia1:22:53

       ... 1.8, it would be an 80% increase. 2.2 would be 120% increase. How are you doing that? You're taking 2.2, you're subtracting one, and you get 1.2, and you multiply by 100%. And what you did earlier was when I gave you 0.82, you took one minus 0.82, and you got negative 0.18, which is a reduction of 18%. So again, y-you can just play with these for like five minutes. It's, it's, it's actually not that complicated, um, but you just have to do a bunch of them and become familiar with what those numbers mean. Um, now let's bring it back to the, um, ARR thing. There's another common theme you'll hear about in trials called the number needed to treat or the NNT analysis, and this gets back to the importance of absolute risk reduction. So remember the, uh... L-Let's say there's an example of, um... Well, let's use the same numbers we used earlier just 'cause I, they're familiar to me. But you've got, um, a, a drug that, uh, the people who take it have four heart attacks per thousand people over a five-year period, and then there's another -- and then the placebo, they have five events over that same period of time per thousand people, right? So the, the drug reduces the events from five out of a thousand to four out of a thousand. So what's the relative risk reduction there? The relative risk reduction is 20%, right? That's the f-four minus five divided by five in this case is a 20% relative risk reduction. So you might say, "This is something we should be putting in the drinking water. This is-

   17. BKBob Kaplan1:24:50

       [chuckles]

   18. PAPeter Attia1:24:51

       ... such an important thing," right? But you want to calculate how many people do you need to treat to prevent the event, and to do that, you have to take one and divide it by the absolute risk reduction, not the relative risk reduction. And the absolute risk reduction here is 0.01%, and one divided by 0.01% is 1,000. So now you have to treat 1,000 people to achieve the effect, which means you better figure out what the side effects are of that thing, what the cost of that thing is, what the complexity of it to justify it. There may be certain things for which a NNT of 1,000 is valuable. But you wouldn't say that across the board. So, um, you know, conversely, if you have a drug that reduces the risk of death from 4% to 2%, or say 4% to 3%, then you would say four minus three is 1%. One divided by 1% is 100. If it took it from 4 to 2%, it would be one divided by 2% is 50. If it went from 4% to 1%, a reduction of death from 4% to 1%, your difference is 3%. You're now talking about an NNT of 33. As a general rule, we love to see, you know, drugs in that sub 100 range of NNT. Um, we tend to not get that impressed when the NNT of something is like 1,000. So again, it's another way to think about the effect size.

   19. BKBob Kaplan1:26:26

       Okay. That, and that makes... Uh, the, I like the number needed to treat because it really, from a clinician's perspective or from a practical perspective, it's really telling. It, embedded in there obviously is, is the absolute risk and not just relative risk.

10. 1:26:39 – 1:34:15

    How to interpret confidence intervals

    1. BKBob Kaplan1:26:39

       Um, so we went over p-values and confidence intervals a little. I don't think we went over confidence intervals as much as p-values. Do you wanna stop and talk about confidence intervals?

    2. PAPeter Attia1:26:49

       Sure. Why don't you take this one? I need a drink.

    3. BKBob Kaplan1:26:51

       [laughs] Okay.

    4. PAPeter Attia1:26:53

       By the way, non-alcoholic just for those listening. Drinking my-

    5. BKBob Kaplan1:26:57

       What are you drinking again?

    6. PAPeter Attia1:26:59

       Drinking my Ghia with Topo Chico.

    7. BKBob Kaplan1:27:02

       So confidence intervals are, you know, technically intervals in which the population statistic could lie. So you've got, typically I think what you see on a paper is this 95% CI, it's usually abbreviated, but it's a 95% confidence interval, and it's usually reported next to the, that, that hazard ratio that we just, we just talked about. So the ha- whether say the hazard ratio is, uh, zero, the hazard ratio is 0.5, or yeah, 0.5, which means basically a, a hav- a halving of the risk, say in the experimental group versus the control group. And then you'll see this 95% confidence interval, and it might say, um, it might give you this, these two, they'll give you these two numbers. Um, and for example, let's just say it's 0.2 to 1.2 is your confidence interval. And the confident that... What, what that is, is that's the flip side of the significance level, which is one minus alpha. So it's one minus, so we talked about alpha being the p-value, but also being the, the false positive rate. So it's the flip side. So when you see .05 for your p-value, that's a tip-off that your confidence interval is 95%. Um, and I think a lot of people think about the word confidence, um, in this definition, and they, they, they take it to mean the probability that a specific confidence interval, so my example that goes 0.4 to say 1.2, that that interval between those two numbers or between those two ratios, um, contains the population parameter. Um, so they think, they think, "Okay, we could be 95% confident that the true, the true effect, um, of say meat, meat consumption and, and cancer, uh, is between, between these, these two numbers." Um, but that's not really what the confidence interval suggests. It's really, it's, it's more of a suggestion. I don't think this often happens in practice, but if you were to take 100 different samples and compute this confidence interval, um, then approximately 95 out of those 100, uh, will con- will contain the true mean value.

    8. PAPeter Attia1:29:08

       Y- yeah, I mean, uh, uh, it's been described by some as an uncertainty interval rather than a confidence interval. There's another way to do this, which is just kind of the quick and dirty way to do this, is just to look at the confidence interval and ask if the interval contains one or not. So, um, y- you, you gave an example a second ago, Bob. You said your hazard ratio was what?

    9. BKBob Kaplan1:29:29

       Uh, hazard ratio was 0.5-

    10. PAPeter Attia1:29:30

        0.5

    11. BKBob Kaplan1:29:30

        ... with the confidence-

    12. PAPeter Attia1:29:32

        Okay.

    13. BKBob Kaplan1:29:32

        Yep, with a confidence interval of 0.4 to 1.2.

    14. PAPeter Attia1:29:37

        Ah, okay. So that would not be significant. So even though your hazard ratio, you might look at that and say, "Oh look, that's a big reduction." 0.5 hazard ratio means a 50% reduction. But your confidence interval was very wide, right? It was all the way from .4 up to one-point-something, so it crosses over unity. Um, conversely, if you had a hazard ratio of .5 but your confidence interval was .4 to .6 or .7 or even up to .9, you would say indeed that is n- at the level of 95%, that is confident. So the other thing you'll notice, by the way, is the closer one edge of the confidence interval comes to one, the closer the p-value is to .05. So when you have like a confidence interval that runs from 1.01 up to two, your p-value is probably about .049 or something like that.

    15. BKBob Kaplan1:30:43

        Mm-hmm.

    16. PAPeter Attia1:30:43

        Whereas when you have confidence intervals that are miles away from one, the p-values tend to be very small.

    17. BKBob Kaplan1:30:50

        Yeah. And I think, so that, that statisti- statistician, Andrew Gelman, he talked about uncertainty in intervals, and the reason why he says that is imagine you've got a, a huge confidence interval is what we call it, so a big confidence interval, meaning instead of 0.4 to 1.2, it was like a 40% reduction at 0.4, or it went, it went all the way out to like, say, 1,000. He would say like that's a huge uncertainty interval, but the way that we think about confidence is, oh, that's a huge confidence interval, and it's, it's maybe intuitively backwards for some people to think about it that way.

    18. PAPeter Attia1:31:23

        Yep. Absolutely agree. Um, the, the tighter the interval, the more confidence you actually have in it. As-

    19. BKBob Kaplan1:31:30

        Yep

    20. PAPeter Attia1:31:30

        ... and obviously it can't cross one.

    21. BKBob Kaplan1:31:31

        The less uncertainty.

    22. PAPeter Attia1:31:32

        That's right.

    23. BKBob Kaplan1:31:32

        Yeah.

    24. PAPeter Attia1:31:32

        The less uncertainty there is. And when you get these monster ones, and this is why I like those sort of tornado graphs that you see in meta-analyses, where you visually get to see how much uncertainty existed in a given study. You know, the confidence interval went... So the hazard ratio was Here's a great example. Hazard ratio was 1.4. Oh, wow, 40% increase. The 95% confidence interval went from 1.1 to 17. Well, I mean, do I really have a lot of confidence in that? No. That's an enormous uncertainty interval.

    25. BKBob Kaplan1:32:11

        Yeah. And of course, you would want to know, okay, what, what's the, what are we talking about here? Absolutely, and not just relatively.

    26. PAPeter Attia1:32:16

        Yeah. Yep.

    27. BKBob Kaplan1:32:17

        Yeah. Yep.

    28. PAPeter Attia1:32:18

        I, I mean, look, I think the takeaway of this entire section is if you make the decision that you want to pay attention to science, you, you, you just have to roll up your sleeves and accept the fact that you're, you're not going to be able to read these things in the bathtub, uh, you know, on a lazy Sunday morning. Like, you, you kind of have to roll up your sleeves and pay attention to all of this little stuff. Now, it gets easier the more you do it. It, it, it, it, you know ... I, I think when I read a paper today, it's so much easier than it was 25 years ago. Um, but you still have to kind of have your, your guard up for all of these things.

    29. BKBob Kaplan1:32:57

        Yeah, I think you learn something, you might learn something new from virtually every paper you that r- you read. So I'm sure you haven't ... Well, I, you didn't read a paper 25 years ago and then you read your second paper today. You've read a multitude of papers, and in each one, there's probably some- something educational in there that you, you might pick up, and that's, that probably goes to, I think, the beginning of the, beginning of the episode. You were talking about, I think, Tim asking, you know, how do I get better at this? And it's probably like, it's like be consistent, repetition.

    30. PAPeter Attia1:33:22

        Yeah, you-
11. 1:34:15 – 1:43:01

    Why a study might be stopped before its completion

    1. BKBob Kaplan1:34:15

       So another question we got is, do studies ever stop midway through? If so, what are the reasons?

    2. PAPeter Attia1:34:23

       Yes, they, they do. Uh, and there, there are generally kind of three reasons that studies are, are stopped. And again, we're really talking about, you know, kind of prospective clinical trials here. So, um, the first and most important of these is safety. So remember, we talked about phase one, phase two, phase three. Well, phase one is all about safety. Phase two is about efficacy and safety. Phase three is really about effectiveness and safety. Um, but notice safety is in all of those. So absolutely anytime there's a safety breach, which means there is a statistically significant difference between an important safety metric in the, between the groups, that'll just stop the study. Um, the second thing that, uh, will stop a study is, is benefit. So, uh, again, the PREDIMED example of when it was first done is it stopped two-thirds of the way through because it was deemed that there was such a benefit to the group on the Mediterranean diet relative to the low-fat diet that it would've been unethical to let those people in the low-fat diet continue for another two and a half years, uh, on a diet that was, uh, so clearly increasing their risk of mortality. And then the final thing that will stop a study prematurely is futility. This is a little bit harder to understand, but it actually comes down to that hazard ratio concept, which is able to measure risk temporally and, and sort of in an aggregate fashion. So if two-thirds of the way through a study there's no benefit and statistically you know that nothing that's going to happen in the remainder of the study is going to change that, you stop the study. It's futile to continue the study. So, so, so those are basically your big three reasons why a study is going to be stopped.

    3. BKBob Kaplan1:36:12

       I think a good example of stopping a trial for safety was, uh, actually there's been several, but one of the CTP inhibitors, torcetrapib. I don't know if I pronounced that correctly.

    4. PAPeter Attia1:36:22

       Y- yes, I remem- this, I, I remember this really well. I, this is one of the few moments in science where I remember where I was standing when the result was announced. It was, uh, it was Q4 of 2006. I was at McKinsey at the time, and I was walking up Kearny towards California Street, and, um, I heard the news of this, and I couldn't believe. I, I was so sure this was going to be a home run study.

    5. BKBob Kaplan1:36:50

       Yeah, so in this case they, the, the, the trial was set up with 7,500 patients about in each group. So they had a, they were on a CTP inhibitor and they're all on statins, so they're on Lipitor in particular. And so they compared the CTP inhibitor to just Lipitor alone, which is, serves as a control group.

    6. PAPeter Attia1:37:08

       Well, it, it was, it was actually the CTP inhibitor plus Lipitor versus Lipitor alone, and this was a Pfizer-

    7. BKBob Kaplan1:37:13

       Yes

    8. PAPeter Attia1:37:13

       ... study, which everybody thought was very cheeky of Pfizer because Lipitor was about to come off patent, and their way of sort of extending the life of it was saying, "Hey, when you pair the CTP inhibitor with Lipitor, it's gonna have a benefit," 'cause the, the background on this is that CTP inhibitors raised HDL cholesterol. So it's like we're gonna take a drug, Lipitor-

    9. BKBob Kaplan1:37:34

       Mm

    10. PAPeter Attia1:37:34

        ... that lowers LDL cholesterol, we're gonna pair it with a drug that raises HDL cholesterol. How could this possibly go wrong?

    11. BKBob Kaplan1:37:41

        Famous last words. [laughs]

    12. PAPeter Attia1:37:43

        Yeah.

    13. BKBob Kaplan1:37:45

        Yeah, so they, they intended to follow, follow up patients. It's gonna be about a, almost a five-year trial, four and a half years, and along the way they'll have a review board that's looking at the results of the study. And in this case, they had a, a monitoring board that- Was looking at, in this case, they're looking at death, all-cause mortality, and they f- they found that 82 p- [clears throat] 82 patients receiving the drug combination had died compared with only 51 on Lipitor alone. And so they advised Pfizer to halt the trial at that point, uh, which it did immediately, which it was just a little over a year into the trial, um, when they did it. And it, in a way, it gets back to when you're talking about power analysis. The way that they do this is they, they have pre-specified p-values where they're, they're, they're kind of sneaking looks at the data. Like, we talked about multiple hypothesis testing, that they're actually taking a few shots on goal in a way, because after 12 months, they're gonna actually compare the, compare the two groups, see if this thing is a p-value of less than .01, depending on what they're looking at. Um, and in this case, they had this pre-specified, uh, p-value of less than .01, um, based on a test for death from any cause, and they, they found that. And actually, the paper, um, there was still a published paper even though the trial only went for 12 months in the New England Journal of Medicine, and they, they report those, those endpoints where the study was stopped.

    14. PAPeter Attia1:39:09

        Yeah. Yeah, it's, it's, um, another discussion. In fact, I, I feel like, I feel like Tom Dayspring and I or Ron Krauss and I talked about this at length on, on one of our podcasts. I think it was, I think it was Tom and I-

    15. BKBob Kaplan1:39:18

        I think you did

    16. PAPeter Attia1:39:18

        ... yeah, that talked about this.

    17. BKBob Kaplan1:39:19

        Yeah, I think it was Tom.

    18. PAPeter Attia1:39:20

        Super, super interesting.

    19. BKBob Kaplan1:39:21

        I remember you.

    20. PAPeter Attia1:39:21

        Yeah.

    21. BKBob Kaplan1:39:21

        So I remember you telling that story. I had just a quick follow-up question. I guess, A, was this the first CTEPH inhibitor that-

    22. PAPeter Attia1:39:28

        Yes

    23. BKBob Kaplan1:39:28

        ... was, was tested? Okay, and then you had the follow-up ones, and this, this, I don't think we've really addressed this, but, you know, why would you do a, an observational study over a randomized controlled trial? In some cases, like, you know, a good example is try doing a random-- try getting it past the IRB, uh, a randomized controlled trial, and you're gonna see, you know, give people a carton of Marlboros, they're gonna smoke that over, you know, each day compared to a placebo group. It's, it's unethical. And so with this, it's, I don't know how many times they saw these adverse events with each CTEPH inhibitor, and it's, I guess, the same drug class that might have different mechanisms, but does there, does there become a point where, where it's, I guess maybe that's why they have the phase one, phase two, phase three, and they get past those, those barriers. But in a way, is it almost, do you almost assume it might be unethical to run another CTEPH inhibitor trial if you're seeing, you know, differences in death the last however many times?

    24. PAPeter Attia1:40:17

        Well, I mean, I don't think it's unethical because I think they are, they are basically saying, "Look, it's a different drug," and it's, you know, you can change one molecule, uh, on a drug, and it completely changes the way it works. Uh, you know, you look at the COX-2 inhibitors, you look at Celebrex versus Vioxx. I mean, notwithstanding my views on that-

    25. BKBob Kaplan1:40:35

        Yeah

    26. PAPeter Attia1:40:35

        ... which I talk about with Eric Topol on our podcast. But, um, you know, basically two drugs v- nearly identical, and one was far more efficacious, but also had, uh, side effects in a subset of people with hypertension. So, um, I, I think the real question is, at what point do pharma companies say enough is enough? And I lost track. I f- I feel like there were three CTEPH inhibitors that were brought to phase three. Um, and ultimately, there was a Mendelian randomization that looked at CTEPH, uh, mutations and, and really found that this was not going to be a good strategy, so.

    27. BKBob Kaplan1:41:10

        Mm.

    28. PAPeter Attia1:41:11

        Um, okay, so that's a good example that you brought up with respect to, um, safety, and then we talked about, um, benefit, which was Predimed. And then, uh, th- I, I think w- what happened in the Look AHEAD Trial? 'Cause I think Look AHEAD was one that got stopped for futility, right?

    29. BKBob Kaplan1:41:29

        Yes. Yep. That was one where they, they randomly assigned about 5,000 overweight or obese patients with Type 2 diabetes, and it was an intensive lifestyle intervention. Um, and they were looking at, um, what was the, the other-- That was the intervention group, and then you had diabetes support and education in the control group, and their, their, their primary outcome, what they were looking at is, uh, basically, uh, what's called MACE. So death from cardiovascular causes major adverse cardiovascular events. Um, and I think the, they were gonna look for 13 and a half, long trial, almost 14 years, and in this case, the trial was stopped just under 10 years, and it was based off of what, what's called a futility analysis, um, that you explained.

    30. PAPeter Attia1:42:16

        Yeah, which basically means no matter what happens from this point on, this study will not be significant. So at the time that it was stopped, the hazard ratio was .95, so there was a suggestion of a 5% reduction in, in, in, in the risk of death from cardiovascular events, so in the right direction. But the 95% confidence interval, uh, or uncertainty interval, if we're gonna adopt that terminology, was .83 to 1.09, so it crossed one. And so you know the p-value is going to be, uh, greater than .05. In fact, the p-value was .5, .51 or something like that. I mean, it was basically complete chance. There was absolutely no effect, and again-
12. 1:43:01 – 1:53:45

    Why only a fraction of studies are ever published and how to combat publication bias

    1. BKBob Kaplan1:43:02

       Okay, so moving on to review process. What is the review process once a study is done to get a paper published in a journal?

    2. PAPeter Attia1:43:12

       S- s- so, you know, once the, once the study is done and they've done their analysis and they write up a manuscript, they'll, they'll submit it to a journal for publication, and, and that, that journal will have an editor, uh, who will, you know, look to see if the, you know, paper meets their criteria, um, and if they think it's, you know, original and interesting, right? It, uh, is this, is this paper adding something to the body of knowledge? Um, at that point, the editor might just say, you know, "Hey, this is not really a good fit for our journal," or for whatever reason, "This is, you know, something we're not interested in any further. You, you're free to go and submit this elsewhere." Um, but otherwise, the editor is going to invite individuals that are typically part of a, an editorial board to, to peer review the manuscript. So you, you hear this term all the time, right? Which is, is this a peer-reviewed publication? And that's important because not all things that get published have been peer reviewed, and that's obviously the highest standard. So the, um, reviewers are basically invited not randomly, but because they have some expertise in this area. Uh, but other things are important, right? You know, you have to consider the conflicts of interest, um, and that they might have to decline if they're conflicted. Uh, and that's, that's kind of a sticky topic because there are some really obvious conflicts like, um, you know, financial conflicts of interest. But I think, I think there's a whole deeper discussion about when you have sort of philosophical conflicts of interest with the person, and that gets into another area which is peer review can be, um, blinded or, or not blinded, right? It can be single blinded where the reviewer knows who the author is, but the author doesn't know who the feedback is from. That tends to be very common. I think that's probably the most common one I've seen. It can be double blinded where the reviewer doesn't know who it's being written by and vice versa, and they can be completely open. Um, but again, the, the most common one that I've seen is, is single blinded. Um, and, and so you'll typically have three reviewers review something and they can either accept it outright, reject it outright, or make recommendations for revisions. And a- again, I, I, I think you'll see that as probably the most common thing where they say, you know, uh, "We're, we're still interested in this paper, but, you know, did you, did you actually consider this hypothesis?" So sometimes the revisions are just repeat your analysis. Sometimes it's do another experiment. Uh, that won't be the case in a clinical trial, but if you're... You know, I, I've had papers where that happened where I've done a series of experiments and I'd written it all up and I'd submitted and the reviewer came back and said, "Well, you really should have done this experiment as well, um, 'cause this would've served as another control." So you go and repeat that experiment.

    3. BKBob Kaplan1:45:53

       Mm.

    4. PAPeter Attia1:45:53

       Of course, when you're working in cell culture or something like that, it's not, it's not that onerous. Um, but, and this process can go on several times. Uh, but ultimately, you know, the editor makes a decision to, um, accept that paper and publish it or reject it again, and that, and that's basically the process. And, and again, y- you, you're typically gonna start at the, at the top of the food chain, so you're typically as a, as a, as an author you're going to try to get your paper published in the most prestigious journal. Um, I guess that's something we can talk about what determines what, what determines the prestige of a journal. Um, but you'll, you'll, you'll sort of keep going down the pecking order until you can get it into the right journal and, and sometimes right out of the gate you just sort of know like, "Hey, this is, this is a, this is a publication that is really mechanistic and it's really gonna be geared towards, you know, proceedings of the National Academy of Science versus something that has really got enormous clinical, um, implications and should go to JAMA or The New England Journal of Medicine." So, so there's a, there's a little bit of that that's going on as well.

    5. BKBob Kaplan1:46:55

       Hmm. So, so to that end, do all, every study that's, that's out there, do they end up getting published?

    6. PAPeter Attia1:47:01

       No. Uh, in fact many don't, and this is, I think this is a really big problem which is, um, you have this thing called publication bias. So there's a very, very famous example of this that you and I have spoken about which is the Minnesota Heart Study, right? And this is an example where a study was done, um, God it was, it ran from what? 1967 to 1973 if my memory serves me correctly.

    7. BKBob Kaplan1:47:30

       I think so.

    8. PAPeter Attia1:47:31

       And it was looking at people who were in a residential care facility, so they had complete control over what these patients ate and they were randomized to a diet of either normal saturated fat consumption or very low saturated fat consumption where the saturated fat was substituted with polyunsaturated fats. And at the end of this seven-year study, of course the hypothesis being the group that was substituted saturated fat for polyunsaturated fat would have lower cholesterol levels and lower, uh, cardiovascular death rate. And at the end of, in 1973 when the study concluded, they found that indeed the subjects who were given high amounts of polyunsaturated fats and low saturated fats did in fact have lower cholesterol levels, but their rates of cardiovascular deaths were, were significantly greater. Um, and they didn't publish the study. Uh, that study would remain unpublished until 1989, some 16 years later. Um, when asked why a 16-year delay in publishing that study, the lead author who's, I don't even remember who it was. Fr- uh, begins with an F.

    9. BKBob Kaplan1:48:36

       I think it's Ivan, Ivan Frantz.

    10. PAPeter Attia1:48:38

        Yeah. Uh, Frantz. That's right.

    11. BKBob Kaplan1:48:39

        Ivan Frantz.

    12. PAPeter Attia1:48:39

        Yep.

    13. BKBob Kaplan1:48:40

        There's a, there's a senior and a junior. Yeah.

    14. PAPeter Attia1:48:41

        Yeah, yeah. Uh, he said, "The study didn't turn out the way we wanted it to." So that's, that's kind of an egregious example of publication bias. Um, in this case a negative study, but y- you know, I think there are a lot of studies that don't get published even if they're negative, and that's a shame because when something doesn't work it is just as important as when it does work. So it is unfortunate that not all studies get, get published, um, because again, just think about it this way. If you wanna go out and do an experiment and 10 people have done that experiment before you and it's always failed, wouldn't it be great to know that? Would that impact your decision on whether or not you wanna do the experiment a certain way or would you wanna try something a little bit different? So you can see very quickly this becomes problematic when, when papers don't get published.

    15. BKBob Kaplan1:49:35

        Okay. Are there-- So you've got this massive problem, publication bias. Do you know of any ways that, that can combat this?

    16. PAPeter Attia1:49:43

        I think there are a lot of people working on this problem. Um, and, and I think, um, o- one of the important steps is pre-registration, which we talked about at the outset, right? Which is you force investigators to pre-register their experiments on clinicaltrials.gov so that... And, and that just, that's not just, "Here's my experiment," it's, "Here are my statistical methods. Here is my, uh, number of subjects. Here's my primary outcome. Here are my secondary outcomes," et cetera. And that basically makes it a lot harder to say, "I'm not going to publish this when it comes out if it doesn't turn out the way I wanted it to."

    17. BKBob Kaplan1:50:22

        Hmm. And I think I, and I don't know if there are particular journals that participate in this, but I, I imagine that they could. They could make it a, a prerequisite that your, your trial must be preregistered in order to be published in our journal. And if it's a journal, you know, worth publishing in, it's probably not a bad idea.

    18. PAPeter Attia1:50:36

        Correct. Um, so there's, there's both requirements of journals and there's also requirements of funding entities which say-

    19. BKBob Kaplan1:50:41

        Hmm

    20. PAPeter Attia1:50:41

        ... you're, you can't-- y- we won't fund you unless the study is preregistered. Um, so, uh, y- y- you know, I, I think that, uh, you know, registered reports is a, is a publishing format that an organization called the Center for Open Science, and it, I think that's the one founded by, um, [lip smack] Brian Nosek. Is that his name?

    21. BKBob Kaplan1:51:01

        I think that's right, yeah.

    22. PAPeter Attia1:51:03

        Yeah, Brian would be a great guy to have on the, uh, on the podcast actually at some point.

    23. BKBob Kaplan1:51:06

        Mm-hmm. So with registered reports, um, it, it, it pulls back, basically you submit your protocol almost like the preregistration. You submit that and at that point, instead of after all the data is collected, it's peer-reviewed. And if it's peer-reviewed and accepted, um, based on you've got a high qual- high-quality protocol, everything looks good, um, then it's provisionally accepted for publication. And so, like you said, like, if it's a negative result and, and maybe that doesn't seem like it's worth publishing or, uh, the journal's not gonna publish it, they're basically making the decision if this is a positive or a negative trial, whatever it, however it turns out, your, your protocol, your plan looks really good, and so we're going to accept it. We're gonna basically accept it provisionally, provided that you don't, you don't start cutting corners and go away from this plan that we accepted. So you follow your plan and it's like however the cards fall, it's already been accepted for publication.

    24. PAPeter Attia1:52:02

        Yeah. That, that's a, that's a pretty novel concept actually. Um, but again, I think it's all in the spirit of how do we make sure that we get rid of publication bias, positive result bias. Again, going back to what we said a second ago, you're far more likely to see something get published if it is a positive finding than if it's a negative finding, although negative findings are, are just as important for the establishment of knowledge, right? Let's use the CTEPH inhibitor.

    25. BKBob Kaplan1:52:29

        Yeah.

    26. PAPeter Attia1:52:29

        Imagine no one had ever published the studies demonstrating that CTEPH inhibitors were at best neutral, at worst harmful. Um, again, studies of that magnitude can't escape publication, but think of all the bench research that can be going on, or the small early phase one trials that can be going on, or the preclinical stuff that's going on. Uh, it's very easy to kind of, um, underreport things that are, that are negative.

    27. BKBob Kaplan1:52:56

        Yeah. And one, one other thing about the registers-- registered reports I was just thinking about, which is, gets to your power analysis. Like as an example, let's say that your study's underpowered. It would be great to have a, a, a group of your peers, you know, more or le-- you know, say tear your study apart, but see if there's anything wrong with it. And they say like, "Your, your p- your study is powered to detect like a, a, a 70% difference in, in all-cause mortality." In some, you know, they might be pointing something out that basically saying, "Your study's dead on arrival if you actually run it this way."

    28. PAPeter Attia1:53:26

        R- right. Because i- there's no way you're gonna see an effect size greater than 30%, and yet you're only powered to detect it if it's 70%, which is crazy. So either change the experimental design, figure out a way to raise more money to do this study correctly, um, but yeah, that's, that, that's a, a valuable tool.

13. 1:53:45 – 1:57:54

    Why certain journals are more respected than others

    1. BKBob Kaplan1:53:45

       Okay. And we, we touched on this. I think you talked about, you know, more reputable journals and that's one of the questions is I think people know that certain journals are more respected than others, but i- is there like a m- d- is there a reason why in particular something's respected over another journal?

    2. PAPeter Attia1:54:01

       Y- yeah. So, so th- there's something called an impact factor, and it, it's, it's usually, it's something that changes each year, um, meaning it's usually evaluated on a per year basis. So it's, um, it's the ratio between the number of citations, total citations that point-- So to cite, citing something is referencing that paper. So when you, if you're writing a paper you would say, "Well, you know, Kaplan wrote such and such," and you, you cite that paper. It's a reference. So it's, it's-

    3. BKBob Kaplan1:54:29

       Great, great paper.

    4. PAPeter Attia1:54:29

       Yeah. Great, great paper by Kaplan. [laughs] So it's the, it's the ratio between the total number of citations that come to all articles published by that journal by the total number of articles published by that journal over a previous period of time. So it's typically done over a year. So you would say, okay, um, you know, 27,000 citations came into the journals published out of that article out of 10,000 articles. So 27,000 divided by 10,000 would be 2.7. The impact factor would be 2.7. So to put this in context, there's, you know, like 13,000 journals out there. Um, most all, you know, 98% of them have an impact factor less than 10. Uh, about 95% of them have an impact factor less than five. Um, and about half of them have, have an impact factor less than two. Uh, just to give you kind of a sense of what impact factor looks like. And by the way, there's a t-- the tail on that is very asymmetric. So I mean, the number of journals that have an impact factor of like .4, .7, .8, I mean, is, is, is incredibly high. So this is, these are, um, uh, if, if you look at the distribution of this, there's obviously a very long tail on the small end of this.

    5. BKBob Kaplan1:55:48

       I've got a table here-

    6. PAPeter Attia1:55:49

       Okay

    7. BKBob Kaplan1:55:50

       ... that I can pull up.

    8. PAPeter Attia1:55:50

       Yeah, yeah. Let's, let's take a look at that 'cause I think it's, it's pretty cool to look at this actually.

    9. BKBob Kaplan1:55:55

       Got it.

    10. PAPeter Attia1:55:56

        So if, so if you look at this table, um, you've highlighted the, the journals that have more than 100,000 citations. Uh, this is over, what year is this? 2019.

    11. BKBob Kaplan1:56:05

        Yes.

    12. PAPeter Attia1:56:05

        So you've got, you know, the New England Journal of Medicine, uh, which is kind of staggering, right? Nearly 350,000 citations and we could do the math, but, uh, you can tell how many articles were published because if you divide 347,000 by that number, you get 74.699. So that's the, that's the impact factor for the New England Journal of Medicine. Um, you know, The Lancet 250,000 citations, impact factor 60. So you can sort of see like these are the, whatever, top 28 journals by impact factor. Uh, there's kind of an outlier here, right? Which is the, um, the Cancer Journal for Clinicians, which has a staggering impact factor of 292 despite only having 40,000 citations. So, uh, that's a little bit of a skew. I don't really consider, um... I, I don't really consider that to be sort of, uh, in the same league because it's, it's, it's basically the Global Cancer Society statistic article, and therefore it reports on tons of cancer statistics, and therefore it doesn't really publish that much, but it, um, gets referenced so much. Because anytime someone is basically referencing a cancer statistic, they're gonna reference that. So again, I, I kind of put that in its own little category. Um, but what you can see here is, y- you know... And, and the same, by the way, notice that the WHO Technical Report Series has an impact factor-

    13. BKBob Kaplan1:57:34

        Mm-hmm

    14. PAPeter Attia1:57:34

        ... of 59, but it's only cited like 3,500 times, and so it's cited a lot for a very few number of publications. But again, I think the ones that really matter here clinically, um, the New England Journal of Medicine, obviously Lancet, JAMA, um, are, are sort of your, your, your, your huge clinical ones.

14. 1:57:54 – 2:02:02

    Peter’s process when reading a scientific paper

    1. BKBob Kaplan1:57:55

       Okay. So one more question about reading a scientific paper. So do you have a particular process when you read a, read a paper? Do you just print it out and start to finish, start from the abstract and work your way through, or do you have a particular process in general?

    2. PAPeter Attia1:58:10

       Um, yeah, kind of. Um, I mean, I generally do read the abstract first. Um, and, and that kind of gives me a sense of am I interested in this paper. The, the title of the paper is usually not sufficient for me to know if I'm going to be interested, but the abstract usually is my go, no-go on that. So I could, you know, read 10 abstracts in a, a matter of minutes and decide, you know, how m- do I wanna read three of these papers? Um, the next decision I make is how familiar am I with this subject matter? And if I'm not really familiar with it, I will read the introduction section. A lot of times I am relatively familiar with the subject matter, so I'll just skip the introduction section altogether, and I usually go straight to the methods section. Um, and that gets into the details. So okay, you know, this, if this was a, an, an exercise study, um, and they did muscle biopsies, I just want to really get right down to it. So how many subjects were there? How were they randomized? What were the interventions? When were the biopsies taken? Was there a crossover? You know, blah, blah, I just want to get into all that detail. The next thing I do is I look at the results section, but s- I start with the figures. So I, um, I, I just kind of go right into read the, look at the figure and, and read the legend. And it, and if the authors have done a good job, it's almost standalone at that point. So figures and tables should, uh, in my opinion, be standalone. So the legend should explain everything you need to know. Um, and of course, then reading the pros of the results section kind of adds a, a little bit more, more color to that. Um, and then the last thing I do is I'll, I'll read the discussion section because I'll by this point have formulated my own thoughts on what the strengths and weaknesses of the studies are, what questions remain, et cetera. Um, but you know, oftentimes the authors will have thought of things that I haven't thought of, or they'll thought of things that I disagree with and, and so I'll, I'll, I'll kind of want to go through and, and do that. So, uh, that's my general framework for it, and you'll notice it's, it's, um, quasi-linear but not entirely linear.

    3. BKBob Kaplan2:00:21

       Yeah. I like that. I'd like your example of figures. I can't remember, uh, I might be misremembering, but did Steve Rosenberg, and this probably talks about like the importance of mentorship as well, did he have advice as far as, I think probably when you're, when you're writing a paper, how, like your figures and what they should represent?

    4. PAPeter Attia2:00:37

       Yeah, that was, that was our, our process. So when you finished an experiment, the very first thing you did was you made the figures and legends for your, um, your f- the figures and tables, you made the, those and the legends first, and that's what you would go in and present to him, and you'd present that at journal club, or not journal club, at lab meeting rather. Um, and you wouldn't really take pen to paper to write anything until you had that down. So you, you had to sort of know what are the relevant figures, what are the relevant tables, can I explain them very concisely in a legend? And once you got that down, the paper kind of wrote itself, right? The methods-

    5. BKBob Kaplan2:01:13

       Mm

    6. PAPeter Attia2:01:13

       ... are really easy to write. The results is easy to write. And, and sort of the last thing you would write would be the intro and the abstract. Um, so, a- and again, I don't know that that's the... That was just the way that I was taught to do it, and I, and I found that to be very-

    7. BKBob Kaplan2:01:26

       Yeah

    8. PAPeter Attia2:01:26

       ... productive.

    9. BKBob Kaplan2:01:28

       Okay. I think we've run the list of questions.

    10. PAPeter Attia2:01:30

        We got through them all, man. This was, uh, there's a lot in here, but, um, I, I'm glad we, I'm glad we did this. We, we've been sort of, um, pushing this off for a couple of months now, but, but I, I think it, I'm glad we bit the bullet here.

    11. BKBob Kaplan2:01:44

        Me too. Great topic.

    12. PAPeter Attia2:01:47

        All right, Bob. I'll see you next time.

    13. BKBob Kaplan2:01:49

        All right, Peter. Thanks.

    14. PAPeter Attia2:01:51

        Bye. [outro music]

Episode duration: 2:02:02