By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Peter Attia MD269 - Good vs. bad science: how to read and understand scientific studies
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
- 0:02 – 6:39
Why a primer on reading studies, and how a study goes from idea to execution
Peter Attia and Bob Kaplan explain why they're doing a Q&A on making sense of scientific literature amid conflicting headlines. Peter then walks through the steps before a study runs: null and alternative hypotheses, experimental design, power analysis, IRB approval, outcomes, statistics plan, pre-registration and funding.
- •Prompted by Tim Ferriss asking how to get better at reading research
- •Good science is hypothesis driven, framed as a null versus an alternative hypothesis
- •Design choices include duration, frequency, measurement and subject numbers
- •IRB approval, primary and secondary outcomes, and pre-registration happen before the study
- •Not every study type follows all of these steps
- 6:39 – 20:29
Types of studies: case reports, cohorts, experiments and meta-analyses
Peter lays out three categories of studies: observational, experimental, and reviews or analyses. He covers case reports, case series, retrospective and prospective cohorts, non-randomized versus randomized trials, and why a meta-analysis is not automatically better than a good randomized trial.
- •Case reports and series generate hypotheses but are not generalizable
- •Cohort studies can be retrospective or prospective, and observational
- •Non-randomized trials introduce selection bias; randomization is the gold standard
- •Meta-analyses weight studies by precision, but garbage in means garbage out
- •Each study within a meta-analysis still needs to be examined
- 20:29 – 28:32
The phases of clinical trials (phase 1 through 4)
Peter explains the FDA-style phases of human drug testing, starting with the IND filing. Phase 1 tests safety and dose escalation, phase 2 looks at early efficacy, phase 3 is large and randomized, and phase 4 is post-marketing, with semaglutide as an example.
- •IND application follows preclinical animal work
- •Phase 1 is small dose-escalation for toxicity
- •Phase 2 is typically open label and can be biased without randomization
- •Phase 3 is large, randomized, and blinded when possible
- •Phase 4 expands indications and detects missed side effects
- 28:32 – 38:23
Observational studies: healthy user bias, recall bias and food questionnaires
Peter and Bob discuss what lowers confidence in observational research. They cover selection and healthy user bias with bacon and diet soda examples, and the unreliability of food frequency questionnaires, with Peter arguing they should be abandoned.
- •Healthy user bias distorts associations such as meat consumption and diet soda
- •Controlling statistically cannot strip out all lifestyle differences
- •Food frequency questionnaires rely on poor recall and often a single baseline survey
- •Recall is more reasonable for profound events, but causality still can't be inferred
- •Peter thinks epidemiology's role has been overvalued
- 38:23 – 49:27
Performance bias, primary vs. secondary outcomes, multiple testing and confounding
The conversation turns to biases that affect trials: the Hawthorne and Avis effects and the subtle performance bias in lifestyle RCTs, such as the potato diet example. Peter then covers primary versus secondary outcomes, multiple hypothesis testing with the Bonferroni correction, and confounding.
- •Hawthorne effect and Avis effect change behavior in participants
- •Unequal attention between arms can produce a biased RCT result
- •Primary outcomes are what a study is powered for; failing it makes a study null
- •Multiple looks at data raise false positives, which the Bonferroni correction addresses
- •Confounders can't all be identified, as the hot chocolate and skiing example illustrates
- 49:27 – 59:39
What to look for in experiments: randomization, PREDIMED, crossover and blinding
Peter and Bob discuss what boosts or lowers confidence in experimental studies. They use the PREDIMED reanalysis to show randomization problems and cluster randomization, then cover control groups, crossover designs, and blinding.
- •PREDIMED was reanalyzed after some clinics were randomized as whole units
- •Cluster randomization requires different statistical adjustment
- •Crossover designs need fewer subjects but must account for washout and order effects
- •Single and double blinding reduce bias, though psychedelics are hard to blind
- •Every study will have some mistakes
- 59:39 – 1:03:41
Generalizability, effect size, funding and conflicts of interest
Peter lists further factors: sample size, duration, who is in the study (including the historical exclusion of women), multi-site versus single-site, clinical versus statistical significance, adverse events, funding and authors' conflicts of interest.
- •Heterogeneous populations are more generalizable but harder to show effects in
- •Many drugs were approved for women after being studied only in men
- •Multi-site studies add heterogeneity but are harder to control
- •A one mmHg blood pressure drop may be statistically but not clinically significant
- •Funding sources and declared conflicts of interest matter
- 1:03:41 – 1:16:20
Power, p-values and statistical significance
Peter explains power as one minus the false negative rate, and the p-value as the false positive threshold. Using a sample size table, he shows how effect size drives needed subjects, and why both underpowered and overpowered studies mislead. Bob adds tips for finding power details in protocols.
- •P-value of 0.05 is an arbitrary ceiling on false positives
- •Power is typically 80-90%, meaning a 10-20% false negative rate
- •Smaller expected effects require far more subjects per arm
- •Underpowered null studies don't show there is no difference
- •Overpowered studies can find clinically irrelevant effects
- 1:16:20 – 1:27:02
Effect size: relative vs. absolute risk, hazard ratios and number needed to treat
Bob asks how effect sizes are measured. Peter explains absolute versus relative risk reduction with the Women's Health Initiative example, how to read hazard ratios, and how number needed to treat is derived from absolute risk reduction.
- •Relative risk is misleading without absolute risk
- •Women's Health Initiative: a 25% relative increase equaled one extra case per 1,000 women
- •Hazard ratios capture risk over time; subtract one and multiply by 100 for percent change
- •NNT equals one divided by the absolute risk reduction
- •Sub-100 NNTs are impressive, while an NNT of 1,000 is less so
- 1:27:02 – 1:34:15
Interpreting confidence intervals and the takeaway on reading science
Bob explains confidence intervals and the common misreading of them, and Peter offers the quick check of whether the interval crosses one. They discuss framing them as uncertainty intervals, and Peter closes the section by saying that following science requires effort and practice.
- •A 95% CI corresponds to a 0.05 p-value threshold
- •Correct meaning: across 100 samples, about 95 intervals would contain the true value
- •If the interval crosses one, the result is not significant
- •Wide intervals mean greater uncertainty, a framing Andrew Gelman uses
- •Reading papers gets easier with practice, and media reports may lack proper analysis
- 1:34:15 – 1:43:02
Why studies are stopped early: safety, benefit and futility
Peter outlines three reasons for stopping a trial early: safety, benefit and futility. Examples include the torcetrapib CTEP inhibitor trial stopped for excess deaths, PREDIMED for benefit, and Look AHEAD for futility.
- •Safety breaches stop trials, as in torcetrapib with Lipitor where deaths were higher in the combination arm
- •Benefit stops trials when continuing would be unethical, as in PREDIMED
- •Futility means nothing later can change the result, as in Look AHEAD
- •Monitoring boards use pre-specified p-value thresholds for interim looks
- •Different drugs in a class can behave differently
- 1:43:02 – 1:46:55
The journal review process and peer review
Bob asks about the review process once a study is done. Peter describes submission, editor screening, peer review by invited experts, conflicts of interest, blinding options, and revision rounds, as authors work down from prestigious journals.
- •Editors screen for fit and originality before peer review
- •Reviewers are chosen for expertise, and conflicts may require declining
- •Peer review can be single blinded, double blinded or open
- •Typically about three reviewers accept, reject or request revisions
- •Authors typically start at the most prestigious journal and work down
- 1:46:55 – 1:53:45
Publication bias and how to combat it
Peter discusses publication bias using the Minnesota study that went unpublished for 16 years. Pre-registration, funder and journal requirements, and registered reports are discussed as remedies.
- •Negative results are often unpublished though they are just as important
- •Minnesota study was published 16 years late because the lead author said it didn't turn out as wanted
- •Pre-registration on clinicaltrials.gov locks in methods and outcomes
- •Registered reports provisionally accept a paper based on its protocol
- •Early peer review of protocols can catch underpowered designs
- 1:53:45 – 1:57:57
Why some journals are more respected: impact factor
Peter explains impact factor as citations divided by articles published, and shows how most journals have low values. Looking at a table, he notes NEJM, The Lancet and JAMA as key clinical journals and flags outliers with high impact factors from few articles.
- •Impact factor is total citations over number of articles published
- •Most journals have an impact factor below five, and half below two
- •NEJM's impact factor is about 74.7, and The Lancet's about 60
- •CA: A Cancer Journal for Clinicians is an outlier because it publishes cancer statistics
- •NEJM, Lancet and JAMA are the major clinical journals
- 1:57:57 – 2:02:02
Peter's process for reading a scientific paper
Peter describes how he reads a paper: abstract first, introduction only if unfamiliar, then methods, figures with legends, results and finally discussion. He adds the figures-first approach to writing papers he learned from Steve Rosenberg.
- •The abstract is his go or no-go filter
- •He skips the introduction when familiar with the topic and reads methods in detail
- •He looks at figures and legends before the results text
- •The discussion comes last, after forming his own view
- •In Rosenberg's lab, figures and legends came first and the intro and abstract last
