By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Peter Attia MD#143 - John Ioannidis, M.D., D.Sc.: Why most biomedical research is flawed, and how to improve it
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
- 0:11 – 11:13
Guest Introduction and John's Path Through Math, Medicine and Mentors
Peter Attia introduces John Ioannidis and the episode's themes, then asks about his background. Ioannidis describes growing up in Athens, combining mathematics with medicine, his physician-scientist parents, and mentors such as Trichopoulos, Mellinger and Chalmers who shaped his view of evidence-based medicine.
- •Ioannidis describes himself as trying to be a scientist who keeps correcting his previous self
- •Medicine appealed as a way to save lives, but needs rigorous science and quantitative tools
- •Both parents were physician scientists, giving him early exposure to research and computing
- •Meeting Tom Chalmers and Joe Lau showed him mathematics, rigorous methods, evidence and medicine combined in one whole
- •Early clinical exposure showed little data and a lot of expert opinion guiding practice
- 11:13 – 19:39
Why Most Published Research Findings Are False
Attia raises Ioannidis's 2005 PLOS Medicine paper. Ioannidis explains the mathematical model, which weighs prior odds, statistical power, bias and the number of scientists attacking a question to estimate how often a significant finding is a false positive.
- •The model tries to match empirical data on unreliable and overturned results
- •Prior chance of discovery differs across fields
- •Power, bias and the multi-team research environment all feed into the false positive rate
- •In most circumstances a nominally significant result near p=0.05 has a greater than 50% chance of being a false positive
- 19:39 – 26:09
P Values, Statistical Significance and Clinical Significance
Attia asks Ioannidis to explain p values and how statistical significance differs from clinical significance. Ioannidis discusses misconceptions, proposals for stricter thresholds, the debate over banning statistical significance, and why he thinks pre-specified rules still make sense for some designs.
- •Medicine cares about clinical significance, not just statistical significance
- •A p value does not account for bias, so astronomically small p values can simply reflect bias
- •He and others suggested a default threshold of 0.005
- •He is reluctant to ban statistical significance because most scientists are not trained in alternatives
- •Pre-registered designs like randomized trials benefit from clear rules of the game
- 26:09 – 31:01
Under-Powered and Over-Powered Studies
Attia asks how under-powered studies can yield false positives. Ioannidis explains that low power exaggerates effect estimates and that competition and funding incentives push toward small studies, while big data now creates the opposite problem of over-powered studies that merely measure bias.
- •Under-powered fields produce more false positives and exaggerated effect sizes
- •Competition and scarce resources push scientists toward small, selectively reported studies
- •There is a disincentive to refute incorrect results
- •Big data can produce significant but meaningless signals driven by even slight bias
- 31:01 – 38:47
Genetics vs. Nutritional Epidemiology: Why One Reformed
Ioannidis describes how genetics moved from false-positive candidate gene studies to large, shared, genome-wide work. Attia asks why nutritional epidemiology has not followed, and Ioannidis cites measurement quality, entrenched beliefs, outdated methods and the Cornell case.
- •Genetics adopted very large studies, whole-genome analysis and data sharing in consortia
- •Genotyping is highly accurate, while nutrition relies on biased questionnaires
- •Strong expert opinion and cultural beliefs shape nutrition far more than genetics
- •Genetics uses stringent thresholds like 10 to the minus eight and independent analyst teams
- 38:47 – 44:50
Bradford Hill's Criteria and Tobacco as a Test Case
Attia asks about Austin Bradford Hill and goes through his criteria using tobacco. Ioannidis says Hill was cautious and never meant them as hard rules, and that strength and consistency, as with tobacco, are the exception in modern epidemiology.
- •Hill noted that none of his criteria is bulletproof
- •Ioannidis does not want to give a nihilistic view of epidemiology and cites tobacco as solid evidence
- •Tobacco shows huge odds ratios of 10 to 30 and strong consistency
- •Odds ratios of 20 are rare today, and in genetics effects of 1.01 are common
- •Softer criteria like specificity and analogy matter less when strength and consistency are strong
- 44:50 – 52:42
Getting Better Answers in Nutrition: Trials, Efficacy and Effectiveness
Facing weak criteria in nutritional epidemiology, Attia asks where to look next. Ioannidis points to randomized trials, noting that large ones rarely confirm observational signals, while Attia argues efficacy matters too. Ioannidis outlines controlled experiments, Mendelian randomization and exposure-wide association studies.
- •Over 200 large nutrition trials mostly fail to show the benefits suggested by observational data
- •Non-adherence is part of real life, so effectiveness is what matters to Ioannidis
- •Attia argues efficacy knowledge is needed to inform policy and the food environment
- •Options include controlled metabolic-clinic trials and Mendelian randomization
- •Exposure-wide and outcome-wide analyses account for multiplicity and correlation
- 52:42 – 59:36
The Cookbook Review, Cherry-Picked Meta-Analyses and Institutional Beliefs
Attia raises Ioannidis's 2012 cookbook ingredients paper showing that nearly every ingredient had studies linking it to cancer risk. Ioannidis discusses the reaction, warns that meta-analyses built from cherry-picked teams magnify bias, and talks about schools of thought like Harvard's.
- •Nearly all of 50 randomly chosen ingredients had published studies on cancer risk
- •The paper made both enemies and friends, and some now use exposure-wide approaches
- •Inviting only teams that already found an association turns meta-analysis into solidified cherry-picking
- •More evidence alone does not make things better
- •Ioannidis says he is not committed to any result and worries about resistance to doing it the right way
- 59:36 – 1:08:22
Beliefs, Conflicts of Interest and Questionable Research Practices
Attia argues that entrenched prior beliefs make epidemiology hard to trust, using the D.B. Cooper documentary as an analogy. Ioannidis discusses transparency about beliefs, financial and non-financial conflicts, the Wansink case, and why questionable practices rather than fraud are the common problem, to be addressed through training rather than punishment.
- •Ioannidis says beliefs are fine if scientists are transparent and add safeguards against fooling themselves
- •Financial conflicts are highly prevalent, and non-financial conflicts also matter
- •The Wansink case involved urging students to torture data until they got nice results
- •Fraud is uncommon, while questionable research practices are extremely common
- •Remedies are training and a grassroots movement, not firing the workforce
- 1:08:22 – 1:16:53
Improving Science Without Cutting It Back: Workforce, Funding and Risk
Attia asks how many of the 35 million authors are fit to lead research. Ioannidis rejects simply cutting research and argues for better standards and incentives for risk-taking. Attia suggests philanthropy fills the risk gap, and Ioannidis says public funding should rise, with industry paying for translational work and public funds testing products.
- •Most of the 35 million author IDs are one-time co-authors, with fewer than one million principal investigators
- •Cutting research could worsen incentives to cut corners
- •Researchers should be incentivized to try risky ideas rather than chase significant results
- •No-strings philanthropy can be catalytic, but public funding should increase
- •Product testing should be done with unconflicted funding and investigators
- 1:16:53 – 1:21:38
Communicating Science Honestly to the Public
Attia compares science's funding narrative to NASA's and the Manhattan Project and says scientists need to be better communicators. Ioannidis says communication is hard in polarized environments and that scientists should stay honest about uncertainty and likely failure, while Attia warns that a vacuum gets filled by charlatans.
- •Most grants will find nothing, and a few out of a thousand transform the world
- •Scientists' voices are drowned out by lobbies, social media and agendas
- •Ioannidis favors defending the method and honesty over exaggerated promises
- •Attia stresses that scientific thinking is not innate and that good scientists must speak up
- 1:21:38 – 1:31:03
Absurd Nutrition Claims and the PREDIMED Trial
Attia raises Ioannidis's paper on single-food claims, such as hazelnuts, bacon and eggs affecting lifespan. They then discuss PREDIMED, where Ioannidis explains the randomization was subverted, the study was retracted and republished, and why that undermines confidence even in top trials, including multicenter trials run by industry.
- •Ioannidis calls the lifespan estimates tongue in cheek translations of reported relative risks
- •Complicated systems with inaccurate measurements and selective reporting produce implausible big pictures
- •PREDIMED showed large benefits for Mediterranean arms, and Ioannidis disclosed his personal bias in favor of the diet
- •Randomization was found to be subverted, leading to retraction and republication with similar results
- •Multicenter trials can suffer from untrained sites and a lack of allocation concealment
- 1:31:03 – 1:45:15
Santa Clara Seroprevalence Study and the Political Backlash
Attia recalls the idea of measuring seroprevalence in 2020, and Ioannidis describes the Santa Clara and LA County studies, which found the virus about 50 times more common than documented cases. He says later studies validated them, and describes the hate mail, smears and threats that followed, including to his family.
- •The studies were led by others, with Ioannidis as one of many co-investigators
- •Prevalence was about 50 times higher than documented cases suggested
- •Later seroprevalence studies, he says, largely validated the results
- •Ioannidis says criticism had value, but much of the animosity was political
- •He argues scientists must be protected from attacks so they are not silenced
- 1:45:15 – 1:52:11
Science and Politics, and Ioannidis's Optimism for the Future
Attia contrasts changing one's mind in science with how it is seen in politics. Ioannidis says the pandemic has brought science to the limelight but also made it a hostage of lobbying forces, and he closes by expressing excitement about all he does not yet know before the episode wraps up.
- •Changing your mind is a scientific virtue but seen as a political weakness
- •The pandemic created broad public interest in science, mostly without science education
- •This is an explosive mix and an opportunity to communicate better science
- •Ioannidis is optimistic as long as he keeps finding out what he doesn't know
- •He says we are just at the beginning of knowledge
