By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Latent SpaceThe Watchdogs of AGI — Rune Kvist of AI Underwriting Company
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
- 0:00 – 0:46
Robots, strict liability, and why AI will need new trust systems
Rune opens with a forward-looking view: as AI moves from software to physical systems, the tolerance for failure drops sharply. He argues that longer-horizon agents and robot-to-human incidents will trigger tougher liability regimes and new kinds of failure modes. He also suggests autonomous agent-to-agent interactions will create legal and trust questions that current systems aren’t built for.
- •Physical AI will likely face much stricter liability expectations than software-only systems
- •As agents become more autonomous and long-horizon, new failure modes and risk surface emerge
- •Agent-to-agent interactions without humans raise new trust and legal questions
- •A future with autonomous agents may require new mechanisms for recourse and accountability
- 0:46 – 1:50
AIUC’s $40M raise and the claim that risk is now the adoption bottleneck
The hosts introduce Rune and AI Underwriting Company (AIUC), and he announces a $40M fundraise led by Ribbit Capital and First Monarch. He explains that AIUC’s founding hypothesis—risk would slow AI adoption—has shifted from speculative to obvious in the current moment. He also names a set of customers AIUC is working with.
- •Announcement of a $40M round led by Ribbit Capital and First Monarch
- •Rune says risk, not interest, is now the binding constraint on adoption
- •AIUC highlights work with customers including Cursor, Harvey, Lovable, and ElevenLabs
- •The conversation frames recent events as making the risk bottleneck undeniable
- 1:50 – 7:31
From scaling laws to Anthropic: a path into frontier AI
Rune recounts selling an ed-tech company, reading the Scaling Laws paper, and realizing its implications for investment and the pace of progress. He describes traveling to San Francisco to find the authors and joining a small early Anthropic team. He portrays that period as uncertain about products and value accrual, but full of conviction that scaling compute and data would drive capability gains.
- •Reading the Scaling Laws paper convinced Rune that ‘bigger models get smarter’ would trigger predictable investment cycles
- •He moved to San Francisco to find the people behind the work and met Anthropic early
- •Anthropic was debating how to deploy models, engage with the world, and whether to pursue revenue
- •Rune describes early 2022 as a ‘fog of war’ with unclear product and stack outcomes
- 7:31 – 8:48
Anthropic’s early culture: cohesive mission under race pressure
Rune and Vibhu discuss what it felt like inside Anthropic: a strong, cohesive mission paired with constant tension about outcomes going well or badly. Rune emphasizes the incentive challenge of racing while trying not to cut corners. He also notes the team’s deep conviction in the core scaling thesis even as specifics remained unclear.
- •Anthropic is described as cohesive and mission-oriented, with sustained core beliefs
- •Internal tension came from believing AI could go very well or very badly
- •Rune highlights the risk of race dynamics pushing teams to cut corners
- •The team held strong conviction that scaling compute and data yields smarter models
- 8:48 – 10:20
Why trust, not capability, may limit adoption—and why AIUC exists
Rune defines AIUC as building “confidence infrastructure” for frontier AI via standards and insurance. He uses Waymo as an example of systems that can be “superhuman” yet still constrained by trust, liability, and permissioning rather than raw capability. He argues the trust problem worsens as AI gets more capable and autonomous, and that regulated sectors need clear promises about behavior.
- •AIUC’s mission is confidence infrastructure through standards and insurance
- •Waymo is used to illustrate that capability can outpace trust and liability acceptance
- •Rune argues better AI increases autonomy and expands risk surface
- •Governments, banks, hospitals, and militaries need clarity on what AI will/won’t do
- 10:20 – 13:02
Standards + insurance as a recurring pattern in technology adoption
Rune traces a historical pattern: new technologies create risks before regulation catches up, and markets build confidence mechanisms to enable adoption. He claims a common blueprint is standards (rules and tests) paired with insurance (financial accountability and incentives to quantify and reduce risk). He explains how this maps to enterprise procurement needs and a “golden sentence” for trust-building.
- •Markets often build confidence infrastructure ahead of regulation
- •Standards define rules of the road and specify tests; insurance aligns incentives by paying losses
- •Insurers are positioned to quantify risk and push for risk reduction to protect profit
- •AI vendors selling into enterprises need credible third-party testing plus insurer backing
- 13:02 – 15:30
Founding story and team: working with family, connecting insurance and evals
Vibhu asks how Rune and Rajiv came together; Rune explains Rajiv is his soon-to-be brother-in-law and they’ve known each other for a decade. He describes Rajiv’s background in insurance consulting and frontier model testing and government work, which complemented Rune’s trust-infrastructure thesis. Rune notes they’re now a 20-person company.
- •Rajiv is Rune’s soon-to-be brother-in-law; they’ve known each other for a decade
- •Rajiv’s experience spans insurance and model testing work connected to frontier releases
- •Rune frames AIUC as solving a market problem the government struggles to match in pace
- •AIUC’s team size is stated as 20 people
- 15:30 – 19:34
AIUC-1 certification: categories, evidence, and quarterly refresh cadence
Rune walks through AIUC-1 as a standard for agent security, safety, and reliability built around the concerns of Fortune 1000 security leaders. He emphasizes that the standard is grounded in technical testing rather than paperwork, requiring quarterly testing and simulations. He also describes a consortium model where risk leaders help drive quarterly updates based on emerging concerns.
- •AIUC-1 is positioned as a comprehensive framework based on enterprise risk concerns
- •A design goal is avoiding ‘paperwork theater’ by grounding requirements in testing
- •Passing requires quarterly testing with large-scale simulations and remediation loops
- •A consortium of risk leaders meets regularly to drive updates and keep pace
- 19:34 – 24:09
How audits and stress tests work: auditors check controls; AIUC runs the tests
Vibhu probes how detailed requirements translate into testing. Rune explains three requirement types—technical controls, test controls, and policy controls—and distinguishes between auditors verifying evidence (e.g., whether a control exists) and AIUC performing effectiveness testing. He outlines typical timelines, remediation, and what it means to be certified with quarterly refreshes.
- •Requirements span technical controls, third-party testing, and policy/accountability controls
- •Auditors (e.g., KPMG or Schellman) review evidence that controls exist
- •AIUC performs testing to evaluate whether safeguards actually work
- •End-to-end certification often takes weeks, including remediation and retesting
- 24:09 – 26:09
What companies miss: corner cases, weak guardrails, and adversarial stress testing
Asked what distinguishes companies that pass, Rune says many haven’t seriously stress-tested adversarial behavior and corner cases. He notes many teams implement guardrails but they often don’t work well in practice. He frames expertise as “finicky” work: reading relevant literature, enumerating tricks, and systematically testing whether controls hold up under attack.
- •Many teams optimize for the average case and underinvest in adversarial stress testing
- •Guardrails are commonly present, but effectiveness is the real gap
- •Robustness requires exploring corner cases and known attacker framings
- •Passing is about being able to make credible promises to risk-sensitive customers
- 26:09 – 34:15
From agents to frontier models: government trust gaps and the need for a broker
Rune shifts to model-level risks, describing them as national-security-relevant and analogous in shape to enterprise trust problems. He argues governments must make go/no-go decisions with limited technical capacity and that labs have incentives that can hinder trust. He proposes an independent third party, supported by a public standard and consistent audit methods, to produce legible, repeatable risk reports.
- •Model risks are framed as national security concerns distinct from enterprise agent concerns
- •Rune describes a trust gap between governments and labs driven by incentives
- •He argues self-auditing is unusual in other industries and a neutral third party is needed
- •A public standard plus consistent, explainable audit methodology is central to trust
- 34:15 – 38:57
Which risks matter at the model layer: cyber, child safety, and biological risk
Rune lists categories of model risks that are becoming salient: cyber as the immediate focus, child safety as both practical and political, and biology-related risks as emerging but concerning. He contrasts agents as deliberately narrowed systems with models that accept arbitrary inputs and create an effectively unbounded evaluation surface. He highlights the coordination challenge of assembling many subject-matter experts into one coherent framework and output.
- •Cyber is described as the most tangible ‘risk of the day’
- •Child safety is framed as increasingly important and politically salient
- •Biology-related risks are discussed as a pipeline concern tied to weaponization accessibility
- •Models have a much broader risk surface than narrow agents, requiring many expert inputs
- 38:57 – 51:35
Why AIUC pairs standards with underwriting: incentives, Lloyd’s partnership, and policy basics
The hosts press on the “money part,” and Rune argues for-profit standards can work when incentives are aligned with insurers. He cites historical patterns where insurers helped create safety standards and positions AIUC’s name as an homage to that lineage. He then describes how AI insurance demand sits between AI builders and buyers, explains what policies look like (perils, limits, premiums), and discusses the role of Lloyd’s as the risk-capital provider.
- •Rune argues nonprofit standards can be unresponsive; for-profit models can be effective with the right incentives
- •Insurers are framed as well-incentivized partners because they pay losses and want risk reduction
- •Insurance policies specify perils, limits, and pricing; the value includes trust signaling
- •Lloyd’s is described as providing trusted risk capital while AIUC provides underwriting/testing inputs
- 51:35 – 54:36
Liability, precedent, and why standards clarify ‘duty of care’
Using a hypothetical about a small subscription leading to massive downstream damages, the conversation turns to limits, ambiguity, and how courts clarify responsibility over time. Rune cites an example where a company was held responsible for its chatbot’s promises to customers, framing it as precedent that shapes future expectations. He explains how standards influence negligence determinations by defining best practices that courts can reference.
- •Coverage depends on policy limits; larger limits require higher premiums
- •Early ambiguity in AI liability gets resolved through court precedent over time
- •Rune describes courts treating chatbot statements as binding promises by the deploying company
- •Standards can shape negligence and duty-of-care judgments by defining expected practices
- 54:36 – 1:12:35
Ongoing monitoring, eval awareness, and where testing is headed
The hosts discuss how certification interacts with rapid shipping cycles and the limits of quarterly audits. Rune points to requirements around customer-facing promises and internal testing processes, while acknowledging uncertainty and software bugs as a reality. He also discusses eval awareness—agents behaving differently when they detect evaluation—and argues monitoring and incident detection may become more important despite being more intrusive.
- •Quarterly audits don’t cover daily releases, but standards can require internal pre-release processes
- •Smaller companies need third-party trust signals beyond self-authored security claims
- •Eval awareness is presented as a growing challenge that can weaken eval reliability
- •Monitoring and fast detection of real-world failures is framed as a key truth source
- 1:12:35 – 1:26:57
Prediction markets vs. audits, certifying engineers, and the ‘watchdogs of AGI’ question
The host proposes prediction markets as an alternative; Rune argues they rely on public information and can’t generate the specific private information enterprises and governments need, unlike audits. They then explore whether certifying AI engineers would help shift security considerations “left” in development, with Rune supporting training aligned to a clear grading scheme but noting the difficulty of strong guarantees. The episode closes on AIUC’s roadmap (agents → models → robotics), hiring needs, and Rune’s view that even if “AGI” is declared, independent watchdogs remain necessary—while acknowledging the need to scrutinize watchdogs too.
- •Prediction markets are framed as limited by lack of public info and inability to generate bespoke assurance
- •Engineer certification could help if it teaches builders the security spec earlier, but guarantees are hard
- •AIUC’s roadmap is described as expanding from agents to models to robotics as risk stringency rises
- •Rune argues labs can’t be their own watchdogs; insurers are positioned as a stabilizing incentive against a race to the bottom
