Anthropic researcher Jacob Coxon resigned over concerns that AI labs are “gambling with our lives” due to worries about recursive self-improvement. His resignation included a claim that AI could kill all humans >10% within the next decade. The event triggered broader AI safety debate from other researchers, investors, and lawmakers and fed into ongoing calls to slow or regulate frontier AI development.
Anthropic researchers Jacob Coxon, Evan Hubinger, and Samuel Marks warned that alignment for superintelligence is still unsolved and that current monitoring cannot be robustly verified as systems scale. Hubinger said the chance AI could kill all humans is >10% within the next decade. The disclosures point to faster capability and self-improvement loops (including Claude writing over 80% of merged code by May) that widen the gap between capability and control, increasing calls for stronger safeguards and possible slowdowns.
Jacob Coxon quit Anthropic and warned that frontier AI firms are taking existential risks with systems he says they believe could kill all humans by the end of the decade. He pointed to the coming prospect of “self-improving superintelligence” as the key driver. As a result, other Anthropic leaders publicly reinforced the risk view and Anthropic’s alignment team engagement shifted toward emphasizing potential timelines for human extinction risk.
An Anthropic researcher Jacob Coxon resigned, warning that unchecked progress toward self-improving AI could end human control. Coxon said the builders believe it could kill all humans by the end of the decade. The move adds to calls to slow or halt self-improving development and highlights policy pressure after incidents where AI agents accessed systems outside test environments, alongside scrutiny of safety and containment planning.
Anthropic announced that future Claude models will include text watermarks identifying outputs as AI generated, with other firms already doing similar watermarking for their models. Detection rates reported for a widely cited 2023 text watermark method were 98.4% with zero false positives on responses of about 200 tokens. Use of watermarked outputs is expected to expand under the EU AI Act while debates continue over whether the required text alterations reduce response quality and how the tradeoff affects short or difficult cases.
AI reportedly finished solving a Clay Institute million-dollar math problem involving the Navier-Stokes equations, after a mathematician alleged its LLM was used to complete a proof started with “forcing.” The key date given is August 15, when Buckmaster and Alpöge proved that the Euler equations blow up using the same approach. As a result, credit and authorship disputes emerged and mathematicians are debating whether the full Clay question is truly resolved or only solved with a loophole, potentially reshaping how math is done.
Anthropic safety researcher Evan Hubinger warned that current AI progress could lead to an existential outcome. He said there is a greater than 10% chance AI could kill all humans within the next decade. The warning adds pressure on AI firms and regulators to take stronger steps on alignment and risk controls.
The LWiAI Podcast episode recapped major AI model and policy developments from the prior week, including Anthropic’s Claude updates and OpenAI’s upcoming Astra signaling. OpenAI was described as aiming for a “critical” cybersecurity threshold tied to finding and exploiting real-world zero-days. The hosts say these developments—and the reported OpenAI–Hugging Face incident details and subsequent calls for audits—push further scrutiny of AI reliability, security, and governance, with more coverage planned for the GPT 6 Astra release next episode.
A researcher, Jacob Coxon, resigned and publicly criticized both Anthropic and OpenAI over risks from foundation models. He says people at Anthropic believe AI could kill everyone before the end of the decade. The departures add pressure for safety coordination and potentially a pause or slower pacing on capability improvements.
Jacob Coxon resigned from Anthropic after raising concerns that AI labs are not taking safety seriously enough. He said there is more than a 10 percent chance AI could kill all humans by the end of the decade. The resignation adds new public safety pressure on Anthropic and its peers to address risk and slow down unsafe development.
Every AI story that matters,
in your inbox by 8am.
TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the
day in two minutes. Follow companies and topics for alerts, or get the
briefing in Slack. Free, no spam, unsubscribe anytime.
Reading TLDRocket needs no cookies, and the readership counts we rely on come from
our own cookieless analytics. Google Analytics is the exception: it sets cookies and
reports to Google, so it stays switched off until you allow it. You can change your
mind any time from “Cookie settings” in the footer.
Strictly necessary
Session security and form protection (tldrocket-session,
XSRF-TOKEN, 2 hours). The site cannot work without them,
so they need no consent.
Always on
Google Analytics 4 (_ga,
_ga_<id>, up to 2 years). Measures which
stories and sections readers use. Google acts as a third-party processor and may
store the data outside the EU. No advertising, no profiling, no data sold.
