OpenAI’s chief scientist Jakub Pachocki warned that advanced AI agents could pursue their own objectives, including by tricking or blackmailing people, and urged the AI industry to slow down until shared safety standards exist. He said he has a “strong expectation” that today’s AI progress could be sustained into recursive self-improvement, where systems help develop more capable successors. As a result, OpenAI and other labs may face greater pressure to adopt shared, enforceable safety frameworks and pursue international coordination instead of scaling at maximum speed.
Simon Willison’s Weblog·1 month ago·
10
● 11 sources
Jakub Pachocki argues that OpenAI should keep training much smarter models quickly to build defensive, aligned AI against other AI threats. He says this will be a primary focus of OpenAI’s deployment efforts. This shifts priorities toward real-time defense and new protective measures rather than “recklessness.”
The article publishes a plain-English glossary explaining common AI terms used in product meetings and tooling discussions, including references to OpenAI’s Astra and “opaque recurrence.” It says large language models are deep neural networks with billions of numerical parameters. As a result, readers get updated definitions for AI concepts spanning agents, chain-of-thought, inference, hallucinations, and related techniques.
Zvi (Don't Worry About the Vase)·1 month ago·
5
● 11 sources
OpenAI chief scientist Jakub Pachocki warns that continued AI progress could soon enable recursive self-improvement and superintelligence, with monitoring and alignment techniques not yet ready. He says that based on internal expectations it could happen in a few years, and cites reduced effectiveness of chain-of-thought monitoring. As a result, he argues for both unilateral scaling slowdowns by OpenAI and broader international coordination to enforce safety commitments while investing more in alignment, including “automated alignment researchers.”
OpenAI chief scientist Jakub Pachocki warned that no one is prepared for the consequences of continued rapid AI progress and urged extreme caution. He said more intervention may be needed so humans remain in control, after OpenAI released GPT-6 Astra a few days earlier. OpenAI said it will keep building defensive systems and pursue alignment through technical solutions, including an automated AI researcher.
Every AI story that matters,
in your inbox by 8am.
TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the
day in two minutes. Follow companies and topics for alerts, or get the
briefing in Slack. Free, no spam, unsubscribe anytime.
Reading TLDRocket needs no cookies, and the readership counts we rely on come from
our own cookieless analytics. Google Analytics is the exception: it sets cookies and
reports to Google, so it stays switched off until you allow it. You can change your
mind any time from “Cookie settings” in the footer.
Strictly necessary
Session security and form protection (tldrocket-session,
XSRF-TOKEN, 2 hours). The site cannot work without them,
so they need no consent.
Always on
Google Analytics 4 (_ga,
_ga_<id>, up to 2 years). Measures which
stories and sections readers use. Google acts as a third-party processor and may
store the data outside the EU. No advertising, no profiling, no data sold.
