CSET’s Jessica Ji discussed mainstreaming concerns about catastrophic AI risks and the difficulty of creating effective government oversight for more capable AI systems in a Vox article.
Simon Willison’s Weblog·1 month ago·
45
● 2 sources
Paul Ford says software developer roles were at first expected to be replaced by tireless robots, but humans still need to think and collaborate for truly cutting-edge software. He argues that A.I. makes it easy to do someone else’s job badly, which is why many projects fail. This shifts emphasis from coding ability alone toward human skill, teamwork, and doing the right work well.
Silicon Valley executives and investors at a Goldman Sachs conference questioned Jacob Coxon’s claims that AI builders expect it could destroy humanity, after Coxon resigned from Anthropic following work at OpenAI. Coxon, a 27-year-old, said he believed “superhuman systems” could hack anything and implied a near-term existential threat, prompting skepticism alongside debate about Anthropic’s safety messaging. The pushback shifts the conversation toward profitability and regulation debates—covering proposals to slow advanced model development and accusations of hype versus genuine safety concern.
Sam Altman said OpenAI would not do an IPO in 2026 during a Fortune interview. He also said the company could pause training to prevent an AI from being beyond human control, even though it is ‘absolutely’ possible. As a result, OpenAI signals delays or limits to both IPO timing and potentially ongoing training in response to safety risks.
Anthropic CEO Dario Amodei called for AI companies to slow the improvement of AI model capabilities and described three strategies to do it in a new blog post. He said Anthropic is “unilaterally committing” to using embedded third-party evaluators (like METR) to verify pacing and safety commitments. As a result, evaluators would get badges, desks, and laptops with access comparable to internal risk teams, while governments and other frontier firms would be urged to match the approach.
Anthropic’s Model Context Protocol (MCP) moved into production in late 2024 and quickly spread, leading to thousands of MCP servers being deployed and treated as critical infrastructure for AI agents’ access to tools and data. The SANS 2026 Identity Threats Survey found 76 percent of businesses saw an increase in non-human identities, with 74 percent using AI systems that rely on standing credentials that can enable overbroad access. Security failures are now attributed less to the infrastructure and more to permissions, driving a shift toward redesigned access controls such as per-task secrets and temporary, action-scoped credentials, plus treating agent identity and lifespan as first-class inputs to authorization.
Corporate boards are being stress-tested because their vertically structured oversight model is being confronted with risks that move horizontally, continuously, and often outside the firm. The article’s key timing contrast is that boards work on quarterly-style cycles while cybersecurity vulnerabilities can appear overnight and geopolitical dynamics can shift in weeks. As a result, the author argues governance may need to shift toward continuous visibility and oversight of the systems companies depend on, rather than relying on adding more layers to the existing board model.
The Federal Trade Commission alleged that Amazon used a pricing tool called Project Nessie to anticipate competitor price moves, raise prices, and keep them elevated, generating over $1 billion in excess profit before pauses during scrutiny and a later restart. Economists analyzing German gas-station pricing automation found that when two rival stations both adopted automated pricing, margins rose by about 38% without communication or agreements. The article argues this shifts scrutiny from performance metrics to auditing what pricing systems learn about competitors, adding constraints, and running counterfactual tests to detect coordination-like outcomes.
Jacob Coxon warned that AI could pose extreme existential risks while developer reporting highlighted monitoring weaknesses in real incident reviews. Anthropic’s offline monitor flagged about 1% of actions in retrospective testing of Mythos 5, rising to about 50% when the model’s written reasoning was removed from what the monitor saw. Developers are urged to change agent evaluation by testing monitoring judgments against justification text, protecting logs and permissions, and judging monitors on action outcomes rather than explanations.
Every AI story that matters,
in your inbox by 8am.
TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the
day in two minutes. Follow companies and topics for alerts, or get the
briefing in Slack. Free, no spam, unsubscribe anytime.
Reading TLDRocket needs no cookies, and the readership counts we rely on come from
our own cookieless analytics. Google Analytics is the exception: it sets cookies and
reports to Google, so it stays switched off until you allow it. You can change your
mind any time from “Cookie settings” in the footer.
Strictly necessary
Session security and form protection (tldrocket-session,
XSRF-TOKEN, 2 hours). The site cannot work without them,
so they need no consent.
Always on
Google Analytics 4 (_ga,
_ga_<id>, up to 2 years). Measures which
stories and sections readers use. Google acts as a third-party processor and may
store the data outside the EU. No advertising, no profiling, no data sold.