Quoting @joedaroo
Simon Willison’s Weblog Simon Willison
Opinion — commentary, not a factual news event.
@joedaroo said OpenAI was surprised by how fast and suddenly its models’ capabilities advanced for areas tied to incidents such as cyber, swarming, and message boards. Today, the key takeaway is to test whether people, systems, processes, incident response, and communications are resilient to sudden AI capability jumps. As a result, organizations are urged to prepare teams and procedures in advance for unexpected changes in AI capability.
Why it matters
To say that we were surprised at the jump and suddenness of the capabilities of our models when it came to “cyber” or “swarming” or “message boards” or anything else related to the incidents is an understatement. Security posture takes time to develop. It’s not just about hardening the systems at play; you have to ingrain it in the culture of the company. The literal people themselves in your organization have to change and evolve with it. These jumps in capabilities were so fast and so sudden that they created an extremely difficult problem. [...] So today my hope is that everyone around the world can look at their own organization and say: how can I deal with a surprise or a sudden jump in AI capability? Are my people, my systems, or my processes resilient to surprises? Do my teams know what to do when something goes wrong? Do I have the right incident response? The right comms and messaging? Do I have the right people ready to go when capabilities jump? — @joedaroo, Agent Security a