Open Questions On Open Weights

Open-weights AI works like open-source software: the company hands over the actual model file, no login required, no corporate terms of service standing between you and the machine. Last month more than a hundred companies, including Microsoft, Nvidia, Intel, Amazon, Meta, and Hugging Face, signed a letter defending that approach. The pitch is simple. If you can download the weights, the AI is genuinely yours, not something OpenAI or Anthropic lets you borrow until they decide you've broken a rule.

The obvious problem is that the same download works for anyone, including people who want to build bombs, write malware, or worse. Recent models have gotten scarily good at hacking, good enough that some critics now argue open weights should simply be banned. It doesn't help that the leading open models increasingly come out of China, which turns a technical debate into a geopolitical one. Defenders respond with a version of the old gun-rights slogan: outlaw open weights and only outlaws will have them, while everyone else loses the ability to build defenses.

What's strange is who isn't fighting this fight. No government agency has stepped up to lead the opposition, though parts of the Trump administration grumble about China on hawkish grounds without proposing an outright ban. Anthropic skipped the pro-open-weights letter entirely, offering only a vague line about supporting open models 'that don't have dangerous capabilities' — notable, since most in the industry expect open models to cross that line within a year. Meanwhile the loose network of AI-safety researchers and effective altruists that gets blamed for pushing restrictions has mostly stayed quiet. Nobody's actually running the ban-open-weights campaign; it exists mostly as a suspicion.

The deeper argument here is about timing, not principle. Closed models tend to sit about six months ahead of the best open ones, a gap that's held steady for years. That buffer matters if the real fear is a rogue superintelligence, since it gives aligned systems time to spot trouble before an unaligned open model catches up. But criminal misuse doesn't need a six-month head start to cause damage — some hacker or bioterrorist could hurt real people well before that structural advantage means anything.

History suggests societies don't act on this kind of risk until something actually blows up. Nobody hardened airport security before 9/11, and nobody built pandemic stockpiles before COVID, but both triggered massive overcorrections afterward. Bioterrorism attempts, according to the historical record, kill a median of zero people, which is grim comfort but comfort nonetheless. The likely path isn't civilizational collapse; it's a bad hack or a small-scale attack, followed by lawsuits, headlines, and a sudden, furious clampdown that nobody bothered to organize in advance.