There are failures that don't originate from technical flaws. They are decisions made with information already available, by people who could have acted and chose not to. That's the kind of failure we examine here.
In June 2025, OpenAI's team detected Van Rootselaar's account, flagged it, and suspended it after it described scenarios of violence involving weapons. Human staff internally debated whether to alert authorities. They decided it didn't meet their criteria. Seven months later, eight people died in an attack in Canada, five of them children. Someone saw the signal, processed it, and chose to sit on it.
In a previous article we covered how a group accessed Mythos, Anthropic's most heavily protected cybersecurity model, on the very day of its launch. Two cases, two leading companies. One failed to contain its most dangerous tool. The other withheld information about a real threat and chose not to share it. What emerges isn't a technical problem. It points to failures of governance.
What sets this incident apart is that it wasn't a mistake. OpenAI maintains internal criteria for deciding when to escalate an account to police reporting. Those criteria are designed by them, applied by them, and reviewed by them. There is no external regulatory standard, no independent audit. The organization that builds the tool defines the limits of its own responsibility when that tool helps plan violence. That choice isn't accidental. It's structural.
Sam Altman wrote that he deeply regretted not alerting authorities about the account suspended in June. He announced improvements to protocols with broader criteria. The apology acknowledges that the previous thresholds were insufficient. What it doesn't clarify is who will define the new criteria, who will oversee them, or what guarantees they'll be sufficient next time. The apology closes off the public conversation exactly where it should be widening.
I recognize these patterns in other contexts. Digital platforms promised for years that their internal moderation was enough. They knew about documented harm to teenagers and chose not to disclose it. Policies against incitement to violence were enforced unevenly. Self-regulation sustained the narrative until it failed, and by then the damage was already irreversible. OpenAI shows that this cycle continues — just now in a different industry.
A family from Tumbler Ridge is suing OpenAI. They allege the company had specific knowledge of the planning of a mass-casualty event and did not act. Regardless of whether the lawsuit succeeds, the case matters. If it wins, it creates legal precedent for a duty to report. If it loses, the current model holds: the company decides alone. Either outcome redefines the terrain for every platform that comes after.
Canadian authorities have said they are considering new regulations on artificial intelligence. The wording is telling. After eight deaths, with confirmation that warning signs existed beforehand and the choice was made not to act, the institutional response is still that they're looking into it. Technology advances in months. Regulatory processes operate in years. The current institutional framework responds to a different rhythm of change.
ChatGPT has surpassed four hundred million weekly active users. If OpenAI's internal criteria didn't trigger an alert in this case — with explicit debate and an account already suspended — it's reasonable to ask how many similar cases never even generated a discussion. This isn't rhetorical. It's a technical question whose answer remains inside OpenAI, inaccessible to regulators or outside researchers. That alone should settle the debate over whether self-regulation is viable.
The image evokes a perfected panopticon: structures that observe everything, but whose access and decisions remain in the hands of whoever built them. Bentham imagined a prison where the watchman saw without being seen. What we have now is more sophisticated: the watchman sees, debates, decides, and, at times, doesn't act until the damage is irreversible. Altman's statement is the most direct admission yet that this model has deep cracks. The promised fixes, however, remain under the same internal control.
I still don't have a clear answer for how to build effective accountability that doesn't stifle necessary innovation. I keep exploring these tensions and I recognize there are dimensions more complex than they appear. I've seen similar dynamics in organizations where critical information stayed locked inside internal criteria never audited externally.
At what point did we decide that the companies building the most powerful tools in recent history hold the exclusive right to define the limits of their own responsibility when those tools are used to cause harm?
