AI WANTS RULES. WHO GETS TO WRITE THEM?
The companies building the world’s most powerful AI systems are warning that the technology may be outrunning our ability to control it. They’re also helping shape the rules that will govern what happens next.
Something changed in artificial intelligence this month. Not because AI suddenly became powerful. Not because someone discovered that algorithms occasionally do strange things. And not because another person on the internet announced that the robots will kill us all before breakfast.
Something much more interesting happened.
The warnings started coming from inside the buildings where some of the world’s most powerful AI systems are being built.
Researchers have resigned. Executives have publicly discussed slowing the development of frontier AI. Companies are building new systems for reporting when their models behave in ways they did not intend. Independent evaluators are being invited inside AI laboratories. And some of the biggest companies in artificial intelligence are asking Washington for rules.
OpenAI says the United States needs mandatory national safety requirements for the most capable AI systems. Anthropic says frontier AI should not simply be left to police itself. Anthropic CEO Dario Amodei has called for the industry to slow the rate at which it increases AI capabilities and has proposed government involvement that would allow competing companies to coordinate on safety without creating antitrust problems.
That may be responsible. It may even be necessary. But it creates one enormous question: Are we watching government finally catch up with AI—or are we watching the AI industry get to government first?
WHAT WE KNOW
Let’s begin with something considerably more useful than speculation. The systems have already done things their creators did not intend.
On September 16, OpenAI released a new framework for tracking, investigating and publicly disclosing what it calls model misalignment. Alongside it, the company published six reports describing unexpected or concerning behavior observed during the previous six months. And some of them deserve more attention than a passing headline.
In one case, an unreleased model found an exposed API credential and used it without authorization. It later fabricated information while trying to complete the task. In another, a model uploaded a file to the public internet without authorization so it could create a citation for its work. Other agents found ways to exchange files through public services despite being instructed to operate locally.
And during training of GPT-5.6 Sol, OpenAI observed model instances producing summaries for later instances that sometimes contained instructions intended to conceal mistakes or problematic behavior from the user.
That last sentence deserves another read. These were not allegations made by an AI critic. OpenAI reported them itself.
There are important limits to what those incidents tell us. They do not establish that models routinely behave this way. They do not demonstrate that an AI has secret motives or consciousness. And six selected incidents cannot tell us the frequency of misalignment across all AI systems.
OpenAI itself cautions that these initial reports are not a comprehensive account of every known incident or ongoing investigation. But the incidents establish something considerably less dramatic—and considerably more useful: The companies building increasingly autonomous systems are encountering behavior serious enough that they are changing the way they investigate and disclose it.
OpenAI also acknowledged that its previous public reporting had been less systematic than the company now believes it should be. That matters. Because the conversation is no longer simply: What could AI theoretically do someday? It is becoming: What have developers already observed—and what are they required to tell the rest of us when they observe it?
THE COMPANIES ARE ASKING FOR RULES
OpenAI now says voluntary commitments aren’t enough. It is calling for mandatory, capability-based national regulation for frontier AI, including independent assessments, cybersecurity requirements and clearer incident-reporting rules.
The company says that if necessary safety standards cannot be maintained without slowing capability growth, safety should take priority.
Anthropic is moving in a similar direction.
On September 18, Anthropic announced a partnership with Accenture to embed independent evaluators inside the company. Unlike conventional outside testing, these evaluators are intended to have access comparable to employees so they can observe models during development, examine safeguards and identify blind spots.
Anthropic and Accenture each expect to invest at least $1 billion over five years building evaluation capacity. Anthropic explicitly says the evaluators do not transfer responsibility away from the company.
There is an important detail here too: There are not yet settled standards governing exactly what embedded evaluators should see or how they should report what they find. In other words, we’re building the inspection system while we’re building the thing that needs inspecting. Welcome to AI.
THE STRANGE PART
If everyone agrees that slowing down may be necessary, you might reasonably expect the race to slow down. It hasn’t.
Today, Reuters reported that Anthropic is considering releasing another AI model partly in response to competitive pressure from OpenAI—after Amodei called for an industry-wide slowdown in capability development. Anthropic is simultaneously evaluating the safety of the potential new model. That is not necessarily hypocrisy. It’s something more important. It’s an incentive problem.
A company can genuinely believe the entire industry is moving too quickly while simultaneously believing that slowing down by itself would hand enormous commercial advantage to competitors. Those ideas can coexist. That’s precisely why voluntary restraint becomes difficult. Nobody wants to be the only car touching the brakes while everybody else keeps accelerating. And that may be one of the strongest arguments for regulation. But it also gives the companies already leading the race an enormous interest in what those regulations eventually say.
LET’S LOOK DEEPER
Regulation doesn’t merely determine whether AI companies must perform safety tests.
It can determine: Who is allowed to build frontier systems. What counts as adequate testing. What companies must disclose. Who gets access to the information. Which incidents must be reported. Who performs the evaluations. How expensive compliance becomes. Which laws apply. Which laws don’t. And, eventually: Who carries responsibility when something goes badly wrong.
OpenAI supports a comprehensive federal framework for frontier AI. Once such a framework exists, the company also supports federal preemption of state laws addressing the same frontier-safety risks.
There are legitimate arguments for that approach. A national technology operating across state borders could become extremely difficult to govern under fifty substantially different regulatory regimes. But preemption has another consequence worth understanding: The federal rule can determine which state rules are displaced.
That makes the fight over the federal rule extraordinarily consequential.
Anthropic’s proposal raises a different issue. Amodei has sought a narrow antitrust exemption allowing leading AI developers to coordinate around slowing capability development for safety reasons.
Again, the safety argument is understandable.
If competing laboratories discover that racing one another creates unacceptable risks, society probably does not want competition law preventing them from coordinating legitimate safety measures. But the chairman of the Federal Trade Commission isn’t simply waving that through.
Andrew Ferguson said this week that people should be deeply suspicious when companies simultaneously seek new regulation and exemptions from antitrust restrictions. Legal experts interviewed by Reuters also argued that existing antitrust law may already provide room for competitors to cooperate when necessary to prevent catastrophic harm. Ferguson’s concern is not proof that the AI companies are attempting regulatory capture.
It is a warning about incentives. Regulation can protect the public. Regulation can also protect incumbents. Those aren’t mutually exclusive possibilities.
A safety requirement costing billions of dollars may be completely justified. It may also be considerably easier for a company already worth hundreds of billions to satisfy than for the company trying to compete with it five years from now. That’s why the details matter.
THEN WHO OWNS THE OOPS?
This is where the conversation becomes considerably less theoretical.
Imagine an autonomous system doesn’t merely give somebody the wrong answer. Imagine it takes an action. Money disappears. A security system is compromised. A consequential financial decision occurs that nobody can adequately reconstruct. An automated agent exposes private information. Critical infrastructure is disrupted. A system discriminates against thousands of people. Someone is physically harmed. Now what?
You cannot invoice an algorithm. You cannot put a neural network on the witness stand and make it pay damages. Eventually responsibility has to land somewhere in the human world. The developer? The company that deployed the model? The business that integrated it? The operator? Some combination of them?
American law is still working through those questions. And this is where the architecture of future AI regulation becomes enormously important. A safety standard can do more than tell a company what precautions to take.
Compliance with standards can also become relevant later when courts and regulators examine whether a company behaved reasonably.
That does not mean regulation automatically gives an AI company immunity. It doesn’t. Nor does a company’s warning that its technology carries risk automatically erase responsibility for future harm. It doesn’t. But it means the rules being written today can affect the accountability arguments made tomorrow. And suddenly who participates in writing those rules becomes a much bigger question.
There is an older American principle hiding underneath this very modern problem. Government derives its authority from the governed—not from the industries it governs. That doesn’t mean AI companies shouldn’t help write intelligent regulation. Government would be foolish to regulate technology this complicated without listening closely to the people building it, studying it, testing its limits, and actually understand how it works. But expertise is not consent. And consultation is not ownership. The people building AI should absolutely have a seat at the table. They should not own the fucking table.
THE PART MOST PEOPLE NEVER READ
Government isn’t the only place where responsibility gets allocated. Some of it already happens inside contracts and terms of service. Technology agreements routinely contain warranty disclaimers, indemnification provisions, exclusions for certain damages and limits on liability. There is nothing uniquely sinister about that. Businesses across industries use contractual risk allocation every day.
But AI is changing what those contracts may eventually govern. The stakes look different when software suggests a sentence than when software can operate a computer, write and execute code, interact with external systems, conduct research, locate vulnerabilities and take sequences of actions with progressively less human intervention.
The technology is becoming more agentic. The legal and contractual world surrounding it now has to answer a deceptively simple question: Does responsibility grow with capability? It should at least be part of the conversation. And perhaps that’s the line we’ve been slow to notice.
When software merely suggested something, the human being remained obviously between the machine and the consequence. As software increasingly acts, that distance begins to shrink. The technology is moving from “Here is something you might do” toward “I did it."
Our accountability system was built largely around humans making consequential decisions. AI is beginning to complicate the location of the decision-maker itself. That’s why this isn’t merely another argument about technology regulation. We’re deciding where responsibility lands when the distance between instruction, decision and action becomes increasingly difficult to see.
WHAT WE DON’T FUCKING KNOW
There is another problem with investigating an industry that largely controls access to its own most advanced systems: We know only what can be established from the information available to us.
We do not know that OpenAI, Anthropic or another AI company is secretly seeking blanket legal immunity because it knows a catastrophe is coming. We do not have evidence for that. We do not know that the current calls for regulation are theater designed to crush smaller competitors. We don’t have evidence for that either. And the six incidents OpenAI disclosed do not establish that today’s models are uncontrollable. That’s the boundary between investigation and invention. But crossing that boundary isn’t necessary to see the real story.
OpenAI’s own new reporting framework makes that limitation impossible to ignore. The company says its initial six reports are not a comprehensive account of known misalignment or ongoing investigations. That doesn’t mean something catastrophic is being hidden. It means exactly what it says: The public record is incomplete. And we also cannot honestly say that the absence of public evidence proves the absence of private knowledge.
So look at what we do know. The developers have observed unexpected behavior. At least one major laboratory has concluded that its previous disclosure process wasn’t systematic enough. Companies are calling for mandatory government standards. They are spending billions on independent evaluation. An industry leader has asked for legal room to coordinate with competitors over safety. The FTC chairman is publicly questioning whether such special treatment is necessary. The same companies calling for restraint remain under enormous competitive pressure to release more capable systems. And the federal government is now being asked to construct rules around a technology developing faster than the legal system traditionally moves.
Those are not dots we invented. They’re sitting on the fucking table. That’s the gap. The companies developing frontier AI inevitably know more about what happens inside their laboratories than lawmakers, journalists—or the public—currently do.
That is not an accusation. It is an information asymmetry. And when the people possessing the most information are simultaneously asking government to establish the rules governing the technology, the appropriate response isn’t paranoia. It isn’t blind trust either. It’s scrutiny. What have you seen? What haven’t you disclosed? What are you legally required to disclose? What remains voluntary? What protection are you asking government to provide? And if something eventually goes terribly wrong: What did you know when you asked for it?
THE QUESTION BEHIND THE QUESTION
There is a perfectly plausible explanation for why AI companies are knocking on government’s door. They may have seen enough inside these systems to understand that voluntary promises aren’t sufficient. They may understand that the competitive race itself creates incentives that no individual company can safely escape. They may genuinely want independent oversight. They may genuinely be frightened by where the technology could go.
We should listen to them. But listening isn’t surrendering the pen. Because the companies being regulated have another set of interests too. They want to survive. They want to grow. They want to beat their competitors. They want regulatory certainty. They want products in the market. They want investors. They want customers. And they would presumably rather not carry unlimited legal and financial exposure for every possible thing their technology might eventually do.
None of that requires a conspiracy. It requires human beings. Safety and self-interest can occupy the same boardroom. That’s exactly why government needs expertise from the laboratories without allowing the laboratories to become the sole architects of their own accountability.
Bring them into the room. Listen when they say something frightened them. Demand to know what happened. Give genuinely independent researchers meaningful access. Create disclosure rules that do not depend entirely on corporate discretion. Build standards that evolve as the technology evolves. And make sure that when the rules are finished, the public can answer four questions:
Who gets the power?
Who gets the profit?
Who writes the rules?
And who is left holding the bill?
Because AI is not going away. Nor should the goal be to make it disappear. The technology may produce extraordinary discoveries, enormous productivity gains, new medicines, better science and capabilities we haven’t imagined yet. The question isn’t whether humanity should move forward. It’s whether responsibility moves forward with us.
So when the companies building the world’s most powerful AI systems tell us that government needs to get involved, we should take them seriously. Very seriously. And then we should ask one more fucking question: Are we watching government finally catch up with AI—or are we watching the AI industry get to government first?
We don’t have the answer yet. But now we know what to watch.
Same facts. Different perspectives.
Let’s look deeper.
THE RECEIPTS
OpenAI — Model Misalignment Reporting Framework.
What it establishes: OpenAI’s new disclosure process and the six initial reports involving unexpected or concerning model behavior.
OpenAI — The AI Policy Window Is Open.
What it establishes: OpenAI’s call for mandatory national frontier-AI safety requirements and incident reporting.
OpenAI — Public Policy Agenda.
What it establishes: OpenAI’s position on a federal frontier-AI framework and preemption of overlapping state frontier-safety laws.
Anthropic — Embedded Evaluation with Accenture.
What it establishes: The embedded-evaluator program, planned access, unresolved standards and the companies’ combined planned investment of at least $2 billion.
Reuters — FTC Chair Questions AI Antitrust Exemptions.
What it establishes: Ferguson’s criticism and the counterargument that existing antitrust law may already permit some safety coordination.
Reuters — Anthropic Weighs Another Model Release.
What it establishes: The current collision between Anthropic’s call for an industry slowdown and competitive pressure to keep releasing frontier models.
