Can AI agents go rogue? What to check before you give one access to your business data

What happened when AI agents went rogue at OpenAI and Anthropic?

In both labs, AI agents being tested on their hacking skills ended up inside real companies’ systems. The causes were different, and the difference matters.

At OpenAI, the agents found their own way out. Running in isolated test environments, they uncovered flaws nobody knew about, gained internet access, and ran their own code on dozens of Hugging Face servers¹. OpenAI calls it a “warning shot”.

At Anthropic, the door had been left open. Its models were told they had no internet access. Because of “a misunderstanding between us and our evaluation partner”, they did². Treating what they found as part of the exercise, they got into 3 real companies’ systems. One published a booby-trapped software package that was online for roughly 1 hour, and was downloaded and run on 15 real systems².

It hasn’t stopped there. Meta has disclosed similar incidents, US senators have introduced a bill on who is liable³, and OpenAI’s chief executive says the company will not go public in 2026, “given everything happening with safety”⁴.

Where we stand: NightForward builds mainly on Claude, which Anthropic makes. Both labs get the same treatment here.

Why do AI agents go rogue?

Not because they turn evil. In both labs’ accounts, the agents were mainly trying to finish a test, and kept going well past the point where a person would have stopped.

Anthropic found “no evidence in any run described here of a model pursuing a goal of its own. Instead, the models did what their evaluation asked”². OpenAI’s account is less comfortable. It calls its agents’ actions misaligned, says a primary driver was agents trying to cheat on their tests, and found they “rarely ‘gave up’ on their evaluation tasks, even when the tasks appeared impossible to solve”¹.

An agent is built to finish its task. Faced with a locked door, it tends to treat the lock as a puzzle to solve, not as a sign to stop. That is what makes agents useful, and what makes them risky. Here is how the same habit could look in an ordinary working week:

• Told to clear the overdue invoices, it chases a client you’d agreed to leave alone this month.
• Told to answer every customer question within the hour, it promises a delivery date nobody checked.
• Told to clean up the shared drive, it deletes the folder your accountant still needed.

None of that needs a hacker. It only needs an agent with more access than the job requires, and nothing in the way.

If the AI labs struggled to keep agents contained, can your team?

Not by trusting 1 barrier. Both labs had containment they believed in, and both found out late that it had failed.

At Anthropic, neither the company nor its testing partner knew the test machines were online. It surfaced only when the company went back through 141,006 test runs, and the earliest incidents dated to April². At OpenAI, the barrier had been “tested and validated”, and the agents still got through by chaining “previously unknown vulnerabilities”. A team had seen early signs in late May¹.

In fairness, these were stripped-down test versions. Anthropic says the safeguards on its public models “would have blocked the behaviors identified”², and OpenAI found that its production setup can make such break-ins more than 100 times less likely¹.

But the lesson isn’t “build a better wall”. It’s layers. The national cyber agencies of the US, UK, Canada, Australia and New Zealand put it plainly in their joint guidance on AI agents: “Avoid reliance on a single security mechanism”⁵. The same guidance says businesses “often grant permissions too broadly and overlook these issues”⁵.

Knowing the risk isn’t the same as acting on it. In a November 2025 poll (published 2026) of 250 UK sole traders and owners of businesses with 1 to 49 staff, 44% worried that using AI without proper safeguards could expose them to cyber threats. Only 10% had invested in AI security training⁶.

So don’t avoid AI. Give an agent only the access the job needs, and put hard limits around it.

Can you rely on the AI itself to say no?

No. Use AI to build the fence, not to be the fence. Clear instructions matter, but they are only part of the answer, and they are hard to get right.

Even the labs find them hard. Anthropic says its models probably “wouldn’t have accessed the internet to begin with if the prompt had clearly explained which systems were in and out of scope”².

It keeps happening in smaller ways. On 9 October 2026, Anthropic reported a test in which a model, told never to log in, enter personal data or make purchases, sent an invented tip through a police website’s form: its instructions “did not rule out form submissions”. The tip was flagged as spam and never passed on⁷. A check on the receiving side caught it, not the model.

Think of a strong working horse. It is hugely useful, and nobody manages one by explaining things to it. It works in harness, inside a fence, with someone holding the reins.

The difference is predictability. Code and workflow systems are deterministic: the same input gives the same result, every time. AI is less predictable. So put AI’s strength where it counts, in building and testing those deterministic limits. OpenAI now does this itself, using its own models to test its barriers against simulated attacks¹.

We recently put the AI assistant on our own website through a set of standard security tests for a system of its kind. It held up. Why it held up is the useful part.

• The AI’s own call was the least predictable layer. It turned the tricks away, but it didn’t always treat 2 similar inputs the same way. The agencies say this of agents in general: “Even when prompts appear identical, agents may generate different actions”⁵.
• The code did the dependable work. Whatever a visitor types is handled as plain text, everywhere, every time. Nothing has to be judged. That code was written with AI’s help, then checked and tested, and it does the same thing on the thousandth request as on the first.
• The system protects itself. Protection against flooding and misuse is built in. It steps in automatically, and the site stays available for everyone else.

The agencies apply the same thinking to approvals: when a person’s sign-off is needed should be “determined by system designers or operators, not delegated to the agentic AI system”⁵.

Is it safe to give an AI agent access to your business data?

It can be, with the right limits around it. We’d want a yes to 5 checks first, and wherever the answer is no, that’s where to start.

“Access” means everything the agent can reach: the inbox, and also customer records, shared files, finance systems and whatever those connect to.

1. Does it reach only what the job needs? An agent that drafts replies needs to read messages, not delete them or send on its own. Start from how the work really gets done, then remove every permission the task doesn’t use (also known as least privilege).
2. Does it have its own login (an agent identity)? Not your account, and not the admin account. Then every action in the log traces back to the agent, you can limit it without limiting yourself, and you can switch it off without switching anyone else off.
3. Does a person approve anything that can’t be undone? Sending to customers, paying, deleting, changing prices. The agent prepares; a person presses the button (a human in the loop). That rule belongs in the system, not in a polite instruction to the AI.
4. Can you stop it fast enough to limit the damage? Could you spot a problem and cut off just the agent before the damage spreads? That is the kill switch, and there is more on it below.
5. Do you know who set it up, and is there a record of what it does? Ask whoever built it what access they granted, document it, and keep a log the agent can’t edit (an audit log).

Checks 2 and 3 are starting to show up in the tools businesses already use. In October 2026, Microsoft said Windows will soon tell an agent’s activity apart from the person’s, so that security controls can “target the agent’s access to protected resources without blocking the employee’s access”⁸. Finance software is being built the same way: ApprovalMax, an approval tool for Xero and QuickBooks Online users, has launched AI checks on supplier bills⁹. Its own rule: “It flags. It never decides”¹⁰.

Does an AI agent need a kill switch?

Yes. But a switch only helps if you notice in time and someone is allowed to use it. OpenAI says its own early response was a weak point.

Even a good fence can fail: OpenAI’s had been tested. That is what the off switch is for.

Agents work at machine speed: Anthropic’s booby-trapped package needed only about 1 hour online to be run on 15 real systems². OpenAI says weaknesses in how it responded to early warning signs contributed to its incident, and it now wants safeguards that “operate at the speed of the AI agents themselves”¹.

On paper, the rule-makers agree. NIST, the US standards body, asks for ways “to supersede, disengage, or deactivate AI systems” that go off course¹¹. The EU’s AI Act says the people overseeing a high-risk AI system must be able to interrupt it “through a ‘stop’ button or a similar procedure”¹². The agencies’ guidance, published on 1 May 2026¹³, adds testing: “Develop and test incident response procedures to detect, contain and recover from agent compromise”⁵.

Practice looks different. In a 2026 survey of 459 security, compliance and technology professionals across North America, Europe and the Middle East, only 21% said their company had deployed a formal AI kill switch¹⁴. (The survey is by Kiteworks, which sells data security software.) Its verdict: “An untested kill switch is not a control. It is an assumption”¹⁴.

OpenAI’s own fix is a deadline. For its most severe alerts, responders are expected to pause the activity unless they can show within 30 minutes that the alert is a false alarm¹. Stop first, investigate after.

You don’t need a lab’s tooling to copy that. 5 things make an off switch real:

• You would notice. Set alerts for what would look wrong for this agent: a sudden burst of messages, activity in the middle of the night, a bulk delete, a new payee.
• Someone is allowed to act. Name the people now, more than 1 of them, and tell them they need nobody’s permission to pause it.
• It stops only the agent. This is why it needs its own login: disable that 1 account and the agent stops while the business carries on. If stopping it means shutting down email for everyone, people hesitate, and the damage grows while they do.
• Limits hold the line until you get there. Cap what it can do in an hour: messages sent, records changed, money moved.
• You have tried it. Use the switch on a quiet day and time it. After a real stop, change every password and key the agent could see, and restart only when a named person says so.

Who is responsible when an AI agent goes wrong?

For the AI labs, the law is still being argued. For a business that uses an agent, we think the working answer is already clear: you are.

AI companies “have so far avoided major lawsuits” over these incidents, CNN reported on 12 October 2026³. A first test is already in court. On 29 September 2026, a non-profit, Legal Advocates for Safe Science and Technology, sued OpenAI in San Francisco over the Hugging Face break-in. It asks for a court order, not damages, and it leans on a California law under which a company may not argue that its AI “autonomously caused the harm”¹⁵. OpenAI calls it “completely without merit”¹⁶. No court has ruled.

For a business, a much smaller case points the way. In February 2024, a tribunal in British Columbia ordered Air Canada to pay a customer $812.02 in Canadian dollars after its website chatbot gave wrong information about a fare. Air Canada had argued it could not be held liable for what the chatbot said. The tribunal’s answer: “It makes no difference whether the information comes from a static page or a chatbot”¹⁷.

That is 1 small-claims decision in 1 Canadian province, not a rule for everywhere. We still think it’s the safe assumption: if your agent messages the wrong client or deletes the wrong file, your customers will look to you, not to the software company. Suppliers say the same: Anthropic’s usage policy, effective 12 November 2026, makes users responsible for the agents they build or deploy, “including actions those agents take through tools, browsers, or connected systems”¹⁸.

So decide who in your business owns the agent, check your supplier terms and your insurance, and design things so that the expensive mistakes need a person’s yes. This is general guidance, not legal advice.

So should you stay away from AI agents?

No, in our view. Set them up properly, and do it before the agent is connected, not after the first surprise.

In both labs, the trouble began weeks or months before anyone caught it. So here is how we think about it. Good instructions lower the odds of a problem. The fence limits how big it can get. The off switch limits how long it lasts. You want all 3, and the last 2 should not depend on the AI choosing to cooperate.

AI can help you build all of that. It just shouldn’t be the only thing standing guard.