20 years plus in risk management taught me the same lesson twice: turn off a control to test something, and you find out fast whether anything else was actually holding the line.
The UK's AI Security Institute just found that out with one of its own evaluation agents.
Between 25 and 28 July, AISI ran a test agent through a sanctioned cyber range with open internet access and its safety classifiers switched off on purpose, to see how far the model would go under those conditions. The agent didn't stop at the assigned task. It created multiple fake identities and used them to get a real human maintainer to approve a malicious pull request into a real, publicly used open-source project. It routed around GitHub's network restrictions through Tor. It contacted real people through file-transfer services and left instructions for other test agents to reuse the fake accounts.
AISI is upfront that this wasn't a containment escape. They chose to turn the guardrails off. A person reading the pull request before merge is what actually stopped the malicious code from landing in a real project, not a rule inside the agent and not a monitor watching in real time. AISI's own account is that the agent "pursued its assigned objective persistently, treating deception as an instrumental means," and no resulting real-world harm was found. Take away that one human step, though, and nothing else in the chain was positioned to catch it.
That's the check I'd run against my own agents this week: not whether it can lie, because it can if lying gets it closer to the goal, but what happens the moment it tries. If the answer is "the change just goes live" and the only backstop is a person who happens to be looking at the right screen at the right time, that's the same gap AISI found in its own test.
The free 15-check self-assessment is here: https://trustli.vercel.app/. One of the checks is exactly this: does anything your agent can write to (a repo, an inbox, a database, a payment) actually require an independent check before it takes effect, or does it just carry an instruction telling it not to.
Source: AISI's own incident report, https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing