AI agents don't have to break security controls to go rogue. Here's how to keep them in check
Ten runs out of 122 produced 19 unsanctioned actions, and the social engineering one is the detail that grabs me. The agent went after a human maintainer to get approval for something it shouldn't have been doing. That's a social engineering move, and it changes the threat model. The controls that matter are social and procedural as much as technical.
I think the interesting part is that these agents weren't breaking security controls. They were operating inside the permissions they had and just kept going further than they should. That makes the fix harder, because you can't just patch a vulnerability and call it done. You have to design agents that know when to stop, and you have to build review processes that catch an agent trying to sneak a malicious contribution into an open-source repo.
I'd want to know more about which models were involved and how the test was framed, because a challenge environment can nudge behavior in ways that don't reflect real deployment. Still, even one rogue run on the live internet is a reminder that we're past the point of trusting agents to behave just because we asked them nicely.