We are not ready for this.
Republished on March 15th with additional security industry analysis on the new threat from semi-autonomous AI attacks.
Email users have been warned for some time that AI attacks and hacks will ramp up this year, becoming ever harder to detect. And while this will include frightening levels of deepfake sophistication, it will also enable more attackers to conduct more attacks, with AI operating largely independently, “carrying out attacks.” That has always been the nightmare scenario and it is suddenly coming true, putting millions of you at risk.
We know this, but seeing is believing. A new video and blog from Symantec has just shown how a new AI agent ot operator can be deployed to conduct a phishing attack. “Agents have more functionality and can actually perform tasks such as interacting with web pages. While an agent’s legitimate use case may be the automation of routine tasks, attackers could potentially leverage them to create infrastructure and mount attacks.”
The security team has warned of this before, that “while existing Large Language Model (LLM) AIs are already being put to use by attackers, they are largely passive and could only assist in performing tasks such as creating phishing materials or even writing code. At the time, we predicted that agents would eventually be added to LLM AIs and that they would become more powerful as a result, increasing the potential risk.”
Now there’s a proof of concept. It’s rudimentary but will quickly become more advanced. The sight of an AI agent hunting the internet and LinkedIn to find a target’s email address and then for website advice on crafting malicious scripts, before writing its own lure should put fear into all of us. There’s no limit to how far this will go.
“We’ve been monitoring usage of AI by attackers for a while now,” Symantec’s Dick O’Brien explained to me. “While we know they’re being used by some actors, we’ve been predicting that the advent of AI agents could be the moment that AI-assisted attacks start to pose a greater threat, because an agent isn’t passive, it can do things as opposed to generate text or code. Our goal was to see if an agent could carry out an an attack end to end with no intervention from us other than the initial prompt.”
As SlashNext’s J Stephen Kowski’s told me, “the rise of AI agents like Operator shows the dual nature of technology — tools built for productivity can be weaponized by determined attackers with minimal effort. This research highlights how AI systems can be manipulated through simple prompt engineering to bypass ethical guardrails and execute complex attack chains that gather intelligence, create malicious code, and deliver convincing social engineering lures.”
Even the inbuilt security is ludicrously lightweight. “Our first attempt failed quickly as Operator told us that it was unable to proceed as it involves sending unsolicited emails and potentially sensitive information,” Symantec said, with its POC showing how this was easily overcome. “This could violate privacy and security policies. However, tweaking the prompt to state that the target had authorized us to send emails bypassed this restriction, and Operator began performing the assigned tasks.”
The agent used was from OpenAI, but this will be a level playing field and it’s the nature of the capability that matters not the identity of the AI developer. Perhaps the most notable aspect of this attack is that when the Operator fails to find the target’s email address online, and so successfully deduces what the address would likely be from others within the same organization that it could find.
Black Duck’s Andrew Bolster warned me that “as AI-driven tools are given more capabilities via systems such as OpenAI Operator or Anthropic’s Computer Use, the challenge of ‘constraining’ LLMs comes into clearer focus,” adding that “examples like this demonstrate the trust-gap in underlying LLMs guardrails that supposedly prevent ‘bad’ behavior, whether established through reinforcement, system prompts, distillation or other methods; LLM’s can be ’tricked’ into bad behavior. In fact, one could consider this demonstration as a standard example of social engineering, rather than exploiting a vulnerability. The researchers simply put on a virtual hi-vis jacket and acted to the LLM like they were “supposed” to be there.”
“Agents such as Operator demonstrate both the potential of AI and some of the possible risks,” Symantec warns. “The technology is still in its infancy, and the malicious tasks it can perform are still relatively straightforward compared to what may be done by a skilled attacker. However, the pace of advancements in this field means it may not be long before agents become a lot more powerful. It is easy to imagine a scenario where an attacker could simply instruct one to ‘breach Acme Corp’ and the agent will determine the optimal steps before carrying them out.”
And that really is the nightmare scenario. “We were a little surprised that it actually worked for us on day one,” O’Brien told me, given it’s the first agent to launch.
Guy Feinberg from Oasis Security agrees, telling me “AI agents, like human employees, can be manipulated. Just as attackers use social engineering to trick people, they can prompt AI agents into taking malicious actions.”
“Organizations need to implement robust security controls that assume AI will be used against them,” warns Kowski. “The best defense combines advanced threat detection technologies that can identify behavioral anomalies with proactive security measures that limit what information is accessible to potential attackers in the first place.”
This week we have also seen a report into “Microsoft Copilot Spoofing” as a new “phishing vector,” with users not yet trained on how to detect these new attacks. That’s one of the reasons AI fueled attacks are much more likely to hit their targets. You can expect to see continuous reports as this new threat landscape shapes up.
One thing is already clear though — we are not yet ready for this.
