As a provider of monitoring and takedown services, IP Twins regularly observes fraudulent domains and their application in phishing campaigns. These fraud campaigns increasingly bear the stamp of AI:
- The domain name clearly targets the brand, but its composition is unexpected.
- Phishing emails are nearly indistinguishable from routine, targeted emails
- Copies of brand websites go live quickly and evade detection
Most readers have a passing ability to detect when content appears to be AI-generated, which they can apply in reviewing fraudulent emails and websites. This article will go beyond this “smell test” and summarize recent security research to explain how fraud actors are using AI to mount more effective phishing campaigns.
Phantom Squatting
In this scenario, the AI model “hallucinates” a non-existent domain, then the attacker registers the domain name to capture and misuse the traffic.
Palo Alto Networks[1] found that there are different types of prompts, each used by employees in their daily work, which can produce hallucinated domain names:
- Software created with the help of AI coding assistants
- Guides created with AI research tools
- Autonomous AI agents attempting to complete tasks
In each instance, the AI model produced a domain name that could plausibly host a given resource but did not actually exist. The study gave some examples of domains it had detected, which generally were under the .com extension and used hyphens. AI hallucinations appear less inclined to use cheap domain extensions and misspellings, which have long been hallmarks of abusive domains.
The software or guides containing non-existent domain names do not need to be publicly available for the attackers to detect and exploit them. Both the Palo Alto Networks study and previous work on the same topic[2] find that the hallucinated domain names can be predicted by observing the behavior of the AI model.
The sample of hallucinated domain names tended to include the brand name, so this type of attack can be detected with domain name monitoring. The probing of the AI models used by attackers can also be replicated by security professionals, presenting an opportunity for the company to proactively register domains likely to be hallucinated.
Improved Phishing Emails
Employees are taught to be suspicious of emails with poor grammar and awkward phasing, as these are signs of phishing. Even with the help of automated translation, it was hard for the fraud actor to create natural emails in a language different from their own.
AI programs solve that problem, thereby removing one line of defense from the recipient of the phishing email. However, improving the writing quality is not the only way that AI is helping craft more credible phishing emails.
A joint study by researchers from Harvard and the Avant Research Group found that AI can also be used effectively in the target reconnaissance phase[3]. In other words, with minimal input, an AI program can gather information on a target’s field, workplace superiors and other personal details that can be incorporated into the phishing email.
The same study tested phishing emails written by a human expert against emails written entirely by AI, finding that the AI-generated phishing emails were equally as effective.
On an optimistic note, a separate component of this study analyzed the ability of AI models to detect phishing emails and found that, with the right prompts, their detection rate was very high. The increase in quality of phishing emails created by AI can be mitigated by email security that also incorporates AI-powered review,
Website Clones
The final step in harvesting the victim’s personal data and/or payment credentials resides on a website. Same as poor writing, the poor site design gives a clue to the visitor that they’re probably not on the brand’s legitimate website.
AI website builders create pages based on the user’s prompt. Case studies have shown that they can be exploited by sending prompts to create copies of existing websites. Cybersecurity firm Malware Bytes inspected the code of a clone of its own website and found evidence it had been created by Vercel’s v0.dev AI-powered platform[4]. Another firm also examined Lovable[5] (lovable.dev) and found its AI model especially cooperative in creating clones of existing brand websites.
The use of mainstream AI website creators have another aspect that is conducive to phishing: Links to websites hosted on the infrastructure of Vercel, Lovable do not trigger any red flags, as they are generally trustworthy hosts. Links included in phishing emails can be scanned to reveal the content of the destination website, but that can be evaded by putting a captcha or “verify you are human” challenge before the cloned website.[6]
Conclusion
Fraud actors have figured out how to use AI to improve their phishing campaigns. The new techniques are especially damaging during the current period of AI adoption:
- The hallucination problem in AI models remains unsolved
- Users aren’t expecting such high-quality phishing emails and websites
- AI models aren’t refusing to work on ill-intentioned prompts.
Some of this will be sorted out in time, as the companies and services within the AI economy mature and regulation catches up with the technology. The other part will need to be addressed by IT Security professionals and companies like IP Twins that assist brands in detecting and neutralizing domains and web resources used for fraud.
Browsers and email clients will improve to better detect impersonations that are good enough to deceive the human eye; The studies cited above suggest this can be accomplished by incorporating AI review. AI can also be leveraged in Domain and Web Monitoring Services to detect live attacks, as well as to recommend defensive domain name registrations to prevent attacks before they can materialize.
As attack archetypes evolve, so too do the tools IP Twins develops for our clients. We recognize that AI problems often require AI-based solutions and are rapidly incorporating this technology into our existing suite of services.
About IP Twins
IP Twins helps companies protect their brands against online abuse worldwide. Combining advanced monitoring technologies with legal expertise, our teams detect fraudulent domain names, phishing websites, counterfeit offers and other forms of digital brand abuse. From early detection to takedown and dispute resolution, we help organizations reduce cyber risks and safeguard the trust of their customers.
Notes
[1] Phantom Squatting: AI-Hallucinated Domains as a Software Supply Chain Vector
[2] The Rise of Slopsquatting: How AI Hallucinations Are Fueling…
[3] Evaluating Large Language Models’ Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects
[4] Criminals are using AI website builders to clone major brands | Malwarebytes
[5] “VibeScamming” — From Prompt to Phish: Benchmarking Popular AI Agents’ Resistance to the Dark Side
[6] How AI-Native Development Platforms Enable Fake Captcha Pages | Trend Micro (US)