AI-Powered Phishing and Deepfake
Social Engineering:
The Attack Surface Nobody Prepared For.

The Cost Collapse

Traditional spear phishing required research time — finding the target's colleagues, communication style, current projects, and writing convincing context-specific content. A good spear phishing campaign targeting executives might take a skilled attacker several hours per target. The success rate was higher than generic phishing, but the volume was limited by the labour cost.

LLMs collapsed this cost curve. A prompt that includes the target's LinkedIn, recent company announcements, and the names of their direct reports produces a personalised, grammatically perfect phishing email in seconds. The same infrastructure can be run at scale against thousands of targets simultaneously. What was a boutique, high-effort attack class is now industrial.

The economics: a generic phishing email costs fractions of a penny to generate. An LLM-personalised spear phishing email costs maybe a cent. A voice-cloned phone call to a finance team member takes five minutes of setup and costs a few cents per minute in API calls. The return on investment for attackers, measured against the potential payoffs of wire fraud and credential theft, is now essentially unlimited.

LLM-Generated Spear Phishing

The effective phishing email has three components that LLMs excel at: personalisation (referencing specific projects, colleagues, or recent news), plausibility (context-appropriate reason for the request), and urgency (time pressure that suppresses critical thinking). All three can be generated from publicly available information.

Research published in 2024 compared human-written and LLM-generated phishing emails in controlled corporate settings. LLM-generated emails achieved click-through rates of 54% versus 12% for generic templates — comparable to the best human-crafted spear phishing. The LLM emails were also harder to flag in manual review because they lacked the grammatical tells that filter training relies on.

More sophisticated attacks use LLMs to generate multi-turn phishing campaigns — initial benign contact to establish rapport, followed by the malicious request in a subsequent message where the conversation history provides trust. The first message is often a legitimate-looking question about a business topic. The second message references the previous exchange and delivers the hook.

Voice Cloning Attacks

Modern voice cloning models can produce convincing replicas from as little as three to five seconds of clean audio. Public sources — earnings calls, conference presentations, YouTube interviews, podcasts — provide the training data. The output is real-time synthesised speech that retains the target's accent, cadence, and vocal character.

The vishing (voice phishing) variant: an employee receives a call that sounds like their direct manager, asking them to process an urgent wire transfer, share their MFA code, or provide VPN credentials. The call comes from a spoofed number showing a legitimate internal extension. The employee has heard this voice on calls every week. The social pressure of an urgent request from a manager bypasses their scepticism.

The defence against voice cloning attacks is not better detection — it is process. Any request for a financial transaction or credential sharing received by voice must be verified through a separate, pre-established channel regardless of how convincing the call sounds.

Deepfake Video in Business Email Compromise

Real-time deepfake video has become commercially available through several services and open-source projects. The quality varies — current models handle frontal face views well but struggle with rapid head movements and specific lighting conditions. However, the quality threshold for social engineering is lower than for entertainment: the target is not looking for visual artifacts, they are focused on the conversation content.

The attack pattern: schedule a video call with a finance team member, impersonating an executive or external auditor. Use deepfake face replacement software in real time. The meeting is brief — long enough to establish the instruction but not long enough for the victim to study the video carefully. Follow up with a spoofed email confirming the wire transfer amount and account details.

Indicators that suggest deepfake video: slight lag between lip movements and audio, unnatural blinking patterns, artifacts at face boundaries during head rotation, and resistance to requests to look sideways or make sudden movements. Asking the caller to hold up their hand in front of their face is a simple test — current deepfake models struggle with foreground occlusion.

Detection

Automated deepfake detection tools exist and improve constantly, but so do generation models. It is an arms race with no foreseeable technical winner. Current detection approaches:

• Audio watermarking: Embed imperceptible watermarks in all official communications from executives. Detection tools verify the watermark's presence before acting on a request.
• C2PA (Coalition for Content Provenance and Authenticity): Cryptographically sign media at the point of capture. Verification confirms the media came from a specific device and has not been altered. Requires adopting hardware and software that creates these attestations.
• Deepfake detection APIs: Microsoft, AWS, and several specialist vendors offer APIs that analyse video and audio for generation artifacts. False negative rates on current best-in-class models are still unacceptably high for use as a sole control.

Organisational Defenses That Work

1. Out-of-band verification for all high-stakes requests: Any request for wire transfer, credential sharing, or access change received via phone, video, or email must be verified by calling a pre-stored, verified phone number. Not a number provided in the message or call. This single control defeats almost every vishing and deepfake BEC attack.
2. Verbal code words for high-risk staff: Establish a unique word or phrase between senior staff and their likely targets (PA, finance team). The code word must be included in any legitimate urgent request. An attacker who does not know the code word cannot complete the attack.
3. DMARC, DKIM, and SPF enforcement: Strong email authentication stops the spoofed sender addresses that accompany voice and video attacks in the follow-up email. Without the email spoofing, attackers must use look-alike domains — which are easier to spot.
4. Wire transfer approval workflows: Require multiple approvals for transactions above threshold amounts. A single point of social engineering cannot authorise a transfer when a second approver must log into a separate system and confirm.
5. Security awareness training updated for AI attacks: Traditional phishing training tells employees to look for grammar errors and urgency. Those signals no longer work. Training must teach the process controls (out-of-band verification) rather than content spotting.