Presented at
DEF CON 33 (2025),
Aug. 10, 2025, 10 a.m.
(45 minutes).
Over the past two years, we have witnessed the emergence of a new class of attacks against LLM-powered systems known as Promptware.
Promptware refers to prompts (in the form of text, images, or audio samples) engineered to exploit LLMs at inference time to perform malicious activities within the application context.
While a growing body of research has already warned about a potential shift in the threat landscape posed to applications, Promptware has often been perceived as impractical and exotic due to the presumption that crafting such prompts requires specialized expertise in adversarial machine learning, a cluster of GPUs, and white-box access.
This talk will shatter this misconception forever.
In this talk, we introduce a new variant of Promptware called Targeted Promptware Attacks.
In these attacks, an attacker invites a victim to a Google Calendar meeting whose subject contains an indirect prompt injection.
By doing so, the attacker hijacks the application context, invokes its integrated agents, and exploits their permission to perform malicious activities.
We demonstrate 15 different exploitations of agent hijacking targeting the three most widely used Gemini for Workspace assistants: the web interface (www.gemini.google.com), the mobile application (Gemini for Mobile), and Google Assistant (which is powered by Gemini), which runs with OS permissions on Android devices.
We show that by sending a user an invitation for a meeting (or an email or sharing a Google Doc), attackers could hijack Gemini’s agents and exploit their tools to: Generate toxic content, perform spamming and phishing, delete a victim's calendar events, remotely control a victim's home appliances (connected windows, boiler, and lights), video stream a victim via Zoom, exfiltrate emails and calendar events, geolocate a victim, and launch a worm that tarets Gemini for Workspace clients.
Our demonstrations show that Promptware is capable to perform (1) inter-agent lateral movement (triggering malicious activity between different Gemini agents), and (2) inter-device lateral movement, escaping the boundaries of Gemini and leveraging applications installed on a victim's smartphone to perform malicious activities with physical outcomes (e.g., activating the boiler and lights or opening a window in a victim's apartment).
Finally, we assess the risk posed to end users using a dedicated threat analysis and risk assessment framework we developed.
Our findings indicate that 73% of the identified risks are classified as high-critical, requiring the deployment of immediate mitigations.
Presenters:
-
Or "oryair1999" Yair
Or Yair (@oryair1999) is a security research professional with seven years of experience, currently serving as the Security Research Team Lead at SafeBreach. His primary focus lies in vulnerabilities in the Windows operating system’s components, though his past work also included research of Linux kernel components and some Android components. Or's research is driven by innovation and a commitment to challenging conventional thinking. He enjoys contradicting assumptions and considers creativity as a key skill for research. Or frequently presents his vulnerability and security research discoveries internationally at top conferences he speaks at such as Black Hat, DEF CON, RSAC, SecTor, and many more.
-
Ben Nassi
Dr. Ben Nassi (https://www.linkedin.com/in/ben-nassi-phd-68a743115/) is a Black Hat board member (Asia and Europe), a cybersecurity expert, and a consultant. Ben specializes in AI security, side channel attacks, cyber-physical systems, and threat analysis and risk assessment. His work has been presented at top academic conferences, published in journals and Magazines, and covered by international media. Ben is a frequent speaker at Black Hat (6), RSAC (2), and DEFCON (3) events and won the 2023 Pwnie Award for the Best Crypto Attack for Video-based Cryptanalysis.
-
Stav Cohen
Stav Cohen is a Ph.D. student at the Technion – Israel Institute of Technology who investigates Cyber-Physical Systems (CPS) that integrate GenAI methodologies and feature Human-in-the-loop interactions, with a specific emphasis on their security and operational aspects. He conducts detailed analyses of GenAI models with the aim of identifying potential vulnerabilities and devising effective strategies to mitigate them. Additionally, he takes a proactive approach by exploring how GenAI methodologies can be utilized to improve both the security and operational efficiency of Cyber-Physical Systems.
Similar Presentations: