In the nascent field of AI hacking, indirect prompt injection has become a basic building block for inducing chatbots to exfiltrate sensitive data or perform other malicious actions. Developers of platforms such as Google’s Gemini and OpenAI’s ChatGPT are generally good at plugging these security holes, but hackers keep finding new ways to poke through them again and again.
On Monday, researcher Johann Rehberger demonstrated a new way to override prompt injection defenses Google developers have built into Gemini—specifically, defenses that restrict the invocation of Google Workspace or other sensitive tools when processing untrusted data such as incoming emails or shared documents. The result of Rehberger’s attack is the permanent planting of long-term memories that will be present in all future sessions, opening the potential for the chatbot to act on false information or instructions in perpetuity.
Incurable gullibility
More about the attack later. For now, here is a brief review of indirect prompt injections: Prompts in the context of large language models (LLMs) are instructions, provided either by the chatbot developers or by the person using the chatbot, to perform tasks, such as summarizing an email or drafting a reply. But what if this content contains a malicious instruction? It turns out that chatbots are so eager to follow instructions that they often take their orders from such content, even though there was never an intention for it to act as a prompt.
