Vision-language models, AI systems capable of understanding and reasoning across both images and text, have introduced a category of security vulnerability the enterprise AI community has been slow to address. Organisations have invested significantly in text-based guardrails and prompt injection defences for conventional large language models. The multimodal attack surface remains dangerously underappreciated. A user submitting an image that contains embedded instructions, a handwritten prompt photographed and uploaded as an attachment, or machine-readable but visually invisible markers within a document: these are documented attack vectors against production multimodal AI systems that text-based controls cannot detect. The Aigos Blueprint on Multimodal Model Security, published in 2023, provides the framework enterprises need to address this before it becomes a production incident.
📄 Download the Full Blueprint: Multimodal Model Security 2023
What Vision-Language Models Are and Why They Change the Security Calculus
Vision-language models such as GPT-4V, Alibaba’s QWEN-VL, and LLaVA integrate computer vision capabilities, enabling them to interpret and generate content by processing both visual and textual information simultaneously. This capability, demonstrated by OpenAI’s image-to-website generation in 2023 and subsequently by Google’s Gemini demonstrations, enables use cases that were previously impossible: document analysis spanning structured and unstructured content, visual question-answering over complex imagery, and multimodal chat interfaces that accept photographs, diagrams, and mixed-media documents.
In government and financial services, the security implications are particularly significant. The potential impact of a security breach extends beyond data integrity to national security and financial stability. Securing these applications requires addressing not only traditional data privacy and model robustness concerns but also the fundamentally new vulnerabilities introduced by the visual input channel.
Known Vulnerabilities: Image-Based Prompt Injection and Beyond
Visual prompt injection is the primary novel attack vector. It involves manipulating the visual input provided to a multimodal model by introducing carefully crafted stimuli: screenshots of text containing malicious instructions, images of handwritten commands, or machine-readable markers invisible to the human eye but interpretable by the model. An attacker who controls the visual input to a multimodal system can redirect the model’s behaviour, extract sensitive information from its context, or cause it to execute actions it would otherwise refuse.
The example the Blueprint uses is instructive. In a multi-modal VL chat agent that can write and execute code, a malicious actor uploads a screenshot of a code snippet while using the text prompt to ask the model to “explain” or “improve” it. This causes the model to process and potentially execute adversarial code through the image channel rather than the text channel, bypassing text-based input controls entirely. This demonstrates why guardrails that operate only on text inputs are structurally inadequate for multimodal systems.
The Blueprint also addresses reputational risk from data poisoning. Biased or adversarially curated training data can cause models to exhibit behaviours that damage the deploying organisation’s reputation, even when the immediate security impact appears negligible. Reputational risk is a real consequence of model misbehaviour, not only technical security breaches.
Six Categories of Risk That Multimodal Guardrails Must Address
An effective multimodal guardrail must handle at least six categories of security and privacy risk events: audio-visual code injection at input; text-based code injection at input; audio-visual data exfiltration in output; text-based data exfiltration in output; inappropriate or harmful audio-visual content in inputs or outputs; and privacy violations in model outputs. A guardrail that addresses only a subset of these categories provides partial protection, and adversaries will probe the uncovered dimensions.
Key Challenges and Security Implementation Guidelines
Three principal challenges arise in implementing guardrails for multimodal AI systems: input format complexity, the breadth of risk scenarios that must be addressed, and latency requirements in production. Text inputs screened by a guardrail model are relatively straightforward to process; images, audio, and video require separate processing pipelines operating within the same latency budget. An effective multimodal guardrail must be multimodal itself, capable of screening all input modalities, not only text.
The Blueprint’s security implementation guidelines cover three core components: input handling running parallel to retrieval processes with strict latency constraints; output sanitisation operating independently from the core model to prevent bypass through injection; and data pipeline handling that screens training and ingestion data before it reaches the knowledge base. These guidelines are applicable to in-house development teams and as a framework for vendor security assessment during AI procurement. Guardrails implemented as a locally deployed microservice rather than an external API call satisfy both latency and data residency requirements that API-based guardrail services typically cannot meet for enterprise production environments.
📄 Download the Full Blueprint: Multimodal Model Security 2023