The transition of Large Language Models (LLMs) from experimental novelties to core enterprise assets has been remarkably swift. The technology has carefully treaded the line between a 'stroke of brilliance' and 'threat to society'. Thankfully, global leaders are working to leverage the former and address the latter.
The stakes are exceptionally high. Traditional security perimeters are designed for deterministic code, but LLMs are fluid. They don't just process data, they interpret intent. This nuance creates a massive blind spot where a single, cleverly worded sentence can override millions of dollars' worth of conventional cybersecurity infrastructure.
A study found that 15% of organisations surveyed had had a security breach due to or affecting Gen AI. That makes generative AI security a non-negotiable requirement for business continuity.
The anatomy of the attack: beyond the firewall
To protect an organisation, decision-makers must first understand the weapons used against them. The threats are no longer just about malware, they are about semantic manipulation.
Taxonomy of attack techniques
Prompt engineering: Precision-crafting inputs to trigger unintended or unauthorised model outputs.
Social engineering: Manipulating the AI through deception, such as impersonating authorities or exploiting logical vulnerabilities.
Obfuscation: Encrypting or hiding malicious commands within harmless-looking text to bypass safety filters.
Knowledge poisoning: Corrupting the model's underlying data or knowledge base to compromise its reliability.
A few specific attacks have been identified:
Direct prompt injection (jailbreaking)
This is a front-door assault. An attacker uses "persona adoption" (forcing the AI to act as an unrestricted entity) or character-encoding tricks to bypass safety filters. The goal is usually to force the model to generate toxic content or reveal its internal system instructions.
Implement robust prompt injection prevention by using a secondary, "guardian" model to inspect the intent of user inputs before they reach the primary engine.
Indirect prompt injection
This is far more dangerous for enterprises using RAG (Retrieval-Augmented Generation). An attacker hides malicious instructions inside a website or document that the AI "reads" while performing a search. The AI then unknowingly executes those instructions, such as exfiltrating a user's session data to an external server.
Treat all retrieved data as untrusted. Use strict delimiters to separate external data from system instructions and apply "least privilege" principles to the AI's API access.
Why this is a "wicked" problem
The primary challenge in securing GenAI is its non-deterministic nature. Unlike a standard software patch that fixes a specific bug, a "fix" in an LLM might hold for one prompt but fail when the user changes a single adjective. As models become more integrated into business workflows, the risk of 'Shadow AI' increases the likelihood of PII leakage and IP theft.
When teams bypass official channels to use unvetted tools, they circumvent all pre-deployment security. This 'Shadow AI' exposes the firm to data leaks that no amount of perimeter security can stop. Centralising AI development under a unified security accelerator is the only way to maintain visibility and control.
The pressure to innovate often outpaces the pressure to secure. Many organisations find themselves in a "security debt" cycle, deploying models first and trying to bolt on guardrails later. This reactive approach is precisely what sophisticated threat actors exploit.
Best practices for preventive mitigation
Building a resilient AI ecosystem requires a shift from "incident response" to "secure by design." Industry leaders are now adopting a proactive stance through several critical layers:
Continuous AI red teaming
Before a model goes live, adversarial experts must attempt to "break" it. This isn't a one-off audit. It is a systematic attempt to trigger toxic outputs or bypass safety filters using techniques like persona adoption or obfuscated code. Finding these failure points in a sandbox environment is the only way to ensure the model behaves predictably in the wild.
Semantic guardrails
Move beyond simple "banned word" lists. Modern guardrails must check for context, sentiment, and data patterns. For example, a guardrail should prevent a model from ever outputting a string that looks like a credit card number or a private encryption key.
Establishing a responsible AI framework
Security is just one pillar. A holistic responsible AI framework ensures that the model is also accurate, fair, and transparent. This framework provides the governance necessary to satisfy both internal stakeholders and external regulators, such as those overseeing the EU AI Act.
Privacy-first data handling
Minimise the data the model can access. Use privacy-enhancing technologies to ensure that even if a model is compromised, the underlying sensitive data remains encrypted or anonymised.
How can Infosys BPM help with a responsible AI framework?
From automating ad reviews to implementing cutting-edge prompt injection prevention, Infosys BPM provides the domain expertise needed to safeguard your brand reputation. We help you build a responsible AI framework that balances the need for rapid innovation with the necessity of stringent generative AI security. By automating the review and compliance process, we allow your teams to innovate at scale without compromising on trust or safety.
FAQ
Primary threats include direct prompt injection (jailbreaking via persona adoption), indirect prompt injection (malicious instructions hidden in retrieved data), social engineering, obfuscation to bypass filters, and knowledge poisoning that corrupts model training data. These exploit LLMs' non-deterministic nature to override safety controls.
GenAI threats target semantic interpretation rather than code execution, making attacks fluid and context-dependent rather than deterministic. A single rephrased prompt can bypass static filters, while Shadow AI (unvetted tools) creates blind spots no perimeter security can address.
Continuous red teaming systematically tests models with adversarial prompts, persona tricks, and obfuscated attacks in sandbox environments to identify failure modes before production. Unlike one-time audits, it simulates real-world threat actors to ensure predictable behaviour under attack.
Semantic guardrails analyse context, sentiment, and patterns beyond keyword blocking, preventing outputs like credit card numbers, toxic content, or private keys. Guardian models inspect inputs before primary processing, while strict delimiters protect against indirect injection via RAG-retrieved data.
Best practices include privacy-first data handling with encryption/anonymisation, least-privilege API access, explainable models for auditability, human oversight for high-risk outputs, and centralised governance to eliminate Shadow AI. These balance innovation speed with regulatory compliance like the EU AI Act.
