Game Theory Jailbreaking - A New Black-Box Attack Paradigm for LLMs
Jailbreaking a production LLM is no longer about finding one clever prompt. Black-box models are stochastic systems accessed through unlimited, repeatable interaction — which turns the exchange into a repeated game where retries are nearly free for the attacker and refusal consistency is expensive for the defender. Evaluations across multi-turn adversarial sessions consistently show the same pattern: refusal is statistically high but not absolute, and consistency degrades as context accumulates across turns. A model that refuses correctly nine times out of ten is still a failure mode once an attacker gets ten tries — and API and SaaS access effectively guarantees far more than ten. This article breaks down why opacity does not raise the bar the way teams assume, models the interaction formally as a repeated game between attacker and defender, and sets out what multi-turn adversarial testing actually needs to measure instead of single-prompt refusal rates.
Tested Your LLM With One Prompt? That's Not a Test.
Sustained, adversarial multi-turn sessions that measure refusal stability, not just refusal presence. Concrete quote in the scoping meeting.
For a long time, jailbreaks were treated as a curiosity. Someone found a clever prompt, it circulated online, a filter was adjusted, and the incident got written off as an edge case. If you've spent years in application security, this probably felt familiar another input validation issue, just expressed in natural language.
That framing is now insufficient.
What breaks in real systems today is not a single prompt but the interaction itself. Modern jailbreaks unfold across multiple turns, exploiting how language models adapt, hedge, and rebalance priorities over time. The prompt is no longer the exploit; it's the probe. This distinction explains a pattern we see repeatedly in real-world evaluations: models that look robust in single-turn testing often fail under sustained, adversarial interaction.
Why Opacity Doesn't Buy You Security
Most production LLM deployments are intentionally opaque. Model weights are hidden, safety classifiers are abstracted, and system prompts are treated as sensitive implementation details. There's an implicit assumption that this opacity raises the bar for attackers.
In practice, it rarely does.
A black-box adversary doesn't need internal access to learn how a model behaves. Given enough interaction, outputs become a reliable proxy for internal state a well-understood failure mode in security, from timing side channels to ML model extraction. Language models amplify the problem because the feedback channel is unusually rich. During assessments, the same patterns show up again and again: refusals that soften after context accumulates, explanations that grow more detailed over time, and inconsistent outcomes across near-identical inputs. None of it requires insider knowledge, only repetition.
In environments where models are accessed via APIs or SaaS platforms, attackers effectively get unlimited samples. Under those conditions, even small behavioral inconsistencies become exploitable.
Stop Thinking in Prompts
One of the most persistent mistakes we see in LLM security reviews is treating the prompt as the unit of analysis. It isn't.
What matters is how the system behaves across a sequence of decisions. The attacker observes a response, adjusts framing, and applies pressure wherever the model shows uncertainty. The model responds probabilistically, balancing safety, usefulness, and conversational coherence turn by turn.
This is why one-shot jailbreak metrics are misleading. In multi-turn evaluations, models that refuse correctly on the first turn often fail later — not because a rule is missing, but because earlier responses constrain later ones. Context accumulates. Refusal states degrade. What was "out of scope" in turn one becomes "conditionally acceptable" by turn six.
From a security perspective, this isn't a content moderation problem. It's a sequential decision-making problem under uncertainty which is exactly the setup game theory was built to analyze.
The Interaction as a Repeated Game
Treat it formally for a second. There are two players. The attacker chooses, at each turn, whether to persist (escalate, reframe, apply pressure) or disengage. The defender the model, shaped by training and system prompt behaves somewhere on a spectrum from consistent refusal to variable, degrading refusal as context builds.
The payoffs aren't symmetric. Each additional attacker turn costs the attacker close to nothing a little time, maybe a slightly longer chat log. Each additional point of refusal consistency costs the defender something real: usability, false-positive friction, and, at the extreme, users who leave for a less restrictive product. That asymmetry is the whole game.
Under these payoffs, "persist" weakly dominates "disengage" for the attacker whenever there's any observable variance in the defender's behavior and stochastic models always have some variance. This is why treating a 95% refusal rate as "safe" misreads the game. It isn't a single draw. It's a repeated one, and the attacker only needs to win once.
Reasonable Decisions, Bad Outcomes
One uncomfortable reality of LLM failures is that they rarely look like bugs. They look like judgment calls.
In isolation, most problematic responses are defensible. The model avoids being overly restrictive. It tries to be helpful. It handles ambiguity politely. The failure emerges only when those decisions are chained together across a session.
This mirrors patterns security teams have seen for years in fraud detection and abuse-prevention systems: local optimization producing global failure. Guardrails get tuned for explicit violations — keywords, direct intent, obvious misuse. Attackers operate in the gray space between them, where framing and context matter more than syntax.
Interpretation. Risk is not concentrated at the first interaction. It emerges as context accumulates and earlier responses reshape the model's decision space over the course of a session which is exactly the qualitative pattern shown above, based on repeated multi-turn adversarial evaluations rather than a single published dataset.
Probabilistic Safety Is Still Unsafe
Another misconception we run into constantly: the idea that rare failures are acceptable because they're rare.
Language models are stochastic systems. Sampling, temperature, and internal uncertainty mean the same input doesn't always produce the same output. From a reliability standpoint, that's expected. From a security standpoint, it's exploitable.
During repeated testing, refusal behavior is often statistically consistent but not absolute. A response that's blocked nine times out of ten is still a failure mode if the tenth response matters. Under repeated interaction, probability stops being a safety margin and becomes an inevitability. Attackers don't need certainty. They need variance.
Most vendor red-team reports and internal QA cycles still measure refusal presence on isolated prompts. If yours hasn't been tested under sustained, escalating, multi-turn pressure, you don't actually know your refusal consistency — only that it worked once.
Schedule a Scoping Meeting →This Isn't Patchable
There's no single prompt to block here. No static policy fixes this class of issue.
Once interaction itself becomes the attack surface, defense has to become part of the system's behavior, not a filter bolted onto it. Attackers get unlimited retries; defenders pay for false positives, degraded user experience, and regulatory exposure. That imbalance is structural, and it's the same asymmetry the repeated-game framing above describes — it doesn't go away because a new keyword gets blocklisted.
This is also why governance frameworks are moving away from checklist-style controls and toward behavioral assurance. Standards like ISO/IEC 42001 emphasize whether AI systems behave predictably and responsibly over time, not just whether safeguards exist on paper, as covered in our piece on AI compliance and governance under ISO/IEC 42001. The same logic underpins EU AI Act Article 15(5), which names adversarial resilience as a required control category rather than a one-time checkbox.
Single-Turn vs Multi-Turn Evaluation: What Changes
The practical consequence of treating this as a repeated game is that the unit of testing has to change. The table below shows what shifts when evaluation moves from single-prompt filters to sustained adversarial sessions.
| Dimension | Single-Prompt Testing | Multi-Turn Adversarial Testing |
|---|---|---|
| Unit of analysis | The prompt | The session, across sequential turns |
| What gets measured | Refusal presence on a single input | Refusal stability across repeated and escalating inputs |
| Sampling variance | Treated as noise, averaged away | Treated as a security parameter to be measured directly |
| Context effects | Not captured | Explicitly tracked as context accumulates turn to turn |
| Typical blind spot | Guardrails that only trigger after several turns of pressure | Sustained-pressure failure modes surfaced directly |
Framework developed by SecurityWall based on patterns observed across multi-turn AI red team engagements.
What This Means in Practice
From a security engineering perspective, this changes what "testing" actually means.
- Multi-turn adversarial scenarios, not single inputs
- Measuring refusal stability, not just refusal presence
- Treating sampling variance as a security parameter, not noise to average away
- Assessing how guardrails behave under sustained, escalating pressure
This is why organizations increasingly pair traditional penetration testing with AI-specific risk and maturity assessments, rather than treating LLMs as just another API endpoint.
How SecurityWall Tests for Multi-Turn Jailbreak Resilience
SecurityWall's AI and LLM security testing service is built around the repeated-game dynamic described above, not single-prompt filter checks.
Sustained adversarial sessions. Multi-turn engagements that escalate framing and pressure across a conversation, the same way a persistent real-world attacker would, rather than a one-shot prompt list.
Refusal stability measurement. We report refusal consistency across repeated and near-identical prompts, not just a pass or fail on a single attempt, so you see the variance an attacker would eventually find.
Prompt injection and jailbreak assessment. Direct and indirect prompt injection, system prompt extraction attempts, and jailbreak resistance testing, detailed further in our prompt injection testing guide.
AI red teaming. Scenario-based adversarial simulation synthesizing the categories above into production-like conditions. Our LLM red teaming methodology walks through the approach in full.
Framework-mapped output. Reports aligned to OWASP Top 10 for Agentic AI, EU AI Act Article 15(5), and, where relevant, ISO 42001 controls.
Concrete pricing is quoted in the 30 minute scoping meeting because it depends on model architecture, session length, and framework coverage. What's standard across every engagement: independent testing, measured refusal stability, and a report built for engineers who need to fix the behavior, not just read about it.Frequently Asked Questions
What is game theory jailbreaking? It's a framing for multi-turn LLM jailbreaks as a repeated game between an attacker and a defender (the model), rather than a single malicious prompt. The attacker chooses to persist or disengage at each turn; the defender's refusal behavior sits somewhere between consistent and variable. Because retries cost the attacker almost nothing while refusal consistency costs the defender usability, the game structurally favors persistence whenever the defender shows any variance in refusal behavior.
Why does opacity (hidden weights, hidden system prompts) not stop this kind of attack? Because a black-box attacker doesn't need internal access to learn how a model behaves. Given enough interaction, outputs alone are a reliable proxy for internal state, the same failure mode security teams already know from timing side channels and ML model extraction. API and SaaS access effectively gives an attacker unlimited samples to build that picture from.
A model refuses 90 to 95 percent of the time. Isn't that good enough? Not under repeated interaction. A response blocked nine times out of ten is still a failure mode if the tenth response matters, and an attacker with unlimited retries only needs to find that tenth response once. Treating a high refusal rate as "safe" mistakes a repeated game for a single draw.
How is multi-turn jailbreak testing different from a standard LLM penetration test? A standard test often checks whether a model refuses a list of known-bad single prompts. Multi-turn testing runs sustained, escalating adversarial sessions and measures refusal stability across turns and across repeated near-identical prompts, which is where most production failures actually show up. See our LLM penetration testing guide for how the two approaches fit together.
Can this be fixed with a single prompt filter or system prompt update? No. There's no static policy that closes this class of issue, because the interaction itself is the attack surface. Defense has to become part of the system's ongoing behavior across a session, evaluated the same way it's attacked: over sustained, multi-turn interaction, not a keyword blocklist.
How often should multi-turn adversarial testing be repeated? Treat it as continuous rather than one-time, similar to how EU AI Act Article 9 frames risk management as an iterative lifecycle process. Any model update, system prompt change, or new deployment surface changes the game's payoffs and warrants a fresh round of testing.
Related reading:
