Jailbreak

A jailbreak is a prompting technique that attempts to get a language model to bypass its safety training and produce output it was trained to refuse, typically by reframing the request — through role-play, hypothetical framing, or splitting a disallowed request into innocuous-looking pieces — rather than through any code-level exploit.

Safety behavior in a language model isn't a hard-coded rule that either fires or doesn't; it's trained behavior, a learned tendency to refuse certain requests that generalizes across the countless ways a request can be phrased, but generalizes imperfectly. That imperfection is what a jailbreak is searching for: a framing that shifts the model's response away from refusal without ever asking, in the way its training recognizes, for the thing it was trained to decline.

How it works

Rather than detailing techniques, it's worth understanding the shape of the problem: because refusal is a learned distribution over "comply" versus "decline" rather than an explicit filter, any context that changes how a request reads to the model can shift that distribution — a fictional frame, a claimed legitimate purpose, an incremental build-up across several turns that never states the disallowed goal outright in any single message. Effective defense doesn't try to enumerate and block every possible framing; it trains refusal to generalize better, and backstops that training with independent checks that don't rely on the generative model's own judgment.

Why it matters

No model ships with a jailbreak success rate of zero, and treating "we trained it to refuse" as the end of the story is how gaps go undetected until an attacker finds them first. The layered defense that actually works looks like this: training-time methods that generalize refusal across framings rather than memorizing specific phrasings, inference-time classifiers that scan input and output independently of the generative model (so a jailbroken generation can still be caught downstream), and systematic red-teaming that hunts for gaps before real users or attackers do. The honest measure of jailbreak resistance is a tested success rate, not a claimed one.

A worked example

A team runs an automated jailbreak-evaluation harness — 250 adversarial prompts drawn from a benchmark set — against a model before and after a safety-training update paired with a new input classifier. Before the patch: 71 of 250 prompts succeed (28.4%). After: 9 of 250 succeed (3.6%) — an 87% relative reduction in success rate, measured, not assumed. That residual 3.6% is also the honest finding: no patch reaches zero, which is precisely why red-teaming has to keep running after a fix ships, not stop once one round of testing looks good.

How Miatz teaches it

The security module teaches jailbreaks strictly as a measurement problem: learners are given an eval harness and a labeled prompt set, run it against a model, and score the success rate — the same exercise as the worked example above — rather than being handed a list of techniques to try. The point is building the instinct to test and measure a model's safety behavior empirically, the same instinct llm-evals teaches for quality more broadly.