OpenAI open-weight models
OpenAI described its new open-weight models as hardened against jailbreaks. The pseudonymous red-teamer Pliny the Liberator published a working jailbreak within hours of release.
OpenAI described its new open-weight models as hardened against jailbreaks. The pseudonymous red-teamer Pliny the Liberator published a working jailbreak within hours of release.
Source: Decrypt ↗. Follow it for the full account.
A user talks the model out of its own safeguards.
Framework entries that cover it: OWASP LLM01: Prompt Injection (jailbreaking) · MITRE ATLAS: LLM Jailbreak
- 1
What would have caught this one
Assume the model's own safeguards will be bypassed, and test the controls you put around it.
- 2
Prevent the whole class
Assume the model's built-in safeguards will be bypassed, and put your own controls around it.
- 3
Test for it
Run hundreds of rephrasings of each disallowed request, not a handful.