[Varun Poojari (Varunpoojari)]

Description

Adds _CyberSecEvalPromptInjectionDataset, a remote loader for the English textual prompt injection benchmark of Meta's CyberSecEval (Purple Llama), following the MITRE loader from #3028. Closes #3157 and contributes to the CyberSecEval parity tracked in #511.

• Fetches prompt_injection/prompt_injection.json from the same pinned PurpleLlama commit as the MITRE loader (172c1074069eb88ec834124272c1b1c4f8893445, MIT licensed): 251 English cases, 196 direct and 55 indirect, 180 security-violating and 71 logic-violating, 15 injection variants.
• Emits two SeedPrompts per row sharing a prompt_group_id: a system seed (sequence 0) with test_case_prompt and a user seed (sequence 1) with user_input, both verbatim.
• Preserves prompt_id, injection_type, injection_variant, risk_category and judge_question in metadata (separate dict per seed), so a scorer can reproduce the benchmark's success check.
• Exposes CyberSecEvalInjectionType and CyberSecEvalRiskCategory enum filters. An empty result raises an explicit error that names the filter.
• Validates rows (required keys, non-empty strings, integer prompt_id, enum values). Malformed rows are reported even when a filter would exclude them.
• Fetching goes through asyncio.to_thread, as in the MITRE loader.
• harm_categories is left empty because HarmCategory has no prompt-injection category (see the issue). Size medium; tags safety, cybersecurity, prompt_injection.

Out of scope: the judge step, the multilingual set and the visual prompt injection benchmark.

I opened #3157 first to agree on the approach (system seed + user seed in one group). It has no maintainer feedback yet, so I'm happy to restructure, for example by moving the system prompt into metadata.

Tests and Documentation

Added 71 unit tests, covering roles/sequences, group sharing, verbatim values, metadata, filters, the empty-filter message, enum and row validation (including a malformed row that a filter would exclude), non-blocking fetch, source/cache passthrough, class metadata and a real SQLite memory round-trip.

Added a "CyberSecEval: Prompt Injection" section to doc/code/datasets/1_loading_datasets.py and the paired notebook. Only the two new cells have outputs; existing outputs are untouched.

Validated locally:

• new tests: 71 passed; all of tests/unit/datasets: 5191 passed
• real-source tests/end_to_end/test_all_datasets.py -k CyberSecEvalPromptInjection: passed
• ruff, ty, build_scripts.validate_docs and pre-commit: passed

🤖 Generated with Claude Code

https://claude.ai/code/session_01MuspaiiGPPX4nXN7WCZYAx