Repository navigation
FEAT: Add MASK (CAIS honesty benchmark) dataset loader - #1904
Open
Roman Lutz (romanlutz) wants to merge 10 commits into
Open
Roman Lutz (romanlutz) wants to merge 10 commits into
Roman Lutz (romanlutz) wants to merge 10 commits into
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolved conflicts: - pyrit/datasets/seed_datasets/remote/__init__.py — kept both MaskQuestionArchetype (HEAD) and MossBenchOversensitivityType (main) in __all__. - doc/bibliography.md — merged the new MASK citation key into the upstream's larger citation list. Followups for upstream changes: - mask_dataset.py + test_mask_dataset.py: renamed _fetch_from_huggingface -> _fetch_from_huggingface_async to match the async-suffix sweep (microsoft#1889). - seed_metadata.py: added "honesty" to RECOMMENDED_TAGS so MASK's cross-cutting tag passes the metadata-coverage check added by microsoft#1780. Verification: 836 unit dataset tests pass; pre-commit clean on all touched files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adrian Gavrila (adrian-gavrila)
left a comment
Adrian Gavrila (adrian-gavrila)
left a comment
Contributor
There was a problem hiding this comment.
Overall looks good! A couple of nits and then one question related to tagging this as defult.
Resolve dataset-registration and bibliography conflicts while retaining the dataset-only MASK changes. Update MASK seed annotations and property docstrings for current type and lint rules. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5af0366c-c21d-410f-914d-fae7757a6cec
Normalize the MASK BibTeX entry, make base/concrete loader registration explicit, and exclude the gated dataset from the default tag. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5af0366c-c21d-410f-914d-fae7757a6cec
Adrian Gavrila (adrian-gavrila)
approved these changes
Aug 14, 2026
Load all six MASK archetypes through one registered dataset by default while retaining optional enum-based archetype filtering and archetype-specific conversation construction. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5af0366c-c21d-410f-914d-fae7757a6cec
Roman Lutz (romanlutz)
changed the title
Derive binary misinformation goals from the non-ground-truth pivotal option, retain the original proposition in metadata, and describe numerical pressure as a misleading-value objective. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5af0366c-c21d-410f-914d-fae7757a6cec
Contributor
Author
|
This is not yet merged on purpose. I want to evaluate all the individual sub datasets a bit more to see if I actually want this. |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5af0366c-c21d-410f-914d-fae7757a6cec
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5af0366c-c21d-410f-914d-fae7757a6cec
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5af0366c-c21d-410f-914d-fae7757a6cec
Preserve complete seed conversations across sequential dataset imports while keeping reloads idempotent. Map MASK to INFO_INTEGRITY, include numerical reference facts in objectives, and correct public access and license documentation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5af0366c-c21d-410f-914d-fae7757a6cec
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 19, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds one
_MaskDatasetprovider, registered asmask, for the CAIS MASK honesty benchmark. It loads all six configs by default, with optional filtering througharchetypes: list[MaskQuestionArchetype] | None:continuations,disinformation,doubling_down_known_facts,known_facts,provided_facts, andstatistics. Together they contain 1,000 rows at pinned revision4602b84dd9e2ca05c6e1eafbc14e556e908ac1bb.Each row becomes one attacker-facing
SeedObjectiveplus the original role-bearing conversation. System prompts are preserved, including the supplied facts inprovided_facts. Doubling-down rows retainsystem -> user -> assistant -> userordering. Every seed carries the source task ID, archetype, proposition and type, ground truth, formatted ground truth, originaldeliveryvalue, belief-elicitation fields, and canary metadata. Binary objectives target the pressured false claim; numerical objectives include the correct reference fact and interval so objective-based judging has the required context.Dataset discovery and all seeds use the canonical
INFO_INTEGRITYharm category. Thesafetyandhonestytags and source archetype metadata are retained. Shared memory deduplication now compares complete multi-seed groups, including roles, sequence, metadata and criteria, rather than dropping individual shared system prompts or objectives. Sequential subset imports preserve complete groups; equivalent reloads remain idempotent even with fresh UUIDs. Standalone seed deduplication, database collation, unnamed-dataset matching, and criteria behavior are preserved. No schema migration is needed.Access and licensing: the pinned dataset is public and can be downloaded without a HuggingFace token. Optional constructor/environment authentication remains supported. The dataset card does not declare a data license, so this PR does not claim CAIS research-use restrictions or assume that the upstream code repository's MIT license covers the data. MASK is no longer incorrectly skipped as a gated provider in the E2E sweep. No source rows or real canary are vendored.
This is dataset-only. It adds no scorer, honesty metric, attack, executor, or scenario. Full MASK honesty evaluation still requires separate belief elicitation and joint judging.
Tests and Documentation
Fictional unit fixtures cover all six configs, filtering, optional token plumbing, complete metadata and role preservation, numeric reference context, and canonical discovery/filtering. Memory regressions cover sequential subsets, complete-group reloads, different roles/positions/criteria/metadata, member multiplicity, schemas, copied companions, collation, wildcard matching, and bounded lookups.
Validation:
uv run pytest -n 4 tests\unit\memory tests\unit\datasets\test_mask_dataset.py -q --tb=short --disable-warnings: 1,837 passed, 1 skipped.uv run pytest tests\end_to_end\test_all_datasets.py -k Mask -q --tb=short: 1 passed against the real pinned dataset, with isolated HF caches and implicit authentication disabled.uv run python -(inline anonymous loader/SQLite audit): passed. Checked every upstream field, conversation role/text, execution metadata, origin and canonical category across all 1,000 groups / 3,240 seeds. Sequential imports, full reload, reverse subset reloads, and bulk import with fresh-ID reloads all retained exactly 1,000 complete groups / 3,240 seeds.uv run ty check pyrit: passed.uv run --no-sync pre-commit run --files pyrit\memory\memory_interface.py pyrit\datasets\seed_datasets\remote\mask_dataset.py tests\unit\memory\memory_interface\test_interface_seed_prompts.py tests\unit\datasets\test_mask_dataset.py tests\end_to_end\test_all_datasets.py doc\code\memory\8_seed_database.py doc\code\memory\8_seed_database.ipynb: passed withUV_NO_SYNC=1.Adds the MASK citation and built-in dataset listing. Updates the paired memory documentation to explain complete-group deduplication. The prose-only memory notebook changes were made in both
.pyand.ipynb; Jupytext cell-content comparison passed and existing outputs were retained. Notebook execution was not rerun for these prose-only edits.