Skip to content

FEAT: Add OutputFormatConverter and CharSplitOutputConverter - #3381

Open
Absarahmedofficial wants to merge 1 commit into
microsoft:mainfrom
Absarahmedofficial:feat/char-split-output-converter
Open

Absarahmedofficial wants to merge 1 commit into
microsoft:mainfrom
Absarahmedofficial:feat/char-split-output-converter

Conversation

@Absarahmedofficial

Copy link
Copy Markdown
Contributor

Description

Adds OutputFormatConverter and CharSplitOutputConverter (text -> text) for output-format steering and response normalization in PyRIT converter pipelines.

  • OutputFormatConverter: Abstract base class for converters that append output-formatting instructions to a prompt (action="encode") and normalize the formatted target response back into plain text (action="decode") before scoring. Follows the same action="encode" | "decode" pattern and public self.action interface as SmugglerConverter (pyrit/converter/token_smuggling/base.py).
  • CharSplitOutputConverter: Instructs the target to insert a single non-whitespace delimiter (. by default) between adjacent characters within each word while preserving spaces, hyphens, and punctuation, and strips intra-word delimiters (unescaping doubled literal delimiters) on action="decode".
  • Exported both classes via TYPE_CHECKING and _LAZY_EXPORTS in pyrit/converter/__init__.py.
encoder = CharSplitOutputConverter(action="encode")
prompt = (await encoder.convert_async(prompt=user_prompt)).output_text
response = await target.send_prompt_async(message=Message.from_prompt(prompt=prompt, role="user"))
normalized = (await CharSplitOutputConverter(action="decode").convert_async(prompt=response[0].get_value())).output_text
scores = await scorer.score_text_async(text=normalized)

cc Roman Lutz (@romanlutz)

Tests and Documentation

  • Tests: Added 62 unit tests in tests/unit/converter/test_output_format_converter.py covering abstract instantiation checks, action and modality validation, valid/invalid delimiter checks (empty, multi-char, whitespace, non-string, NUL byte), printable and non-printable format instructions, intra-word delimiter stripping, doubled-delimiter unescaping, and ComponentIdentifier params. Verified with pytest tests/unit/converter/test_output_format_converter.py, pytest tests/unit/docs/test_converter_documentation.py, pytest tests/unit/common/test_lazy_package_imports.py, ruff check, ruff format --check, and ty check.
  • Documentation & JupyText: Added a CharSplitOutputConverter encode/decode example to doc/code/converters/1_text_to_text_converters.py, synced the paired doc/code/converters/1_text_to_text_converters.ipynb with jupytext --sync, and executed the new cell so the notebook carries the actual output while leaving existing cell outputs untouched.

from pyrit.models import ComponentIdentifier, PromptDataType


class OutputFormatConverter(Converter):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absarahmedofficial Could we compose the existing converters here instead of adding a new public base class?

PromptTemplateConverter or SuffixAppendConverter can append the formatting instructions, and SearchReplaceConverter can decode the response using the same regex. AttackConverterConfig already lets us configure these separately as request and response converters.

Could we package that configuration as a reusable attack technique instead? That would keep the character-splitting behavior without adding another converter hierarchy to maintain. Is there a use case this composition cannot support that I'm missing?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants