Skip to content

FIX: Reject invalid scalar values in UnicodeSubstitutionConverter - #3881

Open
KVR Vishnu (VISHNU0906) wants to merge 2 commits into
microsoft:mainfrom
VISHNU0906:fix-unicode-sub-invalid-scalars
Open

KVR Vishnu (VISHNU0906) wants to merge 2 commits into
microsoft:mainfrom
VISHNU0906:fix-unicode-sub-invalid-scalars

Conversation

@VISHNU0906

Copy link
Copy Markdown

Fixes #3853

UnicodeSubstitutionConverter adds start_value to each code point and passes the sum straight to chr(). With some accepted start values (or with the default and a high input character, like U+30000) the sum goes past U+10FFFF or lands in the surrogate range, so you either get a bare chr() arg not in range error or output that blows up later on UTF-8 encode.

convert_async now checks each shifted value and raises a ValueError that names the start value, the character and the result. The constructor check stays the same, since whether a start value works depends on the prompt.

Tests cover the two cases from the issue, the surrogate start, the default start with a high input character, and the valid edges right next to the surrogate range and at U+10FFFF.

Local checks: pre-commit run --all-files, ty check pyrit tests/unit, and the full unit suite (34263 passed).

cc Roman Lutz (@romanlutz)

Shifting a character by start_value can land above U+10FFFF or inside the surrogate range. That used to surface as a raw chr() error or as text that cannot be UTF-8 encoded. Raise a clear ValueError instead.

Signed-off-by: KVR Vishnu <46535652+VISHNU0906@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 3, 2026 12:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Both new parametrized tests omit required parameter and return annotations and use positional parameters, contrary to the repository style guide.

1 open issue
What changed in this PR

This PR addresses #3853 by validating Unicode scalar values in UnicodeSubstitutionConverter, preventing out-of-range errors and surrogate output that cannot be UTF-8 encoded.

Changes:

  • Rejects invalid shifted values with a descriptive ValueError, while preserving constructor validation.
  • Adds regression tests for invalid combinations and valid Unicode boundaries.
File Description
tests/​unit/​converter/​test_unicode_sub_converter.py Tests invalid scalar rejection and valid boundary output.
pyrit/​converter/​unicode_sub_converter.py Validates shifted code points and documents the new error behavior.

Review effort: Balanced

Comment on lines +40 to +54
async def test_unicode_sub_rejects_invalid_scalar_output_async(start_value, prompt):
converter = UnicodeSubstitutionConverter(start_value=start_value)
with pytest.raises(ValueError, match="not a valid Unicode scalar value"):
await converter.convert_async(prompt=prompt, input_type="text")


@pytest.mark.parametrize(
("start_value", "prompt"),
[
(0x10FFFF, "\x00"),
(0xD7FF - ord("a"), "a"),
(0xE000 - ord("a"), "a"),
],
)
async def test_unicode_sub_accepts_scalar_boundaries_async(start_value, prompt):
… keyword-only

Signed-off-by: KVR Vishnu <46535652+VISHNU0906@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UnicodeSubstitutionConverter can emit invalid scalar values from accepted starts

2 participants