Repository navigation
FIX: Reject invalid scalar values in UnicodeSubstitutionConverter - #3881
Open
KVR Vishnu (VISHNU0906) wants to merge 2 commits into
Open
KVR Vishnu (VISHNU0906) wants to merge 2 commits into
KVR Vishnu (VISHNU0906) wants to merge 2 commits into
Conversation
Shifting a character by start_value can land above U+10FFFF or inside the surrogate range. That used to surface as a raw chr() error or as text that cannot be UTF-8 encoded. Raise a clear ValueError instead. Signed-off-by: KVR Vishnu <46535652+VISHNU0906@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Both new parametrized tests omit required parameter and return annotations and use positional parameters, contrary to the repository style guide.
1 open issue
What changed in this PR
This PR addresses #3853 by validating Unicode scalar values in UnicodeSubstitutionConverter, preventing out-of-range errors and surrogate output that cannot be UTF-8 encoded.
Changes:
- Rejects invalid shifted values with a descriptive
ValueError, while preserving constructor validation. - Adds regression tests for invalid combinations and valid Unicode boundaries.
| File | Description |
|---|---|
tests/unit/converter/test_unicode_sub_converter.py |
Tests invalid scalar rejection and valid boundary output. |
pyrit/converter/unicode_sub_converter.py |
Validates shifted code points and documents the new error behavior. |
Review effort: Balanced
Comment on lines
+40
to
+54
| async def test_unicode_sub_rejects_invalid_scalar_output_async(start_value, prompt): | ||
| converter = UnicodeSubstitutionConverter(start_value=start_value) | ||
| with pytest.raises(ValueError, match="not a valid Unicode scalar value"): | ||
| await converter.convert_async(prompt=prompt, input_type="text") | ||
|
|
||
|
|
||
| @pytest.mark.parametrize( | ||
| ("start_value", "prompt"), | ||
| [ | ||
| (0x10FFFF, "\x00"), | ||
| (0xD7FF - ord("a"), "a"), | ||
| (0xE000 - ord("a"), "a"), | ||
| ], | ||
| ) | ||
| async def test_unicode_sub_accepts_scalar_boundaries_async(start_value, prompt): |
… keyword-only Signed-off-by: KVR Vishnu <46535652+VISHNU0906@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fixes #3853
UnicodeSubstitutionConverteraddsstart_valueto each code point and passes the sum straight tochr(). With some accepted start values (or with the default and a high input character, like U+30000) the sum goes past U+10FFFF or lands in the surrogate range, so you either get a barechr() arg not in rangeerror or output that blows up later on UTF-8 encode.convert_asyncnow checks each shifted value and raises aValueErrorthat names the start value, the character and the result. The constructor check stays the same, since whether a start value works depends on the prompt.Tests cover the two cases from the issue, the surrogate start, the default start with a high input character, and the valid edges right next to the surrogate range and at U+10FFFF.
Local checks:
pre-commit run --all-files,ty check pyrit tests/unit, and the full unit suite (34263 passed).cc Roman Lutz (@romanlutz)