Repository navigation
Speed up UTF-8 validation in tokenizer and ASCII scanning in UTF-8 decoding #161145
Description
Feature or enhancement
- The tokenizer checks source is utf-8 with
_PyTokenizer_ensure_utf8one character at a time. Reuse_Py_find_first_nonasciito scan faster. That makescompile()forLib/typing.pyabout 1% faster. - Modify
_Py_find_first_nonasciito test 64 bytes (32 on 32-bit platforms) of ASCII per branch. The code gets vectorized by some compilers and uses no intrinsics (https://godbolt.org/z/8bM83vx3a).(b'a' * 4096).decode()gets aout 1.5x faster.
Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
No response
Activity
picnixz commentedon Nov 1, 2026
We have a similar feature for stringlib with
find_max_char. Possibility to reuse it maybe?cmaloney commentedon Nov 1, 2026
I looked at that but the
unicodeobject.ccode andstringlib/codecs.hassume copying the whole value to an output buffer in their API currently and would take a lot bigger refactor to deduplicate than I want to do currently.pythongh-161145: Use _Py_find_first_nonascii in the tokenizer
pythongh-161145: Test 32-64 bytes check in find_first_nonascii
pythongh-161145: Test more bytes simultaneously find_first_nonascii
pythongh-161145: Add NEWS entry