Skip to content

fix(domain): match IDN TLDs against IANA list in IDNA-encoded form - #524

Open
rahul05ranjan wants to merge 1 commit into
python-validators:masterfrom
rahul05ranjan:fix/idn-tld-consider-tld
Open

rahul05ranjan wants to merge 1 commit into
python-validators:masterfrom
rahul05ranjan:fix/idn-tld-consider-tld

Conversation

@rahul05ranjan

Copy link
Copy Markdown

Summary

With consider_tld=True, the TLD was looked up in the IANA list as written in the input. The IANA list stores internationalized TLDs in their IDNA-encoded ASCII form (e.g. XN--P1AI), so a domain like пример.рф was rejected even though the identical punycode form was accepted:

>>> validators.domain("пример.рф", consider_tld=True)
ValidationError(func=domain, args={'value': 'пример.рф', 'consider_tld': True})
>>> validators.domain("xn--e1afmkfd.xn--p1ai", consider_tld=True)
True

This encodes the TLD with IDNA before the IANA lookup. When IDNA encoding of the TLD fails, the raw value is kept so the lookup still rejects it, matching the previous behavior for unencodable/unknown TLDs. hostname() and url() pass consider_tld through to domain(), so they pick up the fix as well.

Tests

Added regression cases to tests/test_domain.py:

  • valid: пример.рф and trailing-dot form with rfc_1034=True
  • invalid: пример.испытание (a non-IANA IDN TLD that encodes fine, so it must still be rejected after encoding)

Verification

PYTHONPATH=src python3 -m pytest tests/ -q
18 failed, 1098 passed

The 18 failures are pre-existing on current master (all in tests/crypto_addresses/test_eth_address.py, verified by stashing this change and re-running: same 18 failed, 1095 passed — this PR adds 3 passing tests). All domain/hostname/url tests pass, including the new IDN TLD cases.

Fixes #522

With consider_tld=True, the TLD was looked up in the IANA list as
written. The list stores internationalized TLDs in their ASCII
(IDNA-encoded) form, so domains like пример.рф were rejected even
though their punycode form xn--e1afmkfd.xn--p1ai was accepted.

Encode the TLD before the lookup (falling back to the raw value
when IDNA encoding fails, so unencodable TLDs are still rejected
exactly like before). hostname() and url() pass consider_tld through
to domain(), so they pick up the fix as well.

Fixes python-validators#522

Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

consider_tld rejects Unicode IDN TLDs (e.g. пример.рф)

1 participant