Skip to content

Read each cell as its declared type, or say why not - #31

Merged
turukawa merged 8 commits into
v2from
typing
Nov 20, 2026
Merged

turukawa merged 8 commits into
v2from
typing

Conversation

@turukawa

Copy link
Copy Markdown
Member

A field with a declared type now reads each cell as that type, deterministically, by what is declared and never by the
machine's locale, and a transform either conforms to its destination schema or is not built. This replaces
coerce_to_schema and the pandas-era value parsers.

What changes

  • One reader for every type: ReadingParser, in Polars, reads numbers, integers, booleans, strings, dates, times,
    datetimes, periods, and durations. Each cell is a value, a blank, or a refusal with its reason.
  • A declaration settles what a cell cannot say. A field's policy holds CSV on the Web's base and format, with
    whyqd's own rules beside them. A date pattern may be typed as Excel, strptime, or UAX 35 writes it, and is saved
    as UAX 35's.
  • What has one sensible reading reads, with a warning: £5, (5), 5%, 1,234.50, 1e3, 13/04/2024,
    1 April 2024, and a bare 45295 in a date field. Each warning says what was read, and declaring it confirms the
    reading and tightens the field.
  • What cannot say halts: 01/02/2024, 1,234, 24 as a year, <5, TBC. A column never votes, so one cell
    never settles another.
  • A transform conforms or is not built. It reads every cell and checks every constraint, then halts with one
    report, raising ConformanceError, a ValueError. process(strict=True) treats each warning as a fault.
  • Presence as a boolean: CATEGORISE assigning True takes any value of the field's type. A blank is never
    present, and a zero only where zeroAsTrue, set by set_categories(..., zero_as_true=True), counts it.
  • Month and weekday names come from a fixed extract of CLDR, whyqd/config/names.json, so no library or machine
    changes a read.

Breaking changes

  • coerce_to_schema and the DataSourceParser value parsers (parse_dates, parse_float, and the rest) are removed.
  • A stored transform may halt where 1.x read silently, and builds again once its source fields declare how to read
    what halted. TransformDefinition.report says what and where.
  • A 1.x usdate upgrades to a date with month-first patterns, and saves back to 1.x as it was.
  • A month, quarter, or year reads as the date it ends on.
  • An integer field rounds half away from zero, with a warning, where 1.x truncated.

Tested

  • tests/test_reading.py reads every case in tests/data/synthetic-typing/typing-key.json, 427 of them, and converts
    13 synthetic files written as different locales' publishers write, under three machine locales.
  • tests/test_conformance.py covers halting, warnings, constraints, actions, presence, and a stored sample.
  • Suite: 1,464 passed, 1 skipped, 18 xfailed. Every stored definition reads, writes, and verifies its 1.x checksum.

The wrangler's account is docs/strategies/reading.md.

@turukawa
turukawa merged commit e127687 into v2 Nov 20, 2026
3 checks passed
@turukawa
turukawa deleted the typing branch November 20, 2026 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant