Skip to content

Read every source through the new readers - #28

Merged
turukawa merged 8 commits into
v2from
load
Oct 31, 2026
Merged

turukawa merged 8 commits into
v2from
load

Conversation

@turukawa

Copy link
Copy Markdown
Member

Every format is read into one grid of cells and a safe table under whyqd's own dialect: CSV with its encoding ladder, Excel cell by cell, PDF, Parquet, and Feather. Each column is profiled, and each source is fingerprinted by its bytes and its cells.

The 1.x read is frozen in whyqd/parsers/v1.py, so every stored 1.x checksum still verifies. A 2.0 schema carries no row count; row-level actions are checked against the data source's row count. A derive reads its file once.

Test data now keeps its bytes, restoring the line-ending fixtures git had normalised.
Record each column's cell types and formats, verify a file by its bytes or cells, and check row-level actions against the data source's row count.
Feed the parser lines matched from the decoded text, and keep the derived table for the first get_data, so a derive reads the file once.
Install all extras in the workflow, and skip the PDF tests where pdfplumber is missing.
@turukawa
turukawa merged commit ae3b6ac into v2 Oct 31, 2026
3 checks passed
@turukawa
turukawa deleted the load branch October 31, 2026 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant