Repository navigation
feat(stack-encrypt, eql): json_tree tokens derivable from the path, a self tag, client-side wildcards - #1363
Conversation
A member's token depended on its ordinal, a keyed rank among its
siblings' keys, so the client could not derive the token of any node
below a top-level field. Reads therefore sent the head's token and walked
the rest through gates: a read of `$.profile.address.city` handed the
database a capability over all of `$.profile`, in every row, for the life
of the column, and a read could not return the node it named, so `{}`,
`[]`, a scalar and an absent field were one answer (#1356).
A member's token is now H16("t" ‖ t(parent) ‖ s), an element's keeps its
index, and every entry carries a self tag x = H8("x" ‖ iv ‖ t) by which
the server finds the node a token names, the root and the heads
included. Ordinals stay, in the parent tag, for enumeration only. The
subtree key is the list of tokens of the concrete paths a path names: one
for a path with no wildcard, `$` included under the sealed root, and for
a wildcard the positions below a bound (64 by default,
`json_subtree_key_with_bound`) and a probe at it. A probe that finds an
element fails the read with `Error::JsonWildcardTruncated` rather than
return part of the list, walked or fallen back to the sealed root, so a
row's answer does not depend on the budget.
The client checks each row: the depth-0 row's selector and decrypted
label against its token's path, each deeper row at the next position
with its parent's path's selector extended by its label, and every
row's self tag against the token it recomputes. The self tag is
unauthenticated, so that last check is for consistency; the selector,
position and label are the binding. The rows open as `Value`,
`Option<Value>` (None for no node) and `Vec<Value>`.
`TermError::JsonSealedRoot` had no use left and is removed;
`TermError::JsonWildcardTooWide` refuses a read past 65,536 tokens. The
derivations are pinned under the deterministic key source, and property
tests cover the read's metamorphic relations, the independence of
tokens from values and iv, and exactly which tokens a key sends.
JsonHeader::FORMAT_VERSION is unchanged: the family is unreleased.
Refs #1360
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
With every token derivable from its node's path, the server no longer
walks from a head through gates to reach a node: it is handed the
token of exactly the node read, finds it by its self tag, and returns it
at d = 0 with everything beneath it. `json_tree_subtree(doc, tokens
text[], budget DEFAULT 64)` replaces `json_tree_subtree(doc, token,
budget)` and `json_tree_subtree_at`, and answers `{}`, `[]`, a scalar
and an absent node as four answers, as `jsonb` does (#1356). Under the
member rule a hit's token no longer says whether it is a member or an
element, so the walk keeps whichever rule's self tag matches the hit's
`x`. The budget counts rows across the tokens; the recursion's LIMIT is
the entry count times the token count, since forged member tags can
make one subtree be visited twice. `json_tree_child_count` is NULL for
a token that names no node, and for the sealed root, whose members
chain from nothing.
The entry validator requires `x`; `JsonTreeEntry` gains it and
`JsonTreeSubtree` opens to `Option<Value>`. The PostgreSQL oracle holds
every read with no wildcard to strict `jsonb_path_query_first`, wildcard
reads to `jsonb_path_query` with truncation past the bound, the child
functions to `jsonb_object_keys` and `jsonb_array_length`, and reads of
randomly tampered rows to the right answer, a refusal or a documented
allowance.
Closes #1356
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The design change section left eight questions open; the owner settled them before the implementation, and the section now records them as "Decisions (2026-10-24)": keep the consecutive-position check beside the self-tag check and drop the parent-tag one, choose the member or element rule by the self tag with no stored kind bit, send probes as ordinary tokens, apply the bound in the sealed-root fallback, edit the derivation in place without a version bump, allow `$` under the sealed root, return NULL for an absent node's child count, and drop the gates. The leakage page's read and authentication sections now say what the check binds and what it leaves to the database by design: the row an entry comes from, the element a node at a selector sits under, and whether rows were withheld. Refs #1360 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
cargo-mutants found that nothing failed when a read's rows could match the same token more than once: `next = at + 1` survived as `at * 1`. A server that rewrites a second node's self tag to the first token's would then have two genuine nodes at one selector answer one token. The test forges exactly that and expects the refusal. Refs #1360 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
🦋 Changeset detectedLatest commit: 8c843ee The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 100 | 0 | 40 | 0 |
Every mutant in the changed lines was caught by a test.
cipherstash/stack#1363 changes the json_tree stored form (a self tag on every entry, ordinal-free member tokens, reads by token list, wildcards expanded on the client). Deciding whether to ship it needs the new form measured against its base on the same documents and the same machine, with the noise the comparison carries stated. The crate now builds either format from one source, by feature: `derivable` (default, stack 683802bcc) or `gates` (its base, edabee625), each a git dependency pinned by rev and renamed so both resolve in one Cargo.lock. What differs between them is confined to `ReadKey` in lib.rs. The plan builder's rename (`encrypt_with_layout`) is picked up. - dt_bench: `plan` derives each format's scenarios against its own database; `time` runs both plans from one process, alternating which format goes first, per call and per document in-server. - check_json_tree_v4 runs in both formats and opens every read through the key that derived it; it now also covers deep member paths, positions, wildcards at a bound, `$`, and the open-root and wide tables. - Two corpora for what the others cannot show: the nested corpus with the root open, and a wide one (300 members, 300 elements). - The ingest reports seal and INSERT time separately. - mise tasks bench:dt:{build,ingest,check,plan,time} and report:dt. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The measurement cipherstash/stack#1363 asked for, so the owner can decide whether to ship the format change: stack 683802bcc against its base edabee625, the same documents in two databases on one cluster, timed interleaved, 10,000 rows in full and 100,000 rows for sizes and the headline reads. Wildcard-free deep paths, single-level wildcards and wide enumeration got faster. Whole-subtree reads, the budget fallback, a full walk of a wide array, a two-level wildcard at N = 64 and json_tree_child_count on a small node got slower: the per-hit self-tag test, and a second whole-document map built on every read. Stored documents are 11% larger on disk (nested) and 7% (flat). Both formats pass every correctness check at both tiers. report/JSON_TREE_DERIVABLE_TOKENS.md holds the measurement. The JSON_TREE_COMPARISON.md Results tables are regenerated on the new format (the SteVec columns are the second measurement's, not re-run), and a fifth-measurement section there summarises it. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Benchmarks: this PR (683802b) against its base (edabee6)I measured the derivable-token format against the gated walk on the same documents, on the same machine. Each format had its own database on one PostgreSQL 17.10 cluster, and the two were timed interleaved. Full report, with methodology, the noise floor and the EXPLAIN plans: SummaryBetter
Worse
Unchanged within noise
Where the cost comes from
Stored size
Each entry gains 21 characters of Ingest (seconds; 10k rows is the median of 3 interleaved runs, 100k one run)
Reads with no wildcard (nested, 10,000 rows)These are per-call client times by id over 600 calls (200 rows × 3 passes), which include about 0.1 ms of round trip. The in-server column is one query over 500 documents.
Wildcard reads (nested, 10,000 rows)The base sends the same head token and gates at every N. 33% of rows hold more than 8 orders, so at N = 8 the derivable read refuses on those rows (
Enumeration and the member/element testThe first five rows are the wide corpus (1,000 rows, 300 members and 300 elements per document). The last two are nested
Containment,
|
| read | sent (gates → derivable) | gates, median (p95) ms | derivable, median (p95) ms | per call | in-server per doc |
|---|---|---|---|---|---|
@> one array element (nested, 1 row) |
– → – | 0.64 (0.79) | 0.64 (0.75) | +1% | — |
@> deep object (nested, LIMIT 10) |
– → – | 1.84 (2.03) | 1.92 (2.13) | +4% | — |
@> flat |
– → – | 0.44 (0.62) | 0.43 (0.58) | -1% | — |
-> nested |
– → – | 0.46 (0.57) | 0.49 (0.68) | +7% | +10% |
-> flat |
– → – | 0.18 (0.31) | 0.18 (0.32) | +0% | +5% |
ORDER BY ord_term(… -> sel) LIMIT 10, nested |
– → – | 2.17 (2.45) | 2.33 (2.59) | +7% | — |
| the self-tag map alone (built by every derivable read) | – | n/a (no such map) | 0.55 (0.72) | n/a | n/a |
| detoast and parse the document (floor) | – → – | 0.14 (0.19) | 0.15 (0.20) | +4% | +9% |
100,000 rows, headline reads
| read | sent (gates → derivable) | gates, median (p95) ms | derivable, median (p95) ms | per call | in-server per doc |
|---|---|---|---|---|---|
$.user.contact.emergency_contact.name |
1 tok + 3 gates → 1 tok | 2.37 (2.65) | 1.84 (2.32) | -22% | -27% |
$.company |
1 tok → 1 tok | 1.90 (2.15) | 2.68 (3.14) | +41% | +61% |
$.orders, budget 0 |
1 tok → 1 tok | 16.6 (24.8) | 24.1 (36.1) | +45% | +47% |
$.orders, default budget |
1 tok → 1 tok | 6.03 (6.35) | 9.19 (9.75) | +53% | +63% |
$.orders[*].total, N = 64 |
1 tok + 2 gates → 65 tok | 4.95 (6.63) | 2.47 (3.12) | -50% | -60% |
$.orders[*].items[*].price, N = 64 |
1 tok + 4 gates → 4161 tok | 16.4 (24.3) | 24.8 (26.1) | +51% | +50% |
json_tree_child_count, $.orders |
1 tok → 1 tok | 1.74 (2.13) | 2.75 (3.58) | +58% | +69% |
@> deep object, LIMIT 10 |
– → – | 6.12 (6.29) | 6.20 (6.25) | +1% | — |
-> nested |
– → – | 0.46 (0.54) | 0.48 (0.54) | +4% | +9% |
The noise floor is run 1 against run 2 of the whole 10k matrix in the same format. The median change was under 1%. Above 1 ms, every gates-to-derivable change reproduced within 3.5 points. Below 1 ms, round-trip noise reaches ±20% per call, so read the in-server column there.
… index once The benchmarks of the derivable-token format (#1363) found every read that enumerates slower than the gated walk: whole subtrees by 38 to 46%, a 300-element array by 78%, json_tree_child_count on a small node by 59%, with a per-hit cost inside the walk far above the hashes. None of it was the format. The post-install step pins search_path on every function without the eql-inline-critical marker, and a pinned SQL function is never inlined, so each of the seven chain primitives ran as a full function call, about fourteen per level of the walk, 160 µs a level on the wide corpus. json_tree_hit_token could not inline even unpinned: a FROM clause, then STRICT with a CASE body, then arguments the planner had flattened into expensive expressions. The two maps copied every entry to build and were built twice by child_count. And every reference to the argument decompressed the whole document again. The seven primitives carry the marker. hit_token is FROM-less and not STRICT, and the walk derives a hit's token in the probe beneath it from the working table's columns. json_tree_index replaces json_tree_by_parent and json_tree_by_self: one pass, tag to position, set-returning so it inlines as an aggregate subquery computed once. Each read decompresses the document once into a MATERIALIZED CTE. No stored byte changes. Measured on the benchmark's own pass queries (500 documents per scenario, in-server time per document, interleaved, medians), the SQL as the PR first had it against the bundle built from the tree, on the same rows on one cluster, identical rows from both: read/first_name 0.10 ms -> 0.11 ms +8.9% read/company.name 1.27 ms -> 0.39 ms -69.5% read/user.contact.emergency_contact.name 1.27 ms -> 0.37 ms -70.5% read/orders[2] 4.26 ms -> 0.83 ms -80.5% read/orders[2].status 1.26 ms -> 0.37 ms -70.3% read/company 2.06 ms -> 0.49 ms -76.2% read/user 2.77 ms -> 0.59 ms -78.6% read/orders (fallback) 8.64 ms -> 0.84 ms -90.3% read/orders (walked) 24.03 ms -> 3.57 ms -85.1% read/$ (sealed root) 1.26 ms -> 0.39 ms -69.1% read/$ (root open) 28.21 ms -> 4.20 ms -85.1% wildcard/orders[*].total N=8 2.41 ms -> 0.60 ms -75.0% wildcard/orders[*].total N=64 1.79 ms -> 0.50 ms -71.9% wildcard/orders[*].total N=256 2.81 ms -> 0.66 ms -76.5% wildcard/orders[*].items[*].price N=8 3.57 ms -> 0.95 ms -73.3% wildcard/orders[*].items[*].price N=64 25.03 ms -> 4.08 ms -83.7% children/orders (nested) 1.35 ms -> 0.42 ms -68.6% child_count/orders (nested) 2.39 ms -> 0.31 ms -87.1% children/attrs 12.33 ms -> 5.12 ms -58.4% child_count/attrs 14.27 ms -> 1.20 ms -91.6% read/attrs (walked) 40.74 ms -> 7.22 ms -82.3% children/list 12.59 ms -> 5.08 ms -59.7% child_count/list 14.23 ms -> 1.21 ms -91.5% read/list (walked) 55.61 ms -> 8.22 ms -85.2% The oracle's tamper SQL reads the index's new shape. The Postgres suite (11 json_tree tests), documented-sql and the eight SQL tasks pass. The cipherstash/benches harness still names the two removed maps as floors and moves to json_tree_index before the next full run. Refs #1363 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
What the walk was paying for, and the fix (f94a6eb)The benchmarks of the implementation (cipherstash/benches
Measured on the benchmark's own pass queries (500 documents per scenario,
The gated-walk column is the benchmark's in-server figure for the same scenario, from a different run, and is there for orientation only. Every row the first comparison had as a regression against the gated walk is now faster than it. The Verification on the commit: |
The cap on what one subtree read may send was set where nothing would ever reach it. A wildcard sends its bound plus one tokens under each node it sits below, so two levels at the default bound of 64 send 4,161, the largest read any measurement made. The server looks every token up in every row before it knows whether the budget is blown, so a wide read costs its tokens even when the whole document comes back instead, and the entries-times-tokens LIMIT on the walk scales with the cap. 8,192 admits every one-level wildcard up to 8,191 elements, two levels up to a bound of 90, the default included, and never a third; 1,024 was considered and would have refused the default two-level read. Owner's decision, 2026-10-25, recorded in the design doc with the reasons. The same note records the decision not to chain the sealed root's top-level entries from its token: a walk from the root is a capability over the whole document's shape, so it would add leakage, and it waits for a customer to ask. Refs #1363 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
tests/fixtures/json_tree_vectors.json is written by the pinned-vectors test under STACK_ENCRYPT_UPDATE_JSON_TREE_VECTORS=1, in serde_json's pretty form, and Biome formats every JSON file it is not told to skip: the Lint (Biome) job on #1363 failed with one error, the formatter wanting the fixture's one-element arrays on one line. Reformatting it would be undone by the next regeneration, so it joins the other generated fixtures the config already skips. Refs #1363 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Decisions locked in by the owner (2026-10-25)Review of this PR raised six points that were not in the design's decisions D1–D10. The owner settled them as follows; each is also recorded in
Also on this branch since the review began: f94a6eb (the walk's SQL cost: inlining the chain primitives, one index built once, one decompression per read; numbers in the comment above) and 0ec0c73 (the |
The crates job on #1363 failed on rustfmt alone: one assertion in the cap test fits on a line. No behaviour changes. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…rivable-tokens Brings in the review fixes on #1358 through #1361. Conflicts in `json/derive.rs`, `json/query.rs` and `tests/json_read_checks.rs`, all where this branch replaced the gated and ungated walks with token reads. This branch's `place` and `JsonSubtreeKey::open` are kept: the leading-element check #1358 added belongs to the walk with no gates, which no longer exists here, and every read now returns the node it names with the child count that bounds its children. The truncation fix is carried over by hand: `rebuild_walked` here still refused rows that ran out before a child count was met as `OpenedType`, while the leakage page lists a truncated row among the `JsonWrongEntry` refusals. It is now `JsonWrongEntry`, like a gap. `a_truncated_read_is_refused` now expects that. #1358's ungated-walk tests are replaced by one that pins what this design closes: an array read missing its first or its last element is refused. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
… wrong entry A subtree read's rows were refused as `Error::OpenedType` when the database returned one entry twice or rows carrying two headers, while every other refusal of a row the read did not ask for is `Error::JsonWrongEntry`, and the json_tree leakage page lists a repeated row among those. `OpenedType` says the ciphertext is authentic and was sealed as another shape, "not tampered with"; neither case is that. An honest database never returns either, which is exactly what `JsonWrongEntry` documents, so both now return it, still before any key is retrieved. The two checks sit in `open_subtree` ahead of the per-row placement. The one assertion in tests/json_index.rs that pinned the old variant moves with them (a one-line change; that file conflicts with #1345 anyway), and two tests in tests/json_read_checks.rs pin the new answer and its `expected` text. The error, `JsonSubtreeKey::open` and the CHANGELOG list both. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…d model
`JsonDocument::find`, `children` and `subtree` are the in-crate model of
`json_tree_subtree` and `json_tree_children`, and the read-check tests
build their rows from it. Where a document repeats a self tag or a parent
tag, which no sealing produces but a hostile row can, the model took the
first entry carrying it. The SQL index is built by `jsonb_object_agg`,
which keeps the last value of a repeated key (checked on PostgreSQL 17:
`{"a": 2}` from `('a', 1), ('a', 2)`), so the server finds the last. The
model now searches from the end, and a test appends a copy of an entry
and checks both lookups land on the copy, as the SQL's do.
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…te from MAX_ORDINAL Three rustdoc statements described the format before the derivable-token change or miscounted it: - `MAX_ORDINAL` still justified itself by "a position gate" PostgreSQL would refuse to cast. Gates are gone; a position reaches the server only inside a token, so the refusal now stands on what it says: no stored document has a node there, and the read would find nothing. - `JsonSubtreeKey`, `subtree_key_for` and `json_subtree_key_with_bound` said a key carries a probe per wildcard level. It carries one for each array a wildcard expands: N + 1 for two levels, as `probes()` and the `nested_wildcards_multiply` test count them. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…unds "The bound", the comment every json_tree read function points to, said each read takes its recursion through LIMIT the entry count times the number of tokens. Only json_tree_subtree does; json_tree_children and json_tree_child_count probe one level, find each entry at most once, and take LIMIT the entry count plus one, which is right for them. The comment now names the function and says why the other two need less. Comment only: no function body changes. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…of one row Two gaps on the json_tree leakage page's "What is authenticated": - Withheld rows are an allowed deviation, and the page said a wildcard read with elements left out opens to the rest. It did not say the probe is one of them: a database that leaves out the probe's rows makes an array longer than the bound open as its first N elements rather than fail with `JsonWildcardTruncated`. That follows from the same allowance, and a reader relying on the truncation error should know it. - The client refuses rows carrying two headers in one read (as `JsonWrongEntry` since the previous commit on this branch), which the list of refusals did not name; a whole read answered from another row still opens, as the page says. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…isions Since the owner's decisions replaced the open questions, section 3 still sent the reader to "the open questions" for the fallback's answer to a long array, and stated the walk's limit as the entry count plus one, which decision 9 changed to the entry count times the number of tokens. The status line says the decisions override, but a bound stated twice with two values reads as two answers, so each sentence now names the decision that settled it. Decision 3 also said a probe is sent per wildcard level; it is sent for each array a wildcard expands. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Review of the implementation (as of f42496c)This review covers the derivations, the row check, the SQL, the wildcard expansion, the key's surface, the tests, and the docs and changelogs. It leaves out the owner's six decisions of 2026-10-25, the known non-findings in AGENTS.md (ADR 0010), the format version,
Found and left for the owner (not fixed):
Checked and found sound:
Fixes, and the mutants run, follow. |
…rivable-tokens Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The reversed lookup in JsonDocument::find is past rustfmt's chain width. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…akage-profile Brings in b992e78, #1363's correction to the v3 column of the leakage page's comparison table. It touches a different part of the page from the sentence this branch adds, and no code. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…lement Two sentences the leakage-profile port (#1362) showed to be wrong. The json_tree_subtree comment said the server cannot tell a probe from any other token: it can, since the probe is the last token of its level, the token count gives the bound away, and a hit on it carries the element's selector rather than the leaf's; the executable profile pins this. And the leakage page's q row still called element grouping the one thing v4 reveals that v3 did not, after the comparison table below it was corrected to say v3 reveals it through document order and the a flag. Comments and prose only; no SQL body changes. Refs #1362, #1363 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
cipherstash-agent
left a comment
cipherstash-agent
left a comment
There was a problem hiding this comment.
Automated review attestation: /code-review medium + cross-check-review-findings
Reviewed head b992e787. No critical or blocking issues. One non-blocking new finding.
Scope reviewed: Rust derivation, row check and wildcard expansion (stack-encrypt/src/json, eql-bindings); EQL SQL (json_tree/functions.sql, types.sql, search_path/inline markers, NULL/STRICT, recursion bounds); changeset, fuzz seeds, vectors fixture, docs and removed-API callers.
New finding
- major
packages/stack-encrypt/src/json/derive.rs:605:MAX_DEPTH = 127is off by one for empty containers. An empty[]/{}at depth 127 is the 128th nested container;visitaccepts it (depth > MAX_DEPTH), but the sealed root's JSON is read back withserde_json, which rejects 128 nested containers (verified: 127 nested arrays parse, 128 fail). The seal succeeds and every later open (full decrypt,$read, sealed-root fallback) fails withOpenedType, which theMAX_DEPTHdoc says the bound prevents. Refuse whendepth > MAX_DEPTH || (value is a container && depth >= MAX_DEPTH), and add a test withnested(127)aroundjson!([])andjson!({})(existing test attests/json_index.rs:322only uses a scalar leaf).
Already covered: every other confirmed observation was already raised and handled in the PR's existing review comments.
Not blocking, low confidence: json_tree_sealed_root (functions.sql:189-194) raises "index out of range" on a forged header under 2 bytes (forged rows only); json_tree_iv and json_tree_sealed_root lack the eql-inline-critical marker the comment at functions.sql:478-481 implies (negligible cost).
Limits: SQL was reviewed by reading (no PostgreSQL run); Rust findings by trace plus a standalone serde_json check; existing PR feedback was read via the GitHub API (no review threads existed).
Generated by Claude Code
…xception in the token Answers the two review threads on #1361. The probe thread (open question 3): the question said a separate probes argument, answered with existence only, would remove the leakage exception for a probe that hits. It would not. A probe is the element's token, and whoever holds a token can find that element in any row by its self tag, probe its children's parent tags and derive every descendant's token from the public derivations of sections 1 and 2, whatever the SQL function returns, in rows written later too. The capability is in the token, so the exception holds for every variant that sends it. Only a different derivation, such as a one-way existence value sent in place of the token and a tag stored from it, would be existence-only; that is a format change, recorded as a future option. Section 6 and the leakage page now say the same. The question also said the server cannot tell a probe from any other token, which #1363 corrected in the SQL comment: it can. The consecutive-position thread (open question 1 and section 4): the example given for keeping the check was a swap the check does not see in the form described. The new example is one it does refuse and nothing else does: a read of $.items[0] answered with the element's own entry and then $.items[1].price, its self tag rewritten, which has the right selector, label and child count and the wrong position. The question now also says what the rule does not reach (a depth-0 row, such as a read of $.items[1].price itself). Section 4 no longer says the self-tag check replaces the rule: x is a consistency check, and the bound fields with the consecutive-position rule are the security check, as the leakage page has it. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…rivable-tokens Brings in 0edead7, the answers to the two review threads on #1361: the probe exception is in the token, not in what the SQL does with it, so it holds for every variant that sends the element's token; and the consecutive-position check is part of the security check, which the self-tag check does not replace. The design doc conflicted in section 4 and where #1361's open questions became this branch's decisions. This branch's text is kept in both, and the corrections are carried into it: section 4 says nothing replaces the consecutive-position rule; decision 3 says a separate existence-only argument would not remove the probe exception and records an existence-only derivation as a future option; decision 8 gains the example the rule alone refuses (a read of $.items[0] answered with $.items[1].price) and says the allowed swap is of a depth-0 row; and section 6 points at decision 3. The leakage page merged cleanly. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
|
Merged #1361's branch at 0edead7 into this branch (ba9a87f). It carries the answers to #1361's two review threads: the probe exception is in the token, not the SQL (section 6, decision 3, leakage page), and the self-tag check does not replace the consecutive-position rule (section 4, decision 8, with an example only that rule refuses). Only the two documents changed. |
…catches Design decision 8 keeps the consecutive-position check, and the design record's example for it (corrected on #1361) is a read of one array element answered with the element itself and then the other element's child in place of its own, self tag rewritten: selector, label, depth and child count all agree, and only the row's authenticated position refuses it. No committed test isolated that case; the closest ones are also caught by the child counts or the selectors. This one opens to {"x": 2} for {"x": 1} with the j comparison in rebuild_walked removed. Refs #1363 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…akage-profile Takes #1363's probe-exception sentence on the wildcard bullet and keeps this branch's hit-rule sentence on the token bullet. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…assertions messages After the merge of #1363, json_tree_subtree's comment says the server "can pick it out"; mod.rs and the doc comment of a_read_gives_away_its_bound_and_its_probes still quoted the old claim as standing. Both now say it was corrected. The test's assert_eq! calls gain the messages the rust-testing convention asks for, and mod.rs names the two harness tests that show features no mutant can reach do fire. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The net paragraph said the member-or-element rule "was already disclosed by the gated walk's ordinals and #n: gates". Before #1363 every child's token took one rule, and an only child's ordinal is 0 either way, so a node that is an only child in every row had its kind disclosed by nothing. The note's own finding said so. The paragraph now says what was and was not disclosed, and that the net rests on the ancestor token and the gates. The SQL-comment paragraph records that #1363 corrected the comment, and a closing section records what the review added: the ordering terms' shared prefixes, which both profiles missed, the new checks, mutants and corpus, and what is left to the owner. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
| let rt = runtime(); | ||
| rt.block_on(async { |
There was a problem hiding this comment.
Why not use tokio::test?
There was a problem hiding this comment.
Because these are proptest tests. The proptest! macro generates ordinary synchronous #[test] functions and runs each body once per generated case through TestRunner::run, which takes a plain closure; there is no async form, so #[tokio::test] cannot be applied to a proptest! body, and an async block inside it has to be driven by hand. Hence runtime(): one current-thread Tokio runtime per test function, and block_on around each case's body. The non-proptest tests in the same crate use #[tokio::test] as usual. I'll put that reason in a doc comment on runtime() so it is not a question next time.
… seals
MAX_DEPTH is 127 because serde_json reads a sealed root back to 128
nested containers: a scalar 127 levels down is inside 127 containers and
reads back. An empty [] or {} at depth 127 is itself the 128th nested
container, and the check let it through, so such a document sealed and
then never opened from its sealed root. The review attestation on #1363
found it. A container at MAX_DEPTH is now refused as JsonTooDeep, under
either root option, and a test shows serde_json rejecting the document's
own text before asserting the refusal.
Also: a doc comment on the proptest runtime helper answering why those
tests do not use tokio::test, and the SQL inlining comment now names
which chain primitives carry the marker (not json_tree_iv and
json_tree_sealed_root, read once per call).
Refs #1363
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
|
On the review attestation's finding at |
…akage-profile Carries 8c843ee (the deepest-level container refusal) so #1362 holds #1363's final head before either merges. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Summary
EQL v4's
json_treeis the encrypted JSON column Stack Encrypt writes: one encrypted entry per JSON node, so PostgreSQL can answer containment, path extraction and "give me the subtree at this path" without the key. A subtree read sends the database a token, a keyed hash that lets it find a node's entries and the entries beneath it in every row.Until now the client could only derive tokens for top-level fields, so reading
$.profile.address.citysent$.profile's token, which exposes the structure of all of$.profilein every row, for as long as the column lives, and walked down through "gates". It also could not return the node itself, so{},[], a scalar and a missing field all came back as "no rows" (#1356).This PR implements the agreed design change (the design-doc section "Design change (2026-10-24)" on this PR's base, #1361): every token is now derivable from the path alone, every stored entry carries a self tag
xby which the database finds a node, and wildcards are expanded on the client into concrete paths up to a bound. A read now sends exactly the token of the node it reads, returns that node, and answers the four cases the wayjsonbdoes. The owner's decisions D1–D10 are folded into the design doc as "Decisions (2026-10-24)".Changes
Stack Encrypt (
packages/stack-encrypt)H16("t" ‖ t(parent) ‖ s)(no ordinal); element token keeps its index;x = H8("x" ‖ iv ‖ t)on every entry, the root and heads included. The parent tagpis unchanged and used for enumeration only. A wire entry is{s, c, x, p?, op?}.JsonHeader::FORMAT_VERSIONandvare unchanged (D5).JsonSubtreeKeyis a list of tokens: one per concrete path, plus a probe per wildcard level at the boundN(default 64,JsonSubtreeKey::DEFAULT_BOUND, set per read withKeysetCipher::json_subtree_key_with_bound).tokens(),token_bytes(),probes(),bound(). Gates are gone.Value,Option<Value>(Nonemeans no node) andVec<Value>(one value per concrete path). A probe hit fails with the newError::JsonWildcardTruncated { bound }, walked or via the sealed-root fallback (D3, D4).jwith the parent's path's selector plus their label; every row'sxagainst the recomputed token (a consistency check only, and documented as such). Thepcheck is dropped.$is a read under the sealed root (D6);TermError::JsonSealedRootis removed (it had no other use); the newTermError::JsonWildcardTooWiderefuses a read past 8,192 tokens (the owner lowered it from 65,536 in f3400a2).JsonDocument::find/children/subtree(&[tokens]), the in-crate model of the SQL, follow the new rules.x, the stale mutants exclusion for the removed(0, true)arm dropped,proptestadded as a dev-dependency (already in the lockfile).EQL SQL (
packages/eql/src/v3/json_tree)eql_v4.json_tree_subtree(e, tokens text[], budget integer DEFAULT 64);json_tree_subtree_atremoved (D9). The budget counts rows across all tokens; the recursion'sLIMITis entries × tokens, and the comment "The bound" is rewritten.json_tree_children(e, token)andjson_tree_child_count(e, token)losegates; the count is NULL for an absent node and for the sealed root (D7).json_tree_self_tag,json_tree_member_token,json_tree_element_token,json_tree_hit_token(the member-or-element choice byx, D2), andjson_tree_index, which maps every entry'sxand every chained entry'spto its position in one pass;json_tree_child_tokenandjson_tree_by_parentremoved. The chain primitives andjson_tree_indexcarry theeql-inline-criticalmarker, and each read decompresses the document once (f94a6eb). The entry CHECK requiresx.eql-bindings:
JsonTreeEntrygainsx;JsonTreeSubtreeopens toOption<Value>; docs and CHANGELOG.Docs: the design doc section gets "Decisions (2026-10-24)"; the leakage page's read bullets and "What is authenticated" are updated, including the two allowances (another row; the same selector under another element) and that a database can withhold rows.
Changeset:
@cipherstash/eqlminor.Verification
From the worktree root:
mise x --env test -- cargo nextest run -p stack-encrypt --all-features→858 tests run: 858 passed, 1 skipped(includes the trybuilduibinary; re-run by the review on 2026-10-25)mise x --env test -- cargo test -p stack-encrypt --doc --all-features(whattest:doc:stack-encryptruns) →50 passed; 0 failedcargo clippy -p stack-encrypt --all-features --all-targets -- -D warnings→ exit 0cargo fmt --check→ exit 0 (root andpackages/eql)RUSTDOCFLAGS="-D warnings" cargo doc -p stack-encrypt --no-deps --all-features(whatdoc:stack-encryptruns) → no warningspnpm run test:scripts(.cargo/mutants.tomlchanged) →Test Files 81 passed (81),Tests 1432 passed | 29 skipped. A first run alongside the mutants job timed out two unrelated tests at 5 s; the rerun on an idle machine passed.mise run fuzz:json-wire -- -runs=0(seed replay, as CI runs it) →Done 6 runs, no crash, with the regenerated seeds.From
packages/eql:cargo test --locked -p eql-bindings --all-features→ every suiteokmise run test:encryption→ exit 0 (both feature sets, the encryption tests, both clippy runs)mise run test:encryption:postgresagainst a disposable PostgreSQL 17 container on port 5455 → exit 0; thejson_treesuite11 passedmise run types:check→ exit 0 (run after committing: before that it diffs the hand-writtenjson_tree.rs)mise run docs:validate:source,test:docs_v3_grep,test:self_contained_v3,test:installer_complete,test:symbol_order_v3,test:v4_isolation,test:build_ordering_helpers,test:doc-anchors→ all exit 0tasks/docs/validate/documented-sql.shagainst the same container →Validated: 290 Errors: 0Mutants:
mise x --env test -- cargo mutants --in-diff <diff vs docs/json-tree-derivable-tokens> -p stack-encrypt -j 4(settings from.cargo/mutants.toml; a cold build in a fresh worktree) →132 mutants tested in 40m: 1 missed, 91 caught, 40 unviable. The one miss (next = at + 1→at * 1inplace, letting two rows match one token) is killed byone_token_answered_twice_is_refused; applying the mutant by hand fails that test. Re-run by the review on 2026-10-25 at 4ce771c's code:134 mutants tested in 39m: 94 caught, 40 unviable, none missed.Not run: benchmarks (step 3 of the plan, for a separate agent;
check:v4:json_treein cipherstash/benches calls the gated walk and has to move to the token list). No new fuzz target: the existingjson_wire_decodealready fuzzes theJsonDocumentandJsonNodedecoders the client runs on every row the database returns, and is wired intofuzz:json-wireand the CI replay. A target for the row check itself would need a key source inside the fuzz crate.New tests, by what they hold:
tests/json_tree_vectors.rs, fixturetests/fixtures/json_tree_vectors.json):s,xandpper entry, and the tokens for the root, a head, a member, an array, an element and a member under an element, sealed and open, under the deterministic key source. Regenerate only withSTACK_ENCRYPT_UPDATE_JSON_TREE_VECTORS=1(exactly1).tests/json_properties.rs, proptest, and unit proptests inderive.rs): a member's token is independent of its siblings; element tokens are distinct by index and from every member (keys include"","0",.,[*],],$, quotes, a backslash and unicode); every path's read equalsselect, and the reads reassemble the document; a read then a step equals the longer read, and$(walked, or sealed and opened whole) plus any path equals that path's read;->equals the read for every scalar;childrenequals the child count, and is empty under a scalar; selectors, layout and tokens are independent of values and iv, with everyx,pandcdistinct;xon every entry andpabsent only on the root and heads; a key sends exactly the tokens of the concrete paths (checked against a witness document through the stored self tags), the probes, and no ancestor's token; the serialized key is exactly those tokens.tests/json_read_checks.rs): every refusal of D8 (wrong depth-0 selector, wrong label, non-consecutivej, wrongxat depth 0 and deeper, a splice, a gap, an orphan, a wrong depth, truncation, the root among rows, one token answered twice, one entry returned twice, rows of two documents), the two allowed cases, a 127-deep document, and the four answers asValue,Option<Value>andVec<Value>.N − 1,NandN + 1forN= 1, 3 and 64, walked and fallen back; nested wildcards sendN² + N + 1; the expansion limit.xadds +9.8% characters on the nested benchmark shape and +9.4% on the flat one; the bounds are 8–12% and 7–12%.packages/eql/tests/encryption/tests/json_tree.rs):postgres_every_subtree_read_answers_as_jsonb_does(ported from fix(eql): return the node itself from every json_tree subtree read #1359 and extended with nested empties, degenerate roots, the stress keys, numbers,nullagainst absent, a 300-key object and 40-level nesting; sealed and open; budgets 0, 1 and 64;Value,OptionandVec;childrenandchild_countagainstjsonb_object_keysandjsonb_array_length, NULL included;->against the read);postgres_wildcard_reads_answer_as_jsonb_path_query;postgres_reads_never_answer_wrongly_under_tampering(120 random rows, nine kinds of tampering; observed 512 refusals, 1,402 right answers, 98 withheld); the hostile tests rewritten for tokens andx(a forgedx, another position, a moved node, swapped entries, truncated and reordered rows, the two allowances); andpostgres_walks_do_not_multiply_on_forged_member_tags(new) beside the duplicated-entries and deep-chain tests.Rewritten rather than kept: every test that used gates,
json_tree_subtree_at,JsonSubtreeKey::token()orgates(), or expected a read to return only a node's descendants;the_sealed_root_has_no_token_and_its_fields_are_heads(the sealed root has a token now) anda_position_past_postgres_integer_is_refused_not_sent(no position reaches the server as an integer any more; the refusal stays, since no stored element sits there).Related
Closes #1360
Closes #1356
Refs #1358
Refs #1359
Refs #1357
Review notes
placeandrebuild_walkedinpackages/stack-encrypt/src/json/derive.rs(the row check),subtree_key_for(the expansion), andjson_tree_subtreeinfunctions.sql.TermError::JsonWildcardTooWide(at most 8,192 tokens per read, the owner's cap) is not in D1–D10; without it, three nested wildcards at the default bound would send 266,305 tokens.json_tree_child_countof the sealed root is NULL, not 0: its members chain from nothing, so its token cannot count them, and 0 would be a wrong answer that looks right.None, the database withholding rows. Nothing in a row says how many nodes a read should find, so a read answered with no rows opens as absent, and a wildcard read can lose elements the same way. The leakage page now says so. Closing it would need an authenticated count, for example reading the array node's child count beside a wildcard.LIMITis entries × tokens rather than entries + 1: forged member tags can make one token's subtree be visited more than once. D9 asked to keep the per-row bound and reword it; this is that bound restated for many tokens.jsonb_path_query_firstper concrete path rather than one strictjsonb_path_query: strict mode aborts the whole query when one element does not fit a step (.xon a number), where the index answers per element.float_roundtripcan return anf64one unit in the last place off (-926809.5051008083came back as…085). The new property tests found it, and now draw exactly representable floats; filed as json_tree: a stored float can come back one ulp off because numbers are re-parsed without serde_jsonfloat_roundtrip#1364.https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu