feat(stack-encrypt): cascade the JSON tree's AEAD tags and store its entries in a random order by coderdan · Pull Request #1379 · cipherstash/stack

coderdan · 2026-10-26T02:11:03Z

Summary

EQL v4's json_tree stores an encrypted JSON document in PostgreSQL as one small AES-GCM ciphertext per node, plus search terms the database can match without the key. This change makes the stored form reveal less and makes the integrity check independent of storage order:

a document's entries are stored in a random order (the root first), an object's members get random ordinals, and the containment selectors are stored and sent in a random order, so a stored row no longer shows each row's tree, the size of any subtree, or the order of sibling key names (json_tree: the stored entry order reveals each row's shape and the alphabetical order of sibling keys #1357);
every node's ciphertext commits to its children's ciphertexts (the cascade), so a client reading part of a document can tell when the database has swapped, dropped, added or reordered anything beneath what it asked for, whatever order the rows arrive in.

The format is unreleased, so there is no migration: documents sealed by the earlier build do not open. Nothing stored changes shape; one SQL function (json_tree_children) returns a node's children in ordinal order, where it returned them by stored slot.

Changes

Stack Encrypt (packages/stack-encrypt)

The cascade (src/json/cascade.rs, place in derive.rs). Each entry's AAD binds the SHA-256 digest of its children's AEAD tags, in ordinal order. Sealing runs leaves up; a read places the rows by the chain, as the server finds them, and opens each node over the children placed under it. The module's rustdoc carries a diagram.
Random permutations (src/json/shuffle.rs). One Randomness source per cipher draws, per document or query, a SafeRand seeded from the operating system; the shuffles are rand's SliceRandom::shuffle (the rand vitaminc-random wraps). The stored slots (root at 0), each object's member ordinals (member_ordinals), the stored selector set q and a query's selector set are all drawn from it. The deterministic stream for fixtures (StackCipherBuilder::json_seed) exists only under the test-support feature.
Names. JsonProjection is JsonSelectorSet (the selector pairs of a document's nodes, grouped by array element: what a row stores as q and what a containment query sends), with OpaqueDebug and constant-time set equality through vitaminc's TimingSafeEq; JsonDocument::selector_set and JsonQuery::selector_set. JsonNode::slot replaces ordinal. On the read side a ReturnedRow, an ExpectedSelector and a ChainWalk say what they hold, and entry_aad takes key_tag, a JsonSelector and slot.
Owned documents are Protected<PlainJson>, the crate's existing zeroizing newtype, wherever the index owns one (sealing, the query checks, the owned-source release).
Keyset load on a root-open decrypt. Opening a whole document whose root is open derives the root's token under the keyset it was sealed under, so the pending asks for the keyset beside the data key (Request::load_keyset, answered through the cipher's own load_keyset path). The loads run beside the data-key retrieval, and a load that fails is Error::KeysetLoad on the documents under that keyset alone. Awaiting a Pending now bounds K: DataKeySource + IndexKeySource, which every StackCipher already has.
-> on a container with members is Error::JsonNotALeaf, not Error::Aead: the entry does not open alone (its ciphertext commits to children -> does not return), and the client cannot tell that from an altered entry before the open, so the error names both causes and claims neither. The Go guest maps it to STATUS_ENCODING.
Heap hygiene. Key names, scalar values and data keys are wiped before freeing; tests/heap_hygiene.rs holds it, and now zeroes what it allocates and scrubs the stack before each watch, so it reports what the watched operation itself wrote to the heap.
Tests: hostile-row tests assert the exact error; proptests cover the slot permutation, the member ordinals and reassembly under them; the pinned vectors (tests/fixtures/json_tree_vectors.json) are regenerated and the tags and tokens are unchanged; the leakage profile's simulator draws per-row member orders.

EQL (packages/eql)

json_tree_children orders by the probe's ordinal (ORDER BY p.o), so an array's elements come back in position order; three doc comments stop promising document order; the PostgreSQL oracle compares the elements' order unsorted and asserts JsonNotALeaf for a moved entry read through ->.
docs/reference/json-tree-leakage.md opens with what the database needs to answer queries and that the values stay secret, says json_tree reveals strictly less than ste_vec, and records what a read proves and what a token's holder learns from random ordinals.

Go guest (languages/golang/encrypt/guest): the generic operations bound IndexKeySource too; the status map covers KeysetLoad (as its cause) and JsonNotALeaf.

Docs: docs/plans/2026-10-17-json-tree-index.md (the 2026-10-26 notes), ADR 0010 (the -> exception under a restored Consequences heading), the stack-encrypt changelog and a @cipherstash/eql changeset.

Verification

Run locally on this branch (b540fa0):

cargo fmt --all --check: clean.
RUSTFLAGS=-D warnings cargo clippy --workspace --all-features --tests and --workspace --tests (default features): clean.
mise x --env test -- cargo nextest run --workspace --all-features: 1496 run, 1496 passed, 1 skipped. tests/heap_hygiene.rs passes repeatedly (six consecutive runs of the binary).
mise run test:doc: 50 doctests in stack-encrypt pass; mise run doc: clean.
packages/eql: mise run test:encryption: 526 passed, 0 failed. mise run test:encryption:postgres against a fresh PostgreSQL 17.10 database with the v4 bundle built from the tree: every suite passes, the json_tree oracle 14 of 14 (42 PostgreSQL tests in all, 0 failed).
Go: mise run wasm:guest:build, wasm:guest:build:eql, both deterministic variants and wasm:auth-guest:build build; mise run go:test: every package passes (11 ok, none failed).
cargo metadata --locked passes in the root workspace, both Go guest workspaces, the EQL workspace and the stack-encrypt fuzz crate (each gained rand, and the root futures).
cargo mutants --no-shuffle -vV -p stack-encrypt --in-diff <this pass's diff> --jobs 4: 124 mutants, 62 caught, 62 unviable, 0 missed, 0 timed out.

Review notes

Start with cascade.rs (the diagram), then seal and place / ChainWalk::node in derive.rs, then shuffle.rs.
vitaminc-permutation was considered for the shuffle: its PermutationKey<N> is defined for N of 8, 16, 32, 64 and 128 only, and a document has any number of nodes, so the shuffle is rand's (the one vitaminc-random wraps). A slice shuffle in vitaminc-random would let shuffle.rs call that instead.
-> at a wildcard path (any path with an array step) returns an unspecified element of the row, the same one every time that row is read; every element is reached through json_tree_path_query or a subtree read.
Not done, deliberately: upward verification of a read's top node (ADR 0010's allowances stand).
Benchmarks were not re-run: nothing stored changed shape, and the one SQL change is an ORDER BY over rows already computed.

https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

…n-list merge The test module #1366 added to `tests/json_index.rs` was written against the gated walk: it called `subtree_nodes(doc, token)`, which the token-list read (#1363) replaced with `read_rows(doc, key)`, and opened an extracted element with `decrypt_as::<Value, JsonNode>`, which has no implementation (an extracted entry opens through `JsonExtracted`, which checks its selector first). Merged on top of #1363 the file no longer compiled, and the `tests-crates` workflow is red on `next` at ebdb5de. Both calls now use the helpers the rest of the file uses. The test's assertions are unchanged. Refs #1366, #1363. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

…its AAD A json_tree read used to be checked by position: the rows under the node a token named had to sit at consecutive slots in depth-first order, and each entry's AAD bound its slot. That rule is sound only while the stored order is the depth-first order, and the stored order is what reveals each row's tree (#1357). To store the entries in any other order, the client needs an integrity check that does not read the slot as a position. Each entry's AAD now binds the digest of its children's AEAD tags, SHA-256("k" ‖ tag(child_0) ‖ tag(child_1) ‖ …) in ordinal order, under the new domain json/entry/v2. A node's tag covers its AAD, so it covers its children's tags, which cover theirs: a node commits to its whole subtree. Sealing runs from the leaves up. A leaf, an empty container and the sealed root (whose members chain from nothing) open under the empty digest. Nothing stored changes: the AAD is not stored and the tag is already the last sixteen bytes of each ciphertext. Opening a read places its rows by the chain, as the server finds them: the node a token names by its self tag, a node's children by its parent tags at ordinals 0, 1, 2, … Each node then opens under the digest of the children the tags gave it, so a row swapped, dropped, added, reordered or moved under another parent is refused by the cipher (Error::Aead). The client still refuses, as JsonWrongEntry, a row placed twice, a row no read reaches, a reported depth the chain does not give, a selector or label that is not the path's, a child under the wrong rule, the sealed root anywhere but at `$`, and a container whose child count disagrees. The consecutive-position rule is gone. Opening a whole document whose root is open walks the chain the same way, so it derives the root's token; for now it takes the keyset from the cipher's cache. `->` of a container with children no longer opens alone (its AAD covers tags `->` does not return) and is refused by the cipher where it was refused by kind. The new code lives in json/cascade.rs (the digest and the seal order) and derive.rs (`place` and `Placing`). The hostile-row tests that asserted the position check's errors are rewritten in a later commit. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

A json_tree row stored its entries depth-first, root first, members in key byte order. Every entry carries its path selector, so the stored order alone gave anyone holding the table each row's tree: the parent of every node, the size of every subtree, and which of two sibling key names sorts first (#1357). The containment projection q restated it. With the cascade (the previous commit) nothing reads a slot as a position any more: the server finds entries by tag and reads q as sets, and the client places a read's rows by the chain and verifies them by the tags. So the entries are now stored in a uniformly random order per document, with the root kept at slot 0 where the sealed-root fallback looks for it, and q's pair lists and element groups are shuffled too. The AAD still binds each entry's slot. This is the format, not an option: no header flag, no data-plan option, no switch. The permutation is drawn at sealing time and thrown away; nothing derives it, keyed or not, because nothing needs it and anything it were derived from would have to be kept from the database. Production draws from vitaminc's SafeRand::from_entropy, one generator per document, Fisher–Yates over slots 1..n with next_below (Lemire's reduction, bias at most n / 2^64). The generator is not part of the format: nothing stored depends on it, so it can be replaced without a version change. For fixtures, `StackCipherBuilder::json_order_seed` (behind the `test-support` feature) draws every document's order from one generator started at a fixed seed (SafeRand::from_controlled_seed), and the test helper that builds a cipher over the deterministic key source sets it, so a pinned vector repeats. The one SQL meaning that moves: `->`, `->>` and ordering by `eql_v4.ord_term(col -> sel)` at a wildcard path return the first entry in the stored order carrying the selector, now an unspecified element of the row (the same one every time) rather than element 0. Every element is reached through all its matches; the rustdoc of the extraction says so. Tests that pick an entry by its depth-first slot are rewritten in a later commit. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

…h its root open Under the cascade, opening a whole JSON document whose root is open (`sealed_root: false`) walks the document's chain from the root's token, as the server would for a read of `$`, so the client needs the index key of the keyset the document was sealed under. The cascade commit took it from the cipher's keyset cache and refused when the keyset was not loaded, so a document sealed under a tenant keyset failed to open through a cipher that had not selected that tenant first. The open now asks for the keyset beside the data key, as a new kind of request a Pending carries (`Request::load_keyset`). Settling answers it the way `StackCipher::keyset` selects a keyset by id: from the cache, or loaded from ZeroKMS and cached on a miss. The request is held to a pending's keyset scope like a retrieve, so a keyset cipher refuses a document of another keyset before any I/O. Settling a pending asks only for a DataKeySource, and adding IndexKeySource there would tighten public bounds across the crate and its bindings. So the cipher captures the backend's `load_index_key` at `init`, where `K: IndexKeySource` is known, as a plain function pointer returning a boxed future (Send on native, as every pending's future is), and the keyset request goes through it. No public bound changes. The sealed root (the default) opens from its root alone and loads nothing. Tested with a reader cipher over the deterministic key source that has loaded only its default keyset: it opens an open-root document of another keyset, loading that keyset once and looking it up after. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

…fusal comes from With the entries stored in a random order, a test can no longer name a node by its depth-first slot. Every test that did now finds the row by its path, through the self tag of the path's token as the server does: the hostile-row tests (tests/json_read_checks.rs), tests/json_index.rs, tests/json_properties.rs, and the PostgreSQL oracle (packages/eql/tests/encryption/tests/json_tree.rs, whose two slot assertions now compare as sets, since the SQL returns a token's rows by slot). `q` and the selector lists compare as sets. Each hostile-row test asserts the error the new code produces, not either of two. Beneath the node a read names, a swapped entry, a dropped child, an added child, reordered children, a child moved under another parent, a truncated subtree and another column are refused by the cipher (Aead); a repeated row, a row no read reaches, a wrong depth, a self tag no rule derives (or not the one the label names), the wrong selector or label at the top of a read, the sealed root out of place and a wrong child count are JsonWrongEntry. Tests that forge rows compute the chain's public hashes the way a database holding the read's tokens can. New properties: a read's rows open to the same value in any order; any one row of a whole-document read changed, dropped, repeated or misreported is refused with the error that names how; the stored order is a permutation with the root at slot 0, and every seal draws its own. The pinned vectors (tests/json_tree_vectors.rs, behind `test-support`) now hold, per root option, the entries in their stored order, the path stored at each slot (the permutation the seeded generator draws), the selector multiset, `q` as sets and the tokens. The tags, the iv and the tokens are unchanged from the previous pin; only the order moved. The leakage profile (tests/json_leakage) models the random order: a new `StoredOrder` clause, a simulator that draws the same permutation, and a canonical order (canonical.rs) that every distinguisher reads the world in, built only from what an observer sees without the stored order. Distinguishers that read the depth-first order are retired; a mutant claiming that order is caught by `walked_subtrees_are_contiguous`. It found that a containment needle is still sent in key byte order, which the leakage page now states. Rewriting the tests also tightened the placement in derive.rs: rows of two documents are refused before repeated slots (two documents' entries can share a slot); a re-entry check that no key's disjoint paths can reach is gone; and unit tests cover the whole-document chain walk, a forged child count and another column. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

The design record gains a dated note on the change (#1357): the cascade as the integrity rule and which refusal comes from where, the random stored order as the format and why it is derived from nothing, the wildcard first-match semantics the owner chose (an unspecified element, with the element-0 fix-up and the sealed root's heads digest considered and dropped), the keyset load on a root-open decrypt, and the two SQL doc comments that still say "document order". The leakage page drops its stored-order row, says what a random order leaves visible (the selector multiset, q's element grouping, the head count, ciphertext lengths and the ordering terms) and what it no longer shows, rewrites "What is authenticated" around the subtree guarantee with the three allowances kept, says what `->` returns at a wildcard path, and notes that a containment needle is still sent in key order. ADR 0010 says what a json_tree document authenticates and keeps the three allowances: another row's entry, another element's node at the top of a read, and withheld rows. The json module's rustdoc and eql-bindings' JsonTreeDocument doc stop describing a depth-first order. A changeset records the change for @cipherstash/eql. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

changeset-bot · 2026-10-26T02:11:10Z

🦋 Changeset detected

Latest commit: f783f8e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@cipherstash/eql	Patch
stash	Patch
@cipherstash/stack-prisma	Patch
@cipherstash/basic-example	Patch
@cipherstash/e2e	Patch
@cipherstash/prisma-example	Patch
@cipherstash/stack-drizzle	Patch
@cipherstash/stack-supabase	Patch
@cipherstash/stack	Patch
@cipherstash/wizard	Patch
@cipherstash/bench	Patch
@cipherstash/test-kit	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

coderabbitai · 2026-10-26T02:11:12Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI; Enforced overrides: Organization
Review profile: CHILL
Plan: Advanced
Run ID: dff1ab82-91c6-4a2a-9fa9-7cfcaa2cc9ea

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-26T02:32:50Z

Mutation testing (cargo-mutants, `--in-diff`, stack-auth + stack-encrypt)

caught	missed	unviable	timeout
169	0	110	0

Every mutant in the changed lines was caught by a test.

…s sets From the review of #1379: - A keyset request is now answered once per keyset in a batch, however many documents ask: a column of root-open documents under one keyset costs one cache lookup, or one ZeroKMS load on a miss, not one each. - A load by id that ZeroKMS (or a custom source) answers for another id is refused as KeysetMismatch. It would otherwise walk the document under another keyset's PRF and cache the state under an id no later lookup asks for. - The three places that built a KeysetState from an index key share one constructor. - JsonProjection's equality is now as sets, the way `jsonb @>` reads a projection: two seals of one document carry projections that differ only in their shuffled order, and compared unequal under the derived equality. - Opening a whole root-open document moves its entries into the rows instead of cloning every ciphertext. - `no_children`'s rustdoc said the sealed root commits to its heads in its plaintext, which this change does not do. It commits to nothing beneath it; the heads are each checked as the top of a read. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

coderdan · 2026-10-26T02:33:19Z

Review (/code-review high, posted inline above), and what was done with each finding, in 7690219:

-> of a container is Aead, not OpenedType (derive.rs). Not changed. This follows from the cascade: a container's AAD covers its children's tags, which -> does not return, and the client cannot see the node's kind before it opens. The owner accepted the change of error ("changes an error, not a capability", in the report on json_tree: the stored entry order reveals each row's shape and the alphabetical order of sibling keys #1357); a container's value is read with a subtree read. The rustdoc of the -> decryption says so. For the Go mapping (STATUS_AUTH), worth a note when feat(go): generate encrypted JSON document fields #1374 rebases.
Keyset loads per request, sequential, batch-wide failure (pending.rs). Fixed in part: keyset requests are now answered once per keyset per batch (a test holds one load for two requests, then the cache). The load still runs after the retrieves rather than alongside them, and a failed load fails the batch, as a failed retrieve does; both only arise on a root-open document under an uncached keyset, and the sealed root (the default) loads nothing.
no_children rustdoc claimed the sealed root commits to its heads. Fixed: a leftover from a variant the owner withdrew. It now says the sealed root commits to nothing beneath it.
Needles are not shuffled, so a query shows sibling key order. Fixed in 810852c, at the owner's request: every needle is shuffled per query (see the follow-up comment below).
SQL doc comments still say "document order". Not changed: this change makes no SQL edits, by the owner's scope. Recorded in the design note and the PR body as a follow-up.
keyset_state trusted the id load_index_key returned. Fixed: an answer for another id is refused as KeysetMismatch and nothing is cached (test added).
JsonProjection's derived equality compared order. Fixed: equality is now as sets (pairs and groups, recursively), with a test; JsonDocument equality follows.
open_whole_chain cloned every entry. Fixed: the entries move into the rows.
Three copies of the KeysetState construction. Fixed: one keyset_state_of. The two cache miss/insert flows stay separate because StackCipher::keyset resolves names and handles name-binding races, which the by-id load does not need.

cargo clippy -D warnings and the stack-encrypt suite (920 passed) are clean after the fixes.

https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

Three doc comments in functions.sql said rows come back in document order. Under the random stored order the path query returns entries in stored order, which says nothing about the document; the children walk returns them by ordinal, which is position for an array and keyed rank for an object; and the subtree walk promises no order at all, since the client places every row by its tags. Comments only; no function changes. Refs #1357. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

With the stored order random, the client's containment and equality needles were the last thing in key order: built in the walk's order, an object's members in their keys' byte order, so a query with two sibling keys told the database which sorts first, at selectors it also sees in the stored rows. The leakage model found it. `JsonQuery::of_terms` and `at_path` now shuffle the needle's projection as a stored `q` is shuffled: every pair list and element-group list in a uniformly random order, recursively, with a fresh permutation per query. The seed is drawn from the cipher's OrderSource when the query's terms are derived (`terms_pending`), so it is the operating system's entropy outside tests and the seeded stream under `test-support`, and it travels in `JsonTerms` until the needle is built. `jsonb @>` is set containment over arrays, so no answer changes; `JsonQuery` equality is already as sets, and two queries for one value now serialize differently and compare equal. Nothing in SQL, the header or the data plan changes. Tests: two needles for one value under the entropy source differ in order and are equal as sets; a proptest over the walk's projections shows a shuffled needle matches exactly the stored projections (in either order) the walk-order needle matched; the generated containment property derives every needle twice and holds both to `jsonb @>`. The PostgreSQL json_tree oracle passes unchanged. The leakage model gains a `SentOrder` clause; a mutant claiming needles go in the walk's order is caught by `repeated_needles_sent_alike`, and the key-order reading test now asserts that nothing the database sees tells the two readings apart. The leakage page, the design note and the changeset say so. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

coderdan · 2026-10-26T04:24:57Z

Follow-up on finding 4 of the review (needles not shuffled): this is no longer declined. At the owner's request, 810852c shuffles every containment and equality needle with a fresh permutation per query, from the same OrderSource as the stored order (pair lists and element groups, recursively). jsonb @> is set containment over arrays, so no answer changes, and the PostgreSQL oracle's containment results are unchanged. The leakage page, the design note and the leakage profile now say a needle shows no more of its keys' order than a stored row does.

https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

chatgpt-codex-connector · 2026-10-26T05:12:19Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-26T05:14:06.492903Z	`810852c`	Draft marked ready
🔒 Security Review	✅ Completed 2026-10-26T05:18:00.098059Z	`810852c`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

cipherstash-agent

Review attestation: Claude Code /code-review medium (two parallel reviewers: cascade/order/derive; cipher/pending/query/docs) at head 810852c, cross-checked with company-skills:cross-check-review-findings against the 9 existing threads and 5 PR comments. Findings were verified against head code.

Critical / blocking: none. The cascade digest encoding, Fisher–Yates shuffle, panic/allocation paths, needle seed entropy and keyset dispatch checked clean.

New (all minor, non-blocking):

derive.rs:1256 — root-token diagnostic now always yields Error::Aead for a container root (inline).
cipher.rs:652 — by-id keyset load can overwrite a named cache entry with a nameless state (inline).
sem/mod.rs:350 and json/query.rs:762 (unchanged files/lines, so not inlined) — JsonOrderingAtPosition rustdoc and miette help still say -> compares "the first element at the path". After this PR it is an unspecified element at [*] paths. Matches the wording already used in the changeset and leakage page.
docs/plans/2026-10-17-json-tree-index.md:504 — "Open for the owner: the stored entry order" is not marked superseded by the 2026-10-26 notes.

Already covered by existing threads (dropped): -> of a container → Aead; keyset load sequencing and batch-wide failure; keyset id mismatch; JsonProjection equality; open_whole_chain clone; KeysetState duplication. Open thread on the SQL "document order" doc comments is unchanged (the author declined it as out of scope).

Not run: build and tests (the PR reports them green; the mutation report shows 0 missed).

Generated by Claude Code

…keys before freeing them The data-plan work in #1372 found that sealing a JSON document freed key names, scalar values and the data key unwiped, and left it for the index work. A heap spy over this branch's seal, open, read, extraction and needle paths, under the deterministic key source, found 42 blocks for one sealed-root seal (43 with the root open), 4 for each whole open and each `->`, 7 for each subtree read, 40 for a containment needle and 20 for an equality needle. Each holds a marker the document carries or the data key the seal used. After this change every one of those paths frees none. Where they came from, and what holds them now: - JsonPath carries key names, and the walk builds one per node: a path now wipes its key names when it drops, and `selector_bytes` and `token_bytes` are sized before they are written (a growing buffer freed a copy) and come back in `Zeroizing`. - The value selector framed the scalar, and the rank framed the key name, through `Context::pae`, whose buffer is not wiped: both now frame into a sized `Zeroizing` buffer with the same bytes (a unit test holds the equality), and the canonical value is `Zeroizing` too. The rank map no longer clones key names. - The document `seal_pending` owns (a clone, in borrowed mode) is now wiped when sealing returns (`wipe_value`); a query's needle, owned by the terms operation, is wiped through a new `SourceMode::release`; the copies `AtPath` checks make are wiped after the check. - The data key was moved out of the response buffers (`pop_front`, `drain`, `into_iter`), leaving its bytes in memory freed later. Each response is now handed out as a clone and the original drops in place, where `DataKey` wipes itself; `dispatch` clones retrieved keys the same way. This is the shared settling path, so every decryption benefits. `tests/heap_hygiene.rs` gains the JSON index's tests, with a key source that records each data key so the spy looks for it as well as the marker, and a control showing a freed key copy fails the watch. Each fix was checked by removing it and watching a test fail. One case is left out and written down: the path an `AtPath` caller passes as an owned `String` is cloned by the query builder and dropped by the plan chain, which knows only `P: AsRef<str>`. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

The EQL workspace, the Go encrypt guest and the stack-encrypt fuzz crate resolve stack-encrypt by path, so each lockfile gains the one `rand` line; `cargo metadata --locked` passes in all three again. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

…k hold `entry_aad` took a `tag` that was a byte slice (the data key's ZeroKMS tag), an `s` of type `Tag` that is a selector, and a `j`; it now takes `key_tag`, a `JsonSelector` and `slot`, and its rustdoc says where each comes from. The entry AAD domain is `v1` again: nothing of this format has shipped, so there is no earlier derivation to tell apart. `retrieval` and `retrieval_of` are one `key_retrieval` whose rustdoc says what it builds from the header and the context. On the read side a `Row` is a `ReturnedRow` with a `slot` and a `depth`, `Under` is the `ExpectedSelector` a node must carry, `Placing` is the `ChainWalk` the client runs over a read's rows, and `JsonNode::ordinal` is `slot`, which is what it returns. Sealing asserts the pre-order it relies on. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

…he cipher's keyset path A keyset request was answered through a function pointer the cipher captured at `init`, with its own boxed-future aliases per target, so that settling a pending could keep asking only for a `DataKeySource`. Every `StackCipher` is built over a source that is also an `IndexKeySource`, so the settling path now says so and `keyset_state` calls `load_keyset`, the function `StackCipher::keyset` loads a keyset with; the pointer, the aliases and the second constructor go. The `LoadKeyset` request kind stays, and its definition says why a request rather than a call: the open that needs the keyset runs inside a pending's fulfilment, after dispatch, which is the one place a pending reaches ZeroKMS. The load's future is boxed on its own, so the pending's boxed future does not grow by it: a generator is copied into its box with the bytes it does not initialise, and the heap-hygiene spy reads every freed block whole. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

…nly the documents that need one A batch that opened root-open JSON documents under an uncached keyset waited for `retrieve_keys`, then loaded the keyset in a second round trip, and a load that failed failed every fulfilment in the batch, scalar decrypts included. `dispatch` now runs the retrieves and the keyset loads together (`futures::future::join`, one `keyset_state` per distinct keyset through `join_all`), and a keyset request settles to a `KeysetAnswer`: the loaded state, or the load's error, shared by every request that named that keyset. `Responses::next_keyset` returns it as `Error::KeysetLoad { keyset_id, source }`, so the documents under that keyset fail and the fulfilments beside them draw their keys. The Go guest maps the new error as its cause. The joined sides, the per-keyset futures and `keyset_state`'s load are built before any data key has been through the stack, and boxed: a future is built on the stack and moved with the bytes it does not initialise, and a join replaces a finished side's slot from a stack temporary as wide as the side. `tests/heap_hygiene.rs` is what sees the difference; it now zeroes what it allocates and scrubs the stack before each watch, so what it reports is what the watched operation itself wrote to the heap. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

…n alone without claiming tampering `->` on an object or array with members was `Error::Aead`, which the Go guest maps to `STATUS_AUTH`: an honest query for a container read as tampering. The client cannot know the entry's kind before it opens (the kind is inside the ciphertext, and the entry is sealed over children `->` does not return), so the open under the empty children digest stays and its refusal is a new `Error::JsonNotALeaf`, which names both causes, a container with children and an altered entry, and claims neither. The guest maps it to `STATUS_ENCODING`, as it mapped a container before the cascade. ADR 0010 records the exception under its restored Consequences heading; the changeset, the stack-encrypt changelog, the leakage page and the design note say the same. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

A member's ordinal, the `o` in its parent tag that the server probes from zero to enumerate a node's children, was its keyed rank among its siblings. Nothing reads a rank back: placement probes the parent tags and reads what is there, and a member's token does not depend on its ordinal. So each object's members now get a uniformly random permutation of `0..n` as their ordinals (`member_ordinals`, through `shuffle::ordinals`), drawn from the document's generator beside its stored order; `rank` and `RANK_DOMAIN` go. A token's holder learns nothing from the ordinals it probes, and cannot link a key to an ordinal from row to row. The pinned vectors move in `order` and in each entry's `p` alone; the self tags, selectors, tokens and selector set are unchanged. The leakage profile's simulator draws member orders per row, and its distinguisher for a consistent member order across rows is now chance-bounded, as the one across parents is. The design note, the leakage page and the changeset say the same. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

`json_tree_children` ordered its rows by stored slot, which was the ordinal's order while entries were stored depth-first and is a random order now, so an array's elements came back shuffled where the function is meant to be `jsonb_array_elements` for an encrypted array. It orders by the probe's ordinal, which the server already computes, so an array's elements come back in position order and an object's members in the random order their ordinals were drawn in; the comment says so and no longer promises that every child opens on its own. The PostgreSQL test compares the elements' order unsorted. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

…e does The cascade module's rustdoc carries a diagram: a small document, what each node's AAD binds, sealing leaves up and opening root down. The leakage page opens with what the database needs to answer queries and that the values stay secret, and that `json_tree` reveals strictly less than `ste_vec`, before listing what each capability costs; its closing line and the selector set's rustdoc no longer narrate what an earlier order showed. The crate's front-page example and changelog say that awaiting a pending needs an `IndexKeySource` beside the `DataKeySource`, which every cipher already has. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

The guest is a detached Cargo workspace, which the root `cargo fmt --all` does not reach; CI's `wasm:guest:test` runs rustfmt there and refused the line wrap in `a_failed_keyset_load_maps_as_its_cause`. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

Brings in the mutants gate fix from #1381, which reads the pull request's real base from the API and checks out the head. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

Brings in #1379 (the json_tree cascade and random stored order). The one conflict was tests/json_index.rs, where this branch and #1379 each carried a fix for the float round-trip test; next's version is taken, since it also finds the node by path rather than by its stored slot. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

…profile Brings in next with #1379 (the json_tree cascade and random stored order). Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

Brings in next's JSON tree cascade and random storage order (#1379). Conflicts resolved to #1379's names and wording, keeping this branch's data plan target, its JSON text parser and the JsonSearch rename: JsonProjection is JsonSelectorSet in the fuzz table and the JsonQuery docs, the leakage page takes #1379's opening under the eql_v4_json_search name, json_tree_path_query's comment says stored order under the json_entry domain, and the oracle's hostile-row updates find each entry's slot (the order is random) and cast to eql_v4_json_search. The changelog keeps both sides' entries. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

Brings in next's JSON tree cascade and random storage order (#1379) through the base, with the base's review fixes. The guest's status map keeps both sides' arms: JsonWrongEntry and JsonWildcardTruncated from this branch, JsonNotALeaf from #1379, all STATUS_ENCODING. The operators' derivation follows #1379's renames (a path builds the document `holding` a value; JsonQuery::of_terms and at_path take the terms by value), and the guest's json_open bounds IndexKeySource, which awaiting a pending now needs. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

…in the JSON operators #1379 stores a document's entries in a random order, shuffles every containment query's selector set, and makes an entry's ciphertext commit to its children's. Three things here assumed otherwise: - the data plan's operand parity test compared a containment or equality operand with the typed chain's byte for byte; it now reads both back as JsonQuery, which compares as sets; - `->` on a container with members now fails as JsonNotALeaf (Go: ErrEncoding, code stack_encrypt::json_not_a_leaf), so a selector read of `$.visits` is pinned to that refusal in the data plan test and through the Go guest, and json_open's and Cipher.JSONOpen's docs say it; - the docs said an ordering or `->` through a wildcard path compares the first element; it sees an unspecified one, and an ordering over every element is written over all the matches (json_tree_path_query). The same wording is corrected in JsonOptions::ordered, JsonOrderingAtPosition and the typed orderings' position check. The data plan's identifiers follow #1379's naming: a containment operand is a document's selector set, not a needle (DocumentInput, Input::Document, Binds::Query). Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

Brings in next's JSON tree cascade and random storage order (#1379) through the base, with the base's review fixes. The per-type PostgreSQL test's comment takes the base's wording, which names this branch's TestJSONDocumentsQueryAndReadInPostgres. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

A JSON document's entries are now stored in a random order (#1379), so `col -> sel` at a wildcard path, and an ordering through it, sees an unspecified element of each row rather than the first. The At accessors stay: their operands are what a comparison over every match takes, and the engine stores ordering terms at a wildcard path when asked to, so refusing ordered= there would leave those terms unreachable from Go. What changes is the SQL they are documented and tested with. - The generated XPath comment gives both forms: col -> Selector() at a path with no array step, and at a wildcard path EXISTS over eql_v4.json_tree_path_query for a comparison and the least or greatest match's ordering term for ORDER BY. Lt, Lte, Gt, Gte and Between get doc comments of their own, which they lacked, and Selector says which form it binds into. The READMEs and the CHANGELOG say the same. - TestJSONDocumentsQueryAndReadInPostgres compares at $.visits[*].at with EXISTS (some visit > 2, and some visit < 2, which only Alice's second visit meets), orders rows by their least visit ascending and greatest descending against the plaintexts' order, and reads a wildcard past its bound from PostgreSQL's own rows, which Open refuses as ErrEncoding. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

Brings in next's JSON tree cascade and random storage order (#1379) through the base, with the base's review fixes and its orderings at a wildcard path written over every match. Resolved: the generated ordering accessors keep this branch's `any` operands and take the base's doc comments, the path type's comment keeps this branch's sentence on Go operands, planJSON keeps the base's refusal of ordered paths the engine answers no ordering on without the base's encoding/json import (a document of a Go type does not need it), the encrypt README's JSON paragraph keeps both sides' sentences, and json.go imports what both sides use. The generated test models are regenerated. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

A containment query now sends its selector set in a random order of its own (#1379), so two derivations of one document, typed or as text, serialize differently. The test that holds a typed needle to the query its JSON text derives compared bytes; it now sorts every list of both queries, recursively, and compares those. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

coderdan added 6 commits October 26, 2026 11:39

coderdan commented Oct 26, 2026

View reviewed changes

Comment thread packages/stack-encrypt/src/json/derive.rs

coderdan commented Oct 26, 2026

View reviewed changes

Comment thread packages/stack-encrypt/src/target/pending.rs Outdated

coderdan commented Oct 26, 2026

View reviewed changes

Comment thread packages/stack-encrypt/src/json/cascade.rs Outdated

coderdan commented Oct 26, 2026

View reviewed changes

Comment thread packages/stack-encrypt/src/json/query.rs

coderdan commented Oct 26, 2026

View reviewed changes

Comment thread docs/plans/2026-10-17-json-tree-index.md Outdated

coderdan commented Oct 26, 2026

View reviewed changes

Comment thread packages/stack-encrypt/src/cipher.rs Outdated

coderdan commented Oct 26, 2026

View reviewed changes

Comment thread packages/stack-encrypt/src/json/query.rs

coderdan commented Oct 26, 2026

View reviewed changes

Comment thread packages/stack-encrypt/src/json/derive.rs Outdated

coderdan commented Oct 26, 2026

View reviewed changes

Comment thread packages/stack-encrypt/src/cipher.rs Outdated

coderdan added 2 commits October 26, 2026 13:56

coderdan marked this pull request as ready for review October 26, 2026 05:12

coderdan requested a review from a team as a code owner October 26, 2026 05:12

cipherstash-agent reviewed Oct 26, 2026

View reviewed changes

Comment thread packages/stack-encrypt/src/json/derive.rs

Comment thread packages/stack-encrypt/src/cipher.rs Outdated

coderdan commented Oct 26, 2026

View reviewed changes

Comment thread packages/eql/docs/reference/json-tree-leakage.md Outdated

coderdan commented Oct 26, 2026

View reviewed changes

Comment thread packages/stack-encrypt/src/json/cascade.rs

coderdan added 4 commits October 26, 2026 18:10

coderdan mentioned this pull request Oct 26, 2026

ci(mutants): diff against the pull request's real base, read from the API #1381

Merged

coderdan added 6 commits October 26, 2026 19:03

Merge next into feat/json-tree-cascade-and-random-order

f783f8e

Brings in the mutants gate fix from #1381, which reads the pull request's real base from the API and checks out the head. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

This was referenced Oct 26, 2026

vitaminc-random: no uniform random permutation of arbitrary size, so stack-encrypt reaches for rand's shuffle cipherstash/vitaminc#542

Open

spike(stack-encrypt): cascading AEAD tags and a random stored order for json_tree #1371

Closed

coderdan merged commit 02ae76b into next Oct 26, 2026
54 checks passed

coderdan deleted the feat/json-tree-cascade-and-random-order branch October 26, 2026 09:29

coderdan mentioned this pull request Oct 26, 2026

feat(stack-encrypt,eql): store a JSON document through a data plan as JsonSearch #1372

Merged

Conversation

coderdan commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Verification

Related

Review notes

Uh oh!

changeset-bot Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

coderabbitai Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

Uh oh!

coderdan commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderdan commented Oct 26, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

cipherstash-agent left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coderdan commented Oct 26, 2026 •

edited

Loading

changeset-bot Bot commented Oct 26, 2026 •

edited

Loading

coderabbitai Bot commented Oct 26, 2026 •

edited

Loading

github-actions Bot commented Oct 26, 2026 •

edited

Loading

Mutation testing (cargo-mutants, `--in-diff`, stack-auth + stack-encrypt)

coderdan commented Oct 26, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 26, 2026 •

edited

Loading