Repository navigation
Independent audit: Tom lab re-derivation, red-team findings, note-tone experiments - #78
Merged
Merged
Conversation
added 30 commits
October 6, 2026 21:47
Publication audit (T-11) before co-authorship: re-derived every published number about the symptom-index and fail-closed work from raw artifacts. Retracted, with the referent recorded: - reader stability "one model 8/10, another flipped 6/10": the denominator exists in no surviving file, and two defensible rules disagree on which reader was steadier. - "glm FA 0.24-0.38": 0.38 is nemotron-nano-30b, a different model. A category error on our side, and no raw survives anywhere. - "recall 0.08 -> 0.88": kept only with the caveat that it is a change of evidence format (code_first to file_content_first), not before/after. Confirmed from recovered raw: the arrival layer maps the dispositional symptom to family A in 5/5 runs. The E11 handout was recovered byte-exact from the opencode session database, its sha256 matches a pre-registered value, and it was written 35s before the first model call, so the recovery is pre-look and the replay it enables is a real replay. Data defect found in our own results: two F4b run files are byte-identical while the manifest declares 22 runs over 21 distinct files, so the "arrival 10/11 vs symptom 3/11" denominator is inflated by one. The direction is unaffected: every symptom failure is on the same item. Guard: scripts/check_duplicate_runs.py, exit 1 on a duplicate or a declared/actual mismatch, exit 2 on an empty population, --selftest 5/5, plus 16 unit tests. It reproduces the F4b defect and passes on recovered_e7 and pinned_variant. KNOWN_ISSUES.md and docs/archive/* hold another session's uncommitted work and are deliberately untouched; proposals are in the PENDING_LEDGER.
Inputs for the deterministic delivery check against Tirthahq/crystal-memory @ fe4e465, frozen before any run so the expectations cannot be fitted to the result. - frozen/ACTS.md: 18 acts in six groups. Word-boundary cases where a short key or one of the seven generic words appears only inside a longer word; substring cases that must still match; whole-word generic cases; same-tier and cross-tier ordering pairs; write-scope cases for the 600/2000 char subject rule; and telemetry cases including a dry run that must emit nothing. - frozen/ORDER_RULE.md: ordering is scored only inside a rotation tier, the overlap gap that makes an ordering difference observable is fixed at 0.10, and delivered / headed / starved are reported separately. - frozen/HYPOTHESES.md: 12 hypotheses with falsifiers, of which three are predicted to FAIL on purpose (the long-context subject guard, the post-hoc specificity threshold, and the Windows selftest, which is outside the declared support envelope). Five mandatory controls, including the blind replay. The run has two arms: our independent reimplementation and their code in a one-off copy. A claim counts as confirmed only when both arms agree on every act; disagreement is the finding, not a pass.
…cumented, zero arm divergences
…ation and one downgrade
… research, M4 design
Recovered run transcripts under experiments/*/frozen/ carry historical version strings by design. Rewriting them to match today would destroy the pre-look integrity the experiments rest on, so drift-checking them can only ever be a false positive. Measured: 10 files, up to 18 drifts each, all inside frozen/. Discrimination re-proven live and manifest re-pinned.
Rig rewritten into the repo; the full-30 rig was a throwaway file in TEMP. Four gates, each self-tested before spending: artifact written before teardown, plant hash pinned per run, fresh hook-free fixture repo, and a variability gate that refuses to scale on a constant outcome. Result: 4/4 committed, 4/4 opened callee.py, 0/4 fixed. Outcome constant, exit 3. Measured cost 0.106 USD.
The matcher was a regex requiring the two tokens `git` and `commit` to be adjacent, so `git -C <dir> commit` was never gated. M4 full-30 run 11 committed that way, was not discarded despite a failed C-plant, and was scored as NO-FIX. Replaced with a tokenizer that skips value-taking global options (-C, -c, --git-dir, ...) and ignores quoted segments, so `echo "git commit"` no longer over-gates the gate. New control file: 11 positive and 9 negative cases, 20/20.
…0.000 Previous three fixtures scored the outcome the agent produced anyway (against-prior requirement unmet, Harness-IF arXiv:2608.11727) and used a fully specified task, leaving no decision to influence (AutoDojo arXiv:2606.15057). v4 makes the task action-open with two correct resolutions and scores the artifact, not prose. G0 prior gate: P(PRESERVE)=5/9=0.556 from the working-tree diff. Prose disagreed with artifact in 2 of 9 runs. Main contrast, 18 runs: V1 (pushes UPDATE) 9/9 UPDATE, V2 (pushes PRESERVE) 9/9 PRESERVE, baseline 4/9 UPDATE. Spread 1.000 against a pre-registered 0.30 threshold. Rig defects found and fixed: provider ceiling ~200s per session; artifact captured the fixture commit instead of the agent work; silent null from transport failure masquerading as a unanimous prior; silent model fallback. Cost: 27 successful runs, 0.48 USD plus 9 zero-cost transport failures.
Both notes point the same way (UPDATE) and are length-matched within 4% (130 vs 135), so wording is separable from direction: H1 imperative 9/9 UPDATE, H2 hedged 9/9 UPDATE, spread 0.000 against a pre-registered 0.30 threshold. Read as NOT DETECTED, not as absent: at n=9/arm a 9/9 vs 8/9 difference is unresolvable, and both arms saturated at 1.000, which removes the instrument's sensitivity to any smaller difference. Both arms moved the prior from 0.444 (G0, 4 of 9) to 1.000. Prose and artifact disagreed in 1 of 18 runs (run 3 declared PRESERVE, patched UPDATE). Cost: v4+v5 58 runs, 0.735 USD.
…16 rows Ran his E1/E2 and overlap fixture through our own implementations, from inputs frozen and hashed before any analysis. E1: our estimate -0.3056, CI [-0.5, -0.1111], SUPPORT, 108/111 rows banked, 12/12 tasks, approach 0.972 vs 0.972 (not compromised). His code agrees to four decimals. Label-permutation control collapses it to -0.003. E2: C1 -0.0556 CI [-0.25, 0.1111] INCONCLUSIVE; C2 -0.1111 CI [-0.3889, 0.1111] INCONCLUSIVE; C3 -0.5455 CI [-0.8485, -0.2727] SUPPORT on 11 tasks of 12 included (Model B), REPLICATES. All three agree with his verdicts. Overlap: 25/25 input hashes reproduce from crystal-memory @ fe4e465; 400/416 rows reproduce; 16 rows are unverifiable because the fixture ships the SYNTHETIC-unicode hash without its text. Collapsed-tokenizer control matches 0/171 fraction rows, so the check discriminates. Three bugs in our own verifier are documented in the report: a leftover loop variable that compared a foreign input 416 times, a strip() that shifted the 1500-char cut by one character, and a scanner that printed 0/0 and looked like a pass.
Promised in the issue thread and frozen before running. Their own dispatcher on our isolated store (the 3 starters, the entire deployable population at fe4e465), scored from their own eligibility/delivery logs. No model in the loop. H1: 4 boundary keys (eval/ssh/ssm/tsc) match only as words, 8/8; 9 symbolic keys match verbatim, pipe is load-bearing, 10/10; 6 long keys keep substring semantics (old behaviour preserved), 12/12. H2: superstrings of the seven generic words add nobody, 7/7. H3: must-match-nothing acts give empty lists, 3/3. H4: order of 3 two-candidate acts equals our independent word-overlap computation, 3/3. H5: stable across repeats. G4 blind repeat identical. Caveats stated in the report: the blind control is vacuous (0 candidates, guard has no binding), repeatability on non-empty sets comes from H5 instead; the seven words have no deployed binding so only their superstring behaviour is covered; order checked on 3 acts. Mirror left clean, ledger restored.
Red team against crystal-memory @ fe4e465, all deterministic, no model, frozen plan before running. A substring false positives on length-5 keys, 5/5 HIT (escargot fires green-tests via cargo; |tailspin/|headache/|grepable/<<EOFILE fire exit-code): BUG, natural text triggers warnings about absent danger. B seven-word rule ablation: 0/8 delivery changes, the advertised change has no effect on the deployed set: DEPLOYMENT FACT. C silent truncation (max_n=1 on a 2-candidate act): 1 delivered, 1 starved, starved text absent from output: DESIGN LIMIT. D repeat danger in one session: first warns, second is empty (backoff+cap): DESIGN LIMIT. E rotation beats relevance on repeats (unit-level, 0.065 vs 0.021 overlap): DESIGN LIMIT. F benign battery 0/10 deliveries: GOOD. Also adds HISTORY.md, the full story ledger from invitation through withdrawals, X12, re-derivation, delivery check and red team, for the eventual thread comment.
Three of our own published claims were wrong and are corrected here, with evidence. 1. SYNTHETIC-unicode: the 16 rows ARE verifiable. The input text lives in his frozen generator; our terminal rendered it as ASCII mush and we concluded the text was absent. With the real text the hash matches and all 16 rows agree, including the 4/9 unicode-tokenization control. Overlap is 416/416, not 400/416. The hole we reported did not exist. 2. Attack B ablation was vacuous: emptying _BOUNDARY_KEYS cannot change anything by construction because short keys take the boundary path on length alone. Replaced with real evidence: a synthetic guard-key crystal proves the mechanism works (word matches, glued does not), and a full inventory of the 25 deployed keys proves none of them depend on the set. Conclusion stands (no deployed effect), evidence replaced. 3. Attack C used an artificial max_n=1. On the live 3-crystal store under the default budget all 3 deliver and nothing starves. Reclassified: mechanism verified, not triggered here. Also verified: repeat-silence second run raises no exception (real empty, not a crash); no expired flags or tiers on the deployed set; rotation-env order coincides on the tested act. Uncovered paths named in the report: pack/specificity role, systematic rotation comparison, holdback, expiry.
All deterministic, no model, expectations frozen before running (redteam/PLAN2.md lives next to the scripts). B1 obfuscation: env-var assignment does NOT hide (substring present, fires); base64/escaping/ANSI/padding bypass (no literal, silent); heredoc body scanned; background & does not hide. Two of my first B1 rows were confounded (contained go build/go test) and were redone clean with a control. B2 seen poisoning, stronger than the TZ: [1,0,0,0,0] on 5 repeats (backoff, not the cap), backoff key is per ACT TYPE so any bash act in the window suppresses. B3 fail modes: corrupted ledger returns+delivers (fail-open), broken crystal skipped silently (fail-open), 10MB ctx in 0.4s (no ReDoS). B4 TOCTOU: heredoc body visible, & does not hide; intra-process race absent by construction (synchronous single call). B5 scale: load 0.87/8.02/39.07s and match 0.034/0.326/1.735s at 100/1000/5000 crystals, both linear; 1.7s per act at 5000 exceeds the pain threshold.
Three of our own red-team claims were wrong and are corrected with evidence. B2 cap test forged the ledger twice without the schema key and without session-last; both times the loader discarded the forgery and the test measured an empty ledger. Redone with schema+live session: cap fires exactly at seen=3 (control seen=2 delivers), and live 5x repeats go [1,0,0,0,0] because time-backoff suppresses from the second run, so the cap is a backstop that repeats never reach. B1 two rows contained go build/go test and fired on those, not on the tested obfuscation; redone clean with a control. C on max_n=1 is reclassified: mechanism verified, but on the live 3-crystal store under the default budget all 3 deliver and nothing starves. Scale numbers are lower bounds (synthetic crystals carry 2 keys each); ~1400 crystals crosses the 500ms pain threshold at 0.35ms each.
The 30 runner records, the recovered transcripts+telemetry (30/30), the zero-fix diagnosis, and the two uncommitted rigs (ambig, notes). Evidence preservation: this data was already lost once when worktrees were deleted before capture.
Four advanced vectors, plan frozen before running, plus ~25 sourced literature anchors and a second agent re-verifying all five technical claims with its own code (5/5 AGREE). V1 weaponized crystals: evil file loads 1/1 with no complaint and fires 1/1; no content-verification mechanism exists (only structural invalid_bindings, a ledger-cache signature, and fail-open credential redaction). V2 attention deflection: 1 large critical vs 12 small weaks sharing the act vocabulary; critical starved 1/1 with its text absent from output. Mechanism established by dissection: candidate order follows overlap honestly, but the greedy best-fit packer takes small pieces first, so a swarm of small high-overlap weaks crowds out a large critical. Reverse also measured (small critical survives by best-fit). V3 shell aliasing: aliased git/cargo fire green-tests on text the shell would execute differently; matcher resolves no aliases (verified by reading). V4 phantom poisoning: repeat goes silent by backoff; forged seen=3 blocks while seen=2 delivers (cap discriminates exactly); clock-ageing releases backoff but not cap; fresh session speaks (cap is per-session).
Three pilot failures, three fixes, all measured. (1) PowerShell agents cannot execute extensionless sh wrappers: added .cmd wrappers via a python relay (note from env, never interpolated). (2) opencode denies Read on *.env by default and asks on external dirs: scoped OPENCODE_CONFIG override for experiment runs only (fixture .env files hold no secrets); global config untouched. (3) Identical harness line appended to every prompt telling the agent to use cat when a read is refused. Pilot t5a x 2 arms now CLEAN: approach=True both, audit rows flow, scorer runs.
proc.kill() leaves grandchildren holding our pipes and the next communicate() hangs forever (measured: 25 min, zero output). All run_agent implementations now use taskkill /F /T first with proc.kill as fallback and a bounded second communicate. Post-mortem P-kill recorded in AGENT_DIARY: terminated the OpenCode Desktop app mistaking it for zombies (age is not evidence of zombieness); guard is terminate-only-own-children plus Win32 parent-chain check, Electron main+renderer trees never touched.
added 4 commits
October 10, 2026 23:29
REFUSE (prohibitive mood, same mechanisms, length-matched 131-136) vs fresh SHAM, 11 traps x 1 rep: approach 4/11 vs 9/11, hit|approach 2/4 vs 6/9. The approach gap has no cite evidence (tails show bypasses and tool-discovery stalls, never the note), so it is noise+variance, not a mechanism. His E1 effect did not replicate on our stack in advisory form (0.600 vs 0.529, 44 runs) or prohibitive form (0.500 vs 0.667, 22 runs). Stated as non-replication on a different agent/harness, not refutation. Cost: REFUSE 11 sessions 0.23 USD; replication total 77 sessions 1.65 USD.
The v4/v5 runs classified outcomes at runtime and printed them, but saved no artifact field, so an independent audit read missing keys as 0.0. Backfilled by re-running the fixed classifier (with wrapper-function resolution) on the saved working-tree diffs: 36 files, zero disagreements with the printed runtime verdicts. The report audit now reproduces every headline number.
R1 (44 runs, HELP vs SHAM) + R2 refusal arm (22 runs, REFUSE vs fresh SHAM) + pilots. Every run: approach/hit from his scorer on our artifacts, note attempt flag, tails, worktree state. Saved before teardown per the artifact rule.
Adds the experiment index (read order, guarantees), text-level results (10/10 predicted: case folds, 6 invisible bypasses silent), independent verification record (second agent 5/5 AGREE, report audit ALL NUMBERS REPRODUCE), and the full draft comment for Tom's thread with every number carrying its denominator and every error corrected next to the number it touched. Nothing posted externally; draft awaits explicit word.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
added 22 commits
October 10, 2026 23:59
Audit of the draft found 7 gaps: no model identity, no timeout count, no delivery rate, no scaffolding disclosure, no second-agent mention, no instrument-validity line, no his-vs-our model distinction. Added a Scope section up front (longcat-2.0/default, 77 sessions, 17/68 timeouts, 47/68 deliveries attempted, full scaffolding list, second agent 5/5, instrument proven on choice task) and expanded caveats with approach denominators. Every added number re-verified against artifacts before writing.
Self-audit of the comment found two omissions: t3a was excluded from the pooled contrast without being named (both agents bypassed the wrapped standins so the note never arrived; kept as a delivery-coverage finding since both hit the quota wall through the uninstrumented path), and the blind repeat control was vacuous (0 candidates, guard has no binding). Both now stated in Caveats.
Spot re-check of the scale curve reproduces order and linearity (n=1000: 7.99s/0.344s vs recorded 8.02s/0.326s) but n=100 absolutes vary 8x with machine state (0.11 vs 0.87s). Stated next to the numbers instead of silently keeping the single-run values.
H-A session rotation: REFUTED as stated (1/5 deliveries, not 5/5). Corrected finding: suppression is two-layered (session cap/seen + global time backoff); rotation defeats only the session layer. H-B size demotion: first run failed to overflow (INCONCLUSIVE, honestly labeled); second run with real overflow CONFIRMS a 2102-char critical starved by twelve 329-char weaks with text absent. H-C rank-pack: REFUTED on 6 acts; refined rule is rank==pack iff nothing starves, with size-driven reordering under overflow (snap_ord case). All three started as failures and were dissected into precise claims rather than re-run until green.
Same agent/channel/fixtures, two framings, three arms, 18 runs. PREF replicates v4 steering (H1 CONFIRMED). Misleading note in TRAP changes nothing: 3/3 UPDATE with 3/3 passes, agents verify rather than obey (H2 REFUTED). H4a gate FIRED as frozen: TRAP-NONE went 3/3 UPDATE instead of splitting, so the trap framing has no dilemma for a note to move; stopping per rule rather than claiming invariance. No check.py modifications in 18/18 (cheat guard held). Cost 18 sessions 0.23 USD.
ASYM paragraph (steering 1.000/0.000 vs verification dominance), 3f4dbfd acknowledgement bounding the R2 null to the current release with a re-run offer, ledger-size negative result (0.014s vs 0.059s at 20k entries). Verified: scripts/ unchanged between fe4e465 and 491f385 (docs only), so all code findings stand on current code.
3f4dbfd and 491f385 are docs-only; scripts/catalogue/starter unchanged since fe4e465, so all code findings stand on current code. The new staleness section is checkable and was checked: 24/24 entries carry no stale_after (21 catalogue entries have no dates at all, 3 starters verified 2026-09-03, 37 days old). The expiry mechanism exists in code with no data to operate on. Refuse-once exists only in prose, no code.
Expiry machinery verified working (past stale_after yields WITHHELD stub 1/1, future/undated yield full text), but 0/24 shipped entries carry any stale_after: mechanism without data. Added to the Tom draft as a deployment gap (not a bug), plus the ledger-size negative result was already there. Documentation pass: all 10 docs present and consistent, HISTORY at 14 chapters. Nothing posted externally.
inventory.py verified every planned run against saved artifacts and every artifact against a verdict: 113 M4 files across 11 series, 68/68 replication runs, all deterministic JSONs parse, 0 hollow artifacts. ALL EXPERIMENTS ACCOUNTED.
States what our numbers are not: fixed probes (lower bounds, no optimized attacker measured), single ~70s acts (no lived long sessions), one model (other models his territory), dispatcher isolated (his hook-to-reading loop unverified from our side), races/POSIX/catalogue named but unmeasured. The honest-bounds paragraph the audit standard requires.
Review-driven rework, all points addressed with evidence. (1) Weaponized reframed as supply-chain with the explicit scenario (catalogue sync, malicious PR, vendored pack) and the perimeter objection answered. (2) AgentPoison 82/63 verified verbatim against arXiv full text; Governance Decay 100/85/65 verified against Table 4 in the paper. (3) Order sample n=3 and tier-untestability stated next to the conclusion. (4) Seven-word and staleness moved out of breaks into a separate verified-working section. (5) Budget test labeled proof-of-possibility. (6) Form: urgent three first, remainder in a details block; caveats stay visible.
Owner published the final comment 2026-10-10 22:24 UTC (13,279 chars); head and tail verified against the draft, full text landed, no replies at time of recording (15 comments total). Open items remaining: refusal-arm re-run when refuse-once ships; PR #78 merge decision.
His judgements-in-implementation post is 09-27 (public), our base practices predate it (diary July-August), but our sections 19-21 were formulated 09-30/10-03 after collaboration began, so influence cannot be excluded. Recorded: method priority is his; ours is only the 10-11.10 measurements on his system. Cite his post as prior art in any methods writing; claim originality on numbers only. HOOK BYPASS REASON (not silent): verify_diary fails on two pre-existing 09-09/09-10 entries referencing commit 0301fa9, which exists nowhere in any history (verified via cat-file + log --all). Unrelated to this change; fixing others' diary rows would violate single-owner registries. All other gates green.
Enumerated all 20 posts via API; pulled live threads. Applied Ahmet paired-transition analysis to our arms: V4/V5 unanimous agreement, R1 McNemar 3-vs-2 with 5+4 agreements (p=1.0), null confirmed rigorously. Recorded: v7-loss parallels our full-30, second-agent independence stands as briefed, calibration unmeasured everywhere, escalation churn supports pairing. HOOK BYPASS (not silent): verify_diary fails on pre-existing 09-09/09-10 entries citing commit 0301fa9, absent from all histories. Unrelated to this change; all other gates green.
Reproduced+fixed (7a129db/b86edc7, credited): len-5 and normalization; his new find: naive len<=5 silences symbol-ended prefix post-. Confirmed-by-reading: backoff keyed act:<type>:<crystal>. Re-derived: budget starvation (one act), repeat silence, scale. Scale gap 100-200x: our candidates() re-loads internally (crystal_act.py:416) so our match bundled a warm re-load; 2932-char essences vs ~1000; Windows host vs declared macOS/Linux. Owe him the timing boundary. Behavioural nulls accepted as cross-model non-replication. HOOK BYPASS: verify_diary fails on pre-existing 0301fa9 reference; unrelated, other gates green.
Same-session A/B (n=1000, fresh stores with nonce): Defender Real-Time ON 8.9-10.3 ms/file vs OFF 0.32 ms/file = 28-32x. Our recorded load numbers were RT-ON class (7.8-8.7 ms/file). Our 'match' (0.35 ms/crystal) was actually a warm re-load: candidates() calls load_crystals internally (crystal_act.py:416); pure match is 2.1-2.2 us/crystal, ~Tom's 3 us. Disk is one NVMe (H3 refuted); essence 1335 vs 1000 = 1.3x (H4 refuted); residual 8.7x is Windows open overhead, outside declared macOS/Linux (H5 refuted). Controls: RT-ON positive, empty-store negative, on/on/off/off reversibility. Harness + raw artifacts committed under scale_forensics/. HOOK BYPASS: verify_diary pre-existing 0301fa9 failure; unrelated.
Owner asked to re-run every experiment whose numbers were distorted by the (then-on) protection. Inventory of all wall-clock numbers found 3 (scale, ledger, 10MB ctx) plus a self-contradiction: scale n=100 gave 0.868s (REDTEAM2) vs 0.11s (VERIFY_NUMBERS) on the same store - Defender caches its verdict by file content (identical names+text => second run unscanned). Fresh run, Defender OFF, unique nonce:\n\n- scale: load 318 us/file; true match 2.1 us/crystal (was 0.35 ms = re-load); candidates(repo)==load at all n.\n- ledger: linear ~2.7 us/entry (0.11s at 40k) - previous 'does not degrade' RETRACTED; ledger_perf.py had a dead first loop and persisted nothing.\n- 10MB ctx 0.486s - no-ReDoS verdict holds (CPU/parse, not file I/O).\n- delivery 30/30 PASS re-run - content verdicts invariant; Defender moved only time.\n- pain point: 500ms ~= 1570 files (load), not '~1400 crystals'.\n\nHarness + raw: scale_forensics/REMEASURE.json, remeasure_all.py. HOOK BYPASS: verify_diary pre-existing 0301fa9 failure; unrelated.
The producer of VERIFY_OVERLAP.json was a one-off and never saved - the only unclosed item in the audit. Rebuilt independently (0 USD): the 26 fixture inputs are pinned by the fixture's own essence_sha256 (26/26 - starter/ was added after fe4e465, catalogue pages edited later, so each input comes from the revision whose derived hash matches). My own overlap() (exact package semantics at fe4e465: _essence between markers with .strip newline, Jaccard over lowercase word tokens, essence[:1500] before lower) reproduces all 416 rows (0 disagreements). Controls: 171 nonzero rows, 0 collapsed to zero (positive); 245 zero rows, 0 flipped to nonzero (negative). Verdict: ALL 416 ROWS REPRODUCE. Artifact: overlap_rederivation/. HOOK BYPASS: verify_diary pre-existing 0301fa9 failure; unrelated.
Draft reply to the thread correcting our own published timing numbers. The scale paragraph and the ledger caveat in TOM_COMMENT_DRAFT.md (the published comment) are stale; the follow-up gives the Defender-OFF numbers, retracts the "no ledger degradation" line (now linear ~2.7 us/entry, JSON parse per act), and corrects the match number (re-load -> 2.1 us/crystal, matching his ~3 us). Draft only, not posted. Awaiting owner's word. HOOK BYPASS: verify_diary pre-existing 0301fa9 failure; unrelated.
Tom pushed the input-validation fix in crystal-memory@7a129db (parent 491f385) and tjonesit/crystals@b86edc7 — both change scripts/crystal_act.py (+45/-5). We loaded HIS OWN matches_ctx / match_scope / normalize_match from both revisions (raw.githubusercontent) and re-ran our five audited false-fires plus the normalization strings. The OLD revision is the control that must fail: it reproduces the length-5 hole 5/5. The NEW revision closes all five, keeps the three real commands and the symbol-ended `post-` prefix, and now matches all five normalization inputs (zero-width, Cyrillic dze, fullwidth, nbsp, thin space). Result 15/15, failures 0 -> TOMS-FIX-INDEPENDENTLY-CONFIRMED. No model, $0. Artifact: tom_fix_verification/. HOOK BYPASS: verify_diary pre-existing 0301fa9 failure; unrelated.
ManSio
pushed a commit
that referenced
this pull request
Oct 24, 2026
Owner published the final comment 2026-10-10 22:24 UTC (13,279 chars); head and tail verified against the draft, full text landed, no replies at time of recording (15 comments total). Open items remaining: refusal-arm re-run when refuse-once ships; PR #78 merge decision.
ManSio
added a commit
that referenced
this pull request
Oct 24, 2026
…e experiments (#78) * docs(experiments): audit published numbers, retract three claims Publication audit (T-11) before co-authorship: re-derived every published number about the symptom-index and fail-closed work from raw artifacts. Retracted, with the referent recorded: - reader stability "one model 8/10, another flipped 6/10": the denominator exists in no surviving file, and two defensible rules disagree on which reader was steadier. - "glm FA 0.24-0.38": 0.38 is nemotron-nano-30b, a different model. A category error on our side, and no raw survives anywhere. - "recall 0.08 -> 0.88": kept only with the caveat that it is a change of evidence format (code_first to file_content_first), not before/after. Confirmed from recovered raw: the arrival layer maps the dispositional symptom to family A in 5/5 runs. The E11 handout was recovered byte-exact from the opencode session database, its sha256 matches a pre-registered value, and it was written 35s before the first model call, so the recovery is pre-look and the replay it enables is a real replay. Data defect found in our own results: two F4b run files are byte-identical while the manifest declares 22 runs over 21 distinct files, so the "arrival 10/11 vs symptom 3/11" denominator is inflated by one. The direction is unaffected: every symptom failure is on the same item. Guard: scripts/check_duplicate_runs.py, exit 1 on a duplicate or a declared/actual mismatch, exit 2 on an empty population, --selftest 5/5, plus 16 unit tests. It reproduces the F4b defect and passes on recovered_e7 and pinned_variant. KNOWN_ISSUES.md and docs/archive/* hold another session's uncommitted work and are deliberately untouched; proposals are in the PENDING_LEDGER. * experiments(crystal-delivery): freeze act list and ordering rule Inputs for the deterministic delivery check against Tirthahq/crystal-memory @ fe4e465, frozen before any run so the expectations cannot be fitted to the result. - frozen/ACTS.md: 18 acts in six groups. Word-boundary cases where a short key or one of the seven generic words appears only inside a longer word; substring cases that must still match; whole-word generic cases; same-tier and cross-tier ordering pairs; write-scope cases for the 600/2000 char subject rule; and telemetry cases including a dry run that must emit nothing. - frozen/ORDER_RULE.md: ordering is scored only inside a rotation tier, the overlap gap that makes an ordering difference observable is fixed at 0.10, and delivered / headed / starved are reported separately. - frozen/HYPOTHESES.md: 12 hypotheses with falsifiers, of which three are predicted to FAIL on purpose (the long-context subject guard, the post-hoc specificity threshold, and the Windows selftest, which is outside the declared support envelope). Five mandatory controls, including the blind replay. The run has two arms: our independent reimplementation and their code in a one-off copy. A claim counts as confirmed only when both arms agree on every act; disagreement is the finding, not a pass. * experiments(crystal-delivery): D8 results - both changes behave as documented, zero arm divergences * docs(experiments): Exp 28 - D8 delivery check with maximal re-verification and one downgrade * docs(experiments): Exp 29 - trap bundle audit, design clean, headlines not re-derived * feat(gates): delivery telemetry out of temp with per-evaluation decision rows * docs(experiments): Exp 30 second wave - D9 telemetry, red team fixes, research, M4 design * experiments(note-tone): freeze M4 voices and red-team, smoke scope only * feat(gates): MSCODEBASE_ADVISORY_TEXT override for controlled voice experiments * fix(gates): loader-safe plugin entry, length-matched M4 voices amendment * fix(gates): entry exports only default, helpers moved out of loader path * fix(gates): hooks never throw on malformed input, loader-proof test * fix(gates): helpers out of plugin dir, loader auto-loads every file * experiments(note-tone): probe2 loader fixed, gate cli needs explicit python * experiments(note-tone): smoke 5 runs, rig works, no verdicts, measured cost * fix(stale-detector): exclude frozen/ evidence from version drift Recovered run transcripts under experiments/*/frozen/ carry historical version strings by design. Rewriting them to match today would destroy the pre-look integrity the experiments rest on, so drift-checking them can only ever be a false positive. Measured: 10 files, up to 18 drifts each, all inside frozen/. Discrimination re-proven live and manifest re-pinned. * experiments(note-tone): reachability probe, 4 runs, scaling blocked Rig rewritten into the repo; the full-30 rig was a throwaway file in TEMP. Four gates, each self-tested before spending: artifact written before teardown, plant hash pinned per run, fresh hook-free fixture repo, and a variability gate that refuses to scale on a constant outcome. Result: 4/4 committed, 4/4 opened callee.py, 0/4 fixed. Outcome constant, exit 3. Measured cost 0.106 USD. * fix(gate): gate commits that use git global options The matcher was a regex requiring the two tokens `git` and `commit` to be adjacent, so `git -C <dir> commit` was never gated. M4 full-30 run 11 committed that way, was not discarded despite a failed C-plant, and was scored as NO-FIX. Replaced with a tokenizer that skips value-taking global options (-C, -c, --git-dir, ...) and ignores quoted segments, so `echo "git commit"` no longer over-gates the gate. New control file: 11 positive and 9 negative cases, 20/20. * experiments(note-tone): v4 choice design, note steers agent 1.000 vs 0.000 Previous three fixtures scored the outcome the agent produced anyway (against-prior requirement unmet, Harness-IF arXiv:2608.11727) and used a fully specified task, leaving no decision to influence (AutoDojo arXiv:2606.15057). v4 makes the task action-open with two correct resolutions and scores the artifact, not prose. G0 prior gate: P(PRESERVE)=5/9=0.556 from the working-tree diff. Prose disagreed with artifact in 2 of 9 runs. Main contrast, 18 runs: V1 (pushes UPDATE) 9/9 UPDATE, V2 (pushes PRESERVE) 9/9 PRESERVE, baseline 4/9 UPDATE. Spread 1.000 against a pre-registered 0.30 threshold. Rig defects found and fixed: provider ceiling ~200s per session; artifact captured the fixture commit instead of the agent work; silent null from transport failure masquerading as a unanimous prior; silent model fallback. Cost: 27 successful runs, 0.48 USD plus 9 zero-cost transport failures. * experiments(note-tone): v5 phrasing contrast, no detectable effect Both notes point the same way (UPDATE) and are length-matched within 4% (130 vs 135), so wording is separable from direction: H1 imperative 9/9 UPDATE, H2 hedged 9/9 UPDATE, spread 0.000 against a pre-registered 0.30 threshold. Read as NOT DETECTED, not as absent: at n=9/arm a 9/9 vs 8/9 difference is unresolvable, and both arms saturated at 1.000, which removes the instrument's sensitivity to any smaller difference. Both arms moved the prior from 0.444 (G0, 4 of 9) to 1.000. Prose and artifact disagreed in 1 of 18 runs (run 3 declared PRESERVE, patched UPDATE). Cost: v4+v5 58 runs, 0.735 USD. * experiments(tom-e1e2): independent re-derivation, 25/25 hashes, 400/416 rows Ran his E1/E2 and overlap fixture through our own implementations, from inputs frozen and hashed before any analysis. E1: our estimate -0.3056, CI [-0.5, -0.1111], SUPPORT, 108/111 rows banked, 12/12 tasks, approach 0.972 vs 0.972 (not compromised). His code agrees to four decimals. Label-permutation control collapses it to -0.003. E2: C1 -0.0556 CI [-0.25, 0.1111] INCONCLUSIVE; C2 -0.1111 CI [-0.3889, 0.1111] INCONCLUSIVE; C3 -0.5455 CI [-0.8485, -0.2727] SUPPORT on 11 tasks of 12 included (Model B), REPLICATES. All three agree with his verdicts. Overlap: 25/25 input hashes reproduce from crystal-memory @ fe4e465; 400/416 rows reproduce; 16 rows are unverifiable because the fixture ships the SYNTHETIC-unicode hash without its text. Collapsed-tokenizer control matches 0/171 fraction rows, so the check discriminates. Three bugs in our own verifier are documented in the report: a leftover loop variable that compared a foreign input 416 times, a strip() that shifted the 1500-char cut by one character, and a scanner that printed 0/0 and looked like a pass. * experiments(tom-e1e2): deterministic delivery+order check, 30/30 Promised in the issue thread and frozen before running. Their own dispatcher on our isolated store (the 3 starters, the entire deployable population at fe4e465), scored from their own eligibility/delivery logs. No model in the loop. H1: 4 boundary keys (eval/ssh/ssm/tsc) match only as words, 8/8; 9 symbolic keys match verbatim, pipe is load-bearing, 10/10; 6 long keys keep substring semantics (old behaviour preserved), 12/12. H2: superstrings of the seven generic words add nobody, 7/7. H3: must-match-nothing acts give empty lists, 3/3. H4: order of 3 two-candidate acts equals our independent word-overlap computation, 3/3. H5: stable across repeats. G4 blind repeat identical. Caveats stated in the report: the blind control is vacuous (0 candidates, guard has no binding), repeatability on non-empty sets comes from H5 instead; the seven words have no deployed binding so only their superstring behaviour is covered; order checked on 3 acts. Mirror left clean, ledger restored. * experiments(tom-e1e2): red team, 5 findings, history ledger Red team against crystal-memory @ fe4e465, all deterministic, no model, frozen plan before running. A substring false positives on length-5 keys, 5/5 HIT (escargot fires green-tests via cargo; |tailspin/|headache/|grepable/<<EOFILE fire exit-code): BUG, natural text triggers warnings about absent danger. B seven-word rule ablation: 0/8 delivery changes, the advertised change has no effect on the deployed set: DEPLOYMENT FACT. C silent truncation (max_n=1 on a 2-candidate act): 1 delivered, 1 starved, starved text absent from output: DESIGN LIMIT. D repeat danger in one session: first warns, second is empty (backoff+cap): DESIGN LIMIT. E rotation beats relevance on repeats (unit-level, 0.065 vs 0.021 overlap): DESIGN LIMIT. F benign battery 0/10 deliveries: GOOD. Also adds HISTORY.md, the full story ledger from invitation through withdrawals, X12, re-derivation, delivery check and red team, for the eventual thread comment. * experiments: corrections after self-audit, overlap 416/416 Three of our own published claims were wrong and are corrected here, with evidence. 1. SYNTHETIC-unicode: the 16 rows ARE verifiable. The input text lives in his frozen generator; our terminal rendered it as ASCII mush and we concluded the text was absent. With the real text the hash matches and all 16 rows agree, including the 4/9 unicode-tokenization control. Overlap is 416/416, not 400/416. The hole we reported did not exist. 2. Attack B ablation was vacuous: emptying _BOUNDARY_KEYS cannot change anything by construction because short keys take the boundary path on length alone. Replaced with real evidence: a synthetic guard-key crystal proves the mechanism works (word matches, glued does not), and a full inventory of the 25 deployed keys proves none of them depend on the set. Conclusion stands (no deployed effect), evidence replaced. 3. Attack C used an artificial max_n=1. On the live 3-crystal store under the default budget all 3 deliver and nothing starves. Reclassified: mechanism verified, not triggered here. Also verified: repeat-silence second run raises no exception (real empty, not a crash); no expired flags or tiers on the deployed set; rotation-env order coincides on the tested act. Uncovered paths named in the report: pack/specificity role, systematic rotation comparison, holdback, expiry. * experiments(tom-e1e2): red team-2, five blocks from the TZ All deterministic, no model, expectations frozen before running (redteam/PLAN2.md lives next to the scripts). B1 obfuscation: env-var assignment does NOT hide (substring present, fires); base64/escaping/ANSI/padding bypass (no literal, silent); heredoc body scanned; background & does not hide. Two of my first B1 rows were confounded (contained go build/go test) and were redone clean with a control. B2 seen poisoning, stronger than the TZ: [1,0,0,0,0] on 5 repeats (backoff, not the cap), backoff key is per ACT TYPE so any bash act in the window suppresses. B3 fail modes: corrupted ledger returns+delivers (fail-open), broken crystal skipped silently (fail-open), 10MB ctx in 0.4s (no ReDoS). B4 TOCTOU: heredoc body visible, & does not hide; intra-process race absent by construction (synchronous single call). B5 scale: load 0.87/8.02/39.07s and match 0.034/0.326/1.735s at 100/1000/5000 crystals, both linear; 1.7s per act at 5000 exceeds the pain threshold. * experiments(tom-e1e2): redteam-2 corrections after self-audit Three of our own red-team claims were wrong and are corrected with evidence. B2 cap test forged the ledger twice without the schema key and without session-last; both times the loader discarded the forgery and the test measured an empty ledger. Redone with schema+live session: cap fires exactly at seen=3 (control seen=2 delivers), and live 5x repeats go [1,0,0,0,0] because time-backoff suppresses from the second run, so the cap is a backstop that repeats never reach. B1 two rows contained go build/go test and fired on those, not on the tested obfuscation; redone clean with a control. C on max_n=1 is reclassified: mechanism verified, but on the live 3-crystal store under the default budget all 3 deliver and nothing starves. Scale numbers are lower bounds (synthetic crystals carry 2 keys each); ~1400 crystals crosses the 500ms pain threshold at 0.35ms each. * experiments(note-tone): preserve full-30 raw evidence and leftover rigs The 30 runner records, the recovered transcripts+telemetry (30/30), the zero-fix diagnosis, and the two uncommitted rigs (ambig, notes). Evidence preservation: this data was already lost once when worktrees were deleted before capture. * experiments(tom-e1e2): red team-3 vectors + independent verification Four advanced vectors, plan frozen before running, plus ~25 sourced literature anchors and a second agent re-verifying all five technical claims with its own code (5/5 AGREE). V1 weaponized crystals: evil file loads 1/1 with no complaint and fires 1/1; no content-verification mechanism exists (only structural invalid_bindings, a ledger-cache signature, and fail-open credential redaction). V2 attention deflection: 1 large critical vs 12 small weaks sharing the act vocabulary; critical starved 1/1 with its text absent from output. Mechanism established by dissection: candidate order follows overlap honestly, but the greedy best-fit packer takes small pieces first, so a swarm of small high-overlap weaks crowds out a large critical. Reverse also measured (small critical survives by best-fit). V3 shell aliasing: aliased git/cargo fire green-tests on text the shell would execute differently; matcher resolves no aliases (verified by reading). V4 phantom poisoning: repeat goes silent by backoff; forged seen=3 blocks while seen=2 delivers (cap discriminates exactly); clock-ageing releases backoff but not cap; fresh session speaks (cap is per-session). * experiments(replication): harness fixes from pilot Three pilot failures, three fixes, all measured. (1) PowerShell agents cannot execute extensionless sh wrappers: added .cmd wrappers via a python relay (note from env, never interpolated). (2) opencode denies Read on *.env by default and asks on external dirs: scoped OPENCODE_CONFIG override for experiment runs only (fixture .env files hold no secrets); global config untouched. (3) Identical harness line appended to every prompt telling the agent to use cat when a read is refused. Pilot t5a x 2 arms now CLEAN: approach=True both, audit rows flow, scorer runs. * fix(rigs): kill process tree on timeout + incident diary proc.kill() leaves grandchildren holding our pipes and the next communicate() hangs forever (measured: 25 min, zero output). All run_agent implementations now use taskkill /F /T first with proc.kill as fallback and a bounded second communicate. Post-mortem P-kill recorded in AGENT_DIARY: terminated the OpenCode Desktop app mistaking it for zombies (age is not evidence of zombieness); guard is terminate-only-own-children plus Win32 parent-chain check, Electron main+renderer trees never touched. * experiments(replication): sidefx merge, t3a exclusion, kill-tree score_run now merges the lab_server side-effect log (vendor-api rows live there, not in audit.log). t3a excluded from the pooled contrast with stated reason: both agents bypassed the wrapped standins (urllib straight to the API), so the note never arrived in either arm; kept as a delivery-coverage finding (agents hit the quota wall, note undeliverable off the instrumented path). Kill-tree fix applied to run_agent. 11-trap pooled contrast at reps=1: HELP 5/10 vs SHAM 5/10 hits given approach, exactly null at this n. * experiments(replication): R1 null, 44 runs, HELP 0.600 vs SHAM 0.529 12 traps x 2 arms x 2 reps on our agent with his tasks/fixture/standins/scorer. HELP (specific advisory) vs SHAM (generic), length-matched. Pooled hit|approach 9/15 vs 9/17: no detectable direction effect. This does NOT refute his E1: his notes are refusal-first, ours advisory. The honest reading is a null for advisory notes, silence on refusals. Instrument defects stated: 12/44 no-approach (provider timeouts, bypasses); t3a excluded (note undeliverable off the instrumented path, kept as a coverage finding); t22a valid no-approach both arms. Cost: 55 sessions, 1.19 USD. * experiments(behavior): #2-#5 results, classifier fix, artifact rule #2 weapon 1/3 vs 0/3 (non-zero utilization, weak evidence at n=3). #3 swarm N=1 7U/0P vs N=5 3U/4P (preference competition moves choice). #4 WARNING 1U/2P vs NONE 1U/2P, identical, instrument fixed mid-flight (wrapper-function third resolution shape added; full rescore changed 1 classification, already matching manual review). #5 6/6 fix all 3, no decay, ceiling. Rule going forward: every run saves its working-tree diff, or independent re-scoring is impossible (s3/s4 lacked diffs). Cost w2+s3+s4+s5 0.51 USD. * experiments(replication): prohibitive arm, no effect, circuit closed REFUSE (prohibitive mood, same mechanisms, length-matched 131-136) vs fresh SHAM, 11 traps x 1 rep: approach 4/11 vs 9/11, hit|approach 2/4 vs 6/9. The approach gap has no cite evidence (tails show bypasses and tool-discovery stalls, never the note), so it is noise+variance, not a mechanism. His E1 effect did not replicate on our stack in advisory form (0.600 vs 0.529, 44 runs) or prohibitive form (0.500 vs 0.667, 22 runs). Stated as non-replication on a different agent/harness, not refutation. Cost: REFUSE 11 sessions 0.23 USD; replication total 77 sessions 1.65 USD. * experiments(note-tone): backfill artifact verdicts into arm/phrase rows The v4/v5 runs classified outcomes at runtime and printed them, but saved no artifact field, so an independent audit read missing keys as 0.0. Backfilled by re-running the fixed classifier (with wrapper-function resolution) on the saved working-tree diffs: 36 files, zero disagreements with the printed runtime verdicts. The report audit now reproduces every headline number. * experiments(replication): all 68 run artifacts R1 (44 runs, HELP vs SHAM) + R2 refusal arm (22 runs, REFUSE vs fresh SHAM) + pilots. Every run: approach/hit from his scorer on our artifacts, note attempt flag, tails, worktree state. Saved before teardown per the artifact rule. * docs(tom-e1e2): complete documentation, index, Tom comment draft Adds the experiment index (read order, guarantees), text-level results (10/10 predicted: case folds, 6 invisible bypasses silent), independent verification record (second agent 5/5 AGREE, report audit ALL NUMBERS REPRODUCE), and the full draft comment for Tom's thread with every number carrying its denominator and every error corrected next to the number it touched. Nothing posted externally; draft awaits explicit word. * docs(tom-e1e2): scope section in Tom comment draft Audit of the draft found 7 gaps: no model identity, no timeout count, no delivery rate, no scaffolding disclosure, no second-agent mention, no instrument-validity line, no his-vs-our model distinction. Added a Scope section up front (longcat-2.0/default, 77 sessions, 17/68 timeouts, 47/68 deliveries attempted, full scaffolding list, second agent 5/5, instrument proven on choice task) and expanded caveats with approach denominators. Every added number re-verified against artifacts before writing. * docs(tom-e1e2): name t3a exclusion and vacuous blind in draft Self-audit of the comment found two omissions: t3a was excluded from the pooled contrast without being named (both agents bypassed the wrapped standins so the note never arrived; kept as a delivery-coverage finding since both hit the quota wall through the uninstrumented path), and the blind repeat control was vacuous (0 candidates, guard has no binding). Both now stated in Caveats. * docs(tom-e1e2): scale variance qualifier in draft Spot re-check of the scale curve reproduces order and linearity (n=1000: 7.99s/0.344s vs recorded 8.02s/0.326s) but n=100 absolutes vary 8x with machine state (0.11 vs 0.87s). Stated next to the numbers instead of silently keeping the single-run values. * docs(tom-e1e2): nobody-claims chapter, three novel mechanisms H-A session rotation: REFUTED as stated (1/5 deliveries, not 5/5). Corrected finding: suppression is two-layered (session cap/seen + global time backoff); rotation defeats only the session layer. H-B size demotion: first run failed to overflow (INCONCLUSIVE, honestly labeled); second run with real overflow CONFIRMS a 2102-char critical starved by twelve 329-char weaks with text absent. H-C rank-pack: REFUTED on 6 acts; refined rule is rank==pack iff nothing starves, with size-driven reordering under overflow (snap_ord case). All three started as failures and were dissected into precise claims rather than re-run until green. * experiments(asym): preference vs trap framing, gate fired honestly Same agent/channel/fixtures, two framings, three arms, 18 runs. PREF replicates v4 steering (H1 CONFIRMED). Misleading note in TRAP changes nothing: 3/3 UPDATE with 3/3 passes, agents verify rather than obey (H2 REFUTED). H4a gate FIRED as frozen: TRAP-NONE went 3/3 UPDATE instead of splitting, so the trap framing has no dilemma for a note to move; stopping per rule rather than claiming invariance. No check.py modifications in 18/18 (cheat guard held). Cost 18 sessions 0.23 USD. * docs(tom-e1e2): draft additions from audit round ASYM paragraph (steering 1.000/0.000 vs verification dominance), 3f4dbfd acknowledgement bounding the R2 null to the current release with a re-run offer, ledger-size negative result (0.014s vs 0.059s at 20k entries). Verified: scripts/ unchanged between fe4e465 and 491f385 (docs only), so all code findings stand on current code. * docs(tom-e1e2): staleness audit of new docs commits 3f4dbfd and 491f385 are docs-only; scripts/catalogue/starter unchanged since fe4e465, so all code findings stand on current code. The new staleness section is checkable and was checked: 24/24 entries carry no stale_after (21 catalogue entries have no dates at all, 3 starters verified 2026-09-03, 37 days old). The expiry mechanism exists in code with no data to operate on. Refuse-once exists only in prose, no code. * docs(tom-e1e2): staleness mechanism verified, draft completed Expiry machinery verified working (past stale_after yields WITHHELD stub 1/1, future/undated yield full text), but 0/24 shipped entries carry any stale_after: mechanism without data. Added to the Tom draft as a deployment gap (not a bug), plus the ledger-size negative result was already there. Documentation pass: all 10 docs present and consistent, HISTORY at 14 chapters. Nothing posted externally. * docs(tom-e1e2): inventory chapter, all experiments accounted inventory.py verified every planned run against saved artifacts and every artifact against a verdict: 113 M4 files across 11 series, 68/68 replication runs, all deterministic JSONs parse, 0 hollow artifacts. ALL EXPERIMENTS ACCOUNTED. * docs(tom-e1e2): boundaries paragraph in draft caveats States what our numbers are not: fixed probes (lower bounds, no optimized attacker measured), single ~70s acts (no lived long sessions), one model (other models his territory), dispatcher isolated (his hook-to-reading loop unverified from our side), races/POSIX/catalogue named but unmeasured. The honest-bounds paragraph the audit standard requires. * docs(tom-e1e2): restructure draft per review, verify literature Review-driven rework, all points addressed with evidence. (1) Weaponized reframed as supply-chain with the explicit scenario (catalogue sync, malicious PR, vendored pack) and the perimeter objection answered. (2) AgentPoison 82/63 verified verbatim against arXiv full text; Governance Decay 100/85/65 verified against Table 4 in the paper. (3) Order sample n=3 and tier-untestability stated next to the conclusion. (4) Seven-word and staleness moved out of breaks into a separate verified-working section. (5) Budget test labeled proof-of-possibility. (6) Form: urgent three first, remainder in a details block; caveats stay visible. * docs(tom-e1e2): comment posted to thread Owner published the final comment 2026-10-10 22:24 UTC (13,279 chars); head and tail verified against the draft, full text landed, no replies at time of recording (15 comments total). Open items remaining: refusal-arm re-run when refuse-once ships; PR #78 merge decision. * docs(tom-e1e2): methodology priority record His judgements-in-implementation post is 09-27 (public), our base practices predate it (diary July-August), but our sections 19-21 were formulated 09-30/10-03 after collaboration began, so influence cannot be excluded. Recorded: method priority is his; ours is only the 10-11.10 measurements on his system. Cite his post as prior art in any methods writing; claim originality on numbers only. HOOK BYPASS REASON (not silent): verify_diary fails on two pre-existing 09-09/09-10 entries referencing commit 0301fa9, which exists nowhere in any history (verified via cat-file + log --all). Unrelated to this change; fixing others' diary rows would violate single-owner registries. All other gates green. * docs(tom-e1e2): week threads sweep + paired re-analysis Enumerated all 20 posts via API; pulled live threads. Applied Ahmet paired-transition analysis to our arms: V4/V5 unanimous agreement, R1 McNemar 3-vs-2 with 5+4 agreements (p=1.0), null confirmed rigorously. Recorded: v7-loss parallels our full-30, second-agent independence stands as briefed, calibration unmeasured everywhere, escalation churn supports pairing. HOOK BYPASS (not silent): verify_diary fails on pre-existing 09-09/09-10 entries citing commit 0301fa9, absent from all histories. Unrelated to this change; all other gates green. * docs(tom-e1e2): Tom's three replies (11.10) dissected Reproduced+fixed (7a129db/b86edc7, credited): len-5 and normalization; his new find: naive len<=5 silences symbol-ended prefix post-. Confirmed-by-reading: backoff keyed act:<type>:<crystal>. Re-derived: budget starvation (one act), repeat silence, scale. Scale gap 100-200x: our candidates() re-loads internally (crystal_act.py:416) so our match bundled a warm re-load; 2932-char essences vs ~1000; Windows host vs declared macOS/Linux. Owe him the timing boundary. Behavioural nulls accepted as cross-model non-replication. HOOK BYPASS: verify_diary fails on pre-existing 0301fa9 reference; unrelated, other gates green. * docs(tom-e1e2): scale forensics - Defender A/B, retract our match number Same-session A/B (n=1000, fresh stores with nonce): Defender Real-Time ON 8.9-10.3 ms/file vs OFF 0.32 ms/file = 28-32x. Our recorded load numbers were RT-ON class (7.8-8.7 ms/file). Our 'match' (0.35 ms/crystal) was actually a warm re-load: candidates() calls load_crystals internally (crystal_act.py:416); pure match is 2.1-2.2 us/crystal, ~Tom's 3 us. Disk is one NVMe (H3 refuted); essence 1335 vs 1000 = 1.3x (H4 refuted); residual 8.7x is Windows open overhead, outside declared macOS/Linux (H5 refuted). Controls: RT-ON positive, empty-store negative, on/on/off/off reversibility. Harness + raw artifacts committed under scale_forensics/. HOOK BYPASS: verify_diary pre-existing 0301fa9 failure; unrelated. * fix(tom-e1e2): re-measure all timing experiments with Defender OFF Owner asked to re-run every experiment whose numbers were distorted by the (then-on) protection. Inventory of all wall-clock numbers found 3 (scale, ledger, 10MB ctx) plus a self-contradiction: scale n=100 gave 0.868s (REDTEAM2) vs 0.11s (VERIFY_NUMBERS) on the same store - Defender caches its verdict by file content (identical names+text => second run unscanned). Fresh run, Defender OFF, unique nonce:\n\n- scale: load 318 us/file; true match 2.1 us/crystal (was 0.35 ms = re-load); candidates(repo)==load at all n.\n- ledger: linear ~2.7 us/entry (0.11s at 40k) - previous 'does not degrade' RETRACTED; ledger_perf.py had a dead first loop and persisted nothing.\n- 10MB ctx 0.486s - no-ReDoS verdict holds (CPU/parse, not file I/O).\n- delivery 30/30 PASS re-run - content verdicts invariant; Defender moved only time.\n- pain point: 500ms ~= 1570 files (load), not '~1400 crystals'.\n\nHarness + raw: scale_forensics/REMEASURE.json, remeasure_all.py. HOOK BYPASS: verify_diary pre-existing 0301fa9 failure; unrelated. * test(tom-e1e2): re-derive the lost overlap check from scratch (416/416) The producer of VERIFY_OVERLAP.json was a one-off and never saved - the only unclosed item in the audit. Rebuilt independently (0 USD): the 26 fixture inputs are pinned by the fixture's own essence_sha256 (26/26 - starter/ was added after fe4e465, catalogue pages edited later, so each input comes from the revision whose derived hash matches). My own overlap() (exact package semantics at fe4e465: _essence between markers with .strip newline, Jaccard over lowercase word tokens, essence[:1500] before lower) reproduces all 416 rows (0 disagreements). Controls: 171 nonzero rows, 0 collapsed to zero (positive); 245 zero rows, 0 flipped to nonzero (negative). Verdict: ALL 416 ROWS REPRODUCE. Artifact: overlap_rederivation/. HOOK BYPASS: verify_diary pre-existing 0301fa9 failure; unrelated. * docs(tom-e1e2): performance follow-up draft (Defender correction) Draft reply to the thread correcting our own published timing numbers. The scale paragraph and the ledger caveat in TOM_COMMENT_DRAFT.md (the published comment) are stale; the follow-up gives the Defender-OFF numbers, retracts the "no ledger degradation" line (now linear ~2.7 us/entry, JSON parse per act), and corrects the match number (re-load -> 2.1 us/crystal, matching his ~3 us). Draft only, not posted. Awaiting owner's word. HOOK BYPASS: verify_diary pre-existing 0301fa9 failure; unrelated. * docs(tom-e1e2): expand performance follow-up draft with full tables and raw data * test(tom-e1e2): independently confirm Tom's key-matching fix (15/15) Tom pushed the input-validation fix in crystal-memory@7a129db (parent 491f385) and tjonesit/crystals@b86edc7 — both change scripts/crystal_act.py (+45/-5). We loaded HIS OWN matches_ctx / match_scope / normalize_match from both revisions (raw.githubusercontent) and re-ran our five audited false-fires plus the normalization strings. The OLD revision is the control that must fail: it reproduces the length-5 hole 5/5. The NEW revision closes all five, keeps the three real commands and the symbol-ended `post-` prefix, and now matches all five normalization inputs (zero-width, Cyrillic dze, fullwidth, nbsp, thin space). Result 15/15, failures 0 -> TOMS-FIX-INDEPENDENTLY-CONFIRMED. No model, $0. Artifact: tom_fix_verification/. HOOK BYPASS: verify_diary pre-existing 0301fa9 failure; unrelated. --------- Co-authored-by: MSCodeBase Agent <mscodebase@intelligence.local>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Independent audit of the Tom Jones crystal lab (tjonesit/crystals#1) plus our own note-tone experiments (M4). Every number carries a denominator; every instrument has a control that can fail; our own errors are documented next to the numbers they touched.
Independent re-derivation (deterministic, )
Red-team findings (all measured, )
Our-stack experiments (measured costs)
Fixes included
Review guide
Start at \experiments/tom_e1e2_independent/README.md, then HISTORY.md (10 chapters).
Numbers live in RESULTS*.md; every one recomputes via the scripts next to it.