[johnml1135]

Start here: conformance/docs/what-it-is.md, then .github/workflows/conformance.yml: what the suite claims and the gate that enforces it.

Status: stacked on #500 — merge in order #556 → #500 → #480 → #578.

Status: CI is red at head 262ab300. Both Build jobs fail ConformanceFixtureGateTests.EveryConformanceFixturePassesSelfCheck on four edge-cases/truncate-morphotactic* fixtures (word analyses differ from expectation), and both Conformance jobs fail, with the workflow reporting conformance/coverage.csv or rules.csv stale against the fixtures. The branch is behind master. Work is held until after the Hermit Crab Optimizations release; nothing stacks on this PR.

A conformance suite of synthetic HermitCrab grammars, run in-process with HermitCrab as the oracle, with CI gates for parse results, coverage claims and generated ledgers. The latest revision adds named parser-settings variants: the runner builds a fresh morpher per run, applies the variant's settings, and compares complete parse-identity multisets and statuses against that run's own expectations. Variants earn no fixture or semantic-coverage credit.

Why is it so big? The change is about 940 files. Harness code and its tests are the reviewable part; the rest is fixtures, generated ledgers and evidence cards, and docs. Generated files are rewritten by hc-conformance --write-* and pinned by tests, so check freshness, do not read line by line. No parser engine changed.

Where to look:

• .github/workflows/conformance.yml: the gate (self-check, coverage report, freshness of generated CSVs).
• conformance/dead-rules-allowed.tsv: a rule no word exercises fails the coverage report unless listed with a reason.
• conformance/PROTOCOL.md and WordsYamlLoader.cs: the adapter contract; settings are closed, so an unsupported setting fails rather than being silently ignored. Pinned by WordsYamlMorpherSettingsTests.
• src/SIL.Machine.Morphology.HermitCrab.Conformance/Runner.cs: self-check.
• global.json: new; pins SDK 10.0.100 with latestFeature roll-forward for the whole repository.

Deliberately not here: MaxAlternatives (PanGloss lacks HC's typed overflow behaviour) and MergeMSAs (needs the FieldWorks project loader) are unsupported variant settings. MergeAnalyses verifies output preservation only, not merge-fold counts. Suite-side work for #534 (open) waits for its merge.

Verification: the earlier numbers (local_check.sh --agent-strict, C# and PanGloss variant matrices) were taken at 184bcd46, not at the current head, and are not repeated here; the red jobs above are the current evidence. Broader Rust workspace and Foma suites were not run.

Next: fix the four truncate-morphotactic* fixtures and regenerate the CSVs, then rebase; or say if the red gate should block holding this branch.

---

Reading this a year from now -- start here

The suite is an executable adequacy argument: synthetic grammars whose expected parse-identity multisets are checked against HermitCrab (self-check) and against a second engine (PanGloss) through an adapter protocol. The decisions below explain why the gates look the way they do.

Decisions and why

• Variants as run-level settings. Settings come from constructor defaults and per-call GuessRoots; supported: DelReapps, MaxRoots, MakePartialsFinal, MergeAnalyses, GuessRoots. The main risk is claiming support for a setting an adapter cannot apply or treating incomplete output as an empty result, so unsupported settings, missing rows and zero-comparison variants fail, and PanGloss also rejects capped or timed-out output.
• Source-generator loading. PCRE.NET ships an IIncrementalGenerator; the compilation-graph loader admits it only by content hash, and a probe runs it and fails closed on any generated output.
• Crash fixtures. An expect_crash fixture passes only if the same engine completes an ordinary fixture in the same run, so a broken adapter command cannot pass as a crash (conformance/PROTOCOL.md).
• Binary .fwdata. Marked binary in .gitattributes because its pinned hash depended on core.autocrlf. Four fixture grammars keep trailing whitespace because their hashes are pinned in the manifest. Hand-maintained fixture totals were removed from docs; the manifest is the count.
• Trace.FailureAllomorph dropped. An engine API change does not belong in a test-suite PR; Trace HermitCrab failure reasons and template slot outcomes #534 adds the same fact through its own trace field.
• ci.yml back to master. Its added fixture run duplicated ConformanceFixtureGateTests under dotnet test.

Surprising findings

• The coverage report could not fail; it now gates dead rules and uncovered constructs.
• The counterfactual sweep read mutant output in the console code page, so a mutation that changed nothing looked like evidence. It now reads UTF-8 (CounterfactualGate, with a regression test). Regenerating the ledger demoted two surfaces that had only that false evidence (Gloss to Unobservable, HeadFeatures to RequiredByDtd).
• local_check.sh ran --semantic-coverage, which exits 1 by design while the catalog is incomplete, so it could never pass; it now runs the same gates as conformance.yml.
• chained-output-feature-override-loss claimed to fail on master on purpose; it has passed since Change Add to PriorityUnion #494 (merged).

Deferred, and what would unblock it

#540, #522 and #519 are merged and #543 is closed; their suite-side work is already in this branch or no longer stacks here. #534 is open: once it merges, FailureRuleAttributor should credit through its new trace field and the FieldWorks-producibility ledger needs rows for its four new failure reasons. #543 would have added fixtures for the #504 and #507 evidence gaps; they are not in this branch. Unsupported variant settings (MaxAlternatives, MergeMSAs) need PanGloss overflow behaviour and a FieldWorks project loader respectively.

Evidence and oracle observations

C# HC observations for the variants were collected on 2026-10-22 with the engine unchanged; the five measured fixture/settings pairs cover deletion reapplications (DelReapps), compounding breadth (MaxRoots), final-template partial discriminators (MakePartialsFinal), the final-template merge key (MergeAnalyses) and guess-roots (GuessRoots). Exact witness outputs and reproduction commands are in conformance/docs/parser-settings-variants.md. The variant protocol forbids capped or timed-out output from matching an expected empty result; the PanGloss tests include that negative control. The companion PanGloss adapter was tested at e2627c9 on feat/conformance-settings-variants. Counts quoted in earlier revisions of this description (tests, runs, fixtures, words) were taken at an older head and are not carried forward.

🤖 Generated with Claude Code

---

This change is

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.11%. Comparing base (b86120f) to head (9fd3f8a).
⚠️ Report is 1 commits behind head on fix/signature-drops-id-under-identity-rule.

Additional details and impacted files

@@                              Coverage Diff                               @@
##           fix/signature-drops-id-under-identity-rule     #480      +/-   ##
==============================================================================
+ Coverage                                       74.62%   75.11%   +0.48%     
==============================================================================
  Files                                             453      556     +103     
  Lines                                           38345    56531   +18186     
  Branches                                         5285     7719    +2434     
==============================================================================
+ Hits                                            28614    42461   +13847     
- Misses                                           8568    12210    +3642     
- Partials                                         1163     1860     +697     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[johnml1135]

Added two independently justified conformance fixtures in a20bce1: final-template-partial-discriminators (10 words; #507) and template-category-sharing (6 words; #505). Fresh Release build: zero warnings/errors. Both fixtures pass with memoization on and off, zero skipped; 22 focused manifest/coverage tests pass. Four grammar counterfactuals produce the independently predicted mismatches, documented under conformance/docs. These are grammar-sensitivity controls, not proof that pruning, cache hits, or same-key feature collisions execute. No parser code changed. Generated inventory also restores previously omitted rows for the already-existing Exact-inverse fixture; its valid zudiua loss remains tracked in #504, not relabeled as correct.

[johnml1135]

Added the shared-template unconstrained-suffix conformance fixture in f150e2a. The grammar has a slot-only feature-unconstrained suffix shared by mandatory TV/IV templates, an ordinary N-to-IV verbalizer, four positive rows, and six rejection controls. The companion rationale explains complete forward derivations and loader reachability (not a FieldWorks UI round trip).

Fresh C# harness: all 10 rows pass in both template orders, memoization on/off (40 comparisons; no skips). Focused template/manifest tests: 22 passed, no skips. Regenerated manifest is current at 40 fixtures; coverage/rule reports updated. No Machine parser code changed.

PanGloss main now contains the narrow gate-only alignment (fc357daf), with this fixture mirrored unchanged until its old Machine submodule pin is upgraded. Direct state assertions provide red/green evidence; full-parse fixture checks pass both before and after the change. Issue #505 remains open for collision/correlation and broader merge-equivalence evidence.

[claude]

FYI: F40. The regenerated engine-gate-inventory.tsv header (line 9) still says '-' is a genuine finding for six gates, but adding AllomorphNotApplicable this round makes seven. The others are DisjunctiveAllomorph, Environments, HeadPattern, NonHeadPattern, Pattern and SurfaceFormMismatch. This line writes that header. The class summary at line 18 also still says 23 gates. Change them to seven and 24, then regenerate.

[claude]

FYI: F41. This index still reports 14 of 46 gate arms evidenced, 5 of 23 gates with both MC/DC arms. what-it-claims.md, updated this round, reports 18 of 48 and 7 of 24, and GateObligationLedgerTests pins those numbers. A reader who starts here gets the old figures. Copy the current ones over.

[johnml1135]

Claude-drafted comment, posted on behalf of @johnml1135.

Pushed 9b2ff58 (fast-forward from cf84539): four new edge-case fixtures from PanGloss, each replayed on C# hc.dll and matching (boundary-narrowing-probe 3/3; rewrite-default-environment, -epenthesis, -narrow 7/7 each). Each pins a bug PanGloss had and C# did not (UseDefaults in rewrite matching, sillsdev/PanGloss#41; boundary-initial narrowing probe, sillsdev/PanGloss#14). The ledgers gain only rows for these fixtures. Self-check 154/154, coverage 32/32, manifest 144 fixtures.

Pre-existing staleness found on cf84539 and left alone:

• semantic-coverage-evidence.tsv ordering witness for optional-template-composite t2pfx~t2mandatory: the current engine changes 0/11 results when those slots are swapped.
• interface-witness.tsv recomputation drift for truncate-morphotactic*, and a varying stack-overflow excerpt for suffixing-extension-slot-ordering.
• conformance/parity-check.py fails on guess-roots-settings/words.yaml, which has no requires.
• The weekly counterfactual CLI compares witness arrays by reference (190 spurious changed entries), plus 6 unresolved and 3 rejected surface items.

[claude]

FYI: F42. pr1 is listed in dead-rules-allowed.tsv as never exercised, but bdt here is rejected because of pr1. The lexicon has bdt (grammar.xml:127), and pr1 deletes its t, so it never surfaces. Add blocked_by: [pr1] to this row: the dead-rule gate credits blocked_by (CoverageReport.cs:166), so the allowlist entry can go and a regression in pr1 stays visible.

[johnml1135]

Claude-drafted. Added rewrite-default-full-set (bdd369f). It comes from PanGloss#85: the Rust port treated a char def that writes a feature's whole symbol set out in full as absent under rewrite defaults. Replayed on hc.dll at aab4c59: bat and bdt both give |b[ad(aa)]t, and bt, ba, b, a and d give no parse, matching words.yaml. grammar.xml and words.yaml are byte-identical to the PanGloss staging copy. The witness-ledger regeneration also wanted to change pre-existing stale rows (truncate-morphotactic*, one suffixing-extension-slot-ordering timeout); I left those at their checked-in values, as in the previous batch.

[claude]

FYI: F43. This says the seven expectations were recorded at Machine commit aab4c598, but that commit is from 2026-10-22 and PanGloss#85, which this fixture follows up, was opened 2026-10-30. The expectations are right at this head (self-check passes), so only the provenance is in doubt. Name the build actually used, or drop the commit. Unverified: I could not tell which build produced them.

[johnml1135]

Claude-drafted.

Added five ad hoc prohibition cases in 49bbe7f (fast-forward from bdd369f). Each pins a copy HCLoader registers that a port must also cover:

Case	Copy
adhoc-prohibition-stem-msa-copies	Every MSA of a multi-MSA stem
adhoc-prohibition-affix-msa-copies	Every MSA of a multi-MSA affix entry
adhoc-prohibition-affix-environment-copies	Every environment of an affix allomorph
adhoc-prohibition-circumfix-pairings	Circumfix pairings
adhoc-prohibition-variant-msa-copies	Variant entries that borrow a main entry's MSA

These came from sillsdev/PanGloss#99 and sillsdev/PanGloss#100, fixed in sillsdev/PanGloss#101 and sillsdev/PanGloss#102. Each case carries a LibLCM-authored fieldworks/ witness, per PROTOCOL.md section 10.

Run on the commit before project.fwdata was force-added (the files were on disk; a local exclude had kept them out of the commit): conformance self-check 159 runs, 0 failures, 150 fixtures; 149 count-pinning, framework and fixture-gate tests, 0 failures; construct coverage 32 of 32. All 45 words replay on hc.dll at aab4c598 to words.yaml. After the amend, each committed project.fwdata matches its base_sha256. No test was rerun on the amended commit; CI on this head is the first run with the files in the tree.

The interface inventory moves from 52 to 54 typed edges, because Allomorph is now a target of AllomorphCoOccurrenceRule.otherAllomorphs.

[claude]

FYI: F44. These 18 new rows call each fixture's prohibition rules dead, yet those rules are what the fixtures test: tada/taba (and the circumfix and variant equivalents) fail because of them (affix-msa-copies/grammar.xml:84-87 names subTa1, subTa2, aBa as primaries). suffixing-evidential-adjacency-chain/words.yaml:263 already credits this kind of rule with blocked_by: [aKanta], which CoverageReport.cs:166 counts. Add blocked_by to the expect_fail words and drop these rows. The reason text also names tadi/tabi, which only exist in the variant fixture.

[claude]

Approve with fixes. Follow-up on bdd369f. One new commit, 49bbe7f (fast-forward). It adds five adhoc-prohibition-* fixtures and regenerates the ledgers and test pins. No harness or engine code changed.

1. Verdict: approve with fixes. The five fixtures are consistent with their grammars: each prohibited word is expect_fail, each has a control that still parses, and each README records the negative controls run against a partly wired loader.
2. Most important: F35 (conformance/interface-witness.tsv:709) is still open and this commit does not touch it. New this round: F44 (conformance/dead-rules-allowed.tsv:24). The new allowlist rows mark the prohibition rules of their own fixtures as dead, when blocked_by would credit them.
3. Counts: 0 Critical. 2 Important open (F35, F38). 7 Low open (F36, F37, F39, F40, F41, F42, F43), plus 1 new Low (F44).
4. What I ran:
   • gh pr checks 480: both Conformance suite runs and both Comment hygiene runs pass. Both Build jobs and claude-review are pending.
   • Locally, dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests -c Release, filtered to the conformance fixture gate, manifest, interface-witness, interface-inventory, grammar-coverage, construct-corroboration and ordering-generator tests: 78 of 78 passed.
   • I read the new words.yaml, READMEs and allowlist rows, and checked the co-occurrence pseudo-ids in GrammarRuleIndex.cs:96-103. I did not run /code-review.
5. What I could not verify:
   • the .fwdata witnesses or PanGloss behaviour;
   • the HCLoader.cs line ranges, which are FieldWorks source and are cited inconsistently: LoadAllomorphCoOccurrenceRules is 2163-2189 in two fixtures and 2212-2236 in two others;
   • which build recorded the expectations. The new fixtures repeat the aab4c598 provenance that F43 questions. Their verified-date is 2026-10-30/31, but that commit is from 2026-10-22.

F39 grew this round: the five new witnesses add 20 more full-CLDR WritingSystemStore/*.ldml files, 614,775 lines and about 27 MB. The thread is unchanged, so I did not post a reply.

Public API, target framework, package, or sillsdev/machine.py parity: None verified.

Finding	Severity	Status
F1	Critical	addressed (cc947db)
F2	Important	addressed (bcb3d62)
F3	Important	addressed (bcb3d62)
F4	Important	accepted, unverified
F5	Low	addressed (bcb3d62)
F6	Low	addressed (067de6f)
F7	Low	addressed (0a1212b)
F8	Low	addressed (067de6f)
F9	Low	addressed (067de6f)
F10	Important	addressed (cd0c909)
F11	Low	accepted, unverified
F12	Low	addressed (cd0c909)
F13	Low	addressed (0a1212b)
F14	Low	addressed (0a1212b)
F15	Low	addressed (067de6f)
F16	Low	addressed (067de6f)
F17	Low	addressed (0a1212b)
F18	Low	addressed (0a1212b)
F19	Critical	addressed (e7fe733)
F20	Low	addressed (8b5d74e)
F21	Low	addressed (1c5c3a5)
F22	Low	addressed (a3b4c0e)
F23	Important	addressed (49ba481)
F24	Important	addressed (49ba481)
F25	Low	addressed (9671b28)
F26	Low	addressed (641f613)
F27	Low	addressed (8b5d74e)
F28	Critical	addressed (dd5f00f, 2cd68e8)
F29	Low	addressed (bb42ff4)
F30	Low	addressed (f592706)
F31	Low	addressed (6c6fbb3)
F32	Critical	addressed (cf84539, stacked on #500 and #556)
F33	Low	addressed (b9c7c8a)
F34	Low	addressed (382168a)
F35	Important	open
F36	Low	open, unverified
F37	Low	open
F38	Important	open, unverified
F39	Low	open
F40	Low	open
F41	Low	open
F42	Low	open
F43	Low	open, unverified
F44	Low	new

Reviewed at 49bbe7f

[claude]

No new findings. Reviewed at 9fd3f8a

[johnml1135]

Claude-drafted.

Correction to the note on 49bbe7f: that commit failed OpenCoverageDebtMatchesTheNamedItemAndResolutionBaselineExactly on both Build jobs. The new adhoc-prohibition-* fixtures sort first, so the generator gave them 43 dtd: rows, while semantic-coverage-evidence.tsv and grammar-coverage-ledger.tsv still named the old fixtures.

9fd3f8a regenerates those two ledgers with --write-coverage-evidence and --write-coverage-traceability. The rows for truncate-morphotactic*, suffixing-extension-slot-ordering and optional-template-composite stay at their previous values, so the diff is only the 43 moved rows. All checks pass at 9fd3f8a, including both Build jobs and both Conformance suite runs.