test(hermitcrab): opt-in stress grammars for optimization parity (not conformance) by johnml1135 · Pull Request #578 · sillsdev/machine

johnml1135 · 2026-10-28T13:44:16Z

Start here: conformance-stress/README.md -- it lists the 18 grammars, what each exercises and which optimization it was built for.

Status: stacked on #480 — merge in order #556 → #500 → #480 → #578.

Status: draft. Stacked on #480 (open); merge #480 first, and the base integrate-conformance-framework has moved since this was cut. CI is red: the Build jobs (ubuntu and windows) fail EveryConformanceFixturePassesSelfCheck, and the Conformance suite job fails 3 truncate-morphotactic fixtures (truncate-morphotactic-copy-u, -identity, -subsumption). Both name the truncate-morphotactic family that #480 already treats as nondeterministic, not these grammars; not compared against #480's own CI.

Adds 18 HermitCrab stress grammars in a new conformance-stress/ directory, a sibling of conformance/, for optimization work. They are not conformance fixtures and do not run by default. Each models a language type where one analysis or synthesis optimization matters: Bantu slot templates, agglutinative early-lexicon roots, dense co-occurrence, MPR concord, environment-conditioned allomorphs, and others. Expected parses are master's output, so a fixture is a parity check: an optimization must leave parses unchanged. Work is compared in step counts, not time.

The unknown a reviewer arrives with is "does this change CI or the conformance numbers?". It should not: nothing that enumerates fixtures scans conformance-stress/, and no workflow selects it.

Where to look

tests/SIL.Machine.Morphology.HermitCrab.Tests/StressFixtureParityTests.cs: [Explicit] and [Category("Stress")], so a plain dotnet test reports it not executed. Opt in with dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests --filter TestCategory=Stress.
Runner: opt in with --fixtures conformance-stress --include-pathological; the conformance runner needs --fixtures <path> and is otherwise pointed at conformance/.
Isolation: GrammarCoverageGate, ledgers, manifests and counterfactual coverage read conformance/ only; no workflow (ci.yml, conformance.yml, counterfactual-coverage.yml) mentions conformance-stress.
Marking: each of the 18 words.yaml files starts with # kind: stress.

Not here: polundanasak in early-lexicon-agglutinative (engines before #548, merged, trace an extra phonological rule for it); a trace-memory fixture with up to 864 parses per word; any marking of the flaky truncate fixtures (#480 has expect_fail, expect_skip and expect_crash, and no known-nondeterministic state).

Verification: parity runs and the plain dotnet test and runner comparisons against #480's head were done on the earlier 16-grammar version; they were not re-run for the 18-grammar head, so no pass counts are claimed here. csharpier and comment hygiene were clean then, and the comment-hygiene jobs pass now.

Next: confirm conformance-stress/ as an opt-in directory here, and settle #480 so this can leave draft.

Reading this a year from now -- start here

These grammars exist so an analysis or synthesis optimization can be checked for unchanged parses and measured in steps, without adding to the conformance suite's runtime or its coverage ledgers. The first 16 were added together; bantu-tam-fusion (built for the impure-slot look-ahead stacked on #576) and truncating-slot-chain (the worst case for the supply-set look-ahead stacked on #579) were added after. Both new grammars are synthetic, not real languages.

Decisions and why

Separate directory, not a flag inside conformance/: the coverage gate, ledgers, manifests and counterfactual coverage all enumerate conformance/, so nothing there has to learn to skip these.
[Explicit] plus a category: the test reuses Fixture.DiscoverAll and Runner.RunSelfCheck from the conformance runner, so no new harness.
Expected parses come from master, three identical runs required (per the README).

Paths not taken

Marking the flaky truncate fixtures was left out because #480 has no known-nondeterministic state. A trace-memory fixture was left out because its words have up to 864 parses each.

Deferred

Whether the red Build and Conformance jobs match #480's own was not checked against #480's CI. A known-nondeterministic state in the conformance framework would let the truncate family stop failing CI; that belongs to #480.

🤖 Generated with Claude Code

This change is

Adds the round-doc section "Combined stack and R2's cost": the full stack against master (work, unification, matcher per grammar), additivity (near-additive on Mbugwe, overlapping on Sena, super-additive on Aweti), R2's root cause (its own probe, low prune rate) and the witness-reuse fix folded into #572 @ 01bdf5a, the 1-4 point residual, and the landing decision: #572 is a draft and lands only on top of #577 and #574. Stress grammars are PR #578 (opt-in, [Explicit], not run in CI). Ledger: row 40 (R2) Outcome records the fold and the landing decision; new row 47 records the combined stack measurement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude · 2026-10-28T13:49:37Z

+        RunReport report = Runner.RunSelfCheck(
+            fixtures,
+            includePathological: true,
+            engineCapabilities: new HashSet<string>(StringComparer.Ordinal) { "phonology", "morpher-settings" },


FYI: F1. This copies the capability set instead of using the engine's own, so once SelfCheckEngine gains a capability that a stress grammar requires, this test skips that fixture, but the README's CLI command still runs it. new SelfCheckEngine().Capabilities stays in step (SelfCheckEngine.cs:22).

claude · 2026-10-28T13:49:39Z

+            proposeOutput: TextWriter.Null
+        );
+        Assert.That(report.Failed, Is.EqualTo(0), string.Join("; ", report.Results.Select(r => r.Reason)));
+        Assert.That(report.Passed, Is.EqualTo(1));


FYI: F2. A skipped fixture fails here with only Expected: 1 But was: 0. The skip reason (Runner.cs:98-100, "missing [...]") is in report.Results but is not passed to this assert. Pass the same Reason join used at line 39 as the message.

claude · 2026-10-28T13:49:41Z

+[Category("Stress")]
+public sealed class StressFixtureParityTests
+{
+    private static string StressRoot()


FYI: F3. This is a seventh copy of the ancestor walk that six test files here already have as RepositoryRoot(), which anchors on conformance/constructs.txt. This copy stops at the first ancestor with any conformance-stress directory. Path.Combine(RepositoryRoot(), "conformance-stress") would anchor on the same root as the others.

claude · 2026-10-28T13:49:43Z

@@ -0,0 +1,554 @@
+# kind: stress (timing and optimization parity fixture, not conformance; see conformance-stress/README.md)
+# StressAthabaskanPrefixComplex: dense-exclusion stress fixture, generated by gen_dense_exclusions.py. The expected parses


FYI: F4. The provenance can't be reproduced: gen_dense_exclusions.py, gen_excluded_cooccurrence.py, Morpher.PruneExcludedCoOccurrence, PruneByTemplateReplay and "the E1 cutoff" are not in the tree (searched *.cs, *.py, *.md outside conformance-stress/). The same applies to bantu-extension-chain, excluded-cooccurrence, required-template-slot and prefix-class-density. Either link the branch or PR that has them, or say the expectations are master's output, as the README does.

F4 still applies at 2bf16fa: the generator scripts are still not in the tree. This commit also regenerated the S6e rows here from the engine, so the header line saying the parses are "not from the engine" is now out of date as well.

claude · 2026-10-28T15:10:14Z

+fieldworks_producible_notes: "research stress fixture; not authored through FieldWorks"
+budget_ms: 60000
+words:
+  - word: wnonemileila


Minor: F5. No expected parse uses a fusion allomorph, so the parity check cannot catch an impure-slot look-ahead that wrongly prunes one. All 16 words shaped like a fused subrule output fail on master, for example tkokaawabani = fsub_subj_*_to on kaawabani. That accounts for 28 no-parse words here against 12 in bantu-tam-templates. The fsub_* output (grammar.xml:46) also is not vowel fusion: it copies the first stem segment on both sides of the inserted vowel. Could you make the fused forms parse on master, or note in the README that they never do?

codecov-commenter · 2026-10-29T19:03:56Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.10%. Comparing base (f459717) to head (2bf16fa).

Additional details and impacted files

@@                       Coverage Diff                        @@
##           integrate-conformance-framework     #578   +/-   ##
================================================================
  Coverage                            75.10%   75.10%           
================================================================
  Files                                  556      556           
  Lines                                56531    56531           
  Branches                              7719     7719           
================================================================
  Hits                                 42458    42458           
  Misses                               12213    12213           
  Partials                              1860     1860

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…nce-stress, outside the conformance suite and CI Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…for the impure-slot look-ahead stacked on #576. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…he worst case for the supply-set look-ahead stacked on #579. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude · 2026-11-01T00:57:45Z

Verdict: approve with fixes. The new commit regenerates the five null-affix fixtures using FLEx boundaries, and all 18 stress fixtures now pass.
Most important: F5, conformance-stress/edge-cases/bantu-tam-fusion/words.yaml:9. None of the expected parses uses a fusion allomorph, so the fixture still cannot catch the pruning it was added for.
Counts: Critical 0, Important 1, Low 4.
Ran dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests --filter TestCategory=Stress on the merge checkout: 18 passed, 0 failed, in 13 s. Compared tree hashes against the last reviewed commit a37fa2af: StressFixtureParityTests.cs is identical, and only 2bf16fa changes conformance-stress/. Per gh pr checks 578, Conformance suite and Comment hygiene pass; both Builds and claude-review are pending.
Not verified: whether master, rather than the Add a conformance suite for HermitCrab, with its adequacy argument #480 base this PR targets, produces the regenerated rows. The README now names Add a conformance suite for HermitCrab, with its adequacy argument #480 as their source. Also not verified: the Build results while they are pending.

Public API, target framework, package, machine.py parity: None verified. Since the last round, only fixture data and the README changed.

Findings: F1 open, F2 open, F3 open, F4 open, F5 open (mechanism unverified), F6 addressed.

Reviewed at 2bf16fa

claude Bot reviewed Oct 28, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

claude Bot mentioned this pull request Oct 28, 2026

HermitCrab optimization round record: what was tried, and a faster package that is not mergeable #490

Closed

johnml1135 mentioned this pull request Oct 28, 2026

perf(hermitcrab): keep the required-slot look-ahead testing behind slots whose unmatched parts are unbounded runs #579

Merged

claude Bot reviewed Oct 28, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

johnml1135 mentioned this pull request Oct 28, 2026

perf(hermitcrab): keep the required-slot look-ahead testing behind slots that write back deleted stem segments #580

Draft

This comment has been minimized.

Sign in to view

jtmaxwell3 added the Hermit Crab label Oct 28, 2026

johnml1135 mentioned this pull request Oct 29, 2026

Add a conformance suite for HermitCrab, with its adequacy argument #480

Open

johnml1135 force-pushed the integrate-conformance-framework branch from 262ab30 to cf84539 Compare October 29, 2026 18:46

johnml1135 force-pushed the conformance/analysis-stress-fixtures branch from df134b0 to a37fa2a Compare October 29, 2026 18:46

claude Bot reviewed Oct 29, 2026

View reviewed changes

Comment thread conformance-stress/edge-cases/env-bantu-concord/words.yaml Resolved

This comment has been minimized.

Sign in to view

This was referenced Oct 29, 2026

Fix directional annotation tie boundaries #556

Open

Fix a dropped morpheme ID when a zero-width rule wraps one ending at the same node #500

Open

johnml1135 and others added 4 commits October 31, 2026 15:56

Add 16 analysis stress grammars as opt-in timing fixtures in conforma…

7a56ac5

…nce-stress, outside the conformance suite and CI Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add the bantu-tam-fusion stress grammar to conformance-stress, built …

89a1ff5

…for the impure-slot look-ahead stacked on #576. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add the truncating-slot-chain stress grammar to conformance-stress, t…

ced6669

…he worst case for the supply-set look-ahead stacked on #579. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Represent stress null affixes with FLEx boundaries

2bf16fa

johnml1135 force-pushed the conformance/analysis-stress-fixtures branch from a37fa2a to 2bf16fa Compare November 1, 2026 00:53

Navigation Menu

Repository navigation

test(hermitcrab): opt-in stress grammars for optimization parity (not conformance) - #578

test(hermitcrab): opt-in stress grammars for optimization parity (not conformance)#578
johnml1135 wants to merge 4 commits into
integrate-conformance-frameworkfrom
conformance/analysis-stress-fixtures

johnml1135 commented Oct 28, 2026 •

edited

Loading

Uh oh!

claude Bot Oct 28, 2026

Uh oh!

claude Bot Oct 28, 2026

Uh oh!

claude Bot Oct 28, 2026

Uh oh!

claude Bot Oct 28, 2026

Uh oh!

claude Bot Nov 1, 2026

Uh oh!

This comment has been minimized.

claude Bot Oct 28, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

This comment has been minimized.

codecov-commenter commented Oct 29, 2026 •

edited

Loading

Uh oh!

claude Bot commented Nov 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,554 @@
		# kind: stress (timing and optimization parity fixture, not conformance; see conformance-stress/README.md)
		# StressAthabaskanPrefixComplex: dense-exclusion stress fixture, generated by gen_dense_exclusions.py. The expected parses

Conversation

johnml1135 commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

claude Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

This comment has been minimized.

codecov-commenter commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

claude Bot commented Nov 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

johnml1135 commented Oct 28, 2026 •

edited

Loading

codecov-commenter commented Oct 29, 2026 •

edited

Loading