Skip to content

feat: run NeMo's grammars by default when fst-engine is built in (TN + ITN) - #116

Merged
Alex-Wengg merged 3 commits into
mainfrom
feat/nemo-default-engine
Nov 14, 2026
Merged

Alex-Wengg merged 3 commits into
mainfrom
feat/nemo-default-engine

Conversation

@Alex-Wengg

@Alex-Wengg Alex-Wengg commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

With the fst-engine feature built in, every normalize* and tn_normalize* entry point now runs NeMo's own compiled grammars, so the output is NeMo's instead of an approximation of it. The Swift XCFramework already builds with fst-engine, and this PR adds it to the npm builds.

What changes

  • ITN through the FST engine. NeMo's InverseNormalizer grammars for en, de, es, fr, hi, ja and zh are exported and run by the same driver as TN. Byte-exact against NeMo's own output on NeMo's ITN test inputs:

    en de es fr hi ja zh
    1217/1217 288/288 381/381 307/307 352/352 230/230 390/390

    For zh, 4 ordinals are left out because NeMo's own output leaks tag syntax there (第1億" } } tokens { …). TN stays at 100% in all 8 languages.

  • Punctuated ASR text. NeMo's ITN expects unpunctuated input (January 5 , 2024 .), so its output gets the punctuation post-processing NeMo runs after TN. The Moses step is now a full port of sacremoses' detokenizer. Where a written form's period meets the sentence's own, one is kept (09:00 a.m.).

  • input_case. ITN uses NeMo's default, lower_cased. On 599 earnings22 transcripts, cased wrote no as NO and an opening First, as 1st,.

  • Routing. Custom rules take precedence. A rule for the whole input returns its written form; elsewhere each match is shielded from the grammars and written back verbatim. The rule-based taggers still run in three cases:

    • a build without the feature;
    • the ITN options NeMo has no equivalent of (concat_compound_numbers, disable_bare_second, small_numbers_as_words);
    • after set_rule_based_engine(true), which is nemo_set_rule_based_engine in C and setRuleBasedEngine in Swift and WASM.
  • Re-exporting. scripts/export_nemo_grammars.py and scripts/generate_nemo_fixtures.py regenerate the grammars and goldens from a NeMo checkout at the pinned commit. The export reproduces the bundled TN grammars' FST bytes exactly.

Behaviour changes for XCFramework and npm users. This is NeMo's output:

  • no one knows and three to five people stay words; the taggers wrote no 1 and 04:57.
  • A lone one stays one.
  • 2:05 reads two o five.
  • 200 km/h reads two hundred kilometers per H, which is NeMo's own reading.

Cost

  • Speed: 1–25 ms per sentence, against microseconds.
  • Size: the npm WASM grows from 1.1 to 12.4 MB (0.35 to 9.2 MB gzipped).

Tests

  • New suites:
    • tests/fst_itn_parity.rs holds the goldens.
    • tests/default_engine.rs covers the routing, the option and custom-rule paths, and the switch.
    • Both run in the FST CI job.
  • Rule-based test files are gated off under fst-engine. cargo test with default features, fst-engine, ffi and ffi,fst-engine all pass locally.
  • The Swift smoke test passes 147/147 against an ffi,fst-engine build, with the three NeMo readings above updated. Both WASM smoke tests pass against an fst-engine node build.
  • fmt is clean, and clippy reports the same 220 warnings as main.

🤖 Generated with Claude Code

Export NeMo's InverseNormalizer grammars (input_case=cased) and run them
through the FST driver. NeMo expects unpunctuated ITN input, so the output
gets the same punctuation post-processing NeMo applies after TN; the Moses
step is now a full port of sacremoses' detokenizer instead of the one rule
TN needed. A token NeMo can't verbalize leaves the input unchanged, as in
NeMo. Goldens are NeMo's own output (scripts/generate_nemo_fixtures.py),
minus four zh ordinals where NeMo leaks tag syntax.
With the fst-engine feature, every normalize* and tn_normalize* entry
point now runs NeMo's compiled grammars, so its output is NeMo's: TN in
en, de, es, fr, hi, it, ja, zh and ITN in en, de, es, fr, hi, ja, zh.

- ITN uses NeMo's default input_case="lower_cased". On 599 earnings22
  transcripts, "cased" wrote "no" as "NO" and an opening "First," as
  "1st,".
- Custom rules are written into the input before the grammars run.
- The rule-based taggers still run for builds without the feature, for
  the ITN options NeMo has no equivalent of, and after
  set_rule_based_engine(true) (also nemo_set_rule_based_engine in C,
  setRuleBasedEngine in Swift and WASM).
- One period is kept where a written form ending in "." meets the
  sentence's own ("09:00 a.m.").
- The npm builds include fst-engine. Rule-based test files are gated off
  under the feature, and tests/default_engine.rs covers the routing.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment thread src/lib.rs Outdated
"de" | "es" | "fr" | "hi" | "ja" | "zh" => lang,
_ => "en",
};
fst::inverse_normalize_lang(&custom_rules::substitute(input), lang)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Custom written forms get rewritten again

When a custom rule writes two dollars, nemo_itn feeds it to NeMo, which changes the replacement to $2. Callers lose the written form they registered.

Learn more

Custom rules map a spoken phrase to an exact caller-supplied written form. parse returns that value directly on the rule-based path, while the FST path applies substitution before classifying the entire substituted string. An FST can therefore reinterpret a replacement as another spoken expression, even when the match covers the whole input. This affects Rust, C, Swift, and WASM entry points when fst-engine is built in.

Example: Register add_rule("five dollars", "two dollars"), then call normalize("five dollars"). The rule specifies two dollars, but NeMo reads the substituted phrase and returns $2.

Recommended fix: Preserve matched replacement spans outside the ITN classification step and splice the caller's written forms into the final output. For a whole-input match, return the exact value from custom_rules::parse before invoking the FST.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/custom_rules.rs Outdated
Comment on lines +110 to +115
let first = tokens[i];
let last = tokens[i + spoken.len() - 1];
let lead = &first[..first.len() - first.trim_start_matches(|c: char| !is_word(c)).len()];
let trail = &last[last.trim_end_matches(|c: char| !is_word(c)).len()..];
out.push(format!("{lead}{written}{trail}"));
changed = true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Custom matches delete internal punctuation

With gee pee tee registered, substitute also matches gee, pee tee and drops the comma. Only punctuation outside the matched token range survives.

Learn more

The custom-rule matcher removes leading and trailing punctuation from every whitespace-separated token before comparing words. It then retains punctuation only from the first token's beginning and the last token's end. Any punctuation attached to an interior boundary disappears, even when the rule's spoken phrase never included that punctuation. nemo_itn passes this altered input to the grammar, making the loss visible in the default ITN APIs.

Example: Register add_rule("gee pee tee", "GPT") and normalize I said gee, pee tee yesterday. The three stripped cores match the rule, and the resulting sentence loses the comma after gee.

Recommended fix: Require the original punctuation and separators within a multiword candidate to match the registered phrase, or retain all intervening punctuation when substituting. Add a regression test for punctuation on an interior word.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/custom_rules.rs Outdated
Comment on lines +90 to +93
let cores: Vec<String> = tokens
.iter()
.map(|t| t.trim_matches(|c: char| !is_word(c)).to_lowercase())
.collect();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Punctuated custom rules stop matching

Registering dr. as a custom spoken form cannot match dr. through substitute: its core becomes dr. The FST path ignores the registered override.

Learn more

Custom spoken forms are stored with their punctuation intact by add_rule. The new matcher strips leading and trailing punctuation from input tokens but compares those stripped cores to the unmodified rule tokens. Consequently a spoken rule token with boundary punctuation cannot equal its corresponding input token's core. The rule-based path's parse did not have this mismatch.

Example: Register add_rule("dr.", "Doctor") and call normalize("dr.") in an fst-engine build. The registered dr. never equals the input core dr, so the custom override does not run.

Recommended fix: Apply compatible token normalization to both the registered spoken form and the input while preserving rules that intentionally distinguish punctuation. Test exact punctuated rules through the public FST-backed API.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Review fixes for the custom-rule pre-pass:
- A rule for the whole input returns its written form, as on the
  rule-based path.
- Elsewhere each match becomes a private-use placeholder the grammars
  pass through, and the written form is put back afterwards, so NeMo
  can't re-read it ("two dollars" no longer becomes "$2").
- Words match exactly, so punctuation inside a phrase ("gee, pee tee")
  is no match and a spoken form with its own punctuation ("dr.") matches
  as registered; only punctuation around the phrase may differ.
@Alex-Wengg
Alex-Wengg merged commit 5ec9959 into main Nov 14, 2026
8 checks passed
@Alex-Wengg
Alex-Wengg deleted the feat/nemo-default-engine branch November 14, 2026 01:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant