Repository navigation
feat: run NeMo's grammars by default when fst-engine is built in (TN + ITN) - #116
Conversation
Export NeMo's InverseNormalizer grammars (input_case=cased) and run them through the FST driver. NeMo expects unpunctuated ITN input, so the output gets the same punctuation post-processing NeMo applies after TN; the Moses step is now a full port of sacremoses' detokenizer instead of the one rule TN needed. A token NeMo can't verbalize leaves the input unchanged, as in NeMo. Goldens are NeMo's own output (scripts/generate_nemo_fixtures.py), minus four zh ordinals where NeMo leaks tag syntax.
With the fst-engine feature, every normalize* and tn_normalize* entry
point now runs NeMo's compiled grammars, so its output is NeMo's: TN in
en, de, es, fr, hi, it, ja, zh and ITN in en, de, es, fr, hi, ja, zh.
- ITN uses NeMo's default input_case="lower_cased". On 599 earnings22
transcripts, "cased" wrote "no" as "NO" and an opening "First," as
"1st,".
- Custom rules are written into the input before the grammars run.
- The rule-based taggers still run for builds without the feature, for
the ITN options NeMo has no equivalent of, and after
set_rule_based_engine(true) (also nemo_set_rule_based_engine in C,
setRuleBasedEngine in Swift and WASM).
- One period is kept where a written form ending in "." meets the
sentence's own ("09:00 a.m.").
- The npm builds include fst-engine. Rule-based test files are gated off
under the feature, and tests/default_engine.rs covers the routing.
There was a problem hiding this comment.
Devin Review found 3 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| "de" | "es" | "fr" | "hi" | "ja" | "zh" => lang, | ||
| _ => "en", | ||
| }; | ||
| fst::inverse_normalize_lang(&custom_rules::substitute(input), lang) |
There was a problem hiding this comment.
🟡 Custom written forms get rewritten again
When a custom rule writes two dollars, nemo_itn feeds it to NeMo, which changes the replacement to $2. Callers lose the written form they registered.
Learn more
Custom rules map a spoken phrase to an exact caller-supplied written form. parse returns that value directly on the rule-based path, while the FST path applies substitution before classifying the entire substituted string. An FST can therefore reinterpret a replacement as another spoken expression, even when the match covers the whole input. This affects Rust, C, Swift, and WASM entry points when fst-engine is built in.
Example: Register add_rule("five dollars", "two dollars"), then call normalize("five dollars"). The rule specifies two dollars, but NeMo reads the substituted phrase and returns $2.
Recommended fix: Preserve matched replacement spans outside the ITN classification step and splice the caller's written forms into the final output. For a whole-input match, return the exact value from custom_rules::parse before invoking the FST.
Was this helpful? React with 👍 or 👎 to provide feedback.
| let first = tokens[i]; | ||
| let last = tokens[i + spoken.len() - 1]; | ||
| let lead = &first[..first.len() - first.trim_start_matches(|c: char| !is_word(c)).len()]; | ||
| let trail = &last[last.trim_end_matches(|c: char| !is_word(c)).len()..]; | ||
| out.push(format!("{lead}{written}{trail}")); | ||
| changed = true; |
There was a problem hiding this comment.
🟡 Custom matches delete internal punctuation
With gee pee tee registered, substitute also matches gee, pee tee and drops the comma. Only punctuation outside the matched token range survives.
Learn more
The custom-rule matcher removes leading and trailing punctuation from every whitespace-separated token before comparing words. It then retains punctuation only from the first token's beginning and the last token's end. Any punctuation attached to an interior boundary disappears, even when the rule's spoken phrase never included that punctuation. nemo_itn passes this altered input to the grammar, making the loss visible in the default ITN APIs.
Example: Register add_rule("gee pee tee", "GPT") and normalize I said gee, pee tee yesterday. The three stripped cores match the rule, and the resulting sentence loses the comma after gee.
Recommended fix: Require the original punctuation and separators within a multiword candidate to match the registered phrase, or retain all intervening punctuation when substituting. Add a regression test for punctuation on an interior word.
Was this helpful? React with 👍 or 👎 to provide feedback.
| let cores: Vec<String> = tokens | ||
| .iter() | ||
| .map(|t| t.trim_matches(|c: char| !is_word(c)).to_lowercase()) | ||
| .collect(); |
There was a problem hiding this comment.
🟡 Punctuated custom rules stop matching
Registering dr. as a custom spoken form cannot match dr. through substitute: its core becomes dr. The FST path ignores the registered override.
Learn more
Custom spoken forms are stored with their punctuation intact by add_rule. The new matcher strips leading and trailing punctuation from input tokens but compares those stripped cores to the unmodified rule tokens. Consequently a spoken rule token with boundary punctuation cannot equal its corresponding input token's core. The rule-based path's parse did not have this mismatch.
Example: Register add_rule("dr.", "Doctor") and call normalize("dr.") in an fst-engine build. The registered dr. never equals the input core dr, so the custom override does not run.
Recommended fix: Apply compatible token normalization to both the registered spoken form and the input while preserving rules that intentionally distinguish punctuation. Test exact punctuated rules through the public FST-backed API.
Was this helpful? React with 👍 or 👎 to provide feedback.
Review fixes for the custom-rule pre-pass:
- A rule for the whole input returns its written form, as on the
rule-based path.
- Elsewhere each match becomes a private-use placeholder the grammars
pass through, and the written form is put back afterwards, so NeMo
can't re-read it ("two dollars" no longer becomes "$2").
- Words match exactly, so punctuation inside a phrase ("gee, pee tee")
is no match and a spoken form with its own punctuation ("dr.") matches
as registered; only punctuation around the phrase may differ.
With the
fst-enginefeature built in, everynormalize*andtn_normalize*entry point now runs NeMo's own compiled grammars, so the output is NeMo's instead of an approximation of it. The Swift XCFramework already builds withfst-engine, and this PR adds it to the npm builds.What changes
ITN through the FST engine. NeMo's
InverseNormalizergrammars for en, de, es, fr, hi, ja and zh are exported and run by the same driver as TN. Byte-exact against NeMo's own output on NeMo's ITN test inputs:For zh, 4 ordinals are left out because NeMo's own output leaks tag syntax there (
第1億" } } tokens { …). TN stays at 100% in all 8 languages.Punctuated ASR text. NeMo's ITN expects unpunctuated input (
January 5 , 2024 .), so its output gets the punctuation post-processing NeMo runs after TN. The Moses step is now a full port of sacremoses' detokenizer. Where a written form's period meets the sentence's own, one is kept (09:00 a.m.).input_case. ITN uses NeMo's default,
lower_cased. On 599 earnings22 transcripts,casedwrotenoasNOand an openingFirst,as1st,.Routing. Custom rules take precedence. A rule for the whole input returns its written form; elsewhere each match is shielded from the grammars and written back verbatim. The rule-based taggers still run in three cases:
concat_compound_numbers,disable_bare_second,small_numbers_as_words);set_rule_based_engine(true), which isnemo_set_rule_based_enginein C andsetRuleBasedEnginein Swift and WASM.Re-exporting.
scripts/export_nemo_grammars.pyandscripts/generate_nemo_fixtures.pyregenerate the grammars and goldens from a NeMo checkout at the pinned commit. The export reproduces the bundled TN grammars' FST bytes exactly.Behaviour changes for XCFramework and npm users. This is NeMo's output:
no one knowsandthree to five peoplestay words; the taggers wroteno 1and04:57.onestaysone.2:05readstwo o five.200 km/hreadstwo hundred kilometers per H, which is NeMo's own reading.Cost
Tests
tests/fst_itn_parity.rsholds the goldens.tests/default_engine.rscovers the routing, the option and custom-rule paths, and the switch.fst-engine.cargo testwith default features,fst-engine,ffiandffi,fst-engineall pass locally.ffi,fst-enginebuild, with the three NeMo readings above updated. Both WASM smoke tests pass against anfst-enginenode build.🤖 Generated with Claude Code