Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ jobs:
- run: npm run wasm:ci

fst-engine:
name: FST Engine Parity (8 langs vs NeMo)
name: FST Engine Parity (TN 8 langs, ITN 7 langs vs NeMo)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand All @@ -95,3 +95,7 @@ jobs:
# Source-aligned TN over the same fixtures, plus its C FFI surface.
- name: Run FST-engine alignment tests
run: cargo test --release --features ffi,fst-engine --test fst_alignment
- name: Run FST-engine ITN parity tests
run: cargo test --release --features fst-engine --test fst_itn_parity
- name: Run default-engine routing tests
run: cargo test --release --features ffi,fst-engine --test default_engine --lib
34 changes: 29 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,28 @@ Converts written-form text to spoken form (useful for TTS preprocessing):
| 1st | first |
| 200 km/h | two hundred kilometers per hour |

## Engines

With the `fst-engine` feature, every `normalize*` and `tn_normalize*` entry point
runs NeMo's own compiled grammars, so its output is NeMo's:

- **TN** matches `Normalizer(lang, deterministic=True)` byte for byte in en, de,
es, fr, hi, it, ja and zh.
- **ITN** matches `InverseNormalizer(lang)` (NeMo's default, `input_case="lower_cased"`)
byte for byte in en, de, es, fr, hi, ja and zh. NeMo's ITN expects unpunctuated
input, so punctuated ASR text then gets the punctuation post-processing NeMo
applies after TN (`January 5 , 2024 .` → `January 5, 2024.`).
- Custom rules apply ahead of NeMo's grammars.

The Swift XCFramework and the npm package are built with `fst-engine`. A crate
build without it, the ITN options NeMo doesn't have (`concat_compound_numbers`,
`disable_bare_second`, `small_numbers_as_words`) and
`set_rule_based_engine(true)` (Swift `setRuleBasedEngine`, WASM
`setRuleBasedEngine`, C `nemo_set_rule_based_engine`) use the rule-based taggers
instead. The grammars add about 12 MB to the binary (the npm WASM grows from 1.1
to 12.4 MB) and take 1–25 ms per sentence, where the taggers take microseconds. `scripts/export_nemo_grammars.py`
re-exports them from a NeMo checkout; see [docs/NEMO_PARITY.md](docs/NEMO_PARITY.md).

## Usage

### Rust
Expand Down Expand Up @@ -73,12 +95,11 @@ wasm.addRule("gee pee tee", "GPT");
console.log(wasm.normalize("gee pee tee")); // "GPT"
```

Source-aligned compiled-FST TN (`tnFstNormalizeAligned`) needs the
`fst-engine` feature, which the npm package leaves out. Build it into
`pkg-node-fst/` with `npm run wasm:build:node:fst`:
The packages include the `fst-engine` grammars, so source-aligned
compiled-FST TN (`tnFstNormalizeAligned`) is available too:

```javascript
import * as wasm from "./pkg-node-fst/text_processing_rs.js";
import * as wasm from "./pkg-node/text_processing_rs.js";

const { normalized, spans } = wasm.tnFstNormalizeAligned("The price is $1,234.56.", "en");
// spans[3]: { inputStart: 13, inputEnd: 22, original: "$1,234.56",
Expand Down Expand Up @@ -230,6 +251,9 @@ echo "2:30 PM" | nemo-tn # → two thirty p m

## Compatibility

With `fst-engine`, TN and ITN are byte-exact with NeMo (see [Engines](#engines)).
The tables below describe the rule-based taggers.

### ITN (Spoken → Written)

**98.6% compatible** with NeMo text processing test suite (1200/1217 tests passing).
Expand Down Expand Up @@ -292,7 +316,7 @@ cargo test
### WASM + JavaScript

```bash
# Build + smoke test (Node, with and without fst-engine) + build browser artifact
# Build + smoke test (Node) + build browser artifact, both with fst-engine
npm run wasm:ci

# Create a tarball from the browser package
Expand Down
47 changes: 45 additions & 2 deletions docs/NEMO_PARITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,11 @@ TL;DR:
- The optional **`fst-engine`** feature runs NeMo's *actual* compiled grammars
via [`rustfst`](https://crates.io/crates/rustfst) and reaches **byte-exact
NeMo parity for all eight languages** (zh, ja, fr, es, de, en, hi, it — 100% each
vs NeMo's own deterministic output). Cost: the `rustfst`/`flate2` deps and
~8 MB of gzipped grammars. See "The FST route, shipped" below.
vs NeMo's own deterministic output), and for ITN in seven (en, de, es, fr, hi,
ja, zh). With the feature built in, the default `normalize*` / `tn_normalize*`
entry points run these grammars (see "Default engine" below). Cost: the
`rustfst`/`flate2` deps and ~12 MB of gzipped grammars. See "The FST route,
shipped" below.

## How parity is measured

Expand Down Expand Up @@ -186,3 +189,43 @@ whitespace difference, not a grammar difference.
- Parity test: `tests/fst_parity.rs` runs the bundled NeMo goldens
(`tests/fixtures/<lang>/`) and asserts 100% per language; the `fst-engine` CI
job guards all eight without a NeMo checkout.

## ITN through the FST engine

NeMo's `InverseNormalizer` grammars are exported the same way
(`scripts/export_nemo_grammars.py --mode itn`, `input_case="lower_cased"`, NeMo's
default) into `grammars/<lang>/itn_{classify,verbalize}.fst.gz` and run by the same
driver, minus the post-processing FST. NeMo's ITN expects unpunctuated input and
emits each mark as its own token (`January 5 , 2024 .`), so the output then gets
the punctuation post-processing NeMo runs after TN: a full port of sacremoses'
Moses detokenizer, then `post_process_punct` against the input. One period is
kept where a written form ending in `.` meets the sentence's own (`a.m..`). A
token NeMo can't verbalize leaves the whole input unchanged, as in NeMo.

| lang | parity vs NeMo |
|------|----------------|
| en | **1217/1217** |
| de | **288/288** |
| es | **381/381** |
| fr | **307/307** |
| hi | **352/352** |
| ja | **230/230** |
| zh | **390/390** (4 ordinals left out: NeMo's own output leaks tag syntax, `第1億" } } tokens { …`) |

The goldens in `tests/fixtures/itn/<lang>/` are NeMo's output on NeMo's ITN test
inputs, regenerated by `scripts/generate_nemo_fixtures.py`, and
`tests/fst_itn_parity.rs` asserts them.

Why `lower_cased`: on 599 earnings22 TDT transcripts (cased, punctuated) the two
modes differ on 26. `cased` writes `no` as `NO` and a sentence-opening `First,` as
`1st,`; `lower_cased` misses a capitalized `Forty percent`.

## Default engine

With `fst-engine` built in (the Swift XCFramework and the npm package), every
`normalize*` and `tn_normalize*` entry point runs the grammars above. Custom
rules are written into the input first. The rule-based taggers still run for a
crate build without the feature, for the ITN options NeMo has no equivalent of
(`concat_compound_numbers`, `disable_bare_second`, `small_numbers_as_words`), and
after `set_rule_based_engine(true)`. Language codes without grammars read as
English, as on the rule-based path.
Binary file added grammars/de/itn_classify.fst.gz
Binary file not shown.
Binary file added grammars/de/itn_verbalize.fst.gz
Binary file not shown.
Binary file added grammars/en/itn_classify.fst.gz
Binary file not shown.
Binary file added grammars/en/itn_verbalize.fst.gz
Binary file not shown.
Binary file added grammars/es/itn_classify.fst.gz
Binary file not shown.
Binary file added grammars/es/itn_verbalize.fst.gz
Binary file not shown.
Binary file added grammars/fr/itn_classify.fst.gz
Binary file not shown.
Binary file added grammars/fr/itn_verbalize.fst.gz
Binary file not shown.
Binary file added grammars/hi/itn_classify.fst.gz
Binary file not shown.
Binary file added grammars/hi/itn_verbalize.fst.gz
Binary file not shown.
Binary file added grammars/ja/itn_classify.fst.gz
Binary file not shown.
Binary file added grammars/ja/itn_verbalize.fst.gz
Binary file not shown.
Binary file added grammars/zh/itn_classify.fst.gz
Binary file not shown.
Binary file added grammars/zh/itn_verbalize.fst.gz
Binary file not shown.
7 changes: 3 additions & 4 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,11 @@
"pkg-web/text_processing_rs_bg.wasm.d.ts"
],
"scripts": {
"wasm:build:node": "wasm-pack build --release --target nodejs -- --features wasm && mkdir -p pkg-node && cp -f pkg/* pkg-node/ && node scripts/set-wasm-package-name.mjs pkg-node",
"wasm:build:web": "wasm-pack build --release --target web -- --features wasm && mkdir -p pkg-web && cp -f pkg/* pkg-web/ && node scripts/set-wasm-package-name.mjs pkg-web",
"wasm:build:node": "wasm-pack build --release --target nodejs -- --features wasm,fst-engine && mkdir -p pkg-node && cp -f pkg/* pkg-node/ && node scripts/set-wasm-package-name.mjs pkg-node",
"wasm:build:web": "wasm-pack build --release --target web -- --features wasm,fst-engine && mkdir -p pkg-web && cp -f pkg/* pkg-web/ && node scripts/set-wasm-package-name.mjs pkg-web",
"wasm:test:node": "node wasm-tests/node-smoke.mjs",
"wasm:build:node:fst": "wasm-pack build --release --target nodejs --out-dir pkg-node-fst -- --features wasm,fst-engine",
"wasm:test:node:fst": "node wasm-tests/node-fst-smoke.mjs",
"wasm:ci": "npm run wasm:build:node && npm run wasm:test:node && npm run wasm:build:node:fst && npm run wasm:test:node:fst && npm run wasm:build:web",
"wasm:ci": "npm run wasm:build:node && npm run wasm:test:node && npm run wasm:test:node:fst && npm run wasm:build:web",
"wasm:pack": "npm pack ./pkg-web",
"wasm:publish": "npm publish ./pkg-web --access public"
}
Expand Down
66 changes: 66 additions & 0 deletions scripts/export_nemo_grammars.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
#!/usr/bin/env python3
"""Export NeMo-text-processing's compiled grammars for the `fst-engine` feature.

Runs NeMo's own grammar builders (pynini) and writes the FSTs the Rust engine
loads, gzipped, into grammars/<lang>/:

TN: classify.fst.gz, verbalize.fst.gz, postprocess.fst.gz (en, hi only)
ITN: itn_classify.fst.gz, itn_verbalize.fst.gz

Needs pynini (2.1.7) and a NeMo-text-processing checkout on PYTHONPATH at the
pinned commit (NEMO_REF in .github/workflows/parity.yml). The FSTs are written
untouched: calling properties() before write() changes the header.

PYTHONPATH=/path/to/NeMo-text-processing python scripts/export_nemo_grammars.py --mode itn --lang en
"""

import argparse
import gzip
import os
import shutil
import tempfile


def write_gz(fst, path):
with tempfile.TemporaryDirectory() as tmp:
raw = os.path.join(tmp, "g.fst")
fst.write(raw)
with open(raw, "rb") as src, open(path, "wb") as dst:
# mtime=0 and no filename keep the output reproducible (gzip -n).
with gzip.GzipFile(filename="", mode="wb", fileobj=dst, compresslevel=6, mtime=0) as gz:
shutil.copyfileobj(src, gz)


def export_tn(lang, out_dir, cache_dir):
from nemo_text_processing.text_normalization.normalize import Normalizer

normalizer = Normalizer(input_case="cased", lang=lang, deterministic=True, cache_dir=cache_dir)
write_gz(normalizer.tagger.fst, os.path.join(out_dir, "classify.fst.gz"))
write_gz(normalizer.verbalizer.fst, os.path.join(out_dir, "verbalize.fst.gz"))
post = getattr(normalizer, "post_processor", None)
if post is not None:
write_gz(post.fst, os.path.join(out_dir, "postprocess.fst.gz"))


def export_itn(lang, out_dir, cache_dir):
from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer

normalizer = InverseNormalizer(input_case="lower_cased", lang=lang, cache_dir=cache_dir)
write_gz(normalizer.tagger.fst, os.path.join(out_dir, "itn_classify.fst.gz"))
write_gz(normalizer.verbalizer.fst, os.path.join(out_dir, "itn_verbalize.fst.gz"))


def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--mode", choices=["tn", "itn"], required=True)
parser.add_argument("--lang", required=True)
parser.add_argument("--out", default=None, help="output directory (default grammars/<lang>)")
parser.add_argument("--cache-dir", default=None, help="NeMo .far cache directory")
args = parser.parse_args()
out_dir = args.out or os.path.join(os.path.dirname(__file__), "..", "grammars", args.lang)
os.makedirs(out_dir, exist_ok=True)
(export_tn if args.mode == "tn" else export_itn)(args.lang, out_dir, args.cache_dir)


if __name__ == "__main__":
main()
71 changes: 71 additions & 0 deletions scripts/generate_nemo_fixtures.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
#!/usr/bin/env python3
"""Regenerate the FST engine's ITN goldens from NeMo's own output.

For every NeMo ITN test file (tests/nemo_text_processing/<lang>/
data_inverse_text_normalization/test_cases_*.txt in the NeMo checkout), runs
InverseNormalizer(lang, input_case="lower_cased") on each spoken input, then NeMo's
own punctuation post-processing (the Moses detokenizer and post_process_punct,
as NeMo's TN applies them), and writes `spoken~NeMo output` to
tests/fixtures/itn/<lang>/. The goldens are NeMo's actual output, not the
upstream expected column, so the engine is held to byte-exact NeMo behaviour.

PYTHONPATH=/path/to/NeMo-text-processing python scripts/generate_nemo_fixtures.py --lang en \
--nemo /path/to/NeMo-text-processing
"""

import argparse
import glob
import os

PINNED = "1f1263579fe57ba7ed783cad3dddee710fcc5064"


def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--lang", required=True)
parser.add_argument("--nemo", required=True, help="NeMo-text-processing checkout")
parser.add_argument("--out", default=None)
parser.add_argument("--cache-dir", default=None)
args = parser.parse_args()

from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer
from nemo_text_processing.text_normalization.data_loader_utils import post_process_punct
from sacremoses import MosesDetokenizer

normalizer = InverseNormalizer(input_case="lower_cased", lang=args.lang, cache_dir=args.cache_dir)
moses = MosesDetokenizer(lang=args.lang)
src = os.path.join(args.nemo, "tests", "nemo_text_processing", args.lang, "data_inverse_text_normalization")
out_dir = args.out or os.path.join(os.path.dirname(__file__), "..", "tests", "fixtures", "itn", args.lang)
os.makedirs(out_dir, exist_ok=True)
for path in sorted(glob.glob(os.path.join(src, "test_cases_*.txt"))):
rows, skipped = [], []
with open(path, encoding="utf-8") as f:
for line in f:
line = line.rstrip("\n")
if not line.strip() or "~" not in line:
continue
spoken = line.split("~", 1)[0]
written = normalizer.inverse_normalize(spoken, verbose=False)
written = moses.detokenize([written], unescape=False)
written = post_process_punct(input=spoken, normalized_text=written)
# NeMo's token parser sometimes leaks raw tag syntax into its output
# (zh ordinals: `第1億" } } tokens { cardinal ...`). That is a NeMo bug,
# not behaviour to reproduce, so such cases are left out.
if "tokens {" in written:
skipped.append(spoken)
continue
rows.append((spoken, written))
with open(os.path.join(out_dir, os.path.basename(path)), "w", encoding="utf-8") as f:
f.write(
f"# NeMo ITN output (lang={args.lang}, input_case=lower_cased, punctuation post-processed),"
" regenerated from NeMo-text-processing\n"
)
f.write(f"# pinned commit {PINNED}. Format: spoken~written.\n")
for spoken in skipped:
f.write(f"# left out, NeMo's output leaks tag syntax: {spoken}\n")
for spoken, written in rows:
f.write(f"{spoken}~{written}\n")


if __name__ == "__main__":
main()
Loading
Loading