diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5b4d26d..8b9cd20 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,7 +79,7 @@ jobs: - run: npm run wasm:ci fst-engine: - name: FST Engine Parity (8 langs vs NeMo) + name: FST Engine Parity (TN 8 langs, ITN 7 langs vs NeMo) runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 @@ -95,3 +95,7 @@ jobs: # Source-aligned TN over the same fixtures, plus its C FFI surface. - name: Run FST-engine alignment tests run: cargo test --release --features ffi,fst-engine --test fst_alignment + - name: Run FST-engine ITN parity tests + run: cargo test --release --features fst-engine --test fst_itn_parity + - name: Run default-engine routing tests + run: cargo test --release --features ffi,fst-engine --test default_engine --lib diff --git a/README.md b/README.md index f55f314..8c154f4 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,28 @@ Converts written-form text to spoken form (useful for TTS preprocessing): | 1st | first | | 200 km/h | two hundred kilometers per hour | +## Engines + +With the `fst-engine` feature, every `normalize*` and `tn_normalize*` entry point +runs NeMo's own compiled grammars, so its output is NeMo's: + +- **TN** matches `Normalizer(lang, deterministic=True)` byte for byte in en, de, + es, fr, hi, it, ja and zh. +- **ITN** matches `InverseNormalizer(lang)` (NeMo's default, `input_case="lower_cased"`) + byte for byte in en, de, es, fr, hi, ja and zh. NeMo's ITN expects unpunctuated + input, so punctuated ASR text then gets the punctuation post-processing NeMo + applies after TN (`January 5 , 2024 .` → `January 5, 2024.`). +- Custom rules apply ahead of NeMo's grammars. + +The Swift XCFramework and the npm package are built with `fst-engine`. A crate +build without it, the ITN options NeMo doesn't have (`concat_compound_numbers`, +`disable_bare_second`, `small_numbers_as_words`) and +`set_rule_based_engine(true)` (Swift `setRuleBasedEngine`, WASM +`setRuleBasedEngine`, C `nemo_set_rule_based_engine`) use the rule-based taggers +instead. The grammars add about 12 MB to the binary (the npm WASM grows from 1.1 +to 12.4 MB) and take 1–25 ms per sentence, where the taggers take microseconds. `scripts/export_nemo_grammars.py` +re-exports them from a NeMo checkout; see [docs/NEMO_PARITY.md](docs/NEMO_PARITY.md). + ## Usage ### Rust @@ -73,12 +95,11 @@ wasm.addRule("gee pee tee", "GPT"); console.log(wasm.normalize("gee pee tee")); // "GPT" ``` -Source-aligned compiled-FST TN (`tnFstNormalizeAligned`) needs the -`fst-engine` feature, which the npm package leaves out. Build it into -`pkg-node-fst/` with `npm run wasm:build:node:fst`: +The packages include the `fst-engine` grammars, so source-aligned +compiled-FST TN (`tnFstNormalizeAligned`) is available too: ```javascript -import * as wasm from "./pkg-node-fst/text_processing_rs.js"; +import * as wasm from "./pkg-node/text_processing_rs.js"; const { normalized, spans } = wasm.tnFstNormalizeAligned("The price is $1,234.56.", "en"); // spans[3]: { inputStart: 13, inputEnd: 22, original: "$1,234.56", @@ -230,6 +251,9 @@ echo "2:30 PM" | nemo-tn # → two thirty p m ## Compatibility +With `fst-engine`, TN and ITN are byte-exact with NeMo (see [Engines](#engines)). +The tables below describe the rule-based taggers. + ### ITN (Spoken → Written) **98.6% compatible** with NeMo text processing test suite (1200/1217 tests passing). @@ -292,7 +316,7 @@ cargo test ### WASM + JavaScript ```bash -# Build + smoke test (Node, with and without fst-engine) + build browser artifact +# Build + smoke test (Node) + build browser artifact, both with fst-engine npm run wasm:ci # Create a tarball from the browser package diff --git a/docs/NEMO_PARITY.md b/docs/NEMO_PARITY.md index bc5ecb0..15ddf70 100644 --- a/docs/NEMO_PARITY.md +++ b/docs/NEMO_PARITY.md @@ -14,8 +14,11 @@ TL;DR: - The optional **`fst-engine`** feature runs NeMo's *actual* compiled grammars via [`rustfst`](https://crates.io/crates/rustfst) and reaches **byte-exact NeMo parity for all eight languages** (zh, ja, fr, es, de, en, hi, it — 100% each - vs NeMo's own deterministic output). Cost: the `rustfst`/`flate2` deps and - ~8 MB of gzipped grammars. See "The FST route, shipped" below. + vs NeMo's own deterministic output), and for ITN in seven (en, de, es, fr, hi, + ja, zh). With the feature built in, the default `normalize*` / `tn_normalize*` + entry points run these grammars (see "Default engine" below). Cost: the + `rustfst`/`flate2` deps and ~12 MB of gzipped grammars. See "The FST route, + shipped" below. ## How parity is measured @@ -186,3 +189,43 @@ whitespace difference, not a grammar difference. - Parity test: `tests/fst_parity.rs` runs the bundled NeMo goldens (`tests/fixtures//`) and asserts 100% per language; the `fst-engine` CI job guards all eight without a NeMo checkout. + +## ITN through the FST engine + +NeMo's `InverseNormalizer` grammars are exported the same way +(`scripts/export_nemo_grammars.py --mode itn`, `input_case="lower_cased"`, NeMo's +default) into `grammars//itn_{classify,verbalize}.fst.gz` and run by the same +driver, minus the post-processing FST. NeMo's ITN expects unpunctuated input and +emits each mark as its own token (`January 5 , 2024 .`), so the output then gets +the punctuation post-processing NeMo runs after TN: a full port of sacremoses' +Moses detokenizer, then `post_process_punct` against the input. One period is +kept where a written form ending in `.` meets the sentence's own (`a.m..`). A +token NeMo can't verbalize leaves the whole input unchanged, as in NeMo. + +| lang | parity vs NeMo | +|------|----------------| +| en | **1217/1217** | +| de | **288/288** | +| es | **381/381** | +| fr | **307/307** | +| hi | **352/352** | +| ja | **230/230** | +| zh | **390/390** (4 ordinals left out: NeMo's own output leaks tag syntax, `第1億" } } tokens { …`) | + +The goldens in `tests/fixtures/itn//` are NeMo's output on NeMo's ITN test +inputs, regenerated by `scripts/generate_nemo_fixtures.py`, and +`tests/fst_itn_parity.rs` asserts them. + +Why `lower_cased`: on 599 earnings22 TDT transcripts (cased, punctuated) the two +modes differ on 26. `cased` writes `no` as `NO` and a sentence-opening `First,` as +`1st,`; `lower_cased` misses a capitalized `Forty percent`. + +## Default engine + +With `fst-engine` built in (the Swift XCFramework and the npm package), every +`normalize*` and `tn_normalize*` entry point runs the grammars above. Custom +rules are written into the input first. The rule-based taggers still run for a +crate build without the feature, for the ITN options NeMo has no equivalent of +(`concat_compound_numbers`, `disable_bare_second`, `small_numbers_as_words`), and +after `set_rule_based_engine(true)`. Language codes without grammars read as +English, as on the rule-based path. diff --git a/grammars/de/itn_classify.fst.gz b/grammars/de/itn_classify.fst.gz new file mode 100644 index 0000000..15da0ce Binary files /dev/null and b/grammars/de/itn_classify.fst.gz differ diff --git a/grammars/de/itn_verbalize.fst.gz b/grammars/de/itn_verbalize.fst.gz new file mode 100644 index 0000000..d9a075a Binary files /dev/null and b/grammars/de/itn_verbalize.fst.gz differ diff --git a/grammars/en/itn_classify.fst.gz b/grammars/en/itn_classify.fst.gz new file mode 100644 index 0000000..d1bdf76 Binary files /dev/null and b/grammars/en/itn_classify.fst.gz differ diff --git a/grammars/en/itn_verbalize.fst.gz b/grammars/en/itn_verbalize.fst.gz new file mode 100644 index 0000000..12f9bb5 Binary files /dev/null and b/grammars/en/itn_verbalize.fst.gz differ diff --git a/grammars/es/itn_classify.fst.gz b/grammars/es/itn_classify.fst.gz new file mode 100644 index 0000000..7d4a674 Binary files /dev/null and b/grammars/es/itn_classify.fst.gz differ diff --git a/grammars/es/itn_verbalize.fst.gz b/grammars/es/itn_verbalize.fst.gz new file mode 100644 index 0000000..ab475c9 Binary files /dev/null and b/grammars/es/itn_verbalize.fst.gz differ diff --git a/grammars/fr/itn_classify.fst.gz b/grammars/fr/itn_classify.fst.gz new file mode 100644 index 0000000..34c82ca Binary files /dev/null and b/grammars/fr/itn_classify.fst.gz differ diff --git a/grammars/fr/itn_verbalize.fst.gz b/grammars/fr/itn_verbalize.fst.gz new file mode 100644 index 0000000..db0da7e Binary files /dev/null and b/grammars/fr/itn_verbalize.fst.gz differ diff --git a/grammars/hi/itn_classify.fst.gz b/grammars/hi/itn_classify.fst.gz new file mode 100644 index 0000000..8aa6787 Binary files /dev/null and b/grammars/hi/itn_classify.fst.gz differ diff --git a/grammars/hi/itn_verbalize.fst.gz b/grammars/hi/itn_verbalize.fst.gz new file mode 100644 index 0000000..972247d Binary files /dev/null and b/grammars/hi/itn_verbalize.fst.gz differ diff --git a/grammars/ja/itn_classify.fst.gz b/grammars/ja/itn_classify.fst.gz new file mode 100644 index 0000000..492000b Binary files /dev/null and b/grammars/ja/itn_classify.fst.gz differ diff --git a/grammars/ja/itn_verbalize.fst.gz b/grammars/ja/itn_verbalize.fst.gz new file mode 100644 index 0000000..47b6047 Binary files /dev/null and b/grammars/ja/itn_verbalize.fst.gz differ diff --git a/grammars/zh/itn_classify.fst.gz b/grammars/zh/itn_classify.fst.gz new file mode 100644 index 0000000..a0a4bd5 Binary files /dev/null and b/grammars/zh/itn_classify.fst.gz differ diff --git a/grammars/zh/itn_verbalize.fst.gz b/grammars/zh/itn_verbalize.fst.gz new file mode 100644 index 0000000..cc6bc3d Binary files /dev/null and b/grammars/zh/itn_verbalize.fst.gz differ diff --git a/package.json b/package.json index 9d06c2c..63f04d8 100644 --- a/package.json +++ b/package.json @@ -12,12 +12,11 @@ "pkg-web/text_processing_rs_bg.wasm.d.ts" ], "scripts": { - "wasm:build:node": "wasm-pack build --release --target nodejs -- --features wasm && mkdir -p pkg-node && cp -f pkg/* pkg-node/ && node scripts/set-wasm-package-name.mjs pkg-node", - "wasm:build:web": "wasm-pack build --release --target web -- --features wasm && mkdir -p pkg-web && cp -f pkg/* pkg-web/ && node scripts/set-wasm-package-name.mjs pkg-web", + "wasm:build:node": "wasm-pack build --release --target nodejs -- --features wasm,fst-engine && mkdir -p pkg-node && cp -f pkg/* pkg-node/ && node scripts/set-wasm-package-name.mjs pkg-node", + "wasm:build:web": "wasm-pack build --release --target web -- --features wasm,fst-engine && mkdir -p pkg-web && cp -f pkg/* pkg-web/ && node scripts/set-wasm-package-name.mjs pkg-web", "wasm:test:node": "node wasm-tests/node-smoke.mjs", - "wasm:build:node:fst": "wasm-pack build --release --target nodejs --out-dir pkg-node-fst -- --features wasm,fst-engine", "wasm:test:node:fst": "node wasm-tests/node-fst-smoke.mjs", - "wasm:ci": "npm run wasm:build:node && npm run wasm:test:node && npm run wasm:build:node:fst && npm run wasm:test:node:fst && npm run wasm:build:web", + "wasm:ci": "npm run wasm:build:node && npm run wasm:test:node && npm run wasm:test:node:fst && npm run wasm:build:web", "wasm:pack": "npm pack ./pkg-web", "wasm:publish": "npm publish ./pkg-web --access public" } diff --git a/scripts/export_nemo_grammars.py b/scripts/export_nemo_grammars.py new file mode 100644 index 0000000..210e1f5 --- /dev/null +++ b/scripts/export_nemo_grammars.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python3 +"""Export NeMo-text-processing's compiled grammars for the `fst-engine` feature. + +Runs NeMo's own grammar builders (pynini) and writes the FSTs the Rust engine +loads, gzipped, into grammars//: + + TN: classify.fst.gz, verbalize.fst.gz, postprocess.fst.gz (en, hi only) + ITN: itn_classify.fst.gz, itn_verbalize.fst.gz + +Needs pynini (2.1.7) and a NeMo-text-processing checkout on PYTHONPATH at the +pinned commit (NEMO_REF in .github/workflows/parity.yml). The FSTs are written +untouched: calling properties() before write() changes the header. + + PYTHONPATH=/path/to/NeMo-text-processing python scripts/export_nemo_grammars.py --mode itn --lang en +""" + +import argparse +import gzip +import os +import shutil +import tempfile + + +def write_gz(fst, path): + with tempfile.TemporaryDirectory() as tmp: + raw = os.path.join(tmp, "g.fst") + fst.write(raw) + with open(raw, "rb") as src, open(path, "wb") as dst: + # mtime=0 and no filename keep the output reproducible (gzip -n). + with gzip.GzipFile(filename="", mode="wb", fileobj=dst, compresslevel=6, mtime=0) as gz: + shutil.copyfileobj(src, gz) + + +def export_tn(lang, out_dir, cache_dir): + from nemo_text_processing.text_normalization.normalize import Normalizer + + normalizer = Normalizer(input_case="cased", lang=lang, deterministic=True, cache_dir=cache_dir) + write_gz(normalizer.tagger.fst, os.path.join(out_dir, "classify.fst.gz")) + write_gz(normalizer.verbalizer.fst, os.path.join(out_dir, "verbalize.fst.gz")) + post = getattr(normalizer, "post_processor", None) + if post is not None: + write_gz(post.fst, os.path.join(out_dir, "postprocess.fst.gz")) + + +def export_itn(lang, out_dir, cache_dir): + from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer + + normalizer = InverseNormalizer(input_case="lower_cased", lang=lang, cache_dir=cache_dir) + write_gz(normalizer.tagger.fst, os.path.join(out_dir, "itn_classify.fst.gz")) + write_gz(normalizer.verbalizer.fst, os.path.join(out_dir, "itn_verbalize.fst.gz")) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--mode", choices=["tn", "itn"], required=True) + parser.add_argument("--lang", required=True) + parser.add_argument("--out", default=None, help="output directory (default grammars/)") + parser.add_argument("--cache-dir", default=None, help="NeMo .far cache directory") + args = parser.parse_args() + out_dir = args.out or os.path.join(os.path.dirname(__file__), "..", "grammars", args.lang) + os.makedirs(out_dir, exist_ok=True) + (export_tn if args.mode == "tn" else export_itn)(args.lang, out_dir, args.cache_dir) + + +if __name__ == "__main__": + main() diff --git a/scripts/generate_nemo_fixtures.py b/scripts/generate_nemo_fixtures.py new file mode 100644 index 0000000..62f4015 --- /dev/null +++ b/scripts/generate_nemo_fixtures.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python3 +"""Regenerate the FST engine's ITN goldens from NeMo's own output. + +For every NeMo ITN test file (tests/nemo_text_processing// +data_inverse_text_normalization/test_cases_*.txt in the NeMo checkout), runs +InverseNormalizer(lang, input_case="lower_cased") on each spoken input, then NeMo's +own punctuation post-processing (the Moses detokenizer and post_process_punct, +as NeMo's TN applies them), and writes `spoken~NeMo output` to +tests/fixtures/itn//. The goldens are NeMo's actual output, not the +upstream expected column, so the engine is held to byte-exact NeMo behaviour. + + PYTHONPATH=/path/to/NeMo-text-processing python scripts/generate_nemo_fixtures.py --lang en \ + --nemo /path/to/NeMo-text-processing +""" + +import argparse +import glob +import os + +PINNED = "1f1263579fe57ba7ed783cad3dddee710fcc5064" + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--lang", required=True) + parser.add_argument("--nemo", required=True, help="NeMo-text-processing checkout") + parser.add_argument("--out", default=None) + parser.add_argument("--cache-dir", default=None) + args = parser.parse_args() + + from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer + from nemo_text_processing.text_normalization.data_loader_utils import post_process_punct + from sacremoses import MosesDetokenizer + + normalizer = InverseNormalizer(input_case="lower_cased", lang=args.lang, cache_dir=args.cache_dir) + moses = MosesDetokenizer(lang=args.lang) + src = os.path.join(args.nemo, "tests", "nemo_text_processing", args.lang, "data_inverse_text_normalization") + out_dir = args.out or os.path.join(os.path.dirname(__file__), "..", "tests", "fixtures", "itn", args.lang) + os.makedirs(out_dir, exist_ok=True) + for path in sorted(glob.glob(os.path.join(src, "test_cases_*.txt"))): + rows, skipped = [], [] + with open(path, encoding="utf-8") as f: + for line in f: + line = line.rstrip("\n") + if not line.strip() or "~" not in line: + continue + spoken = line.split("~", 1)[0] + written = normalizer.inverse_normalize(spoken, verbose=False) + written = moses.detokenize([written], unescape=False) + written = post_process_punct(input=spoken, normalized_text=written) + # NeMo's token parser sometimes leaks raw tag syntax into its output + # (zh ordinals: `第1億" } } tokens { cardinal ...`). That is a NeMo bug, + # not behaviour to reproduce, so such cases are left out. + if "tokens {" in written: + skipped.append(spoken) + continue + rows.append((spoken, written)) + with open(os.path.join(out_dir, os.path.basename(path)), "w", encoding="utf-8") as f: + f.write( + f"# NeMo ITN output (lang={args.lang}, input_case=lower_cased, punctuation post-processed)," + " regenerated from NeMo-text-processing\n" + ) + f.write(f"# pinned commit {PINNED}. Format: spoken~written.\n") + for spoken in skipped: + f.write(f"# left out, NeMo's output leaks tag syntax: {spoken}\n") + for spoken, written in rows: + f.write(f"{spoken}~{written}\n") + + +if __name__ == "__main__": + main() diff --git a/src/custom_rules.rs b/src/custom_rules.rs index 0046b9d..5f37b3f 100644 --- a/src/custom_rules.rs +++ b/src/custom_rules.rs @@ -64,6 +64,120 @@ pub fn parse(input: &str) -> Option { None } +/// First private-use code point of [`protect`]'s placeholders; one per match. +const PLACEHOLDER_BASE: u32 = 0xE000; +const PLACEHOLDER_LAST: u32 = 0xF8FF; + +/// Mark every registered spoken form in `input` for NeMo's grammars: each match +/// becomes a private-use placeholder the grammars pass through, and the written +/// forms come back from [`restore`] afterwards, so NeMo can't re-read them +/// (`"two dollars"` stays `"two dollars"`). Words match case-insensitively and +/// exactly, longest rule first; only punctuation around the whole phrase may differ +/// (`"gee pee tee."` matches, `"gee, pee tee"` doesn't), and it is kept. +pub(crate) fn protect(input: &str) -> (std::borrow::Cow<'_, str>, Vec) { + let rules = CUSTOM_RULES.read().unwrap(); + if rules.is_empty() { + return (std::borrow::Cow::Borrowed(input), Vec::new()); + } + let mut rules: Vec<(Vec<&str>, &str)> = rules + .iter() + .map(|(spoken, written)| { + ( + spoken.split_whitespace().collect::>(), + written.as_str(), + ) + }) + .filter(|(spoken, _)| !spoken.is_empty()) + .collect(); + rules.sort_by_key(|(spoken, _)| std::cmp::Reverse(spoken.len())); + + let tokens: Vec = input.split_whitespace().map(str::to_lowercase).collect(); + let originals: Vec<&str> = input.split_whitespace().collect(); + let mut out: Vec = Vec::with_capacity(tokens.len()); + let mut written_forms = Vec::new(); + let mut i = 0; + while i < tokens.len() { + let matched = rules.iter().find_map(|(spoken, written)| { + let n = spoken.len(); + if i + n > tokens.len() + || PLACEHOLDER_BASE + written_forms.len() as u32 > PLACEHOLDER_LAST + { + return None; + } + let (lead, trail) = phrase_affixes(&tokens[i..i + n], spoken)?; + Some((n, lead, trail, *written)) + }); + let Some((n, lead, trail, written)) = matched else { + out.push(originals[i].to_string()); + i += 1; + continue; + }; + let placeholder = char::from_u32(PLACEHOLDER_BASE + written_forms.len() as u32).unwrap(); + out.push(format!("{lead}{placeholder}{trail}")); + written_forms.push(written.to_string()); + i += n; + } + if written_forms.is_empty() { + (std::borrow::Cow::Borrowed(input), written_forms) + } else { + (std::borrow::Cow::Owned(out.join(" ")), written_forms) + } +} + +/// Put [`protect`]'s written forms back in place of their placeholders. +pub(crate) fn restore(output: &str, written_forms: &[String]) -> String { + if written_forms.is_empty() { + return output.to_string(); + } + let mut restored = String::with_capacity(output.len()); + for c in output.chars() { + let index = (c as u32).wrapping_sub(PLACEHOLDER_BASE) as usize; + match written_forms.get(index) { + Some(written) if (c as u32) >= PLACEHOLDER_BASE => restored.push_str(written), + _ => restored.push(c), + } + } + restored +} + +/// The punctuation before and after `tokens` (lowercased input words) when they +/// spell `spoken`: the words must match exactly, and only the first token may +/// carry extra leading punctuation and the last extra trailing punctuation. +fn phrase_affixes<'a>(tokens: &'a [String], spoken: &[&str]) -> Option<(&'a str, &'a str)> { + let is_mark = |c: char| !c.is_alphanumeric() && c != '\''; + let last = spoken.len() - 1; + let mut lead = ""; + let mut trail = ""; + for (k, (token, word)) in tokens.iter().zip(spoken).enumerate() { + let token = token.as_str(); + // The extra leading marks the first token may have, then the word, then the + // extra trailing marks the last token may have. + let start = if k == 0 { + (0..=token.len()) + .filter(|&at| token.is_char_boundary(at) && token[at..].starts_with(word)) + .find(|&at| token[..at].chars().all(is_mark))? + } else if token.starts_with(word) { + 0 + } else { + return None; + }; + let end = start + word.len(); + let rest = &token[end..]; + if k == last { + if !rest.chars().all(is_mark) { + return None; + } + trail = rest; + } else if !rest.is_empty() { + return None; + } + if k == 0 { + lead = &token[..start]; + } + } + Some((lead, trail)) +} + /// Get the number of custom rules currently registered. pub fn rule_count() -> usize { let rules = CUSTOM_RULES.read().unwrap(); diff --git a/src/ffi.rs b/src/ffi.rs index 94764be..686d1b1 100644 --- a/src/ffi.rs +++ b/src/ffi.rs @@ -371,6 +371,13 @@ pub extern "C" fn nemo_rule_count() -> u32 { guard(0, || custom_rules::rule_count() as u32) } +/// Use the rule-based taggers (`enabled != 0`) even when NeMo's grammars are +/// built in, or go back to NeMo's grammars (`0`, the default). Process-wide. +#[no_mangle] +pub extern "C" fn nemo_set_rule_based_engine(enabled: u32) { + guard((), || crate::set_rule_based_engine(enabled != 0)) +} + /// Get the library version. /// /// # Safety @@ -1022,10 +1029,15 @@ mod tests { unsafe { let input = CString::new("no one knows").unwrap(); let en = CString::new("en").unwrap(); - // Default flag (0) keeps today's behavior. + // Default flag (0): NeMo's grammars keep "no one" too; the taggers read 1. let baseline = nemo_normalize_sentence_lang_with_options(input.as_ptr(), en.as_ptr(), 0, 0, 0, 0); - assert_eq!(CStr::from_ptr(baseline).to_str().unwrap(), "no 1 knows"); + let expected = if cfg!(feature = "fst-engine") { + "no one knows" + } else { + "no 1 knows" + }; + assert_eq!(CStr::from_ptr(baseline).to_str().unwrap(), expected); nemo_free_string(baseline); let opted_in = nemo_normalize_sentence_lang_with_options(input.as_ptr(), en.as_ptr(), 0, 0, 0, 1); diff --git a/src/fst/de.rs b/src/fst/de.rs index a33bda1..f321984 100644 --- a/src/fst/de.rs +++ b/src/fst/de.rs @@ -15,6 +15,10 @@ lazy_static! { load_gz(include_bytes!("../../grammars/de/classify.fst.gz")); static ref VERBALIZE: VectorFst = load_gz(include_bytes!("../../grammars/de/verbalize.fst.gz")); + static ref ITN_CLASSIFY: VectorFst = + load_gz(include_bytes!("../../grammars/de/itn_classify.fst.gz")); + static ref ITN_VERBALIZE: VectorFst = + load_gz(include_bytes!("../../grammars/de/itn_verbalize.fst.gz")); } /// Normalize written-form German (de) to spoken form. @@ -28,10 +32,16 @@ lazy_static! { /// # } /// ``` pub fn normalize(input: &str) -> String { - driver::normalize(&CLASSIFY, &VERBALIZE, None, input, " ") + driver::normalize(&CLASSIFY, &VERBALIZE, None, input, " ", "de") } /// Normalize German text and retain the source span for every output token. pub fn normalize_aligned(input: &str) -> Option { - driver::normalize_aligned(&CLASSIFY, &VERBALIZE, None, input, " ") + driver::normalize_aligned(&CLASSIFY, &VERBALIZE, None, input, " ", "de") +} + +/// Inverse-normalize spoken German (de) to written form, byte-exact with NeMo's +/// `InverseNormalizer(lang="de", input_case="cased")`. +pub fn inverse_normalize(input: &str) -> String { + driver::inverse_normalize(&ITN_CLASSIFY, &ITN_VERBALIZE, input, " ", "de") } diff --git a/src/fst/driver.rs b/src/fst/driver.rs index d725f3b..9ea05b2 100644 --- a/src/fst/driver.rs +++ b/src/fst/driver.rs @@ -227,6 +227,7 @@ pub fn normalize( post: Option<&VectorFst>, input: &str, sep: &str, + lang: &str, ) -> String { let Some(tagged) = apply(classify, input) else { return input.to_string(); @@ -235,10 +236,61 @@ pub fn normalize( let mut parts = Vec::with_capacity(tokens.len()); for (k, v) in tokens { - parts.push(verbalize_token(verbalize, k, v).unwrap_or_default()); + // NeMo returns the input unchanged when a token can't be verbalized. + let Some(part) = verbalize_token(verbalize, k, v) else { + return input.trim().to_string(); + }; + parts.push(part); } - finish_normalization(parts, post, input, sep) + finish_normalization(parts, post, input, sep, lang) +} + +/// Run the ITN pipeline of NeMo's `InverseNormalizer.inverse_normalize`: +/// classify → parse → permute → verbalize → join → collapse spaces. +/// +/// NeMo's ITN expects unpunctuated input and emits each mark as its own token +/// (`January 5 , 2024 .`). ASR transcripts carry punctuation, so the output then +/// gets the same punctuation re-alignment NeMo applies after TN (Moses spacing, +/// then `post_process_punct` against the input), which leaves NeMo's output on +/// unpunctuated input unchanged. Returns the input unchanged if classification +/// fails. +pub fn inverse_normalize( + classify: &VectorFst, + verbalize: &VectorFst, + input: &str, + sep: &str, + lang: &str, +) -> String { + let text = input.trim(); + if text.is_empty() { + return text.to_string(); + } + let Some(tagged) = apply(classify, text) else { + return text.to_string(); + }; + let tokens = TagParser::new(&tagged).fields(); + let mut parts = Vec::with_capacity(tokens.len()); + for (k, v) in tokens { + // NeMo returns the input unchanged when a token can't be verbalized + // (Chinese tags a lone 年 as a date it can't write out). + let Some(part) = verbalize_token(verbalize, k, v) else { + return text.to_string(); + }; + parts.push(part); + } + let output = finish_normalization(parts, None, text, sep, lang); + // A written form ending in a period (`a.m.`, `Mr.`) before the sentence's own + // period reads `09:00 a.m..`; keep one, unless the input had a run of periods. + if text.contains("..") { + output + } else { + let mut collapsed = output; + while collapsed.contains("..") { + collapsed = collapsed.replace("..", "."); + } + collapsed + } } /// Run TN while retaining the input span consumed by every top-level NeMo @@ -249,6 +301,7 @@ pub(super) fn normalize_aligned( post: Option<&VectorFst>, input: &str, sep: &str, + lang: &str, ) -> Option { let classified = apply_with_alignment(classify, input)?; let tokens = TagParser::new(&classified.output).aligned_fields(); @@ -272,7 +325,13 @@ pub(super) fn normalize_aligned( let original = input[input_start..input_end].to_string(); let kind = token_kind(&value, &original); - let verbalized = verbalize_token(verbalize, key, value).unwrap_or_default(); + let verbalized = match verbalize_token(verbalize, key, value) { + Some(verbalized) => verbalized, + // English and French classify "" as an empty date that says nothing. + None if original.is_empty() => String::new(), + // Otherwise NeMo returns the input unchanged. + None => return None, + }; let normalized = collapse_spaces(verbalized.clone()); parts.push(verbalized); // English classifies empty input as an empty date token. A token that @@ -290,7 +349,7 @@ pub(super) fn normalize_aligned( } Some(AlignedNormalization { - normalized: finish_normalization(parts, post, input, sep), + normalized: finish_normalization(parts, post, input, sep, lang), spans, }) } @@ -321,6 +380,7 @@ fn finish_normalization( post: Option<&VectorFst>, input: &str, sep: &str, + lang: &str, ) -> String { // NeMo's post-verbalization steps (normalize.py): collapse spaces, apply the // post-processing FST, Moses-detokenize, then re-align punctuation spacing to @@ -330,7 +390,7 @@ fn finish_normalization( Some(pp) => apply(pp, &joined).unwrap_or(joined), None => joined, }; - post_process_punct(input, &moses_despace(&processed)) + post_process_punct(input, &moses_detokenize(&processed, lang)) } fn trim_source_span(input: &str, start: usize, end: usize) -> (usize, usize) { @@ -396,15 +456,144 @@ fn collapse_spaces(s: String) -> String { out.trim().to_string() } -/// The one Moses-detokenizer behavior NeMo's outputs depend on: drop a space -/// before sentence/clause punctuation. `post_process_punct` then restores any -/// spacing the original input actually had. -fn moses_despace(s: &str) -> String { - let mut out = s.to_string(); - for p in [" .", " ,", " !", " ?", " ;", " :"] { - out = out.replace(p, &p[1..]); +/// Port of sacremoses' `MosesDetokenizer(lang).detokenize([s], unescape=False)`, +/// which NeMo runs on the joined output before `post_process_punct`. Rules for +/// languages the engine doesn't load (Czech, Finnish, Korean) are left out. +fn moses_detokenize(s: &str, lang: &str) -> String { + // Undo the aggressive hyphen split, then work on whitespace tokens. + let padded = format!(" {} ", s).replace(" @-@ ", "-"); + let tokens: Vec<&str> = padded.split_whitespace().collect(); + let mut out = String::with_capacity(s.len()); + let mut prepend_space = " "; + let mut quote_counts: Vec<(String, usize)> = Vec::new(); + for (i, &token) in tokens.iter().enumerate() { + let first = token.chars().next().unwrap_or(' '); + if is_cjk(first) { + // A CJK word right after another CJK word joins it. + if i > 0 && tokens[i - 1].chars().last().is_some_and(is_cjk) { + out.push_str(token); + } else { + out.push_str(prepend_space); + out.push_str(token); + } + prepend_space = " "; + } else if token.chars().all(is_moses_currency_or_open) { + out.push_str(prepend_space); + out.push_str(token); + prepend_space = ""; + } else if token.chars().all(|c| ",.?!:;\\%}])".contains(c)) { + // French puts a (non-breaking) space before these marks. + if lang == "fr" && token.chars().count() == 1 && "?!:;\\%".contains(token) { + out.push(' '); + } + out.push_str(token); + prepend_space = " "; + } else if lang == "en" && i > 0 && is_english_contraction(token) { + out.push_str(token); + prepend_space = " "; + } else if (lang == "fr" || lang == "it") + && i + 1 < tokens.len() + && is_french_contraction(token) + && tokens[i + 1] + .chars() + .next() + .is_some_and(char::is_alphabetic) + { + out.push_str(prepend_space); + out.push_str(token); + prepend_space = ""; + } else if !token.is_empty() && token.chars().all(|c| "'\"„“`".contains(c)) { + let normalized = if token.chars().all(|c| "„“”".contains(c)) { + "\"".to_string() + } else { + token.to_string() + }; + let count = match quote_counts.iter_mut().find(|(q, _)| *q == normalized) { + Some((_, n)) => n, + None => { + quote_counts.push((normalized, 0)); + &mut quote_counts.last_mut().unwrap().1 + } + }; + if *count % 2 == 0 { + if lang == "en" && token == "'" && i > 0 && tokens[i - 1].ends_with('s') { + // Possessive after "s": "The Jones' house". + out.push_str(token); + prepend_space = " "; + } else { + out.push_str(prepend_space); + out.push_str(token); + prepend_space = ""; + *count += 1; + } + } else { + out.push_str(token); + prepend_space = " "; + *count += 1; + } + } else { + out.push_str(prepend_space); + out.push_str(token); + prepend_space = " "; + } + } + let mut collapsed = String::with_capacity(out.len()); + let mut prev_space = false; + for c in out.chars() { + if c == ' ' { + if !prev_space { + collapsed.push(' '); + } + prev_space = true; + } else { + collapsed.push(c); + prev_space = false; + } + } + collapsed.trim().to_string() +} + +/// sacremoses' `is_cjk`, including its quirk that a range's first code point +/// doesn't count. +fn is_cjk(c: char) -> bool { + const RANGES: [(u32, u32); 11] = [ + (4352, 4607), + (11904, 42191), + (43072, 43135), + (44032, 55215), + (63744, 64255), + (65072, 65103), + (65381, 65500), + (94208, 101119), + (110592, 110895), + (110960, 111359), + (131072, 196607), + ]; + let code = c as u32; + for (start, end) in RANGES { + if code < end { + return code > start; + } } - out + false +} + +/// Moses' `IS_CURRENCY_SYMBOL` class: Unicode currency symbols (Perl `IsSc`) +/// and the opening marks `( [ { ¿ ¡`. +fn is_moses_currency_or_open(c: char) -> bool { + "$¢£¤¥֏؋৲৳৻૱௹฿៛₠₡₢₣₤₥₦₧₨₩₪₫€₭₮₯₰₱₲₳₴₵₶₷₸₹₺₻₼₽꠸﷼﹩$¢£¥₩([{¿¡".contains(c) +} + +/// Moses' `IS_ENGLISH_CONTRACTION`: an apostrophe and then a letter (`'s`). +fn is_english_contraction(token: &str) -> bool { + let mut chars = token.chars(); + chars.next() == Some('\'') && chars.next().is_some_and(char::is_alphabetic) +} + +/// Moses' `IS_FRENCH_CONRTACTION`: a letter and then a final apostrophe (`l'`). +fn is_french_contraction(token: &str) -> bool { + let chars: Vec = token.chars().collect(); + chars.len() >= 2 && chars[chars.len() - 1] == '\'' && chars[chars.len() - 2].is_alphabetic() } /// Port of NeMo's `data_loader_utils.post_process_punct`: for each punctuation diff --git a/src/fst/en.rs b/src/fst/en.rs index b158aec..78ce7ff 100644 --- a/src/fst/en.rs +++ b/src/fst/en.rs @@ -17,6 +17,10 @@ lazy_static! { load_gz(include_bytes!("../../grammars/en/verbalize.fst.gz")); static ref POST: VectorFst = load_gz(include_bytes!("../../grammars/en/postprocess.fst.gz")); + static ref ITN_CLASSIFY: VectorFst = + load_gz(include_bytes!("../../grammars/en/itn_classify.fst.gz")); + static ref ITN_VERBALIZE: VectorFst = + load_gz(include_bytes!("../../grammars/en/itn_verbalize.fst.gz")); } /// Normalize written-form English (en) to spoken form. @@ -30,10 +34,24 @@ lazy_static! { /// # } /// ``` pub fn normalize(input: &str) -> String { - driver::normalize(&CLASSIFY, &VERBALIZE, Some(&POST), input, " ") + driver::normalize(&CLASSIFY, &VERBALIZE, Some(&POST), input, " ", "en") } /// Normalize English text and retain the source span for every output token. pub fn normalize_aligned(input: &str) -> Option { - driver::normalize_aligned(&CLASSIFY, &VERBALIZE, Some(&POST), input, " ") + driver::normalize_aligned(&CLASSIFY, &VERBALIZE, Some(&POST), input, " ", "en") +} + +/// Inverse-normalize spoken English (en) to written form, byte-exact with NeMo's +/// `InverseNormalizer(lang="en", input_case="cased")`. +/// +/// ``` +/// # #[cfg(feature = "fst-engine")] +/// # { +/// use text_processing_rs::fst::en; +/// assert_eq!(en::inverse_normalize("two dollars"), "$2"); +/// # } +/// ``` +pub fn inverse_normalize(input: &str) -> String { + driver::inverse_normalize(&ITN_CLASSIFY, &ITN_VERBALIZE, input, " ", "en") } diff --git a/src/fst/es.rs b/src/fst/es.rs index 84a6191..3d16408 100644 --- a/src/fst/es.rs +++ b/src/fst/es.rs @@ -15,6 +15,10 @@ lazy_static! { load_gz(include_bytes!("../../grammars/es/classify.fst.gz")); static ref VERBALIZE: VectorFst = load_gz(include_bytes!("../../grammars/es/verbalize.fst.gz")); + static ref ITN_CLASSIFY: VectorFst = + load_gz(include_bytes!("../../grammars/es/itn_classify.fst.gz")); + static ref ITN_VERBALIZE: VectorFst = + load_gz(include_bytes!("../../grammars/es/itn_verbalize.fst.gz")); } /// Normalize written-form Spanish (es) to spoken form. @@ -28,10 +32,16 @@ lazy_static! { /// # } /// ``` pub fn normalize(input: &str) -> String { - driver::normalize(&CLASSIFY, &VERBALIZE, None, input, " ") + driver::normalize(&CLASSIFY, &VERBALIZE, None, input, " ", "es") } /// Normalize Spanish text and retain the source span for every output token. pub fn normalize_aligned(input: &str) -> Option { - driver::normalize_aligned(&CLASSIFY, &VERBALIZE, None, input, " ") + driver::normalize_aligned(&CLASSIFY, &VERBALIZE, None, input, " ", "es") +} + +/// Inverse-normalize spoken Spanish (es) to written form, byte-exact with NeMo's +/// `InverseNormalizer(lang="es", input_case="cased")`. +pub fn inverse_normalize(input: &str) -> String { + driver::inverse_normalize(&ITN_CLASSIFY, &ITN_VERBALIZE, input, " ", "es") } diff --git a/src/fst/fr.rs b/src/fst/fr.rs index 8a5e908..1b184d7 100644 --- a/src/fst/fr.rs +++ b/src/fst/fr.rs @@ -15,6 +15,10 @@ lazy_static! { load_gz(include_bytes!("../../grammars/fr/classify.fst.gz")); static ref VERBALIZE: VectorFst = load_gz(include_bytes!("../../grammars/fr/verbalize.fst.gz")); + static ref ITN_CLASSIFY: VectorFst = + load_gz(include_bytes!("../../grammars/fr/itn_classify.fst.gz")); + static ref ITN_VERBALIZE: VectorFst = + load_gz(include_bytes!("../../grammars/fr/itn_verbalize.fst.gz")); } /// Normalize written-form French (fr) to spoken form. @@ -28,10 +32,16 @@ lazy_static! { /// # } /// ``` pub fn normalize(input: &str) -> String { - driver::normalize(&CLASSIFY, &VERBALIZE, None, input, " ") + driver::normalize(&CLASSIFY, &VERBALIZE, None, input, " ", "fr") } /// Normalize French text and retain the source span for every output token. pub fn normalize_aligned(input: &str) -> Option { - driver::normalize_aligned(&CLASSIFY, &VERBALIZE, None, input, " ") + driver::normalize_aligned(&CLASSIFY, &VERBALIZE, None, input, " ", "fr") +} + +/// Inverse-normalize spoken French (fr) to written form, byte-exact with NeMo's +/// `InverseNormalizer(lang="fr", input_case="cased")`. +pub fn inverse_normalize(input: &str) -> String { + driver::inverse_normalize(&ITN_CLASSIFY, &ITN_VERBALIZE, input, " ", "fr") } diff --git a/src/fst/hi.rs b/src/fst/hi.rs index 3595de3..b313668 100644 --- a/src/fst/hi.rs +++ b/src/fst/hi.rs @@ -17,6 +17,10 @@ lazy_static! { load_gz(include_bytes!("../../grammars/hi/verbalize.fst.gz")); static ref POST: VectorFst = load_gz(include_bytes!("../../grammars/hi/postprocess.fst.gz")); + static ref ITN_CLASSIFY: VectorFst = + load_gz(include_bytes!("../../grammars/hi/itn_classify.fst.gz")); + static ref ITN_VERBALIZE: VectorFst = + load_gz(include_bytes!("../../grammars/hi/itn_verbalize.fst.gz")); } /// Normalize written-form Hindi (hi) to spoken form. @@ -29,10 +33,16 @@ lazy_static! { /// # } /// ``` pub fn normalize(input: &str) -> String { - driver::normalize(&CLASSIFY, &VERBALIZE, Some(&POST), input, " ") + driver::normalize(&CLASSIFY, &VERBALIZE, Some(&POST), input, " ", "hi") } /// Normalize Hindi text and retain the source span for every output token. pub fn normalize_aligned(input: &str) -> Option { - driver::normalize_aligned(&CLASSIFY, &VERBALIZE, Some(&POST), input, " ") + driver::normalize_aligned(&CLASSIFY, &VERBALIZE, Some(&POST), input, " ", "hi") +} + +/// Inverse-normalize spoken Hindi (hi) to written form, byte-exact with NeMo's +/// `InverseNormalizer(lang="hi", input_case="cased")`. +pub fn inverse_normalize(input: &str) -> String { + driver::inverse_normalize(&ITN_CLASSIFY, &ITN_VERBALIZE, input, " ", "hi") } diff --git a/src/fst/it.rs b/src/fst/it.rs index 999d247..17ec9dc 100644 --- a/src/fst/it.rs +++ b/src/fst/it.rs @@ -29,10 +29,10 @@ lazy_static! { /// # } /// ``` pub fn normalize(input: &str) -> String { - driver::normalize(&CLASSIFY, &VERBALIZE, None, input, " ") + driver::normalize(&CLASSIFY, &VERBALIZE, None, input, " ", "it") } /// Normalize Italian text and retain the source span for every output token. pub fn normalize_aligned(input: &str) -> Option { - driver::normalize_aligned(&CLASSIFY, &VERBALIZE, None, input, " ") + driver::normalize_aligned(&CLASSIFY, &VERBALIZE, None, input, " ", "it") } diff --git a/src/fst/ja.rs b/src/fst/ja.rs index 89f066d..1e50b29 100644 --- a/src/fst/ja.rs +++ b/src/fst/ja.rs @@ -15,6 +15,10 @@ lazy_static! { load_gz(include_bytes!("../../grammars/ja/classify.fst.gz")); static ref VERBALIZE: VectorFst = load_gz(include_bytes!("../../grammars/ja/verbalize.fst.gz")); + static ref ITN_CLASSIFY: VectorFst = + load_gz(include_bytes!("../../grammars/ja/itn_classify.fst.gz")); + static ref ITN_VERBALIZE: VectorFst = + load_gz(include_bytes!("../../grammars/ja/itn_verbalize.fst.gz")); } /// Normalize written-form Japanese (ja) to spoken form. @@ -28,10 +32,16 @@ lazy_static! { /// # } /// ``` pub fn normalize(input: &str) -> String { - driver::normalize(&CLASSIFY, &VERBALIZE, None, input, "") + driver::normalize(&CLASSIFY, &VERBALIZE, None, input, "", "ja") } /// Normalize Japanese text and retain the source span for every output token. pub fn normalize_aligned(input: &str) -> Option { - driver::normalize_aligned(&CLASSIFY, &VERBALIZE, None, input, "") + driver::normalize_aligned(&CLASSIFY, &VERBALIZE, None, input, "", "ja") +} + +/// Inverse-normalize spoken Japanese (ja) to written form, byte-exact with NeMo's +/// `InverseNormalizer(lang="ja", input_case="cased")`. +pub fn inverse_normalize(input: &str) -> String { + driver::inverse_normalize(&ITN_CLASSIFY, &ITN_VERBALIZE, input, "", "ja") } diff --git a/src/fst/mod.rs b/src/fst/mod.rs index 0da8a95..b47557b 100644 --- a/src/fst/mod.rs +++ b/src/fst/mod.rs @@ -12,11 +12,15 @@ //! Every language reaches byte-exact parity with //! `Normalizer(lang=…, deterministic=True)`: //! zh 367/367, fr 116/116, ja 542/542, en 506/506, hi 677/677, es 536/536, -//! de 314/314, it 52/52. +//! de 314/314, it 52/52. ITN ([`inverse_normalize_lang`]) runs NeMo's +//! `InverseNormalizer` grammars the same way: en 1217/1217, de 288/288, +//! es 381/381, fr 307/307, hi 352/352, ja 230/230, zh 390/390. //! //! This trades the crate's pure-Rust, tiny-bundle shape for byte-exactness and -//! is therefore optional and off by default. Grammars are bundled gzipped -//! (~8 MB total for all languages) and decompressed once at first use. +//! is therefore an optional feature, off by default. When it is built in, the +//! crate's `normalize*` and `tn_normalize*` entry points run these grammars. +//! Grammars are bundled gzipped (~12 MB total for all languages) and +//! decompressed once at first use. //! //! See `docs/NEMO_PARITY.md` for the measured ceilings and the fidelity fixes. @@ -53,6 +57,21 @@ pub fn normalize_lang(input: &str, lang: &str) -> Option { }) } +/// Inverse-normalize spoken `input` (ITN) with NeMo's grammars for `lang`; +/// `None` for a language without ITN grammars. +pub fn inverse_normalize_lang(input: &str, lang: &str) -> Option { + Some(match lang { + "en" => en::inverse_normalize(input), + "de" => de::inverse_normalize(input), + "es" => es::inverse_normalize(input), + "fr" => fr::inverse_normalize(input), + "hi" => hi::inverse_normalize(input), + "ja" => ja::inverse_normalize(input), + "zh" => zh::inverse_normalize(input), + _ => return None, + }) +} + /// [`normalize_lang`] with the option-gated pre-pass applied first. Only /// [`NormalizeOptions::roman_enumerators`] affects this path; with every flag /// off the output is byte-exact NeMo. diff --git a/src/fst/zh.rs b/src/fst/zh.rs index ba0804a..93fd438 100644 --- a/src/fst/zh.rs +++ b/src/fst/zh.rs @@ -15,6 +15,10 @@ lazy_static! { load_gz(include_bytes!("../../grammars/zh/classify.fst.gz")); static ref VERBALIZE: VectorFst = load_gz(include_bytes!("../../grammars/zh/verbalize.fst.gz")); + static ref ITN_CLASSIFY: VectorFst = + load_gz(include_bytes!("../../grammars/zh/itn_classify.fst.gz")); + static ref ITN_VERBALIZE: VectorFst = + load_gz(include_bytes!("../../grammars/zh/itn_verbalize.fst.gz")); } /// Normalize written-form Mandarin (zh) to spoken form. @@ -29,10 +33,16 @@ lazy_static! { /// # } /// ``` pub fn normalize(input: &str) -> String { - driver::normalize(&CLASSIFY, &VERBALIZE, None, input, "") + driver::normalize(&CLASSIFY, &VERBALIZE, None, input, "", "zh") } /// Normalize Mandarin text and retain the source span for every output token. pub fn normalize_aligned(input: &str) -> Option { - driver::normalize_aligned(&CLASSIFY, &VERBALIZE, None, input, "") + driver::normalize_aligned(&CLASSIFY, &VERBALIZE, None, input, "", "zh") +} + +/// Inverse-normalize spoken Chinese (zh) to written form, byte-exact with NeMo's +/// `InverseNormalizer(lang="zh", input_case="cased")`. +pub fn inverse_normalize(input: &str) -> String { + driver::inverse_normalize(&ITN_CLASSIFY, &ITN_VERBALIZE, input, "", "zh") } diff --git a/src/lib.rs b/src/lib.rs index 25c6094..56df02d 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -36,12 +36,84 @@ use itn::en::{ time, whitelist, word, }; +// With the `fst-engine` feature built in, the entry points below run NeMo's +// compiled grammars, byte-exact with NeMo's `Normalizer` / `InverseNormalizer`. +// Custom rules take precedence over them. The rule-based taggers cover what NeMo +// doesn't: the ITN options other than `max_span_tokens`, and builds without the +// feature. + +static RULE_BASED_ENGINE: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false); + +/// Use the rule-based taggers even when NeMo's grammars are built in +/// (`fst-engine`), for callers that depend on their output. Process-wide; off by +/// default. Without the feature the taggers always run. +pub fn set_rule_based_engine(enabled: bool) { + RULE_BASED_ENGINE.store(enabled, std::sync::atomic::Ordering::Relaxed); +} + +/// Whether [`set_rule_based_engine`] is on. +pub fn rule_based_engine() -> bool { + RULE_BASED_ENGINE.load(std::sync::atomic::Ordering::Relaxed) +} + +/// NeMo ITN for `lang` (other codes read as English, as on the rule-based +/// path), or `None` to use the rule-based taggers. +#[cfg(feature = "fst-engine")] +fn nemo_itn(input: &str, lang: &str) -> Option { + if rule_based_engine() { + return None; + } + // A rule for the whole input is the answer, as on the rule-based path. + if let Some(written) = custom_rules::parse(input) { + return Some(written); + } + let lang = match lang { + "de" | "es" | "fr" | "hi" | "ja" | "zh" => lang, + _ => "en", + }; + let (protected, written_forms) = custom_rules::protect(input); + let output = fst::inverse_normalize_lang(&protected, lang)?; + Some(custom_rules::restore(&output, &written_forms)) +} + +#[cfg(not(feature = "fst-engine"))] +fn nemo_itn(_input: &str, _lang: &str) -> Option { + None +} + +/// NeMo TN for `lang` (other codes read as English), or `None` to use the +/// rule-based taggers. +#[cfg(feature = "fst-engine")] +fn nemo_tn(input: &str, lang: &str) -> Option { + if rule_based_engine() { + return None; + } + let lang = match lang { + "de" | "es" | "fr" | "hi" | "it" | "ja" | "zh" => lang, + _ => "en", + }; + fst::normalize_lang(input, lang) +} + +#[cfg(not(feature = "fst-engine"))] +fn nemo_tn(_input: &str, _lang: &str) -> Option { + None +} + +/// Whether ITN with `options` can run on NeMo's grammars: every ITN flag is off. +/// `max_span_tokens` doesn't matter, since NeMo reads the whole sentence. +fn nemo_handles_itn_options(options: NormalizeOptions) -> bool { + !options.concat_compound_numbers + && !options.disable_bare_second + && !options.small_numbers_as_words +} + /// Normalize spoken-form text to written form. /// /// Tries taggers in order of specificity (most specific first). /// Returns original text if no tagger matches. pub fn normalize(input: &str) -> String { - normalize_inner(input, false) + nemo_itn(input, "en").unwrap_or_else(|| normalize_inner(input, false)) } /// Single-expression dispatch with the `disable_bare_second` flag plumbed in. @@ -155,6 +227,11 @@ fn normalize_inner(input: &str, disable_bare_second: bool) -> String { /// assert_eq!(normalize_with_options("hello world", opts), "hello world"); /// ``` pub fn normalize_with_options(input: &str, options: NormalizeOptions) -> String { + if nemo_handles_itn_options(options) { + if let Some(result) = nemo_itn(input, "en") { + return result; + } + } if !options.concat_compound_numbers { return normalize_inner(input, options.disable_bare_second); } @@ -193,6 +270,9 @@ pub fn normalize_with_options(input: &str, options: NormalizeOptions) -> String /// Supported languages: "en" (default), "fr" (French), "de" (German), /// "es" (Spanish), "hi" (Hindi), "ja" (Japanese), "zh" (Chinese). pub fn normalize_with_lang(input: &str, lang: &str) -> String { + if let Some(result) = nemo_itn(input, lang) { + return result; + } let input = input.trim(); match lang { @@ -1065,7 +1145,9 @@ fn is_zero_to_twelve(word: &str) -> bool { /// assert_eq!(normalize_sentence("hello world"), "hello world"); /// ``` pub fn normalize_sentence(input: &str) -> String { - normalize_sentence_inner(input, DEFAULT_MAX_SPAN_TOKENS, false, false, false) + nemo_itn(input, "en").unwrap_or_else(|| { + normalize_sentence_inner(input, DEFAULT_MAX_SPAN_TOKENS, false, false, false) + }) } /// Unified sentence-mode entry point. @@ -1093,6 +1175,11 @@ pub fn normalize_sentence(input: &str) -> String { /// ); /// ``` pub fn normalize_sentence_with_options(input: &str, options: NormalizeOptions) -> String { + if nemo_handles_itn_options(options) { + if let Some(result) = nemo_itn(input, "en") { + return result; + } + } let max_span = options.max_span_tokens.unwrap_or(DEFAULT_MAX_SPAN_TOKENS); normalize_sentence_inner( input, @@ -1140,6 +1227,9 @@ pub fn normalize_sentence_with_max_span_lang( lang: &str, max_span_tokens: usize, ) -> String { + if let Some(result) = nemo_itn(input, lang) { + return result; + } match lang { // Scanning-based ITN: processors already replace spans across the whole // sentence in place, so the sliding window is unnecessary. @@ -1569,6 +1659,9 @@ fn normalize_sentence_inner( /// assert_eq!(result, "five dollars fifty cents"); /// ``` pub fn tn_normalize(input: &str) -> String { + if let Some(result) = nemo_tn(input, "en") { + return result; + } let input = input.trim(); if let Some(result) = tn::en::whitelist::parse(input) { @@ -1713,7 +1806,7 @@ pub fn tn_normalize_sentence(input: &str) -> String { /// assert_eq!(tn_normalize_lang("123", "en"), "one hundred and twenty three"); /// ``` pub fn tn_normalize_lang(input: &str, lang: &str) -> String { - tn_normalize_for_lang(input, lang) + nemo_tn(input, lang).unwrap_or_else(|| tn_normalize_for_lang(input, lang)) } /// Normalize a full sentence (TN) for a specific language. @@ -1772,6 +1865,9 @@ pub fn tn_normalize_sentence_with_max_span_lang( lang: &str, max_span_tokens: usize, ) -> String { + if let Some(result) = nemo_tn(input, lang) { + return result; + } match lang { "en" | "" => tn_normalize_sentence_with_max_span(input, max_span_tokens), _ => { @@ -1791,6 +1887,9 @@ pub fn tn_normalize_sentence_with_max_span_lang( /// Normalize a full sentence (TN) with a configurable max span size. pub fn tn_normalize_sentence_with_max_span(input: &str, max_span_tokens: usize) -> String { + if let Some(result) = nemo_tn(input, "en") { + return result; + } let trimmed = input.trim(); if trimmed.is_empty() { return trimmed.to_string(); @@ -1801,7 +1900,8 @@ pub fn tn_normalize_sentence_with_max_span(input: &str, max_span_tokens: usize) sentence_loop(&pretokens, max_span_tokens, None, tn_parse_span) } -#[cfg(test)] +// Rule-based output; with `fst-engine` the entry points run NeMo's grammars. +#[cfg(all(test, not(feature = "fst-engine")))] mod tests { use super::*; diff --git a/src/wasm.rs b/src/wasm.rs index e1d72c9..495c17d 100644 --- a/src/wasm.rs +++ b/src/wasm.rs @@ -232,3 +232,10 @@ pub fn clear_rules_js() { pub fn rule_count_js() -> u32 { custom_rules::rule_count() as u32 } + +/// Use the rule-based taggers even when NeMo's grammars are built in, or go back +/// to NeMo's grammars (the default). +#[wasm_bindgen(js_name = setRuleBasedEngine)] +pub fn set_rule_based_engine_js(enabled: bool) { + crate::set_rule_based_engine(enabled); +} diff --git a/swift-test/Sources/CNemoTextProcessing/include/nemo_text_processing.h b/swift-test/Sources/CNemoTextProcessing/include/nemo_text_processing.h index f6c4f9e..3bc84ae 100644 --- a/swift-test/Sources/CNemoTextProcessing/include/nemo_text_processing.h +++ b/swift-test/Sources/CNemoTextProcessing/include/nemo_text_processing.h @@ -33,6 +33,12 @@ void nemo_add_rule(const char* spoken, const char* written); int32_t nemo_remove_rule(const char* spoken); void nemo_clear_rules(void); uint32_t nemo_rule_count(void); + +/** + * Use the rule-based taggers (enabled != 0) even when NeMo's grammars are built in, + * or go back to NeMo's grammars (0, the default). Process-wide. + */ +void nemo_set_rule_based_engine(uint32_t enabled); void nemo_free_string(char* s); const char* nemo_version(void); diff --git a/swift-test/Sources/NemoTest/NemoTest.swift b/swift-test/Sources/NemoTest/NemoTest.swift index b5e43dd..3601ded 100644 --- a/swift-test/Sources/NemoTest/NemoTest.swift +++ b/swift-test/Sources/NemoTest/NemoTest.swift @@ -150,7 +150,8 @@ func runGroup(_ name: String, cases: [TC], transform: (String) -> String) -> Tes let cardinalTests: [TC] = [ ("zero", "zero"), - ("one", "1"), + // NeMo keeps a lone small number as a word (the CI build has NeMo's grammars). + ("one", "one"), ("twenty one", "21"), ("one hundred", "100"), ("two hundred and fifty four", "254"), @@ -298,7 +299,7 @@ let tnMoneyTests: [TC] = [ let tnTimeTests: [TC] = [ ("2:30", "two thirty"), - ("2:05", "two oh five"), + ("2:05", "two o five"), // NeMo's own reading ("2:00 PM", "two PM"), ("2:30 PM", "two thirty PM"), ("8:15 AM", "eight fifteen AM"), @@ -320,7 +321,7 @@ let tnDecimalTests: [TC] = [ ] let tnMeasureTests: [TC] = [ - ("200 km/h", "two hundred kilometers per hour"), + ("200 km/h", "two hundred kilometers per H"), // NeMo's own output for km/h ("1 kg", "one kilogram"), ("2 kg", "two kilograms"), ("-66 kg", "minus sixty six kilograms"), diff --git a/swift/NemoTextProcessing.swift b/swift/NemoTextProcessing.swift index 394ea71..4cb5425 100644 --- a/swift/NemoTextProcessing.swift +++ b/swift/NemoTextProcessing.swift @@ -450,6 +450,14 @@ public enum NemoTextProcessing { Int(nemo_rule_count()) } + // MARK: - Engine + + /// Use the rule-based taggers even though NeMo's grammars are built in, for + /// callers that depend on their output. Process-wide; off by default. + public static func setRuleBasedEngine(_ enabled: Bool) { + nemo_set_rule_based_engine(enabled ? 1 : 0) + } + // MARK: - Info /// Get the library version. diff --git a/swift/include/CNemoTextProcessing/nemo_text_processing.h b/swift/include/CNemoTextProcessing/nemo_text_processing.h index f41a18f..6f1f8c2 100644 --- a/swift/include/CNemoTextProcessing/nemo_text_processing.h +++ b/swift/include/CNemoTextProcessing/nemo_text_processing.h @@ -133,6 +133,12 @@ void nemo_clear_rules(void); */ uint32_t nemo_rule_count(void); +/** + * Use the rule-based taggers (enabled != 0) even when NeMo's grammars are built in, + * or go back to NeMo's grammars (0, the default). Process-wide. + */ +void nemo_set_rule_based_engine(uint32_t enabled); + /** * Text Normalization: convert written-form text to spoken form (TTS preprocessing). * diff --git a/tests/default_engine.rs b/tests/default_engine.rs new file mode 100644 index 0000000..c7e29bf --- /dev/null +++ b/tests/default_engine.rs @@ -0,0 +1,168 @@ +//! With `fst-engine`, the default entry points run NeMo's grammars, and the +//! rule-based taggers take over only for what NeMo doesn't cover. +#![cfg(feature = "fst-engine")] + +use std::sync::Mutex; +use text_processing_rs::{ + custom_rules, fst, normalize, normalize_sentence, normalize_sentence_lang, + normalize_sentence_lang_with_options, normalize_sentence_with_options, normalize_with_lang, + set_rule_based_engine, tn_normalize, tn_normalize_lang, tn_normalize_sentence, + tn_normalize_sentence_lang, NormalizeOptions, +}; + +/// Custom rules and the engine switch are process-wide. +static GLOBAL: Mutex<()> = Mutex::new(()); + +#[test] +fn itn_entry_points_run_nemo() { + let _lock = GLOBAL.lock().unwrap(); + for (input, expected) in [ + // NeMo leaves "no one" and "three to five" alone; the taggers read numbers. + ( + "No one knows, but I paid twenty five dollars for it.", + "No one knows, but I paid $25 for it.", + ), + ( + "We need three to five people.", + "We need three to five people.", + ), + ( + "Call me at five five five one two three four five six seven.", + "Call me at 555-123-4567.", + ), + // NeMo's ITN expects unpunctuated input; the sentence keeps its own punctuation, + // and one period ends `p.m.` at the end of a sentence. + ( + "the meeting is at three pm.", + "the meeting is at 03:00 p.m.", + ), + ("I said no, then twenty one...", "I said no, then 21..."), + ] { + assert_eq!(normalize_sentence(input), expected, "{input}"); + assert_eq!(normalize_sentence_lang(input, "en"), expected, "{input}"); + assert_eq!(normalize_with_lang(input, "en"), expected, "{input}"); + assert_eq!(normalize(input), expected, "{input}"); + } + for (lang, input) in [ + ("de", "ich habe zwei hundert euro"), + ("es", "tengo doscientos cincuenta y un euros"), + ("fr", "j'ai vingt et un ans"), + ("hi", "मेरे पास बीस रुपये हैं"), + ("ja", "二十一"), + ("zh", "一百二十三"), + ] { + let nemo = fst::inverse_normalize_lang(input, lang).unwrap(); + assert_eq!( + normalize_sentence_lang(input, lang), + nemo, + "{lang}: {input}" + ); + assert_eq!(normalize_with_lang(input, lang), nemo, "{lang}: {input}"); + } + // Codes without NeMo ITN grammars read as English, as on the rule-based path. + assert_eq!(normalize_sentence_lang("twenty one", "xx"), "21"); +} + +#[test] +fn tn_entry_points_run_nemo() { + let _lock = GLOBAL.lock().unwrap(); + let input = "I have 5 in the box."; + assert_eq!(tn_normalize_sentence(input), "I have five in the box."); + assert_eq!(tn_normalize(input), "I have five in the box."); + for lang in ["en", "de", "es", "fr", "hi", "it", "ja", "zh"] { + let input = "123"; + let nemo = fst::normalize_lang(input, lang).unwrap(); + assert_eq!(tn_normalize_lang(input, lang), nemo, "{lang}"); + assert_eq!(tn_normalize_sentence_lang(input, lang), nemo, "{lang}"); + } +} + +#[test] +fn itn_options_nemo_lacks_use_the_taggers() { + let _lock = GLOBAL.lock().unwrap(); + let input = "United seven eighty eight"; + // NeMo reads this as 7808; the aviation option concatenates. + assert_eq!(normalize_sentence(input), "United 7808"); + let aviation = NormalizeOptions::new().with_concat_compound_numbers(true); + assert_eq!( + normalize_sentence_with_options(input, aviation), + "United 788" + ); + assert_eq!( + normalize_sentence_lang_with_options(input, "en", aviation), + "United 788" + ); + // max_span_tokens alone doesn't change the engine. + let span = NormalizeOptions::new().with_max_span_tokens(4); + assert_eq!(normalize_sentence_with_options(input, span), "United 7808"); +} + +#[test] +fn custom_rules_apply_ahead_of_nemo() { + let _lock = GLOBAL.lock().unwrap(); + custom_rules::clear_rules(); + custom_rules::add_rule("gee pee tee", "GPT"); + custom_rules::add_rule("acme corp", "ACME Corp."); + assert_eq!( + normalize_sentence("I use Gee pee tee at acme corp, for twenty one dollars."), + "I use GPT at ACME Corp., for $21." + ); + assert_eq!(normalize("gee pee tee"), "GPT"); + custom_rules::clear_rules(); + assert_eq!(normalize_sentence("I use gee pee tee"), "I use gee pee tee"); +} + +#[test] +fn custom_written_forms_are_kept_verbatim() { + let _lock = GLOBAL.lock().unwrap(); + custom_rules::clear_rules(); + // NeMo would read the written form again ("two dollars" -> "$2"). + custom_rules::add_rule("five dollars", "two dollars"); + assert_eq!(normalize("five dollars"), "two dollars"); + assert_eq!( + normalize_sentence("it costs five dollars, not twenty one dollars."), + "it costs two dollars, not $21." + ); + for lang in ["de", "es", "fr", "hi", "ja", "zh"] { + assert_eq!( + normalize_sentence_lang("five dollars", lang), + "two dollars", + "{lang}" + ); + } + custom_rules::clear_rules(); +} + +#[test] +fn custom_rules_match_whole_phrases_only() { + let _lock = GLOBAL.lock().unwrap(); + custom_rules::clear_rules(); + custom_rules::add_rule("gee pee tee", "GPT"); + custom_rules::add_rule("dr.", "Doctor"); + // Punctuation inside the phrase means it isn't the phrase. + assert_eq!( + normalize_sentence("I said gee, pee tee yesterday"), + "I said gee, pee tee yesterday" + ); + // Punctuation around it is kept. + assert_eq!(normalize_sentence("(gee pee tee)."), "(GPT)."); + // A spoken form with its own punctuation matches as registered. + assert_eq!(normalize("dr."), "Doctor"); + assert_eq!(normalize_sentence("ask dr. smith"), "ask Doctor smith"); + custom_rules::clear_rules(); +} + +#[test] +fn the_switch_runs_the_taggers() { + let _lock = GLOBAL.lock().unwrap(); + assert_eq!(normalize("five"), "five"); + + set_rule_based_engine(true); + assert_eq!(normalize("five"), "5"); + assert_eq!( + tn_normalize_sentence("I have 5 in the box."), + "I have five in the box." + ); + set_rule_based_engine(false); + assert_eq!(normalize("five"), "five"); +} diff --git a/tests/en_tests.rs b/tests/en_tests.rs index ef08bb6..090478d 100644 --- a/tests/en_tests.rs +++ b/tests/en_tests.rs @@ -2,6 +2,9 @@ //! //! Test cases sourced from NVIDIA NeMo text processing: //! https://github.com/NVIDIA/NeMo-text-processing +// These tests pin the rule-based taggers' output. With `fst-engine` the default +// entry points run NeMo's grammars instead (tests/default_engine.rs). +#![cfg(not(feature = "fst-engine"))] mod common; diff --git a/tests/en_tn_tests.rs b/tests/en_tn_tests.rs index 2120cc9..942eff2 100644 --- a/tests/en_tn_tests.rs +++ b/tests/en_tn_tests.rs @@ -1,6 +1,9 @@ //! English text normalization tests. //! //! Tests for written → spoken conversion (the reverse of ITN). +// These tests pin the rule-based taggers' output. With `fst-engine` the default +// entry points run NeMo's grammars instead (tests/default_engine.rs). +#![cfg(not(feature = "fst-engine"))] mod common; diff --git a/tests/extensive_tests.rs b/tests/extensive_tests.rs index fb0df6a..6f0efa5 100644 --- a/tests/extensive_tests.rs +++ b/tests/extensive_tests.rs @@ -3,6 +3,9 @@ //! Covers both ITN (spoken → written) and TN (written → spoken) directions, //! sentence-level processing, date handling (flagged for extra testing), //! boundary conditions, roundtrip consistency, and cross-tagger interference. +// These tests pin the rule-based taggers' output. With `fst-engine` the default +// entry points run NeMo's grammars instead (tests/default_engine.rs). +#![cfg(not(feature = "fst-engine"))] use text_processing_rs::{ normalize, normalize_sentence, normalize_sentence_with_options, tn_normalize, diff --git a/tests/fixtures/itn/de/test_cases_cardinal.txt b/tests/fixtures/itn/de/test_cases_cardinal.txt new file mode 100644 index 0000000..efb2ed6 --- /dev/null +++ b/tests/fixtures/itn/de/test_cases_cardinal.txt @@ -0,0 +1,64 @@ +# NeMo ITN output (lang=de, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +ein hundert~100 +einhundert~100 +ein hundert und zwei~102 +einhundertzwei~102 +ein hundert und zwanzig~120 +ein hundert und elf~111 +ein tausend~1000 +eintausend~1000 +ein hundert zwanzig~120 +ein tausend zwanzig~1020 +eintausendzwanzig~1020 +neun billionen sieben hundert neun und achtzig milliarden drei hundert zwei und achtzig millionen fünf hundert sechs und dreißig tausend ein hundert dreißig~9789382536130 +zwei hundert vier und fünfzig~254 +ein hundert sieben und vierzig tausend vier hundert ein und fünfzig~147451 +eine million ein hundert sechs und fünfzig tausend ein hundert drei und siebzig~1156173 +eine milliarde fünf hundert drei und neunzig millionen zwei und siebzig tausend neun hundert ein und sechzig~1593072961 +sieben und neunzig billiarden acht hundert acht billionen zwei hundert vier und sechzig milliarden sieben hundert zwei und siebzig millionen sieben hundert zwei und neunzig tausend fünf~97808264772792005 +zehn billiarden zehn billionen zehn millionen ein hundert tausend zehn~10010000010100010 +zehn billiarden zehn billionen zehn millionen einhunderttausendzehn~10010000010100010 +minus fünf und zwanzig tausend sieben und dreißig~-25037 +minus fünf und zwanzig tausend sieben und dreißig~-25037 +minus fünfundzwanzigtausendsiebenunddreißig~-25037 +eine billiarde zwei hundert vier und sechzig billionen drei hundert eins milliarden neun hundert acht und dreißig millionen ein hundert vier~1264301938000104 +eine billiarde zweihundertvierundsechzig billionen dreihunderteins milliarden neunhundertachtunddreißig millionen einhundertvier~1264301938000104 +minus sechzig~-60 +sechs und vierzig tausend sechs hundert vier und sechzig~46664 +sechzig~60 +null~null +eins~eins +ein~ein +eine~eine +einer~einer +zwei~zwei +neun~neun +zehn~10 +elf~11 +zwölf~12 +dreizehn~13 +vierzehn~14 +fünfzehn~15 +sechzehn~16 +siebzehn~17 +achtzehn~18 +zwanzig~20 +dreißig~30 +vierzig~40 +fünfzig~50 +sechzig~60 +siebzig~70 +achtzig~80 +neunzig~90 +zwei millionen drei~2000003 +ein tausend dreizehn~1013 +ein tausend eins~1001 +ein tausend ein hundert~1100 +ein tausend sechs und zwanzig~1026 +ein tausend ein hundert sechs und zwanzig~1126 +achtzehn millionen vier hundert fünfzig tausend neun hundert neunzig~18450990 +achtzehn millionen neun hundert vierzig tausend sieben hundert zwei und zwanzig~18940722 +achtzehn millionen sechs hundert neunzig tausend neun hundert sechzehn~18690916 +achtzehn millionen sechshundertneunzigtausendneunhundertsechzehn~18690916 +achtzehn tausend acht hundert achtzig~18880 diff --git a/tests/fixtures/itn/de/test_cases_date.txt b/tests/fixtures/itn/de/test_cases_date.txt new file mode 100644 index 0000000..d7c41dc --- /dev/null +++ b/tests/fixtures/itn/de/test_cases_date.txt @@ -0,0 +1,24 @@ +# NeMo ITN output (lang=de, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +vierundzwanzigster juli zwei tausend dreizehn~24. Jul. 2013 +vier und zwanzigster juli zwei tausend dreizehn~24. Jul. 2013 +neunzehn achtzig~1980 +neunzehnachtzig~1980 +neunzehnhundertachtzig~1980 +neunzehn hundert achtzig~1980 +neunzehn achtziger~19 achtziger +zwei tausend zwanzig~2020 +zwanzig zwanzig~2020 +zwei tausend neun~2009 +vierzehnter januar~14. Jan. +januarzweitausendneun~januarzweitausendneun +januar zweitausendneun~Jan. 2009 +erster januar~1. Jan. +dreißigster juni~30. Jun. +neunzehn siebzehn~1917 +neunzehn hundert siebzehn~1917 +neunzehn hundert vierundneunzig~1994 +neunzehn hundert vier und neunzig~1994 +neunzehn vierundneunzig~1994 +zwei tausend drei~2003 +ein tausend acht~1008 diff --git a/tests/fixtures/itn/de/test_cases_decimal.txt b/tests/fixtures/itn/de/test_cases_decimal.txt new file mode 100644 index 0000000..14cd5a3 --- /dev/null +++ b/tests/fixtures/itn/de/test_cases_decimal.txt @@ -0,0 +1,12 @@ +# NeMo ITN output (lang=de, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +null komma zwei millionen~0,2 millionen +eine million~1 million +eins komma zwei millionen~1,2 millionen +achtzehn milliarden~18 milliarden +vier hundert sechzig millionen~460 millionen +ein hundert zwanzig millionen~120 millionen +zehn millionen~10 millionen +minus sechzig komma zwei vier null null~-60,2400 +acht hundert achtzehn komma drei null drei~818,303 +achthundertachtzehn komma drei null drei~818,303 diff --git a/tests/fixtures/itn/de/test_cases_electronic.txt b/tests/fixtures/itn/de/test_cases_electronic.txt new file mode 100644 index 0000000..f075bc8 --- /dev/null +++ b/tests/fixtures/itn/de/test_cases_electronic.txt @@ -0,0 +1,11 @@ +# NeMo ITN output (lang=de, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +c d f at a b c punkt e d u~cdf@abc.edu +a b c at g mail punkt a b c~abc@gmail.abc +a b c at a b c punkt com~abc@abc.com +a s d f eins zwei drei at a b c punkt com~asdf123@abc.com +a eins b zwei at a b c punkt com~a1b2@abc.com +a b drei bindestrich s d d bindestrich drei at g mail punkt com~ab3-sdd-3@gmail.com +h t t p s doppelpunkt slash slash w w w punkt a b c punkt com~https://www.abc.com +w w w punkt a b c punkt com~www.abc.com +h t t p s doppelpunkt slash slash w w w punkt a b c punkt com slash a b fragezeichen gleichheitszeichen drei bindestrich slash a b s slash eins~https://www.abc.com/ab?=3-/abs/1 diff --git a/tests/fixtures/itn/de/test_cases_fraction.txt b/tests/fixtures/itn/de/test_cases_fraction.txt new file mode 100644 index 0000000..557b9db --- /dev/null +++ b/tests/fixtures/itn/de/test_cases_fraction.txt @@ -0,0 +1,36 @@ +# NeMo ITN output (lang=de, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +null nulltel~0/0 +ein halb~1/2 +vier halbe~4/2 +ein drittel~1/3 +ein viertel~1/4 +ein fünftel~1/5 +ein sechstel~1/6 +ein siebtel~1/7 +ein achtel~1/8 +zwei neuntel~2/9 +ein ein halb~1 1/2 +ein sechstel~1/6 +ein zehntel~1/10 +ein elftel~1/11 +ein zehntel~1/10 +ein zwölftel~1/12 +ein dreizehntel~1/13 +ein vierzehntel~1/14 +ein fünfzehntel~1/15 +ein sechzehntel~1/16 +ein siebzehntel~1/17 +ein achtzehntel~1/18 +ein neunzehntel~1/19 +ein zwanzigstel~1/20 +ein dreißigstel~1/30 +ein vierzigstel~1/40 +ein fünfzigstel~1/50 +ein sechzigstel~1/60 +ein siebzigstel~1/70 +ein achtzigstel~1/80 +ein neunzigstel~1/90 +ein ein hundertstel~1/100 +ein zwei und zwanzigstel~1/22 +minus ein zwei und zwanzigstel~-1/22 diff --git a/tests/fixtures/itn/de/test_cases_measure.txt b/tests/fixtures/itn/de/test_cases_measure.txt new file mode 100644 index 0000000..1f3211b --- /dev/null +++ b/tests/fixtures/itn/de/test_cases_measure.txt @@ -0,0 +1,30 @@ +# NeMo ITN output (lang=de, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +zwei hundert meter~200 m +sechs und fünfzig komma drei pro quadrat kilometer~56,3 /km² +zwei hundert kilometer pro stunde~200 km/h +zwei und vierzig tausend zwei hundert neun und fünfzig pro quadrat meter~42259 /m² +minus sechs und sechzig kilogramm~-66 kg +minus sechsundsechzig kilogramm~-66 kg +zwei kilowattstunden~2 kwh +eins komma null null null null zwei acht kubik zentimeter~1,000028 cm³ +eins komma eins zentimeter~1,1 cm +drei stunden~3 h +eine stunde~1 h +ein millivolt~1 mv +eine million millivolt~1 million mv +zwei kubik meter~2 m³ +neunzig gramm~90 g +neunzig millionen gramm~90 millionen g +neunzig komma vier millionen gramm~90,4 millionen g +vier hundert vierzig milliliter~440 ml +drei hundert mikrometer~300 μm +fünf und sechzig tausend quadrat kilometer~65000 km² +zwei kilometer pro stunde~2 km/h +zwei millionen kilometer pro stunde~2 millionen km/h +zwei komma zwei millionen kilometer pro stunde~2,2 millionen km/h +sechzig komma zwei vier null null kilogramm~60,2400 kg +null fuß~0 ft +ein halb fuß~1/2 ft +ein ein halb fuß~1 1/2 ft +minus ein ein halb fuß~-1 1/2 ft diff --git a/tests/fixtures/itn/de/test_cases_money.txt b/tests/fixtures/itn/de/test_cases_money.txt new file mode 100644 index 0000000..0acf683 --- /dev/null +++ b/tests/fixtures/itn/de/test_cases_money.txt @@ -0,0 +1,25 @@ +# NeMo ITN output (lang=de, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +zwei dollar~$2 +ein dollar~$1 +eine million dollar~$1 million +zwei komma null null null eins dollar~$2,0001 +zwei komma null eins dollar~$2,01 +zwei komma null null dollar~$2,00 +ein cent~€0,01 +zwei cent~€0,02 +zwanzig cent~€0,20 +zweiundzwanzig cent~€0,22 +einhundert cent~100 cent +zwei dollar zwanzig~$2,20 +zweidollarzwanzig~zweidollarzwanzig +zwei dollar und zwanzig cent~$2,20 +zwei euro und zwanzig cent~€2,20 +zwei pfund und zwanzig pence~£2,20 +zwei euro zwanzig cent~€2,20 +zwei millionen euro~€2 millionen +zwei komma zwei null null millionen euro~€2,200 millionen +zwei komma zwei null eins millionen euro~€2,201 millionen +zwei komma zwei eins millionen euro~€2,21 millionen +zwei pfund und ein penny~£2,01 +zwei pfund und ein hundert penny~£2 und 100 penny diff --git a/tests/fixtures/itn/de/test_cases_ordinal.txt b/tests/fixtures/itn/de/test_cases_ordinal.txt new file mode 100644 index 0000000..94f8d1e --- /dev/null +++ b/tests/fixtures/itn/de/test_cases_ordinal.txt @@ -0,0 +1,22 @@ +# NeMo ITN output (lang=de, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +ein hundertste~100. +fünf und zwanzig tausend ein hundert elftem~25111. +fünfundzwanzigtausendeinhundertelftem~25111. +zweite~zweite +nullte~nullte +erster~erster +zweiter~zweiter +dritter~dritter +vierter~vierter +zehnter~10. +elftem~11. +dreizehntem~13. +ein und zwanzigstes~21. +drei und zwanzigstes~23. +dreiundzwanzigstes~23. +ein hundert elftes~111. +ein tausendstem~1000. +dem ein tausendstem~dem 1000. +ein hundert ein und zwanzigste~121. +einhunderteinundzwanzigste~121. diff --git a/tests/fixtures/itn/de/test_cases_telephone.txt b/tests/fixtures/itn/de/test_cases_telephone.txt new file mode 100644 index 0000000..936ce35 --- /dev/null +++ b/tests/fixtures/itn/de/test_cases_telephone.txt @@ -0,0 +1,3 @@ +# NeMo ITN output (lang=de, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +null vier eins eins eins zwei drei vier eins zwei drei vier~(0411) 1234-1234 diff --git a/tests/fixtures/itn/de/test_cases_time.txt b/tests/fixtures/itn/de/test_cases_time.txt new file mode 100644 index 0000000..7349163 --- /dev/null +++ b/tests/fixtures/itn/de/test_cases_time.txt @@ -0,0 +1,26 @@ +# NeMo ITN output (lang=de, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +acht uhr~8 Uhr +vier und zwanzig uhr~24 Uhr +vierundzwanziguhr~24 Uhr +vierundzwanzig uhr~24 Uhr +vierundzwanziguhrzweiundzwanzigest~24:22 Uhr est +vierundzwanziguhrzweiundzwanzig e s t~24:22 Uhr est +zwölf uhr mittags~12 Uhr mittags +achtzehn uhr~18 Uhr +acht uhr sieben~08:07 Uhr +null uhr siebzehn~00:17 Uhr +halb zwölf~11:30 Uhr +viertel vor zwölf~11:45 Uhr +drei vor zwölf~11:57 Uhr +zwei und zwanzig vor zwölf~11:38 Uhr +zweiundzwanzig vor zwölf~11:38 Uhr +drei nach zwölf~12:03 Uhr +viertel nach zwölf~12:15 Uhr +zehn nach zwölf~12:10 Uhr +zehn vor zwölf~11:50 Uhr +viertel nach zwölf nachts~12:15 Uhr nachts +null uhr null minuten null sekunden~00:00:00 Uhr +ein uhr eine minute eine sekunde e s t~01:01:01 Uhr est +zwei uhr zwei minuten drei und zwanzig sekunden~02:02:23 Uhr +zwei uhr zwei minuten dreiundzwanzig sekunden~02:02:23 Uhr diff --git a/tests/fixtures/itn/de/test_cases_whitelist.txt b/tests/fixtures/itn/de/test_cases_whitelist.txt new file mode 100644 index 0000000..c6e81a1 --- /dev/null +++ b/tests/fixtures/itn/de/test_cases_whitelist.txt @@ -0,0 +1,8 @@ +# NeMo ITN output (lang=de, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +doktor dao~Dr. dao +miss smith~Ms. smith +misses smith~Mrs. smith +mister dao~Mr. dao +ich mag essen zum beispiel eis~ich mag essen z.B. eis +Chanel nummer fünf~Chanel Nr. fünf diff --git a/tests/fixtures/itn/de/test_cases_word.txt b/tests/fixtures/itn/de/test_cases_word.txt new file mode 100644 index 0000000..6c3dc4a --- /dev/null +++ b/tests/fixtures/itn/de/test_cases_word.txt @@ -0,0 +1,51 @@ +# NeMo ITN output (lang=de, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +~ +yahoo!~yahoo! +zwanzig!~20! +x ~x +—~— +aaa~aaa +aabach~aabach +aabenraa~aabenraa +aabye~aabye +aaccessed~aaccessed +aach~aach +aachen's~aachen's +aadri~aadri +aafia~aafia +aagaard~aagaard +aagadu~aagadu +aagard~aagard +aagathadi~aagathadi +aaghart's~aaghart's +aagnes~aagnes +aagomoni~aagomoni +aagon~aagon +aagoo~aagoo +aagot~aagot +aahar~aahar +aahh~aahh +aahperd~aahperd +aaibinterstate~aaibinterstate +aajab~aajab +aakasa~aakasa +aakervik~aakervik +aakirkeby~aakirkeby +aalam~aalam +aalbaek~aalbaek +aaldiu~aaldiu +aalem~aalem +a'ali~a'ali +aalilaassamthey~aalilaassamthey +aalin~aalin +aaliyan~aaliyan +aaliyan's~aaliyan's +aamadu~aamadu +aamara~aamara +aambala~aambala +aamera~aamera +aamer's~aamer's +aamina~aamina +aaminah~aaminah +aamjiwnaang~aamjiwnaang diff --git a/tests/fixtures/itn/en/test_cases_cardinal.txt b/tests/fixtures/itn/en/test_cases_cardinal.txt new file mode 100644 index 0000000..d5b3764 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_cardinal.txt @@ -0,0 +1,31 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +nine trillion seven hundred eighty nine billion three hundred eighty two million five hundred thirty six thousand one hundred thirty~9789382536130 +two hundred and fifty four~254 +one hundred forty seven thousand four hundred fifty one~147451 +one million one hundred fifty six thousand one hundred seventy three~1156173 +one billion five hundred ninety three million seventy two thousand nine hundred sixty one~1593072961 +ninety seven quadrillion eight hundred eight trillion two hundred sixty four billion seven hundred seventy two million seven hundred ninety two thousand five~97808264772792005 +seventeen sextillion eight hundred fifty five quintillion thirty six quadrillion six hundred fifty seven trillion seven billion five hundred ninety six million one hundred ten thousand nine hundred forty nine~17855036657007596110949 +ten quadrillion ten trillion ten million one hundred thousand ten~10010000010100010 +minus twenty five thousand thirty seven~-25037 +one quadrillion two hundred sixty four trillion three hundred one billion nine hundred thirty eight million one hundred four~1264301938000104 +minus sixty~-60 +forty six thousand six hundred sixty four~46664 +sixty~60 +zero~zero +two million three~2000003 +one thousand thirteen~1013 +one thousand one~1001 +one thousand one hundred~1100 +one thousand twenty six~1026 +one thousand one hundred twenty six~1126 +eighteen million four hundred fifty thousand nine hundred ninety~18450990 +eighteen million nine hundred forty thousand seven hundred twenty two~18940722 +eighteen million six hundred ninety thousand nine hundred sixteen~18690916 +eighteen thousand eight hundred eighty~18880 +eleven hundred~1100 +twenty one hundred~2100 +twenty one hundred and eleven~2111 +eleven hundred twenty one~1121 +twenty one crore ninety eight lakh thirty six thousand five hundred and ninety three~219836593 diff --git a/tests/fixtures/itn/en/test_cases_cardinal_cased.txt b/tests/fixtures/itn/en/test_cases_cardinal_cased.txt new file mode 100644 index 0000000..cd81dd6 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_cardinal_cased.txt @@ -0,0 +1,62 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +nine trillion seven hundred eighty nine billion three hundred eighty two million five hundred thirty six thousand one hundred thirty~9789382536130 +two hundred and fifty four~254 +one hundred forty seven thousand four hundred fifty one~147451 +one million one hundred fifty six thousand one hundred seventy three~1156173 +one billion five hundred ninety three million seventy two thousand nine hundred sixty one~1593072961 +One Billion five hundred ninety three Million seventy two thousand nine hundred sixty one~One Billion 593 Million 72961 +ninety seven quadrillion eight hundred eight trillion two hundred sixty four billion seven hundred seventy two million seven hundred ninety two thousand five~97808264772792005 +seventeen sextillion eight hundred fifty five quintillion thirty six quadrillion six hundred fifty seven trillion seven billion five hundred ninety six million one hundred ten thousand nine hundred forty nine~17855036657007596110949 +ten quadrillion ten trillion ten million one hundred thousand ten~10010000010100010 +minus twenty five thousand thirty seven~-25037 +one quadrillion two hundred sixty four trillion three hundred one billion nine hundred thirty eight million one hundred four~1264301938000104 +minus sixty~-60 +forty six thousand six hundred sixty four~46664 +sixty~60 +zero~zero +two million three~2000003 +one thousand thirteen~1013 +one thousand one~1001 +one thousand one hundred~1100 +one thousand twenty six~1026 +one thousand one hundred twenty six~1126 +eighteen million four hundred fifty thousand nine hundred ninety~18450990 +eighteen million nine hundred forty thousand seven hundred twenty two~18940722 +eighteen million six hundred ninety thousand nine hundred sixteen~18690916 +eighteen thousand eight hundred eighty~18880 +eleven hundred~1100 +twenty one hundred~2100 +twenty one hundred and eleven~2111 +eleven hundred twenty one~1121 +Nine trillion seven hundred eighty nine billion three hundred eighty two million five hundred thirty six thousand one hundred thirty~Nine trillion 789382536130 +Two hundred and fifty four~Two hundred and 54 +One hundred forty seven thousand four hundred fifty one~One hundred 47451 +One million one hundred fifty six thousand one hundred seventy three~One million 156173 +One billion five hundred ninety three million seventy two thousand nine hundred sixty one~One billion 593072961 +Ninety seven quadrillion eight hundred eight trillion two hundred sixty four billion seven hundred seventy two million seven hundred ninety two thousand five~Ninety 7808264772792005 +Seventeen sextillion eight hundred fifty five quintillion thirty six quadrillion six hundred fifty seven trillion seven billion five hundred ninety six million one hundred ten thousand nine hundred forty nine~Seventeen sextillion 855036657007596110949 +Ten quadrillion ten trillion ten million one hundred thousand ten~Ten quadrillion 10000010100010 +Minus twenty five thousand thirty seven~-25037 +One quadrillion two hundred sixty four trillion three hundred one billion nine hundred thirty eight million one hundred four~One quadrillion 264301938000104 +Minus sixty~-60 +Forty six thousand six hundred sixty four~Forty 6664 +Sixty~Sixty +Zero~Zero +Two million three~Two million three +One thousand thirteen~One thousand 13 +One thousand one~One thousand one +One thousand one hundred~One thousand 100 +One thousand twenty six~One thousand 26 +One thousand one hundred twenty six~One thousand 126 +Eighteen million four hundred fifty thousand nine hundred ninety~Eighteen million 450990 +Eighteen million nine hundred forty thousand seven hundred twenty two~Eighteen million 940722 +Eighteen million six hundred ninety thousand nine hundred sixteen~Eighteen million 690916 +Eighteen thousand eight hundred eighty~Eighteen thousand 880 +Eleven hundred~Eleven hundred +Twenty one hundred~Twenty 100 +Twenty one hundred and eleven~Twenty 111 +Eleven hundred twenty one~Eleven hundred 21 +Nineteen~Nineteen +Twelve~Twelve +twenty one crore ninety eight lakh thirty six thousand five hundred and ninety three~219836593 diff --git a/tests/fixtures/itn/en/test_cases_date.txt b/tests/fixtures/itn/en/test_cases_date.txt new file mode 100644 index 0000000..574ebe1 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_date.txt @@ -0,0 +1,38 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +july twenty fifth two thousand twelve~july 25 2012 +nineteen eighties~1980s +two thousand and twenty~2020 +two thousand and nine~2009 +the twenty fifth of july twenty twelve~25 july 2012 +the twenty fifth of july two thousand twelve~25 july 2012 +the twenty second of july twenty twelve~22 july 2012 +the fifteenth of january~15 january +the seventeenth of may twenty ten~17 may 2010 +january first~january 1 +july twenty second two thousand eight~july 22 2008 +june thirty~june 30 +july twenty fifth twenty twelve~july 25 2012 +nineteen seventeen~1917 +twenty twelve~2012 +march sixteen sixty five~march 1665 +sixteen sixty five~1665 +july two thousand twelve~july 2012 +october nineteen oh five~october 1905 +july fifteen o six~july 1506 +the twenty fifth of july twenty twelve~25 july 2012 +july twenty fifth twenty twelve~july 25 2012 +july twenty fifth two thousand twelve~july 25 2012 +july one thousand eight hundred seventy six~july 1876 +february twenty fifth twenty sixteen~february 25 2016 +november twenty fourth twenty fourteen~november 24 2014 +nineteen ninety four~1994 +two thousand three~2003 +one thousand eight~1008 +nineteen seventy six~1976 +june twentieth twenty fourteen~june 20 2014 +nineteen seventy three~1973 +nineteen seventy five~1975 +eleven fifty five~1155 +second quarter of twenty twenty two~Q2 2022 +seven fifty b c~750BC diff --git a/tests/fixtures/itn/en/test_cases_date_cased.txt b/tests/fixtures/itn/en/test_cases_date_cased.txt new file mode 100644 index 0000000..90b953b --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_date_cased.txt @@ -0,0 +1,72 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +july twenty fifth two thousand twelve~july 25 2012 +nineteen eighties~1980s +two thousand and twenty~2020 +two thousand and nine~2009 +the twenty fifth of july twenty twelve~25 july 2012 +the twenty fifth of july two thousand twelve~25 july 2012 +the twenty second of july twenty twelve~22 july 2012 +the fifteenth of january~15 january +the seventeenth of may twenty ten~17 may 2010 +january first~january 1 +july twenty second two thousand eight~july 22 2008 +june thirty~june 30 +july twenty fifth twenty twelve~july 25 2012 +nineteen seventeen~1917 +twenty twelve~2012 +march sixteen sixty five~march 1665 +sixteen sixty five~1665 +july two thousand twelve~july 2012 +october nineteen oh five~october 1905 +july fifteen o six~july 1506 +the twenty fifth of july twenty twelve~25 july 2012 +july twenty fifth twenty twelve~july 25 2012 +july twenty fifth two thousand twelve~july 25 2012 +july one thousand eight hundred seventy six~july 1876 +february twenty fifth twenty sixteen~february 25 2016 +november twenty fourth twenty fourteen~november 24 2014 +nineteen ninety four~1994 +two thousand three~2003 +one thousand eight~1008 +nineteen seventy six~1976 +june twentieth twenty fourteen~june 20 2014 +nineteen seventy three~1973 +nineteen seventy five~1975 +eleven fifty five~1155 +July twenty fifth two thousand twelve~July 25th 2012 +Nineteen eighties~1980s +Two thousand and twenty~Two thousand and 20 +Two thousand and nine~Two thousand and nine +The twenty fifth of july twenty twelve~The 25th of july 2012 +The twenty fifth of july two thousand twelve~The 25th of july 2012 +The twenty second of july twenty twelve~The 22nd of july 2012 +The fifteenth of january~The 15th of january +The fifteenth of January~The 15th of January +The seventeenth of may twenty ten~The 17th of may 2010 +January first~January 1st +July twenty second two thousand eight~July 22nd 2008 +June thirty~June 30 +July twenty fifth twenty twelve~July 25th 2012 +Nineteen seventeen~Nineteen 17 +Twenty twelve~Twenty twelve +March sixteen sixty five~March 1665 +Sixteen sixty five~Sixteen 65 +July two thousand twelve~July 2012 +October nineteen oh five~October 1905 +July fifteen o six~July 1506 +The twenty fifth of july twenty twelve~The 25th of july 2012 +The twenty fifth of July twenty twelve~The 25th of July 2012 +July twenty fifth twenty twelve~July 25th 2012 +July twenty fifth two thousand twelve~July 25th 2012 +July one thousand eight hundred seventy six~July 1876 +February twenty fifth twenty sixteen~February 25th 2016 +November twenty fourth twenty fourteen~November 24th 2014 +Nineteen ninety four~Nineteen 94 +Two thousand three~Two thousand three +One thousand eight~One thousand eight +Nineteen seventy six~Nineteen 76 +June twentieth twenty fourteen~June 20th 2014 +Nineteen seventy three~Nineteen 73 +Nineteen seventy five~Nineteen 75 +Eleven fifty five~Eleven 55 diff --git a/tests/fixtures/itn/en/test_cases_decimal.txt b/tests/fixtures/itn/en/test_cases_decimal.txt new file mode 100644 index 0000000..aafec45 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_decimal.txt @@ -0,0 +1,65 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +five point two million~5.2 million +one hundred sixty four point five eight thousand~164.58 thousand +four hundred million~400 million +fifty billion~50 billion +four hundred five billion~405 billion +four point eight five billion~4.85 billion +one hundred billion~100 billion +one hundred ten billion~110 billion +one hundred thirty two billion~132 billion +one point eight four billion~1.84 billion +one point eight one billion~1.81 billion +one point five nine billion~1.59 billion +one point four five three billion~1.453 billion +one point seven two billion~1.72 billion +one point two five billion~1.25 billion +thirteen billion~13 billion +thirty billion~30 billion +two thousand eight hundred five point eight seven three billion~2805.873 billion +seventy trillion~70 trillion +thirteen million~13 million +eighteen billion~18 billion +four hundred fifty million~450 million +one hundred thirty million~130 million +ten million~10 million +four hundred million~400 million +five million~5 million +five hundred million~500 million +twelve million~12 million +thirteen million~13 million +four million~4 million +forty five million~45 million +fifteen million~15 million +fifteen trillion~15 trillion +fifteen billion~15 billion +two million~2 million +eight million~8 million +point one two o five~.1205 +minus sixty point two four zero zero~-60.2400 +zero point two six~0.26 +point zero two~.02 +sixty point two~60.2 +eighteen~18 +eighteen point eight five~18.85 +eighteen point five o~18.50 +eighteen point five six~18.56 +eighteen point nine~18.9 +eighteen point o five~18.05 +eighteen point one two~18.12 +eighteen point o one~18.01 +eighteen point o o o~18.000 +eighteen point six~18.6 +eighteen point three o o~18.300 +eighteen point three six~18.36 +eighteen point two five~18.25 +eighteen point two two~18.22 +eight hundred eighteen point three o three~818.303 +eight hundred eight point eight~808.8 +eight hundred eight point zero~808.0 +eight hundred eighty eight point one~888.1 +eight hundred eighty four point three~884.3 +eight hundred eighty two point eight~882.8 +eight hundred eighty two point zero~882.0 +eight hundred forty five point nine four~845.94 diff --git a/tests/fixtures/itn/en/test_cases_decimal_cased.txt b/tests/fixtures/itn/en/test_cases_decimal_cased.txt new file mode 100644 index 0000000..e4f15db --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_decimal_cased.txt @@ -0,0 +1,131 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +five point two million~5.2 million +one hundred sixty four point five eight thousand~164.58 thousand +four hundred million~400 million +fifty billion~50 billion +four hundred five billion~405 billion +four point eight five billion~4.85 billion +one hundred billion~100 billion +one hundred ten billion~110 billion +one hundred thirty two billion~132 billion +one point eight four billion~1.84 billion +one point eight one billion~1.81 billion +one point five nine billion~1.59 billion +one point four five three billion~1.453 billion +one point seven two billion~1.72 billion +one point two five billion~1.25 billion +thirteen billion~13 billion +thirty billion~30 billion +two thousand eight hundred five point eight seven three billion~2805.873 billion +seventy trillion~70 trillion +thirteen million~13 million +eighteen billion~18 billion +four hundred fifty million~450 million +one hundred thirty million~130 million +ten million~10 million +four hundred million~400 million +five million~5 million +five hundred million~500 million +twelve million~12 million +thirteen million~13 million +four million~4 million +forty five million~45 million +fifteen million~15 million +fifteen trillion~15 trillion +fifteen billion~15 billion +two million~2 million +eight million~8 million +point one two o five~.1205 +minus sixty point two four zero zero~-60.2400 +zero point two six~0.26 +point zero two~.02 +sixty point two~60.2 +eighteen~18 +eighteen point eight five~18.85 +eighteen point five o~18.50 +eighteen point five six~18.56 +eighteen point nine~18.9 +eighteen point o five~18.05 +eighteen point one two~18.12 +eighteen point o one~18.01 +eighteen point o o o~18.000 +eighteen point six~18.6 +eighteen point three o o~18.300 +eighteen point three six~18.36 +eighteen point two five~18.25 +eighteen point two two~18.22 +eight hundred eighteen point three o three~818.303 +eight hundred eight point eight~808.8 +eight hundred eight point zero~808.0 +eight hundred eighty eight point one~888.1 +eight hundred eighty four point three~884.3 +eight hundred eighty two point eight~882.8 +eight hundred eighty two point zero~882.0 +eight hundred forty five point nine four~845.94 +Five point two million~Five .2 million +One hundred sixty four point five eight thousand~One hundred 64.58 thousand +Four hundred million~Four hundred million +Fifty billion~Fifty billion +Four hundred five billion~Four hundred 5 billion +Four point eight five billion~Four .85 billion +One hundred billion~One hundred billion +One hundred ten billion~One hundred 10 billion +One hundred thirty two billion~One hundred 32 billion +One point eight four billion~One .84 billion +One point eight four Billion~One .84 Billion +One point eight one billion~One .81 billion +One point five nine billion~One .59 billion +One point four five three billion~One .453 billion +One point seven two billion~One .72 billion +One point two five billion~One .25 billion +Thirteen billion~Thirteen billion +Thirty billion~Thirty billion +Thirty Billion~Thirty Billion +Two thousand eight hundred five point eight seven three billion~Two thousand 805.873 billion +Seventy trillion~Seventy trillion +Thirteen million~Thirteen million +Eighteen billion~Eighteen billion +Four hundred fifty million~Four hundred 50 million +One hundred thirty million~One hundred 30 million +Ten million~Ten million +Four hundred million~Four hundred million +Five million~Five million +Five hundred million~Five hundred million +Twelve million~Twelve million +Thirteen million~Thirteen million +Four million~Four million +Forty five million~Forty 5 million +Fifteen million~Fifteen million +Fifteen trillion~Fifteen trillion +Fifteen billion~Fifteen billion +Fifteen Billion~Fifteen Billion +Two million~Two million +Eight million~Eight million +Point one two o five~Point 12o5 +Minus sixty point two four zero zero~-60.2400 +Zero point two six~Zero .26 +Point zero two~Point zero two +Sixty point two~Sixty .2 +Eighteen~Eighteen +Eighteen point eight five~Eighteen .85 +Eighteen point five o~Eighteen .50 +Eighteen point five six~Eighteen .56 +Eighteen point nine~Eighteen .9 +Eighteen point o five~Eighteen .05 +Eighteen point one two~Eighteen .12 +Eighteen point o one~Eighteen .01 +Eighteen point o o o~Eighteen .000 +Eighteen point six~Eighteen .6 +Eighteen point three o o~Eighteen .300 +Eighteen point three six~Eighteen .36 +Eighteen point two five~Eighteen .25 +Eighteen point two two~Eighteen .22 +Eight hundred eighteen point three o three~Eight hundred 18.303 +Eight hundred eight point eight~Eight hundred 8.8 +Eight hundred eight point zero~Eight hundred 8.0 +Eight hundred eighty eight point one~Eight hundred 88.1 +Eight hundred eighty four point three~Eight hundred 84.3 +Eight hundred eighty two point eight~Eight hundred 82.8 +Eight hundred eighty two point zero~Eight hundred 82.0 +Eight hundred forty five point nine four~Eight hundred 45.94 diff --git a/tests/fixtures/itn/en/test_cases_electronic.txt b/tests/fixtures/itn/en/test_cases_electronic.txt new file mode 100644 index 0000000..894c2af --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_electronic.txt @@ -0,0 +1,27 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +a dot b c at g mail dot com~a.bc@gmail.com +a at gmail dot com~a@gmail.com +a at m s n dot fr~a@msn.fr +a at a o l dot com~a@aol.com +a at m s n dot com~a@msn.com +a at nvidia dot com~a@nvidia.com +a dot b c at nvidia dot com~a.bc@nvidia.com +c d f at a b c dot e d u~cdf@abc.edu +a b c at g mail dot a b c~abc@gmail.abc +a b c at a b c dot com~abc@abc.com +a s d f one two three at a b c dot com~asdf123@abc.com +a one b two at a b c dot com~a1b2@abc.com +a b three dot s d d dot three at g mail dot com~ab3.sdd.3@gmail.com +dot three at g mail dot com~dot 3@gmail.com +one three at g mail dot com~13@gmail.com +a b three hyphen s d d dash three at g mail dot com~ab3-sdd-3@gmail.com +h t t p colon slash slash w w w dot o u r d a i l y n e w s dot com dot s m~http://www.ourdailynews.com.sm +h t t p colon slash slash w w w dot c o m d a i l y n e w s dot a b dot s m~http://www.comdailynews.ab.sm +h t t p colon slash slash w w w dot c o m d a i l y n e w s dot a b slash s m~http://www.comdailynews.ab/sm +w w w dot c o m d a i l y n e w s dot a b slash s m~www.comdailynews.ab/sm +c o m d a i l y n e w s dot a b slash s m~comdailynews.ab/sm +n vidia dot com~nvidia.com +abc at gmail dot com~abc@gmail.com +athreed at gmail dot com~athreed@gmail.com +kore dot ai~kore.ai diff --git a/tests/fixtures/itn/en/test_cases_electronic_cased.txt b/tests/fixtures/itn/en/test_cases_electronic_cased.txt new file mode 100644 index 0000000..3f110de --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_electronic_cased.txt @@ -0,0 +1,61 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +a dot b c at g mail dot com~a.bc@gmail.com +a at gmail dot com~a@gmail.com +a at m s n dot fr~a@msn.fr +a at a o l dot com~a@aol.com +a at m s n dot com~a@msn.com +a at nvidia dot com~a@nvidia.com +a dot b c at nvidia dot com~a.bc@nvidia.com +c d f at a b c dot e d u~cdf@abc.edu +a b c at g mail dot a b c~abc@gmail.abc +a b c at a b c dot com~abc@abc.com +a s d f one two three at a b c dot com~asdf123@abc.com +a one b two at a b c dot com~a1b2@abc.com +a b three dot s d d dot three at g mail dot com~ab3.sdd.3@gmail.com +dot three at g mail dot com~dot 3@gmail.com +one three at g mail dot com~13@gmail.com +a b three hyphen s d d dash three at g mail dot com~ab3-sdd-3@gmail.com +h t t p colon slash slash w w w dot o u r d a i l y n e w s dot com dot s m~http://www.ourdailynews.com.sm +h t t p colon slash slash w w w dot c o m d a i l y n e w s dot a b dot s m~http://www.comdailynews.ab.sm +h t t p colon slash slash w w w dot c o m d a i l y n e w s dot a b slash s m~http://www.comdailynews.ab/sm +w w w dot c o m d a i l y n e w s dot a b slash s m~www.comdailynews.ab/sm +c o m d a i l y n e w s dot a b slash s m~comdailynews.ab/sm +n vidia dot com~nvidia.com +abc at gmail dot com~abc@gmail.com +athreed at gmail dot com~athreed@gmail.com +kore dot ai~kore.ai +A dot b c at g mail dot com~A.bc@gmail.com +A at gmail dot com~A@gmail.com +A at m s n dot fr~A@msn.fr +A at a o l dot com~A@aol.com +A at m s n dot com~A@msn.com +A at nvidia dot com~A@nvidia.com +A dot b c at nvidia dot com~A.bc@nvidia.com +C d f at a b c dot e d u~Cdf@abc.edu +A b c at g mail dot a b c~Abc@gmail.abc +A b c at a b c dot com~Abc@abc.com +A s d f one two three at a b c dot com~Asdf123@abc.com +A one b two at a b c dot com~A1b2@abc.com +A b three dot s d d dot three at g mail dot com~Ab3.sdd.3@gmail.com +Dot three at g mail dot com~Dot 3@gmail.com +One three at g mail dot com~One 3@gmail.com +A b three hyphen s d d dash three at g mail dot com~Ab3-sdd-3@gmail.com +H t t p colon slash slash w w w dot o u r d a i l y n e w s dot com dot s m~H t t p colon//www.ourdailynews.com.sm +HTTP colon slash slash w w w dot o u r d a i l y n e w s dot com dot s m~HTTP colon//www.ourdailynews.com.sm +HTTP colon slash slash w w w dot o u r d a i l y n e w s dot com dot Zero one~HTTP colon//www.ourdailynews.com.Zero one +http colon slash slash w w w dot o u r d a i l y n e w s dot com dot SM and s ~http colon//www.ourdailynews.com.SM and s +Http colon slash slash w w w dot o u r d a i l y n e w s dot COM dot s m~Http colon//www.ourdailynews.COM.sm +H t t p colon slash slash w w w dot c o m d a i l y n e w s dot a b dot s m~H t t p colon//www.comdailynews.ab.sm +H t t p colon slash slash w w w dot c o m d a i l y n e w s dot a b slash s m~H t t p colon//www.comdailynews.ab/sm +W w w dot c o m d a i l y n e w s dot a b slash s m~Www.comdailynews.ab/sm +C o m d a i l y n e w s dot a b slash s m~Comdailynews.ab/sm +N vidia dot com~N vidia.com +Abc at gmail dot com~Abc@gmail.com +Athreed at gmail dot com~Athreed@gmail.com +Kore dot ai~Kore.ai +NVIDIA dot com~NVIDIA.com +NVIDIA dot COM~NVIDIA.COM +WWW.A B C at A B C dot com~WWW.A BC@ABC.com +W W W. A B C dot com~W W W. ABC.com +w w w . o u r d a i l y n e w s dot com . s m~w w w . ourdailynews.com. s m diff --git a/tests/fixtures/itn/en/test_cases_measure.txt b/tests/fixtures/itn/en/test_cases_measure.txt new file mode 100644 index 0000000..c1c9a49 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_measure.txt @@ -0,0 +1,114 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +two hundred meters~200 m +fifty six point three per square kilometer~56.3 /km² +two hundred kilometers per hour~200 km/h +twenty eight kilograms force per square centimeter~28 kgf/cm² +forty two thousand two hundred fifty nine per square meter~42259 /m² +minus two thousand twelve kilo liters~-2012 kl +minus sixty six kilograms~-66 kg +two kilo watt hours~2 kWh +one point o o o o two eight cubic deci meters~1.000028 dm³ +seven point five peta bytes~7.5 pb +three hours~3 h +one milli volt~1 mv +two cubic meters~2 m³ +ninety grams~90 g +one hundred twenty four point three lumens~124.3 lm +four hundred forty milliliters~440 ml +thirty one thousand four hundred eighty square feet~31480 sq ft +one thousand six hundred hours~1600 h +thirty one thousand four hundred eighty square feet~31480 sq ft +two square miles~2 sq mi +zero point one nine square miles~0.19 sq mi +one thousand five hundred thirty one c c~1531 cc +three hundred micrometers~300 μm +sixty five thousand square kilometers~65000 km² +two miles per hour~2 mph +two hundred forty five miles per hour~245 mph +one hundred fifty c c~150 cc +sixty point two four zero zero kilograms~60.2400 kg +zero feet~0 ft +zero foot~0 ft +two feet~2 ft +twenty foot~20 ft +point two meters~.2 m +two square meters~2 m² +eighteen feet~18 ft +eighteen mega siemens~18 ms +eighteen ounces~18 oz +eighteen point five kilometers~18.5 km +eighteen point five two square kilometers~18.52 km² +eighteen point nine one square kilometers~18.91 km² +eighteen point one four percent~18.14% +eighteen point one six percent~18.16% +eighteen point one square kilometers~18.1 km² +eighteen point six percent~18.6% +eighteen point two two kilometers~18.22 km +eighteen point zero kilometers~18.0 km +eighteen point zero percent~18.0% +eighteen square kilometers~18 km² +eighteen thousand eight hundred giga watt hours~18800 gWh +eighteen thousand seven hundred hectares~18700 ha +eight hectares~8 ha +eight hundred eighty five astronomical units~885 au +eight hundred eighty hectares~880 ha +eight hundred eighty kilobytes~880 kb +eight hundred eighty kilometers~880 km +eight hundred eighty nine feet~889 ft +eight hundred eighty six kilometers~886 km +eight hundred eighty two megawatts~882 mW +eight hundred feet~800 ft +eight hundred fifty five square kilometers~855 km² +eight hundred fifty megahertz~850 mhz +eight hundred fifty meters~850 m +eight hundred fifty nanometers~850 nm +eight hundred fifty one meters~851 m +eight hundred fifty seven square kilometers~857 km² +eight hundred fifty three meters~853 m +eight hundred fifty three point six meters~853.6 m +eight hundred five point four six square kilometers~805.46 km² +eight hundred forty two point nine meters~842.9 m +eight hundred forty two square kilometers~842 km² +eight hundred gigabytes~800 gb +eight hundred horsepower~800 hp +eight hundred kilograms~800 kg +eight hundred kilo watt hours~800 kWh +eight hundred kilowatts~800 kW +eight hundred megahertz~800 mhz +eight hundred ninety four c c~894 cc +eight hundred ninety kilowatts~890 kW +eight hundred ninety millimeters~890 mm +eight hundred ninety two square kilometers~892 km² +eight hundred seventy horsepower~870 hp +eight hundred seventy meters~870 m +eight hundred sixty kilograms~860 kg +eight hundred sixty kilometers~860 km +eight hundred sixty miles~860 mi +eight hundred sixty six feet~866 ft +eight hundred ten hectares~810 ha +eight hundred ten kilohertz~810 khz +eight hundred thirty eight point two millimeters~838.2 mm +eight hundred thirty five kilometers~835 km +eight hundred thirty kilohertz~830 khz +eight hundred thirty megawatts~830 mW +eight hundred thirty nine kilometers~839 km +eight hundred thirty six meters~836 m +eight hundred twenty feet~820 ft +eight hundred twenty kilometers~820 km +eight hundred twenty meters~820 m +eight hundred twenty one point zero feet~821.0 ft +eight hundred two point eight nine kilometers~802.89 km +eight hundred volts~800 v +eight kilobits~8 kb +eight kilograms~8 kg +eight million two hundred thousand feet~8200000 ft +eight point eight kilometers~8.8 km +eight point eight meters~8.8 m +eight point eight miles~8.8 mi +eight point five centimeters~8.5 cm +eight point five five percent~8.55% +eight point five megawatts~8.5 mW +eight point five meters~8.5 m +eight point five two percent~8.52% +eight point four four percent~8.44% diff --git a/tests/fixtures/itn/en/test_cases_measure_cased.txt b/tests/fixtures/itn/en/test_cases_measure_cased.txt new file mode 100644 index 0000000..bb5fee2 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_measure_cased.txt @@ -0,0 +1,234 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +two hundred meters~200 m +fifty six point three per square kilometer~56.3 /km² +two hundred kilometers per hour~200 km/h +twenty eight kilograms force per square centimeter~28 kgf/cm² +forty two thousand two hundred fifty nine per square meter~42259 /m² +minus two thousand twelve kilo liters~-2012 kl +minus sixty six kilograms~-66 kg +two kilo watt hours~2 kWh +one point o o o o two eight cubic deci meters~1.000028 dm³ +seven point five peta bytes~7.5 pb +three hours~3 h +one milli volt~1 mv +two cubic meters~2 m³ +ninety grams~90 g +one hundred twenty four point three lumens~124.3 lm +four hundred forty milliliters~440 ml +thirty one thousand four hundred eighty square feet~31480 sq ft +one thousand six hundred hours~1600 h +thirty one thousand four hundred eighty square feet~31480 sq ft +two square miles~2 sq mi +zero point one nine square miles~0.19 sq mi +one thousand five hundred thirty one c c~1531 cc +three hundred micrometers~300 μm +sixty five thousand square kilometers~65000 km² +two miles per hour~2 mph +two hundred forty five miles per hour~245 mph +one hundred fifty c c~150 cc +sixty point two four zero zero kilograms~60.2400 kg +zero feet~0 ft +zero foot~0 ft +two feet~2 ft +twenty foot~20 ft +point two meters~.2 m +two square meters~2 m² +eighteen feet~18 ft +eighteen mega siemens~18 ms +eighteen ounces~18 oz +eighteen point five kilometers~18.5 km +eighteen point five two square kilometers~18.52 km² +eighteen point nine one square kilometers~18.91 km² +eighteen point one four percent~18.14% +eighteen point one six percent~18.16% +eighteen point one square kilometers~18.1 km² +eighteen point six percent~18.6% +eighteen point two two kilometers~18.22 km +eighteen point zero kilometers~18.0 km +eighteen point zero percent~18.0% +eighteen square kilometers~18 km² +eighteen thousand eight hundred giga watt hours~18800 gWh +eighteen thousand seven hundred hectares~18700 ha +eight hectares~8 ha +eight hundred eighty five astronomical units~885 au +eight hundred eighty hectares~880 ha +eight hundred eighty kilobytes~880 kb +eight hundred eighty kilometers~880 km +eight hundred eighty nine feet~889 ft +eight hundred eighty six kilometers~886 km +eight hundred eighty two megawatts~882 mW +eight hundred feet~800 ft +eight hundred fifty five square kilometers~855 km² +eight hundred fifty megahertz~850 mhz +eight hundred fifty meters~850 m +eight hundred fifty nanometers~850 nm +eight hundred fifty one meters~851 m +eight hundred fifty seven square kilometers~857 km² +eight hundred fifty three meters~853 m +eight hundred fifty three point six meters~853.6 m +eight hundred five point four six square kilometers~805.46 km² +eight hundred forty two point nine meters~842.9 m +eight hundred forty two square kilometers~842 km² +eight hundred gigabytes~800 gb +eight hundred horsepower~800 hp +eight hundred kilograms~800 kg +eight hundred kilo watt hours~800 kWh +eight hundred kilowatts~800 kW +eight hundred megahertz~800 mhz +eight hundred ninety four c c~894 cc +eight hundred ninety kilowatts~890 kW +eight hundred ninety millimeters~890 mm +eight hundred ninety two square kilometers~892 km² +eight hundred seventy horsepower~870 hp +eight hundred seventy meters~870 m +eight hundred sixty kilograms~860 kg +eight hundred sixty kilometers~860 km +eight hundred sixty miles~860 mi +eight hundred sixty six feet~866 ft +eight hundred ten hectares~810 ha +eight hundred ten kilohertz~810 khz +eight hundred thirty eight point two millimeters~838.2 mm +eight hundred thirty five kilometers~835 km +eight hundred thirty kilohertz~830 khz +eight hundred thirty megawatts~830 mW +eight hundred thirty nine kilometers~839 km +eight hundred thirty six meters~836 m +eight hundred twenty feet~820 ft +eight hundred twenty kilometers~820 km +eight hundred twenty meters~820 m +eight hundred twenty one point zero feet~821.0 ft +eight hundred two point eight nine kilometers~802.89 km +eight hundred volts~800 v +eight kilobits~8 kb +eight kilograms~8 kg +eight million two hundred thousand feet~8200000 ft +eight point eight kilometers~8.8 km +eight point eight meters~8.8 m +eight point eight miles~8.8 mi +eight point five centimeters~8.5 cm +eight point five five percent~8.55% +eight point five megawatts~8.5 mW +eight point five meters~8.5 m +eight point five two percent~8.52% +eight point four four percent~8.44% +Two hundred meters~Two hundred meters +Fifty six point three per square kilometer~56.3 /km² +Two hundred kilometers per hour~Two hundred kilometers per hour +Twenty eight kilograms force per square centimeter~Twenty 8 kgf/cm² +Forty two thousand two hundred fifty nine per square meter~Forty 2259 /m² +Minus two thousand twelve kilo liters~-2012 kl +Minus sixty six kilograms~-66 kg +Two kilo watt hours~Two kilo watt hours +One point o o o o two eight cubic deci meters~1.000028 dm³ +Seven point five peta bytes~7.5 pb +Three hours~Three hours +One milli volt~One milli volt +Two cubic meters~Two cubic meters +Ninety grams~Ninety grams +One hundred twenty four point three lumens~124.3 lm +Four hundred forty milliliters~Four hundred 40 ml +Thirty one thousand four hundred eighty square feet~Thirty 1480 sq ft +One thousand six hundred hours~One thousand 06:00 +Thirty one thousand four hundred eighty square feet~Thirty 1480 sq ft +Two square miles~Two square miles +Zero point one nine square miles~0.19 sq mi +One thousand five hundred thirty one c c~One thousand 531 cc +Three hundred micrometers~Three hundred micrometers +Sixty five thousand square kilometers~Sixty 5000 km² +Two miles per hour~Two miles per hour +Two hundred forty five miles per hour~Two hundred 45 mph +One hundred fifty c c~One hundred 50cc +Sixty point two four zero zero kilograms~60.2400 kg +Zero feet~Zero feet +Zero foot~Zero foot +Two feet~Two feet +Twenty foot~Twenty foot +Point two meters~.2 m +Two square meters~Two square meters +Eighteen feet~Eighteen feet +Eighteen mega siemens~Eighteen mega siemens +Eighteen ounces~Eighteen ounces +Eighteen point five kilometers~18.5 km +Eighteen point five two square kilometers~18.52 km² +Eighteen point nine one square kilometers~18.91 km² +Eighteen point one four percent~18.14% +Eighteen point one six percent~18.16% +Eighteen point one square kilometers~18.1 km² +Eighteen point six percent~18.6% +Eighteen point two two kilometers~18.22 km +Eighteen point zero kilometers~18.0 km +Eighteen point zero percent~18.0% +Eighteen square kilometers~Eighteen square kilometers +Eighteen thousand eight hundred giga watt hours~Eighteen thousand 800 gWh +Eighteen thousand seven hundred hectares~Eighteen thousand 700 ha +Eight hectares~Eight hectares +Eight hundred eighty five astronomical units~Eight hundred 85 au +Eight hundred eighty hectares~Eight hundred 80 ha +Eight hundred eighty kilobytes~Eight hundred 80 kb +Eight hundred eighty kilometers~Eight hundred 80 km +Eight hundred eighty nine feet~Eight hundred 89 ft +Eight hundred eighty six kilometers~Eight hundred 86 km +Eight hundred eighty two megawatts~Eight hundred 82 mW +Eight hundred feet~Eight hundred feet +Eight hundred fifty five square kilometers~Eight hundred 55 km² +Eight hundred fifty megahertz~Eight hundred 50 mhz +Eight hundred fifty meters~Eight hundred 50 m +Eight hundred fifty nanometers~Eight hundred 50 nm +Eight hundred fifty one meters~Eight hundred 51 m +Eight hundred fifty seven square kilometers~Eight hundred 57 km² +Eight hundred fifty three meters~Eight hundred 53 m +Eight hundred fifty three point six meters~853.6 m +Eight hundred five point four six square kilometers~805.46 km² +Eight hundred forty two point nine meters~842.9 m +Eight hundred forty two square kilometers~Eight hundred 42 km² +Eight hundred gigabytes~Eight hundred gigabytes +Eight hundred horsepower~Eight hundred horsepower +Eight hundred kilograms~Eight hundred kilograms +Eight hundred kilo watt hours~Eight hundred kilo watt hours +Eight hundred kilowatts~Eight hundred kilowatts +Eight hundred megahertz~Eight hundred megahertz +Eight hundred ninety four c c~Eight hundred 94cc +Eight hundred ninety kilowatts~Eight hundred 90 kW +Eight hundred ninety millimeters~Eight hundred 90 mm +Eight hundred ninety two square kilometers~Eight hundred 92 km² +Eight hundred seventy horsepower~Eight hundred 70 hp +Eight hundred seventy meters~Eight hundred 70 m +Eight hundred sixty kilograms~Eight hundred 60 kg +Eight hundred sixty kilometers~Eight hundred 60 km +Eight hundred sixty miles~Eight hundred 60 mi +Eight hundred sixty six feet~Eight hundred 66 ft +Eight hundred ten hectares~Eight hundred 10 ha +Eight hundred ten kilohertz~Eight hundred 10 khz +Eight hundred thirty eight point two millimeters~838.2 mm +Eight hundred thirty five kilometers~Eight hundred 35 km +Eight hundred thirty kilohertz~Eight hundred 30 khz +Eight hundred thirty megawatts~Eight hundred 30 mW +Eight hundred thirty nine kilometers~Eight hundred 39 km +Eight hundred thirty six meters~Eight hundred 36 m +Eight hundred twenty feet~Eight hundred 20 ft +Eight hundred twenty kilometers~Eight hundred 20 km +Eight hundred twenty meters~Eight hundred 20 m +Eight hundred twenty one point zero feet~821.0 ft +Eight hundred two point eight nine kilometers~802.89 km +Eight hundred volts~Eight hundred volts +Eight kilobits~Eight kilobits +Eight KiloBits~Eight KiloBits +Eight kilograms~Eight kilograms +Eight million two hundred thousand feet~Eight million 200000 ft +Eight point eight kilometers~8.8 km +Eight point eight meters~8.8 m +Eight point eight miles~8.8 mi +Eight point five centimeters~8.5 cm +Eight point five five percent~8.55% +Eight point five megawatts~8.5 mW +Eight point five meters~8.5 m +Eight point five two percent~8.52% +Eight point four four percent~8.44% +one gigabit per second~1 gbps +nine gigabits per second~9 gbps +five degrees celsius~5 °C +seventy two degrees fahrenheit~72 °F +Seventy two Degrees Fahrenheit~Seventy 2 °F +two hundred seventy three kelvin~273 K +Nine GigaBits per second~Nine GigaBits per 2nd diff --git a/tests/fixtures/itn/en/test_cases_money.txt b/tests/fixtures/itn/en/test_cases_money.txt new file mode 100644 index 0000000..9b8ffe7 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_money.txt @@ -0,0 +1,54 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +two dollars~$2 +one cent~$0.01 +four united states dollars and sixty nine cents~$4.69 +seventy five dollars sixty three~$75.63 +twenty nine dollars fifty cents~$29.50 +eleven dollars and fifty one cents~$11.51 +nine hundred ninety three dollars and ninety two cents~$993.92 +four hundred sixty billion won~₩460 billion +thirty billion yen~¥30 billion +two point five billion dollars~$2.5 billion +forty five billion dollars~$45 billion +fifty million dollars~$50 million +fifty billion dollars~$50 billion +zero point two million dollars~$0.2 million +fifteen point two billion dollars~$15.2 billion +one point six nine billion yuan~1.69 billion yuan +one point four three six billion yuan~1.436 billion yuan +four million yuan~4 million yuan +one dollar~$1 +fifteen thousand dollars~$15000 +one dollars~one dollars +twenty dollar~$20 +twenty point five o six dollars~$20.506 +point five o six dollars~$.506 +eighteen dollars~$18 +eighteen million nine hundred twenty five thousand dollars~$18925000 +eighteen thousand eight hundred fifty four dollars~$18854 +eighteen thousand eight hundred one dollars~$18801 +eighteen thousand eight hundred seventy five dollars~$18875 +eighteen thousand eighty one dollars~$18081 +eighteen thousand fifty two dollars~$18052 +eighteen thousand five hundred forty two dollars~$18542 +eighteen thousand five hundred nineteen dollars~$18519 +eighteen thousand five hundred seventy dollars~$18570 +eighteen thousand five hundred seventy eight dollars~$18578 +eighteen thousand five hundred sixteen dollars~$18516 +eighteen thousand four hundred eighty two dollars~$18482 +eighteen thousand four hundred seventy eight dollars~$18478 +eighteen thousand four hundred sixty eight dollars~$18468 +eighteen thousand nine hundred three dollars~$18903 +eighteen thousand nine hundred twenty nine dollars~$18929 +eighteen thousand ninety five dollars~$18095 +eighteen thousand one hundred seventeen dollars~$18117 +eighteen thousand one hundred twenty eight dollars~$18128 +eighteen thousand one hundred twenty five dollars~$18125 +eighteen thousand one hundred twenty four dollars~$18124 +eighteen thousand one hundred twenty nine dollars~$18129 +one thousand fifty five dollars~$1055 +one fifty five dollars~$155 +fifteen hundred dollars~$1500 +ninety nine hundred dollars~$9900 +ninety nine hundred and fifteen dollars and one cent~$9915.01 diff --git a/tests/fixtures/itn/en/test_cases_money_cased.txt b/tests/fixtures/itn/en/test_cases_money_cased.txt new file mode 100644 index 0000000..b312e2b --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_money_cased.txt @@ -0,0 +1,56 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +Two dollars~Two dollars +One cent~One cent +Four united states dollars and sixty nine cents~Four united states dollars and $0.69 +Seventy five dollars sixty three~Seventy $5.63 +Twenty nine dollars fifty cents~Twenty $9.50 +Eleven dollars and fifty one cents~Eleven dollars and $0.51 +Nine hundred ninety three dollars and ninety two cents~Nine hundred $93.92 +Four hundred sixty billion won~₩460 billion +Thirty billion yen~¥30 billion +Two point five billion dollars~$2.5 billion +Forty five billion dollars~$45 billion +Fifty million dollars~$50 million +Fifty billion dollars~$50 billion +Zero point two million dollars~$0.2 million +Fifteen point two billion dollars~$15.2 billion +One point six nine billion yuan~One .69 billion yuan +One point four three six billion yuan~One .436 billion yuan +Four million yuan~Four million yuan +One dollar~One dollar +Fifteen thousand dollars~Fifteen thousand dollars +One dollars~One dollars +Twenty dollar~Twenty dollar +Twenty point five o six dollars~$20.506 +Point five o six dollars~$.506 +Eighteen dollars~Eighteen dollars +Eighteen million nine hundred twenty five thousand dollars~Eighteen million $925000 +Eighteen thousand eight hundred fifty four dollars~Eighteen thousand $854 +Eighteen thousand eight hundred one dollars~Eighteen thousand $801 +Eighteen thousand eight hundred seventy five dollars~Eighteen thousand $875 +Eighteen thousand eighty one dollars~Eighteen thousand $81 +Eighteen thousand fifty two dollars~Eighteen thousand $52 +Eighteen thousand five hundred forty two dollars~Eighteen thousand $542 +Eighteen thousand five hundred nineteen dollars~Eighteen thousand $519 +Eighteen thousand five hundred seventy dollars~Eighteen thousand $570 +Eighteen thousand five hundred seventy eight dollars~Eighteen thousand $578 +Eighteen thousand five hundred sixteen dollars~Eighteen thousand $516 +Eighteen thousand four hundred eighty two dollars~Eighteen thousand $482 +Eighteen thousand four hundred seventy eight dollars~Eighteen thousand $478 +Eighteen thousand four hundred sixty eight dollars~Eighteen thousand $468 +Eighteen thousand nine hundred three dollars~Eighteen thousand $903 +Eighteen thousand nine hundred twenty nine dollars~Eighteen thousand $929 +Eighteen thousand ninety five dollars~Eighteen thousand $95 +Eighteen thousand one hundred seventeen dollars~Eighteen thousand $117 +Eighteen thousand one hundred twenty eight dollars~Eighteen thousand $128 +Eighteen thousand one hundred twenty five dollars~Eighteen thousand $125 +Eighteen thousand one hundred twenty four dollars~Eighteen thousand $124 +Eighteen thousand one hundred twenty nine dollars~Eighteen thousand $129 +One thousand fifty five dollars~One thousand $55 +One fifty five dollars~One $55 +Fifteen hundred dollars~Fifteen hundred dollars +Ninety nine hundred dollars~Ninety $900 +Ninety nine hundred and fifteen dollars and one cent~Ninety $915.01 +zero point two Million Dollars~0.2 Million Dollars +One Cent~One Cent diff --git a/tests/fixtures/itn/en/test_cases_ordinal.txt b/tests/fixtures/itn/en/test_cases_ordinal.txt new file mode 100644 index 0000000..a047972 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_ordinal.txt @@ -0,0 +1,36 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +one hundredth~100th +twenty five thousand one hundred eleventh~25111th +second~2nd +zeroth~0th +first~1st +second~2nd +third~3rd +fourth~4th +eleventh~11th +twelfth~12th +thirteenth~13th +twenty first~21st +twenty third~23rd +one hundred eleventh~111th +one thousandth~1000th +one hundred twenty first~121st +eleven hundred twenty first~1121st +second~2nd +tenth~10th +sixth~6th +third~3rd +nineteenth~19th +third~3rd +twelfth~12th +forty eighth~48th +seventy first~71st +third~3rd +forty second~42nd +seventeenth~17th +twentieth~20th +twenty first~21st +seventh~7th +second~2nd +fifth~5th diff --git a/tests/fixtures/itn/en/test_cases_ordinal_cased.txt b/tests/fixtures/itn/en/test_cases_ordinal_cased.txt new file mode 100644 index 0000000..d9a8c77 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_ordinal_cased.txt @@ -0,0 +1,36 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +One hundredth~One hundredth +Twenty five thousand one hundred eleventh~Twenty 5111th +Second~Second +Zeroth~Zeroth +First~First +Second~Second +Third~Third +Fourth~Fourth +Eleventh~Eleventh +Twelfth~Twelfth +Thirteenth~Thirteenth +Twenty first~Twenty 1st +Twenty third~Twenty 3rd +One hundred eleventh~One hundred 11th +One thousandth~One thousandth +One hundred twenty first~One hundred 21st +Eleven hundred twenty first~Eleven hundred 21st +Second~Second +Tenth~Tenth +Sixth~Sixth +Third~Third +Nineteenth~Nineteenth +Third~Third +Twelfth~Twelfth +Forty eighth~Forty 8th +Seventy first~Seventy 1st +Third~Third +Forty second~Forty 2nd +Seventeenth~Seventeenth +Twentieth~Twentieth +Twenty first~Twenty 1st +Seventh~Seventh +Second~Second +Fifth~Fifth diff --git a/tests/fixtures/itn/en/test_cases_telephone.txt b/tests/fixtures/itn/en/test_cases_telephone.txt new file mode 100644 index 0000000..aec4d8c --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_telephone.txt @@ -0,0 +1,25 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +one two three one two three five six seven eight~123-123-5678 +plus nine one one two three one two three five six seven eight~+91 123-123-5678 +plus forty four one two three one two three five six seven eight~+44 123-123-5678 +four one two three one two three five six seven eight~4 123-123-5678 +zero two three one two three five six seven eight~023-123-5678 +o two three one two three five six seven eight~023-123-5678 +oh two three one two three five six seven eight~023-123-5678 +double oh three one two three five six seven eight~003-123-5678 +four three two double seven three two one four three two one four three double zero five~432 7732 143214 3005 +one two three dot one two three dot o dot four o~123.123.0.40 +one twenty three dot one two three dot o dot four o~123.123.0.40 +two two five dot double five dot o dot four o~225.55.0.40 +two two five dot double five dot o dot forty five~225.55.0.45 +ssn is seven double nine one two three double one three~ssn is 799-12-3113 +seven nine nine~799 +a b nine~ab9 +a b c~a b c +five w k r a three one~5wkra31 +x eighty six~x86 +x three eighty six~x386 +a thirty six~a 36 +a ten eighty p display~a 1080p display +r t x forty fifty t i~RTX 4050ti diff --git a/tests/fixtures/itn/en/test_cases_telephone_cased.txt b/tests/fixtures/itn/en/test_cases_telephone_cased.txt new file mode 100644 index 0000000..b4d3820 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_telephone_cased.txt @@ -0,0 +1,20 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +One two three one two three five six seven eight~One 231-23-5678 +Plus nine one one two three one two three five six seven eight~Plus 91 123-123-5678 +Plus forty four one two three one two three five six seven eight~Plus 44 123-123-5678 +Four one two three one two three five six seven eight~Four 123-123-5678 +Zero two three one two three five six seven eight~Zero 231-23-5678 +O two three one two three five six seven eight~O231235678 +Oh two three one two three five six seven eight~Oh 231-23-5678 +Double oh three one two three five six seven eight~Double 031-23-5678 +Four three two double seven three two one four three two one four three double zero five~Four 3277 3214 3214 3005 +One two three dot one two three dot o dot four o~One 23.123.0.40 +One twenty three dot one two three dot o dot four o~One 23.123.0.40 +Two two five dot double five dot o dot four o~Two 25.55.0.40 +Two two five dot double five dot o dot forty five~Two 25.55.0.45 +SSN is seven double nine one two three double one three~SSN is 799-12-3113 +Seven nine nine~Seven nine nine +A b nine~Ab9 +A b c~A b c +Five w k r a three one~Five wkra31 diff --git a/tests/fixtures/itn/en/test_cases_time.txt b/tests/fixtures/itn/en/test_cases_time.txt new file mode 100644 index 0000000..fe78eb0 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_time.txt @@ -0,0 +1,31 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +eight oclock g m t~08:00 gmt +seven a m e s t~07:00 a.m. est +two p m~02:00 p.m. +two thirty~02:30 +three o'clock~03:00 +quarter past one~01:15 +half past three~03:30 +eight fifty one~08:51 +eight fifty two~08:52 +eight forty~08:40 +eight nineteen~08:19 +eight o six~08:06 +eight thirty eight~08:38 +eight thirty two~08:32 +eight twenty nine~08:29 +eleven fifty five p m~11:55 p.m. +eleven fifty three p m~11:53 p.m. +eleven forty a m~11:40 a.m. +eleven forty five a m~11:45 a.m. +eleven forty p m~11:40 p.m. +eleven forty six a m~11:46 a.m. +eleven o six p m~11:06 p.m. +eleven thirteen a m~11:13 a.m. +half past twelve~12:30 +quarter past one~01:15 +quarter to one~12:45 +quarter to twelve~11:45 +set alarm at ten to eleven pm~set alarm at 10:50 p.m. +one min to one am~12:59 a.m. diff --git a/tests/fixtures/itn/en/test_cases_time_cased.txt b/tests/fixtures/itn/en/test_cases_time_cased.txt new file mode 100644 index 0000000..eb19ddc --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_time_cased.txt @@ -0,0 +1,35 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +Eight oclock g m t~Eight oclock g m t +Seven a m e s t~Seven a m e s t +Two p m~Two p m +Seven A M E S T~Seven A M E S T +Two P M~Two P M +Two thirty~Two 30 +Three o'clock~Three o'clock +Quarter past one~Quarter past one +Half past three~Half past three +Eight fifty one~Eight 51 +Eight fifty two~Eight 52 +Eight forty~Eight 40 +Eight nineteen~Eight 19 +Eight o six~Eight o six +Eight thirty eight~Eight 38 +Eight thirty two~Eight 32 +Eight twenty nine~Eight 29 +Eleven fifty five p m~Eleven 55pm +Eleven fifty three p m~Eleven 53pm +Eleven forty a m~Eleven 40am +Eleven forty five a m~Eleven 45am +Eleven forty p m~Eleven 40pm +Eleven forty six a m~Eleven 46am +Eleven o six p m~Eleven o6pm +Eleven thirteen a m~Eleven 13:00 a.m. +Half past twelve~Half past twelve +Quarter past one~Quarter past one +Quarter to one~Quarter to one +Quarter to twelve~Quarter to twelve +Set alarm at ten to eleven pm~Set alarm at 10:50 p.m. +One min to one am~One min to 01:00 a.m. +eleven Forty six A M~eleven Forty 6AM +eleven forty six AM~1146 AM diff --git a/tests/fixtures/itn/en/test_cases_whitelist.txt b/tests/fixtures/itn/en/test_cases_whitelist.txt new file mode 100644 index 0000000..7a0d766 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_whitelist.txt @@ -0,0 +1,14 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +doctor dao~dr. dao +misses smith~mrs. smith +mister dao~mr. dao +saint george~st. george +i like for example ice cream~i like e.g. ice cream +s and p five hundred~S&P 500 +seven eleven stores~7-eleven stores +r t x~RTX +cat five e~CAT5e +c u d n n~cuDNN +p c i e x eight~PCIe x8 +l g a eleven fifty~LGA 1150 diff --git a/tests/fixtures/itn/en/test_cases_whitelist_cased.txt b/tests/fixtures/itn/en/test_cases_whitelist_cased.txt new file mode 100644 index 0000000..fb3f62d --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_whitelist_cased.txt @@ -0,0 +1,11 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +Doctor dao~Doctor dao +Misses Smith~Misses Smith +Misses smith~Misses smith +Mister Dao~Mister Dao +Saint George~Saint George +I like for example ice cream~I like e.g. ice cream +S and p five hundred~S and p 500 +S and P Five hundred~S and P Five hundred +Seven eleven stores~Seven eleven stores diff --git a/tests/fixtures/itn/en/test_cases_word.txt b/tests/fixtures/itn/en/test_cases_word.txt new file mode 100644 index 0000000..d947902 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_word.txt @@ -0,0 +1,57 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +~ +, one~, one +, one , two , three , four~, one , two , three , four +e s three~es3 +yahoo!~yahoo! +twenty!~20! +x ~x +—~— +aaa~aaa +aabach~aabach +aabenraa~aabenraa +aabye~aabye +aaccessed~aaccessed +aach~aach +aachen's~aachen's +aadri~aadri +aafia~aafia +aagaard~aagaard +aagadu~aagadu +aagard~aagard +aagathadi~aagathadi +aaghart's~aaghart's +aagnes~aagnes +aagomoni~aagomoni +aagon~aagon +aagoo~aagoo +aagot~aagot +aahar~aahar +aahh~aahh +aahperd~aahperd +aaibinterstate~aaibinterstate +aajab~aajab +aakasa~aakasa +aakervik~aakervik +aakirkeby~aakirkeby +aalam~aalam +aalbaek~aalbaek +aaldiu~aaldiu +aalem~aalem +a'ali~a'ali +aalilaassamthey~aalilaassamthey +aalin~aalin +aaliyan~aaliyan +aaliyan's~aaliyan's +aamadu~aamadu +aamara~aamara +aambala~aambala +aamera~aamera +aamer's~aamer's +aamina~aamina +aaminah~aaminah +aamjiwnaang~aamjiwnaang +~ +, one~, one +, one , two , three , four~, one , two , three , four diff --git a/tests/fixtures/itn/en/test_cases_word_cased.txt b/tests/fixtures/itn/en/test_cases_word_cased.txt new file mode 100644 index 0000000..06525b7 --- /dev/null +++ b/tests/fixtures/itn/en/test_cases_word_cased.txt @@ -0,0 +1,51 @@ +# NeMo ITN output (lang=en, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +E s three~Es3 +Yahoo!~Yahoo! +Twenty!~Twenty! +X ~X +—~— +Aaa~Aaa +Aabach~Aabach +Aabenraa~Aabenraa +Aabye~Aabye +Aaccessed~Aaccessed +Aach~Aach +Aachen's~Aachen's +Aadri~Aadri +Aafia~Aafia +Aagaard~Aagaard +Aagadu~Aagadu +Aagard~Aagard +Aagathadi~Aagathadi +Aaghart's~Aaghart's +Aagnes~Aagnes +Aagomoni~Aagomoni +Aagon~Aagon +Aagoo~Aagoo +Aagot~Aagot +Aahar~Aahar +Aahh~Aahh +Aahperd~Aahperd +Aaibinterstate~Aaibinterstate +Aajab~Aajab +Aakasa~Aakasa +Aakervik~Aakervik +Aakirkeby~Aakirkeby +Aalam~Aalam +Aalbaek~Aalbaek +Aaldiu~Aaldiu +Aalem~Aalem +A'ali~A'ali +Aalilaassamthey~Aalilaassamthey +Aalin~Aalin +Aaliyan~Aaliyan +Aaliyan's~Aaliyan's +Aamadu~Aamadu +Aamara~Aamara +Aambala~Aambala +Aamera~Aamera +Aamer's~Aamer's +Aamina~Aamina +Aaminah~Aaminah +Aamjiwnaang~Aamjiwnaang diff --git a/tests/fixtures/itn/es/test_cases_cardinal.txt b/tests/fixtures/itn/es/test_cases_cardinal.txt new file mode 100644 index 0000000..f04d04b --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_cardinal.txt @@ -0,0 +1,53 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +doscientos cincuenta y uno~251 +novecientos noventa y nueve millones novecientos noventa y nueve mil novecientos noventa y nueve~999999999 +cero~cero +uno~uno +una~una +dos~dos +nueve~nueve +diez~10 +, uno~, uno +, diez~, 10 +menos veintitrés~-23 +cien~100 +ciento uno~101 +ciento un~101 +ciento una~101 +mil y uno~1001 +mil uno~1001 +nueve billones setecientos ochenta y nueve mil trescientos ochenta y dos millones quinientos treinta y seis mil ciento treinta~9789382536130 +doscientos cincuenta y cuatro~254 +ciento cuarenta y siete mil cuatrocientos cincuenta y uno~147451 +un millón ciento cincuenta y seis mil ciento setenta y tres~1156173 +mil quinientos noventa y tres millones setenta y dos mil novecientos sesenta y uno~1593072961 +noventa y siete mil ochocientos ocho billones doscientos sesenta y cuatro mil setecientos setenta y dos millones setecientos noventa y dos mil cinco~97808264772792005 +diecisiete mil ochocientos cincuenta y cinco trillones treinta y seis mil seiscientos cincuenta y siete billones siete mil quinientos noventa y seis millones ciento diez mil novecientos cuarenta y nueve~17855036657007596110949 +diez mil diez billones diez millones cien mil diez~10010000010100010 +menos veinticinco mil treinta y siete~-25037 +mil doscientos sesenta y cuatro billones trescientos un mil novecientos treinta y ocho millones ciento cuatro~1264301938000104 +menos sesenta~-60 +cuarenta y seis mil seiscientos sesenta y cuatro~46664 +sesenta~60 +dos millones tres~2000003 +mil trece~1013 +mil cien~1100 +mil veintiséis~1026 +mil ciento veintiséis~1126 +dieciocho millones cuatrocientos cincuenta mil novecientos noventa~18450990 +dieciocho millones novecientos cuarenta mil setecientos veintidós~18940722 +dieciocho millones seiscientos noventa mil novecientos dieciséis~18690916 +dieciocho mil ochocientos ochenta~18880 +un millardo uno~1000000001 +mil millones uno~1000000001 +mil millones ciento uno~1000000101 +mil millones mil ciento uno~1000001101 +mil millones diez mil ciento uno~1000010101 +mil un millón diez mil ciento uno~1001010101 +dos millardos cincuenta y dos~2000000052 +muchas millones~muchas millones +mil billones uno~1000000000000001 +mil trillones uno~1000000000000000000001 +veintiacuátro~veintiacuátro +entre dieciséis mil y dieciocho mil~entre 16000 y 18000 diff --git a/tests/fixtures/itn/es/test_cases_cardinal_cased.txt b/tests/fixtures/itn/es/test_cases_cardinal_cased.txt new file mode 100644 index 0000000..429931a --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_cardinal_cased.txt @@ -0,0 +1,32 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +Doscientos cincuenta y uno~Doscientos 51 +Novecientos noventa y nueve millones novecientos noventa y nueve mil novecientos noventa y nueve~Novecientos 99999999 +Cero~Cero +Uno~Uno +una~una +dos~dos +Nueve~Nueve +Diez~Diez +, uno~, uno +, diez~, 10 +Menos veintitrés~-23 +cien~100 +ciento uno~101 +ciento un~101 +ciento una~101 +mil y uno~1001 +Mil una~Mil una +nueve billones setecientos ochenta y nueve mil trescientos ochenta y dos millones quinientos treinta y seis mil ciento treinta~9789382536130 +Doscientos cincuenta y cuatro~Doscientos 54 +ciento cuarenta y siete mil cuatrocientos cincuenta y uno~147451 +Un Millón ciento cincuenta y seis mil ciento setenta y tres~Un Millón 156173 +Mil quinientos noventa y tres millones setenta y dos mil novecientos sesenta y uno~Mil 593072961 +noventa y siete mil ochocientos ocho billones doscientos sesenta y cuatro mil setecientos setenta y dos millones setecientos noventa y dos mil cinco~97808264772792005 +diecisiete mil ochocientos cincuenta y cinco trillones treinta y seis mil seiscientos cincuenta y siete billones siete mil quinientos noventa y seis millones ciento diez mil novecientos cuarenta y nueve~17855036657007596110949 +diez mil diez billones diez millones cien mil diez~10010000010100010 +Menos veinticinco mil treinta y siete~-25037 +mil doscientos sesenta y cuatro billones trescientos un mil novecientos treinta y ocho millones ciento cuatro~1264301938000104 +menos sesenta~-60 +cuarenta y seis mil seiscientos sesenta y cuatro~46664 +sesenta~60 diff --git a/tests/fixtures/itn/es/test_cases_date.txt b/tests/fixtures/itn/es/test_cases_date.txt new file mode 100644 index 0000000..2b49254 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_date.txt @@ -0,0 +1,10 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +primero de enero~1 de enero +uno de enero~1 de enero +el uno de diciembre~el 1 de diciembre +el primero de diciembre~el 1 de diciembre +domingo veintiséis de octubre~domingo 26 de octubre +treinta y uno de diciembre de mil novecientos noventa y dos~31 de diciembre de 1992 +siglo diecinueve~siglo xix +doscientos tres antes de cristo~203 a. c. diff --git a/tests/fixtures/itn/es/test_cases_date_cased.txt b/tests/fixtures/itn/es/test_cases_date_cased.txt new file mode 100644 index 0000000..cf82baa --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_date_cased.txt @@ -0,0 +1,10 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +Primero De Enero~Primero De Enero +Uno de enero~Uno de enero +el uno de Diciembre~el uno de Diciembre +El primero de diciembre~El 1 de diciembre +Domingo Veintiséis De Octubre~Domingo Veintiséis De Octubre +treinta y uno de diciembre de mil novecientos noventa y dos~31 de diciembre de 1992 +Siglo diecinueve~Siglo 19 +doscientos tres antes de Cristo~203 antes de Cristo diff --git a/tests/fixtures/itn/es/test_cases_decimal.txt b/tests/fixtures/itn/es/test_cases_decimal.txt new file mode 100644 index 0000000..32a6c03 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_decimal.txt @@ -0,0 +1,31 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +uno coma dos seis~1,26 +menos uno coma dos seis~-1,26 +uno coma veintiséis~1,26 +cero coma dos seis~0,26 +cero coma veintiséis~0,26 +tres coma ciento cuarenta y uno~3,141 +tres coma cero ciento cuarenta y uno~3,0141 +tres coma ciento cuarenta y uno cincuenta y nueve~3,14159 +tres coma catorce ciento cincuenta y nueve~3,14159 +tres coma catorce quince noventa y dos sesenta y cinco tres~3,141592653 +tres coma catorce quince cero noventa y dos sesenta y cinco treinta y cinco~3,14150926535 +tres coma catorce quince cero novecientos veintiséis cero quinientos treinta y cinco~3,141509260535 +cuatrocientos millones~400 millones +uno punto treinta y tres~1.33 +uno punto treinta y tres millones~1.33 millones +cero coma seis millones~0,6 millones +mil ochocientos veinticuatro millón~1824 millón +mil ochocientos veinticuatro millones~1824 millones +punto dos seis~.26 +un millón~1 millón +dos millones~2 millones +un millardo~1 millardo +dos millardos~2 millardos +un billón~1 billón +dos billones~2 billones +un trillón~1 trillón +dos trillones~2 trillones +un cuatrillón~1 cuatrillón +dos cuatrillones~2 cuatrillones diff --git a/tests/fixtures/itn/es/test_cases_decimal_cased.txt b/tests/fixtures/itn/es/test_cases_decimal_cased.txt new file mode 100644 index 0000000..668105e --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_decimal_cased.txt @@ -0,0 +1,8 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +Uno coma dos seis~Uno,26 +Menos uno coma dos seis~-1,26 +Uno Coma Veintiséis~Uno Coma Veintiséis +Cero coma Dos seis~Cero,Dos seis +cero coma veintiséis~0,26 +tres coma ciento cuarenta y uno~3,141 diff --git a/tests/fixtures/itn/es/test_cases_electronic.txt b/tests/fixtures/itn/es/test_cases_electronic.txt new file mode 100644 index 0000000..fa8fea9 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_electronic.txt @@ -0,0 +1,18 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +a punto b c arroba g mail punto com~a.bc@gmail.com +c d f arroba a b c punto e d u~cdf@abc.edu +a b c arroba g mail punto a b c~abc@gmail.abc +a b c arroba a b c punto com~abc@abc.com +a s d f uno dos tres arroba a b c punto com~asdf123@abc.com +a uno b dos arroba a b c punto com~a1b2@abc.com +a b tres punto s d d punto tres arroba g mail punto com~ab3.sdd.3@gmail.com +hache te te pe ese dos puntos barra barra doble ve doble ve doble ve punto n vidia punto com~https://www.nvidia.com +doble ve doble ve doble ve punto n vidia punto com~www.nvidia.com +doble ve doble ve doble ve punto nvidia punto com~www.nvidia.com +w w w punto nvidia punto com~www.nvidia.com +doble ve doble ve doble ve punto a b c punto es barra e f g~www.abc.es/efg +doble ve doble ve doble ve punto a b c punto e s~www.abc.es +doble ve doble ve doble ve punto a b c punto es barra e f g signo de interrogación i d signo igual a b c~www.abc.es/efg?id=abc +doble ve doble ve doble ve punto a b c punto gob~www.abc.gob +doble ve doble ve doble ve punto a b c punto d e f~www.abc.def diff --git a/tests/fixtures/itn/es/test_cases_electronic_cased.txt b/tests/fixtures/itn/es/test_cases_electronic_cased.txt new file mode 100644 index 0000000..788dd82 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_electronic_cased.txt @@ -0,0 +1,7 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +A punto B C Arroba G mail punto com~A.BC Arroba G mail.com +c d f Arroba a b c Punto e d u~c d f Arroba a b c Punto e d u +W W W Punto N vidia Punto com~W W W Punto N vidia Punto com +Doble ve doble ve doble ve punto a b c punto es barra e f g~Doble ve doble ve doble ve.abc.es/efg +Doble Ve Doble Ve Doble Ve Punto a b c Punto e s~Doble Ve Doble Ve Doble Ve Punto a b c Punto e s diff --git a/tests/fixtures/itn/es/test_cases_fraction.txt b/tests/fixtures/itn/es/test_cases_fraction.txt new file mode 100644 index 0000000..cc6af34 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_fraction.txt @@ -0,0 +1,14 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +medio~medio +un cuarto~un cuarto +ocho tercios~8/3 +dos quintos~2/5 +diez treintavos~10/30 +tres vigésimos~3/20 +once cientounavos~11/101 +un décimo~1/10 +un cuarentiunavo~1/41 +dos y dos tercios~2 2/3 +menos cuatro y un quinto~-4 1/5 +menos diez veinteavos~-10/20 diff --git a/tests/fixtures/itn/es/test_cases_measure.txt b/tests/fixtures/itn/es/test_cases_measure.txt new file mode 100644 index 0000000..5b78410 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_measure.txt @@ -0,0 +1,22 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +doscientos metros~200 m +tres horas~3 h +una hora~1 h +doscientos cuarenta y cinco millas por hora~245 mph +dos kilos~2 kg +sesenta coma dos cuatro cero cero kilogramos~60,2400 kg +menos sesenta coma veinticuatro cero cero kilogramos~-60,2400 kg +ocho coma cinco dos por ciento~8,52% +menos ocho coma cinco dos por ciento~-8,52% +uno porciento~1% +tres centímetros~3 cm +cuatro segundos~4 s +cinco litros~5 l +tres metros cúbicos~3 m³ +dos kilómetros por hora~2 kph +diez grados farenheit~10 ° F +dos metros y medio~2 1/2 m +tres quintos de metro~3/5 m +menos tres y medio metros por hora~-3 1/2 m/h +dos más dos es igual a cuatro~2 + 2 = 4 diff --git a/tests/fixtures/itn/es/test_cases_measure_cased.txt b/tests/fixtures/itn/es/test_cases_measure_cased.txt new file mode 100644 index 0000000..9fea61b --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_measure_cased.txt @@ -0,0 +1,13 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +Doscientos metros~Doscientos metros +tres horas~3 h +una hora~1 h +Doscientos cuarenta y cinco Millas Por Hora~Doscientos 45 mph +Dos Kilos~Dos Kilos +sesenta coma dos cuatro cero cero kilogramos~60,2400 kg +Menos sesenta coma veinticuatro cero cero kilogramos~-60,2400 kg +menos Ocho Coma Cinco Dos por ciento~menos Ocho Coma Cinco Dos por 100 +uno Porciento~1% +tres centímetros~3 cm +dos más dos es igual a cuatro~2 + 2 = 4 diff --git a/tests/fixtures/itn/es/test_cases_money.txt b/tests/fixtures/itn/es/test_cases_money.txt new file mode 100644 index 0000000..51d56ef --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_money.txt @@ -0,0 +1,26 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +doce dólares y cinco centavos~$12,05 +doce dólares y cinco céntimos~$12,05 +setenta y cinco dólares sesenta y tres~$75,63 +setenta y cinco dólares y sesenta y tres centavos~$75,63 +setenta y cinco dólares con sesenta y tres centavos~$75,63 +setenta y cinco dólares con sesenta y tres~$75,63 +veintinueve dólares cincuenta centavos~$29,50 +un dólar~$1 +veinticinco centavos~$0,25 +veinticinco céntimos~€0,25 +doce euros y cinco centavos~€12,05 +doce dólares estadounidenses y cinco centavos~US$12,05 +doce dólares americanos y cinco centavos~US$12,05 +doce pesos y cinco centavos~$12,05 +doce yenes y cinco centavos~¥12,05 +dos dólares y sesenta y tres dólares~$2 y $63 +diez pesetas~₧10 +un colón~₡1 +un chon~₩0,01 +tres wones con veinte~₩3,20 +cien quetzales~q100 +nueve punto cinco millones de pesos~$9.5 millones +catorce millones quinientos mil pesos mexicanos~Mex$14500000 +diez pesos mexicanos~Mex$10 diff --git a/tests/fixtures/itn/es/test_cases_money_cased.txt b/tests/fixtures/itn/es/test_cases_money_cased.txt new file mode 100644 index 0000000..f7e13fa --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_money_cased.txt @@ -0,0 +1,8 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +doce dólares y cinco centavos~$12,05 +Doce Dólares Y Cinco Céntimos~Doce Dólares Y Cinco Céntimos +setenta y cinco Dólares sesenta y tres~75 Dólares 63 +Veintinueve dólares cincuenta centavos~Veintinueve dólares $0,50 +Catorce millones quinientos mil Pesos mexicanos~Catorce 500000 Pesos mexicanos +diez pesos Mexicanos~$10 Mexicanos diff --git a/tests/fixtures/itn/es/test_cases_ordinal.txt b/tests/fixtures/itn/es/test_cases_ordinal.txt new file mode 100644 index 0000000..da0c645 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_ordinal.txt @@ -0,0 +1,32 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +primero~primero +tercera~tercera +primer~primer +tercer~tercer +noveno~noveno +novena~novena +décimo~10.º +décima~10.ª +undécimo~11.º +undécima~11.ª +decimoprimero~11.º +décimo primero~11.º +decimoprimer~11.ᵉʳ +décimo primer~11.ᵉʳ +decimoprimera~11.ª +décima primera~11.ª +(technically ungrammatical) décimo primera~(technically ungrammatical) 11.ª +decimotercero~13.º +vigésimo primero~21.º +vigésima primera~21.ª +(technically ungrammatical) vigésimo primera~(technically ungrammatical) 21.ª +vigésimo primer~21.ᵉʳ +vigesimosegundo~22.º +vigésimo segundo~22.º +vigesimosegunda~22.ª +vigésima segunda~22.ª +vigésimo tercero~23.º +centésimo undécimo~111.º +centésimo trigésimo cuarto~134.º +vigesimoctavo~28.º diff --git a/tests/fixtures/itn/es/test_cases_ordinal_cased.txt b/tests/fixtures/itn/es/test_cases_ordinal_cased.txt new file mode 100644 index 0000000..8574814 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_ordinal_cased.txt @@ -0,0 +1,13 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +primero~primero +Tercera~Tercera +Primer~Primer +tercer~tercer +Décima~Décima +undécimo~11.º +Decimoprimer~Decimoprimer +Décimo primer~Décimo primer +Décima Primera~Décima Primera +(technically ungrammatical) décimo primera~(technically ungrammatical) 11.ª +decimotercero~13.º diff --git a/tests/fixtures/itn/es/test_cases_telephone.txt b/tests/fixtures/itn/es/test_cases_telephone.txt new file mode 100644 index 0000000..d14a864 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_telephone.txt @@ -0,0 +1,11 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +uno dos tres uno dos tres cinco seis siete ocho~123-123-5678 +uno veintitrés uno veintitrés cincuenta y seis setenta y ocho~123-123-5678 +uno dos tres cuatro cinco seis siete ocho nueve~123-456-789 +uno veintitrés cuatro cincuenta y seis siete ochenta y nueve~123-456-789 +uno dos tres cuatro cinco seis siete ocho~1234-5678 +doce treinta y cuatro cincuenta y seis setenta y ocho~1234-5678 +triple tres uno dos tres cinco seis siete ocho~333-123-5678 +más uno uno dos tres uno dos tres cinco seis siete ocho~+1-123-123-5678 +más cincuenta y cuatro uno dos tres uno dos tres cinco seis siete ocho extensión doce~+54-123-123-5678 ext. 12 diff --git a/tests/fixtures/itn/es/test_cases_telephone_cased.txt b/tests/fixtures/itn/es/test_cases_telephone_cased.txt new file mode 100644 index 0000000..c6cdd97 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_telephone_cased.txt @@ -0,0 +1,8 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +Uno dos tres uno dos tres cinco seis siete ocho~Uno 231-235-678 +uno veintitrés uno veintitrés cincuenta y seis setenta y ocho~123-123-5678 +Uno Dos Tres Cuatro Cinco Seis Siete Ocho Nueve~Uno Dos Tres Cuatro Cinco Seis Siete Ocho Nueve +Triple tres uno dos tres cinco seis siete ocho~Triple 3123-5678 +Más uno uno dos tres uno dos tres cinco seis siete ocho~Más 112-312-3567 ocho +más cincuenta y cuatro uno dos tres uno dos tres cinco seis siete ocho Extensión doce~+54-123-123-5678 Extensión 12 diff --git a/tests/fixtures/itn/es/test_cases_time.txt b/tests/fixtures/itn/es/test_cases_time.txt new file mode 100644 index 0000000..a578bb6 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_time.txt @@ -0,0 +1,27 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +las dieciséis cincuenta~las 16:50 +la una~la una +las dos~las dos +las tres personas~las tres personas +las dos a eme~las 2:00 a.m. +la una pe eme~la 1:00 p.m. +la una y diez~la 1:10 +la una y diez a eme~la 1:10 a.m. +la una y diez pe eme~la 1:10 p.m. +la una diez~la 1:10 +la una con diez~la 1:10 +la una y cuarto~la 1:15 +la una y media~la 1:30 +las dos menos veinte~la 1:40 +las dos menos cuarto~la 1:45 +cuarto para las dos~la 1:45 +un cuarto para las dos~la 1:45 +las veintitrés y media~las 23:30 +las veintitrés y cincuenta y nueve~las 23:59 +las dos de la tarde~las 2:00 p.m. +cuarto para las cero~las 23:45 +cuarto para las veinticuatro~las 23:45 +diez para las doce~las 11:50 +dos y media de la tarde~2:30 p.m. +la una de la tarde u t c más cuatro~la 1:00 p.m. UTC+4 diff --git a/tests/fixtures/itn/es/test_cases_time_cased.txt b/tests/fixtures/itn/es/test_cases_time_cased.txt new file mode 100644 index 0000000..c212e12 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_time_cased.txt @@ -0,0 +1,11 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +las dieciséis cincuenta~las 16:50 +la una~la una +Las dos~Las dos +Las tres personas~Las tres personas +Las Dos a eme~Las Dos a eme +la una Pe Eme~la una Pe Eme +la una y diez~la 1:10 +la una y Diez a eme~la una y Diez a eme +La Una Y Diez pe eme~La Una Y Diez pe eme diff --git a/tests/fixtures/itn/es/test_cases_whitelist.txt b/tests/fixtures/itn/es/test_cases_whitelist.txt new file mode 100644 index 0000000..0b66cf8 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_whitelist.txt @@ -0,0 +1,7 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +usted~Ud. +ustedes~Uds. +habla usted español~habla Ud. español +hablan ustedes español~hablan Uds. español +estados unidos~EE. UU. diff --git a/tests/fixtures/itn/es/test_cases_word.txt b/tests/fixtures/itn/es/test_cases_word.txt new file mode 100644 index 0000000..3e5d6b4 --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_word.txt @@ -0,0 +1,51 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +~ +yahoo!~yahoo! +veinte!~20! +x ~x +—~— +aaa~aaa +aabach~aabach +aabenraa~aabenraa +aabye~aabye +aaccessed~aaccessed +aach~aach +aachen's~aachen's +aadri~aadri +aafia~aafia +aagaard~aagaard +aagadu~aagadu +aagard~aagard +aagathadi~aagathadi +aaghart's~aaghart's +aagnes~aagnes +aagomoni~aagomoni +aagon~aagon +aagoo~aagoo +aagot~aagot +aahar~aahar +aahh~aahh +aahperd~aahperd +aaibinterstate~aaibinterstate +aajab~aajab +aakasa~aakasa +aakervik~aakervik +aakirkeby~aakirkeby +aalam~aalam +aalbaek~aalbaek +aaldiu~aaldiu +aalem~aalem +a'ali~a'ali +aalilaassamthey~aalilaassamthey +aalin~aalin +aaliyan~aaliyan +aaliyan's~aaliyan's +aamadu~aamadu +aamara~aamara +aambala~aambala +aamera~aamera +aamer's~aamer's +aamina~aamina +aaminah~aaminah +aamjiwnaang~aamjiwnaang diff --git a/tests/fixtures/itn/es/test_cases_word_cased.txt b/tests/fixtures/itn/es/test_cases_word_cased.txt new file mode 100644 index 0000000..3d4806d --- /dev/null +++ b/tests/fixtures/itn/es/test_cases_word_cased.txt @@ -0,0 +1,13 @@ +# NeMo ITN output (lang=es, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +~ +Yahoo!~Yahoo! +Veinte!~Veinte! +X ~X +—~— +AAA~AAA +Aabach~Aabach +aabenraa~aabenraa +Aachen's~Aachen's +aadri~aadri +aaliyan's~aaliyan's diff --git a/tests/fixtures/itn/fr/test_cases_cardinal.txt b/tests/fixtures/itn/fr/test_cases_cardinal.txt new file mode 100644 index 0000000..5c54554 --- /dev/null +++ b/tests/fixtures/itn/fr/test_cases_cardinal.txt @@ -0,0 +1,108 @@ +# NeMo ITN output (lang=fr, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +cent~100 +dix-huit~18 +vingt et un~21 +vingt-et-un~21 +trente et un~31 +trente-et-un~31 +quarante-trois~43 +quarante trois~40 trois +cinquante et un~51 +cinquante-et-un~51 +soixante et un~61 +soixante-et-un~61 +soixante-dix~70 +soixante-douze~72 +quatre-vingts~80 +quatre-vingt-dix-huit~98 +cent~100 +cent deux~102 +cent-deux~102 +cent vingt~120 +cent-vingt~120 +deux-cents~200 +deux cent neuf~209 +deux-cent-neuf~209 +cent onze~111 +cent-onze~111 +mille~1000 +cent vingt~120 +cent-vingt~120 +mille vingt~1020 +mille-vingt~1020 +neuf billion sept cent quatre-vingt-neuf milliard trois cent quatre-vingt-deux million cinq cent trente-six mille cent trente~9789382536130 +neuf-billion-sept-cent-quatre-vingt-neuf-milliard-trois-cent-quatre-vingt-deux-million-cinq-cent-trente-six-mille-cent-trente~9789382536130 +deux cent cinquante-quatre~254 +deux-cent-cinquante-quatre~254 +cent quarante-sept mille quatre cent cinquante et une~147451 +cent-quarante-sept-mille-quatre-cent-cinquante-et-une~147451 +un million cent cinquante-six mille cent soixante-treize~1156173 +un-million-cent-cinquante-six-mille-cent-soixante-treize~1156173 +un milliard cinq cent quatre-vingt-treize million soixante-douze mille neuf cent soixante et un~1593072961 +un-milliard-cinq-cent-quatre-vingt-treize-million-soixante-douze-mille-neuf-cent-soixante-et-un~1593072961 +un milliard cinq cent quatre-vingt-treize million septante-deux mille neuf cent soixante et un~1593072961 +un-milliard-cinq-cent-quatre-vingt-treize-million-septante-deux-mille-neuf-cent-soixante-et-un~1593072961 +quatre-vingt-dix-sept billiard huit cent huit billion deux cent soixante-quatre milliard sept cent soixante-douze million sept cent quatre-vingt-douze mille cinq~97808264772792005 +quatre-vingt-dix-sept-billiard-huit-cent-huit-billion-deux-cent-soixante-quatre-milliard-sept-cent-soixante-douze-million-sept-cent-quatre-vingt-douze-mille-cinq~97808264772792005 +dix billiard dix billion dix million cent mille dix~10010000010100010 +dix-billiard-dix-billion-dix-million-cent-mille-dix~10010000010100010 +moins vingt-cinq mille trente-sept~-25037 +moins vingt-cinq-mille-trente-sept~-25037 +moins dix-neuf cent trente-sept~-1937 +moins dix-neuf-cent-trente-sept~-1937 +un billiard deux cent soixante-quatre billion trois cent un milliard neuf cent trente-huit million cent quatre~1264301938000104 +un-billiard-deux-cent-soixante-quatre-billion-trois-cent-un-milliard-neuf-cent-trente-huit-million-cent-quatre~1264301938000104 +moins soixante~-60 +quarante-six mille six cent soixante-quatre~46664 +quarante-six-mille-six-cent-soixante-quatre~46664 +soixante~60 +zéro~zéro +un~un +une~une +deux~deux +neuf~neuf +dix~10 +onze~11 +douze~12 +treize~13 +quatorze~14 +quinze~15 +seize~16 +dix-sept~17 +dix-huit~18 +vingt~20 +trente~30 +quarante~40 +cinquante~50 +soixante~60 +soixante-dix~70 +septante~70 +quatre-vingts~80 +huitante~80 +quatre-vingt-dix~90 +deux million dix~2000010 +deux-million-dix~2000010 +mille treize~1013 +mille-treize~1013 +mille un~1001 +mille-un~1001 +mille cent~1100 +mille-cent~1100 +onze cents~1100 +onze-cents~1100 +dix-huit mille treize~18013 +dix-huit-mille-treize~18013 +mille vingt-six~1026 +mille-vingt-six~1026 +mille cent vingt-six~1126 +mille-cent-vingt-six~1126 +onze cent vingt-six~1126 +onze-cent-vingt-six~1126 +dix-huit million quatre cent cinquante mille neuf cent quatre-vingt-dix~18450990 +dix-huit-million-quatre-cent-cinquante-mille-neuf-cent-quatre-vingt-dix~18450990 +dix-huit-million-quatre-cent-cinquante-mille-neuf-cent-nonante~18450990 +dix-huit mille huit cent quatre-vingts~18880 +dix-huit-mille-huit-cent-quatre-vingts~18880 +dix-huit mille huit cent huitante~18880 +dix-huit-mille-huit-cent-huitante~18880 diff --git a/tests/fixtures/itn/fr/test_cases_date.txt b/tests/fixtures/itn/fr/test_cases_date.txt new file mode 100644 index 0000000..5c8c534 --- /dev/null +++ b/tests/fixtures/itn/fr/test_cases_date.txt @@ -0,0 +1,8 @@ +# NeMo ITN output (lang=fr, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +vingt-quatre juillet deux-mille-treize~24 juillet 2013 +vingt-quatre juillet~24 juillet +quatorze janvier~14 janvier +premier janvier~1ᵉʳ janvier +trente juin~30 juin +dix-huit mai dix-neuf cent trente~18 mai 1930 diff --git a/tests/fixtures/itn/fr/test_cases_decimal.txt b/tests/fixtures/itn/fr/test_cases_decimal.txt new file mode 100644 index 0000000..c51921d --- /dev/null +++ b/tests/fixtures/itn/fr/test_cases_decimal.txt @@ -0,0 +1,17 @@ +# NeMo ITN output (lang=fr, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +zéro virgule deux million~0,2 million +dix-huit milliards~18 milliards +quatre cent soixante millions~460 millions +quatre-cent-soixante millions~460 millions +quatre-cent-soixante-millions~460 millions +cent vingt millions~120 millions +cent-vingt-millions~120 millions +cent vingt millions~120 millions +dix billions~10 billions +dix-billions~10 billions +moins soixante virgule deux quatre zéro zéro~-60,240 0 +huit cent dix-huit virgule trois zéro trois~818,303 +huit-cent-dix-huit virgule trois zéro trois~818,303 +huit-cent-dix-huit virgule trente trois~818,303 +mille-huit-cent-dix-huit virgule trois zéro trois trois quatre~1 818,303 34 diff --git a/tests/fixtures/itn/fr/test_cases_electronic.txt b/tests/fixtures/itn/fr/test_cases_electronic.txt new file mode 100644 index 0000000..4e36320 --- /dev/null +++ b/tests/fixtures/itn/fr/test_cases_electronic.txt @@ -0,0 +1,12 @@ +# NeMo ITN output (lang=fr, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +a point b c arobase g mail point com~a.bc@gmail.com +a point b c at g mail point com~a.bc@gmail.com +c d f at a b c point e d u~cdf@abc.edu +a b c at g mail point a b c~abc@gmail.abc +a b c arobase g mail point a b c~abc@gmail.abc +a b c at a b c point com~abc@abc.com +a s d f un deux trois at a b c point com~asdf123@abc.com +a un b deux arobase a b c point com~a1b2@abc.com +a b trois point s d d point trois at g mail point com~ab3.sdd.3@gmail.com +a b trois point s d d point trois arobase g mail point com~ab3.sdd.3@gmail.com diff --git a/tests/fixtures/itn/fr/test_cases_fraction.txt b/tests/fixtures/itn/fr/test_cases_fraction.txt new file mode 100644 index 0000000..2637f99 --- /dev/null +++ b/tests/fixtures/itn/fr/test_cases_fraction.txt @@ -0,0 +1,32 @@ +# NeMo ITN output (lang=fr, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +demi~1/2 +un tiers~1/3 +un quart~1/4 +un cinquième~1/5 +un sixième~1/6 +un septième~1/7 +un huitième~1/8 +deux neuvièmes~2/9 +un et demi~1 1/2 +un dixième~1/10 +un onzième~1/11 +un douzième~1/12 +un treizième~1/13 +un quatrième~1/4 +un quatorzième~1/14 +un quinzième~1/15 +un seizième~1/16 +un dix-septième~1/17 +un dix-huitième~1/18 +un dix-neuvième~1/19 +un vingtième~1/20 +un trentième~1/30 +un quarantième~1/40 +un cinquantième~1/50 +un soixantième~1/60 +un soixante-dixième~1/70 +un quatre-vingtième~1/80 +un quatre-vingt-dixième~1/90 +un centième~1/100 +quatre et deux quatrièmes~4 2/4 diff --git a/tests/fixtures/itn/fr/test_cases_measure.txt b/tests/fixtures/itn/fr/test_cases_measure.txt new file mode 100644 index 0000000..d634a28 --- /dev/null +++ b/tests/fixtures/itn/fr/test_cases_measure.txt @@ -0,0 +1,17 @@ +# NeMo ITN output (lang=fr, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +deux cents mètres~200 m +cinquante-six virgule trois par kilomètre carré~56,3 /km² +deux-cents kilomètres par heure~200 km/h +deux-cents kilomètres heure~200 km/h +quarante-deux-mille-deux-cent-cinquante-neuf par mètre carré~42 259 /m² +moins soixante-six kilogrammes~-66 kg +un virgule zéro zéro zéro zéro vingt-huit centimètre cube~1,000 028 cm³ +cinquante minutes~50 min +deux mètres cubes~2 m³ +quatre-vingt-dix grammes~90 g +quatre-cent-quarante millilitres~440 ml +trois cents micromètres~300 µm +soixante-cinq kilomètres carrés~65 km² +deux kilomètres par heure~2 km/h +soixante virgule vingt-quatre zéro zéro kilogrammes~60,240 0 kg diff --git a/tests/fixtures/itn/fr/test_cases_money.txt b/tests/fixtures/itn/fr/test_cases_money.txt new file mode 100644 index 0000000..7022b6a --- /dev/null +++ b/tests/fixtures/itn/fr/test_cases_money.txt @@ -0,0 +1,24 @@ +# NeMo ITN output (lang=fr, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +deux dollars~2 $ +un centime~0,01 € +vingt centimes~0,20 € +vingt-deux centimes~0,22 € +deux dollars vingt~2,20 $ +deux euros et vingt centimes~2,20 € +vingt euros~20 € +un franc suisse~1 CHF +vingt euro cinq~20,05 € +un euro~1 € +deux euro~2 € +cinq euro et soixante~5,60 € +cinquante centimes~0,50 € +quatre-vingt mille won~80 000 ₩ +quatre-vingt-mille won~80 000 ₩ +quatre-vingt-millions de wons~80 millions de wons +trois livre~3 £ +trois pence~0,03 £ +zéro euro~0 € +zéro euro quatre-vingt~0,80 € +deux-millions de dollars~2 millions de dollars +quatre virgule quatre-vingt milliards d'euros~4,80 milliards d'euros diff --git a/tests/fixtures/itn/fr/test_cases_ordinal.txt b/tests/fixtures/itn/fr/test_cases_ordinal.txt new file mode 100644 index 0000000..2f5af9f --- /dev/null +++ b/tests/fixtures/itn/fr/test_cases_ordinal.txt @@ -0,0 +1,25 @@ +# NeMo ITN output (lang=fr, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +centième~100ᵉ +centièmes~100ᵉˢ +vingt-cinq-mille-cent-onzième~25111ᵉ +première~1ʳᵉ +premières~1ʳᵉˢ +premier~1ᵉʳ +premiers~1ᵉʳˢ +second~2ᵈ +seconds~2ᵈˢ +seconde~2ᵈᵉ +secondes~2ᵈᵉˢ +deuxième~2ᵉ +troisième~3ᵉ +quatrième~4ᵉ +onzièmes~11ᵉˢ +treizième~13ᵉ +vingt-et-unième~21ᵉ +vingt-troisièmes~23ᵉˢ +cent-onzième~111ᵉ +cent onzième~111ᵉ +millième~1000ᵉ +dix-neuvième siècle~XIXᵉ siècle +vingtième siècle~XXᵉ siècle diff --git a/tests/fixtures/itn/fr/test_cases_telephone.txt b/tests/fixtures/itn/fr/test_cases_telephone.txt new file mode 100644 index 0000000..d312f81 --- /dev/null +++ b/tests/fixtures/itn/fr/test_cases_telephone.txt @@ -0,0 +1,7 @@ +# NeMo ITN output (lang=fr, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +zéro deux douze trente-deux trente trente~02 12 32 30 30 +zéro deux une deux trois deux trois zéro trois zéro~02 12 32 30 30 +deux douze trente-deux trente trente~02 12 32 30 30 +deux une deux trois deux trois zéro trois zéro~02 12 32 30 30 +double neuf douze trente-deux trente trente~99 12 32 30 30 diff --git a/tests/fixtures/itn/fr/test_cases_time.txt b/tests/fixtures/itn/fr/test_cases_time.txt new file mode 100644 index 0000000..eedd328 --- /dev/null +++ b/tests/fixtures/itn/fr/test_cases_time.txt @@ -0,0 +1,20 @@ +# NeMo ITN output (lang=fr, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +huit heures~8 h +huit heures du matin~8 h +huit heures du soir~20 h +minuit~0 h +deux heures de l'après-midi~14 h +quatorze heures~14 h +midi~12 h +dix-huit heures~18 h +huit heures sept~8 h 07 +minuit dix-sept~0 h 17 +douze heures~12 h +onze heures et demie~11 h 30 +midi moins le quart~11 h 45 +onze heures et trois quarts~11 h 45 +midi moins trois~11 h 57 +onze heures cinquante-sept~11 h 57 +onze heures trente-huit~11 h 38 +midi moins vingt-deux~11 h 38 diff --git a/tests/fixtures/itn/fr/test_cases_whitelist.txt b/tests/fixtures/itn/fr/test_cases_whitelist.txt new file mode 100644 index 0000000..a8e9b5d --- /dev/null +++ b/tests/fixtures/itn/fr/test_cases_whitelist.txt @@ -0,0 +1,10 @@ +# NeMo ITN output (lang=fr, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +docteur~Dʳ +docteures~Dʳᵉˢ +monsieur~M. +messieurs~MM. +madame~Mᵐᵉ +mesdames~Mᵐᵉˢ +mademoiselle~Mˡˡᵉ +mademoiselles~Mˡˡᵉˢ diff --git a/tests/fixtures/itn/fr/test_cases_word.txt b/tests/fixtures/itn/fr/test_cases_word.txt new file mode 100644 index 0000000..3fe0081 --- /dev/null +++ b/tests/fixtures/itn/fr/test_cases_word.txt @@ -0,0 +1,51 @@ +# NeMo ITN output (lang=fr, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +~ +yahoo!~yahoo! +vingt!~20! +x ~x +—~— +aaa~aaa +aabach~aabach +aabenraa~aabenraa +aabye~aabye +aaccessed~aaccessed +aach~aach +aachen's~aachen's +aadri~aadri +aafia~aafia +aagaard~aagaard +aagadu~aagadu +aagard~aagard +aagathadi~aagathadi +aaghart's~aaghart's +aagnes~aagnes +aagomoni~aagomoni +aagon~aagon +aagoo~aagoo +aagot~aagot +aahar~aahar +aahh~aahh +aahperd~aahperd +aaibinterstate~aaibinterstate +aajab~aajab +aakasa~aakasa +aakervik~aakervik +aakirkeby~aakirkeby +aalam~aalam +aalbaek~aalbaek +aaldiu~aaldiu +aalem~aalem +a'ali~a'ali +aalilaassamthey~aalilaassamthey +aalin~aalin +aaliyan~aaliyan +aaliyan's~aaliyan's +aamadu~aamadu +aamara~aamara +aambala~aambala +aamera~aamera +aamer's~aamer's +aamina~aamina +aaminah~aaminah +aamjiwnaang~aamjiwnaang diff --git a/tests/fixtures/itn/hi/test_cases_address.txt b/tests/fixtures/itn/hi/test_cases_address.txt new file mode 100644 index 0000000..23980a7 --- /dev/null +++ b/tests/fixtures/itn/hi/test_cases_address.txt @@ -0,0 +1,27 @@ +# NeMo ITN output (lang=hi, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +सात शून्य शून्य ओक स्ट्रीट~७०० ओक स्ट्रीट +एक एक जंगल रोड~११ जंगल रोड +तीन शून्य एक पार्क एवेन्यू~३०१ पार्क एवेन्यू +गली नंबर एक सात जीएकगढ़~गली नंबर १७ जीएकगढ़ +अदनान अपार्टमेंट फ्लैट नंबर पाँच पाँच~अदनान अपार्टमेंट फ्लैट नंबर ५५ +प्लॉट नंबर आठ बालाजी मार्केट~प्लॉट नंबर ८ बालाजी मार्केट +बूथ सात शून्य, सेक्टर आठ, चंडीगढ़~बूथ ७०, सेक्टर ८, चंडीगढ़ +दो दो दो एक सदर्न स्ट्रीट~२२२१ सदर्न स्ट्रीट +छह दो पाँच स्कूल स्ट्रीट~६२५ स्कूल स्ट्रीट +पाँच शून्य छह स्टेट रोड~५०६ स्टेट रोड +छह छह हाइफ़न चार, पार्कहर्स्ट रोड~६६-४, पार्कहर्स्ट रोड +एक चार बटा तीन, मथुरा रोड~१४/३, मथुरा रोड +अमरावती छह पाँच पाँच नौ तीन शून्य~अमरावती ६५५९३० +अमरावती चार छह आठ दो पाँच दो~अमरावती ४६८२५२ +शिमला, हिमाचल प्रदेश पाँच नौ तीन नौ आठ आठ~शिमला, हिमाचल प्रदेश ५९३९८८ +रांची, झारखंड सात तीन छह पाँच पाँच सात~रांची, झारखंड ७३६५५७ +कोहिमा, नागालैंड चार चार आठ तीन सात सात~कोहिमा, नागालैंड ४४८३७७ +मुंबई, महाराष्ट्र आठ तीन नौ चार आठ आठ~मुंबई, महाराष्ट्र ८३९४८८ +मुंबई, महाराष्ट्र दो नौ शून्य नौ तीन सात~मुंबई, महाराष्ट्र २९०९३७ +गांधीनगर, गुजरात आठ शून्य आठ तीन सात चार~गांधीनगर, गुजरात ८०८३७४ +रायपुर, छत्तीसगढ़ एक एक शून्य छह तीन पाँच~रायपुर, छत्तीसगढ़ ११०६३५ +भोपाल, मध्य प्रदेश सात पाँच एक दो दो पाँच~भोपाल, मध्य प्रदेश ७५१२२५ +अगरतला, त्रिपुरा नौ एक पाँच तीन शून्य पाँच~अगरतला, त्रिपुरा ९१५३०५ +लखनऊ, उत्तर प्रदेश आठ शून्य दो चार आठ एक~लखनऊ, उत्तर प्रदेश ८०२४८१ +श्रीनगर, जम्मू और कश्मीर नौ छह चार पाँच दो तीन~श्रीनगर, जम्मू और कश्मीर ९६४५२३ diff --git a/tests/fixtures/itn/hi/test_cases_cardinal.txt b/tests/fixtures/itn/hi/test_cases_cardinal.txt new file mode 100644 index 0000000..b7106c0 --- /dev/null +++ b/tests/fixtures/itn/hi/test_cases_cardinal.txt @@ -0,0 +1,56 @@ +# NeMo ITN output (lang=hi, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +चार चौके~४ चौके +छः खिलाड़ी आउट~६ खिलाड़ी आउट +वनप्लस आठ प्रो~वनप्लस ८ प्रो +पाँच चार्जर~५ चार्जर +चार ओवर में सत्रह रन~४ ओवर में १७ रन +पाँच चॉकलेट्स नौ टॉफ़िज़~५ चॉकलेट्स ९ टॉफ़िज़ +दस हजार निन्यानवे~१००९९ +एक लाख एक~१००००१ +एक सौ~१०० +तीन सौ नौ~३०९ +सात सौ अट्ठानवे~७९८ +पाँच हज़ार~५००० +आठ हज़ार चार~८००४ +नौ हज़ार सोलह~९०१६ +उन्नीस सौ बारह~१९१२ +दो हज़ार दो सौ बाईस~२२२२ +चौदह हज़ार~१४००० +अठारह हज़ार छह~१८००६ +छब्बीस हज़ार इक्कीस~२६०२१ +छियानवे हज़ार आठ सौ ग्यारह~९६८११ +चार लाख~४००००० +दो लाख दो~२००००२ +सात लाख बीस~७०००२० +नौ लाख तीन सौ इक्कीस~९००३२१ +आठ लाख पाँच हज़ार तीन सौ इक्कीस~८०५३२१ +तेईस लाख~२३००००० +पन्द्रह लाख एक~१५००००१ +सत्ताईस लाख आठ सौ बीस~२७००८२० +इक्यानवे लाख इकतीस हज़ार आठ सौ उनतीस~९१३१८२९ +तीन करोड़~३००००००० +एक करोड़ एक~१००००००१ +सात करोड़ तेरह~७०००००१३ +चार करोड़ नौ सौ ग्यारह~४००००९११ +छः करोड़ पाँच हज़ार नौ सौ ग्यारह~६०००५९११ +छः करोड़ पच्चीस हज़ार नौ सौ ग्यारह~६००२५९११ +तीन करोड़ एक लाख पच्चीस हज़ार नौ सौ ग्यारह~३०१२५९११ +दो करोड़ सत्रह लाख पच्चीस हज़ार नौ सौ ग्यारह~२१७२५९११ +तीस करोड़~३०००००००० +अट्ठानवे लाख छिहत्तर हज़ार सात सौ नवासी~९८७६७८९ +तेईस लाख पैंतालीस हज़ार पाँच सौ सड़सठ~२३४५५६७ +एक करोड़ तेईस लाख पैंतालीस हज़ार पाँच सौ सड़सठ~१२३४५५६७ +एक करोड़ इक्कीस लाख इक्कीस हज़ार दो सौ बारह~१२१२१२१२ +एक अरब बारह करोड़ तेईस लाख पैंतालीस हज़ार पाँच सौ सड़सठ~११२२३४५५६७ +एक अरब दो करोड़ तेईस लाख पैंतालीस हज़ार पाँच सौ सड़सठ~१०२२३४५५६७ +ग्यारह अरब दो करोड़ तेईस लाख पैंतालीस हज़ार पाँच सौ सड़सठ~११०२२३४५५६७ +इक्यावन अरब दो करोड़ तेईस लाख पैंतालीस हज़ार पाँच सौ सड़सठ~५१०२२३४५५६७ +सवा सात सौ~७२५ +साढ़े सात सौ~७५० +साढ़े सात हज़ार~७५०० +सवा सात हज़ार~७२५० +डेढ़ सौ~१५० +ढाई सौ~२५० +साढ़े सोलह सौ~१६५० +सवा सोलह सौ~१६२५ diff --git a/tests/fixtures/itn/hi/test_cases_date.txt b/tests/fixtures/itn/hi/test_cases_date.txt new file mode 100644 index 0000000..a375009 --- /dev/null +++ b/tests/fixtures/itn/hi/test_cases_date.txt @@ -0,0 +1,44 @@ +# NeMo ITN output (lang=hi, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +छः मई~६ मई +तीस जून~३० जून +पच्चीस मार्च दो हज़ार दस~२५ मार्च, २०१० +तीस मार्च उन्नीस सौ नब्बे~३० मार्च, १९९० +मार्च तीस उन्नीस सौ नब्बे~मार्च ३०, १९९० +उन्नीस जून दो हज़ार पाँच~१९ जून, २००५ +पन्द्रह जून दो हज़ार उन्नीस~१५ जून, २०१९ +आठ जनवरी~८ जनवरी +अठारह जुलाई~१८ जुलाई +छब्बीस नवंबर~२६ नवंबर +तीन अप्रैल~३ अप्रैल +चार जनवरी~४ जनवरी +एक अक्टूबर~१ अक्टूबर +तेरह सितंबर~१३ सितंबर +मार्च दो हज़ार दस~मार्च २०१० +दस मार्च~१० मार्च +बारह दिसंबर~१२ दिसंबर +दिसंबर बारह~दिसंबर १२ +एक सितंबर~१ सितंबर +तीन फ़रवरी~३ फ़रवरी +सात जून~७ जून +सत्ताईस जुलाई दो हज़ार ग्यारह~२७ जुलाई, २०११ +जुलाई सत्ताईस~जुलाई २७ +वर्ष दो हज़ार उन्नीस~वर्ष २०१९ +सन उन्नीस सौ नब्बे~सन १९९० +उन्नीस सौ नब्बे से उन्नीस सौ इक्यानबे~१९९०-१९९१ +दो हज़ार पाँच से दो हज़ार उन्नीस~२००५-२०१९ +दो हज़ार पाँच से उन्नीस~२००५-१९ +चौंतीस सौ ईसा पूर्व~३४०० ई.पू. +उन्नीस सौ बीस ईस्वी~१९२० ई. +पच्चीस जनवरी अठारह सौ तिरेपन ईसवी~२५ जनवरी, १८५३ ई. +इकत्तीस मई उन्नीस सौ नब्बे ईसवी~३१ मई, १९९० ई. +पच्चीस ईसा पूर्व~२५ ई.पू. +मार्च की दो~मार्च २ +फ़रवरी की बीस~फ़रवरी २० +उन्नीस सौ नब्बे से उन्नीस सौ इक्यानबे ईसवी~१९९०-१९९१ ई. +दो हज़ार पाँच से दो हज़ार उन्नीस ईसा पूर्व~२००५-२०१९ ई.पू. +दसवें शताब्दी~१०वें शताब्दी +अठाहरवीं शताब्दी~१८वीं शताब्दी +एक हज़ार एकवीं शताब्दी~१००१वीं शताब्दी +एक सौ उन्नीसवां शताब्दी~११९वां शताब्दी +उन्नीस सौ बीस से छब्बीस तक~१९२०-२६ तक diff --git a/tests/fixtures/itn/hi/test_cases_decimal.txt b/tests/fixtures/itn/hi/test_cases_decimal.txt new file mode 100644 index 0000000..bd08e31 --- /dev/null +++ b/tests/fixtures/itn/hi/test_cases_decimal.txt @@ -0,0 +1,15 @@ +# NeMo ITN output (lang=hi, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +दो सौ छह दशमलव दो नौ~२०६.२९ +एक सौ एक दशमलव छह~१०१.६ +एक सौ नौ दशमलव आठ~१०९.८ +एक सौ आठ दशमलव सात पाँच~१०८.७५ +एक सौ दस दशमलव सात पाँच~११०.७५ +एक सौ दो दशमलव तीन~१०२.३ +एक सौ छह दशमलव पाँच~१०६.५ +साढ़े तीन सौ दशमलव दो दो~३५०.२२ +सवा तीन सौ दशमलव दो~३२५.२ +साढ़े चार सौ दशमलव सात पाँच~४५०.७५ +सवा चार सौ दशमलव सात पाँच~४२५.७५ +ढाई सौ दशमलव छह~२५०.६ +डेढ़ सौ दशमलव सात पाँच~१५०.७५ diff --git a/tests/fixtures/itn/hi/test_cases_fraction.txt b/tests/fixtures/itn/hi/test_cases_fraction.txt new file mode 100644 index 0000000..f391d38 --- /dev/null +++ b/tests/fixtures/itn/hi/test_cases_fraction.txt @@ -0,0 +1,33 @@ +# NeMo ITN output (lang=hi, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +एक सौ नौ बटा एक सौ चौबीस~१०९/१२४ +एक सौ एक बटा दो~१०१/२ +दो सौ एक बटा दो~२०१/२ +एक सौ एक बटा चार~१०१/४ +दो सौ बटा पाँच सौ~२००/५०० +दो सौ बटा बारह~२००/१२ +एक सौ तेईस बटा एक सौ पच्चीस~१२३/१२५ +छह सौ बासठ बटा एक~६६२/१ +एक सौ पाँच बटा सात~१०५/७ +छह सौ चौवन बटा तीन~६५४/३ +एक सौ तैंतीस सही एक बटा दो~१३३ १/२ +एक सौ तैंतीस सही दो बटा तीन~१३३ २/३ +एक सही छह बटा छह~१ ६/६ +दो सही एक बटा छह~२ १/६ +तीन सही तीन बटा चार~३ ३/४ +एक सौ बीस सही तीन बटा चार~१२० ३/४ +एक सौ बीस सही पिछत्तर बटा नब्बे~१२० ७५/९० +तीन सही तीन बटा चार~३ ३/४ +सवा चौरासी~८४ १/४ +डेढ़~१ १/२ +ढाई~२ १/२ +आधा~१/२ +साढ़े~१/२ +सवा~१/४ +पौन~३/४ +पौना~३/४ +सवा पैंतीस~३५ १/४ +साढ़े चार सौ बटा दस~४५०/१० +तीन चौथाई~३/४ +दो तिहाई~२/३ +एक चौथाई~१/४ diff --git a/tests/fixtures/itn/hi/test_cases_measure.txt b/tests/fixtures/itn/hi/test_cases_measure.txt new file mode 100644 index 0000000..95326de --- /dev/null +++ b/tests/fixtures/itn/hi/test_cases_measure.txt @@ -0,0 +1,50 @@ +# NeMo ITN output (lang=hi, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +दो सौ छह दशमलव दो नौ ग्राम~२०६.२९ g +दो सौ छह ग्राम~२०६ g +इक्कीस दशमलव शून्य सेल्सियस~२१.० °C +इक्कीस सेल्सियस~२१ °C +बारह हज़ार तेरह दशमलव सात सात सात डेसिग्राम~१२०१३.७७७ dg +बारह हज़ार तेरह डेसिग्राम~१२०१३ dg +चार सौ उनतीस दशमलव एक कैल्विन~४२९.१ K +चार सौ उनतीस कैल्विन~४२९ K +बाईस दशमलव शून्य पाँच मिलिग्राम~२२.०५ mg +बाईस मिलिग्राम~२२ mg +नौ हज़ार दशमलव शून्य शून्य मीट्रिक टन~९०००.०० t +पच्चीस दशमलव एक किग्रा~२५.१ kg +पच्चीस किग्रा~२५ kg +बानवे हज़ार तीन सौ तिरानवे दशमलव शून्य शून्य चार मिलीमीटर~९२३९३.००४ mm +बानवे हज़ार तीन सौ तिरानवे मिलीमीटर~९२३९३ mm +सात दशमलव सात इंच~७.७ in +पाँच सौ दशमलव आठ नौ तीन माइक्रॉन~५००.८९३ µm +पाँच सौ माइक्रॉन~५०० µm +पच्चीस सौ दशमलव छः छः फुट~२५००.६६ ft +पच्चीस सौ फुट~२५०० ft +छप्पन हज़ार तीस दशमलव दो वर्गसेंटीमीटर~५६०३०.२ cm² +छप्पन हज़ार तीस वर्ग सेंटीमीटर~५६०३० cm² +छियासठ दशमलव एक एकड़~६६.१ ac +छियासठ एकड़~६६ ac +चौंतीस सौ नौ दशमलव सात पाँच क्यूबिकमिलीमीटर~३४०९.७५ mm³ +छे सौ अठारह दशमलव दो दो लीटर~६१८.२२ L +चार हज़ार दशमलव शून्य शून्य गैलन~४०००.०० gal +चार हज़ार गैलन~४००० gal +तैंतीस दशमलव तीन तीन किलोमीटर प्रति घंटा~३३.३३ km/h +चौदह हज़ार इकहत्तर दशमलव नौ नौ पिंट~१४०७१.९९ pt +बहत्तर दशमलव आठ तीन मील प्रति घंटा~७२.८३ mi/h +बहत्तर मील प्रति घंटा~७२ mi/h +पौने ग्यारह घंटे~१०.७५ h +साढ़े सात वर्ष~७.५ yr +सवा ग्यारह सौ मीटर~११२५ m +पौने चार सौ हेक्टेयर~३७५ ha +साढ़े दस घन फीट~१०.५ ft³ +पौने पांच सौ किलोमीटर~४७५ km +ढाई सौ गैलन~२५० gal +डेढ़ दर्जन~१.५ doz +साढ़े सात ऐंपीयर~७.५ A +पौने तीन हजार एकड़~२७५० ac +साढ़े बारह वर्ग माइक्रोमीटर~१२.५ µm² +ढाई महीने~२.५ mo +दो बाई दो~२x२ +दो बाई दो~२x२ +पाँच बाई पाँच~५x५ +बाईस बाई पाँच घन फीट~२२x५ ft³ diff --git a/tests/fixtures/itn/hi/test_cases_money.txt b/tests/fixtures/itn/hi/test_cases_money.txt new file mode 100644 index 0000000..32c30e1 --- /dev/null +++ b/tests/fixtures/itn/hi/test_cases_money.txt @@ -0,0 +1,52 @@ +# NeMo ITN output (lang=hi, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +तैंतीस अल्जीरियाई दिनार~دج३३ +बारह हज़ार तेरह डॉलर~$१२०१३ +चौदह हज़ार इकहत्तर दशमलव नौ नौ बेलारूसी रूबल~br१४०७१.९९ +छे सौ अठारह चीनी युआन~元६१८ +अट्ठाईस सौ दशमलव शून्य आठ आर्मेनियाई ड्राम~֏२८००.०८ +पच्चीस सौ छः अरूबान फ्लोरिन~ƒ२५०६ +बहत्तर त्रिनिदाद और टोबैगो डॉलर~tt$७२ +छियासठ तुर्की लिरा~₺६६ +चार सौ उनतीस युगांडा शिलिंग~ush४२९ +बाईस दशमलव शून्य पाँच यूक्रेनी ग्रिव्ना~₴२२.०५ +पच्चीस वॉन~₩२५ +छप्पन हज़ार तीस वेनेजुएलन बोलिवार~bs.५६०३० +चौंतीस सौ नौ साइप्रस पाउंड~cyp३४०९ +बानवे हज़ार तीन सौ तिरानवे दशमलव शून्य शून्य चार लिलांगेनी~l९२३९३.००४ +छे सौ अठारह बहरीन दिरहम~.د.ب६१८ +दो सौ छह रुपये दो सौ छह पैसे~₹२०६.२०६ +अड़तीस रुपिया~₹३८ +इक्यानबे सौ रुपेया और दो सौ पैसा~₹९१००.२०० +नौ हज़ार दशमलव शून्य शून्य पैसे~p९०००.०० +चौदह हज़ार इकहत्तर अजरबैजानी मनात~₼१४०७१ +इकहत्तर हज़ार इकहत्तर बिटकॉइन~₿७१०७१ +बत्तीस बुरुंडी फ्रैंक~fbu३२ +पन्द्रह सौ कैमन आइलैंड्स डॉलर~ci$१५०० +छह सौ पच्चीस रुपये दो पैसे~₹६२५.२ +साढ़े सात सौ डॉलर~$७५० +सवा दो सौ यूक्रेनी ग्रिव्ना~₴२२५ +साढ़े छः लाख रुपए~₹६५०००० +सवा छः लाख अल्जीरियाई दिनार~دج६२५००० +सवा पंद्रह लाख युगांडा शिलिंग~ush१५२५००० +साढ़े पंद्रह लाख रुपए~₹१५५०००० +साढ़े पाँच हज़ार लीरा~₺५५०० +ढाई सौ यूरो~€२५० +ढाई हजार बुरुंडी फ्रैंक~fbu२५०० +ढाई करोड़ रुपए~₹२५०००००० +ढाई लाख रुपए~₹२५०००० +डेढ़ सौ यूरो~€१५० +डेढ़ हजार रुपए~₹१५०० +डेढ़ करोड़ रुपए~₹१५०००००० +डेढ़ लाख रुपए~₹१५०००० +पौने तीन सौ रुपए~₹२७५ +पौने पंद्रह सौ रुपए~₹१४७५ +पौने तीन हजार रुपए~₹२७५० +पौने पंद्रह हजार यूरो~€१४७५० +पौने पैंतालिस हजार यूरो~€४४७५० +पौने तीन लाख रुपए~₹२७५००० +पौने पंद्रह लाख रुपए~₹१४७५००० +पौने पैंतालिस लाख रुपए~₹४४७५००० +पौने तीन करोड़ रुपए~₹२७५००००० +पौने पंद्रह करोड़ रुपए~₹१४७५००००० +पौने पैंतालिस करोड़ रुपए~₹४४७५००००० diff --git a/tests/fixtures/itn/hi/test_cases_ordinal.txt b/tests/fixtures/itn/hi/test_cases_ordinal.txt new file mode 100644 index 0000000..4bfa451 --- /dev/null +++ b/tests/fixtures/itn/hi/test_cases_ordinal.txt @@ -0,0 +1,15 @@ +# NeMo ITN output (lang=hi, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +एक हज़ार एकवीं~१००१वीं +सौवां~१००वां +एक सौ एकवां~१०१वां +दसवां~१०वां +दसवीं~१०वीं +दसवें~१०वें +एक सौ उन्नीसवां~११९वां +एक सौ उन्नीसवीं~११९वीं +एक सौ उन्नीसवें~११९वें +अट्ठानवे सौ छब्बीसवीं~९८२६वीं +अट्ठानवेवीं~९८वीं +निन्यानवेवां~९९वां +छे सौ चालीसवीं~६४०वीं diff --git a/tests/fixtures/itn/hi/test_cases_telephone.txt b/tests/fixtures/itn/hi/test_cases_telephone.txt new file mode 100644 index 0000000..5a8643f --- /dev/null +++ b/tests/fixtures/itn/hi/test_cases_telephone.txt @@ -0,0 +1,30 @@ +# NeMo ITN output (lang=hi, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +एक एक एक एक एक एक~११११११ +पाँच शून्य शून्य शून्य एक दो~५०००१२ +एक दो तीन चार पाँच छह~१२३४५६ +चार शून्य शून्य शून्य एक शून्य~४०००१० +सात पाँच शून्य शून्य शून्य दो~७५०००२ +आठ आठ शून्य नौ नौ शून्य~८८०९९० +नौ आठ सात छह पाँच चार तीन दो एक शून्य~९८७६५४३२१० +सात शून्य एक दो तीन चार पाँच छह सात आठ~७०१२३४५६७८ +आठ आठ आठ सात सात सात छह छह छह छह~८८८७७७६६६६ +छह दो नौ शून्य एक पाँच सात तीन चार आठ~६२९०१५७३४८ +नौ नौ आठ आठ सात सात छह छह पाँच पाँच~९९८८७७६६५५ +प्लस नौ एक नौ आठ सात छह पाँच चार तीन दो एक शून्य~+९१ ९८७६५४३२१० +प्लस नौ एक सात शून्य एक दो तीन चार पाँच छह सात आठ~+९१ ७०१२३४५६७८ +प्लस नौ एक आठ आठ आठ सात सात सात छह छह छह छह~+९१ ८८८७७७६६६६ +प्लस नौ एक एक एक एक एक एक एक एक एक एक एक~+९१ ११११११११११ +शून्य दो शून्य दो चार तीन सात एक पाँच चार दो~०२०२४३७१५४२ +शून्य एक एक दो छह एक दो तीन चार पाँच छह~०११२६१२३४५६ +चार चार दो दो आठ आठ छह छह चार चार~४४२२८८६६४४ +शून्य आठ शून्य चार एक दो तीन चार पाँच छह सात~०८०४१२३४५६७ +दो दो छह छह पांच चार तीन दो एक शून्य~२२६६५४३२१० +zero one three three six two three four five six seven~०१३३६२३४५६७ +zero one three four two three two one five four eight~०१३४२३२१५४८ +एक दो तीन चार~१२३४ +पाँच शून्य शून्य नौ~५००९ +चार चार चार चार~४४४४ +सात आठ नौ एक~७८९१ +एक शून्य दो शून्य~१०२० +नौ आठ सात छह~९८७६ diff --git a/tests/fixtures/itn/hi/test_cases_time.txt b/tests/fixtures/itn/hi/test_cases_time.txt new file mode 100644 index 0000000..ce1b24f --- /dev/null +++ b/tests/fixtures/itn/hi/test_cases_time.txt @@ -0,0 +1,27 @@ +# NeMo ITN output (lang=hi, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +एक बजे सात मिनट~१:०७ +दो बजकर ग्यारह मिनट~२:११ +दो बजके इकतालीस मिनट~२:४१ +बारह बजकर चौवन मिनट~१२:५४ +ग्यारह बजे~११:०० +सात बजे~७:०० +चार बजके नौ मिनट~४:०९ +आठ बजकर पैंतालीस मिनट~८:४५ +छः बजके पाँच मिनट~६:०५ +छह बजे~६:०० +बारह पन्द्रह~१२:१५ +दस अठारह~१०:१८ +चार बजे पाँच सेकंड~४:००:०५ +नौ घंटा दो सेकंड~९:००:०२ +सोलह घंटा एक मिनट सत्ताईस सेकंड~१६:०१:२७ +दस बजकर चौवन मिनट आठ सेकंड~१०:५४:०८ +तीन मिनट उन्नीस सेकंड~००:०३:१९ +ढाई बजे~२:३० +डेढ़ बजे~१:३० +डेढ़ घंटा~१:३० +साढ़े पाँच बजे~५:३० +सवा चार बजे~४:१५ +साढ़े ग्यारह~११:३० +पौने पाँच~४:४५ +पौने तीन घंटा~२:४५ diff --git a/tests/fixtures/itn/hi/test_cases_whitelist.txt b/tests/fixtures/itn/hi/test_cases_whitelist.txt new file mode 100644 index 0000000..eddf5c6 --- /dev/null +++ b/tests/fixtures/itn/hi/test_cases_whitelist.txt @@ -0,0 +1,10 @@ +# NeMo ITN output (lang=hi, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +मास्टर निखिल तनिष~मा. निखिल तनिष +पाव~१/४ +श्रीमती ज्योत्सना~स्मि. ज्योत्सना +डॉक्टर~डॉ. +आधा कप चाय~१/२ कप चाय +श्रीमान भारत कुमार~श्री. भारत कुमार +डॉक्टर प्रशांत~डॉ. प्रशांत +कुमारी~कु. diff --git a/tests/fixtures/itn/hi/test_cases_word.txt b/tests/fixtures/itn/hi/test_cases_word.txt new file mode 100644 index 0000000..7e7a833 --- /dev/null +++ b/tests/fixtures/itn/hi/test_cases_word.txt @@ -0,0 +1,17 @@ +# NeMo ITN output (lang=hi, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +नींद~नींद +याहू!~याहू! +-~- +आआआ~आआआ +आकाशगंगा~आकाशगंगा +लटरपटर~लटरपटर +कच्चा-पक्का~कच्चा-पक्का +गुब्बारा~गुब्बारा +चिट्ठी~चिट्ठी +ढूंढना~ढूंढना +लोहे का!~लोहे का! +टाटा~टाटा +~ +झ~झ +संगीत~संगीत diff --git a/tests/fixtures/itn/ja/test_cases_cardinal.txt b/tests/fixtures/itn/ja/test_cases_cardinal.txt new file mode 100644 index 0000000..0908258 --- /dev/null +++ b/tests/fixtures/itn/ja/test_cases_cardinal.txt @@ -0,0 +1,30 @@ +# NeMo ITN output (lang=ja, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +一~1 +百~100 +五千億~500,000,000,000 +五万~50,000 +五兆~5,000,000,000,000 +十一兆~11,000,000,000,000 +十一兆一~11,000,000,000,001 +九十九兆~99,000,000,000,000 +一兆~1,000,000,000,000 +一兆一~1,000,000,000,001 +一兆十~1,000,000,000,010 +一兆百~1,000,000,000,100 +一兆千~1,000,000,001,000 +一兆一万~1,000,000,010,000 +一兆十万~1,000,000,100,000 +一兆百万~1,000,001,000,000 +一兆一千万~1,000,010,000,000 +そこに鳥一羽がいます~そこに鳥1羽がいます +これから百数えてください~これから100数えてください +生産に掛かる費用は五千億になります~生産に掛かる費用は500,000,000,000になります +お年玉五万あげる~お年玉50,000あげる +五兆円分の株式を買った~5,000,000,000,000円分の株式を買った +今年の収益は十一兆になる~今年の収益は11,000,000,000,000になる +隣の会社の年収益は九十九兆だそうだ~隣の会社の年収益は99,000,000,000,000だそうだ +政府は一兆の赤字で困っている~政府は1,000,000,000,000の赤字で困っている +兵士五百人を派遣する~兵士500人を派遣する +お寺に一万寄付した~お寺に10,000寄付した +クラスに二十人いる~クラスに20人いる diff --git a/tests/fixtures/itn/ja/test_cases_date.txt b/tests/fixtures/itn/ja/test_cases_date.txt new file mode 100644 index 0000000..7512388 --- /dev/null +++ b/tests/fixtures/itn/ja/test_cases_date.txt @@ -0,0 +1,33 @@ +# NeMo ITN output (lang=ja, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +一日~1日 +一月~1月 +一月一日~1月1日 +一月二十二日~1月22日 +七十から八十年代~70〜80年代 +七十年代~70年代 +七月~7月 +七月二十三日~7月23日 +八月四日~8月4日 +五から九日~5〜9日 +九月~9月 +三から四月~3〜4月 +三月一日水曜日~3月1日(水) +四月三十日日曜日~4月30日(日) +三月二十日~3月20日 +三月~3月 +九十年代~90年代 +九月~9月 +九月五日~9月5日 +二十一世紀~21世紀 +二十一日月曜日~21日(月) +今日は一月二十二日~今日は1月22日 +毎月の三十日はゴミの日~毎月の30日はゴミの日 +七十年代はロックがはやってた~70年代はロックがはやってた +二十一世紀でやることじゃない~21世紀でやることじゃない +正月は一月一日から始まる~正月は1月1日から始まる +五から九日は休みの日~5〜9日は休みの日 +誕生日は千九百九十九年三月二十日~誕生日は1999年3月20日 +八月と九月はまだ夏~8月と9月はまだ夏 +放送日は三月一日水曜日~放送日は3月1日(水) +十月になるとすずしい~10月になるとすずしい diff --git a/tests/fixtures/itn/ja/test_cases_decimal.txt b/tests/fixtures/itn/ja/test_cases_decimal.txt new file mode 100644 index 0000000..7466513 --- /dev/null +++ b/tests/fixtures/itn/ja/test_cases_decimal.txt @@ -0,0 +1,34 @@ +# NeMo ITN output (lang=ja, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +マイナス一点零六~-1.06 +マイナス七点零零六~-7.006 +マイナス三十九点五七四~-39.574 +マイナス三点八六~-3.86 +マイナス九十二点一五七四~-92.1574 +マイナス九点零三八~-9.038 +マイナス二点八七四一~-2.8741 +マイナス二百三十一点四六零九~-231.4609 +マイナス五十二点一八~-52.18 +マイナス五点三~-5.3 +マイナス五百七十九点三零零二~-579.3002 +マイナス八十六点四~-86.4 +マイナス八点四零九~-8.409 +マイナス八百二十一点七九五四~-821.7954 +マイナス八百五十二点七~-852.7 +マイナス六十一点零七~-61.07 +マイナス六点八一四~-6.814 +マイナス六百五十七点三零二四~-657.3024 +マイナス四十二点六零五~-42.605 +マイナス四百八十九点零五二一~-489.0521 +答えはマイナス一点零六~答えは-1.06 +計算の結果はマイナス七点零零六~計算の結果は-7.006 +マイナス二点八七四はかなり悪いスコア~-2.874はかなり悪いスコア +五点三は平均点~5.3は平均点 +テストの点数は八十六点四~テストの点数は86.4 +マイナス三十九点五七四は低すぎる~-39.574は低すぎる +答えはマイナス一点零六~答えは-1.06 +計算の結果はマイナス八十六点四~計算の結果は-86.4 +マイナス五十二点一八はかなり悪いスコア~-52.18はかなり悪いスコア +六点八一四は平均点~6.814は平均点 +テストの点数は九十二点一五七四~テストの点数は92.1574 +マイナス七点零零六は低すぎる~-7.006は低すぎる diff --git a/tests/fixtures/itn/ja/test_cases_fraction.txt b/tests/fixtures/itn/ja/test_cases_fraction.txt new file mode 100644 index 0000000..85069de --- /dev/null +++ b/tests/fixtures/itn/ja/test_cases_fraction.txt @@ -0,0 +1,36 @@ +# NeMo ITN output (lang=ja, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +マイナス一と四分の三~-1 3/4 +一と四分の三~1 3/4 +マイナス一分の九~-9/1 +マイナス一分の六十~-60/1 +マイナス一分の百二十三~-123/1 +マイナス一荷四分の三~-1 3/4 +マイナス七百二十分の一~-1/720 +マイナス三十二分の三十一~-31/32 +マイナス三百九十七分の四~-4/397 +マイナス三百五十分の一~-1/350 +マイナス九十八分の四百七十一~-471/98 +マイナス二と五分の三~-2 3/5 +マイナス二十分の九~-9/20 +マイナス二十分の二十一~-21/20 +マイナス二十四分の一~-1/24 +マイナス二百二十分の一~-1/220 +マイナス二百五十二分の百四十七~-147/252 +マイナス二百五十六分の一~-1/256 +マイナス二荷五分の三~-2 3/5 +マイナス五分の七~-7/5 +マイナス五分の八~-8/5 +マイナス五分の十四~-14/5 +マイナス五分の百三十二~-132/5 +マイナス八分の五~-5/8 +答えはマイナス八分の五~答えは-5/8 +三分の一の人がその場を離れた~1/3の人がその場を離れた +約二分の一を削る~約1/2を削る +十分の三を削って吟醸をつくる~3/10を削って吟醸をつくる +一人三分の一ぐらい取る~1人1/3ぐらい取る +答えは九分の一~答えは1/9 +三分の二の人がその場を離れた~2/3の人がその場を離れた +約十分の一を削る~約1/10を削る +三分の一を削って吟醸をつくる~1/3を削って吟醸をつくる +一人二分の一とぐらい取る~1人1/2とぐらい取る diff --git a/tests/fixtures/itn/ja/test_cases_ordinal.txt b/tests/fixtures/itn/ja/test_cases_ordinal.txt new file mode 100644 index 0000000..67e742e --- /dev/null +++ b/tests/fixtures/itn/ja/test_cases_ordinal.txt @@ -0,0 +1,67 @@ +# NeMo ITN output (lang=ja, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +一万一番目~10001番目 +一万番目~10000番目 +一番目~1番目 +七十番目~70番目 +七千番目~7000番目 +七番目~7番目 +七百番目~700番目 +三十番目~30番目 +三千三百三十番目~3330番目 +三千番目~3000番目 +三番目~3番目 +三百番目~300番目 +九十番目~90番目 +九千九百九十九番目~9999番目 +九千番目~9000番目 +九番目~9番目 +九百番目~900番目 +二十番目~20番目 +二千二百番目~2200番目 +二千番目~2000番目 +二番目~2番目 +二百番目~200番目 +五十番目~50番目 +五千番目~5000番目 +五番目~5番目 +五百番目~500番目 +八十番目~80番目 +八千番目~8000番目 +第一~第1 +第一万~第10000 +第一万一~第10001 +第一万九千~第19000 +第一万九千八百~第19800 +第七~第7 +第七万~第70000 +第七万二千六~第72006 +第七十~第70 +第七十二~第72 +第七千~第7000 +第七千八十九~第7089 +第七百~第700 +第七百三十~第730 +第七百三十五~第735 +第七百九~第709 +第三~第3 +第三万四~第30004 +第三十~第30 +第三千~第3000 +第三千三百二十二~第3322 +第三千十七~第3017 +第三千四百一~第3401 +第三百~第300 +第九~第9 +第九万~第90000 +第九十~第90 +五番目私の席~5番目私の席 +第七班に任務を任せる~第7班に任務を任せる +この角からまっすぐ行って三番目の交差点で曲がる~この角からまっすぐ行って3番目の交差点で曲がる +田中君は二番目の席~田中君は2番目の席 +トップから数えて第七十二~トップから数えて第72 +八番目私の席~8番目私の席 +第十三班に任務を任せる~第13班に任務を任せる +この角からまっすぐ行って二番目の交差点で曲がる~この角からまっすぐ行って2番目の交差点で曲がる +田中君は五番目の席~田中君は5番目の席 +トップから数えて第89~トップから数えて第89 diff --git a/tests/fixtures/itn/ja/test_cases_time.txt b/tests/fixtures/itn/ja/test_cases_time.txt new file mode 100644 index 0000000..9c31b12 --- /dev/null +++ b/tests/fixtures/itn/ja/test_cases_time.txt @@ -0,0 +1,42 @@ +# NeMo ITN output (lang=ja, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +七時一分~7時1分 +七時四分~7時4分 +九時五十八分~9時58分 +九時十分前~9時10分前 +九時四十分~9時40分 +五時二十六分~5時26分 +六時五十五分~6時55分 +三時~3時 +三時~3時 +正午一分前~正午1分前 +正午十分過ぎ~正午10分過ぎ +九時三十分~9時30分 +七時五十分頃~7時50分頃 +一時~1時 +一時十分~1時10分 +三時~3時 +十七時~17時 +二十時~20時 +二十一時~21時 +二時~2時 +十二時三十分~12時30分 +零時~0時 +零時一分前~0時1分前 +二時~2時 +十二時~12時 +二十時~20時 +二十三時~23時 +二十四時~24時 +零時~0時 +四時~4時 +毎日五時に起きる~毎日5時に起きる +九時四十分の予約になります~9時40分の予約になります +現在の時間は十二時三十分~現在の時間は12時30分 +ちょうど零時になった~ちょうど0時になった +四時で店を閉める~4時で店を閉める +毎日六時に起きる~毎日6時に起きる +十時三十分の予約になります~10時30分の予約になります +現在の時間は十時三分~現在の時間は10時3分 +ちょうど一時になった~ちょうど1時になった +七時で店を閉める~7時で店を閉める diff --git a/tests/fixtures/itn/zh/test_cases_cardinal.txt b/tests/fixtures/itn/zh/test_cases_cardinal.txt new file mode 100644 index 0000000..052127c --- /dev/null +++ b/tests/fixtures/itn/zh/test_cases_cardinal.txt @@ -0,0 +1,132 @@ +# NeMo ITN output (lang=zh, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +一百~100 +一百零一~101 +一百一十一~111 +两百~200 +九百~900 +九百五十~950 +九百五十一~951 +一千~1,000 +一千零一~1,001 +一千一百~1,100 +一千一百零一~1,101 +一千零五十~1,050 +一千一百一十~1,110 +一千一百十~1,110 +一千一百一十一~1,111 +两千~2,000 +九千九百九十九~9,999 +一万一千~11,000 +一万一千一百~11,100 +一万一千一百一十~11,110 +一万一千一百一十一~11,111 +一万零一百~10,100 +一万零一百五十~10,150 +一万零一百五十一~10,151 +一万零一~10,001 +一万零五十~10,050 +一万零五十一~10,051 +一万~1万 +两万~2万 +三万~3万 +四万~4万 +五万~5万 +六万~6万 +七万~7万 +八万~8万 +九万~9万 +十万~10万 +十萬~10萬 +九十万~90万 +九十一万~91万 +九十万五千八百二十五~905,825 +九十一万五千八百二十五~915,825 +十一万~11万 +十万一千一百一十一~101,111 +十万一千一百~101,100 +十万一千~101,000 +十万零一百~100,100 +十万零十~100,010 +十万零一~100,001 +一百万~100万 +一百一十万~110万 +一百一十一万~111万 +两百万~200万 +两百一十万~210万 +两百零一万~201万 +一百一十九万~119万 +一百一十九万九千~1,199,000 +一百一十九万九千九百~1,199,900 +一百一十九万九千九百九十~1,199,990 +一百一十九万九千九百九十九~1,199,999 +一百一十九万零九~1,190,009 +一百一十九万零九百九十一~1,190,991 +一千万~1,000万 +一千一百万~1,100万 +一千一百一十万~1,110万 +一千一百一十一万~1,111万 +一千一百一十一万九千~11,119,000 +一千一百一十一万九千一百~11,119,100 +一千一百一十一万九千一百二十~11,119,120 +一千一百一十一万九千一百二十一~11,119,121 +一千一百一十一万零一~11,110,001 +一千一百一十一万零一十~11,110,010 +一千一百一十一万零一百~11,110,100 +一千零一十万零一百~10,100,100 +一千零一十一万零一百~10,110,100 +一千零一万零一百~10,010,100 +一億~1億 +一億一千萬~1億1,000萬 +一億一千一百萬~1億1,100萬 +一億一千一百一十萬~1億1,110萬 +一億一千一百一十一萬~1億1,111萬 +一億零一百萬~101,000,000 +一億零一百一十萬~101,100,000 +一億零一百一十一萬~101,110,000 +一億零一十萬~100,100,000 +一億零一十一萬~100,110,000 +一億零一萬~1億1萬 +一億零一萬一千~100,011,000 +一億零一萬一千一百~100,011,100 +一億零一萬一千一百一~100,011,101 +一億零一萬一千一百一十一~100,011,111 +一億零一萬一千一百零五~100,011,105 +一億零一萬一千零五~100,011,005 +十億~10億 +十一億~11億 +十一億九千萬~11億9,000萬 +十一億九千一百萬~11億9,100萬 +十一億九千一百一十萬~11億9,110萬 +十一億九千一百一十一萬~11億9,111萬 +十一億零一百一十萬~1,101,100,000 +十一億零一十萬~1,100,100,000 +十一億零一萬~11億1萬 +十一億零十萬~1,100,100,000 +十一億零九千~1,100,009,000 +十一億零九百~1,100,000,900 +十一億零九十~1,100,000,090 +十一億零九~1,100,000,009 +一百億~100億 +一百一十億~110億 +一百一十一億~111億 +一百一十一億九千萬~111億9,000萬 +一百一十一億九千九百萬~111億9,900萬 +一百一十一億九千九百一十萬~111億9,910萬 +一百一十一億九千九百一十一萬~111億9,911萬 +一百一十一億九千九百一十一萬九千~11,199,119,000 +一百一十一億九千九百一十一萬九千九百一十一~11,199,119,911 +一百零一億~101億 +一百零一億零九百萬~10,109,000,000 +一百零一億零九十萬~10,100,900,000 +一百零一億零九萬~101億9萬 +一百零一億零九萬零一百~10,100,090,100 +一千億~1,000億 +一千一百億~1,100億 +一千零五十億~1,050億 +一千零五億~1,005億 +一千億九千萬~1,000億9,000萬 +一千億零九百萬~100,009,000,000 +一千億零九十萬~100,000,900,000 +一千億零九萬~1,000億9萬 +一千億零九十萬零五百~100,000,900,500 diff --git a/tests/fixtures/itn/zh/test_cases_date.txt b/tests/fixtures/itn/zh/test_cases_date.txt new file mode 100644 index 0000000..a125727 --- /dev/null +++ b/tests/fixtures/itn/zh/test_cases_date.txt @@ -0,0 +1,33 @@ +# NeMo ITN output (lang=zh, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +一七九八年五月三十日~1798年5月30日 +五月三十日~5月30日 +一七九八年五月~1798年5月 +八月~8月 +一七九八年~1798年 +十九日~19日 +一九九四年一月二日~1994年1月2日 +一九九五年二月三日~1995年2月3日 +二零零零年三月五日~2000年3月5日 +二零零一年四月六日~2001年4月6日 +公元一七九八年五月三十日~公元1798年5月30日 +公元一八三五年~公元1835年 +公元一八三四年八月~公元1834年8月 +公元一九九四年一月二日~公元1994年1月2日 +公元一九九五年二月三日~公元1995年2月3日 +公元二零零零年三月五日~公元2000年3月5日 +公元二零零一年四月六日~公元2001年4月6日 +公元前一七九八年~公元前1798年 +公元前二八零九年~公元前2809年 +公元前一九九四年一月二日~公元前1994年1月2日 +公元前一九九五年二月三日~公元前1995年2月3日 +公元前二零零零年三月五日~公元前2000年3月5日 +公元前二零零一年四月六日~公元前2001年4月6日 +纪元前一九三四年一月二日~公元前1934年1月2日 +纪元前一九九八年三月三日~公元前1998年3月3日 +纪元前二零零零年三月五日~公元前2000年3月5日 +纪元前二零零一年四月六日~公元前2001年4月6日 +纪元一二三四年一月二日~公元1234年1月2日 +纪元二零五六年二月三日~公元2056年2月3日 +纪元二零零零年三月五日~公元2000年3月5日 +纪元二零零一年四月六日~公元2001年4月6日 diff --git a/tests/fixtures/itn/zh/test_cases_decimal.txt b/tests/fixtures/itn/zh/test_cases_decimal.txt new file mode 100644 index 0000000..8cf9a3f --- /dev/null +++ b/tests/fixtures/itn/zh/test_cases_decimal.txt @@ -0,0 +1,44 @@ +# NeMo ITN output (lang=zh, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +一点零~1.0 +十五点零~15.0 +一百点零~100.0 +一百零一点五~101.5 +一点零五六~1.056 +一点零零五六~1.0056 +一点零零零五六~1.00056 +两百点一~200.1 +三千点五~3,000.5 +四万点六~40,000.6 +一點零零五~1.005 +九十九點零零零五~99.0005 +一百點五七三五~100.5735 +一千五百点零一~1,500.01 +负五万点二四五~-50,000.245 +负十五万点三七九~-150,000.379 +负一点一~-1.1 +负十点五~-10.5 +負十點五~-10.5 +負九十九點九五~-99.95 +負一百五十點一二~-150.12 +負一千五百零九點五一~-1,509.51 +負五萬點三~-50,000.3 +負五點零一~-5.01 +負十點零零一~-10.001 +負十點零零零三~-10.0003 +負一百點零零零零四~-100.00004 +一点一二三四五六七八九~1.123456789 +负五点一零二~-5.102 +负三点一二零三~-3.1203 +负十点一二三零五~-10.12305 +伍拾壹点肆~51.4 +壹佰点叁肆~100.34 +贰拾点伍陆~20.56 +柒拾捌点玖~78.9 +负叁拾壹点肆~-31.4 +负壹佰点叁肆~-100.34 +负贰拾点伍陆~-20.56 +负柒拾点玖~-70.9 +負贰拾点叁肆~-20.34 +負玖点玖~-9.9 +負壹佰贰拾点叁肆~-120.34 diff --git a/tests/fixtures/itn/zh/test_cases_fraction.txt b/tests/fixtures/itn/zh/test_cases_fraction.txt new file mode 100644 index 0000000..13066df --- /dev/null +++ b/tests/fixtures/itn/zh/test_cases_fraction.txt @@ -0,0 +1,22 @@ +# NeMo ITN output (lang=zh, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +五分之一~1/5 +二分之一~1/2 +三分之一~1/3 +十分之一~1/10 +一百分之一~1/100 +一千分之一~1/1000 +五分之二~2/5 +三分之二~2/3 +十分之五~5/10 +一千分之五~5/1000 +三又五分之一~3又1/5 +一又二分之一~1又1/2 +一又三分之一~1又1/3 +三又十分之一~3又1/10 +五十又一百分之一~50又1/100 +三又一千分之五~3又5/1000 +六又十分之五~6又5/10 +八又七分之五~8又5/7 +九又四分之三~9又3/4 +五分之四~4/5 diff --git a/tests/fixtures/itn/zh/test_cases_money.txt b/tests/fixtures/itn/zh/test_cases_money.txt new file mode 100644 index 0000000..f059b33 --- /dev/null +++ b/tests/fixtures/itn/zh/test_cases_money.txt @@ -0,0 +1,51 @@ +# NeMo ITN output (lang=zh, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +一千美元~US$1000 +五千美元~US$5000 +一万美元~US$1万 +一点五万美元~US$1.5万 +五十万美元~US$50万 +一百万美元~US$100万 +一千万美元~US$1000万 +一千元~¥1000 +五千元~¥5000 +一万元~¥1万 +一千五万元~¥1005万 +五十万元~¥50万 +一百万元~¥100万 +一千万元~¥1000万 +一千欧元~€1000 +五千欧元~€5000 +一万欧元~€1万 +一点五万欧元~€1.5万 +五十万欧元~€50万 +一百万欧元~€100万 +一千万欧元~€1000万 +一千英镑~£1000 +五千英镑~£5000 +一万英镑~£1万 +一点五万英镑~£1.5万 +五十万英镑~£50万 +一百万英镑~£100万 +一千万英镑~£1000万 +一千韩元~₩1000 +五千韩元~₩5000 +一万韩元~₩1万 +一点五万韩元~₩1.5万 +五十万韩元~₩50万 +一百万韩元~₩100万 +一千万韩元~₩1000万 +一千印度卢布~₹1000 +五千印度卢布~₹5000 +一万印度卢布~₹1万 +一点五万印度卢布~₹1.5万 +五十万印度卢布~₹50万 +一百万印度卢布~₹100万 +一千万印度卢布~₹1000万 +一千日元~JPY¥1000 +五千日元~JPY¥5000 +一万日元~JPY¥1万 +一点五万日元~JPY¥1.5万 +五十万日元~JPY¥50万 +一百万日元~JPY¥100万 +一千万日元~JPY¥1000万 diff --git a/tests/fixtures/itn/zh/test_cases_ordinal.txt b/tests/fixtures/itn/zh/test_cases_ordinal.txt new file mode 100644 index 0000000..3134858 --- /dev/null +++ b/tests/fixtures/itn/zh/test_cases_ordinal.txt @@ -0,0 +1,59 @@ +# NeMo ITN output (lang=zh, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +# left out, NeMo's output leaks tag syntax: 第一億一千萬 +# left out, NeMo's output leaks tag syntax: 第一億一千一百萬 +# left out, NeMo's output leaks tag syntax: 第一億一千一百一十萬 +# left out, NeMo's output leaks tag syntax: 第一億一千一百一十一萬 +第一百~第100 +第五百~第500 +第兩萬一千一百一十一~第21111 +第一百~第100 +第二百~第200 +第兩千~第2000 +第两万~第2万 +第十万~第10万 +第一百万~第100万 +第一千万~第1000万 +第一亿~第1亿 +第一百零一~第101 +第十亿~第10亿 +第五十万~第50万 +第一百一十一~第111 +第十万一千一百一十一~第101111 +第十万一千一百~第101100 +第十万一千~第101000 +第十万零一百~第100100 +第十万零十~第100010 +第十万零一~第100001 +第一百万~第100万 +第一百一十万~第110万 +第一百一十一万~第111万 +第两百万~第200万 +第两百一十万~第210万 +第两百零一万~第201万 +第一百一十九万~第119万 +第一百一十九万九千~第1199000 +第一百一十九万九千九百~第1199900 +第一百一十九万九千九百九十~第1199990 +第一百一十九万九千九百九十九~第1199999 +第一百一十九万零九~第1190009 +第一百一十九万零九十~第1190090 +第一百一十九万零九十一~第1190091 +第一百一十九万零九百九十一~第1190991 +第一千万~第1000万 +第一千一百万~第1100万 +第一千一百一十万~第1110万 +第一千一百一十一万~第1111万 +第一千一百一十一万九千~第11119000 +第一千一百一十一万九千一百~第11119100 +第一千一百一十一万九千一百二十~第11119120 +第一千一百一十一万九千一百二十一~第11119121 +第一千一百一十一万零一~第11110001 +第一千一百一十一万零一十~第11110010 +第一千一百一十一万零一百~第11110100 +第一千零一十万零一百~第10100100 +第一千零一十一万零一百~第10110100 +第一千零一万零一百~第10010100 +第一億~第1億 +第一億零一百萬~第101000000 +第一億零一百一十萬~第101100000 diff --git a/tests/fixtures/itn/zh/test_cases_time.txt b/tests/fixtures/itn/zh/test_cases_time.txt new file mode 100644 index 0000000..b33d607 --- /dev/null +++ b/tests/fixtures/itn/zh/test_cases_time.txt @@ -0,0 +1,25 @@ +# NeMo ITN output (lang=zh, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +五点五分~05:05 +五点一刻~5点1刻 +两点二刻~2点2刻 +三点三刻~3点3刻 +六点~6点 +五点五分~05:05 +五点半~5点半 +五点一刻~5点1刻 +两点三刻~2点3刻 +三点三刻~3点3刻 +五点五分~05:05 +两点一刻~2点1刻 +三点二刻~3点2刻 +四点~4点 +一点五分十秒~01:05:10 +十三点五分十秒~13:05:10 +十点~10点 +五分钟~5分钟 +五秒钟~5秒钟 +十三点五分~13:05 +十三点零五分~13:05 +五点二十五分~05:25 +十一点三十四分~11:34 diff --git a/tests/fixtures/itn/zh/test_cases_whitelist.txt b/tests/fixtures/itn/zh/test_cases_whitelist.txt new file mode 100644 index 0000000..4de89ca --- /dev/null +++ b/tests/fixtures/itn/zh/test_cases_whitelist.txt @@ -0,0 +1,23 @@ +# NeMo ITN output (lang=zh, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +人力资源~HR +自动取款机~ATM +人力资源~HR +首席执行官~CEO +美国研究生入学考试~GRE +研究生管理专业入学考试~GMAT +全球定位系统~GPS +刷卡机~POS机 +数位多功能光碟~DVD +镭射唱片~CD +通用串行总线~USB +统一资源定位符~URL +虚拟专用网络~VPN +网络互联协议~IP +脱氧核糖核酸~DNA +核糖核酸~RNA +平均学分绩点~GPA +发光二极管~LED +可移植文档格式~PDF +社会性网络服务~SNS +博士~PhD diff --git a/tests/fixtures/itn/zh/test_cases_word.txt b/tests/fixtures/itn/zh/test_cases_word.txt new file mode 100644 index 0000000..589af31 --- /dev/null +++ b/tests/fixtures/itn/zh/test_cases_word.txt @@ -0,0 +1,23 @@ +# NeMo ITN output (lang=zh, input_case=lower_cased, punctuation post-processed), regenerated from NeMo-text-processing +# pinned commit 1f1263579fe57ba7ed783cad3dddee710fcc5064. Format: spoken~written. +你好~你好 +年级~年级 +秘密~秘密 +键盘~键盘 +借口~借口 +学生~学生 +人力~人力 +转移~转移 +徘徊~徘徊 +冤枉~冤枉 +浏览~浏览 +珍藏~珍藏 +患难 ~患难 +湿~湿 +眼眶~眼眶 +遗产~遗产 +流浪~流浪 +信仰~信仰 +戒指~戒指 +义无反顾~义无反顾 +交换~交换 diff --git a/tests/fst_itn_parity.rs b/tests/fst_itn_parity.rs new file mode 100644 index 0000000..a469db6 --- /dev/null +++ b/tests/fst_itn_parity.rs @@ -0,0 +1,74 @@ +//! Byte-exact parity of the FST engine's ITN against NeMo's own output. +//! +//! Fixtures in `tests/fixtures/itn//` hold NeMo's `InverseNormalizer` +//! output (input_case=cased) on NeMo-text-processing's ITN test inputs (Apache-2.0, +//! pinned commit `1f1263579fe57ba7ed783cad3dddee710fcc5064`), regenerated by +//! `scripts/generate_nemo_fixtures.py`, so this runs in CI without a NeMo checkout. +//! +//! Requires the `fst-engine` feature: `cargo test --features fst-engine`. +#![cfg(feature = "fst-engine")] + +mod common; + +use std::fs; +use std::path::Path; +use text_processing_rs::fst; + +/// Assert every `test_cases_*.txt` under `tests/fixtures/itn//` inverse-normalizes +/// byte-for-byte, and that a meaningful number of cases actually ran. +fn assert_lang_parity(lang: &str, min_cases: usize) { + let dir = Path::new(env!("CARGO_MANIFEST_DIR")) + .join("tests/fixtures/itn") + .join(lang); + let mut total = 0; + for entry in fs::read_dir(&dir).expect("fixtures dir") { + let path = entry.unwrap().path(); + let name = path.file_name().unwrap().to_string_lossy().to_string(); + if !name.starts_with("test_cases_") { + continue; + } + total += common::parse_test_file(&path).len(); + common::assert_test_file(&path, |input| { + fst::inverse_normalize_lang(input, lang).expect("supported language") + }); + } + assert!( + total >= min_cases, + "{lang}: expected >= {min_cases} cases, saw only {total}" + ); +} + +#[test] +fn en_itn_matches_nemo() { + assert_lang_parity("en", 1200); +} + +#[test] +fn de_itn_matches_nemo() { + assert_lang_parity("de", 280); +} + +#[test] +fn es_itn_matches_nemo() { + assert_lang_parity("es", 370); +} + +#[test] +fn fr_itn_matches_nemo() { + assert_lang_parity("fr", 300); +} + +#[test] +fn hi_itn_matches_nemo() { + assert_lang_parity("hi", 340); +} + +#[test] +fn ja_itn_matches_nemo() { + assert_lang_parity("ja", 220); +} + +#[test] +fn zh_itn_matches_nemo() { + assert_lang_parity("zh", 380); +} diff --git a/tests/multilang_itn_tests.rs b/tests/multilang_itn_tests.rs index e3142a6..04a53c9 100644 --- a/tests/multilang_itn_tests.rs +++ b/tests/multilang_itn_tests.rs @@ -7,6 +7,9 @@ //! Two tagger architectures are exercised: //! - en/fr/de/es: expression-level taggers scanned with the sliding window. //! - hi/ja/zh: whole-sentence scanning taggers (sentence == single-expression). +// These tests pin the rule-based taggers' output. With `fst-engine` the default +// entry points run NeMo's grammars instead (tests/default_engine.rs). +#![cfg(not(feature = "fst-engine"))] use text_processing_rs::{normalize_sentence_lang, normalize_sentence_with_max_span_lang}; diff --git a/tests/multilang_tn_tests.rs b/tests/multilang_tn_tests.rs index c558348..d6e7a97 100644 --- a/tests/multilang_tn_tests.rs +++ b/tests/multilang_tn_tests.rs @@ -2,6 +2,9 @@ //! //! Tests the language dispatch API: tn_normalize_lang() and tn_normalize_sentence_lang(). //! Ensures all 6 languages are wired up and producing correct output through the public API. +// These tests pin the rule-based taggers' output. With `fst-engine` the default +// entry points run NeMo's grammars instead (tests/default_engine.rs). +#![cfg(not(feature = "fst-engine"))] use std::fs; use std::path::Path; diff --git a/wasm-tests/node-fst-smoke.mjs b/wasm-tests/node-fst-smoke.mjs index 72bc920..2327cdd 100644 --- a/wasm-tests/node-fst-smoke.mjs +++ b/wasm-tests/node-fst-smoke.mjs @@ -1,4 +1,4 @@ -import * as wasm from '../pkg-node-fst/text_processing_rs.js'; +import * as wasm from '../pkg-node/text_processing_rs.js'; function assertEqual(actual, expected, message) { if (actual !== expected) {