Repository navigation
Conversation
The measure tagger only accepted a whole or decimal number before the unit, so "4 1/2 lbs" was left to the fraction tagger and the unit stayed as written: "four and a half lbs". NeMo reads a fraction before a unit with the unit in the plural (subgraph_fraction with unit_plural in its measure tagger): "four and a half pounds". The NeMo English TN measure parity goes from 14/21 to 15/21; the fixture line for it is added and the baseline updated.
| if num_part.contains('/') { | ||
| if let Some(fraction) = super::fraction::parse(num_part) { | ||
| return Some(format!("{} {}", fraction, unit_info.plural)); |
There was a problem hiding this comment.
🔴 Unicode denominators crash measurement normalization
With the rule-based engine, tn_normalize("1/2 kg") now panics instead of preserving the input. Removing the unit exposes an unchecked UTF-8 split in strip_ordinal_suffix.
Learn more
The new call passes the number alone to fraction::parse. For a full-width denominator such as 2, strip_ordinal_suffix attempts split_at(denom.len() - 2) before validating the suffix. That character occupies three UTF-8 bytes, so the split falls inside the character and panics. Previously, measurement parsing rejected this input; fraction parsing of the complete measurement then rejected its apparent mixed-number whole part without reaching the unsafe split. This changes a previously preserved whole expression into a panic.
Example: In a default build, pass 1/2 kg to tn_normalize. The measurement parser removes kg, and fraction parsing panics at byte index one of 2 instead of returning the unchanged input.
Recommended fix: Make strip_ordinal_suffix use checked string slicing, such as get, before testing an ASCII suffix. Return the original denominator when the split is not a character boundary, allowing numeric parsing to return None. Add regression tests for both measurement parsing and whole-expression normalization with this input.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Thanks, confirmed. The split in strip_ordinal_suffix already panicked on main for tn_normalize("1/2") and in sentence mode for 1/2 kg; this PR made tn_normalize("1/2 kg") reach it as well. Fixed in c3a048c: the suffix check is skipped when the split isn't on a character boundary, so these inputs come back unchanged. Tests in fraction.rs, measure.rs and tests/en_tn_tests.rs (whole input and sentence mode) panic without the fix.
strip_ordinal_suffix split the denominator two bytes from the end
without checking for a character boundary, so a full-width digit
("1/2") panicked. Reading a fraction before a unit made "1/2 kg"
reach it in tn_normalize as well. The split is now skipped when it
would fall inside a character, and the input is left unchanged.
The measure tagger only accepted a whole or decimal number before the unit. A fraction before a unit was left as written by
tn_normalize, and in sentence mode the fraction tagger read the number and the unit stayed as written:4 1/2 lbs(NeMoenTN measure)four and a half pounds4 1/2 lbs(tn_normalize),four and a half lbs(tn_normalize_sentence)NeMo reads a fraction before a unit with the unit in the plural (
subgraph_fractionwithunit_pluralintaggers/measure.py). The measure tagger now does the same when the number before the unit is a fraction, using the fraction tagger:4 1/2 lbs->four and a half pounds,1/2 kg->one half kilograms.This only changes the rule-based path (crate builds without
fst-engine, andset_rule_based_engine(true)). Withfst-enginethe default path from #116 already reads4 1/2 lbsasfour and a half pounds, so after this change both engines give the same output for it.In sentence mode,
It weighs 4 1/2 lbs.now also keeps its final period (It weighs four and a half pounds.). On main the period is dropped.Tests
tests/data/en/tn_measure.txtlisted this case as omitted; the header no longer does, and4 1/2 lbs~four and a half poundsis added.test_tn_measurefails on main ('4 1/2 lbs' => '4 1/2 lbs') and passes with the change.test_fraction_before_unitinsrc/tn/en/measure.rschecksparse.1/2) panicked instrip_ordinal_suffix, already on main fortn_normalize("1/2")and in sentence mode; with this changetn_normalize("1/2 kg")reached it too (found by the Devin review). The split now checks for a character boundary and these inputs come back unchanged.test_non_ascii_denominatorinsrc/tn/en/fraction.rs, a line intest_fraction_before_unitandtest_non_ascii_fraction_denominator_unchangedintests/en_tn_tests.rscover it; they panic without the fix.Checks
cargo testpasses, also with--features ffiand--release, and thefst-enginejobs from CI (fst_parity,fst_alignment,fst_itn_parity,default_engine --lib) pass.en tn measuregoes from 14/21 to 15/21, andtests/parity_baseline.tsvis updated for that line.cargo fmt --all --checkpasses, and clippy reports the same warnings as main.AI disclosure: written with Claude Code (Claude Opus 5.5), which found the issue, wrote the change, the tests and this description. I reviewed it and ran the checks on my Mac.