Skip to content

fix(tn/en): read a fraction before a measure unit - #117

Open
unaltolga wants to merge 2 commits into
FluidInference:mainfrom
unaltolga:fix-tn-en-fraction-before-unit
Open

unaltolga wants to merge 2 commits into
FluidInference:mainfrom
unaltolga:fix-tn-en-fraction-before-unit

Conversation

@unaltolga

@unaltolga unaltolga commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

The measure tagger only accepted a whole or decimal number before the unit. A fraction before a unit was left as written by tn_normalize, and in sentence mode the fraction tagger read the number and the unit stayed as written:

input NeMo main (5ec9959)
4 1/2 lbs (NeMo en TN measure) four and a half pounds 4 1/2 lbs (tn_normalize), four and a half lbs (tn_normalize_sentence)

NeMo reads a fraction before a unit with the unit in the plural (subgraph_fraction with unit_plural in taggers/measure.py). The measure tagger now does the same when the number before the unit is a fraction, using the fraction tagger: 4 1/2 lbs -> four and a half pounds, 1/2 kg -> one half kilograms.

This only changes the rule-based path (crate builds without fst-engine, and set_rule_based_engine(true)). With fst-engine the default path from #116 already reads 4 1/2 lbs as four and a half pounds, so after this change both engines give the same output for it.

In sentence mode, It weighs 4 1/2 lbs. now also keeps its final period (It weighs four and a half pounds.). On main the period is dropped.

Tests

  • tests/data/en/tn_measure.txt listed this case as omitted; the header no longer does, and 4 1/2 lbs~four and a half pounds is added. test_tn_measure fails on main ('4 1/2 lbs' => '4 1/2 lbs') and passes with the change.
  • test_fraction_before_unit in src/tn/en/measure.rs checks parse.
  • A full-width digit after the slash (1/2) panicked in strip_ordinal_suffix, already on main for tn_normalize("1/2") and in sentence mode; with this change tn_normalize("1/2 kg") reached it too (found by the Devin review). The split now checks for a character boundary and these inputs come back unchanged. test_non_ascii_denominator in src/tn/en/fraction.rs, a line in test_fraction_before_unit and test_non_ascii_fraction_denominator_unchanged in tests/en_tn_tests.rs cover it; they panic without the fix.

Checks

  • cargo test passes, also with --features ffi and --release, and the fst-engine jobs from CI (fst_parity, fst_alignment, fst_itn_parity, default_engine --lib) pass.
  • The NeMo parity check reports no regressions; en tn measure goes from 14/21 to 15/21, and tests/parity_baseline.tsv is updated for that line.
  • cargo fmt --all --check passes, and clippy reports the same warnings as main.

AI disclosure: written with Claude Code (Claude Opus 5.5), which found the issue, wrote the change, the tests and this description. I reviewed it and ran the checks on my Mac.


Devin Review

The measure tagger only accepted a whole or decimal number before the
unit, so "4 1/2 lbs" was left to the fraction tagger and the unit stayed
as written: "four and a half lbs". NeMo reads a fraction before a unit
with the unit in the plural (subgraph_fraction with unit_plural in its
measure tagger): "four and a half pounds". The NeMo English TN measure
parity goes from 14/21 to 15/21; the fixture line for it is added and
the baseline updated.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Devin Review

Comment thread src/tn/en/measure.rs
Comment on lines +180 to +182
if num_part.contains('/') {
if let Some(fraction) = super::fraction::parse(num_part) {
return Some(format!("{} {}", fraction, unit_info.plural));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Unicode denominators crash measurement normalization

With the rule-based engine, tn_normalize("1/2 kg") now panics instead of preserving the input. Removing the unit exposes an unchecked UTF-8 split in strip_ordinal_suffix.

Learn more

The new call passes the number alone to fraction::parse. For a full-width denominator such as 2, strip_ordinal_suffix attempts split_at(denom.len() - 2) before validating the suffix. That character occupies three UTF-8 bytes, so the split falls inside the character and panics. Previously, measurement parsing rejected this input; fraction parsing of the complete measurement then rejected its apparent mixed-number whole part without reaching the unsafe split. This changes a previously preserved whole expression into a panic.

Example: In a default build, pass 1/2 kg to tn_normalize. The measurement parser removes kg, and fraction parsing panics at byte index one of 2 instead of returning the unchanged input.

Recommended fix: Make strip_ordinal_suffix use checked string slicing, such as get, before testing an ASCII suffix. Return the original denominator when the split is not a character boundary, allowing numeric parsing to return None. Add regression tests for both measurement parsing and whole-expression normalization with this input.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, confirmed. The split in strip_ordinal_suffix already panicked on main for tn_normalize("1/2") and in sentence mode for 1/2 kg; this PR made tn_normalize("1/2 kg") reach it as well. Fixed in c3a048c: the suffix check is skipped when the split isn't on a character boundary, so these inputs come back unchanged. Tests in fraction.rs, measure.rs and tests/en_tn_tests.rs (whole input and sentence mode) panic without the fix.

strip_ordinal_suffix split the denominator two bytes from the end
without checking for a character boundary, so a full-width digit
("1/2") panicked. Reading a fraction before a unit made "1/2 kg"
reach it in tn_normalize as well. The split is now skipped when it
would fall inside a character, and the input is left unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant