text-processing-rs

A Rust port of NVIDIA NeMo Text Processing supporting both Inverse Text Normalization (ITN) and Text Normalization (TN).

What it does

ITN: Spoken → Written

Converts spoken-form ASR output to written form:

Input	Output
two hundred thirty two	232
five dollars and fifty cents	$5.50
january fifth twenty twenty five	January 5, 2025
quarter past two pm	02:15 p.m.
one point five billion dollars	$1.5 billion
seventy two degrees fahrenheit	72 °F

TN: Written → Spoken

Converts written-form text to spoken form (useful for TTS preprocessing):

Input	Output
123	one hundred twenty three
$5.50	five dollars and fifty cents
January 5, 2025	january fifth twenty twenty five
2:30 PM	two thirty p m
1st	first
200 km/h	two hundred kilometers per hour

Engines

With the fst-engine feature, every normalize* and tn_normalize* entry point runs NeMo's own compiled grammars, so its output is NeMo's:

• TN matches Normalizer(lang, deterministic=True) byte for byte in en, de, es, fr, hi, it, ja and zh.
• ITN matches InverseNormalizer(lang) (NeMo's default, input_case="lower_cased") byte for byte in en, de, es, fr, hi, ja and zh. NeMo's ITN expects unpunctuated input, so punctuated ASR text then gets the punctuation post-processing NeMo applies after TN (January 5 , 2024 . → January 5, 2024.).
• Custom rules apply ahead of NeMo's grammars.

The Swift XCFramework and the npm package are built with fst-engine. A crate build without it, the ITN options NeMo doesn't have (concat_compound_numbers, disable_bare_second, small_numbers_as_words) and set_rule_based_engine(true) (Swift setRuleBasedEngine, WASM setRuleBasedEngine, C nemo_set_rule_based_engine) use the rule-based taggers instead. The grammars add about 12 MB to the binary (the npm WASM grows from 1.1 to 12.4 MB) and take 1–25 ms per sentence, where the taggers take microseconds. scripts/export_nemo_grammars.py re-exports them from a NeMo checkout; see docs/NEMO_PARITY.md.

Usage

Rust

use text_processing_rs::{normalize, tn_normalize};

// ITN: spoken → written
let result = normalize("two hundred");
assert_eq!(result, "200");

let result = normalize("five dollars and fifty cents");
assert_eq!(result, "$5.50");

// TN: written → spoken
let result = tn_normalize("$5.50");
assert_eq!(result, "five dollars and fifty cents");

let result = tn_normalize("123");
assert_eq!(result, "one hundred twenty three");

JavaScript (WASM)

Build wasm artifacts:

npm run wasm:build:node
npm run wasm:build:web

Node usage:

import * as wasm from "./pkg-node/text_processing_rs.js";

console.log(wasm.normalize("two hundred")); // "200"
console.log(wasm.tnNormalize("$5.50")); // "five dollars and fifty cents"

wasm.addRule("gee pee tee", "GPT");
console.log(wasm.normalize("gee pee tee")); // "GPT"

The packages include the fst-engine grammars, so source-aligned compiled-FST TN (tnFstNormalizeAligned) is available too:

import * as wasm from "./pkg-node/text_processing_rs.js";

const { normalized, spans } = wasm.tnFstNormalizeAligned("The price is $1,234.56.", "en");
// spans[3]: { inputStart: 13, inputEnd: 22, original: "$1,234.56",
//             normalized: "one thousand ... fifty six cents", kind: "money" }
// Offsets are UTF-8 byte offsets; slice new TextEncoder().encode(input).

The generated npm package name is @fluidinference/text-processing-rs.

Web project usage (Vite / Next.js / webpack):

npm install @fluidinference/text-processing-rs

import init, * as wasm from "@fluidinference/text-processing-rs";

async function run() {
  // Loads and initializes the .wasm module (required once at startup)
  await init();

  const itn = wasm.normalize("two hundred");
  const tn = wasm.tnNormalize("$5.50");

  console.log(itn); // "200"
  console.log(tn); // "five dollars and fifty cents"

  wasm.addRule("gee pee tee", "GPT");
  console.log(wasm.normalize("gee pee tee")); // "GPT"
}

run();

If your framework supports top-level await, you can initialize at module load time:

import init, * as wasm from "@fluidinference/text-processing-rs";
await init();

The npm package is a browser (--target web) build, so in Node await init() fails with fetch failed. Hand the .wasm bytes to initSync instead:

import { readFile } from "node:fs/promises";
import { initSync, tnNormalizeSentence } from "@fluidinference/text-processing-rs";

const wasmUrl = new URL(
  "text_processing_rs_bg.wasm",
  import.meta.resolve("@fluidinference/text-processing-rs")
);
initSync({ module: await readFile(wasmUrl) });
console.log(tnNormalizeSentence("I paid $5 for 23 items"));

Sentence-level normalization scans for normalizable spans within a larger sentence:

use text_processing_rs::{normalize_sentence, normalize_sentence_lang, tn_normalize_sentence};

// ITN sentence mode
let result = normalize_sentence("I have twenty one apples");
assert_eq!(result, "I have 21 apples");

// ITN sentence mode, language-aware ("en", "fr", "es", "de", "zh", "hi", "ja")
let result = normalize_sentence_lang("j'ai vingt et un ans", "fr");
assert_eq!(result, "j'ai 21 ans");

// TN sentence mode
let result = tn_normalize_sentence("I paid $5 for 23 items");
assert_eq!(result, "I paid five dollars for twenty three items");

Compiled-FST TN can also retain the source range and semantic class for every normalized span. Offsets are half-open UTF-8 byte offsets:

use text_processing_rs::fst;

let result = fst::normalize_aligned("The price is $1,234.56.", "en").unwrap();
assert_eq!(
    result.normalized,
    "The price is one thousand two hundred and thirty four dollars fifty six cents."
);

let money = &result.spans[3];
assert_eq!(money.input_start, 13);
assert_eq!(money.input_end, 22);
assert_eq!(money.original, "$1,234.56");
assert_eq!(money.kind.as_str(), "money");

Build this API with --features fst-engine. It uses the same compiled NeMo classifier and verbalizer as fst::<lang>::normalize, rather than recovering alignment by diffing the final strings.

Swift

import NemoTextProcessing

// ITN: spoken → written
let result = NemoTextProcessing.normalize("two hundred")
// "200"

// TN: written → spoken
let spoken = NemoTextProcessing.tnNormalize("$5.50")
// "five dollars and fifty cents"

// Sentence modes
let itn = NemoTextProcessing.normalizeSentence("I have twenty one apples")
// "I have 21 apples"

// Language-aware ITN sentence mode ("en", "fr", "es", "de", "zh", "hi", "ja")
let itnFr = NemoTextProcessing.normalizeSentence("j'ai vingt et un ans", language: "fr")
// "j'ai 21 ans"

let tn = NemoTextProcessing.tnNormalizeSentence("I paid $5 for 23 items")
// "I paid five dollars for twenty three items"

if let aligned = NemoTextProcessing.tnNormalizeAligned(
    "The price is $1,234.56.",
    language: "en"
) {
    let money = aligned.spans[3]
    // money.original == "$1,234.56"
    // money.normalized == "one thousand ... dollars fifty six cents"
    // money.inputRange == 13..<22, money.kind == "money"
}

CLI

# ITN
nemo-itn two hundred thirty two        # → 232
nemo-itn -s "I have twenty one apples" # → I have 21 apples

# TN
nemo-tn 123                            # → one hundred twenty three
nemo-tn '$5.50'                        # → five dollars and fifty cents
nemo-tn -s 'I paid $5 for 23 items'    # → I paid five dollars for twenty three items

# Pipe from stdin
echo "2:30 PM" | nemo-tn               # → two thirty p m

Compatibility

With fst-engine, TN and ITN are byte-exact with NeMo (see Engines). The tables below describe the rule-based taggers.

ITN (Spoken → Written)

98.6% compatible with NeMo text processing test suite (1200/1217 tests passing).

Category	Status
Cardinal numbers	100%
Ordinal numbers	100%
Decimal numbers	100%
Money	100%
Measurements	100%
Dates	100%
Time	97%
Electronic (email/URL)	96%
Telephone/IP	96%
Whitelist terms	100%

TN (Written → Spoken)

Category	Examples
Cardinal numbers	123 → one hundred twenty three
Ordinal numbers	1st → first, 21st → twenty first
Decimal numbers	3.14 → three point one four
Money	$5.50 → five dollars and fifty cents
Measurements	200 km/h → two hundred kilometers per hour
Dates	January 5, 2025 → january fifth twenty twenty five
Time	2:30 PM → two thirty p m
Electronic (email/URL)	test@gmail.com → t e s t at g m a i l dot c o m
Telephone	123-456-7890 → one two three, four five six, seven eight nine zero
Whitelist terms	Dr. → doctor, Mr. → mister

Features

• ITN (Inverse Text Normalization): spoken → written form for ASR post-processing
• TN (Text Normalization): written → spoken form for TTS preprocessing
• Cardinal and ordinal number conversion (both directions)
• Decimal numbers with scale words (million, billion)
• Currency formatting (USD, GBP, EUR, JPY, and more)
• Measurements including temperature (°C, °F, K) and data rates (gbps)
• Date parsing (multiple formats) and decade verbalization (1980s → nineteen eighties)
• Time parsing with AM/PM, 24-hour format, and timezone preservation
• Email and URL normalization
• Phone numbers, IP addresses, SSN
• Case preservation for proper nouns and abbreviations
• Sentence-level normalization with sliding window span matching
• Source-to-normalized span alignment with semantic classes (compiled FST)
• Custom rules for domain-specific terms
• C FFI for integration with Swift, Python, and other languages

Building

Rust

cargo build
cargo test

WASM + JavaScript

# Build + smoke test (Node) + build browser artifact, both with fst-engine
npm run wasm:ci

# Create a tarball from the browser package
npm run wasm:pack

# Publish browser package to npm (requires npm auth)
npm run wasm:publish

CLI Tools

# Build the Rust library for this Mac's architecture.
RUST_TARGET="$(rustc -vV | sed -n 's/^host: //p')"
cargo build --release --target "$RUST_TARGET" --features "ffi,fst-engine"

# Build Swift CLI tools
cd swift-test && swift build

Binaries are at swift-test/.build/debug/nemo-itn, swift-test/.build/debug/nemo-tn, and swift-test/.build/debug/nemo-tn-aligned.

nemo-tn-aligned

The aligned CLI emits compact JSON for argument input and JSON Lines for stdin:

swift-test/.build/debug/nemo-tn-aligned --lang en 'The price is $1,234.56.'
# output: {"input":"The price is $1,234.56.","language":"en","normalized":"The price is one thousand two hundred and thirty four dollars fifty six cents.","spans":[{"input_end":3,"input_start":0,"kind":"word","normalized":"The","original":"The"},{"input_end":9,"input_start":4,"kind":"word","normalized":"price","original":"price"},{"input_end":12,"input_start":10,"kind":"word","normalized":"is","original":"is"},{"input_end":22,"input_start":13,"kind":"money","normalized":"one thousand two hundred and thirty four dollars fifty six cents","original":"$1,234.56"},{"input_end":23,"input_start":22,"kind":"punctuation","normalized":".","original":"."}]}

Each result contains input, language, the complete normalized text, and spans with input_start, input_end, original, normalized, and kind. Offsets are half-open UTF-8 byte offsets.

Swift (XCFramework)

# Install Rust targets
rustup target add aarch64-apple-darwin x86_64-apple-darwin
rustup target add aarch64-apple-ios aarch64-apple-ios-sim

# Build XCFramework
./build-xcframework.sh

Output:

• output/NemoTextProcessing.xcframework - Add to Xcode project
• output/NemoTextProcessing.swift - Swift wrapper

License

Apache 2.0

Acknowledgments

This project is a Rust implementation based on the inverse text normalization grammars from NVIDIA NeMo Text Processing. All credit for the original algorithms and test cases goes to the NVIDIA NeMo team.