Repository navigation
funasr_onnx / funasr_torch Paraformer: with batch_size > 1, the last timestamp of a shorter file runs past the end of its audio #3870
Description
With a timestamp export (4 outputs: BiCif Paraformer such as paraformer-large-vad-punc, or SeACo Paraformer) and batch_size > 1, Paraformer.__call__ and ContextualParaformer.__call__ in runtime/python/onnxruntime/funasr_onnx/paraformer_bin.py (SeacoParaformer inherits it), and Paraformer.__call__ in runtime/python/libtorch/funasr_torch/paraformer_bin.py, pass each row of us_peaks to time_stamp_lfr6_onnx with the batch padding still attached:
for pred, us_peaks_ in zip(preds, us_peaks):
timestamp, timestamp_raw = time_stamp_lfr6_onnx(us_peaks_, copy.copy(raw_tokens))time_stamp_lfr6_onnx takes the frame count from the array length. The last token ends halfway between its peak and that end, and the end silence runs to it. So the last token of every file shorter than the longest file in its batch ends past the end of its own audio.
The in-tree models cut each row to its own length first: us_peaks[i][: encoder_out_lens[i] * 3] in funasr/models/bicif_paraformer/model.py (:403-404) and funasr/models/seaco_paraformer/model.py (:559-560). The C++ libtorch runtime does the same: paraformer-torch.cpp copies paraformer_length[index]*3 values per row (:435-441). #3554 made the same per-sample trim for the main Paraformer.
Reproduction on main (7651777), no model needed. The wrappers are real; only the model session is replaced by BiCif-shaped outputs (tokens 你 好, three CIF peaks inside each row's own 3 * feats_len frames, rows zero-padded to the longest one, like the exported graph). Save as repro.py, then run from an empty directory:
PYTHONPATH=<FunASR>/runtime/python/onnxruntime python repro.py onnx
PYTHONPATH=<FunASR>/runtime/python/libtorch python repro.py torchimport importlib
import sys
import wave
import numpy as np
backend = sys.argv[1] # "onnx" or "torch"
pkg = importlib.import_module("funasr_" + backend)
utils = importlib.import_module("funasr_" + backend + ".utils.utils")
frontend = importlib.import_module("funasr_" + backend + ".utils.frontend")
paths = []
for seconds in (1, 3):
paths.append(f"tone_{seconds}s.wav")
tone = np.sin(np.arange(16000 * seconds) * 2 * np.pi * 440 / 16000) * 3000
with wave.open(paths[-1], "wb") as out:
out.setnchannels(1)
out.setsampwidth(2)
out.setframerate(16000)
out.writeframes(tone.astype("<i2").tobytes())
def bicif_outputs(feats, feats_len):
# Tokens 你 好; each row's 3 CIF peaks lie inside its own 3 * feats_len frames,
# and the rows are zero-padded to the longest one, like the exported graph.
am = np.zeros((len(feats_len), 3, 8), dtype=np.float32)
am[:, [0, 1, 2], [3, 4, 2]] = 1.0
us_alphas = np.zeros((len(feats_len), 3 * feats.shape[1]), dtype=np.float32)
us_peaks = np.zeros_like(us_alphas)
for row, n in enumerate(np.asarray(feats_len).tolist()):
us_alphas[row, : 3 * n] = 1.0 / n
us_peaks[row, np.linspace(5, 3 * n - 6, 3).round().astype(int)] = 1.0
return am, np.full(len(feats_len), 3, dtype=np.int32), us_alphas, us_peaks
def wrapper(name, batch_size):
cls = getattr(pkg, name)
model = cls.__new__(cls) # skip loading model files
tokens = ["<blank>", "<s>", "</s>", "你", "好", "我", "们", "<unk>"]
model.converter = utils.TokenIDConverter(tokens)
model.frontend = frontend.WavFrontend(cmvn_file=None, lfr_m=7, lfr_n=6, dither=0.0)
model.batch_size, model.pred_bias, model.language = batch_size, 1, None
model.plot_timestamp_to = ""
if backend == "torch":
import torch
model.device = "cpu"
model.ort_infer = lambda f, n: tuple(map(torch.from_numpy, bicif_outputs(f, n.numpy())))
else:
model.infer = model.bb_infer = lambda f, n, *_: bicif_outputs(f, n)
model.proc_hotword = lambda _: (np.zeros((1, 10), np.int32), np.array([0]))
model.eb_infer = lambda *_: [np.zeros((10, 1, 4), np.float32)]
return model
names = ["Paraformer"] + (["ContextualParaformer"] if backend == "onnx" else [])
for name in names:
hotwords = ("",) if name != "Paraformer" else ()
alone = [wrapper(name, 1)(path, *hotwords)[0]["timestamp"] for path in paths]
batched = [row["timestamp"] for row in wrapper(name, 2)(paths, *hotwords)]
print(f"funasr_{backend}.{name}")
print(" batch_size=1:", alone)
print(" batch_size=2:", batched)Output:
funasr_onnx.Paraformer
batch_size=1: [[[70, 470], [470, 945]], [[70, 670], [1450, 2925]]]
batch_size=2: [[[70, 470], [470, 1935]], [[70, 670], [1450, 2925]]]
funasr_onnx.ContextualParaformer
batch_size=1: [[[70, 470], [470, 945]], [[70, 670], [1450, 2925]]]
batch_size=2: [[[70, 470], [470, 1935]], [[70, 670], [1450, 2925]]]
funasr_torch.Paraformer
batch_size=1: [[[70, 470], [470, 945]], [[70, 670], [1450, 2925]]]
batch_size=2: [[[70, 470], [470, 1935]], [[70, 670], [1450, 2925]]]
The 1 s file has 17 LFR frames (51 upsampled), the 3 s file 50 (150). Batched with the 3 s file, the 1 s file's 好 ends at 1935 ms, 935 ms past the end of the 1000 ms file. Alone, it ends at 945 ms. The longest file in the batch is not affected. SeacoParaformer gives the same output as ContextualParaformer. The same happens with the real constructors and only the session replaced (ort_infer, or ort_infer_bb for the hotword classes).
Expected: the same timestamps for every file whether it is decoded alone or in a batch, as funasr/models and the C++ runtime give.
I did not run a real model, and the outputs above are synthetic. From reading get_upsample_timestmap, a real export masks the padded frames before CIF, so the padding of a row holds no peaks there either, only extra frames.
Environment: Windows 11, Python 3.11.17, numpy 2.4.6, torch 2.10.0+cpu, sources from main.
Activity