Skip to content

funasr_onnx / funasr_torch: spelled-out acronyms come back as separate words with one timestamp, so preds and timestamp no longer line up #3868

Closed

Description

@tanishqbaweja

In runtime/python/onnxruntime/funasr_onnx/utils/postprocess_utils.py and runtime/python/libtorch/funasr_torch/utils/postprocess_utils.py, abbr_dispose gives a run of single-letter tokens (an acronym such as AI, IBM, CCTV) one timestamp, but appends every letter as its own word. funasr/utils/postprocess_utils.py builds one word for the acronym (#270, 2ab8b7f, 2023); that change never reached the runtime copies.

With a timestamp export (4 outputs), preds then has one more word than timestamp for each extra letter. Pairing preds.split() with timestamp gives every word after the acronym the next word's time, and the last word gets none.

Reproduction on main (7651777), no model needed. Run from the repository root; the files are loaded the same way as in tests/test_sentence_postprocess_timestamps.py:

import importlib.util

def load(path):
    spec = importlib.util.spec_from_file_location(path.replace("/", "_")[:-3], path)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module.sentence_postprocess

tokens = ["我", "用", "a", "i", "写", "代", "码"]
timestamps = [[i * 100, (i + 1) * 100] for i in range(len(tokens))]
for path in (
    "funasr/utils/postprocess_utils.py",
    "runtime/python/onnxruntime/funasr_onnx/utils/postprocess_utils.py",
    "runtime/python/libtorch/funasr_torch/utils/postprocess_utils.py",
):
    text, stamps, words = load(path)(tokens, timestamps)
    print(path, repr(text), len(words), "words", len(stamps), "timestamps")

Output:

funasr/utils/postprocess_utils.py '我 用 AI 写 代 码' 6 words 6 timestamps
runtime/python/onnxruntime/funasr_onnx/utils/postprocess_utils.py '我 用 A I 写 代 码' 7 words 6 timestamps
runtime/python/libtorch/funasr_torch/utils/postprocess_utils.py '我 用 A I 写 代 码' 7 words 6 timestamps

Other inputs:

  • ["i", "b", "m", "is", "here"]: IBM is here in-tree, I B M is here (5 words, 3 timestamps) in both runtimes.
  • ["c", "c", "t", "v"]: CCTV in-tree, C C T V (4 words, 1 timestamp) in both runtimes.
  • Without timestamps the text is the same, but the runtime word list is still split: ('我用AI写代码', ['我', '用', 'A', 'I', '写', '代', '码']).

Through the real funasr_onnx Paraformer.__call__, with only the ONNX session replaced by BiCif-shaped outputs for these tokens (2 s input), the result is preds='我 用 A I 写 代 码' with 6 timestamps: I gets 写's span, 写 gets 代's, and 码 gets none. funasr_torch Paraformer gives the same result. I did not run a real model.

Expected: one word per acronym and one timestamp per word, as funasr/utils/postprocess_utils.py returns.

Environment: Windows 11, Python 3.11.17, sources from main.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions