[tanishqbaweja]

Summary

Fixes #3879

runtime/python/websocket/funasr_wss_server.py passed the hotwords field of the config message to the offline model unchanged. The websocket protocol (runtime/docs/websocket_protocol.md) sends it as a JSON object string with weights, {"阿里巴巴":20,"通义实验室":30}, and that is what the HTML5, Java, Android and C++ clients and funasr_wss_client.py in microphone mode send. AutoModel's hotword models read a hotword string as space-separated words: SeACo-Paraformer (paraformer-zh, the server's offline model) and Contextual Paraformer split it on whitespace in generate_hotwords_list. So the JSON text became <unk> hotwords, and the client's hotwords never reached the model as words (#2471, #2514).

This PR adds _model_hotwords to the server. It turns a JSON object string into its words joined with spaces and returns any other value unchanged. ws_serve calls it where it stores the hotwords for the offline and online models (one line). The weights are dropped because AutoModel takes no hotword weights. The C++ server does the same for its NN hotword model (websocket-server.cpp:314-355, websocket-server-2pass.cpp:469-510): it passes the words of the JSON object joined with spaces, and uses a string that is not JSON as it is.

What the offline model's generate receives, and the ids that SeACo-Paraformer's generate_hotwords_list makes from it (toy vocabulary, <unk> = 3, <s> = 1; the script is in the issue):

client sends	main	this PR
{"阿里巴巴":20,"通义实验室":30} (HTML5, Java, Android, C++)	the same string: [[3], [1]]	阿里巴巴 通义实验室: [[4, 5, 6, 6], [7, 8, 9, 10, 11], [1]]
{"阿里巴巴": 20, "通义实验室": 30} (funasr_wss_client.py, microphone)	the same string: [[3], [3], [3], [3], [1]]	阿里巴巴 通义实验室: [[4, 5, 6, 6], [7, 8, 9, 10, 11], [1]]
阿里巴巴 通义实验室	unchanged: [[4, 5, 6, 6], [7, 8, 9, 10, 11], [1]]	unchanged

Type of change

• Bug fix
• Documentation
• Example or demo
• Runtime or deployment
• Benchmark or evaluation
• Model/training change

Validation

The new tests/test_websocket_server_hotwords.py loads the real funasr_wss_server.py with only AutoModel replaced (the server builds its models when it is imported). It sends one offline request through ws_serve (the config message, 0.1 s of PCM, then {"is_speaking": false, "is_end": true}) and checks the hotword that the offline model's generate receives:

• {"阿里巴巴":20,"通义实验室":30} and {"阿里巴巴": 20, "通义实验室": 30} give 阿里巴巴 通义实验室, and {} gives "" (no hotwords);
• 阿里巴巴 通义实验室 and "" (what funasr_wss_client.py sends without --hotword) are passed on unchanged.

Windows 11, CPU, Python 3.11.17, torch 2.10.0+cpu, numpy 2.4.6, scipy 1.17.1, websockets 17.4:

python -m pytest -q tests/test_websocket_server_hotwords.py tests/test_websocket_file_finalization.py

• main (0e52c3c) source with the new test: 3 failed, 30 passed. The three JSON cases fail, for example assert '{"阿里巴巴":20,"通义实验室":30}' == '阿里巴巴 通义实验室'.
• This branch: 33 passed. tests/test_paraformer_hotword_file.py passes as well.

I also served ws_serve over a real socket on 127.0.0.1 with websockets.serve(..., subprotocols=["binary"], ping_interval=None), as main() does, with the models replaced, and sent both JSON forms from a websockets client. The offline model got '阿里巴巴 通义实验室' on this branch and the JSON text on main. Every request ended with the final is_end acknowledgement.

• python -m compileall funasr examples tests
• Docs or links checked
• Runtime/deployment command tested (the server over a local websocket, CPU, models replaced)

black --line-length=100 (24.4.0) leaves the new test unchanged and does not touch the changed lines of funasr_wss_server.py (it would reformat other lines of that file, on main too). No real model was run.

User impact

Users of the Python websocket server who send hotwords from the HTML5, Java, Android or C++ clients, or from funasr_wss_client.py in microphone mode, get those hotwords passed to the offline model as words. Before, the model got the JSON text, which became <unk> hotwords. Space-separated hotword strings and requests without hotwords are not changed.

Notes for reviewers

• The weights are dropped because AutoModel has no hotword weights. The C++ server uses them only for its FST hotwords.
• A hotword with a space in it, such as an English phrase, becomes separate hotwords, because AutoModel splits the joined string on whitespace. The C++ server's NN hotword path does the same.
• No workflow's path filters cover runtime/python/websocket/**, so CI does not run this test (or tests/test_websocket_file_finalization.py). The test needs websockets, scipy and torch, which the server imports, and skips without them. I can add it to a workflow if you want.
• git merge-tree --write-tree of this branch with the head of Fix/security hardening #3877, which also edits this file, gives no conflicts.