This repository contains code for the ACL 2026 paper On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation.
Instead of global perplexity, we report localized, windowed, normalized perplexity, and generation-based (Model-as-a-judge) alternatives for measuring SALMon performance, which has been shown to correlate more with human perception.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txtIf your environment does not already install the classifier-script imports through NeMo, install the explicit extras as well:
python -m pip install numpy tqdm transformersTo obtain loss arrays and generation results, please refer to the original repos of the SLM models. We used cached arrays stored in the SpeechPPL Hugging Face collection.
Supported model IDs can be passed either by index or by name:
| Index | Model |
|---|---|
| 0 | GSLM |
| 1 | TWIST-350M |
| 2 | TWIST-1.3B |
| 3 | TWIST-7B |
| 4 | pGSLM |
| 5 | Spirit-LM-Base |
| 6 | Spirit-LM-Expressive |
| 7 | Flow-SLM-270M |
| 8 | Flow-SLM-1B |
| 9 | Flow-SLM-1B-Extended |
| 10 | Llama-Mimi1.3B |
| 11 | TASLM |
Run likelihood-based SALMon evaluation for a SLM to test:
python3 salmon_eval.py 10
python3 salmon_eval.py Llama-Mimi1.3BThe default output directory is:
output/model_perplexity_eval_summary/
Run generation-based (Model-as-a-judge) evaluation for one model:
python3 salmon_gold_classifier_eval_generation.py 10
python3 salmon_gold_classifier_eval_generation.py Llama-Mimi1.3BThe default output directory is:
output/model_classifier_eval_summary/
To run the experiments:
bash run_salmon_eval_all.shThe alternative evaluation results below highlight two important conclusions:
- Scores computed with
global-pplcan differ substantially from scores computed with the alternative methods. For example, Llama-Mimi averages 73.5 acrossglobal-ppltasks, compared with 92.1 acrosslocalized-ppltasks, signifincantly closes the gap to human performance. - Model rankings also change under the alternative evaluations.
| Model | Config | Method | Sentiment | Speaker | Gender | Bg domain | Bg rand. | Room | Align sentiment | Align background |
|---|---|---|---|---|---|---|---|---|---|---|
| GSLM | - | Global-PPL | 52.5 | 50.5 | 53.0 | 47.5 | 50.5 | 48.0 | 55.0 | 52.5 |
| + Normalization (prop.) | 48.5 | 52.0 | 52.0 | 45.0 | 52.0 | 50.0 | - | - | ||
| Windowed-PPL (prop.) | 53.2 | 50.5 | 49.8 | 52.8 | 52.0 | 47.8 | 56.0 | 43.0 | ||
| Localized-PPL (prop.) | 52.0 | 49.0 | 48.5 | 47.5 | 46.5 | 42.5 | - | - | ||
| + Normalization (prop.) | 55.5 | 47.0 | 49.0 | 46.5 | 45.5 | 51.0 | - | - | ||
| Model-as-a-judge | 55.0 | 55.0 | 50.5 | 59.5 | 55.5 | 53.0 | - | - | ||
| TWIST | 350M | Global-PPL | 59.0 | 69.5 | 68.0 | 54.0 | 61.5 | 59.0 | 51.0 | 56.5 |
| + Normalization (prop.) | 58.0 | 74.5 | 71.5 | 56.0 | 62.0 | 70.5 | - | - | ||
| Windowed-PPL (prop.) | 59.8 | 57.0 | 55.8 | 52.2 | 64.0 | 65.5 | 51.5 | 58.0 | ||
| Localized-PPL (prop.) | 53.0 | 60.5 | 63.5 | 53.2 | 53.5 | 57.2 | - | - | ||
| + Normalization (prop.) | 55.5 | 63.0 | 60.5 | 56.5 | 61.5 | 66.0 | - | - | ||
| Model-as-a-judge | 48.0 | 53.0 | 52.5 | 61.5 | 53.0 | 52.5 | - | - | ||
| TWIST | 1.3B | Global-PPL | 61.0 | 69.0 | 69.5 | 54.5 | 60.5 | 59.5 | 52.5 | 57.0 |
| + Normalization (prop.) | 54.0 | 72.5 | 73.0 | 57.5 | 63.5 | 69.5 | - | - | ||
| Windowed-PPL (prop.) | 58.2 | 55.2 | 55.0 | 55.2 | 61.8 | 66.0 | 54.0 | 55.0 | ||
| Localized-PPL (prop.) | 50.0 | 61.5 | 68.0 | 54.8 | 53.0 | 54.2 | - | - | ||
| + Normalization (prop.) | 52.5 | 65.5 | 62.0 | 60.5 | 62.5 | 62.0 | - | - | ||
| Model-as-a-judge | 48.0 | 55.0 | 54.5 | 62.5 | 54.0 | 53.5 | - | - | ||
| TWIST | 7B | Global-PPL | 61.5 | 70.5 | 70.0 | 55.5 | 59.5 | 61.5 | 51.0 | 53.5 |
| + Normalization (prop.) | 56.0 | 74.0 | 72.0 | 59.0 | 64.5 | 69.0 | - | - | ||
| Windowed-PPL (prop.) | 60.0 | 57.8 | 53.8 | 55.8 | 59.2 | 69.8 | 51.5 | 56.5 | ||
| Localized-PPL (prop.) | 53.0 | 55.5 | 65.0 | 58.8 | 56.0 | 57.8 | - | - | ||
| + Normalization (prop.) | 53.0 | 67.0 | 63.5 | 58.0 | 65.0 | 65.5 | - | - | ||
| Model-as-a-judge | 48.0 | 54.5 | 52.0 | 59.5 | 54.0 | 51.5 | - | - | ||
| pGSLM | - | Global-PPL | 56.0 | 74.0 | 69.0 | 65.5 | 71.5 | 52.0 | 53.0 | 54.0 |
| + Normalization (prop.) | 60.5 | 80.0 | 89.0 | 62.5 | 71.0 | 69.0 | - | - | ||
| Windowed-PPL (prop.) | 59.0 | 61.8 | 63.2 | 64.5 | 61.5 | 63.5 | 49.0 | 52.5 | ||
| Localized-PPL (prop.) | 70.5 | 78.5 | 68.0 | 50.5 | 57.5 | 56.0 | - | - | ||
| + Normalization (prop.) | 66.0 | 82.5 | 87.5 | 57.0 | 66.5 | 68.5 | - | - | ||
| Model-as-a-judge | 46.5 | 49.5 | 55.5 | 51.0 | 54.0 | 52.5 | - | - | ||
| Spirit LM | base | Global-PPL | 52.5 | 67.5 | 67.0 | 53.0 | 55.5 | 54.5 | 47.5 | 51.0 |
| + Normalization (prop.) | 51.5 | 60.0 | 67.0 | 49.0 | 52.0 | 64.5 | - | - | ||
| Windowed-PPL (prop.) | 48.8 | 50.8 | 58.5 | 53.8 | 56.0 | 68.5 | 46.0 | 60.8 | ||
| Localized-PPL (prop.) | 54.5 | 62.0 | 64.0 | 56.0 | 48.5 | 52.8 | - | - | ||
| + Normalization (prop.) | 54.0 | 57.0 | 59.5 | 55.5 | 57.0 | 52.0 | - | - | ||
| Model-as-a-judge | 47.5 | 52.0 | 50.5 | 50.5 | 52.5 | 50.0 | - | - | ||
| Spirit LM | Expr. | Global-PPL | 71.0 | 81.5 | 85.5 | 55.5 | 64.0 | 55.5 | 53.0 | 59.5 |
| + Normalization (prop.) | 73.5 | 80.0 | 87.0 | 53.5 | 56.5 | 67.5 | - | - | ||
| Windowed-PPL (prop.) | 58.5 | 66.0 | 67.5 | 50.0 | 57.5 | 64.8 | 53.0 | 55.0 | ||
| Localized-PPL (prop.) | 73.5 | 80.5 | 84.5 | 54.5 | 55.0 | 58.5 | - | - | ||
| + Normalization (prop.) | 66.5 | 75.5 | 81.0 | 55.5 | 57.0 | 60.5 | - | - | ||
| Model-as-a-judge | 72.0 | 54.0 | 56.0 | 58.5 | 55.5 | 54.5 | - | - | ||
| TASTE | emb | Global-PPL | 56.5 | 69.5 | 75.5 | 39.5 | 48.5 | 55.5 | 54.0 | 42.0 |
| + Normalization (prop.) | 62.0 | 67.0 | 68.0 | 49.0 | 49.0 | 55.0 | - | - | ||
| Windowed-PPL (prop.) | 57.2 | 52.2 | 57.5 | 48.2 | 49.5 | 51.0 | 51.5 | 48.0 | ||
| Localized-PPL (prop.) | 54.5 | 57.0 | 60.5 | 44.5 | 43.8 | 54.0 | - | - | ||
| + Normalization (prop.) | 55.5 | 52.0 | 55.5 | 47.0 | 48.5 | 53.5 | - | - | ||
| Model-as-a-judge | 95.0 | 100.0 | 100.0 | 61.5 | 54.5 | 57.5 | - | - | ||
| Flow-SLM | 270M | Global-PPL | 61.5 | 75.0 | 76.5 | 66.0 | 65.0 | 56.5 | 59.0 | 53.5 |
| + Normalization (prop.) | 77.5 | 86.0 | 82.5 | 56.0 | 63.5 | 80.5 | - | - | ||
| Windowed-PPL (prop.) | 65.5 | 68.2 | 71.8 | 55.5 | 60.8 | 66.5 | 54.5 | 55.0 | ||
| Localized-PPL (prop.) | 75.0 | 72.0 | 77.5 | 62.5 | 62.0 | 67.5 | - | - | ||
| + Normalization (prop.) | 75.5 | 68.5 | 77.0 | 54.5 | 63.0 | 78.0 | - | - | ||
| Model-as-a-judge | 85.0 | 87.0 | 92.5 | 77.0 | 86.0 | 69.5 | - | - | ||
| Flow-SLM | 1B | Global-PPL | 65.5 | 77.0 | 79.0 | 69.0 | 66.0 | 76.0 | 58.5 | 54.5 |
| + Normalization (prop.) | 78.0 | 85.0 | 87.5 | 58.0 | 69.0 | 88.5 | - | - | ||
| Windowed-PPL (prop.) | 67.8 | 66.8 | 75.2 | 59.0 | 61.2 | 77.2 | 58.0 | 53.5 | ||
| Localized-PPL (prop.) | 79.5 | 73.5 | 79.0 | 64.5 | 68.0 | 81.5 | - | - | ||
| + Normalization (prop.) | 80.5 | 69.0 | 80.0 | 53.0 | 63.0 | 78.5 | - | - | ||
| Model-as-a-judge | 92.5 | 99.5 | 99.0 | 77.0 | 84.5 | 77.0 | - | - | ||
| Flow-SLM | 1B-ext. | Global-PPL | 64.5 | 76.0 | 80.5 | 69.5 | 64.5 | 73.5 | 57.5 | 53.5 |
| + Normalization (prop.) | 82.5 | 85.0 | 88.5 | 58.5 | 73.5 | 85.5 | - | - | ||
| Windowed-PPL (prop.) | 66.5 | 71.2 | 73.0 | 61.5 | 61.8 | 77.0 | 52.0 | 57.0 | ||
| Localized-PPL (prop.) | 79.5 | 76.0 | 81.5 | 65.5 | 69.5 | 83.5 | - | - | ||
| + Normalization (prop.) | 82.0 | 74.5 | 81.0 | 54.0 | 67.5 | 81.0 | - | - | ||
| Model-as-a-judge | 93.0 | 98.0 | 100.0 | 77.0 | 86.5 | 85.0 | - | - | ||
| Llama-mimi | 1.3B | Global-PPL | 79.5 | 85.5 | 82.0 | 75.0 | 72.0 | 92.0 | 53.0 | 49.0 |
| + Normalization (prop.) | 89.0 | 95.0 | 99.5 | 73.5 | 87.0 | 98.5 | - | - | ||
| Windowed-PPL (prop.) | 80.5 | 91.2 | 96.2 | 65.2 | 66.5 | 81.2 | 51.0 | 53.5 | ||
| Localized-PPL (prop.) | 95.5 | 95.5 | 100.0 | 79.5 | 84.5 | 97.5 | - | - | ||
| + Normalization (prop.) | 92.0 | 98.0 | 99.0 | 81.0 | 85.0 | 97.0 | - | - | ||
| Model-as-a-judge | 93.0 | 90.0 | 95.0 | 81.0 | 90.0 | 78.5 | - | - | ||
| Human | - | Human topline | 97.2 | 91.5 | 98.6 | 83.1 | 88.7 | 94.4 | 93.3 | 95.7 |
If you like our work, please cite:
@inproceedings{hsu-etal-2026-fallacy,
title = "On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation",
author = "Hsu, Chan-Jan and
Tseng, Liang-Hsuan and
Lin, Yi-Cheng and
Kuo, Yen-Chun and
Chou, Ju-Chieh and
Chang, Kai-Wei and
Lee, Hung-yi and
Busso, Carlos",
editor = "Liakata, Maria and
Moreira, Viviane P. and
Zhang, Jiajun and
Jurgens, David",
booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
month = jul,
year = "2026",
address = "San Diego, California, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.findings-acl.1844/",
pages = "37000--37020",
ISBN = "979-8-89176-395-1",
abstract = "Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content while preserving attributes like speaker and emotion, serving as foundation models for spoken dialogue. In prior literature, these models are often evaluated using ``global token perplexity'', which directly applies the text perplexity formulation to speech tokens. However, this practice overlooks fundamental differences between speech and text modalities, possibly leading to an underestimation of the speech characteristics. In this work, we propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity. We demonstrate that the proposed evaluations more faithfully reflect perceived generation quality, as evidenced by stronger correlations with human-rated mean opinion scores (MOS). When assessed under the new metrics, the relative performance landscape of spoken language models is reshaped, revealing a significantly reduced gap between the best-performing model and the human topline. Together, these results suggest that appropriate evaluation is critical for accurately assessing progress in spoken language modeling."
}