NVVSpeech Challenge

Background and Motivation

Human spoken communication extends beyond linguistic content. In natural conversations, speakers frequently produce nonverbal vocalizations (NVVs), such as laughter, sighs, crying, coughing, breathing, gasps, yawns, and other paralinguistic sounds. These vocalizations convey emotion, attitude, feedback, turn-taking cues, social intent, and, in some cases, physiological states. However, most current spoken language processing systems remain largely transcript-centric: automatic speech recognition systems often omit or normalize nonverbal vocalizations, while text-to-speech systems usually focus on generating fluent linguistic content without reliably modeling or controlling such expressive vocal events.

Recent progress in speech language models, expressive speech synthesis, and multimodal conversational agents makes it timely to revisit speech with nonverbal vocalizations as a core spoken language processing problem. For understanding, systems need to detect, classify, and locate nonverbal vocalizations relative to the accompanying transcript. For generation, systems need to synthesize speech that naturally integrates specified nonverbal vocalizations while preserving speech intelligibility, naturalness, and acoustic continuity between verbal and nonverbal segments.

NVVSpeech challenge aims to promote nonverbal vocalizations as a crucial yet under-explored component of spoken language communication. It defines shared task settings for both Mandarin Chinese and English, provides held-out test sets, baseline implementations, and standardized evaluation protocols, and enables the community to examine the current capability boundaries of speech understanding and generation systems with nonverbal vocalizations. NVVSpeech will help identify remaining technical bottlenecks and establish more reproducible comparisons across systems, while encouraging research on paralinguistic speech understanding, expressive and controllable speech generation, speech language models, and evaluation methods for more human-like spoken interaction.

Challenge Overview

The challenge contains two tracks. Both tracks will be evaluated on Mandarin Chinese and English test sets. The evaluation design is adapted from NVV-SuperBench. Further details of the evaluation framework are available in the paper. Code, evaluation scripts, and related resources are available on GitHub.

{
  "utt_id": "example_001",
  "text_with_nvvs": "That was really surprising. [laugh]"
}

{
  "utt_id": "example_002",
  "text_with_nvvs": "That was really surprising. [laugh]"
}

Challenge Rules

1. Open-resource setting. The organizers will provide the evaluation inputs and task formats, but not an official training set. Participants may use publicly available datasets, pretrained models, and other resources that are legally permitted for academic research.
2. External resources and disclosure. All external datasets, pretrained models, synthetic data, data augmentation methods, external tools, and APIs used in a submission must be clearly disclosed in the system description paper. Participants are responsible for complying with the licenses and terms of use of all external resources.
3. Evaluation data and hidden labels. Participants may download and use the officially released evaluation inputs. The corresponding reference labels and evaluation answers will remain hidden. Participants must not obtain or use any hidden test annotations, reference outputs, or other undisclosed evaluation information through data leakage, unauthorized access, or other improper means.
4. No manual test-set intervention. All submitted predictions and generated waveforms must be produced automatically by the submitted system. Manual, sample-by-sample annotation, correction, selection, or modification of test predictions or evaluation outputs is prohibited.
5. Track 2 generation requirements. Each submitted waveform must be fully generated by the submitted system from the provided test input. Manual waveform editing, manually selected sample-level splicing, or test-instance-specific recording is prohibited. Automatic post-processing that is part of the submitted pipeline is allowed but must be disclosed.
6. Reproducibility verification. Teams provisionally ranked among the top submissions may be required to provide inference scripts, model configurations, and environment specifications for verification. Teams that fail to provide the requested materials within the specified deadline may be removed from the official final ranking.
7. System description paper. A system description paper must be submitted by the stated deadline for a team to be eligible for the official final ranking, awards, and presentation in the challenge session. Teams without a submitted paper may appear on the public leaderboard but will be treated as unofficial.

Leaderboard

The leaderboard will open after the evaluation set is released on 22 July 2026. Official rankings for Track 1 and Track 2 will be published here following the evaluation protocol described above.

Contact

If you have any questions, feel free to contact us:

Email: nvvspeech_iscslp2026@googlegroups.com

Show WeChat group QR code

Citation

The evaluation framework behind this challenge is described in the NVV-SuperBench paper. If you use this challenge or its evaluation protocol in your work, please cite:

@article{xue2026nvvsuperbench,
  title = {{NVV-SuperBench}: Beyond Words, Beyond Quality---Benchmarking Nonverbal Vocalizations in Speech Generation},
  author = {Xue, Liumeng and Bian, Weizhen and Pan, Jiahao and Wu, Wenxuan and Ren, Yilin and Kang, Boyi and Hu, Jingbin and Ma, Ziyang and Wang, Shuai and Qian, Xinyuan and Lee, Hung-yi and Guo, Yike},
  journal = {arXiv preprint arXiv:2604.16211},
  year = {2026}
}