Track 1 — Nonverbal Vocalizations Understanding in Speech

Task definition. Given an input speech recording, participants are required to generate a tagged transcript that identifies the spoken content and all nonverbal vocalizations (NVVs), including their categories and transcript-relative positions.

Input. Audio recordings containing natural speech with one, or multiple NVVs. For the evaluation set, only the audio recordings will be released. Reference transcripts and NVV annotations will remain hidden for official evaluation.

Output. Participants submit a structured prediction file for each utterance, for example:

{
  "utt_id": "example_001",
  "text_with_nvvs": "That was really surprising. [laugh]"
}

Evaluation metrics. Track 1 evaluates the ability to jointly recognize NVV categories, predict their transcript-relative positions, and generate accurate tagged transcripts.

The official ranking score is defined as:

$$\text{Track1Score} = 100 \times \big[0.70 \cdot F1_{\text{micro}} + 0.20 \cdot (1-\text{mNTD}) + 0.10 \cdot (1-\min(\text{Err}_{\text{tagged}},1))\big]$$

• Event-based Micro F1 is computed by pooling all NVV events across categories. A predicted event is considered correct only when it has the correct NVV category and transcript-relative position after text normalization and alignment.
• Multi-event Normalized Tag Distance (mNTD) measures the placement error between predicted and reference NVV tags. Lower mNTD indicates more accurate NVV placement.
• Tagged-transcript Error is computed on the complete transcript with NVV tags, where each NVV tag is treated as a single atomic token. For Mandarin Chinese, we report tag-aware character error rate (CER); for English, we report tag-aware word error rate (WER). Lower error rates indicate more accurate recognition of both linguistic content and NVV tags.

The final bilingual ranking score is the average of the Chinese and English scores:

$$\text{FinalTrack1Score} = \frac{\text{Track1Score}_{ZH} + \text{Track1Score}_{EN}}{2}$$

Track 2 — Controllable Speech Generation with Nonverbal Vocalizations

Task definition. Given a transcript containing one or more NVV tags, participants are required to generate speech that naturally integrates the specified NVVs while maintaining intelligibility, naturalness, quality, and appropriate expressive delivery.

Input. Each test sample contains a tagged transcript, for example:

{
  "utt_id": "example_002",
  "text_with_nvvs": "That was really surprising. [laugh]"
}

Output. Participants must submit one generated waveform for each test utterance in 16 kHz, mono, 16-bit PCM WAV format. Each file must be named using the corresponding utt_id (e.g., example_002.wav). Systems with different native sampling rates may resample their outputs to the required format before submission.

Evaluation metrics. Track 2 evaluates whether a system can generate the intended NVVs while maintaining natural, high-quality, and contextually appropriate speech. Evaluation will be conducted using standardized large audio-language model (LALM)-based assessment and human listening tests.

For each language, the official score is defined as:

$$\text{Track2Score} = 100 \times (0.30A + 0.25P + 0.15N + 0.15Q + 0.15E)$$

• NVV Accuracy (A) measures whether the intended NVV types, numbers, and positions are correctly realized. If the target NVV is absent or almost inaudible, the score is 0.
• NVV Perceptual Effect (P) measures whether the intended NVVs are clearly audible, recognizable, and perceptually effective. A score of 0 indicates that the target NVV is absent or almost inaudible.
• Overall Naturalness (N) measures the naturalness, fluency, and acoustic continuity of the full utterance.
• Overall Quality (Q) measures perceived audio quality, including the absence of noticeable artifacts.
• Overall Expression (E) measures whether the expressive delivery is coherent and appropriate for the linguistic content, target NVVs, and conversational context.

The final bilingual ranking score is the average of the Chinese and English scores:

$$\text{FinalTrack2Score} = \frac{\text{Track2Score}_{ZH} + \text{Track2Score}_{EN}}{2}$$

For the public leaderboard, scores will be obtained using a fixed LALM-based multi-rater evaluation protocol. For the final ranking, top-ranked systems will additionally undergo human listening tests. The final scores will combine LALM-based and human ratings according to the official evaluation protocol.

Systems will be ranked by the final Track2Score in descending order. In the event of a tie, higher NVV Accuracy, higher NVV Perceptual Effect, and higher Overall Naturalness will be used sequentially as tie-breaking criteria.