Nemotron-3.5-Transcribe-kk-v2 — Kazakh speech recognition

A Kazakh (қазақ тілі) fine-tune of NVIDIA Nemotron-3.5-Transcribe-30B-A4B, a speech-language model with a hybrid Mamba–Transformer MoE backbone (~4.3B active parameters). Kazakh is not among the base model's 40 languages: out of the box it emits Turkish/Azeri transliteration or Cyrillic gibberish (FLEURS kk WER

100%). After fine-tuning on ~1,200 h of Kazakh speech it transcribes Kazakh with punctuation and capitalization.

	FLEURS kk test	KSC2 test (full, 9,352 utts)	KSC2 test (stratified 2,019 utts)
Base model	168.7 / 140.9	—	182.1 / 138.7
kk_v1 (128 h)	10.98 / 2.67	—	21.61 / 6.44
kk_v2 (this model, 1,193 h)	8.51 / 2.22	14.34 / 4.60	14.13 / 4.48

WER / CER in %, after normalization (NFC, lowercase, ё→е, punctuation and symbols removed). Greedy decoding.

Comparison with public Kazakh ASR models

Numbers for other models are copied from their model cards and were not re-run by us. Their test protocols differ (split size, normalization, filtering), so read this as orientation, not a leaderboard.

Model	Params	FLEURS kk WER	KSC2 test WER	Notes
This model (kk_v2)	30B (4.3B active)	8.51 (7.48 with the GigaAM filter¹)	14.34 (full)	punctuation + case, word timestamps, diarization branch
ai-sage/GigaAM-Multilingual Large	0.6B CTC	4.4¹	—	best open model on FLEURS kk; lowercase output
InflexionLab/VibeVoice-ASR-Kazakh	—	—	~22 (full)	trained on the same KSC2-Structured data
InflexionLab/sybyrla (Whisper-v3 FT)	1.5B	—	~17.7
nur-dev/Qwen3-ASR-0.6B-kk-ru-en	0.6B	17.8	20.4	kk/ru/en
Whisper large-v3 (zero-shot)	1.5B	32.4¹	—

¹ GigaAM protocol: utterances longer than 30 s and references containing digits are excluded.

Takeaways: on in-domain KSC2 this model is clearly ahead of the published fine-tunes. On FLEURS the much smaller GigaAM CTC model is still more accurate. Choose this model when you want readable punctuated text plus timestamps and speaker attribution from one model, and GigaAM when you only need cheap lowercase transcripts.

KSC2 by domain (WER %)

Domain	kk_v1	kk_v2
TV news	17.2	8.6
Crowdsourced (read)	14.9	9.0
Parliament	21.0	11.7
Radio	26.2	20.9
Talk shows	33.1	23.5
Podcasts	32.6	24.4

Read and broadcast speech is now in the single digits. Spontaneous conversational speech (podcasts, talk shows) remains the hardest.

Examples (FLEURS test)

REF: Егер сіз романдық тілді білсеңіз, сізге португал тілін үйрену оңайырақ болатынын айтпай-ақ қойса да болады.
HYP: Егер сіз романдық тілді білсеңіз, сізге португал тілін үйрену оңайырақ болатынын айтпай ақ қойса да болады.

REF: АҚШ президенті Дональд Трамп жексенбі күні кешкі баспасөз хатшысы арқылы жасаған мәлімдемесінде АҚШ қарушы күштерін Сириядан алып шығуы туралы мәлімдеді.
HYP: АҚШ президенті Дональд Трамп жексенбі күні кешкі баспасөз хатшысы арқылы жасаған мәлімдемесінде АҚШ қарушы күштерін Сириядан алып шығуы туралы мәлімдеді.

Word timestamps and diarization

The checkpoint keeps the base model's frozen Sortformer diarization branch and CTC timestamp head. With the ctc_timestamp_middleware (server command below) and "mm_processor_kwargs": {"capture_ctc_timestamps": true} in the request, the response carries a ctc_timestamps object with per-word start/end times, diarization segments, and a speaker mapping, just like the base model.

We tested this end to end on 30 synthetic Kazakh dialogues built from KSC2 test clips: 2 or 3 speakers, sequential or with 1 s of overlap.

Setup	DER %	Speaker count correct	Words inside their speaker's segment	Transcript <spk:k> count correct
2 speakers	11.3	90%	96%	10–20%
3 speakers	14.0	90%	90–91%	0%
2 speakers, overlap	12.3	100%	98–100%	20–30%

• Works: acoustic diarization and word timestamps work on Kazakh. Speech-region DER is 11–14%, with no collar.
• Does not work yet: speaker tags inside the generated text. Fine-tuning used only single-speaker data, so the model usually writes one <spk:0> for the whole recording, and cpWER computed from transcript tags is high.
• Workaround: assign speakers by time. Each word gets the diarization segment that contains its timestamp.
• A multi-speaker Kazakh fine-tune (synthetic mixtures with <spk:k> targets) is planned for the next version.

Note for NeMo users: export_checkpoint.py / merge_lora_checkpoint.py (NeMo Speech main, Nov 2026) write model.safetensors without the ctc_timestamp_* metadata, and vLLM then silently serves without timestamps. This checkpoint has the metadata restored from the base model.

Usage

Serve with vLLM 0.32 and the NeMo Speech plugin, exactly like the base model (see the base model card for installation):

export VLLM_PLUGINS=nemo_speechlm VLLM_USE_V2_MODEL_RUNNER=0
vllm serve Sherkhan243/Nemotron-3.5-Transcribe-kk-v2 \
  --served-model-name kk --trust-remote-code --dtype bfloat16 \
  --max-model-len 65536 --enable-chunked-prefill --no-enable-prefix-caching \
  --use-replayssm --mamba-backend flashinfer --mamba-cache-mode none \
  --mm-processor-cache-gb 0 --no-async-scheduling \
  --speculative-config '{"method":"mtp","num_speculative_tokens":8}' \
  --reasoning-parser nemotron_v3 --trust-request-mm-kwargs \
  --middleware nemo.collections.speechlm2.vllm.salm.ctc_serving.ctc_timestamp_middleware

import base64
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
audio = base64.b64encode(open("audio.wav", "rb").read()).decode()
prompt = ("Audio transcription. Language: Kazakh. Wording: verbatim. "
          "Formatting: punctuation and capitalization. Output: transcript only.")
r = client.chat.completions.create(
    model="kk", temperature=0, max_tokens=2048,
    messages=[{"role": "user", "content": [
        {"type": "text", "text": prompt},
        {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio}"}}]}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}, "skip_special_tokens": False})
print(r.choices[0].message.content)

Prompts the model was trained with (use them verbatim):

Output style	Prompt
Punctuated, cased	Audio transcription. Language: Kazakh. Wording: verbatim. Formatting: punctuation and capitalization. Output: transcript only.
Lowercase, no punctuation	Audio transcription. Language: Kazakh. Wording: verbatim. Formatting: lowercase, no punctuation. Output: transcript only.

Hardware: BF16 weights are 63 GB, so one 80–96 GB GPU (H100 / RTX PRO 6000) is needed. For smaller GPUs, quantize with NVIDIA ModelOpt to FP8/NVFP4 as NVIDIA did for the base model (not done for this checkpoint yet).

Training

Recipe	NeMo Speech examples/speechlm2/finetuning (SALMAutomodel)
Trainable	LoRA r=64, α=128 on attention q/k/v/o, Mamba in_proj, shared-expert up/down_proj (merged into the weights); full ASR encoder; audio projection
Frozen	LLM base weights, routed experts, router, Sortformer diarization branch, CTC timestamp head
Data	ISSAI-KSC2-Structured train (1,158 h; punctuation/case restored with Gemma; 15% of rows lowercased with the lowercase prompt) + FLEURS kk train ×3 — 633,736 utterances, 1,193 h
Decontamination	3,785 train rows whose normalized text equals a KSC2/FLEURS test reference removed; 0 exact test matches remain
Schedule	4,000 steps × 16k packed tokens (~0.6 epoch), AdamW, lr 1e-4, 200 warmup, cosine to 1e-6
Hardware	1× RTX PRO 6000 Blackwell (96 GB), ~2.6 s/step, ~2.9 h
Validation loss	0.201 @ 2000 → 0.1875 @ 3000

The full training config and per-utterance test outputs are in eval/.

Limitations

• Numbers. Spoken numbers are usually written as words, while some references use digits.
• Conversational speech. WER is 21–24% on podcasts and talk shows, versus about 9% on read and broadcast speech.
• Code-switching. Kazakh–Russian mixing has not been evaluated separately.
• Speaker tags in the text. Multi-speaker Kazakh transcripts usually get a single <spk:0>. Use the diarization segments and the word timestamps for speaker attribution (see above).
• Other languages. The model was trained only on Kazakh. Accuracy on the base model's other languages may have dropped and was not measured.

License and attribution

• Model: a derivative of NVIDIA Nemotron-3.5-Transcribe-30B-A4B, distributed under the same OpenMDW-1.1 license. This is a permissive license: use, modification and redistribution are allowed if the license text and notices are kept (see LICENSE, THIRD_PARTY_NOTICES.md and licenses/). No warranty.
• Training data:
  • KSC2 — Kazakh Speech Corpus 2 by ISSAI, CC BY 4.0, doi:10.48342/m90y-aj02.
  • KSC2-Structured — punctuation-restored KSC2 transcripts by Inflexion Lab.
  • FLEURS — by Google, CC BY 4.0.

Citation

If you use this model, please cite the data it was trained on:

@inproceedings{mussakhojayeva22_interspeech,
  title     = {KSC2: An Industrial-Scale Open-Source Kazakh Speech Corpus},
  author    = {Mussakhojayeva, Saida and Khassanov, Yerbolat and Varol, Huseyin Atakan},
  booktitle = {Proc. Interspeech 2022},
  pages     = {1367--1371},
  year      = {2022}
}
@inproceedings{conneau2023fleurs,
  title     = {FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech},
  author    = {Conneau, Alexis and Ma, Min and Khanuja, Simran and Zhang, Yu and Axelrod, Vera and Dalmia, Siddharth and Riesa, Jason and Rivera, Clara and Bapna, Ankur},
  booktitle = {2022 IEEE Spoken Language Technology Workshop (SLT)},
  pages     = {798--805},
  year      = {2023}
}