VoxMedic
Collection
Modele rozpoznawania mowy medycznej. • 1 item • Updated
You need to agree to share your Hugging Face information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
Aby uzyskać dostęp do modelu Eskulap-ASR, musisz zaakceptować Medalion Eskulap-ASR Community License v1.0. Model jest bezpłatny dla badań, użytku prywatnego, niekomercyjnego oraz komercyjnego użytku podmiotów z rocznym obrotem brutto nieprzekraczającym 1 000 000 PLN. Większe podmioty komercyjne wymagają odrębnej licencji komercyjnej od Medalion Technology P.S.A.
Log in or Sign Up to review the conditions and access this model content.
Whisper-large-v3-turbo Polish Medical ASR
A fine-tuned openai/whisper-large-v3-turbo specialized for Polish medical speech using an
anti-forgetting training recipe (knowledge distillation + medical oversampling + general-domain replay)
and partial LoRA merge (α=0.75 weight interpolation) for implicit regularization.
This model reduces content WER on held-out Polish medical test sets by 42–50% relative to the base model while also improving performance on general Polish speech (including out-of-distribution European Parliament audio).
Benchmark results (content WER — lowercase, no punctuation)
Held-out test sets (fair-eval methodology — no train/test text overlap):
| Test Set | Base whisper-large-v3-turbo | This model | Δ (pp) | Relative |
|---|---|---|---|---|
| admed_anoni (medical, synthetic) | 16.58 % | 11.30 % | −5.28 | −32 % |
| admed_human (medical, human read) | 17.07 % | 7.64 % | −9.43 | −55 % |
| gemini (medical test2) | 6.43 % | 4.88 % | −1.55 | −24 % |
| bigos (general Polish) | 5.37 % | 5.50 % | +0.13 | +2 % |
| VoxPopuli (OOD, EU Parliament) | 15.88 % | 9.28 % | −6.60 | −42 % |
No catastrophic forgetting: bigos (general Polish) is essentially unchanged (+0.13pp). The model even improves on out-of-distribution formal Polish (VoxPopuli −6.6pp) because fine-tuning with Polish data fixes a language-detection issue in the base model.
Error quality (qualitative analysis, n=2000 medical samples)
| Metric | This model |
|---|---|
| Regressions (FT worse than base, >5pp per sample) | 59 / 2000 (2.95%) |
| Improvements (FT better than base, >5pp per sample) | 622 / 2000 (31.1%) |
| Unchanged | 1319 / 2000 (66.0%) |
Regression types are predominantly minor: Polish compound-word boundary shifts ("niereagujące" → "nie reagujące"), rare grammar insertions, and 2 repeat-loop hallucinations on audio where the base model also fails. No systematic formatting habits or deploy-hazardous patterns were detected.
Training recipe
| Component | Value |
|---|---|
| Base model | openai/whisper-large-v3-turbo |
| Adapter | LoRA r=64, α=128, dropout=0.0 |
| LoRA targets | encoder + decoder attention + FFN projections (49M trainable params) |
| Learning rate | 2e-4 (cosine, 10% warmup) |
| Epochs | 5 |
| Batch size | 16 × 4 GPUs |
| Precision | fp16, gradient checkpointing |
| Anti-forgetting | KD α=0.3, T=2.0 from frozen base |
| Data mix | Medical × 2 oversampled + bigos 10k general replay |
| Post-training | Partial LoRA merge: α=0.75 weight interpolation |
Partial merge technique
Instead of fully merging the LoRA adapter (which overfits on some test sets), we interpolate weights:
final_weights = 0.75 × merged_lora_weights + 0.25 × base_weights
This acts as implicit regularization, reducing per-sample regressions by ~20% compared to full merge while preserving nearly all of the medical WER improvement. The optimal α=0.75 was found via an 8-point sweep across [0.25, 0.85].
Training: ~6h on 4×A100 (SXM4-40GB). Partial merge adds ~5 min of post-processing.
Datasets
| Dataset | Role | Samples (train) |
|---|---|---|
lion-ai/admed_voice (admed_anoni) |
Medical (synthetic) | 8,516 × 2 |
lion-ai/admed_voice (admed_human) |
Medical (human read) | 5,693 × 2 |
lion-ai/pl_med_asr_test2 |
Medical (test2) | 1,301 × 2 |
lion-ai/bigos |
General Polish (replay) | 10,000 |
Evaluation uses held-out test splits from the datasets above, plus 200 out-of-distribution samples from VoxPopuli European Parliament.
Usage
import librosa
import torch
from transformers import pipeline
asr = pipeline(
"automatic-speech-recognition",
model="lion-ai/VoxMedic-turbo-beta",
torch_dtype=torch.float16,
device="cuda",
)
audio, sr = librosa.load("sample.wav", sr=16000)
# return_timestamps=True enables long-form decoding, so recordings over 30 s are transcribed in full
result = asr(audio, return_timestamps=True, generate_kwargs={"language": "pl", "task": "transcribe"})
print(result["text"])
If you call WhisperProcessor and model.generate yourself, note that the feature extractor
cuts the input at 30 s unless you pass truncation=False, padding="longest", return_attention_mask=True
(and then return_timestamps=True to generate).
Decoding settings
The model writes medical notation: colons, parentheses, slashes and dashes (Powód: …, (…), mg/kg).
Whisper's default decoders suppress these characters as "non-speech" symbols. When the model's preferred
next token is one of them, the remaining probability can go to a timestamp and then to end-of-text, and
decoding of the 30-second window stops early, for example after the first word of a dictation.
generation_config.json in this repo therefore does not suppress the punctuation that occurs in the
training transcripts. transformers (generate, pipeline) picks this up with no extra settings.
Other runtimes do not read generation_config.json:
faster-whisper / CTranslate2 and openai-whisper default to
suppress_tokens=[-1], a hard-coded symbol set that includes:and(. Pass the list from this repo instead:import json from huggingface_hub import hf_hub_download cfg = hf_hub_download("lion-ai/VoxMedic-turbo-beta", "generation_config.json") suppress_tokens = json.load(open(cfg))["suppress_tokens"] # faster-whisper; openai-whisper's model.transcribe() takes the same suppress_tokens argument segments, info = model.transcribe("sample.wav", language="pl", suppress_tokens=suppress_tokens)whisper.cpp (
ggml-model-q8_0.bin) does not suppress these symbols by default; keep--suppress-nstoff.
Why anti-forgetting matters
Naively fine-tuning Whisper on medical-only data destroys performance on general Polish. This recipe combines three techniques:
- Data replay — mixing general-domain (bigos) samples in training
- Knowledge distillation — KL divergence loss to frozen base preserves its output distribution
- Medical oversampling — repeats the medical training data 2× to shift the balance
- Partial merge — weight interpolation at deploy time provides implicit regularization
Result: strong medical WER improvement with no general-domain forgetting.
Known limitations
- Medical terminology with complex Polish morphology (e.g., "pozapiramidowego", "gruczolakorak") remains challenging — both base and fine-tuned models make similar errors on these terms (acoustic limit).
- ~3% of samples show minor regressions vs base (mostly compound-word boundary shifts).
- Trained primarily on read/dictated medical speech; spontaneous conversational medical speech may differ.
Related work
Part of the Eskulap project — Polish medical ASR research.
See also the smaller variant based on openai/whisper-medium.
Intended use and medical disclaimer
This model is an automatic speech recognition tool. It is not a certified medical device, not a diagnostic or therapeutic system, and not a standalone tool for clinical decision-making.
Outputs must not be used as the sole basis for diagnosis, treatment, clinical decisions, administrative decisions about patients, or medical record-keeping without appropriate verification by a qualified human. In any clinical setting, the deployer is responsible for validation, human oversight, risk assessment, and compliance with applicable healthcare, data-protection and patient-rights regulations (including GDPR).
License and usage
This model is provided under the Medalion Eskulap-ASR Community License v1.0. It is free for research, personal, non-commercial use, and commercial use by entities with annual gross revenue not exceeding 1,000,000 PLN. Larger commercial users require a separate commercial license from Medalion Technology P.S.A.
- Full license text: LICENSE
- Commercial licensing inquiries: kontakt@medalion.tech
Made by
TheThelion.ai Research Group.
Project lead: Alekasnder Obuchowski
Special thanks to:
- Maciej Gierczak
- Kinga Marszałkowska
- Mikołaj Badocha
- Downloads last month
- 110
Model tree for lion-ai/VoxMedic-turbo-beta
Base model
openai/whisper-large-v3 Finetuned
openai/whisper-large-v3-turbo