WakeHuBERT extractor
Browse files- README.md +42 -0
- config.json +56 -0
- wakehubert.onnx +3 -0
- wakehubert_int8.onnx +3 -0
README.md
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-4.0
|
| 3 |
+
library_name: onnx
|
| 4 |
+
pipeline_tag: feature-extraction
|
| 5 |
+
tags: [wake-word, keyword-spotting, speech, distillation, hubert, onnx, int8]
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
# WakeHuBERT: mel-TCN with windowed attention
|
| 9 |
+
|
| 10 |
+
A 1.43M-parameter speech feature extractor for wake-word detection, distilled from HuBERT-base: four dilated depthwise-separable convolution blocks with 256 channels followed by two causal self-attention layers over a one-second window, on a fixed log-mel front end (64 bins) and a strided convolution to 50 frames per second. It turns 16 kHz audio into 128-dimensional features, so a small classifier trained on those features from synthetic speech alone can detect a wake word. It is one of the WakeHuBERT extractors that [wakeforge](https://github.com/TigreGotico/wakeforge) can load; [TigreGotico/wakehubert-tiny](https://huggingface.co/TigreGotico/wakehubert-tiny) is the 0.64M mel-TCN of the same family.
|
| 11 |
+
|
| 12 |
+
## Using it
|
| 13 |
+
|
| 14 |
+
The ONNX graph takes `waveform` [batch, samples] (float32, 16 kHz, -1..1) and returns `features` [batch, samples // 320, 128]. It is strictly causal: frame *t* depends only on audio before sample 320·(t+1). It was trained to reproduce each HuBERT frame five frames (100 ms) later, so its features describe the speech with a delay of about 100 ms. `wakehubert_int8.onnx` is a static int8 version whose features agree with float at a mean cosine of 0.993.
|
| 15 |
+
|
| 16 |
+
```python
|
| 17 |
+
import numpy as np, onnxruntime as ort
|
| 18 |
+
from huggingface_hub import hf_hub_download
|
| 19 |
+
sess = ort.InferenceSession(hf_hub_download("TigreGotico/wakehubert-mel-attn", "wakehubert.onnx"))
|
| 20 |
+
feats = sess.run(None, {"waveform": np.zeros((1, 24000), np.float32)})[0] # (1, 75, 128)
|
| 21 |
+
```
|
| 22 |
+
|
| 23 |
+
## How it was made
|
| 24 |
+
|
| 25 |
+
It was trained for 30,000 steps to predict standardised HuBERT-base layers 4, 8 and 12 (L1 plus log-sigmoid cosine, as in DistilHuBERT), with the teacher hearing clean speech while the student heard it with noise, room reverberation and one to three background talkers never louder than the voice; a quarter of the training items were non-speech sounds heard identically by both.
|
| 26 |
+
|
| 27 |
+
Training speech: LibriSpeech (960 h), Multilingual LibriSpeech (seven languages) and a language-balanced sample of Multilingual Spoken Words (41 languages), cut into 600,000 two-second crops.
|
| 28 |
+
|
| 29 |
+
## Results
|
| 30 |
+
|
| 31 |
+
Single run. Classifiers trained only on 900 synthetic "alexa" clips (TTS voices cloned with voice conversion, with noise, babble, reverberation, speed and gain augmentation) were scored on the real speakers of the Picovoice wake-word benchmark (315 recordings), with the threshold chosen for 0.5 false activations per hour on the 6.5 h of held-out streams it is reported on (LibriSpeech test-clean, three-talker test-other babble, held-out non-speech), so false activations hold by construction:
|
| 32 |
+
|
| 33 |
+
| Classifier | Recall, quiet | Recall in babble at 10 / 5 / 0 dB |
|
| 34 |
+
|---|---|---|
|
| 35 |
+
| GRU | 92% | 89 / 83 / 57% |
|
| 36 |
+
| bidirectional GRU | 95% | 92 / 84 / 50% |
|
| 37 |
+
|
| 38 |
+
Differences of a few points between extractors are within noise, and the two classifier types on the same extractor can differ by several points.
|
| 39 |
+
|
| 40 |
+
## Licence and attribution
|
| 41 |
+
|
| 42 |
+
Released under CC BY 4.0. Trained with LibriSpeech, Multilingual LibriSpeech and Multilingual Spoken Words (all CC BY 4.0), MUSAN, AudioSet-derived noise and room impulse responses, and distilled from `facebook/hubert-base-ls960` (Apache 2.0).
|
config.json
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"family": "WakeHuBERT",
|
| 3 |
+
"architecture": "mel-attn",
|
| 4 |
+
"input": {
|
| 5 |
+
"name": "waveform",
|
| 6 |
+
"shape": [
|
| 7 |
+
"batch",
|
| 8 |
+
"samples"
|
| 9 |
+
],
|
| 10 |
+
"dtype": "float32",
|
| 11 |
+
"sample_rate": 16000
|
| 12 |
+
},
|
| 13 |
+
"output": {
|
| 14 |
+
"name": "features",
|
| 15 |
+
"shape": [
|
| 16 |
+
"batch",
|
| 17 |
+
"frames",
|
| 18 |
+
"128"
|
| 19 |
+
],
|
| 20 |
+
"frame_rate_hz": 50.0,
|
| 21 |
+
"hop_samples": 320
|
| 22 |
+
},
|
| 23 |
+
"feature_dim": 128,
|
| 24 |
+
"causal": true,
|
| 25 |
+
"streaming": true,
|
| 26 |
+
"delay_ms": 100.0,
|
| 27 |
+
"params": 1425152,
|
| 28 |
+
"student": {
|
| 29 |
+
"student": "mel-attn",
|
| 30 |
+
"channels": 256,
|
| 31 |
+
"blocks": 4,
|
| 32 |
+
"n_mels": 64,
|
| 33 |
+
"rnn_layers": 2,
|
| 34 |
+
"attn_layers": 2,
|
| 35 |
+
"attn_window": 50,
|
| 36 |
+
"mixconv": 0,
|
| 37 |
+
"mask_prob": 0.0,
|
| 38 |
+
"steps": 30000
|
| 39 |
+
},
|
| 40 |
+
"teacher": {
|
| 41 |
+
"model": "facebook/hubert-base-ls960",
|
| 42 |
+
"layers": [
|
| 43 |
+
4,
|
| 44 |
+
8,
|
| 45 |
+
12
|
| 46 |
+
]
|
| 47 |
+
},
|
| 48 |
+
"int8": {
|
| 49 |
+
"file": "wakehubert_int8.onnx",
|
| 50 |
+
"feature_cosine_mean": 0.9933
|
| 51 |
+
},
|
| 52 |
+
"files": {
|
| 53 |
+
"float32": "wakehubert.onnx",
|
| 54 |
+
"int8": "wakehubert_int8.onnx"
|
| 55 |
+
}
|
| 56 |
+
}
|
wakehubert.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7a469f74e99f24a72768e844302b47363ec6f618dff2e331106b7a4afaef1230
|
| 3 |
+
size 6435582
|
wakehubert_int8.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7ab2641d0349f724c29adbc29c1c8cd9eea5961112dda21c759a34f4c03b38f7
|
| 3 |
+
size 2304274
|