Jarbas commited on
Commit
ac74958
·
verified ·
1 Parent(s): fd2e7e7

WakeHuBERT extractor

Browse files
Files changed (4) hide show
  1. README.md +42 -0
  2. config.json +56 -0
  3. wakehubert.onnx +3 -0
  4. wakehubert_int8.onnx +3 -0
README.md ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-4.0
3
+ library_name: onnx
4
+ pipeline_tag: feature-extraction
5
+ tags: [wake-word, keyword-spotting, speech, distillation, hubert, onnx, int8]
6
+ ---
7
+
8
+ # WakeHuBERT: mel-TCN with windowed attention
9
+
10
+ A 1.43M-parameter speech feature extractor for wake-word detection, distilled from HuBERT-base: four dilated depthwise-separable convolution blocks with 256 channels followed by two causal self-attention layers over a one-second window, on a fixed log-mel front end (64 bins) and a strided convolution to 50 frames per second. It turns 16 kHz audio into 128-dimensional features, so a small classifier trained on those features from synthetic speech alone can detect a wake word. It is one of the WakeHuBERT extractors that [wakeforge](https://github.com/TigreGotico/wakeforge) can load; [TigreGotico/wakehubert-tiny](https://huggingface.co/TigreGotico/wakehubert-tiny) is the 0.64M mel-TCN of the same family.
11
+
12
+ ## Using it
13
+
14
+ The ONNX graph takes `waveform` [batch, samples] (float32, 16 kHz, -1..1) and returns `features` [batch, samples // 320, 128]. It is strictly causal: frame *t* depends only on audio before sample 320·(t+1). It was trained to reproduce each HuBERT frame five frames (100 ms) later, so its features describe the speech with a delay of about 100 ms. `wakehubert_int8.onnx` is a static int8 version whose features agree with float at a mean cosine of 0.993.
15
+
16
+ ```python
17
+ import numpy as np, onnxruntime as ort
18
+ from huggingface_hub import hf_hub_download
19
+ sess = ort.InferenceSession(hf_hub_download("TigreGotico/wakehubert-mel-attn", "wakehubert.onnx"))
20
+ feats = sess.run(None, {"waveform": np.zeros((1, 24000), np.float32)})[0] # (1, 75, 128)
21
+ ```
22
+
23
+ ## How it was made
24
+
25
+ It was trained for 30,000 steps to predict standardised HuBERT-base layers 4, 8 and 12 (L1 plus log-sigmoid cosine, as in DistilHuBERT), with the teacher hearing clean speech while the student heard it with noise, room reverberation and one to three background talkers never louder than the voice; a quarter of the training items were non-speech sounds heard identically by both.
26
+
27
+ Training speech: LibriSpeech (960 h), Multilingual LibriSpeech (seven languages) and a language-balanced sample of Multilingual Spoken Words (41 languages), cut into 600,000 two-second crops.
28
+
29
+ ## Results
30
+
31
+ Single run. Classifiers trained only on 900 synthetic "alexa" clips (TTS voices cloned with voice conversion, with noise, babble, reverberation, speed and gain augmentation) were scored on the real speakers of the Picovoice wake-word benchmark (315 recordings), with the threshold chosen for 0.5 false activations per hour on the 6.5 h of held-out streams it is reported on (LibriSpeech test-clean, three-talker test-other babble, held-out non-speech), so false activations hold by construction:
32
+
33
+ | Classifier | Recall, quiet | Recall in babble at 10 / 5 / 0 dB |
34
+ |---|---|---|
35
+ | GRU | 92% | 89 / 83 / 57% |
36
+ | bidirectional GRU | 95% | 92 / 84 / 50% |
37
+
38
+ Differences of a few points between extractors are within noise, and the two classifier types on the same extractor can differ by several points.
39
+
40
+ ## Licence and attribution
41
+
42
+ Released under CC BY 4.0. Trained with LibriSpeech, Multilingual LibriSpeech and Multilingual Spoken Words (all CC BY 4.0), MUSAN, AudioSet-derived noise and room impulse responses, and distilled from `facebook/hubert-base-ls960` (Apache 2.0).
config.json ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "family": "WakeHuBERT",
3
+ "architecture": "mel-attn",
4
+ "input": {
5
+ "name": "waveform",
6
+ "shape": [
7
+ "batch",
8
+ "samples"
9
+ ],
10
+ "dtype": "float32",
11
+ "sample_rate": 16000
12
+ },
13
+ "output": {
14
+ "name": "features",
15
+ "shape": [
16
+ "batch",
17
+ "frames",
18
+ "128"
19
+ ],
20
+ "frame_rate_hz": 50.0,
21
+ "hop_samples": 320
22
+ },
23
+ "feature_dim": 128,
24
+ "causal": true,
25
+ "streaming": true,
26
+ "delay_ms": 100.0,
27
+ "params": 1425152,
28
+ "student": {
29
+ "student": "mel-attn",
30
+ "channels": 256,
31
+ "blocks": 4,
32
+ "n_mels": 64,
33
+ "rnn_layers": 2,
34
+ "attn_layers": 2,
35
+ "attn_window": 50,
36
+ "mixconv": 0,
37
+ "mask_prob": 0.0,
38
+ "steps": 30000
39
+ },
40
+ "teacher": {
41
+ "model": "facebook/hubert-base-ls960",
42
+ "layers": [
43
+ 4,
44
+ 8,
45
+ 12
46
+ ]
47
+ },
48
+ "int8": {
49
+ "file": "wakehubert_int8.onnx",
50
+ "feature_cosine_mean": 0.9933
51
+ },
52
+ "files": {
53
+ "float32": "wakehubert.onnx",
54
+ "int8": "wakehubert_int8.onnx"
55
+ }
56
+ }
wakehubert.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7a469f74e99f24a72768e844302b47363ec6f618dff2e331106b7a4afaef1230
3
+ size 6435582
wakehubert_int8.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7ab2641d0349f724c29adbc29c1c8cd9eea5961112dda21c759a34f4c03b38f7
3
+ size 2304274