VoxHOD: tiny person detector for drone search and rescue

A CenterNet-style, anchor-free person detector for downward-looking drone cameras: 136,547 parameters, 197 KB full-int8 TFLite, or 37,363 parameters, 75 KB for the nano variant, built for ESP32-S3-class microcontrollers. It marks where people are in a 160×160 RGB tile so a drone can flag places worth a closer look for a rescue team.

Everything here is built and trained from scratch. A larger sibling built on Google's EfficientNet-B0 is a separate project: VoxScout.

Türkçe özet: Arama kurtarma dronları için, mikrodenetleyicide çalışacak kadar küçük (136 bin parametre, int8 dosyası 197 KB) bir insan tespit modeli. HERIDAL test setinde int8 hâli, eşik 0,2'de insanların %83,9'unu buluyor. 75 KB'lık nano sürümü %81,2'sini buluyor. Eğitimde kullanılmayan bir WiSARD klibinde bu oranlar %61 ve %62'ye düşüyor. Hiç görmediği arazide (NOMAD) boş karo başına yanlış alarm 0,34'ten 1,29'a çıkıyor. ESP32-P4'te gerçek karolarda PC ile aynı oranda insan buluyor. CC BY 4.0.

Code and training details: ErdemWilkinson/VoxHOD.

Files

File	What it is
voxhod_qat_int8.tflite	Recommended (v3). Quantization-aware trained, full int8, input [1,160,160,3] int8, output [1,40,40,3] int8
voxhod_int8.tflite	Post-training quantized int8 of the float model
voxhod.keras	Float model
voxhod_nano_int8.tflite, voxhod_nano.keras	Nano. Half width, 75 KB post-training quantized int8, same input and output
evaluation.json, evaluation_wisard.json, evaluation_nano.json	Recall / precision / false alarms per threshold

Input and output

• Input: 160×160 RGB, pixel / 127.5 - 1. A 640×640 crop of a 4000×3000 aerial photo, resized to 160, matches the test setting.
• Output: 40×40 grid (stride 4).
  • Channel 0: person-centre heatmap (0 to 1).
  • Channels 1 and 2: log box width and height in grid cells.
• Detections: local maxima of channel 0 above a threshold. Lower thresholds find more people and give more false alarms.

import numpy as np
import tensorflow as tf
from PIL import Image

interpreter = tf.lite.Interpreter(model_path="voxhod_qat_int8.tflite")
interpreter.allocate_tensors()
inp, out = interpreter.get_input_details()[0], interpreter.get_output_details()[0]
(si, zi), (so, zo) = inp["quantization"], out["quantization"]

tile = np.asarray(Image.open("tile.png").convert("RGB").resize((160, 160)), np.float32) / 127.5 - 1
interpreter.set_tensor(inp["index"], np.clip(np.round(tile / si + zi), -128, 127).astype(np.int8)[None])
interpreter.invoke()
grid = (interpreter.get_tensor(out["index"])[0].astype(np.float32) - zo) * so

heat = grid[..., 0]
peaks = heat == tf.nn.max_pool2d(heat[None, :, :, None], 3, 1, "SAME")[0, :, :, 0].numpy()
for y, x in zip(*np.nonzero(peaks & (heat >= 0.2))):
    w, h = np.exp(grid[y, x, 1:]) * 4
    print(f"person at ({(x + 0.5) * 4:.0f}, {(y + 0.5) * 4:.0f}) px, about {w:.0f}x{h:.0f}, score {heat[y, x]:.2f}")

Results (HERIDAL test set, 3,507 tiles, 372 people)

A detection is a find when its centre is inside a person's box grown by one grid cell.

int8 file, threshold	Recall	Precision	False alarms per tile
v3 QAT, 0.30	76.6%	32.5%	0.17
v3 QAT, 0.20	83.9%	20.4%	0.35
v3 QAT, 0.10	88.7%	7.7%	1.12
nano PTQ, 0.30	72.3%	25.8%	0.22
nano PTQ, 0.20	81.2%	12.7%	0.59
nano PTQ, 0.10	89.5%	4.8%	1.87

External check: WiSARD sample

One clip of 264 correlated frames from another camera (6,336 tiles, 1,271 people), never trained on. Treat it as a first signal.

int8 file, threshold	Recall	Precision	False alarms per tile
v3 QAT, 0.20	61.3%	19.1%	0.52
nano PTQ, 0.20	62.2%	13.4%	0.81

False alarms on unseen terrain: NOMAD

NOMAD (CC BY 4.0) publishes 500 frames with nobody in them from another camera and terrain, 100 at each flight distance from 10 to 90 m. Cut into 4,000 empty tiles like the HERIDAL ones, every detection is a false alarm. Test only, never trained on.

int8 file, threshold 0.2	HERIDAL empty tiles	NOMAD, all	10 m	30 m	50 m	70 m	90 m
v3	0.34	1.29	1.10	1.11	1.39	1.35	1.49
nano	0.60	1.43	1.34	1.25	1.47	1.54	1.56

On unseen terrain v3 raises almost four times as many false alarms per empty tile as on HERIDAL, and more from higher up. At threshold 0.3 it drops to 0.74.

Where the false alarms come from

An analysis of v3 on the HERIDAL test set at threshold 0.2 (2026-11-07):

• 89% of false alarms are on tiles with nobody in them.
• Only 29% of training tiles are empty, against 91% of test tiles, so the model has had little practice at saying "nobody here".
• People 8 to 16 px tall are found 77% of the time, larger ones 90%.

Memory budget on ESP32 chips

The arena (the RAM a model needs while it runs) was measured on the ESP32-P4 and is almost the same on other ESP32 chips. Free RAM on the P4 was measured in the benchmark firmware; free RAM on the S3 and C6 are estimates for typical ESP-IDF projects, and the real value depends on the firmware. The C6 has no PSRAM, so everything must fit in its internal SRAM. Not included: camera frame buffers and the application's own memory.

Arena as a share of the chip's internal SRAM:

Model	Arena	ESP32-S3 (512 KB)	ESP32-C6 (512 KB)	ESP32-P4 (768 KB)
v3	332 KB	65%	65%	43%
nano	218 KB	43%	43%	28%

Arena as a share of the RAM that is actually free (✅ fits comfortably, ⚠️ fits with little room left, ❌ does not fit):

Model	P4, 569 KB free (measured)	S3, Wi-Fi off, ~350 KB (est.)	S3, Wi-Fi on, ~250 KB (est.)	C6, Wi-Fi off, ~400 KB (est.)	C6, Wi-Fi + BLE on, ~200 KB (est.)
v3	58% ✅	95% ❌*	133% ❌*	83% ⚠️	166% ❌
nano	38% ✅	62% ✅	87% ⚠️	55% ✅	109% ❌

* With PSRAM on the S3 the arena can move there and the model runs, more slowly.

Flash is not the limit (v3 takes 5% of a 4 MB flash). On an ESP32-S3 (240 MHz) the latency is estimated at roughly 1.2 to 2.5 times the P4's (v3 about 0.4 to 0.8 s, nano about 0.2 to 0.4 s per tile), not measured. In short: on an ESP32-S3 without PSRAM, use nano.

Roadmap

The file size will not grow; larger models built on someone else's backbone go to VoxScout. Each step must pass its gate before a new version is published here:

Every candidate is compared at a matched operating point: its threshold is chosen on the validation set to find as many people there as v3 does at 0.2, then it is scored on HERIDAL test, WiSARD and NOMAD with a paired bootstrap against v3.

• Fewer false alarms without losing people: tried 2026-11-08/09, not passed. Hard negatives turned out to be no better than a same-budget control; a 1.5× weight on missed people and plain extra training all lowered recall. v3 stays.

• Smaller files by pruning v3: tried 2026-11-09, not passed. At nano size pruning was no better than training from scratch over three runs each, and no run beat nano at the same recall. One run at 70% width (116 KB) matched v3's precision at 80% recall but lost recall at the matched point. Nano stays.

• A better nano: tried 2026-11-09, not passed. Nano with quantization-aware training, nano trained 30 more minutes and four from-scratch runs: the one picked on validation lost 6 points of recall. Matched on the 317 validation people, the candidates still landed between 77% and 87% recall on the test set, so a larger validation split is the next step. Nano stays.
• A held-out validation split: done 2026-11-10. The old validation photos were a random 10% of the training sequences, with near-twins in training. The new validation set is every photo of the five locations that never occur in test (739 people). Matched there, six nano-size runs land within 3 points of their validation recall on test (spread 5.4 points instead of 10). Published v3 and nano saw these photos in training, so they must be retrained before they can be compared on it.
• Is nano too small? Partly. Published v3 and nano both find 94% of the people in their own training tiles and about 84% on test, so recall is not limited by size. But nano raises 0.92 false alarms per training tile against v3's 0.59: separating people from look-alikes is where it runs short.
• Empty NOMAD frames in training: tried 2026-11-10, not passed. Three runs with half of the NOMAD frames added against three without, at the same HERIDAL recall: false alarms on the held-out NOMAD frames fall from 1.24 to 0.53 per tile, HERIDAL and WiSARD recall unchanged. The held-out frames are from the same camera and flights, and at the validation-matched threshold HERIDAL false alarms rose, so nothing is published.
• Better recall on small people. Gate: ≥ 85% on 8-16 px people.
• More varied aerial data and empty terrain with clear licences. Gate: WiSARD recall above 61.3% and fewer false alarms on NOMAD, without losing HERIDAL recall.

A v4 (v3 trained 2 more hours) was not published: HERIDAL 83.6% / 21.7%, but WiSARD recall fell to 58.6%. All numbers and the version history: GitHub README.

On-device measurement (ESP32-P4)

Measured on 2026-11-06 on one ESP32-P4 board (rev v1.3, 360 MHz, 32 MB flash, 32 MB PSRAM) with ESP-IDF 5.3.5 and esp-tflite-micro 1.4.2. Each row is the mean of 20 runs after 2 warm-up runs; the published files were measured byte for byte. One inference is one 160x160 RGB tile.

Model	File	Arena used	Latency, arena in internal SRAM	Latency, arena in PSRAM (200 MHz)
VoxHOD v3	197 KB	332 KB	321 ms	347 ms
VoxHOD nano	75 KB	218 KB	151 ms	159 ms

The weights stay in flash; the arena only holds the activations. Both models fit in internal SRAM. At 321 ms a tile, the P4 scans about 3 tiles a second.

Output on real tiles (2026-11-09). 163 HERIDAL test tiles (149 people) through both published files on the same board, compared with the PC interpreter at threshold 0.2:

File	Heatmap cells bit-identical	Largest difference	Recall, PC	Recall, device
v3	92.1%	3 int8 steps	81.9%	81.9%
nano	48.8%	6 int8 steps	79.9%	80.5%

The device finds the same people as the PC within one person; the output is not bit-identical, because the device's ESP-NN kernels round differently. Firmware and scripts: benchmarks/esp32-p4-check.

What the timing does and does not show:

• Random int8 input for the timing; only the model inference is timed, not image decoding, resizing or box extraction.
• It is an ESP32-P4, not an ESP32-S3. The numbers do not carry over to the S3, which has other cores.
• One board. Source and raw output: benchmarks/esp32-p4.

Limitations

• Trained only on HERIDAL: summer, daytime, mostly Mediterranean terrain. Recall falls by about 20 points on the WiSARD clip, and false alarms on empty tiles are about four times higher on unseen terrain (NOMAD). Snow, forest canopy, water, night and thermal are untested.
• 90% recall is not reached on the test set.
• Measured on one ESP32-P4 only (see above), not yet on an ESP32-S3 or on real flight video.

Data and license

Not trained on the Ferhat Tatık footage. That footage is licensed to this project for non-commercial use only, so models trained on it are published separately as VoxHOD-NC (CC BY-NC 4.0). VoxHOD stays CC BY 4.0.

Trained on HERIDAL (CC BY 3.0): D. Božić-Štulić, Ž. Marušić, S. Gotovac, "Deep Learning Approach in Aerial Imagery for Supporting Land Search and Rescue Missions", IJCV 2019.

External test only, not training: WiSARD sample. D. Broyles, C. R. Hayner, K. Leung, "WiSARD: A Labeled Visual and Thermal Image Dataset for Wilderness Search and Rescue", IROS 2022.

External test only, not in the published weights: the NOMAD true-negative frames (ArtRuss/NOMAD, CC BY 4.0). Half of them were used in an unpublished 2026-11-10 training experiment.

Weights: CC BY 4.0. Training code: Apache License 2.0 with a NOTICE file (GitHub; earlier commits MIT).

Intended use

Search and rescue: helping find people who are lost or injured. It is not intended for targeting or harming people.