LFM3-1.7B — hybrid RKNN for the RK3588 NPU
LiquidAI/LFM3-1.7B ported to the RK3588 / RK3588S NPU as a hybrid RKNN model. rkllm 1.3.1 cannot convert this architecture (native lfm2 support lacks LFM3's sandwich norm, attention gate and partial RoPE; custom_config requires every layer to be attention, but half of LFM3's layers are short convolutions). So the compute graph was rebuilt by hand: every linear op runs on the NPU as fp16 RKNN segments, while the attention core (q·K, softmax, ·V, sigmoid gate) runs in fp32 on the CPU — measured RK3588 NPU fp16 Softmax/Sigmoid error is 14–20× the ideal-fp16 error, and moving attention off the NPU also made generation 60 % faster (no KV cache shuffling).
The weights are modified relative to the base model (re-segmented + fp16). Output was verified token-identical to the official PyTorch implementation (greedy) across unit, segment and on-board checks, including conv-state handling.
Files
| file | what it is |
|---|---|
seg*.rknn |
NPU compute-graph segments (fp16, one per contiguous group of matmuls) |
embed_f16.npy |
fp16 embedding table (kept on the CPU, memory-mapped) |
meta2.json |
segment wiring, layer types, RoPE inv-freq, dims |
tokenizer.json, generation_config.json |
tokenizer (used by runtime/) |
runtime/ |
inference engine + chat CLI (the exact code used for the benchmark) |
Requirements
- RK3588 / RK3588S board,
librknnrt.so2.5.0 (rknn runtime) and therknnlitePython API (aarch64 wheel from airockchip/rknn-toolkit2) pip install numpy tokenizers
python runtime/chat.py . --tok lfm3 --prompt "你好,自我介紹一下"
All three NPU cores are used (NPU_CORE_0_1_2).
Benchmark
Radxa ROCK 5C (RK3588S, 8 GB), greedy, thinking off, 7-question zh-TW benchmark:
| score | generate | prefill | memory |
|---|---|---|---|
| 5 / 7 (same as PyTorch fp32) | 5.6 tok/s | ~5 tok/s | ~3.5 GB |
Prefill is one token at a time (segment inputs are per-token), so long prompts are slow — this format targets decode-bound chat, not long-context agent workloads. Speed was measured without locked clocks.
License
LFM Open License v1.0 — see LICENSE. Notably, commercial use is not permitted for companies with ≥ USD 10M annual revenue. These weights are modified (re-segmented, fp16). The included runtime/ code is the inference glue needed to run the segments.