LFM3-1.7B — hybrid RKNN for the RK3588 NPU

LiquidAI/LFM3-1.7B ported to the RK3588 / RK3588S NPU as a hybrid RKNN model. rkllm 1.3.1 cannot convert this architecture (native lfm2 support lacks LFM3's sandwich norm, attention gate and partial RoPE; custom_config requires every layer to be attention, but half of LFM3's layers are short convolutions). So the compute graph was rebuilt by hand: every linear op runs on the NPU as fp16 RKNN segments, while the attention core (q·K, softmax, ·V, sigmoid gate) runs in fp32 on the CPU — measured RK3588 NPU fp16 Softmax/Sigmoid error is 14–20× the ideal-fp16 error, and moving attention off the NPU also made generation 60 % faster (no KV cache shuffling).

The weights are modified relative to the base model (re-segmented + fp16). Output was verified token-identical to the official PyTorch implementation (greedy) across unit, segment and on-board checks, including conv-state handling.

Files

file what it is
seg*.rknn NPU compute-graph segments (fp16, one per contiguous group of matmuls)
embed_f16.npy fp16 embedding table (kept on the CPU, memory-mapped)
meta2.json segment wiring, layer types, RoPE inv-freq, dims
tokenizer.json, generation_config.json tokenizer (used by runtime/)
runtime/ inference engine + chat CLI (the exact code used for the benchmark)

Requirements

  • RK3588 / RK3588S board, librknnrt.so 2.5.0 (rknn runtime) and the rknnlite Python API (aarch64 wheel from airockchip/rknn-toolkit2)
  • pip install numpy tokenizers
python runtime/chat.py . --tok lfm3 --prompt "你好,自我介紹一下"

All three NPU cores are used (NPU_CORE_0_1_2).

Benchmark

Radxa ROCK 5C (RK3588S, 8 GB), greedy, thinking off, 7-question zh-TW benchmark:

score generate prefill memory
5 / 7 (same as PyTorch fp32) 5.6 tok/s ~5 tok/s ~3.5 GB

Prefill is one token at a time (segment inputs are per-token), so long prompts are slow — this format targets decode-bound chat, not long-context agent workloads. Speed was measured without locked clocks.

License

LFM Open License v1.0 — see LICENSE. Notably, commercial use is not permitted for companies with ≥ USD 10M annual revenue. These weights are modified (re-segmented, fp16). The included runtime/ code is the inference glue needed to run the segments.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for oliver0804/LFM3-1.7B-RK3588-hybrid-rknn

Finetuned
(87)
this model