Document model provenance, conversion and validation

README.md

@@ -4,41 +4,127 @@ pipeline_tag: image-text-to-text
 base_model: openbmb/MiniCPM-V-4.7-1B
 tags:
 - MNN
-- minicpm-v
 - quantized
 - vision-language
 ---
 
 # MiniCPM-V-4.7-1B-MNN
 
-MNN conversion of [OpenBMB/MiniCPM-V-4.7-1B](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B), source revision 379f120c2ff3588e6e06559ec6b9ec15cc05114e.
-The source model and its weights are provided by OpenBMB under Apache-2.0.
+MNN-format conversion of [openbmb/MiniCPM-V-4.7-1B](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B). Model developer: OpenBMB.
 
-LLM: HQQ 4-bit, block size 64, FP16 scales. Embeddings: BF16.
-Vision encoder: HQQ 8-bit, block size 128 (per-channel where the channel count is not divisible by 128).
-The exported model uses default 16x visual compression and supports text, single-image and multi-image prompts, including image slicing. Video input has not been validated.
+## Model overview
 
-## Runtime
+MiniCPM-V 4.7 is OpenBMB's multimodal model family for efficient local inference. The original models combine single-image, multi-image and video understanding with early visual-token compression inside the vision encoder. This reduces computation in later encoder layers; the upstream architecture supports 4x and 16x compression to trade visual detail against inference cost. This is the compact 1B-class dense variant.
 
-Requires an MNN build with the MiniCPM-V-4.7 adapter discussed in [MNN issue #5015](https://github.com/alibaba/MNN/issues/5015), with LLM, LLM_OMNI, OPENCV and IMGCODECS enabled. Older MNN builds do not implement this model's image preprocessing or Canvas M-RoPE.
+This overview summarizes the [original model card](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B). Refer to it for training details, upstream evaluations and intended uses.
 
-Put this prompt in prompt.txt:
+| Property | Value |
+| --- | --- |
+| Language backbone | Qwen3.5 dense |
+| Vision | Patch-14 ViT, early window merger, final MLP merger |
+| This export | 16x visual compression, up to 9 image slices |
+| Inputs / outputs | Text and images / text |
 
-~~~text
-<img>cat.jpg</img>请用一句话描述图片中的动物。
+## MNN conversion
+
+Language linear weights: HQQ4, block 64, FP16 scales. Input embeddings: BF16. Vision weights: HQQ8, block 128, with per-channel quantization where the input channels are not divisible by 128.
+
+- Public converter/runtime baseline: **MNN 3.6.1**, commit [d662ce36175956038cfd6ed13f84cb3343a287d2](https://github.com/alibaba/MNN/commit/d662ce36175956038cfd6ed13f84cb3343a287d2).
+- Source checkpoint: `379f120c2ff3588e6e06559ec6b9ec15cc05114e`, the Hugging Face Git revision used for the source weights.
+- Export environment: Transformers 5.18.0, PyTorch 2.13.0, ONNX 1.16.0, HQQ 0.2.8.post1.
+- Export options are saved in [export_args.json](export_args.json); source provenance and runtime-file SHA256 checksums are in [conversion.json](conversion.json).
+- MiniCPM-V 4.7 adapter source snapshot: `584f2fe333bc1ad221becb75f25ad6fddfe52918`. The public baseline alone predates this adapter. Use an exporter/runtime containing this adaptation; follow [MNN issue #5015](https://github.com/alibaba/MNN/issues/5015) for upstream availability.
+
+### Files
+
+| File | Size (MiB) | Purpose |
+| --- | ---: | --- |
+| `llm.mnn` | 2.48 | Language graph |
+| `llm.mnn.weight` | 403.63 | Language weights, including tied embeddings when configured |
+| `embeddings_bf16.bin` | 484.56 | BF16 input embeddings |
+| `visual.mnn` | 0.47 | Vision graph |
+| `visual.mnn.weight` | 563.71 | Vision weights |
+| `tokenizer.mtok` | 7.00 | MNN tokenizer |
+
+Sizes describe storage, not peak RAM or GPU allocation. Keep graphs, weights, tokenizer and both config files together.
+
+## Download
+
+~~~bash
+hf download taobao-mnn/MiniCPM-V-4.7-1B-MNN --local-dir ./MiniCPM-V-4.7-1B-MNN
 ~~~
 
-Run:
+If your environment uses the mirror, set `HF_ENDPOINT=https://hf-mirror.com` first.
+
+## Build and convert
+
+Use an MNN source tree with IndexedLinear support and the MiniCPM-V 4.7 adapter noted above. From its root:
+
+~~~bash
+cmake -S . -B build -DMNN_BUILD_LLM=ON -DMNN_LOW_MEMORY=ON \
+  -DMNN_BUILD_CONVERTER=ON -DMNN_BUILD_LLM_OMNI=ON -DMNN_BUILD_OPENCV=ON -DMNN_IMGCODECS=ON
+cmake --build build --target llm_demo llm_bench MNNConvert -j4
+~~~
+
+For Metal add `-DMNN_METAL=ON`; for Android OpenCL use the Android NDK toolchain and `-DMNN_OPENCL=ON`. Use matching backend-enabled binaries.
+
+Download the original checkpoint at the recorded revision, install the exporter's dependencies, then run from `transformers/llm/export`:
 
 ~~~bash
-./llm_demo ./config.json ./prompt.txt
+python llmexport.py \
+  --path /path/to/original/MiniCPM-V-4.7-1B \
+  --export mnn --hqq --quant_bit 4 --quant_block 64 --scale_bit 16 \
+  --mnnconvert /path/to/MNN/build/MNNConvert \
+  --dst_path ./MiniCPM-V-4.7-1B-MNN
 ~~~
 
-The supplied config selects CPU, four threads, Low precision for the language model and Normal precision for the vision encoder. Vision window attention keeps crops in independent batches.
+## Inference and tests
+
+Create a greedy test config inside the downloaded model directory:
+
+~~~python
+import json
+from pathlib import Path
+model = Path('./MiniCPM-V-4.7-1B-MNN')
+config = json.loads((model / 'config.json').read_text())
+config.update(backend_type='cpu', precision='low', thread_num=4,
+              sampler_type='greedy', temperature=0.0, top_k=1)
+config['mllm'] = dict(backend_type='cpu', precision='low', thread_num=4, memory='low')
+(model / 'config-test.json').write_text(json.dumps(config, indent=2))
+~~~
+
+For a GPU build change backend_type to metal or opencl in both the top-level config and mllm, keeping separate language and vision runtime managers. Put each prompt on a separate line in prompts.txt:
+
+~~~text
+请用一句话介绍北京。
+<img>/absolute/path/cat.jpg</img>请用一句话描述图中的动物。
+~~~
+
+~~~bash
+./build/llm_demo ./MiniCPM-V-4.7-1B-MNN/config-test.json ./prompts.txt 128
+./build/llm_bench -m ./MiniCPM-V-4.7-1B-MNN/config-test.json \
+  -a cpu -c 2 -t 4 -p 0 -n 0 -pg 512,128 -rep 3 -load false
+~~~
+
+Use `-a metal` or `-a opencl` for the corresponding GPU build. `-pg 512,128` measures prefill followed by decode using the same KV cache, reporting the speeds separately. `-p 512 -n 128` instead measures independent prefill-only and decode-only workloads. Text benchmarks exclude image encoding.
 
 ## Validation
 
-Checked against Transformers 5.18.0 and tested with the MNN C++ CPU runtime on Linux.
-Python Prefill and first Decode logits/hidden states agree within 1e-4 tolerance. Single-image, sliced-image and multi-image token IDs and Canvas positions were checked against the original processor.
+Linux CPU text/image generation and Python prefill/first-decode parity passed. Single-image, sliced-image and multi-image token IDs and Canvas positions were checked against the original processor. Android PJZ110 passed 11 ARM CPU/OpenCL checks, including vision and language at Low precision; standard llm_demo text/image generation passed on both. Metal text/image correctness and GPU execution were verified on an M4 Mac mini.
+
+The MNN adapter is validated for images, including slicing and multiple images. Video has not been validated. Upstream capabilities describe the original model; these functional checks are not a quality benchmark for the quantized MNN files.
+
+### Measured text performance
+
+Low precision, HQQ4 language weights, four threads, one warmup excluded, three measured repetitions and no mmap. These are different devices, not a controlled backend comparison.
+
+| Device / backend | pp512 (token/s) | tg128 (token/s) | pp512+tg128: prefill / decode (token/s) |
+| --- | ---: | ---: | ---: |
+| M4 Mac mini, 16 GB / Metal | 2581.3 ± 1.2 | 188.6 ± 0.2 | 2576.4 / 185.2 |
+| Snapdragon 8 Elite, Adreno 830 / OpenCL | 1104.4 ± 9.6 | 76.6 ± 0.1 | 1099.0 / 76.1 |
+
+For 2048-token prefill followed by 128 decode tokens, Metal measured 2650.3 / 178.1 token/s and OpenCL measured 1033.2 / 75.5 token/s. Results are specific to the recorded snapshot and devices.
+
+## License and attribution
 
-These are functional checks, not a model-quality benchmark.
+Original model and weights: OpenBMB, Apache-2.0. See the complete [LICENSE](LICENSE) and original model card.

conversion.json

@@ -0,0 +1,74 @@
+{
+  "model": "MiniCPM-V-4.7-1B",
+  "mnn_version": "3.6.1",
+  "mnn_converter_runtime_public_commit": "d662ce36175956038cfd6ed13f84cb3343a287d2",
+  "source_repo": "openbmb/MiniCPM-V-4.7-1B",
+  "source_revision": "379f120c2ff3588e6e06559ec6b9ec15cc05114e",
+  "source_url": "https://huggingface.co/openbmb/MiniCPM-V-4.7-1B",
+  "quantization": {
+    "method": "HQQ",
+    "weight_bits": 4,
+    "block_size": 64,
+    "scale_bits": 16,
+    "embedding": {
+      "storage": "BF16 external"
+    }
+  },
+  "export_packages": {
+    "torch": "2.13.0",
+    "transformers": "5.18.0",
+    "onnx": "1.16.0",
+    "hqq": "0.2.8.post1",
+    "safetensors": "0.8.0"
+  },
+  "files": {
+    "llm.mnn.weight": {
+      "size": 423234586,
+      "sha256": "f4c2519d1e96f5a23515c97a7c6cad0db0d8d8c813c44038292803dad59ba2df"
+    },
+    "visual.mnn.weight": {
+      "size": 591096650,
+      "sha256": "d411969d6de971ce38acd9f3692cbcf952b2d0d2cf96bbd609618a0b3edd71d8"
+    },
+    "export_args.json": {
+      "size": 1117,
+      "sha256": "c07d12c6ce7638dbccbbe2d3a67f4ec479c30e79ad1c9f05916c20e9def527d8"
+    },
+    "LICENSE": {
+      "size": 11346,
+      "sha256": "a830d59c4f98d110e0540d0581efada06192ba6e798434307d1794b07a290302"
+    },
+    "llm_config.json": {
+      "size": 8510,
+      "sha256": "90a9ebea9dc5bf8a531ccdea28477cb362379e2049b70e7ff44f3ff27335af15"
+    },
+    "tokenizer.mtok": {
+      "size": 7334903,
+      "sha256": "6d5d90b7fc88cba043a2eb7d7cb909cec5c5911f4d02a9838d12d7787221ddd9"
+    },
+    "config.json": {
+      "size": 634,
+      "sha256": "b81ac7008ba5f894301b7b9265bba882889df52c6e25c86390514c1bd4afe0c4"
+    },
+    "visual.mnn": {
+      "size": 495696,
+      "sha256": "fe88829e2c4fa0c4ff1686ae0b0a2c4e7c5683f94d1621c1fb5924dc5946b959"
+    },
+    "llm.mnn": {
+      "size": 2603920,
+      "sha256": "c6cdbc5a1f7bfe20510d3770d6784d1ef387f1a921cf66ae861f0c0d53edd5a4"
+    },
+    "embeddings_bf16.bin": {
+      "size": 508096512,
+      "sha256": "6e6d9326e7a960b96d3c9fcf07686d26ab3147556a280dceb494806594f1e334"
+    }
+  },
+  "minicpm_adapter_source_commit": "584f2fe333bc1ad221becb75f25ad6fddfe52918",
+  "source_weight_files": [
+    {
+      "name": "model.safetensors",
+      "size": 3109054480,
+      "sha256": "8c4921ac83182c85190ad10c3aeb118d51e385d2d98ec13b88b514bbf6cac002"
+    }
+  ]
+}