Pretraining a 325M Language Model from Zero on an Inference-Only NPU

TL;DR

• We pretrained VKORE-325M-NPU-Base, a 325.6M parameter decoder-only language model, from random initialization on a commercial inference-only NPU. Every weight was learned on the NPU with backpropagation run on the chip. No pretrained weights were used.
• The run covered 5.0 billion byte tokens of FineWeb-Edu in 76,292 steps over about four days.
• In parallel we trained the same model on a B200 GPU with the same initialization, data, batch and learning-rate schedule.
• On 1,024 windows of FineWeb-Edu text that neither model ever saw, the NPU model reaches a loss of 0.7993 and the GPU model 0.7994. The paired difference is -0.0001 with a standard error of 0.0007. The NPU model is lower on 50.1% of windows.
• Both sets of weights and the evaluation script are public, so anyone can check this result in a few minutes: FINAL-Bench/VKORE-325M-NPU-Base.

1. Why this matters

Accelerators sold for inference are now deployed in large numbers. They are cheaper and use less power than training GPUs, and their vendors ship them with a compiler and runtime built for one job: running a forward pass of a model that was trained somewhere else.

The accepted view is that such chips cannot train. Not a LoRA fine-tune, and certainly not a pretraining run from zero. Training needs a backward pass, an optimizer state, and numerically stable updates over tens of thousands of steps, none of which an inference stack is built to provide.

If that view is wrong, the consequences are practical. Organizations that already own inference hardware could adapt and even pretrain models on it, and new data centers could plan for one class of chip instead of two.

We tested the view directly, in two steps:

1. VKORE-46M-NPU (earlier): a 46M model trained on TinyStories. It showed that an inference-only NPU can run backpropagation at all.
2. VKORE-325M-NPU-Base (this post): a model seven times larger, trained on real web text for 76k steps, with a GPU twin to measure the result against.

2. Experimental design: one variable

A claim like "the NPU reached GPU quality" is only meaningful against a reference that differs in nothing but the hardware. We therefore trained the same model twice.

The only difference is the machine: a commercial inference-only NPU for run A, and a single B200 GPU for run B.

3. Model and data

Data. The training corpus is built from FineWeb-Edu sample/10BT. The parquet shards were read in repository order, each document was encoded as UTF-8 bytes, and documents were joined with a blank line. The corpus is the first two shards (000 and 001), 6.94 GB in total. Training drew random 257-byte windows from this file, so 5.0B training tokens correspond to about 0.72 passes over it.

A byte-level vocabulary keeps the comparison clean: there is no tokenizer to tune, and the reported loss is cross entropy per byte, which converts directly to bits per byte.

4. What made this hard

A backward pass is necessary but not sufficient. Our first long runs on the NPU trained correctly for thousands of steps and then diverged. Short tests did not reveal the problem; it only appeared in long runs checked against a GPU reference.

Making long runs stable on an inference-only stack was the main engineering work between the 46M model and this one. The technique was developed in house and is under patent application, so this post does not describe it. What we can show is the result: the 325M run trained for 76,292 steps with no divergence and no loss spikes. The run was interrupted twice by process failures and resumed from saved checkpoints both times, which is visible in the logs and has no visible effect on the curve.

5. Results

5.1 Training curves

The two curves overlap from the first thousand steps to the end. Both fall from about 5.8 to about 0.80. Small local wiggles differ because the two runs see their random windows in a different order; the trend is the same throughout. (The plotted NPU log runs to step 74,360; the final checkpoint is step 76,292.)

5.2 Loss on unseen text

The decisive test is text that was not in the training corpus at all. We used FineWeb-Edu sample/10BT shard 013, the last shard, which was never part of the training file, and evaluated both published models on 1,024 random 256-byte windows from it. We also report the 256 fixed windows from the end of the training corpus that were used during development.

Because both models were evaluated on identical windows, we can compare them window by window. That removes the variation between windows and gives a much tighter test than comparing two averages.

The paired difference is -0.0001 (NPU minus GPU) with a standard error of 0.0007, and the NPU model has the lower loss on 50.1% of windows. In other words, at this resolution the two models cannot be told apart.

A note on evaluation hygiene: the first version of our model card called the 256 training-corpus windows a "held out" set. They are not strictly held out, because training sampled windows from the whole file. We corrected the card and added the unseen-shard evaluation above, which is the number we stand behind.

5.3 Samples

Both models are small base models trained on 5B tokens, so they write fluent text with shaky facts. The point is that they behave alike. Sampling with temperature 0.7, top-p 0.9, repetition penalty 1.15, seed 0:

6. Reproduce it

Both models live in one repository. The B200 reference is in the b200_reference subfolder.

from transformers import AutoModelForCausalLM

npu = AutoModelForCausalLM.from_pretrained("FINAL-Bench/VKORE-325M-NPU-Base")
gpu = AutoModelForCausalLM.from_pretrained("FINAL-Bench/VKORE-325M-NPU-Base", subfolder="b200_reference")

The unseen-text comparison in section 5.2 is one script, evaluate_unseen.py. It downloads shard 013, draws the same 1,024 windows (seed 2026), and prints both losses and the paired difference. It uses only transformers, numpy and pyarrow.

Generating text with the byte-level model:

import torch
prompt = "Photosynthesis is the process"
ids = torch.tensor([list(prompt.encode("utf-8"))])
out = npu.generate(ids, max_new_tokens=200, do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.15)
print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))

7. What we do not claim

• Not a capable assistant. This is a 325M base model trained on 5B tokens, released as a training-hardware result.
• Not a speed record. This post is about whether an inference-only NPU can carry a full pretraining run to GPU quality, not about how fast. Throughput is a separate engineering track.
• Not the method. The training loop and the stability technique are in house and under patent application. The weights, data recipe, hyperparameters and evaluation are fully open.

8. What comes next

• Larger models on the same hardware class, and longer token budgets.
• Fine-tuning and adaptation workloads on inference-only NPUs for customers who already own the hardware.
• A paper with the full method once the patent filing is complete.

Links

• Model and B200 reference: FINAL-Bench/VKORE-325M-NPU-Base
• Previous model: FINAL-Bench/VKORE-46M-NPU
• Training data: HuggingFaceFW/fineweb-edu

VIDRAFT. Part of the VKORE project (NPU acceleration family).