LFM3-1.7B-chv-llm-v1 — a Chuvash language model (stage 1)

The first generative Chuvash language model from the chuvash-ai program: it writes and continues Chuvash text from a short Russian instruction. This is a research release of stage 1 (native-language LM). It is not a translation model — for Chuvash↔Russian translation use LFM3-1.7B-chv-v2. A stage-2 instruction/translation model built on top of this one is planned.

Native format — completion, no chat template

The model was trained on plain prompt→completion pairs with exactly two instruction prefixes (they must be reproduced verbatim):

Напиши текст на чувашском языке.

→ the model writes a new Chuvash sentence, and

Продолжи текст на чувашском языке.

{Chuvash text}

→ the model continues the given text.

The base LFM3 chat template is not used and was removed from the repo — wrap prompts manually:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

m = AutoModelForCausalLM.from_pretrained(
    "chuvash-ai/LFM3-1.7B-chv-llm-v1",
    dtype=torch.bfloat16, trust_remote_code=True, device_map="auto",
)
tok = AutoTokenizer.from_pretrained("chuvash-ai/LFM3-1.7B-chv-llm-v1")

prompt = "Напиши текст на чувашском языке."
ids = tok(prompt, return_tensors="pt").to(m.device)
out = m.generate(
    **ids, max_new_tokens=80, do_sample=True, temperature=0.8, top_p=0.95,
    pad_token_id=tok.eos_token_id,
)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

Sampling settings matter

Use do_sample=True, temperature≈0.8, top_p≈0.95. With greedy decoding the model degrades into word-repetition loops (a known artifact of this LM-only stage) — greedy is not recommended.

Training

  • Base: LiquidAI/LFM3-1.7B-Base with vocabulary surgery: extended to 136,441 tokens with Chuvash subwords (ӑ/ӗ/ҫ diacritics), cutting token fertility from ~5.6 to 1.93 tokens/word.
  • Continued pretraining: 3 epochs over ~5.9M sentences of Chuvash text (web crawls, the elbib literary corpus ≈104M characters, dictionaries, phrasebooks). Held-out perplexity dropped from 13,492 (base) to 139.5 (×96.7).
  • Supervised stage: 2.23M instruction-LM records (lm30/lm50 partial-text continuation, noseed free generation, next next-sentence continuation) via QLoRA, merged into full bf16 weights.

Limitations (read before using)

  • Research quality: the output is morphologically valid Chuvash with real vocabulary, but long-range coherence is weak.
  • Short outputs are normal — the model emits EOS after a complete sentence (that is what it was taught).
  • Only the two fixed Russian instructions above are supported; it is not a chat model and does not follow arbitrary instructions.
  • No translation ability (see v2 for that), no benchmark chrF for this model yet — stage-2 evaluation is pending.

Changes from the base model (LFM Open License §4b)

  • Vocabulary/embedding surgery to 136,441 tokens; all weights further trained (CPT + SFT, QLoRA-merged, bf16).
  • Custom architecture code files (configuration_lfm3.py, modeling_lfm3.py, lfm3_conv.py) are unchanged from the base model.
  • The base chat template was removed (see Native format).

License

LFM Open License v1.0 (inherited from the Liquid AI base, full text in LICENSE): permitted for research, non-profit and commercial use by entities under $10M annual revenue; redistribution must include this license and preserve attribution notices.

Acknowledgements

Part of the chuvash-ai open program for low-resource Chuvash NLP. Base model by Liquid AI.

Downloads last month
31
Safetensors
Model size
2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for chuvash-ai/LFM3-1.7B-chv-llm-v1

Finetuned
(27)
this model
Quantizations
1 model