Instructions to use aiprimebuilds/LFM2.5-24B-A2B-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aiprimebuilds/LFM2.5-24B-A2B-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="aiprimebuilds/LFM2.5-24B-A2B-GPTQ")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("aiprimebuilds/LFM2.5-24B-A2B-GPTQ")
model = AutoModelForCausalLM.from_pretrained("aiprimebuilds/LFM2.5-24B-A2B-GPTQ", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use aiprimebuilds/LFM2.5-24B-A2B-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aiprimebuilds/LFM2.5-24B-A2B-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aiprimebuilds/LFM2.5-24B-A2B-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/aiprimebuilds/LFM2.5-24B-A2B-GPTQ

SGLang

How to use aiprimebuilds/LFM2.5-24B-A2B-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aiprimebuilds/LFM2.5-24B-A2B-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aiprimebuilds/LFM2.5-24B-A2B-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aiprimebuilds/LFM2.5-24B-A2B-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aiprimebuilds/LFM2.5-24B-A2B-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use aiprimebuilds/LFM2.5-24B-A2B-GPTQ with Docker Model Runner:
```
docker model run hf.co/aiprimebuilds/LFM2.5-24B-A2B-GPTQ
```

LFM2.5-24B-A2B-GPTQ

4-bit GPTQ build of LiquidAI/LFM2.5-24B-A2B, built to serve an office of agents from one RTX 4000 Ada 20 GB. Also tested on a 16 GB RTX 4080 SUPER.

Within noise everywhere except GPQA Diamond: 11 fewer correct out of 198.

The RTX 4000 runs all 8 users at once, even with 32k prompts. The 4080 SUPER generates faster but fits only about two long prompts at a time; the rest wait.

RTX 4000 Ada: 4 users comfortably, 6 with slowdowns. 4080 SUPER: 1–2. The limit is how many conversation histories fit in the KV cache (KV cache tokens ÷ average session size).

Run it

Tested on vLLM 0.31.0 with these flags:

Flag	Why
`--quantization gptq_marlin`	fast 4-bit kernels
`--kv-cache-dtype fp8`	halves KV cache memory; every GPTQ result above uses it
`--max-model-len 131072` (`65536` on 16 GB)	what fits next to the weights
`--gpu-memory-utilization 0.95`	gives vLLM 95% of VRAM, so more is left for KV cache
`--max-num-seqs 8`	up to 8 requests at once
`--reasoning-parser qwen3`	puts the `<think>` block in `reasoning_content`
`--enable-auto-tool-choice --tool-call-parser lfm2`	OpenAI-style tool calls

Details

Fidelity: wikitext-2 perplexity 15.69 vs 13.81 for BF16. KL divergence 0.365 mean. 78% top-token agreement.
KV cache: every GPTQ number uses an FP8 KV cache. The BF16 reference uses BF16.
Tool calls: 96–98% valid. On vLLM 0.31.0, streaming sometimes cuts off a call's arguments.
Sampling: the server default is temperature 0.1. For agents, use about 0.4.

Next: a 3-seed Red Hat-style evaluation.

LFM Open License v1.0 · shipleybuilds.ai

Downloads last month: 281

Safetensors

Model size

24B params

Tensor type

BF16

I32

Model tree for aiprimebuilds/LFM2.5-24B-A2B-GPTQ

Base model

LiquidAI/LFM2.5-24B-A2B-Base

Finetuned

LiquidAI/LFM2.5-24B-A2B

Quantized

(86)

this model