Anant

Anant is a small, local-first AI model family by BlueishAI. It is designed to run comfortably on consumer devices while still giving useful assistant behavior and stronger reasoning than the base chat variant when needed.

This repository contains GGUF builds for local inference with Ollama, llama.cpp, LM Studio, and other GGUF-compatible runtimes.

Available Files

File	Variant	Quantization	Best for
anant-1.5b-base-Q4_K_M.gguf	Base	Q4_K_M	Fastest local chat
anant-1.5b-base-Q8_0.gguf	Base	Q8_0	Better base quality with more RAM
anant-3b-reason-Q4_K_M.gguf	Reason	Q4_K_M	Math, logic, planning, and analysis
anant-3b-reason-Q8_0.gguf	Reason	Q8_0	Higher quality reasoning with more RAM

Which Variant Should You Use?

Use anant-1.5b-base-Q4_K_M.gguf if you want the fastest Anant model for normal chat, short answers, summarization, rewriting, and lightweight coding help.

Use anant-3b-reason-Q4_K_M.gguf if the task needs stronger reasoning: math, multi-step logic, debugging, planning, or careful analysis.

Use Q8_0 if you have enough memory and want quality closer to the merged model. Use Q4_K_M if you want the best balance of size, speed, and quality.

What Makes Anant Different?

Anant is not trying to be the largest model. The goal is a practical local model that:

• Runs on normal laptops and desktops.
• Has small GGUF files that are easy to download and run.
• Keeps a fast 1.5B base variant for everyday use.
• Provides a larger 3B reason variant for harder prompts.
• Uses LoRA adaptation instead of training a full model from scratch.
• Uses Q4_K_M and Q8_0 GGUF quantization for local inference.

The base model is intended to be quick and useful. The reason model is intended to be more deliberate, especially on tasks where the base model may make algebra, logic, or constraint-checking mistakes.

Training Overview

Anant is built from Qwen2.5 Instruct checkpoints and adapted using LoRA.

The training process used this broad pipeline:

1. Start from a Qwen2.5 Instruct checkpoint.
2. Train a LoRA adapter on curated instruction or reasoning data.
3. Merge the LoRA adapter back into the base model.
4. Export the merged model to F16 GGUF.
5. Quantize the F16 GGUF into Q4_K_M and Q8_0.
6. Test the final GGUF models locally with llama.cpp/Ollama style prompts.

The models were NOT trained from scratch. They are adaptations of Qwen2.5 models, which keeps the training cost manageable and preserves the strong tokenizer, chat format, and general language ability of the source model. This decision was made because it would've been a waste of time and resources if we had trained it from scratch. So, we took Qwen as the base model since qwen itself is a really strong model.

Base Variant

The base variant is the lightweight general assistant model.

Base model family:

Qwen2.5 Instruct

Training method:

LoRA supervised fine-tuning

Purpose:

• General chat
• Simple explanations
• Rewriting
• Summarization
• Lightweight coding help
• Fast local inference

The base model is optimized for responsiveness. It is the model to use when the prompt does not require deep reasoning.

Reason Variant

The reason variant is the stronger Anant model for multi-step problem solving.

Base model:

Qwen/Qwen2.5-3B-Instruct

Reasoning dataset:

open-r1/OpenR1-Math-220k

Training method:

QLoRA adaptation

Purpose:

• Math reasoning
• Constraint solving
• Logic problems
• Planning
• Debugging
• Programming
• Analysis tasks where checking intermediate steps matters

The reason variant is instructed to decide whether a prompt actually needs reasoning. For simple questions, it should answer directly. For math, logic, debugging, planning, or analysis, it should reason carefully and then give a clear answer.

LoRA And QLoRA

LoRA means Low-Rank Adaptation. Instead of updating every parameter in the base model, LoRA trains small adapter matrices attached to selected layers of the model. This makes training much cheaper and faster while still changing the model's behavior.

QLoRA is a memory-efficient version of this idea. The base model is loaded in 4-bit quantized form during training, while the LoRA adapter remains trainable. This makes it possible to adapt larger models on limited GPU memory.

After training, the adapter is merged into the base model to create a normal standalone model. That merged model is then exported to GGUF for local inference.

Why GGUF?

GGUF is the model format used by llama.cpp and many local inference tools. It is useful because it:

• Runs well on CPU.
• Supports quantized models.
• Works with Ollama and LM Studio.
• Makes local deployment simple.
• Avoids needing a full Python/Transformers stack for inference.

Quantization

The included models use:

Q4_K_M
Q8_0

Q4_K_M is smaller and faster. It is recommended for most users.

Q8_0 is larger and slower, but it keeps more of the original F16 model quality.

Approximate tradeoff:

Quantization	Size	Speed	Quality
Q4_K_M	Smaller	Faster	Good
Q8_0	Larger	Slower	Better

Ollama Usage

Run the base model:

ollama run anant

Run the reason model:

ollama run anant-reason

llama.cpp Usage

Example with the reason model:

llama-cli -m anant-3b-reason-Q4_K_M.gguf -c 4096 -n 512 -p "Solve carefully: A factory makes two products..."

Example with the base model:

llama-cli -m anant-1.5b-base-Q4_K_M.gguf -c 4096 -n 256 -p "Explain LoRA in simple terms."

Example Reasoning Test

Prompt:

A factory makes two products, A and B.
Each unit of A needs 2 hours of machine time and 1 hour of labor.
Each unit of B needs 1 hour of machine time and 3 hours of labor.
The factory has 100 machine hours and 90 labor hours available.
Profit is $40 per unit of A and $50 per unit of B.

How many units of A and B should the factory make to maximize profit? Show the reasoning.

Expected answer:

A = 42
B = 16
Maximum profit = $2480

This kind of prompt is useful because it checks whether the model can set up constraints, solve the intersection, evaluate corner points, and reject infeasible answers.

Web Search

Anant does not browse the internet by itself.

For web search, connect Anant to an external wrapper or application that performs search and injects the retrieved results into the prompt. The model can then answer from those provided results.

Do NOT rely on the model alone for current events, live prices, breaking news, laws, or anything time-sensitive.

Tokenizer And Chat Format

The tokenizer and chat template come from the Qwen2.5 model family. The Anant variants use the same tokenization format, so the base and reason variants are compatible with the same general chat formatting.

Limitations

Anant is small. It can be fast and useful, but it will not match large frontier models.

Known limitations:

• May hallucinate facts.
• Does not have live internet access.
• Can make reasoning mistakes, especially under long or ambiguous prompts.
• May struggle with very long context.
• Coding ability is limited compared with larger code-specialized models.
• Quantized models may lose some quality compared with F16.

For high-stakes medical, legal, financial, or safety-critical work, verify outputs with reliable sources and expert review.

Recommended Settings

General base model:

temperature: 0.7
top_p: 0.9
repeat_penalty: 1.08
context: 4096

Reason model:

temperature: 0.35
top_p: 0.9
repeat_penalty: 1.08
context: 4096

Lower temperature is recommended for reasoning tasks because it reduces random mistakes.

Citation

If you use Anant, cite or link this repository and BlueishAI.

Anant, BlueishAI. Local-first GGUF language models adapted from Qwen2.5 using LoRA.