saxon11/pricer-2025-02-25_22.36.49

Pricer — Llama 3.1 8B Fine-Tuned for Product Price Prediction

A QLoRA fine-tuned version of Meta-Llama-3.1-8B trained to predict the price of a product given its description. The model takes a product title, description, and attributes as input and generates a dollar-amount price estimate.

Intended Use

Given a prompt like:

How much does this cost to the nearest dollar?

[Product title and description]

Price is $

The model completes the text with a predicted price value.

Training Details

Base Model

• Model: meta-llama/Meta-Llama-3.1-8B
• Quantization: 4-bit (NF4) via BitsAndBytes with double quantization and bfloat16 compute dtype

Dataset

• Dataset: saxon11/pricer-data-small
• Train split: 20,000 rows
• Test split: 8,544 rows
• Max sequence length: 182 tokens

QLoRA Configuration

Parameter	Value
LoRA rank (r)	8
LoRA alpha	16
Target modules	q_proj, v_proj, k_proj, o_proj
Dropout	0.1

Training Hyperparameters

Parameter	Value
Epochs	3
Batch size	16
Gradient accumulation steps	1
Learning rate	1e-4
LR scheduler	Cosine
Warmup ratio	0.03
Optimizer	paged_adamw_32bit
Precision	bf16
Max grad norm	0.3

Training Results

Metric	Value
Final training loss	1.1286
Mean token accuracy	0.7063
Total training steps	3,750
Training runtime	3,587 seconds (60 min)
Samples per second	16.73

Training loss decreased steadily from 1.97 at step 50 to 1.13 by the end of training, indicating the model learned the price prediction task effectively.

Data Collation

A completion-only data collator was used with the response template "Price is $", ensuring the model is only trained to predict the price tokens rather than the product description.

Framework & Libraries

• Transformers (HuggingFace)
• PEFT (Parameter-Efficient Fine-Tuning)
• TRL (Transformer Reinforcement Learning — SFTTrainer)
• BitsAndBytes (4-bit quantization)
• Weights & Biases (experiment tracking)

Limitations

• Trained on a relatively small dataset (20K examples) of product descriptions, primarily appliance parts and accessories.
• Prices in the training set are concentrated in the $1–$101 range (87.3% of data), so predictions outside this range may be less reliable.
• This is a LoRA adapter — it requires the base Llama 3.1 8B model to run inference.

Author

Josh Janzen · joshjanzen.com