Bajju360/nanochat_d20base

hf download Bajju360/nanochat_d20base

Create .venv and install transformers and torch

uv venv
source .venv/bin/activate

Run Inference

python base_nano.py

Base model loss

timestamp: 2025-12-14 23:52:17

• train bpb: 0.8162
• val bpb: 0.8135

Base model training

timestamp: 2025-12-14 22:45:05

• run: nanochat_d20
• device_type:
• depth: 20
• max_seq_len: 2048
• num_iterations: -1
• target_flops: -1.0000
• target_param_data_ratio: 20
• device_batch_size: 64
• total_batch_size: 1,048,576
• embedding_lr: 0.4000
• unembedding_lr: 0.0080
• weight_decay: 0.0000
• matrix_lr: 0.0400
• grad_clip: 1.0000
• warmup_ratio: 0.0000
• warmdown_ratio: 0.2000
• final_lr_frac: 0.0000
• resume_from_step: -1
• eval_every: 250
• eval_tokens: 62,914,560
• core_metric_every: 2000
• core_metric_max_per_task: 500
• sample_every: 2000
• save_every: 1000
• model_tag:
• Number of parameters: 560,988,160
• Number of FLOPs per token: 3.491758e+09
• Calculated number of iterations: 10,700
• Number of training tokens: 11,219,763,200
• Tokens : Params ratio: 20.0000
• DDP world size: 1
• warmup_ratio: 0.0000
• warmdown_ratio: 0.2000
• final_lr_frac: 0.0000
• Minimum validation bpb: 0.8169
• Final validation bpb: 0.8169
• CORE metric estimate: 0.2100
• MFU %: 37.51%
• Total training flops: 3.917670e+19
• Total training time: 1758.84m
• Peak memory usage: 145766.77MiB

Base model evaluation

timestamp: 2025-12-15 00:17:50

• Model: base_model (step 10700)
• CORE metric: 0.2036
• hellaswag_zeroshot: 0.2555
• jeopardy: 0.0874
• bigbench_qa_wikidata: 0.5157
• arc_easy: 0.5253
• arc_challenge: 0.1069
• copa: 0.2200
• commonsense_qa: 0.1308
• piqa: 0.3765
• openbook_qa: 0.0987
• lambada_openai: 0.3852
• hellaswag: 0.2591
• winograd: 0.2821
• winogrande: 0.0355
• bigbench_dyck_languages: 0.0890
• agi_eval_lsat_ar: 0.1141
• bigbench_cs_algorithms: 0.4030
• bigbench_operators: 0.1905
• bigbench_repeat_copy_logic: 0.0000
• squad: 0.2085
• coqa: 0.2078
• boolq: -0.1902
• bigbench_language_identification: 0.1770