Configuration Parsing Warning:In config.json: "quantization_config" must be an object

Omoikane-50M-PoC

Omoikane (思兼, the Shinto god of wisdom and thought, "the one who combines many thoughts") is a small hybrid language model built and trained end to end on a single Mac mini (Apple M4, 32 GB unified memory, no CUDA). This repository holds the proof-of-concept (PoC) checkpoint: 48.9M parameters, trained from random initialisation for one epoch on 104M tokens of educational web text, in 6.8 hours.

The PoC's job was to answer one question before spending ~3 days of Mac time on the full 1B-token run: does this architecture, with this training recipe and this Apple-silicon training stack, work? The answer was yes. The details are below.

This is a research checkpoint, not an assistant. It writes fluent English but has limited knowledge and no instruction tuning. A larger 1B-token run, context extension to 4,096, SFT and preference training are in progress in the main project.

Omoikane architecture

Why this architecture?

Omoikane mixes two kinds of sequence mixers, each used where it is cheapest:

  • Gated DeltaNet (GDN) layers (8 of 10): linear-time recurrent layers with a fixed-size state per head, updated by the gated delta rule S_t = α_t (I − β_t k_t k_tᵀ) S_{t−1} + β_t k_t v_tᵀ. They cost O(1) per token at inference and have no notion of absolute position, so they generalise across lengths.
  • Multi-head Latent Attention (MLA) layers (2 of 10, layers 3 and 7): full attention for exact recall, with a low-rank compressed KV cache (64 + 16 values per token per layer), decoupled RoPE and a sigmoid output gate.
  • Fine-grained Mixture-of-Experts feed-forward blocks (layers 1–8): 16 small experts with top-4 group-limited sigmoid routing, a shared expert, and aux-loss-free load balancing (a per-expert bias nudged toward balance after every step).
  • A multi-token-prediction (MTP) head that predicts token t+2. It adds training signal and can serve as a draft model for self-speculative decoding.
Omoikane-50M
Parameters 48.9M total: 33.6M tied embedding + 15.4M trunk (~41.8M active per token)
Layers 10 + MTP; GDN everywhere except MLA at 3 and 7; dense SwiGLU at 0 and 9; MoE in 1–8
Width d = 256; GDN 4 heads (K = V = 64, chunk 128); MLA 4 heads (q-rank 128, kv-rank 64, 48 nope + 16 RoPE dims, θ = 1e5)
MoE 16 experts × 96, top-4 within 2 of 4 groups, + 1 shared expert
Vocabulary 131,072-token byte-level BPE, tied input/output embedding
Trained context 1,024 tokens (works zero-shot up to at least 4,096; see below)

Training

Training pipeline

How this checkpoint came about:

  1. Optimizer / LR sweep on 1,000 documents (10 trials of 59 steps each). Muon beat AdamW by 0.54 nats and Lion by 0.68 at their best learning rates. The winner was Muon, lr 6e-3, with a warmup-stable-decay (WSD) schedule.
  2. Port to MLX with a custom Metal kernel for the gated-delta recurrence. That gave ~4,100 tokens/s on the Mac mini, ~1.8× the PyTorch (MPS) implementation used for the sweep, with identical math (checked layer by layer and gradient by gradient).
  3. PoC pretraining (this checkpoint):
setting value
Data mlx-community/fineweb-edu-100K: 99,936 documents, 103.7M tokens, 1 epoch, packed 1,024-token windows
Steps 6,329 optimizer steps × 16,384 tokens (micro-batch 8 × 1,024, 2 gradient-accumulation steps)
Optimizer Muon (Nesterov 0.95, 5 Newton–Schulz steps, RMS-matched) for hidden matrices; AdamW for embedding, router, norms, scalars; wd 0.1
Schedule 5% warmup → constant 6e-3 → linear decay to 6e-4 over the last 20% (WSD)
Loss next-token CE + 0.1 × MTP CE (25% of positions) + 1e-4 MoE balance loss
Precision / stack fp32 weights, bf16 vocab GEMMs, MLX 0.32 + custom Metal GDN kernel, mx.compile
Hardware / time Mac mini M4, 32 GB, 6.8 hours, ~4,000–4,300 tokens/s, 13.1 GB peak memory

Training curve

step 250 1,000 2,000 3,000 4,000 5,000 6,000 6,329
validation loss 5.847 4.828 4.547 4.435 4.380 4.333 4.094 4.0105
validation perplexity 346 125 94 84 80 76 60 55.2

Training was stable throughout: no loss spikes, gradient norm ~0.2–0.45, expert load within ~1.2× of the mean. The LR-decay phase alone took 0.32 nats off the loss. The final validation loss is 4.0105 nats / 1.239 bits per byte.

Evaluation

Held-out loss

eval set loss perplexity note
fineweb-edu validation (in-domain) 4.011 55.2
Ultra-FineWeb test (web, unseen source) 4.122 61.7 small transfer gap
UltraData-Math test 4.637 103.3 math never seen in training
UltraData-Code test 5.299 200.2 code never seen in training

Zero-shot benchmarks

Zero-shot accuracy

1,000 examples per task (BLiMP: 938 pairs over all 67 phenomena), lm-evaluation-harness-style prompts and loglikelihood scoring (custom MLX harness, so comparisons with other papers are approximate).

task accuracy chance
BLiMP (grammar) 72.9% 50%
SciQ 60.0% 25%
PIQA 54.6% 50%
ARC-Easy 39.2% (norm 36.4%) 25%
LAMBADA (exact last word) 9.8% 0%
HellaSwag 25.8% (norm 27.1%) 25%
Winogrande 50.7% 50%

Grammar is clearly learned, and simple knowledge and reading are above chance. Commonsense and coreference tasks (HellaSwag, Winogrande) are at chance, as expected for a 15M-parameter trunk after 104M tokens.

Does every part of the architecture do work?

Component knock-outs

Zeroing a component's output at inference time shows how much the trained model relies on it. Every GDN layer and both MLA layers carry load, and the routed experts matter (+1.05 nats). The one weak spot is the shared expert (+0.04), which is nearly redundant at this size; that's a design note for the next model. (Knock-outs measure reliance, not what a retrained model without the component would reach.)

The experts are all in use (routing entropy 0.96–0.99 of the maximum, no dead experts), and they start to specialise by domain in deeper layers.

The GDN heads learned long and short memory

GDN memory horizons

Training pushed the recurrent heads apart: some now remember hundreds of tokens (decay α ≈ 0.996–0.999), others only 2–6 tokens. Layer 1 became purely local; the last layer turned 3 of its 4 heads into long-memory heads. The average head's horizon moved ~2.6× from initialisation.

Length generalisation

Loss by position up to 4,096 tokens

Trained only on 1,024-token windows, the model's loss stays flat out to 4,096 tokens with no context-extension training. The recurrent GDN layers carry most of the sequence modelling, and the two RoPE layers extrapolate fine.

Multi-token prediction

The MTP head predicts token t+2 with 26.3% top-1 accuracy (the main head predicts t+1 with 31.1%). Its draft agrees with the main model's own greedy choice 60.5% of the time, which should mean roughly 1.3–1.5× faster greedy decoding with self-speculative decoding.

Usage

🤗 Transformers (PyTorch)

The model uses custom code, so pass trust_remote_code=True. generate(), pipelines, streamers, beam search and batched left-padded prompts are supported.

from threading import Thread

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer

repo = "<this-repo-id>"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval()

prompts = [
    "Photosynthesis is the process by which plants convert light energy into chemical energy.",
    "Once upon a time, there was a little robot who lived in a big city called New York. He loved to play with his friends and help them learn new things.",
    "Question: Sara has 14 apples and gives away 5. How many apples does she have left?\nAnswer:"
]

for prompt in prompts:
    inputs = tokenizer(prompt, return_tensors="pt")
    streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
    thread = Thread(target=model.generate, kwargs=dict(
        **inputs, streamer=streamer, max_new_tokens=64,
        do_sample=True, temperature=0.8, top_p=0.95, repetition_penalty=1.1,
    ))
    thread.start()
    print(f"\n>>> {prompt}", flush=True)
    for chunk in streamer:
        print(chunk, end="", flush=True)
    thread.join()
    print()

Greedy decoding tends to loop at this model size; sampling (as above) with a mild repetition penalty works better. The model keeps a recurrent state, so it uses its own cache (OmoikaneCache); assisted decoding through generate() is not supported. On Apple-silicon PyTorch (MPS), separate runs can differ slightly numerically, so use CPU for bit-exact outputs.

MLX (Apple silicon)

modeling_omoikane_mlx.py is the MLX implementation the model was trained with. It loads the same model.safetensors. Run from inside the downloaded folder:

import mlx.core as mx
from tokenizers import Tokenizer
from modeling_omoikane_mlx import load_model

model, _ = load_model(".")
tok = Tokenizer.from_file("tokenizer.json")

def generate(prompt, max_new_tokens=48, temperature=0.8):
    ids = tok.encode(prompt, add_special_tokens=False).ids
    cache, out = {}, []
    logits = model(mx.array([ids]), cache=cache, offset=0)[:, -1]
    for pos in range(len(ids), len(ids) + max_new_tokens):
        nxt = mx.random.categorical(logits / temperature)
        if nxt.item() == 1:  # <|end|>
            break
        out.append(nxt.item())
        logits = model(nxt[:, None], cache=cache, offset=pos)[:, -1]
    return tok.decode(out)

print(generate("Photosynthesis is the process by which plants convert light energy into chemical energy."))

Limitations

  • Small model, small budget. The trunk has 15.4M parameters and saw 104M tokens (about 2 tokens per parameter). Expect fluent but factually loose text, repetition under greedy decoding, and chance-level commonsense reasoning.
  • Web-only training data. Math and code were never seen (losses 4.64 and 5.30).
  • No baseline yet. The PoC shows the architecture trains stably and that all its parts do work. It does not show that Omoikane beats a standard transformer of the same size; a same-budget dense baseline is still to come.
  • Not aligned. No instruction tuning or safety training; outputs may be wrong or inappropriate.
  • English only (tokenizer and data).

Files

file what
model.safetensors weights (fp32, 48.9M parameters, tied embedding stored once)
config.json, configuration_omoikane.py model config (model_type: omoikane)
modeling_omoikane.py PyTorch / 🤗 Transformers implementation (OmoikaneForCausalLM)
modeling_omoikane_mlx.py MLX implementation used for training
tokenizer.json, tokenizer_config.json 131,072-token byte-level BPE (`<
generation_config.json sampling defaults (T = 0.8, top-p 0.95)
trainer_state.json final step, token count and training arguments
assets/ figures used in this card

Model family

This PoC is the first of a series of checkpoints from the same project:

model what it is status
Omoikane-50M-PoC (this repo) proof of concept: 104M tokens of fineweb-edu, 1,024 context released
Goekdeniz-Guelmez/Omoikane-50M-Base pretraining on a 1B-token mix (70% web, 20% math, 10% code) training
Goekdeniz-Guelmez/Omoikane-50M-Base-4K Base with context extended to 4,096 tokens planned
Goekdeniz-Guelmez/Omoikane-50M-Instruct instruction-tuned (SFT on JOSIE-3) planned
Goekdeniz-Guelmez/Omoikane-50M-Chat Instruct + preference training planned

Project

Code, training logs, the sweep, analysis and the technical report live in the main Omoikane project repository (currently Goekdeniz-Guelmez/Anthropic-Numbat on the Hugging Face Hub, to be renamed Goekdeniz-Guelmez/Omoikane). Author: Gökdeniz Gülmez.

Downloads last month
69
Safetensors
Model size
48.9M params
Tensor type
F32
·
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train Goekdeniz-Guelmez/Omoikane-50M-PoC

Collection including Goekdeniz-Guelmez/Omoikane-50M-PoC