Instructions to use Goekdeniz-Guelmez/Omoikane-50M-PoC with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Goekdeniz-Guelmez/Omoikane-50M-PoC with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Goekdeniz-Guelmez/Omoikane-50M-PoC", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("Goekdeniz-Guelmez/Omoikane-50M-PoC", trust_remote_code=True, device_map="auto") - MLX
How to use Goekdeniz-Guelmez/Omoikane-50M-PoC with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # if on a CUDA device, also pip install mlx[cuda] # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("Goekdeniz-Guelmez/Omoikane-50M-PoC") prompt = "Once upon a time in" text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- vLLM
How to use Goekdeniz-Guelmez/Omoikane-50M-PoC with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Goekdeniz-Guelmez/Omoikane-50M-PoC" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Goekdeniz-Guelmez/Omoikane-50M-PoC", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/Goekdeniz-Guelmez/Omoikane-50M-PoC
- SGLang
How to use Goekdeniz-Guelmez/Omoikane-50M-PoC with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Goekdeniz-Guelmez/Omoikane-50M-PoC" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Goekdeniz-Guelmez/Omoikane-50M-PoC", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Goekdeniz-Guelmez/Omoikane-50M-PoC" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Goekdeniz-Guelmez/Omoikane-50M-PoC", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - MLX LM
How to use Goekdeniz-Guelmez/Omoikane-50M-PoC with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Generate some text mlx_lm.generate --model "Goekdeniz-Guelmez/Omoikane-50M-PoC" --prompt "Once upon a time"
- Docker Model Runner
How to use Goekdeniz-Guelmez/Omoikane-50M-PoC with Docker Model Runner:
docker model run hf.co/Goekdeniz-Guelmez/Omoikane-50M-PoC
- Atomic Chat
Configuration Parsing Warning:In config.json: "quantization_config" must be an object
Omoikane-50M-PoC
Omoikane (思兼, the Shinto god of wisdom and thought, "the one who combines many thoughts") is a small hybrid language model built and trained end to end on a single Mac mini (Apple M4, 32 GB unified memory, no CUDA). This repository holds the proof-of-concept (PoC) checkpoint: 48.9M parameters, trained from random initialisation for one epoch on 104M tokens of educational web text, in 6.8 hours.
The PoC's job was to answer one question before spending ~3 days of Mac time on the full 1B-token run: does this architecture, with this training recipe and this Apple-silicon training stack, work? The answer was yes. The details are below.
This is a research checkpoint, not an assistant. It writes fluent English but has limited knowledge and no instruction tuning. A larger 1B-token run, context extension to 4,096, SFT and preference training are in progress in the main project.
Why this architecture?
Omoikane mixes two kinds of sequence mixers, each used where it is cheapest:
- Gated DeltaNet (GDN) layers (8 of 10): linear-time recurrent layers with a fixed-size state per head, updated by the gated delta rule
S_t = α_t (I − β_t k_t k_tᵀ) S_{t−1} + β_t k_t v_tᵀ. They cost O(1) per token at inference and have no notion of absolute position, so they generalise across lengths. - Multi-head Latent Attention (MLA) layers (2 of 10, layers 3 and 7): full attention for exact recall, with a low-rank compressed KV cache (64 + 16 values per token per layer), decoupled RoPE and a sigmoid output gate.
- Fine-grained Mixture-of-Experts feed-forward blocks (layers 1–8): 16 small experts with top-4 group-limited sigmoid routing, a shared expert, and aux-loss-free load balancing (a per-expert bias nudged toward balance after every step).
- A multi-token-prediction (MTP) head that predicts token t+2. It adds training signal and can serve as a draft model for self-speculative decoding.
| Omoikane-50M | |
|---|---|
| Parameters | 48.9M total: 33.6M tied embedding + 15.4M trunk (~41.8M active per token) |
| Layers | 10 + MTP; GDN everywhere except MLA at 3 and 7; dense SwiGLU at 0 and 9; MoE in 1–8 |
| Width | d = 256; GDN 4 heads (K = V = 64, chunk 128); MLA 4 heads (q-rank 128, kv-rank 64, 48 nope + 16 RoPE dims, θ = 1e5) |
| MoE | 16 experts × 96, top-4 within 2 of 4 groups, + 1 shared expert |
| Vocabulary | 131,072-token byte-level BPE, tied input/output embedding |
| Trained context | 1,024 tokens (works zero-shot up to at least 4,096; see below) |
Training
How this checkpoint came about:
- Optimizer / LR sweep on 1,000 documents (10 trials of 59 steps each). Muon beat AdamW by 0.54 nats and Lion by 0.68 at their best learning rates. The winner was Muon, lr 6e-3, with a warmup-stable-decay (WSD) schedule.
- Port to MLX with a custom Metal kernel for the gated-delta recurrence. That gave ~4,100 tokens/s on the Mac mini, ~1.8× the PyTorch (MPS) implementation used for the sweep, with identical math (checked layer by layer and gradient by gradient).
- PoC pretraining (this checkpoint):
| setting | value |
|---|---|
| Data | mlx-community/fineweb-edu-100K: 99,936 documents, 103.7M tokens, 1 epoch, packed 1,024-token windows |
| Steps | 6,329 optimizer steps × 16,384 tokens (micro-batch 8 × 1,024, 2 gradient-accumulation steps) |
| Optimizer | Muon (Nesterov 0.95, 5 Newton–Schulz steps, RMS-matched) for hidden matrices; AdamW for embedding, router, norms, scalars; wd 0.1 |
| Schedule | 5% warmup → constant 6e-3 → linear decay to 6e-4 over the last 20% (WSD) |
| Loss | next-token CE + 0.1 × MTP CE (25% of positions) + 1e-4 MoE balance loss |
| Precision / stack | fp32 weights, bf16 vocab GEMMs, MLX 0.32 + custom Metal GDN kernel, mx.compile |
| Hardware / time | Mac mini M4, 32 GB, 6.8 hours, ~4,000–4,300 tokens/s, 13.1 GB peak memory |
| step | 250 | 1,000 | 2,000 | 3,000 | 4,000 | 5,000 | 6,000 | 6,329 |
|---|---|---|---|---|---|---|---|---|
| validation loss | 5.847 | 4.828 | 4.547 | 4.435 | 4.380 | 4.333 | 4.094 | 4.0105 |
| validation perplexity | 346 | 125 | 94 | 84 | 80 | 76 | 60 | 55.2 |
Training was stable throughout: no loss spikes, gradient norm ~0.2–0.45, expert load within ~1.2× of the mean. The LR-decay phase alone took 0.32 nats off the loss. The final validation loss is 4.0105 nats / 1.239 bits per byte.
Evaluation
Held-out loss
| eval set | loss | perplexity | note |
|---|---|---|---|
| fineweb-edu validation (in-domain) | 4.011 | 55.2 | |
| Ultra-FineWeb test (web, unseen source) | 4.122 | 61.7 | small transfer gap |
| UltraData-Math test | 4.637 | 103.3 | math never seen in training |
| UltraData-Code test | 5.299 | 200.2 | code never seen in training |
Zero-shot benchmarks
1,000 examples per task (BLiMP: 938 pairs over all 67 phenomena), lm-evaluation-harness-style prompts and loglikelihood scoring (custom MLX harness, so comparisons with other papers are approximate).
| task | accuracy | chance |
|---|---|---|
| BLiMP (grammar) | 72.9% | 50% |
| SciQ | 60.0% | 25% |
| PIQA | 54.6% | 50% |
| ARC-Easy | 39.2% (norm 36.4%) | 25% |
| LAMBADA (exact last word) | 9.8% | 0% |
| HellaSwag | 25.8% (norm 27.1%) | 25% |
| Winogrande | 50.7% | 50% |
Grammar is clearly learned, and simple knowledge and reading are above chance. Commonsense and coreference tasks (HellaSwag, Winogrande) are at chance, as expected for a 15M-parameter trunk after 104M tokens.
Does every part of the architecture do work?
Zeroing a component's output at inference time shows how much the trained model relies on it. Every GDN layer and both MLA layers carry load, and the routed experts matter (+1.05 nats). The one weak spot is the shared expert (+0.04), which is nearly redundant at this size; that's a design note for the next model. (Knock-outs measure reliance, not what a retrained model without the component would reach.)
The experts are all in use (routing entropy 0.96–0.99 of the maximum, no dead experts), and they start to specialise by domain in deeper layers.
The GDN heads learned long and short memory
Training pushed the recurrent heads apart: some now remember hundreds of tokens (decay α ≈ 0.996–0.999), others only 2–6 tokens. Layer 1 became purely local; the last layer turned 3 of its 4 heads into long-memory heads. The average head's horizon moved ~2.6× from initialisation.
Length generalisation
Trained only on 1,024-token windows, the model's loss stays flat out to 4,096 tokens with no context-extension training. The recurrent GDN layers carry most of the sequence modelling, and the two RoPE layers extrapolate fine.
Multi-token prediction
The MTP head predicts token t+2 with 26.3% top-1 accuracy (the main head predicts t+1 with 31.1%). Its draft agrees with the main model's own greedy choice 60.5% of the time, which should mean roughly 1.3–1.5× faster greedy decoding with self-speculative decoding.
Usage
🤗 Transformers (PyTorch)
The model uses custom code, so pass trust_remote_code=True. generate(), pipelines, streamers, beam search and batched
left-padded prompts are supported.
from threading import Thread
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
repo = "<this-repo-id>"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval()
prompts = [
"Photosynthesis is the process by which plants convert light energy into chemical energy.",
"Once upon a time, there was a little robot who lived in a big city called New York. He loved to play with his friends and help them learn new things.",
"Question: Sara has 14 apples and gives away 5. How many apples does she have left?\nAnswer:"
]
for prompt in prompts:
inputs = tokenizer(prompt, return_tensors="pt")
streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
thread = Thread(target=model.generate, kwargs=dict(
**inputs, streamer=streamer, max_new_tokens=64,
do_sample=True, temperature=0.8, top_p=0.95, repetition_penalty=1.1,
))
thread.start()
print(f"\n>>> {prompt}", flush=True)
for chunk in streamer:
print(chunk, end="", flush=True)
thread.join()
print()
Greedy decoding tends to loop at this model size; sampling (as above) with a mild repetition penalty works better. The model keeps a recurrent state, so it uses its own cache (OmoikaneCache); assisted decoding through generate() is not supported. On Apple-silicon PyTorch (MPS), separate runs can differ slightly numerically, so use CPU for bit-exact outputs.
MLX (Apple silicon)
modeling_omoikane_mlx.py is the MLX implementation the model was trained with. It loads the same model.safetensors. Run from inside the downloaded folder:
import mlx.core as mx
from tokenizers import Tokenizer
from modeling_omoikane_mlx import load_model
model, _ = load_model(".")
tok = Tokenizer.from_file("tokenizer.json")
def generate(prompt, max_new_tokens=48, temperature=0.8):
ids = tok.encode(prompt, add_special_tokens=False).ids
cache, out = {}, []
logits = model(mx.array([ids]), cache=cache, offset=0)[:, -1]
for pos in range(len(ids), len(ids) + max_new_tokens):
nxt = mx.random.categorical(logits / temperature)
if nxt.item() == 1: # <|end|>
break
out.append(nxt.item())
logits = model(nxt[:, None], cache=cache, offset=pos)[:, -1]
return tok.decode(out)
print(generate("Photosynthesis is the process by which plants convert light energy into chemical energy."))
Limitations
- Small model, small budget. The trunk has 15.4M parameters and saw 104M tokens (about 2 tokens per parameter). Expect fluent but factually loose text, repetition under greedy decoding, and chance-level commonsense reasoning.
- Web-only training data. Math and code were never seen (losses 4.64 and 5.30).
- No baseline yet. The PoC shows the architecture trains stably and that all its parts do work. It does not show that Omoikane beats a standard transformer of the same size; a same-budget dense baseline is still to come.
- Not aligned. No instruction tuning or safety training; outputs may be wrong or inappropriate.
- English only (tokenizer and data).
Files
| file | what |
|---|---|
model.safetensors |
weights (fp32, 48.9M parameters, tied embedding stored once) |
config.json, configuration_omoikane.py |
model config (model_type: omoikane) |
modeling_omoikane.py |
PyTorch / 🤗 Transformers implementation (OmoikaneForCausalLM) |
modeling_omoikane_mlx.py |
MLX implementation used for training |
tokenizer.json, tokenizer_config.json |
131,072-token byte-level BPE (`< |
generation_config.json |
sampling defaults (T = 0.8, top-p 0.95) |
trainer_state.json |
final step, token count and training arguments |
assets/ |
figures used in this card |
Model family
This PoC is the first of a series of checkpoints from the same project:
| model | what it is | status |
|---|---|---|
| Omoikane-50M-PoC (this repo) | proof of concept: 104M tokens of fineweb-edu, 1,024 context | released |
Goekdeniz-Guelmez/Omoikane-50M-Base |
pretraining on a 1B-token mix (70% web, 20% math, 10% code) | training |
Goekdeniz-Guelmez/Omoikane-50M-Base-4K |
Base with context extended to 4,096 tokens | planned |
Goekdeniz-Guelmez/Omoikane-50M-Instruct |
instruction-tuned (SFT on JOSIE-3) | planned |
Goekdeniz-Guelmez/Omoikane-50M-Chat |
Instruct + preference training | planned |
Project
Code, training logs, the sweep, analysis and the technical report live in the main Omoikane project repository (currently Goekdeniz-Guelmez/Anthropic-Numbat on the Hugging Face Hub, to be renamed Goekdeniz-Guelmez/Omoikane). Author: Gökdeniz Gülmez.
- Downloads last month
- 69
Quantized






