Model
Params
—
Weights
—
Val loss
—
Perplexity
—
Completion acc.
—
Fwd tok/s
—
Training loss idle
step —
— tok/s
lr —
gnorm —
Chat with the model
Agent — tools + LLM
tools: calculator · search_corpus · word_count · llm_complete
Optimizations applied
- Architecture: RMSNorm, RoPE positional encoding, SwiGLU FFN, Grouped-Query Attention (6 Q heads / 2 KV heads), pre-norm residuals, tied embed↔LM-head.
- Kernels: PyTorch SDPA (Flash/MemEff backends), FFN hidden dim padded to multiples of 64.
- Training: bf16 autocast on MPS, AdamW with decoupled weight decay on 2D params only, gradient accumulation (effective batch 16k tokens), grad clip at 1.0, cosine LR + linear warmup, GPT-2 residual-projection init scaling.
- Data: pre-tokenized to uint16 bin (nanoGPT-style), mmap loader, random-window sampling.
- Agent: tool harness with deterministic planner, safe-sandboxed calculator, corpus grep, LLM-backed creative completion, structured trace.
Benchmarks
loading…
Model Builder — drag layers, swap components, run real training click the ? on any component to learn what it does
Config preview
—
Live progress
status: idle
—
Learn more
Architecture — layer-by-layer visibility
| module | shape | params | kind |
|---|
Experiment sweep — which variant wins & why
loading…
| rank | variant | params | val loss | perplexity | tok/s | time (s) | lr |
|---|