Turkic language models · from inscription to inference
A hand-written transformer stack that learns Turkic the way the Orkhon stones learned to keep it — turning raw text into durable, intelligible speech.
Orkhon begins where Turkic history first learned to write itself: the 8th-century Göktürk inscriptions on the Orkhon River, carved to preserve voice, lineage, and law in a language of its own. From those stone lines to a living language system, the stack treats every model stage — tokenization, training, alignment, evaluation, and serving — as one continuous act of inscription: turning raw signal into durable, intelligible Turkic speech. Orkhon is not a metaphor pasted onto AI; it is a claim that the oldest Turkic writing and the newest language model share the same ambition — to make language last, carry memory, and speak clearly across time.
Not a wrapper around a frontier API. Orkhon is a from-scratch decoder-only transformer — GQA, RoPE, RMSNorm, SwiGLU, KV-cache — written in PyTorch and trained on consumer MPS/CUDA. The thesis is simple: the oldest Turkic writing and the newest language model share one ambition, to make language last.
Orkhon is a decoder-only transformer written from scratch in PyTorch — not a wrapper around a frontier API. Every block — grouped-query attention, rotary embeddings, RMSNorm, SwiGLU, the serving-time KV-cache — is hand-coded and auditable. You can read the whole stack top to bottom.
The full pipeline lives in one repo: byte-level BPE tokenization, pretraining, supervised fine-tuning, DPO and GRPO alignment, evaluation, and an OpenAI-compatible serving layer. The same code that carves the weights ships them.
It runs on your machine — Apple MPS or NVIDIA CUDA — and it trains on consumer hardware. TinyStories in roughly 43 minutes on an M-series GPU; a 4M smoke model in about a minute. No cluster required to hold the lineage in your hands.
Every model passes through the same inscription — each stage a chisel, each checkpoint a stone. Ten stages, from byte-level BPE to a safetensors export an OpenAI-compatible server can serve.
tokenizerdata_preppretrainsftdpogrpo_rlvrevaltools_rag_agentserveexportSeven Turkic voices, each named for a figure or word from the Göktürk inscriptions. Base models continue text; instruct models answer; the imported base brings outside knowledge back into the lineage.
tangriThe default specialist - one assistant for EN, TR, and Kokturk transliteration.
Tangri is the first larger unified Orkhon assistant, tuned from the mixed 100M base after the earlier bilingual and transliteration experiments.
bunghuBilingual branch, upgraded into the same specialist assistant.
Bunghu is the clean ASCII replacement for the former bilingual branch name, upgraded into the same unified assistant behavior.
teginThe old story voice, retrained to answer instead of drifting into stories.
Tegin replaces the former story-instruction public name and is tuned away from unwanted story replies into general assistant behavior.
tonyukCompact reasoning/story base, converted into a chat assistant.
Tonyuk is the concise clean name for the old story base after unified assistant fine-tuning.
istemThe web-text branch, aligned for assistant answers.
Istem is the clean name for the former web-text base after it is aligned into the unified assistant line.
bumin-miniTiny smoke assistant for fast local checks.
Bumin Mini keeps the first tiny proof-of-pipeline model but names it honestly as a compact smoke member.
kashgarImported reference slot, not live until weights are archived.
Kashgar is the imported/open-base slot; it becomes selectable only after local weights are archived and served.
Perplexity on held-out Turkic text, parameter counts, and release dates — the honest ledger of what the stack has actually learned so far.
Once upon a time, there was a little girl named Lily. She loved to play in the garden behind her house. One day she found a small bird that could not fly. Lily picked up the bird and took it home. She gave it water and seeds, and every day the bird grew stronger.
A from-scratch transformer producing coherent, grammatical English narrative after a 43-minute training run on consumer MPS — proof the stack learns to write, not just to count tokens.
<|system|>
You are Orkhon, a helpful assistant.
<|user|>
What is the Göktürk alphabet?
<|assistant|>
The Göktürk alphabet is the script used to write Old Turkic on the Orkhon inscriptions.A four-million-parameter smoke model learns the chat template in about a minute — the fastest proof the pipeline, tokenizer, and SFT loop are wired end to end.
Hand-written, no framework. Decoder-only transformer with grouped-query attention, rotary positional embeddings, RMSNorm, SwiGLU MLPs, and a serving-time KV-cache. Runs on Apple MPS and NVIDIA CUDA.
Queries share key/value heads
Multiple query heads attend over a smaller set of shared key/value heads — most of the attention quality at a fraction of the KV memory. The reason the cache fits on a laptop.
Positions encoded as rotations
Position is injected by rotating query/key pairs in complex space — no learned position table, and relative distances generalize cleanly to longer contexts than training saw.
Scale-only, mean-subtracted-free norm
LayerNorm without the mean subtraction — cheaper, faster, and empirically as stable. Pre-norm placement keeps the residual highway clean through deep stacks.
Gated, Swish-activated feed-forward
The feed-forward block gates its up-projection with a Swish-activated path — a gated linear unit that outperforms plain ReLU/GELU MLPs at the same parameter budget.
Past keys/values kept for incremental decode
At serve time, computed keys and values are cached so each new token only attends to history instead of recomputing it. Turns quadratic generation into linear, token-by-token.
Train, evaluate, export, and serve — all from one command surface. The same tooling that built the stones ships them.
$ whoami
umut
$ orkhon chat --model tangri
# drop into an interactive REPL with the unified specialist
$ orkhon serve --port 8000
# start an OpenAI-compatible HTTP server on :8000
$ orkhon train --config sft_tangri_unified.yaml
# fine-tune the unified EN/TR/Kokturk assistant recipe
$ orkhon eval --split val --metric ppl
# score a checkpoint on held-out text
$ orkhon export --format safetensors
# carve weights into the artifact the server serves
From inscription to inference
Open the model Lab, or read every line of the stack that built it.