Thinking Machines Lab just released Inkling, their first model trained from scratch, weights are open, fine-tunable on Tinker. The lab pitches it as a base for customization.
What is Inkling?
Inkling is a Mixture-of-Experts transformer with 975B total parameters and 41B active. It supports a context window of up to 1M tokens. Pretraining covered 45 trillion tokens of text, images, audio, and video. Inputs accept text, images, and audio; output is UTF-8 text only.
The research team also previewed Inkling-Small, a 276B-parameter MoE with 12B active parameters. It matches or exceeds its larger sibling on many benchmarks, and its weights arrive once testing finishes. Because customization/finetuning is the key differentiator, the architecture matters here very much.
Inside The Architecture
The model architecture includes a 66-layer decoder-only transformer with a sparse MoE feed-forward backbone. Each MoE layer holds 256 routed experts plus 2 shared experts. Six routed experts activate per token, and both shared experts activate on every token. A sigmoid-based router handles selection, using an auxiliary-loss-free load-balancing bias. Routed and shared scores are normalized jointly, then used to weight combined outputs. The MoE design largely follows DeepSeek-V3.
Attention departs from convention. Sliding-window and global layers interleave at a 5:1 ratio with 8 KV heads. Position uses a relative positional embedding rather than RoPE, which the lab reports extrapolates better. Short convolutions are applied after key and value projections, and on residual branch outputs.
Multimodality is encoder-free. Audio enters as dMel spectrograms, and images become 40×40 pixel patches through a four-layer hMLP. A lightweight embedding layer projects both, then the decoder processes them jointly with text tokens.
Training used Muon for large matrix weights and Adam for other parameters, on NVIDIA GB300 NVL72 systems. Post-training bootstrapped from SFT on synthetic data, including data generated by Kimi K2.5. Most compute went to asynchronous RL, scaled past 30M rollouts, improving log-linearly throughout. That RL run also produced the model’s main control surface.