Introduction
A reasoning LLM is a language model trained not just to predict the next word, but to work through a problem step by step and verify its own conclusions before answering the approach behind models like OpenAI’s o1 and DeepSeek’s R1. This guide condenses a practical path to building a compact 300–400M parameter GPT-style model, teaching it to reason, and reward-tuning it for correctness, all on a single rented GPU for under $100.
1. Building a Reasoning LLM : Decide What You’re Building
Scale: 300–400M parameters small enough to pretrain on one GPU in a day or two, large enough to show real reasoning after tuning.
Domain: pick one with automatically verifiable answers. Math (MATH dataset) works well since correctness is checkable symbolically, with no human labeling needed.
Compute: rent a single high-VRAM GPU (H100 or MI300X) by the hour rather than a cluster.
Scaling: use Chinchilla-style compute-optimal ratios for a 350M model, aim for roughly 5–10B pretraining tokens.
2. The Role of Architecture in Reasoning LLMs: Harnessing the Power of Modern Transformer Design
The forward pass: token embedding → N transformer blocks (pre-norm residual, each with RMSNorm + GQA attention with RoPE and QK-norm, plus RMSNorm + SwiGLU FFN) → final RMSNorm → linear projection to vocabulary logits.
A 350M-scale build typically uses ~22 blocks, 1024-dim embeddings, 16 attention heads grouped into 4 KV groups, and a 1024-token context. None of the components are novel individually, what’s notable is that this exact combination (RoPE, RMSNorm, QK-norm, GQA, SwiGLU) is what current efficient open models like Qwen3 converge on, because each piece targets a different bottleneck.
Tokenization: GPT-2 BPE via tiktoken (~50k vocab) mature and fast, avoiding the cost of training a custom tokenizer on a small dataset. Whether to tie the input embedding and output projection weights is a real design decision: tying saves ~51M parameters but couples input and output representations.
RoPE (Rotary Position Embeddings): instead of adding a position vector to embeddings, RoPE rotates query/key vectors by an angle that depends on position, pairing up dimensions and rotating each pair at a different frequency. The key property: the dot product between a rotated query and key depends only on their relative distance, not absolute position making attention relative-position-aware automatically and improving generalization past the trained context length. RoPE applies to queries and keys only, inside every attention layer.
RMSNorm + QK-norm: RMSNorm simplifies LayerNorm by dropping mean-centering and normalizing only by root-mean-square, nearly as effective, cheaper to compute, and now the default in modern LLMs. QK-norm applies RMSNorm directly to queries and keys before computing attention scores, preventing their magnitudes from drifting and causing training instability (e.g., attention collapse or loss spikes) at scale.
Grouped-Query Attention (GQA): standard multi-head attention gives every head its own K/V, which is expensive to cache during inference. Multi-query attention shares one K/V across all heads (cheap but hurts quality). GQA is the middle ground grouping heads (e.g., 16 heads into 4 groups) so each group shares one K/V projection, cutting KV cache size 4x while preserving most of the representational capacity. This matters because the KV cache, not raw compute, is usually the real inference bottleneck. A useful correctness check: token-by-token generation with the KV cache should produce identical logits to a single full forward pass, a mismatch signals a caching or masking bug.
SwiGLU FFN: replaces a simple activation with a gating mechanism two projections of the input, one passed through Swish/SiLU and multiplied elementwise by the other, before a final projection. This needs three weight matrices instead of two, so the hidden dimension is shrunk to about 2/3 size to keep parameter count comparable. Despite the same budget, SwiGLU consistently outperforms ReLU/GELU FFNs, and is now standard across LLaMA, PaLM, and Qwen.
Before touching training data, unit-test the model in isolation: check output logit shapes, and verify the KV-cache-vs-full-forward equivalence. Catching bugs here is far cheaper than debugging them mid-run.
3. Building a Reasoning LLM : Training Stages Involved
Stage 1 Pretraining: standard causal LM cross-entropy on a curated web corpus (FineWeb-Edu is a strong choice). Tune batch size and gradient accumulation to fit VRAM at your context length. Save checkpoints regularly, and build a thin supervisor that auto-restarts from the last checkpoint on crash, stops on loss plateau, and stops at a wall-clock deadline. A healthy loss curve goes from ~10–11 down toward ~3 or lower over several billion tokens.
Stage 2 Supervised fine-tuning on chain-of-thought data: teaches the base model the shape of a reasoning response not correct yet, just format (…reasoning… followed by a boxed final answer). Source this from datasets like GSM8K or MATH paired with CoT solutions. Fine-tune for a couple epochs at a lower learning rate than pretraining. Success looks like validation loss dropping substantially and the model reliably producing the correct format on held-out prompts, even if the reasoning inside is often wrong that’s expected, and it’s what the next stage fixes.
Stage 3 GRPO (reinforcement learning from verifiable rewards): this is the stage that actually produces reasoning behavior. For each problem, sample multiple candidate responses from the current policy, and grade each with a symbolic verifier that extracts the boxed answer and compares it to ground truth (reward = 1 or 0, no human labeling or learned reward model). Compute a group-relative advantage rewarding responses that outperform their siblings within the same group which is what makes this GRPO rather than vanilla PPO (no separate value/critic model needed). Update the policy with a PPO-style clipped objective plus a KL penalty back to the SFT checkpoint, to prevent drifting into incoherent text.
Getting the verifier right is the crux of this stage: it must robustly extract answers from free-form text (handling formatting variance, equivalent numeric representations) before any GPU time is spent, since a buggy verifier silently trains on garbage reward signal.
Watch for two failure modes: reward collapse (mean reward flatlines near zero, often a broken verifier or overly hard problems) and policy drift (outputs become incoherent from over-optimizing reward, usually meaning the KL penalty is too weak). Run a live monitor during training to catch either early. Success looks like mean reward trending upward, and more tellingly the SFT-only model reasoning indefinitely without committing to an answer on hard problems, versus the GRPO model reliably converging to one.
4. Evaluation for Reasoning LLM : Evaluate All Three Checkpoints Together
Compare base, SFT, and GRPO checkpoints side by side on: perplexity on held-out text (should stay similar across all three), format compliance rate (jumps after SFT), pass@k on held-out math problems (should improve from SFT to GRPO), inference throughput (should stay flat, since none of this changes the architecture), and model footprint at different precisions. Qualitative side-by-side samples on a few fixed prompts are often more convincing than any single metric.
5. Building a Reasoning LLM : Package and Publish
Publish base, SFT, and final RL-tuned checkpoints as separate Hugging Face repos so others can see what each stage contributed. Install huggingface_hub, log in, bundle model weights (.safetensors), a config file, and tokenizer files, then create a repo per checkpoint and upload with upload_folder. Write a model card for each repo covering architecture, training data, and sample generations.
6. Looking Ahead
Verifiable-reward training today is dominated by math and code because correctness is easy to check automatically; the next frontier is extending symbolic verification to logic puzzles, planning, and multi-step tool use. Compute-optimal scaling and RL are narrowing the gap between small and large models on reasoning tasks. And as small reasoning models grow more capable, running them entirely on single-GPU infrastructure becomes realistic for far more builders.
Key Challenges
Verifier correctness: a buggy verifier silently corrupts the reward signal to validate it before spending GPU time on RL.
Training instability: RLVR is prone to reward collapse and policy drift, requiring live monitoring and a well-tuned KL penalty.
Compute discipline: unattended runs on rented infrastructure can quietly burn money without automated restarts, plateau detection, and wall-clock limits.
Conclusion
A carefully chosen architecture (RoPE, RMSNorm/QK-norm, GQA, SwiGLU), a compute-optimal pretraining budget, chain-of-thought fine-tuning, and reinforcement learning from verifiable rewards can together produce a genuine reasoning model on a single rented GPU for well under $100 no longer the exclusive domain of large labs.
References: Sebastian Raschka’s Build a Large Language Model (From Scratch) and Build a Reasoning Model (From Scratch) are excellent practical companions to the papers behind RoPE, RMSNorm, SwiGLU, GQA, GRPO, and DeepSeek-R1.
The post Build a Reasoning LLM from Scratch: A Complete Guide to GRPO, RoPE & Pretraining. appeared first on Spritle software.