La Forge

Context

What I built

• Cleaned French Gutenberg corpus, split 98/2.
• Byte-level BPE tokeniser, 16,384 tokens (3.64 characters per token).
• Transformer written from scratch: RoPE, RMSNorm, SwiGLU, tied embeddings.
• Loop with checkpoints, resume, CSV log and sampler.

Architecture 6 blocks

1. 01 · InputCorpus279 French Gutenberg books, cleaned, split 98/2
2. 02 · ToolBPE tokeniserByte-level, 16,384 tokens, 3.64 characters per token
3. 03 · ModelTransformerFrom scratch: RoPE, RMSNorm, SwiGLU, tied embeddings
4. 04 · ToolTrainingAdamW, warmup then cosine, micro-batches of 8
5. 05 · DataCheckpointsEvery 1,000 steps, CSV log, resume
6. 06 · OutputSamplesGenerated text and loss curves

Next stages on the roadmap: SFT and DPO, GRPO, then quantised serving.

Pipeline 4 steps

1. Pre-trainingDone: 4,000 steps, 19 Oct 2026.
2. SFT + DPONext: LoRA on Qwen2.5-0.5B (a different, larger model), hand-written DPO.
3. GRPONext: reinforcement with verifiable rewards.
4. ServingNext: GGUF, quantisation, P50/P95 latencies.

Decisions the choice, the reason, the alternative ruled out

1. D1
   Accumulate gradients in micro-batches of 8, with periodic MPS cache purges.

   Logits of about 1 GB per step plus the MPS cache pushed the process to 11 GB: memory cut by four, same maths.

   Ruled outThe full batch of 32 at once: 17–19 s per step, measured.

2. D2
   Checkpoint every 1,000 steps and support resuming.

   When the machine slept at step 3,750, training resumed from step 3,000.

   Ruled outA single uninterrupted run, to redo after any stop.

3. D3
   A byte-level BPE tokeniser, trained on the corpus.

   3.64 characters per token on literary French, exact round trip checked.

4. D4
   Write the model from scratch, in PyTorch only.

   Every block is understood and checked: initial loss of 9.65, matching ln 16,384 ≈ 9.70.

   Ruled outStarting from an existing model library.

Results

Deliverables 4

• Full code: corpus download, tokeniser, model, training loop, sampler.
• Loss curve (training and validation) and measured throughput over 4,000 steps.
• Raw samples at step 4,000.
• Memory diagnosis logs, kept.

Environment

Reproduce README commands

Private code: read access under NDA, or a 30-minute live demo. Request access →

P=~/Documents/ComfyUI/.venv/bin/python
$P corpus/telecharger.py --objectif-mo 200
$P corpus/preparer.py
$P tokeniseur/entrainer.py
$P train/entrainer.py --pas 4000 --micro 8
$P train/entrainer.py --reprendre train/sortie/dernier.pt
$P train/echantillonner.py --invite "Il était une fois"

• About 1 h 20 of cumulative compute on Apple GPU (MPS), 1.1 to 1.4 s per step.
• Code and documents versioned; corpus and outputs regenerated by the scripts.