Open Source

GPU Training Lab

Production-grade ML training notebooks. DPO, RLHF, LoRA, Diffusion, Audio, Vision — all runnable for free on Kaggle T4x2 or Google Colab.

19
Notebooks
7
Domains
Free
GPU Required
Preference Datachosen vs rejected
→
SFTinstruction tuning
→
DPO / RLHFalignment
→
EvaluationLLM-as-Judge
→
Deployed Modelaligned & evaluated

How DPO Works

DPO skips reward modeling and RL entirely. It trains the model directly on human preference pairs using a reformulation of the RLHF objective.

Old Way: RLHF (3 stages)

1 Collect preference data
2 Train a reward model
3 Run PPO to align LLM
Complex, unstable, slow
→

New Way: DPO (1 stage)

1 Collect preference data
2 Train LLM directly on pairs
_
Simple, stable, fast
loss = -log( sigmoid( β · [ log(P/P_ref)_chosen - log(P/P_ref)_rejected ] ) )

β controls conservatism. Higher β = model stays closer to the original.

DPO KTO ORPO SimPO Free GPU

Alignment Showdown: DPO vs KTO vs ORPO vs SimPO

Train the same model with 4 different alignment methods. Compare quality AND training performance side by side.

Model: Qwen2.5-0.5B • Stack: LoRA + DeepSpeed ZeRO-2
Colab View
RLHF DPO LoRA ZeRO-3 Free GPU

Full Alignment Pipeline: SFT → DPO → RLHF

The complete 3-stage alignment pipeline on Qwen2.5-0.5B. SFT teaches instruction following, DPO aligns offline, PPO/RLHF aligns online.

Model: Qwen2.5-0.5B • Stack: LoRA + ZeRO-3 (model parallelism)
Colab View
LoRA QLoRA Full Fine-Tune Free GPU

LoRA vs QLoRA vs Full Fine-Tuning

Train all 3 methods on the same model and data. Compare GPU memory, speed, and output quality side by side.

Model: TinyLlama 1.1B • Comparison: 100% vs ~1% params
Colab View
DPO LoRA Free GPU

DPO Training from Scratch

Direct Preference Optimization on TinyLlama. Learn how DPO works by training it yourself with real preference pairs.

Model: TinyLlama 1.1B • Stack: TRL DPOTrainer + LoRA
Colab View
DPO DeepSpeed Ray LoRA

Distributed DPO with DeepSpeed & Ray

Production-grade distributed training. Same code scales from 1 GPU to 100 GPUs with Ray Train orchestration.

Stack: TRL + LoRA + DeepSpeed ZeRO-2 + Ray Train
Colab View
SFT + DPO DeepSpeed LoRA Free GPU

Free Distributed SFT + DPO

Two-stage alignment on free Kaggle GPUs. SFT teaches instruction following, then DPO aligns with preferences. Auto-detects your hardware.

Platform: Kaggle 2xT4 (free) • Stack: Accelerate + ZeRO-2
Colab View
Vision LoRA QLoRA DPO Free GPU

Multimodal Fine-Tuning: LoRA + QLoRA + DPO

Fine-tune LLaVA 7B vision-language model. LoRA in 8-bit, QLoRA in 4-bit, then DPO to prefer detailed image descriptions.

Model: LLaVA 1.5 7B • Pipeline: LoRA → QLoRA → DPO → Merge
Colab View
Diffusion LoRA Multi-GPU Free GPU

Stable Diffusion LoRA Fine-Tuning

Teach Stable Diffusion a new style or concept. LoRA on UNet attention layers — tiny adapters (~1MB), same results.

Model: SD 1.5 (860M UNet) • Method: LoRA on attention layers
Colab View
ViT LoRA ZeRO-2 Free GPU

Image Classification with Vision Transformer

LoRA fine-tune ViT-base on Food-101. A text transformer that reads image patches instead of words.

Model: ViT-base 86M • Data: Food-101 (101 categories)
Colab View
Audio LoRA ZeRO-2 Free GPU

Audio: Whisper Speech Recognition

Fine-tune Whisper-small for speech-to-text with LoRA. Multi-GPU with DeepSpeed on freely available FLEURS data.

Model: Whisper-small 244M • Data: FLEURS English (500 examples)
Colab View
VLM LoRA ZeRO-2 Free GPU

Multimodal VLM Training

Fine-tune Qwen2-VL-2B on image+text data with LoRA and multi-GPU acceleration.

Model: Qwen2-VL-2B-Instruct • Stack: LoRA + ZeRO-2
Colab View
Full Fine-Tune ZeRO-3 Free GPU

Full Fine-Tuning (No LoRA)

Actually train all 124M parameters of GPT-2 end-to-end. ZeRO-3 shards the entire model across GPUs + CPU.

Model: GPT-2 124M • Data: OpenWebText subset
Colab View
LoRA ZeRO-2 Free GPU

Simple Multi-GPU Training (Starter)

The simplest possible distributed fine-tuning notebook. GPT-2 + LoRA + DeepSpeed on 500 Alpaca examples. Start here.

Model: GPT-2 124M • Data: 500 Alpaca examples
Colab View
Benchmark DDP ZeRO-2 ZeRO-3 Free GPU

Multi-GPU Benchmark: 1 vs 2 GPU Strategies

Run identical training with DDP, ZeRO-2, and ZeRO-3. See exactly how much speedup and memory savings each gives you.

Configs: 1GPU • 2GPU DDP • ZeRO-2 • ZeRO-3
Colab View
SFT DPO LoRA Free GPU

Post-Training: Watch Every Stage Transform a Model

See a raw model become an assistant. Generate text at Base → SFT → DPO and compare side-by-side. The simplest possible explanation of post-training.

Model: Qwen2.5-0.5B • Stages: Base → SFT → DPO (all visible)
Colab View
GRPO Reasoning QLoRA Free GPU

GRPO: Teach a Model to Reason (DeepSeek-R1 Method)

Train Qwen2.5-1.5B to solve math problems using pure RL. No reward model, no CoT data — just "is the answer correct?" The algorithm behind DeepSeek-R1.

Model: Qwen2.5-1.5B • Dataset: GSM8K • Algorithm: GRPO
Colab View
QLoRA 4 Benchmarks 7B Model A100

Push the Limits: 7B on Colab Pro A100

QLoRA fine-tune Qwen2.5-7B on A100 40GB. 2048-token sequences, 4-bit NF4, benchmarked on 4 Open LLM Leaderboard tasks vs Llama-2, Mistral, Llama-3.

Model: Qwen2.5-7B • Benchmarks: TruthfulQA, ARC, HellaSwag, Winogrande
Colab View
Full Fine-Tune TruthfulQA Benchmark Free GPU

Modern Full Fine-Tuning for Post-Training

Full fine-tuning with modern techniques: cosine LR, gradient checkpointing, weight decay. Benchmarked against TruthfulQA (Open LLM Leaderboard).

Model: TinyLlama 1.1B • Benchmark: TruthfulQA MC2
Colab View
Evaluation LLM Judge VLM Judge Free GPU

LLM-as-Judge Evaluation

Evaluate model outputs using LLM and VLM judges. Three methods: automated metrics, rule-based scoring, and model-as-judge. All local, no API needed.

Models: TinyLlama + LLaVA • Method: Self-contained evaluation
Colab View
Evaluation Agent Traces Retry Loop Free GPU

Multi-Trace Agent Evaluation + Retry

Treat agent reasoning as a search over executable traces. 10 strategies × 3 benchmarks (ARC / SWE / MLE-style), 5-dim weighted scoring, hybrid trace synthesis, beam-search pruning, plus a TraceEvaluator that diagnoses failures and emits concrete retry actions. Rescues 4/4 failing strategies (26/30 → 30/30).

Method: Tree-of-Thought + executable traces • Hooks: Anthropic / OpenAI / HF (optional)
Colab View