PyTorch
PyTorch is a deep learning framework prized for its flexibility and Python-first developer experience. It gives you tensors (n-dimensional arrays with GPU support), automatic differentiation (autograd), and composable neural-network building blocks. Its "define-by-run" approach — models are just Python code that executes eagerly — makes it intuitive to write and debug, which is why it dominates research and is widely used in production.
The learning path is short and well-defined: understand tensors and autograd, build models by subclassing nn.Module, and write an explicit training loop. Once those click, everything else is composition.
TL;DR
- Learn tensors + autograd first.
- Build models with
nn.Module. - Write an explicit training loop (forward → loss → backward → step).
- Make training reproducible — seeds, logging, checkpoints.
Quick Example
Autograd computes gradients automatically from the operations you run:
Core Concepts
- Tensor — an n-dimensional array, optionally on GPU.
- Autograd — automatic differentiation; tracks operations on tensors with
requires_grad=True. nn.Module— the base class for models and layers.- Optimizer — updates parameters from gradients (SGD, Adam).
Defining a Model
The Training Loop
PyTorch makes the loop explicit — and the explicitness is the point:
GPU Acceleration
Move the model and each batch to the GPU when one is available:
Watch GPU memory, and profile to find whether data loading or compute is the bottleneck (often it's the data pipeline, not the model).
Best Practices
- Call
optimizer.zero_grad()every step — gradients accumulate by default. - Toggle
model.train()/model.eval()so dropout and batch-norm behave correctly. - Wrap inference in
torch.no_grad()to save memory and speed. - Log metrics and save checkpoints; validate on held-out data.
- Keep preprocessing identical between training and inference (avoid train/serving skew — see MLOps).
Common Mistakes
Forgetting to zero gradients
Evaluating in train mode (or without no_grad)
FAQ
PyTorch or TensorFlow?
Both are mature and capable. PyTorch has a more Pythonic, eager, debuggable feel and dominates research and a growing share of production. TensorFlow (with Keras) offers a polished high-level API and a strong deployment/serving ecosystem (TF Serving, TF Lite, TF.js). For new projects, PyTorch is often the default for its ergonomics; choose TensorFlow if your team or deployment targets favor its ecosystem. The concepts transfer either way.
Why do I have to call optimizer.zero_grad() every step?
Because PyTorch accumulates gradients into .grad by default rather than overwriting them — a design that's useful for techniques like gradient accumulation across mini-batches. But in a standard loop, you want fresh gradients each step, so you must clear the old ones first. Forgetting it is one of the most common PyTorch bugs and leads to incorrect, unstable updates.
What's the difference between model.train() and model.eval()?
They switch the behavior of layers that act differently during training vs inference — chiefly dropout (active in train, off in eval) and batch normalization (uses batch statistics in train, running statistics in eval). Call model.train() before training and model.eval() before validation/inference. Forgetting eval() gives misleadingly noisy or wrong predictions.
How do I make PyTorch training reproducible?
Seed all the RNGs (torch.manual_seed, numpy, Python's random), set deterministic CUDA behavior where needed, and control DataLoader worker seeding. Also pin your data version, code, and environment. Full bit-for-bit determinism on GPU can cost performance and isn't always achievable, so balance reproducibility against speed based on your needs.
Related Topics
- TensorFlow & Keras — The main alternative
- Python for Data Science — The surrounding toolkit
- Model Evaluation — Validating models
- MLOps — Deploying models
- Computer Vision · NLP — Application domains