PyTorch Tutorial: Learn Neural Networks from Scratch (2026)
PyTorch changed how I think about deep learning. Its define-by-run paradigm means your neural network is built on the fly as Python executes your code, not pre-compiled into a static graph. This makes debugging natural and prototyping fast. By 2026, PyTorch has become the dominant framework in academic research and is rapidly gaining ground in production thanks to TorchServe and the PyTorch ecosystem. What I love most is the transparency: you see exactly what happens at each step, which is invaluable when you are learning.
Setting Up PyTorch and Understanding Tensors
PyTorch installation is straightforward with the official pip command tailored to your hardware. The tensor is the fundamental building block, analogous to NumPy's ndarray but with GPU acceleration built in. You can create tensors from lists, NumPy arrays, or use factory functions like torch.zeros, torch.ones, and torch.rand.
When I first switched from NumPy to PyTorch tensors, the transition was nearly seamless because the API mirrors NumPy closely. The major difference is that PyTorch tensors have a .grad attribute and a .requires_grad flag that enables automatic differentiation, which is the engine behind neural network training.
import torch
x = torch.randn(3, 4, requires_grad=True)
y = x.pow(2).sum()
y.backward()The Autograd Engine: Automatic Differentiation
Autograd is PyTorch's automatic differentiation system. When you set requires_grad=True on a tensor, PyTorch records every operation performed on it in a computational graph. Calling .backward() on the final scalar loss computes gradients for all tensors in the graph. These gradients are accumulated in the .grad attribute of each leaf tensor.
Understanding autograd is the key to mastering PyTorch. I remember being confused about why gradients accumulate instead of resetting each iteration. The reason is efficiency: it allows gradient accumulation across multiple minibatches for large models that do not fit in GPU memory. You just need to call .zero_grad() on your optimizer at the start of each training loop.
x = torch.tensor([1., 2., 3.], requires_grad=True)
y = (x ** 2).mean()
y.backward()
x.gradBuilding Neural Networks with torch.nn
The torch.nn module provides building blocks for neural networks: layers, activation functions, loss functions, and containers. You create a model by subclassing nn.Module and defining layers in __init__ and the forward pass in the forward method. This is where PyTorch's define-by-run nature shines: you can use Python control flow like loops and conditionals inside forward.
I once built a dynamic architecture that used different dropout rates based on the input length. In a static graph framework this would have been extremely difficult, but in PyTorch it was just a conditional statement inside forward. The flexibility is unmatched for research and experimentation.
import torch.nn as nn
class Net(nn.Module):
def __init__(self):
super().__init__()
self.layers = nn.Sequential(
nn.Linear(784, 256),
nn.ReLU(),
nn.Linear(256, 10))
def forward(self, x):
return self.layers(x)Training Loops and Optimization
Writing a training loop manually is the best way to understand what frameworks like Keras do under the hood. The pattern is always the same: forward pass, compute loss, zero gradients, backward pass, and update weights. PyTorch gives you full visibility into each step, which is both empowering and educational.
The torch.optim package provides all the standard optimizers: SGD with momentum, Adam, RMSprop, and more. I have found that Adam with a learning rate of 1e-3 is a solid default for most problems. The key hyperparameters to tune are learning rate, batch size, and weight decay.
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
for epoch in range(10):
logits = model(x_train)
loss = nn.functional.cross_entropy(logits, y_train)
optimizer.zero_grad()
loss.backward()
optimizer.step()Datasets, DataLoaders, and Transforms
PyTorch's data handling is elegant and efficient. The Dataset class abstracts your data source, and the DataLoader wraps it with batching, shuffling, and parallel loading using multiple workers. torchvision and torchaudio extend this with ready-to-use datasets and transforms for images and audio.
One optimization that made a huge difference in my training speed was increasing the num_workers parameter in DataLoader to match my CPU core count. This loads the next batch in parallel while the GPU is processing the current batch, eliminating I/O bottleneck. Simple change, massive speedup.
from torch.utils.data import DataLoader, TensorDataset
dataset = TensorDataset(x_train, y_train)
loader = DataLoader(dataset, batch_size=32, shuffle=True)
x_batch, y_batch = next(iter(loader))GPU Acceleration and Distributed Training
Moving your model to a GPU in PyTorch is as simple as calling .to('cuda') on your model and tensors. PyTorch also supports DistributedDataParallel for multi-GPU training across multiple nodes. The framework handles gradient synchronization so your model trains identically whether on one GPU or a hundred.
I migrated a research model from single GPU to four GPUs using DistributedDataParallel and the only code change was wrapping the model and adjusting the learning rate linearly with the number of GPUs. The training time dropped from 8 hours to just over 2 hours with near-perfect scaling efficiency.
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net().to(device)
x_batch = x_batch.to(device)
model = nn.DataParallel(model)Frequently Asked Questions
What is the difference between torch.nn and torch.nn.functional?
torch.nn contains stateful modules with learnable parameters like Linear and Conv2d. torch.nn.functional provides stateless functions like relu and cross_entropy that do not maintain parameters. Use nn.Module for layers with parameters and F for activation functions.
How do I debug PyTorch models effectively?
Use print statements inside the forward method since PyTorch executes eagerly. You can also use torch.set_anomaly_enabled(True) to detect NaN gradients. For visualizing the computation graph, use torchviz or TensorBoard integration.
What is the difference between model.train() and model.eval()?
model.train() enables dropout and batch normalization training behavior. model.eval() disables them for inference, making the model deterministic. Always call model.eval() before evaluation or deployment.
How do I save and load a PyTorch model?
Save the state_dict with torch.save(model.state_dict(), 'model.pth'). Load by creating a new model instance and calling model.load_state_dict(torch.load('model.pth')). For full checkpointing including optimizer state, save a dictionary.
Originally published on Ayodhyyya. Last updated June 1, 2026.