PyTorch Tutorial: Learn Deep Learning from Scratch (2026)
I switched to PyTorch from TensorFlow in 2019 when I got tired of debugging tf.Session graphs. PyTorch's define-by-run approach — where the computation graph is built on the fly — made debugging feel like normal Python. You can print a tensor's value, inspect its shape, and use standard control flow inside model forward passes. That immediacy has made PyTorch the dominant framework in research and increasingly in production.
We'll build a neural network from the ground up: tensors, autograd, a custom dataset, data loaders, a training loop, and model evaluation. The example is a classifier for a synthetic 2D dataset, so you can visualize decision boundaries and see exactly what the network learns at each epoch.
Tensors: The GPU-Accelerated Array
A PyTorch tensor is like NumPy's ndarray but with GPU support and automatic differentiation. Tensors track operations when requires_grad=True, building a computation graph that enables gradient computation. Many tensor operations mirror NumPy — torch.zeros, torch.randn, tensor.shape, and indexing syntax — but they also support .to('cuda') to move data to a GPU.
import torch
x = torch.tensor([[1.0, 2.0], [3.0, 4.0]], requires_grad=True)
y = x.sum()
y.backward() # Compute gradients
print(x.grad)
# tensor([[1., 1.],
# [1., 1.]])
# GPU check
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f"Using {device}")Autograd: Automatic Differentiation
Autograd is the engine that powers training. When you call .backward() on a scalar loss, it computes gradients for every tensor in the graph that has requires_grad=True. Gradients accumulate by default, so you zero them with optimizer.zero_grad() before each backward pass. Understanding autograd's behavior — especially gradient accumulation and in-place operations — saves hours of debugging.
a = torch.randn(3, requires_grad=True)
b = a * 2
c = b ** 2
loss = c.mean()
loss.backward()
print(a.grad) # d(loss)/d(a)
# Gradients accumulate if not zeroed
a.grad.zero_() # ResetBuilding a Model with nn.Module
The nn.Module base class provides parameter registration, train/eval mode switching, and a container for layers. You override __init__ to define layers and forward to specify the computation. Parameters are automatically tracked and returned by .parameters(). This design makes it straightforward to compose complex architectures from smaller modules.
import torch.nn as nn
class TwoLayerNet(nn.Module):
def __init__(self, input_size, hidden_size, num_classes):
super().__init__()
self.fc1 = nn.Linear(input_size, hidden_size)
self.relu = nn.ReLU()
self.fc2 = nn.Linear(hidden_size, num_classes)
def forward(self, x):
out = self.fc1(x)
out = self.relu(out)
out = self.fc2(out)
return out
model = TwoLayerNet(784, 256, 10)
print(model)Data Loading with Dataset and DataLoader
Dataset abstracts the data source — you implement __len__ and __getitem__. DataLoader wraps a Dataset and adds batching, shuffling, and parallel loading with multiprocessing workers. I often write custom Datasets for CSV files, images on disk, or API responses. DataLoader's pin_memory=True speeds GPU transfers by pinning host memory pages.
from torch.utils.data import Dataset, DataLoader
import pandas as pd
class CustomDataset(Dataset):
def __init__(self, csv_path):
self.df = pd.read_csv(csv_path)
self.features = self.df.iloc[:, :-1].values.astype('float32')
self.labels = self.df.iloc[:, -1].values.astype('int64')
def __len__(self):
return len(self.df)
def __getitem__(self, idx):
return torch.tensor(self.features[idx]), torch.tensor(self.labels[idx])
dataset = CustomDataset('data.csv')
loader = DataLoader(dataset, batch_size=32, shuffle=True, num_workers=4)The Training Loop and Optimization
A typical training loop iterates over epochs, batches, and gradient steps. Each step: forward pass to compute predictions, calculate loss, backward pass for gradients, and optimizer.step() to update weights. I also track running loss and accuracy, and use tqdm for progress bars. The loop structure is the same whether you're training a linear model or a transformer.
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
for epoch in range(10):
running_loss = 0.0
for inputs, labels in loader:
inputs, labels = inputs.to(device), labels.to(device)
optimizer.zero_grad()
outputs = model(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
running_loss += loss.item() * inputs.size(0)
epoch_loss = running_loss / len(loader.dataset)
print(f"Epoch {epoch+1}, Loss: {epoch_loss:.4f}")Model Evaluation and Saving
After training, switch to model.eval() to disable dropout and batch norm updates. Use torch.no_grad() to disable gradient tracking during inference — saves memory and speeds computation. torch.save(model.state_dict(), 'model.pth') saves only the parameters, not the architecture. To load, instantiate the model class first, then load_state_dict().
model.eval()
correct, total = 0, 0
with torch.no_grad():
for inputs, labels in loader:
inputs, labels = inputs.to(device), labels.to(device)
outputs = model(inputs)
_, predicted = torch.max(outputs, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
print(f"Accuracy: {100 * correct / total:.2f}%")
# Save and load
torch.save(model.state_dict(), 'model.pth')
model.load_state_dict(torch.load('model.pth'))Frequently Asked Questions
Should I use PyTorch or TensorFlow in 2026?
PyTorch is the default for research and most production cases. TensorFlow still has an edge in mobile deployment (TFLite) and large-scale serving (TF Serving), but PyTorch's torch.compile and TorchServe have narrowed the gap significantly.
How do I debug NaN losses during training?
Check for exploding gradients by logging gradient norms. Reduce learning rate, add gradient clipping (torch.nn.utils.clip_grad_norm_), and verify input data has no NaN or Inf values. Also check that your loss function matches the output activation.
What is the difference between model.train() and model.eval()?
train() enables dropout and batch norm tracking. eval() freezes them for deterministic inference. Forgetting to switch to eval() before evaluation is one of the most common bugs that inflates accuracy estimates.
How do I use a GPU with PyTorch?
Check torch.cuda.is_available(), then move tensors and models with .to('cuda'). Be aware that CPU-GPU transfers are slow — minimize them. Use pin_memory=True in DataLoader for faster host-to-device transfers.
Originally published on Ayodhyyya. Last updated June 1, 2026.