Deep Learning Tutorial: Learn Neural Networks from Scratch (2026)
Deep learning fascinated me from the moment I saw a neural network learn to recognize handwritten digits. There is something magical about watching mathematical operations give rise to intelligence. By 2026, deep learning has revolutionized computer vision, natural language processing, and scientific computing. But the fundamentals remain the same: neurons, weights, activation functions, and backpropagation. Understanding these building blocks gives you the foundation to work with any architecture from CNNs to Transformers.
The Neuron: Building Block of Neural Networks
An artificial neuron is inspired by biological neurons but much simpler. It takes multiple inputs, multiplies each by a weight, sums them up, adds a bias, and passes the result through an activation function. The activation function introduces non-linearity, which is what allows neural networks to learn complex patterns. Without non-linearity, stacking layers would be equivalent to a single linear transformation.
The first neural network I built had just one neuron. It could learn to classify points on either side of a line. That simple demonstration taught me the core mechanics: forward propagation computes the output, the loss measures error, and backpropagation adjusts weights to reduce the loss.
import numpy as np
def neuron(x, weights, bias):
z = np.dot(x, weights) + bias
return 1 / (1 + np.exp(-z)) # sigmoid activation
x = np.array([0.5, 0.8, -0.2])
w = np.array([0.2, -0.1, 0.5])
print(neuron(x, w, 0.1))Activation Functions and Their Roles
Activation functions decide whether a neuron should fire. The sigmoid function squashes values between 0 and 1, making it useful for binary classification output layers. ReLU, which outputs max(0, x), is the default for hidden layers because it avoids the vanishing gradient problem that plagues sigmoid. Tanh outputs values between -1 and 1, which helps center data.
I spent months using sigmoid in hidden layers and wondering why my deep networks would not train. Switching to ReLU was like unlocking a hidden level. The vanishing gradient problem occurs because sigmoid's gradient is near zero for extreme values, so deep networks stop learning. ReLU's gradient is either 0 or 1, which flows much better through many layers.
def relu(x):
return np.maximum(0, x)
def sigmoid(x):
return 1 / (1 + np.exp(-x))
def tanh(x):
return np.tanh(x)
x = np.linspace(-5, 5, 100)
activation_output = relu(x)Backpropagation and Gradient Descent
Backpropagation is the algorithm that makes deep learning possible. It computes the gradient of the loss with respect to every weight in the network by applying the chain rule from calculus. Starting at the output layer, it calculates how much each weight contributed to the error and propagates that information backward through the network.
Understanding backpropagation mathematically was the hardest part of my deep learning journey. But you do not need to implement it manually in practice, modern frameworks handle it automatically. What matters is understanding the intuition: each weight is adjusted proportionally to its contribution to the error, in the direction that reduces the error.
# Manual backpropagation for a single neuron
z = np.dot(x, w) + b
y_pred = sigmoid(z)
loss = (y_pred - y_true) ** 2
dz = 2 * (y_pred - y_true) * sigmoid(z) * (1 - sigmoid(z))
dw = np.outer(x, dz)
db = dz
w -= learning_rate * dwConvolutional Neural Networks for Computer Vision
CNNs revolutionized computer vision by replacing fully connected layers with convolutional filters that scan across images. Each filter detects a specific pattern: edges, textures, shapes, and eventually complex objects. The hierarchical nature of CNNs means early layers detect low-level features and deeper layers combine them into high-level concepts. Pooling layers downsample the representation, reducing computation and providing translation invariance.
I built my first CNN to classify cats versus dogs and was amazed that the filters learned to detect ears, eyes, and whiskers automatically. The network discovered these features without any human guidance. This is the power of representation learning: deep learning finds the right features for the task.
import tensorflow as tf
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(64, 64, 3)),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(2, activation='softmax')
])Recurrent Neural Networks and LSTMs for Sequences
RNNs process sequential data by maintaining a hidden state that captures information from previous time steps. This makes them suitable for time series, text, and audio. However, simple RNNs suffer from vanishing gradients over long sequences. LSTMs solve this with a gating mechanism that can remember information for thousands of steps, deciding what to keep and what to forget.
I trained an LSTM to generate Shakespearean text character by character. After a few hours of training, it produced grammatically correct sentences that sounded vaguely Elizabethan. The network had learned the rhythm and structure of the language without any explicit grammar rules. LSTMs remain relevant in 2026 for time series forecasting despite Transformers dominating NLP.
model = tf.keras.Sequential([
tf.keras.layers.LSTM(128, return_sequences=True, input_shape=(100, 50)),
tf.keras.layers.LSTM(64),
tf.keras.layers.Dense(vocab_size, activation='softmax')
])
model.compile(optimizer='adam', loss='categorical_crossentropy')Regularization Techniques for Deep Networks
Deep neural networks have millions of parameters, making them prone to overfitting. Regularization techniques prevent this. Dropout randomly deactivates a fraction of neurons during training, forcing the network to learn redundant representations. Batch normalization stabilizes training by normalizing layer inputs. Data augmentation generates training variations to expand your dataset artificially.
In my experience, the combination of dropout with rate 0.5 after dense layers, batch normalization before activation, and aggressive data augmentation for image tasks reliably prevents overfitting. I once improved a model's validation accuracy from 72% to 88% just by adding these three techniques, with zero changes to the architecture.
model = tf.keras.Sequential([
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dropout(0.3),
tf.keras.layers.Dense(10, activation='softmax')
])Frequently Asked Questions
What hardware do I need for deep learning?
You can start with any modern laptop for small models. For serious work, an NVIDIA GPU with at least 8 GB VRAM is recommended. Cloud options like Google Colab, AWS, and Lambda Labs offer GPU instances for hourly rental.
What is the difference between a neural network and deep learning?
Deep learning refers to neural networks with multiple hidden layers typically more than two. A shallow network has one or two hidden layers. Deep networks can learn more complex hierarchical representations.
How do I choose the number of layers and neurons?
Start simple and add complexity only when needed. A good rule is to double the number of neurons until performance plateaus then reduce slightly. Use cross-validation to compare architectures. Transfer learning from pretrained models is often better than designing from scratch.
Why does my neural network not converge?
Common causes: learning rate too high or too low, vanishing/exploding gradients, bad weight initialization, insufficient data, or wrong activation function. Try decreasing learning rate, using batch normalization, and switching to ReLU activations.
Originally published on Ayodhyyya. Last updated June 1, 2026.