Data Mining Tutorial: Learn Pattern Discovery from Scratch (2026)
Data mining is the process of discovering patterns, correlations, and anomalies in large datasets that would otherwise remain hidden. Working with terabyte-scale datasets across finance, healthcare, and e-commerce, I have seen how mining techniques — from association rules to clustering — extract actionable intelligence from raw data. This tutorial covers the core tasks: classification, regression, clustering, association rule mining, anomaly detection, and the preprocessing steps that make these analyses possible.
Each technique is presented with its mathematical foundation, algorithmic implementation, and real-world application. We discuss overfitting, evaluation methodology, and the ethical considerations of automated decision-making based on mined patterns.
Data Preprocessing and Cleaning
Real-world data is messy: missing values, outliers, inconsistent formats, and noise. Preprocessing typically consumes 60-80% of a data mining project's effort. Missing values can be handled by deletion (drop rows/columns), imputation (mean, median, mode, or regression-based), or model-based prediction. Normalization (min-max scaling to [0,1]) and standardization (z-score) ensure features contribute equally to distance-based algorithms. Encoding categorical variables (one-hot encoding, label encoding) converts them to numerical form.
from sklearn.preprocessing import StandardScaler, OneHotEncoder
import pandas as pd
import numpy as np
def preprocess_data(df):
df['age'].fillna(df['age'].median(), inplace=True)
df = df[(np.abs(df.select_dtypes(include=[np.number]).apply(
lambda x: (x - x.mean()) / x.std())) < 3).all(axis=1)]
scaler = StandardScaler()
numeric_cols = df.select_dtypes(include=[np.number]).columns
df[numeric_cols] = scaler.fit_transform(df[numeric_cols])
return dfAssociation Rule Mining: Apriori Algorithm
Association rule mining finds relationships like 'customers who bought X also bought Y'. A rule X -> Y has support = P(X,Y) (frequency of both occurring together) and confidence = P(Y|X) (conditional probability). The Apriori algorithm generates frequent itemsets by leveraging the downward closure property: any subset of a frequent itemset must also be frequent. It iteratively generates candidate itemsets of size k from frequent itemsets of size k-1, pruning those with infrequent subsets. Lift measures the strength of a rule beyond random co-occurrence.
def apriori(transactions, min_support):
from collections import defaultdict
item_counts = defaultdict(int)
for t in transactions:
for item in t:
item_counts[item] += 1
n = len(transactions)
frequent = {frozenset([item]): count/n
for item, count in item_counts.items()
if count/n >= min_support}
all_frequent = dict(frequent)
k = 2
while frequent:
candidates = set()
items = list(frequent.keys())
for i in range(len(items)):
for j in range(i+1, len(items)):
union = items[i] | items[j]
if len(union) == k:
candidates.add(union)
frequent = {}
for cand in candidates:
count = sum(1 for t in transactions if cand.issubset(t))
support = count / n
if support >= min_support:
frequent[cand] = support
all_frequent.update(frequent)
k += 1
return all_frequentClassification: Decision Trees and Random Forests
Classification assigns categorical labels to instances. Decision trees partition the feature space recursively, selecting splits that maximize information gain (reduction in entropy) or minimize Gini impurity. The tree depth controls the bias-variance trade-off — deep trees overfit, shallow trees underfit. Random Forests aggregate many decision trees trained on bootstrapped samples and random feature subsets, reducing variance without increasing bias. Each tree votes, and the majority prediction wins. Feature importance is measured by how much each feature reduces impurity across all trees.
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42)
rf = RandomForestClassifier(
n_estimators=100, max_depth=10, min_samples_split=5, random_state=42)
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}")
print(f"Feature importance: {rf.feature_importances_}")Clustering: K-Means and DBSCAN
Clustering groups similar instances without labeled data. K-Means partitions data into K clusters by minimizing within-cluster variance. It initializes K centroids, assigns each point to the nearest centroid, recalculates centroids as means, and repeats until convergence. The elbow method (plotting inertia vs K) helps choose K. DBSCAN identifies dense regions separated by sparse areas without requiring K — it defines clusters as maximal sets of density-connected points. DBSCAN can find arbitrarily shaped clusters and identifies outliers (noise points).
from sklearn.cluster import KMeans, DBSCAN
def kmeans_cluster(data, k):
km = KMeans(n_clusters=k, init='k-means++', n_init=10, random_state=42)
labels = km.fit_predict(data)
inertia = km.inertia_
return labels, inertia
def dbscan_cluster(data, eps=0.5, min_samples=5):
db = DBSCAN(eps=eps, min_samples=min_samples)
labels = db.fit_predict(data)
n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
n_noise = list(labels).count(-1)
print(f"Clusters: {n_clusters}, Noise points: {n_noise}")
return labelsAnomaly Detection: Isolation Forest
Anomaly detection identifies rare items or events that differ significantly from the majority. Isolation Forest isolates anomalies by randomly selecting a feature and split value, building a tree structure. Anomalies require fewer splits to isolate because they are few and different — they have shorter average path lengths in the tree. The anomaly score is based on the path length compared to the expected length for random data. This approach works well on high-dimensional data and does not assume a particular data distribution.
from sklearn.ensemble import IsolationForest
def detect_anomalies(data, contamination=0.1):
model = IsolationForest(
contamination=contamination,
random_state=42,
n_estimators=100
)
predictions = model.fit_predict(data)
anomaly_scores = model.decision_function(data)
anomalies = data[predictions == -1]
print(f"Detected {len(anomalies)} anomalies "
f"({len(anomalies)/len(data)*100:.1f}%)")
return predictions, anomaly_scoresEvaluation: Cross-Validation and Metrics
Proper evaluation prevents overfitting and provides realistic performance estimates. K-fold cross-validation splits data into K folds, trains on K-1 folds and evaluates on the held-out fold, repeating K times. Stratified folds preserve class distribution. Metrics depend on the task: accuracy, precision, recall, F1-score for classification; mean squared error (MSE) and R-squared for regression; silhouette score for clustering. Confusion matrices reveal false positive/negative patterns. ROC curves and AUC measure classifier performance across thresholds.
from sklearn.model_selection import cross_val_score
from sklearn.metrics import classification_report, confusion_matrix
def evaluate_classifier(clf, X, y):
scores = cross_val_score(clf, X, y, cv=5, scoring='f1_macro')
print(f"Cross-val F1: {scores.mean():.3f} +/- {scores.std():.3f}")
clf.fit(X, y)
y_pred = clf.predict(X)
print(classification_report(y, y_pred))
print(confusion_matrix(y, y_pred))
return scoresFrequently Asked Questions
What is the difference between supervised and unsupervised learning?
Supervised learning uses labeled data (input-output pairs) to learn a mapping from features to targets. Unsupervised learning finds patterns in unlabeled data — clustering groups similar instances, association rules find co-occurring items.
How do you handle imbalanced datasets?
Techniques include resampling (SMOTE oversamples minority, random undersampling reduces majority), class weights (penalize misclassifying minority more), anomaly detection approaches, and using metrics like precision-recall AUC instead of accuracy.
What is the curse of dimensionality?
As the number of features grows, data becomes sparse in high-dimensional space, distances become less meaningful, and models require exponentially more samples to generalize. Dimensionality reduction (PCA, t-SNE) or feature selection mitigates this.
How do you choose between a decision tree and a neural network?
Decision trees are interpretable, handle mixed data types, and require little preprocessing. Neural networks excel with unstructured data (images, text, audio) and large datasets, but are black boxes requiring extensive tuning and computational resources.
Originally published on Ayodhyyya. Last updated June 1, 2026.