Pandas Tutorial: Learn Data Analysis from Scratch (2026)
I started using Pandas when I had to clean messy CSV exports from legacy systems — missing values, inconsistent date formats, duplicate rows. Pandas made hours of manual spreadsheet work collapse into a dozen lines of code. The core ideas are the DataFrame (a labeled, tabular data structure) and the Series (a labeled 1D array). Together they give you SQL-like operations — groupby, join, filter — inside Python.
This tutorial follows a real dataset scenario: analyzing e-commerce transaction logs. You'll load CSV data, clean it, perform aggregations, merge tables, and export results. The patterns here transfer directly to financial data, sensor logs, survey results, or any structured data you encounter.
DataFrames and Series: The Building Blocks
A DataFrame is a collection of Series that share the same index. Each column has a name and a data type (dtype). Creating a DataFrame from a dictionary maps keys to columns. The head() method previews the first rows, and info() shows column dtypes and non-null counts — the first thing I call on any new dataset.
import pandas as pd
data = {
"order_id": [1, 2, 3],
"customer": ["Alice", "Bob", "Charlie"],
"amount": [29.99, 49.99, 15.50],
"date": pd.to_datetime(["2026-01-10", "2026-01-11", "2026-01-12"])
}
df = pd.DataFrame(data)
print(df.head())
print(df.info())
print(df.dtypes)Reading and Writing Data
Pandas handles CSV, Excel, JSON, Parquet, SQL, and many other formats. read_csv() is the workhorse — I use parameters like parse_dates, dtype to enforce types, and na_values to specify missing value markers. For large files, chunking with chunksize lets you process data in streaming fashion instead of loading entirely into memory.
# Read with type hints
orders = pd.read_csv(
"orders.csv",
parse_dates=["order_date"],
dtype={"customer_id": "string"},
na_values=["", "NULL", "N/A"]
)
# Write results
orders.to_parquet("orders_clean.parquet", index=False)
orders.to_excel("orders_report.xlsx", sheet_name="Orders", index=False)Filtering, Selecting, and Sorting
Select columns by name with df['col'] or df.col — but dot notation fails if the column name has spaces or clashes with DataFrame methods. Boolean filtering uses df[df['amount'] > 30] to return rows where the condition is True. isin() and between() are convenient for set membership and range checks. Sorting by one or more columns uses sort_values().
# Filtering
high_value = orders[orders["amount"] > 50]
# Multiple conditions
recent_large = orders[
(orders["amount"] > 50) & (orders["order_date"] >= "2026-01-01")
]
# Top 10 by amount
top10 = orders.nlargest(10, "amount")
# Sort by date descending
sorted_df = orders.sort_values("order_date", ascending=False)GroupBy Operations and Aggregations
GroupBy splits data into groups, applies a function, and combines results. It's the Pandas equivalent of SQL's GROUP BY. I use agg() to apply multiple aggregation functions to different columns at once — sum, mean, count, custom lambdas. The result is a DataFrame with a MultiIndex, which I usually reset for clean output.
summary = orders.groupby("customer").agg(
total_spent=pd.NamedAgg(column="amount", aggfunc="sum"),
order_count=pd.NamedAgg(column="order_id", aggfunc="count"),
avg_order=pd.NamedAgg(column="amount", aggfunc="mean")
).reset_index()
print(summary.sort_values("total_spent", ascending=False))Merging and Joining DataFrames
Combining multiple tables is where Pandas shines for data preparation. merge() works like SQL joins — inner, left, right, outer — on key columns. concat() stacks DataFrames vertically (more rows) or horizontally (more columns). I often load customers and orders from separate files and merge them before analysis.
customers = pd.read_csv("customers.csv") # id, name, city
orders = pd.read_csv("orders.csv") # order_id, customer_id, amount
detailed = orders.merge(
customers,
left_on="customer_id",
right_on="id",
how="left"
)
# City-wise totals
city_totals = detailed.groupby("city")["amount"].sum().reset_index()Handling Missing Data and Duplicates
Real data is messy. isnull().sum() shows missing counts per column. Fill or drop: fillna() with a constant, forward-fill, or interpolate; dropna() removes rows with any or all missing values. duplicated() finds duplicates, and drop_duplicates() removes them while keeping the first or last occurrence. I always check these at the start of an analysis.
print(orders.isnull().sum())
# Fill missing values
orders["amount"].fillna(orders["amount"].median(), inplace=True)
orders["category"].fillna("Unknown", inplace=True)
# Drop duplicates
orders.drop_duplicates(subset=["order_id"], keep="first", inplace=True)
# Forward fill time series
orders.sort_values("order_date", inplace=True)
orders["running_total"] = orders["amount"].cumsum()Frequently Asked Questions
What's the difference between Pandas and SQL?
Pandas is more flexible for complex transformations (custom functions, text processing, time series resampling) and integrates with Python plotting libraries. SQL is better for querying databases directly and for very large datasets that don't fit in memory. I use both: SQL to extract and aggregate, Pandas to clean and visualize.
How do I speed up Pandas on large datasets?
Use appropriate dtypes (category for strings with few unique values), avoid chained indexing, use vectorized operations over apply(), and read only needed columns with usecols. For datasets larger than RAM, use Dask, Polars, or chunked processing.
What is the difference between loc and iloc?
loc selects by label (index value or column name), while iloc selects by integer position. Use df.loc[0:5] for the first 6 rows by label, df.iloc[0:5] for rows by position. iloc is exclusive on the end, loc is inclusive.
How do I apply a custom function to a column?
Use df['col'].apply(lambda x: ...) for element-wise operations, or df['col'].transform() for group-wise transformations. For operations that benefit from vectorization, try to use built-in Pandas methods first — they're faster than apply.
Originally published on Ayodhyyya. Last updated June 1, 2026.