Read, clean, and aggregate CSV, Excel, and DB data with Pandas. Indexing, merging, how groupby works inside, and performance optimization β the core tools for getting data quality right before moving into AI/ML.
Beginner
Updated Oct 1, 2026
~4 min read
11 sections
7 code examples
Web IDE lab included
πΌ
Pandas Web IDE
Run code in the browser with nothing to install and learn through step-by-step examples.
Read, clean, and aggregate CSV, Excel, and DB data with Pandas. Indexing, merging, how groupby works inside, and performance optimization β the core tools for getting data quality right before moving into AI/ML. Rather than listing concepts, this guide is organized so you can follow the content in the order you have to make decisions on a real project.
Key perspective
Backend / systems development
Before syntax, pin down the boundaries a request crosses: validation, processing, storage, and response.
Data cleaningCSV/Excel analysisAggregate reportsML preprocessing
Architecture diagrams
To make what you read stick, it helps to capture the flow as a picture first. The two diagrams below are reference maps you can keep coming back to while learning Pandas.
Learning flow
Rendering diagramβ¦
Architecture view
Rendering diagramβ¦
What is Pandas?
When you first open Pandas, the big picture matters more than individual commands. This section starts with the problems the concepts ahead were created to solve.
Pandas is a Python library for working with tabular data. Centered on the DataFrame, it handles filtering, missing values, group aggregation, joins, and file I/O concisely.
Object
Description
Series
A one-dimensional labeled array.
DataFrame
Two-dimensional table data with rows and columns.
Index
The row/column labels and the basis for alignment.
Loading data
Here you look at Loading data alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Read a variety of data sources such as CSV, Excel, and JSON into a DataFrame.
PYTHON
import pandas as pddf = pd.read_csv("orders.csv")print(df.head())print(df.info())
Indexing & filtering
Here you look at Indexing & filtering alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
`loc` accesses rows/columns by label (name) and `iloc` by integer position. Boolean indexing, where a condition goes inside the square brackets, plays the same role as the WHERE clause in SQL.
indexing.pyPYTHON
# loc β by label (column name, index value)df.loc[0, "amount"]df.loc[df["amount"] > 10000, ["user_id", "amount"]]# iloc β by integer position (like cell coordinates in a spreadsheet)df.iloc[0, 2] # row 0, column 2df.iloc[:5, :3] # first 5 rows, first 3 columns# Boolean indexing β select only the rows that meet the conditionhigh_value = df[(df["amount"] > 10000) & (df["month"] == "2026-01")]
Approach
Based on
Example
loc
Label (column name, index value)
df.loc[3, "amount"]
iloc
Integer position
df.iloc[3, 1]
Boolean indexing
The result of a condition (True/False)
df[df["amount"] > 100]
Cleaning
Here you look at Cleaning alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Analysis results are only stable once missing values, type conversions, and derived columns have been sorted out first.
Here you look at Combining (merge/concat) alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
When combining several tables into one, `concat` simply stacks them top/bottom or left/right, while `merge` joins them on a common key like a SQL JOIN. The `how` option decides what happens to rows that do not match.
merge.pyPYTHON
orders = pd.DataFrame({"order_id": [1, 2, 3], "user_id": [10, 20, 10]})users = pd.DataFrame({"user_id": [10, 20, 30], "name": ["Ada", "Linus", "Grace"]})# INNER JOIN β only user_ids present on both sides remaininner = orders.merge(users, on="user_id", how="inner")# LEFT JOIN β every order is kept, NaN where there is no matchleft = orders.merge(users, on="user_id", how="left")# Plain stacking (data with the same columns, top to bottom)combined = pd.concat([orders, orders], ignore_index=True)
how
Result
inner (default)
Keeps only rows whose key matches on both sides
left
Keeps every row of the left DataFrame, NaN where there is no match
outer
Keeps every row of both sides, NaN where there is no match
Aggregation (Split-Apply-Combine)
Here you look at Aggregation (Split-Apply-Combine) alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
groupby works in three stages: "split into groups (Split) β apply a function to each group (Apply) β combine the results again (Combine)". Once you understand this model, you can predict why the result looks the way it does whatever you pass to agg.
Here you look at Performance optimization alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Row-by-row `apply` and `iterrows` are simple to write but tens to hundreds of times slower than vectorized operations. Express whole-column operations with the vectorized methods of Pandas/NumPy wherever possible.
performance.pyPYTHON
# Slow β loops one row at a time at the Python leveldf["discounted"] = df.apply(lambda row: row["amount"] * 0.9 if row["vip"] else row["amount"], axis=1)# Fast β a vectorized operation (array math implemented in C underneath)df["discounted"] = df["amount"].where(~df["vip"], df["amount"] * 0.9)# For large CSVs, specify only the columns/types you need up front to save memorydf = pd.read_csv("orders.csv", usecols=["user_id", "amount"], dtype={"user_id": "int32"})
Saving
Here you look at Saving alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Save the cleaned result in a format that suits the next stage, such as CSV, Parquet, or Excel.
Pandas practical design is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
With Pandas, separate notebook analysis code from reusable transform functions. Stating the schema, dtype, and missing value policy explicitly makes the pipeline stable.
Decision point
Question to ask
Practical guideline
Boundaries
Which parts of the Pandas code are most likely to change?
Separate input/output, configuration, external integrations, and the core rules.
State
Where is state created and where does it go away?
Make the owner and lifecycle of state visible in the code.
Failure
What does the caller receive on failure?
Decide the timeout, fallback, and error contract first.
Pandas operating standards
This section covers Pandas operating standards from a practical perspective. Instead of memorizing the concept, focus on the situations in which you would reach for it.
With large data, copy cost, the object dtype, groupby/join memory use, and the chunk processing strategy matter.
Pandas verification strategy
Pandas verification strategy is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
For data quality, test the row count, null ratio, unique constraints, and distribution drift.
Quality axis
How to verify
Definition of done
Correctness
Automate the normal/failure cases.
The core scenarios pass reproducibly.
Regression prevention
When fixing a bug, leave the same case behind as a test.
The same failure is not deployed again.
Operability
Check the logs, metrics, and alerts.
There is a path for tracing the cause when a problem occurs.
Pandas: frequently asked questions
Why do I get SettingWithCopyWarning and how do I fix it?
The warning appears when you assign into a selection from a DataFrame and it is unclear whether that selection is a view of the original or a copy. To modify the original, set it in one step with df.loc[condition, column] = value; to work with separate data, add .copy() when you make the selection.
What do I do when memory runs out reading a large CSV file?
Read only the columns you need with usecols, set column dtypes to smaller types or category, and read in pieces with chunksize. For data you use repeatedly, saving it as Parquet cuts both read time and size considerably. If that is still not enough, look at tools that process data out of memory.
When do I use merge, join, and concat?
merge combines two DataFrames side by side on key columns, like a SQL JOIN, and join is a shorthand that combines on the index. concat stacks DataFrames with the same structure vertically or horizontally. Use merge when rows have to be matched by key and concat when you are simply appending.