Skip to main content
AIDevOps
  • Learn
  • Learning Paths
  • Practice
  • Open Source
  • Books
  • Engineering

    AI DevOpsThe full map of building and operating AI servicesLLMOpsLLM deployment Β· evaluation Β· observabilityHands-on ProjectsBuild AI Agent projects

    Knowledge

    DocsTechnical documentationBlogEngineering articlesPloggerDevelopment log feed

    Validate

    Certification3-level skills certification Β· coming soon
AI Models
LlamaTutorial|MistralTutorial|GemmaTutorial|DeepSeekTutorial|QwenTutorial
🧠 AI Core
AI Intro & RoadmapML FundamentalsLLM Fundamentals|Python AIC++|PyTorchTensorFlowJAX
πŸ€– AI Applied Development
Applied AI Intro & RoadmapHugging FaceLangChainLlamaIndexLLMOps|LangGraphMCPMulti-AgentAgent Evaluation
🧠 AI Agent Development
Finance AI AgentLLM API ServerStock Investing AgentAIOps AI AgentEducation AI AgentCoding AI Agent
🌱 Spring Cloud
Spring Intro & RoadmapSpring Cloud GatewaySpring BootJava|Spring AISpring SecuritySpring BatchSpring JPA
🐳 DevOps
DevOps Intro & RoadmapLinuxDockerCI/CD|Kubernetes BasicsK8s AdvancedPrometheusGrafana
🧱 Infrastructure
Infrastructure Intro & RoadmapNginxRedis
☁️ Cloud
Cloud Intro & RoadmapAWSGCPAzureNCPCloudflare
🎨 Frontend
Frontend Intro & RoadmapJavaScriptTypeScript|ReactNext.js|VueNuxt|Electron
πŸ“± Mobile
Mobile Intro & RoadmapKotlinAndroidFlutter
βš™οΈ Backend
Backend Intro & RoadmapPython BasicsFastAPIDjangoFlask|CGoGinNode.js
πŸ’Ύ Database
DB Intro & RoadmapCore SQLOracleMySQLPostgreSQL|MongoDBVector DB
πŸ§ͺ Testing
k6JMeternGrinder
AIDevOps

Engineering AI. From Code to Production.
An engineering learning platform for building and operating AI and AI Agents

Learn

  • All Guides
  • Learning Paths
  • AI Tutorial
  • Practice
  • Books

Resources

  • AI DevOps
  • LLMOps
  • Hands-on Projects
  • Docs
  • Blog
  • Plogger
  • Open Source
  • Certification (coming soon)

Start Here

  • AI Core Roadmap
  • AI Applied Development Roadmap
  • Spring Cloud Roadmap
  • DevOps Roadmap
  • Infrastructure Roadmap

Β 

  • Cloud Roadmap
  • Frontend Roadmap
  • Mobile Roadmap
  • Backend Roadmap
  • Database Roadmap
Β© 2026 AIDevOps. All rights reserved.
Terms of ServicePrivacy PolicySitemaptestforge.kr
  1. Home
  2. Learn
  3. Pandas
The complete guide to Python data analysis

🐼 Pandas Complete Guide

Visitors

Read, clean, and aggregate CSV, Excel, and DB data with Pandas. Indexing, merging, how groupby works inside, and performance optimization β€” the core tools for getting data quality right before moving into AI/ML.

  • Beginner
  • Updated Oct 1, 2026
  • ~4 min read
  • 11 sections
  • 7 code examples
  • Web IDE lab included
🐼

Pandas Web IDE

Run code in the browser with nothing to install and learn through step-by-step examples.

Open the Web IDE β†’
Data cleaningCSV/Excel analysisAggregate reportsML preprocessing

Related frameworks & environments

🐍Python Basicsβ†’

Contents

0 / 14
  1. How to use this guide
  2. Architecture diagrams
  3. What is Pandas?
  4. Loading data
  5. Indexing & filtering
  6. Cleaning
  7. Combining (merge/concat)
  8. Aggregation (Split-Apply-Combine)
  9. Performance optimization
  10. Saving
  11. Pandas design
  12. Operating standards
  13. Verification strategy
  14. Frequently asked questions
Contents 14 sections
  1. How to use this guide
  2. Architecture diagrams
  3. What is Pandas?
  4. Loading data
  5. Indexing & filtering
  6. Cleaning
  7. Combining (merge/concat)
  8. Aggregation (Split-Apply-Combine)
  9. Performance optimization
  10. Saving
  11. Pandas design
  12. Operating standards
  13. Verification strategy
  14. Frequently asked questions

How to use this guide

How to read it

Understanding Pandas as a real-world workflow

Read, clean, and aggregate CSV, Excel, and DB data with Pandas. Indexing, merging, how groupby works inside, and performance optimization β€” the core tools for getting data quality right before moving into AI/ML. Rather than listing concepts, this guide is organized so you can follow the content in the order you have to make decisions on a real project.

Key perspective

Backend / systems development

Before syntax, pin down the boundaries a request crosses: validation, processing, storage, and response.

Data cleaningCSV/Excel analysisAggregate reportsML preprocessing

Architecture diagrams

To make what you read stick, it helps to capture the flow as a picture first. The two diagrams below are reference maps you can keep coming back to while learning Pandas.

Learning flow

Rendering diagram…

Architecture view

Rendering diagram…

What is Pandas?

When you first open Pandas, the big picture matters more than individual commands. This section starts with the problems the concepts ahead were created to solve.

Pandas is a Python library for working with tabular data. Centered on the DataFrame, it handles filtering, missing values, group aggregation, joins, and file I/O concisely.
ObjectDescription
SeriesA one-dimensional labeled array.
DataFrameTwo-dimensional table data with rows and columns.
IndexThe row/column labels and the basis for alignment.

Loading data

Here you look at Loading data alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Read a variety of data sources such as CSV, Excel, and JSON into a DataFrame.
PYTHON
import pandas as pd

df = pd.read_csv("orders.csv")
print(df.head())
print(df.info())

Indexing & filtering

Here you look at Indexing & filtering alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

`loc` accesses rows/columns by label (name) and `iloc` by integer position. Boolean indexing, where a condition goes inside the square brackets, plays the same role as the WHERE clause in SQL.
indexing.pyPYTHON
# loc β€” by label (column name, index value)
df.loc[0, "amount"]
df.loc[df["amount"] > 10000, ["user_id", "amount"]]

# iloc β€” by integer position (like cell coordinates in a spreadsheet)
df.iloc[0, 2]        # row 0, column 2
df.iloc[:5, :3]       # first 5 rows, first 3 columns

# Boolean indexing β€” select only the rows that meet the condition
high_value = df[(df["amount"] > 10000) & (df["month"] == "2026-01")]
ApproachBased onExample
locLabel (column name, index value)df.loc[3, "amount"]
ilocInteger positiondf.iloc[3, 1]
Boolean indexingThe result of a condition (True/False)df[df["amount"] > 100]

Tip

If you confuse loc and iloc and just slice, as in df[0:3], integer position and label semantics get mixed and the result may not be what you expect β€” to make your intent clear, always spell out loc/iloc.

Cleaning

Here you look at Cleaning alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Analysis results are only stable once missing values, type conversions, and derived columns have been sorted out first.
PYTHON
df = df.dropna(subset=["user_id", "amount"])
df["amount"] = df["amount"].astype(float)
df["ordered_at"] = pd.to_datetime(df["ordered_at"])
df["month"] = df["ordered_at"].dt.to_period("M")

Combining (merge/concat)

Here you look at Combining (merge/concat) alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

When combining several tables into one, `concat` simply stacks them top/bottom or left/right, while `merge` joins them on a common key like a SQL JOIN. The `how` option decides what happens to rows that do not match.
merge.pyPYTHON
orders = pd.DataFrame({"order_id": [1, 2, 3], "user_id": [10, 20, 10]})
users  = pd.DataFrame({"user_id": [10, 20, 30], "name": ["Ada", "Linus", "Grace"]})

# INNER JOIN β€” only user_ids present on both sides remain
inner = orders.merge(users, on="user_id", how="inner")

# LEFT JOIN β€” every order is kept, NaN where there is no match
left = orders.merge(users, on="user_id", how="left")

# Plain stacking (data with the same columns, top to bottom)
combined = pd.concat([orders, orders], ignore_index=True)
howResult
inner (default)Keeps only rows whose key matches on both sides
leftKeeps every row of the left DataFrame, NaN where there is no match
outerKeeps every row of both sides, NaN where there is no match

Tip

If there are more rows than expected after a merge, the key column most likely contains duplicates and a many-to-many (N:M) match happened β€” check for duplicates first with `df[key].duplicated().sum()` before merging.

Aggregation (Split-Apply-Combine)

Here you look at Aggregation (Split-Apply-Combine) alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

groupby works in three stages: "split into groups (Split) β†’ apply a function to each group (Apply) β†’ combine the results again (Combine)". Once you understand this model, you can predict why the result looks the way it does whatever you pass to agg.
Rendering diagram…
PYTHON
monthly = (
    df.groupby("month")
      .agg(order_count=("id", "count"), revenue=("amount", "sum"))
      .reset_index()
)
print(monthly)

Performance optimization

Here you look at Performance optimization alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Row-by-row `apply` and `iterrows` are simple to write but tens to hundreds of times slower than vectorized operations. Express whole-column operations with the vectorized methods of Pandas/NumPy wherever possible.
performance.pyPYTHON
# Slow β€” loops one row at a time at the Python level
df["discounted"] = df.apply(lambda row: row["amount"] * 0.9 if row["vip"] else row["amount"], axis=1)

# Fast β€” a vectorized operation (array math implemented in C underneath)
df["discounted"] = df["amount"].where(~df["vip"], df["amount"] * 0.9)

# For large CSVs, specify only the columns/types you need up front to save memory
df = pd.read_csv("orders.csv", usecols=["user_id", "amount"], dtype={"user_id": "int32"})

Tip

Check real memory usage with `df.info(memory_usage="deep")` β€” object (string) columns commonly take far more memory than expected. When string values repeat a lot, switching to the category type can cut memory sharply.

Saving

Here you look at Saving alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Save the cleaned result in a format that suits the next stage, such as CSV, Parquet, or Excel.
PYTHON
monthly.to_csv("monthly_report.csv", index=False)
monthly.to_parquet("monthly_report.parquet", index=False)

Tip

For large data with many columns and frequent repeated reads, Parquet is recommended over CSV β€” column-level compression and a preserved schema (type information) make both file size and reload speed better.

Pandas practical design

Pandas practical design is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.

With Pandas, separate notebook analysis code from reusable transform functions. Stating the schema, dtype, and missing value policy explicitly makes the pipeline stable.
Decision pointQuestion to askPractical guideline
BoundariesWhich parts of the Pandas code are most likely to change?Separate input/output, configuration, external integrations, and the core rules.
StateWhere is state created and where does it go away?Make the owner and lifecycle of state visible in the code.
FailureWhat does the caller receive on failure?Decide the timeout, fallback, and error contract first.

Pandas operating standards

This section covers Pandas operating standards from a practical perspective. Instead of memorizing the concept, focus on the situations in which you would reach for it.

With large data, copy cost, the object dtype, groupby/join memory use, and the chunk processing strategy matter.

Tip

  • dtype contract
  • chunk processing
  • null ratio checks
  • data drift report

Pandas verification strategy

Pandas verification strategy is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.

For data quality, test the row count, null ratio, unique constraints, and distribution drift.
Quality axisHow to verifyDefinition of done
CorrectnessAutomate the normal/failure cases.The core scenarios pass reproducibly.
Regression preventionWhen fixing a bug, leave the same case behind as a test.The same failure is not deployed again.
OperabilityCheck the logs, metrics, and alerts.There is a path for tracing the cause when a problem occurs.

Pandas: frequently asked questions

Why do I get SettingWithCopyWarning and how do I fix it?

The warning appears when you assign into a selection from a DataFrame and it is unclear whether that selection is a view of the original or a copy. To modify the original, set it in one step with df.loc[condition, column] = value; to work with separate data, add .copy() when you make the selection.

What do I do when memory runs out reading a large CSV file?

Read only the columns you need with usecols, set column dtypes to smaller types or category, and read in pieces with chunksize. For data you use repeatedly, saving it as Parquet cuts both read time and size considerably. If that is still not enough, look at tools that process data out of memory.

When do I use merge, join, and concat?

merge combines two DataFrames side by side on key columns, like a SQL JOIN, and join is a shorthand that combines on the index. concat stacks DataFrames with the same structure vertically or horizontally. Use merge when rows have to be matched by key and concat when you are simply appending.