Pandas

Pandas is Python's foundational library for working with tabular data. Its core object, the DataFrame, is a labeled table — like a spreadsheet or SQL table in memory — with fast, expressive operations for filtering, transforming, aggregating, and joining.

Nearly every data workflow in Python passes through Pandas: loading CSVs and database queries, cleaning messy exports, feature preparation for machine learning, and ad-hoc analysis. It's often the second Python library a developer learns, right after the standard library.

TL;DR

Quick Example

A realistic slice of analysis — load, clean, filter, aggregate:

Core Concepts

DataFrame, Series, and Index

Every DataFrame has an index (row labels). Operations align on labels automatically — adding two Series matches rows by index, not position. That alignment is Pandas' superpower and, when indexes unexpectedly differ, a classic source of surprise NaNs.

Selection: loc and iloc

The rule: read or write with a single loc/iloc call. Chained indexing (df[df.price < 0]["price"] = 0) may modify a temporary copy and silently do nothing — that's what SettingWithCopyWarning is about. (Pandas 3's copy-on-write makes chained assignment consistently not work, so the habit matters more, not less.)

GroupBy: Split–Apply–Combine

If you know SQL, groupby().agg() is GROUP BY, and transform is a window function.

Joining Tables

how takes inner, left, right, outer — the same semantics as SQL joins. Watch for unintended many-to-many matches: validate="one_to_many" catches them early.

Missing Data

NaN is contagious in arithmetic and excluded from aggregations by default — decide explicitly whether to fill, drop, or keep.

Performance Habits

Vectorize, Don't Loop

.apply(func, axis=1) is still a row loop in disguise — a last resort, not a default.

Use the Right Dtypes

Categories can cut memory 10× on repetitive string columns; loading only needed columns does the same for wide files.

Prefer Parquet over CSV

CSV loses dtypes and parses slowly; Parquet is the standard interchange format across Pandas, Spark, and DuckDB.

Know When You've Outgrown It

Pandas holds everything in RAM on one core (increasingly multi-core via its Arrow backend, but still one machine). Rules of thumb:

Polars and DuckDB interoperate with Pandas DataFrames directly, so mixing is cheap.

Common Mistakes

Chained Indexing Assignment

Ignoring the Index After Filtering

Filtered frames keep original row labels. Positional assumptions (df.loc[0]) then fail or mislead — reset_index(drop=True) when row identity no longer matters.

Growing a DataFrame Row by Row

Appending in a loop reallocates every iteration. Collect rows in a list of dicts, then build once: pd.DataFrame(rows).

Reading Everything to Use 2 Columns

usecols, nrows, and Parquet column selection exist precisely so a 10 GB file doesn't become a 10 GB DataFrame.

FAQ

Pandas or Polars?

Polars is dramatically faster (Rust, multi-threaded, lazy queries) and its API avoids Pandas' index pitfalls. Pandas still wins on ecosystem: tutorials, Stack Overflow answers, and library integrations (scikit-learn, plotting, Jupyter) assume it. Learn Pandas first; adopt Polars when performance hurts.

Pandas or SQL?

Both, honestly — push heavy filtering/joining into the database (query optimization beats transferring rows), then use Pandas for what SQL is clumsy at: reshaping, time series, plugging into Python ML code. pd.read_sql bridges them.

Why is my DataFrame using so much memory?

Usually object-dtype string columns. Convert low-cardinality strings to category, downcast numerics, and check with df.memory_usage(deep=True).

What is the 2 vs 3 situation?

Pandas 2 introduced the Apache Arrow-backed dtypes; Pandas 3 makes copy-on-write the default (killing chained-assignment ambiguity) and modernizes string handling. Code following the loc-for-assignment rule migrates cleanly.

How do I plot from Pandas?

df.plot() wraps Matplotlib for quick looks; for real charts use Matplotlib/Seaborn/Plotly directly on the columns. See Python for Data Science.

Related Topics

References