Python for Data Science: NumPy, Pandas & scikit-learn

Python for Data Science

Python is the default "glue language" for data science because it ties together fast numeric kernels (NumPy), tabular wrangling (Pandas), visualization (Matplotlib/Seaborn/Plotly), ML tooling (scikit-learn, PyTorch, TensorFlow), and interactive notebooks (Jupyter). No single piece is unique, but the ecosystem — and how smoothly the pieces compose — is why it dominates.

The one tradeoff to internalize: Python itself is slow for tight loops, so you lean on vectorized operations and compiled libraries rather than writing the loops yourself. Get that habit right and the language's speed almost never matters.

TL;DR

Python wins on ecosystem: NumPy, Pandas, viz, scikit-learn, Jupyter.
Vectorize — prefer array/DataFrame ops over Python loops.
Use pipelines so preprocessing and modeling are reproducible.
Watch for leakage, chained assignment, and timezone bugs.

Quick Example

Vectorized math runs in compiled code — no Python loop:

The Core Stack

NumPy — arrays, vectorized math, broadcasting.
Pandas — DataFrames, joins/groupby, time series, IO.
Visualization — Matplotlib (base), Seaborn (statistical), Plotly (interactive).
scikit-learn — classical ML, preprocessing, pipelines, metrics.
Jupyter — exploration + narrative (mind reproducibility).
Data formats — Parquet (columnar), Arrow (in-memory), CSV (last resort).

A Practical Workflow

Define the question + target — what does "good" mean?
Get clean, versioned data — raw → cleaned → features.
Explore — distributions, missingness, leakage checks.
Baseline model — simple and interpretable first.
Iterate — features, cross-validation, error analysis.
Package + reproduce — environment, seeds, splits, saved artifacts.

NumPy & Pandas Essentials

Prefer array operations over loops, and watch shapes — (n,) vs (n,1) is a classic source of bugs. Broadcasting is powerful but easy to misuse; when a result looks too big or small, print .shape for every intermediate.

Cleanup essentials: normalize column names and types (especially timestamps), handle missing values explicitly, and validate assumptions (unique keys, allowed ranges). For joins, prefer explicit keys and validate row counts after — a one-to-many join can silently duplicate rows.

scikit-learn Pipelines

Couple preprocessing and modeling in a pipeline so the whole thing is reproducible and leak-free:

For evaluation: use a proper split (time-based for time series), and for imbalanced classes don't trust accuracy — use AUROC, AUPRC, F1, and calibrated probabilities. See Model Evaluation.

Performance & Scale

When things slow down: move CSV → Parquet, use categorical dtypes, avoid .apply() with Python lambdas (vectorize instead), reach for Polars/DuckDB for heavy tabular analytics, and move ML training to PyTorch/TensorFlow with GPU acceleration.

Best Practices

Vectorize everything you can; loops are the slow path.
Use pipelines to keep preprocessing reproducible and leak-free.
Store UTC, convert timezones at the edges.
Promote exploration to a src/ package for production; keep notebooks for plots.
Make runs deterministic — seeds, versioned data, saved artifacts.

Common Mistakes

Chained assignment in Pandas

Data leakage

FAQ

Why is Python so dominant in data science when it's "slow"?

Because the heavy computation doesn't run in pure Python — NumPy, Pandas, and ML libraries dispatch to optimized C/Fortran/CUDA code. Python is the ergonomic glue: expressive, interactive (Jupyter), and backed by an unmatched ecosystem. As long as you vectorize and lean on compiled libraries instead of writing tight Python loops, the language's raw speed rarely matters.

Pandas, Polars, or DuckDB?

Pandas is the default — ubiquitous, well-documented, huge ecosystem. Polars is a faster, multi-threaded DataFrame library with a lazy API, great when Pandas gets slow or memory-bound. DuckDB is an in-process analytical SQL engine, excellent for heavy aggregations over larger-than-memory data using SQL. Start with Pandas; reach for Polars/DuckDB when performance or scale demands it.

What is data leakage and why is it so dangerous?

Leakage is when information unavailable at prediction time sneaks into training — e.g. scaling on the full dataset before splitting, or a feature that encodes the target. It inflates validation scores, so the model looks great offline but fails in production. Prevent it by splitting first, fitting all preprocessing inside a pipeline on the training fold only, and scrutinizing features that "peek" into the future (common in event/time-series data).

Should I work in notebooks or scripts?

Both, for different phases. Notebooks excel at exploration and narrative — distributions, quick plots, iteration. But production work needs a real src/ package (testable functions/classes), deterministic runs (seeds, versioned data, saved artifacts), and ideally just a thin "driver" notebook for visualization. Notebooks alone are hard to reproduce, review, and test.