Apache Spark
Apache Spark is the standard engine for processing data that doesn't fit on one machine. It presents a DataFrame API much like Pandas or SQL, but executes each query as a distributed job across a cluster — splitting the data into partitions, running transformations in parallel, and moving data between machines only when an operation demands it.
Spark underpins most large-scale data engineering: ETL pipelines feeding warehouses, feature computation for ML, log processing, and the lakehouse platforms (Databricks, EMR, Dataproc) built around it.
TL;DR
- Spark = driver (your program, builds the plan) + executors (workers that process partitions in parallel).
- Execution is lazy: transformations (
filter,groupBy,join) build a plan; nothing runs until an action (count,write,show). - The expensive operation is the shuffle — redistributing data across the network for joins/groupBys. Most Spark tuning is shuffle management.
- Write DataFrame/SQL code, not RDDs — the Catalyst optimizer rewrites your query like a database would.
- Small-data rule: if it fits on one machine, Pandas/Polars/DuckDB is simpler and often faster. Spark's overhead pays off at tens-of-GB scale and up.
- Store data as Parquet (or Delta/Iceberg tables), partitioned by common filter columns.
Quick Example
PySpark looks like Pandas-flavored SQL:
Or the same thing in SQL:
Core Concepts
Cluster Architecture
Data is split into partitions; each executor core processes one partition per task. Parallelism is capped by partition count — 8 partitions on a 100-core cluster leaves 92 cores idle.
Lazy Evaluation
Laziness lets the Catalyst optimizer rearrange your whole query before execution: pushing filters down to the file scan, pruning unread columns, choosing join strategies. It's also why errors surface at the action, far from the line that caused them.
The Shuffle
Narrow transformations (filter, withColumn) stay within partitions. Wide ones (groupBy, join, distinct, orderBy) require a shuffle — every executor writes data grouped by key, every executor reads its share back over the network:
Shuffles dominate job cost. The practical levers:
- Broadcast small tables in joins (
F.broadcast(dim)) — ships the small side to every executor, no shuffle of the big side. - Filter early so less data shuffles.
- Watch data skew: one hot key (e.g.
customer_id = null) makes one task process 100× the data of the others; a job "stuck at 199/200 tasks" is almost always skew.
Storage: Parquet and Table Formats
Columnar Parquet enables column pruning and predicate pushdown. Partitioned layout turns filters into directory pruning:
Modern lakehouses wrap Parquet in Delta Lake or Apache Iceberg for ACID upserts, schema evolution, and time travel.
The Spark Stack
Deployment options range from pip install pyspark locally, to Kubernetes/YARN clusters, to managed platforms — Databricks, AWS EMR, GCP Dataproc — which is how most teams actually run it.
Best Practices
Stay in DataFrames, Avoid Python UDFs
A Python UDF forces rows out of the JVM into a Python worker and back — often 10× slower than built-in functions. pyspark.sql.functions covers far more than people expect; if you must go custom, use pandas_udf (vectorized via Arrow).
Read the Query Plan
Look for BroadcastHashJoin (good for small dims) vs SortMergeJoin (shuffle), and confirm PartitionFilters/PushedFilters show your predicates reaching the scan. The Spark UI (port 4040) shows per-stage shuffle sizes and task skew.
Manage Output File Sizes
Thousands of tiny output files cripple downstream readers. coalesce(n) before writing (or table-format compaction) targets ~128 MB–1 GB files.
Cache Only What You Reuse
df.cache() helps when a DataFrame feeds multiple actions; it hurts when it evicts memory needed for shuffles. Cache deliberately, unpersist() when done.
Spark vs the Alternatives
The most common Spark mistake is organizational: adopting a cluster for data that a laptop running DuckDB handles in seconds. Spark's startup, shuffle, and ops overhead only pay for themselves when the data genuinely exceeds one machine.
Common Mistakes
collect() on Big Data
df.collect() pulls the entire dataset into the driver's memory — the classic OOM. Use show(), take(n), or write results to storage.
Joining Without Broadcasting the Small Side
A 2 TB fact table shuffle-joined against a 10 MB dimension wastes the cluster. Broadcast the dimension; Spark's adaptive execution often catches this, but don't rely on it.
Ignoring Skew Until the Job Hangs
One task running for hours while 199 finish means a hot key. Fix at the source (filter nulls, salt the key, use skew-join hints) rather than adding executors.
Treating Spark Like Pandas
Row iteration, .toPandas() on huge frames, and per-row UDFs all fight the engine. Think in set operations — columns and joins, not loops.
FAQ
Do I need to know Scala/Java?
No — PySpark is the dominant interface and DataFrame code compiles to the same optimized plans regardless of language. Scala only matters for extending Spark itself or rare JVM-level performance work.
Spark or DuckDB/Polars?
Honest sizing: below ~50–100 GB per job, a beefy single machine with DuckDB or Polars is usually simpler, cheaper, and faster end-to-end. Spark wins when data is genuinely distributed-scale, already lives in a lakehouse, or pipelines must scale elastically.
What is Databricks relative to Spark?
The commercial platform from Spark's creators: managed clusters, notebooks, Delta Lake, and a proprietary faster engine (Photon), all running Spark APIs. Most enterprise Spark today runs on Databricks, EMR, or Dataproc rather than self-managed clusters.
Is Spark good for machine learning?
For data prep and features at scale, yes. For model training, MLlib covers classical distributed algorithms, but most teams compute features in Spark and train in scikit-learn/PyTorch — or use Spark to parallelize many single-node trainings.
Batch or streaming?
Spark Structured Streaming processes micro-batches (sub-second to seconds of latency) with the same DataFrame code as batch — ideal for near-realtime ETL from Kafka. For per-event millisecond latency, Flink is the specialist.
Related Topics
- Pandas — The single-machine DataFrame counterpart
- Data Engineering — The pipelines Spark powers
- Apache Airflow — Orchestrating Spark jobs
- Apache Kafka — The streaming source Spark consumes
- ClickHouse — Warehouse-style analytics alternative
- SQL — The query semantics underneath DataFrames
- Scala — Spark's implementation language