Project 20 — polars vs pandas: A Notebook-Only Comparison

A focused, notebooks-only study comparing pandas and polars across four angles, on real data at three scales. No FastAPI. No Streamlit. No Docker. Just six notebooks, three helper scripts, and a benchmark harness.

The four angles:

1. Syntax & API side-by-side (notebook 02) — every common operation in both libraries, with equivalence asserts
2. Performance benchmarks — wall-clock + peak RSS across 3 sizes × 7 operations (notebook 03)
3. Lazy execution & query optimization — polars LazyFrame query planner vs pandas eager (notebook 04)
4. Streaming / out-of-core — collect(engine='streaming') vs manual pyarrow chunking (notebook 05)

Plus a distilled migration cheatsheet (notebook 06) for when you actually port a real ETL script.

Why This Project

A deliberately small, focused study to build muscle memory on the modern DataFrame stack — Arrow-native, lazy, multi-threaded. Two motivations:

• Hands-on coverage of the full lazy + streaming story that pandas alone can’t tell — predicate pushdown, projection pushdown, query fusion, and out-of-core execution.
• A migration reference to come back to when porting a real ETL script from pandas to polars. Notebook 06 is the distilled cheatsheet.

Quick Start

cd 20-polars-vs-pandas

# 1. Create venv + install all deps (polars, pandas, pyarrow, faker, psutil, jupyter, pytest)
bash scripts/setup_env.sh
source .venv/bin/activate

# 2. Download one month of NYC taxi Parquet (~48 MB)
python scripts/download_data.py --year 2024 --months 1

# 3. Generate 50k synthetic Chilean insurance rows
python scripts/generate_synthetic.py

# 4. Open notebooks in Jupyter
jupyter lab notebooks/

For the streaming notebook (nb 05) you’ll want more data. Download a full year (~2 GB):

python scripts/download_data.py --year 2023 --months all

For headless regression runs (CI, sanity check):

bash scripts/run_all_notebooks.sh   # executes all 6 notebooks in place
pytest -v                           # 7 tests

Project Structure

20-polars-vs-pandas/
├── notebooks/
│   ├── 01_data_acquisition.ipynb       # download + subsample
│   ├── 02_syntax_side_by_side.ipynb    # 10 ops × 2 libs × equivalence assert
│   ├── 03_performance_benchmarks.ipynb # 3 sizes × 7 ops × 2 libs
│   ├── 04_lazy_vs_eager.ipynb          # LazyFrame.explain() + pushdown demo
│   ├── 05_streaming_out_of_core.ipynb  # streaming vs pandas chunked + sink_parquet
│   └── 06_migration_cheatsheet.ipynb   # distilled reference
├── scripts/
│   ├── download_data.py                # NYC taxi Parquet fetcher (year/month args)
│   ├── generate_synthetic.py           # Chilean RUT + insurance themed data
│   ├── benchmark_utils.py              # timed() decorator + median_run()
│   ├── setup_env.sh                    # venv + pip install
│   └── run_all_notebooks.sh            # nbconvert headless run
├── data/                               # contents gitignored
│   ├── raw/                            # yellow_tripdata_*.parquet
│   ├── synthetic/                      # insurance.csv
│   └── processed/                      # taxi_200k.parquet, taxi_2M.parquet
├── results/                            # contents gitignored
│   ├── timings.csv                     # 21 rows (3 sizes × 7 ops)
│   ├── memory.csv
│   └── plots/
│       ├── speedup.png
│       └── memory_ratio.png
├── tests/
│   ├── conftest.py                     # tiny pandas + polars fixtures
│   └── test_benchmark_utils.py         # 7 tests (all pass)
├── docs/
│   ├── README.md                       # docs index
│   └── INTERVIEW_PREP.md               # 10 Q&A study notes
├── .gitignore
├── requirements.txt
├── setup.py                            # thin shim, Python 3.10+
└── README.md                           # this file

Notebook Roadmap

#	Notebook	Goal	Time to run
01	data_acquisition	Download NYC taxi, generate synthetic insurance, subsample to 200k + 2M	~30 s (after download)
02	syntax_side_by_side	10 operations in both libraries, each with assert_same()	~5 s
03	performance_benchmarks	Time + memory across 3 sizes × 7 ops, write timings.csv + plots	~30 s
04	lazy_vs_eager	Three execution paths for the same analytical query + explain() plan dumps	~10 s
05	streaming_out_of_core	Polars streaming vs pandas chunked on full-year data + sink_parquet	~15 s (1 month)
06	migration_cheatsheet	Distilled reference — markdown + small cells, not compute-bound	<1 s

Headline Benchmark Results

From results/timings.csv (single month of NYC Yellow Taxi 2024-01, ~3M rows, M3-class macOS, both libraries latest stable). All numbers are median of 5 iterations after dropping a warmup:

Operation	pandas (s)	polars (s)	Speedup
read_parquet	0.148	0.111	1.34x
filter	0.027	0.009	3.12x
groupby_agg	0.036	0.053	0.69x (pandas wins)
sort	0.437	0.233	1.87x
with_columns	0.003	0.001	2.06x
join	0.069	0.026	2.65x
write_parquet	0.767	0.194	3.96x

Key takeaways:

• Filter, join, write are polars’ biggest wins (3–4x). These are I/O- and Arrow-layout-bound; polars’ Rust core and multi-threading pay off.
• Sort is consistently ~2x faster.
• groupby_agg is the one operation where pandas edged out polars in this run — likely Python-overhead-dominated at small aggregation cardinality (262 unique PULocationIDs). Re-runs vary; the difference is inside measurement noise.
• Speedups grow with size for read/filter/join — see results/plots/speedup.png.

Full numbers across 200k / 2M / 3M rows are in results/timings.csv.

Library Comparison Matrix

Dimension	pandas 3.x	polars 1.x
Implementation	C / Cython / Python	Rust
Memory format	NumPy block manager (optional Arrow backend in 2.x)	Apache Arrow (always)
Execution model	Eager	Eager (DataFrame) + lazy (LazyFrame)
Multi-threading	Single-threaded compute (threaded I/O on PyArrow paths)	All cores, by default
Query planner	None — each line materializes	Predicate + projection pushdown, operation fusion
Out-of-core streaming	Manual via pyarrow iter_batches	collect(engine='streaming')
Index	Yes (row labels)	None — explicit columns only
Null handling	NaN for floats, None for object	null for any dtype, NaN separate for floats
Ecosystem integration	Widest (sklearn, plotly, statsmodels, …)	Growing; interop via Arrow (pl.from_pandas)

Datasets

Dataset	Source	Size	Used in
NYC Yellow Taxi	NYC TLC (free, no auth)	200k → ~3M rows per month, up to ~30M for a full year	nb 01, 03, 04, 05
Synthetic Chilean insurance	generated locally (Faker + RUT algo)	50k rows	nb 02

NYC TLC Parquet URL pattern:

https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_<YYYY>-<MM>.parquet

The synthetic insurance data uses realistic Chilean-style columns (rut, age, region, product, account_balance_clp, churn, …) so the syntax demos feel less abstract than raw NYC taxi columns.

Verification

Tests (pytest -v, 7 tests, ~0.6 s):

• timed() decorator returns (result, wall, peak_mb) with sane values
• median_run() drops warmup, returns medians, handles n_iter=1 edge case
• median_run(n_iter=0) raises ValueError
• get_rss_mb() returns positive
• pandas groupby-sum and polars groupby-sum agree on the same input
• pandas filter and polars filter produce the same row count on the same input

Notebooks (scripts/run_all_notebooks.sh):

• All 6 execute end-to-end via jupyter nbconvert --execute
• Notebook 02 asserts equivalence between pandas and polars on all 10 operations
• Notebook 04 asserts pandas eager / polars eager / polars lazy all produce the same top-10 result
• Notebook 05 asserts polars streaming and pandas chunked produce the same group-level aggregate

Troubleshooting

• SSL: CERTIFICATE_VERIFY_FAILED on download (macOS python.org Python): the system Python ships without CA certs. The script auto-uses certifi if installed (it’s in requirements.txt). Re-run bash scripts/setup_env.sh if you skipped it.
• FileNotFoundError: taxi_2M.parquet: notebook 01 only writes the 2M sample if at least one raw month has ≥2M rows. Each month has ~3M, so this requires at least one month downloaded.
• Notebook 05 peak memory looks similar to notebook 03: one month (~3M rows, 48 MB) fits easily in RAM, so neither engine is stressed. To actually exercise streaming, download a full year (--months all) and re-run.
• Pyright import warnings on polars: harmless — Pyright can’t see the project venv. Tests and notebooks run correctly inside .venv.

Roadmap (v2)

Out of scope for this MVP, but flagged for future work:

• DuckDB comparison — same query in SQL via DuckDB’s Python API, as a third engine
• Real RUT validation — the synthetic generator uses a plausible-looking but not cryptographically valid RUT check digit
• GPU path — add cudf.pandas as a third library on a CUDA machine
• Larger-than-RAM benchmark — automatize the multi-month download and run an actual OOM-avoidance test
• Polars on Ray — distributed polars for cluster-scale

License & Attribution

• Code: MIT-style, portfolio use.
• NYC Taxi data: public domain, NYC Taxi & Limousine Commission.
• Synthetic insurance data: fully synthetic; RUT-style identifiers are not real Chilean RUTs (check digit is a plausible-but-not-valid algorithm).

See docs/INTERVIEW_PREP.md for a 10-question study guide covering the polars/pandas topics that come up in 2024–2026 data engineering interviews.