name: polars description: Use when "Polars", "fast dataframe", "lazy evaluation", "Arrow backend", or asking about "pandas alternative", "parallel dataframe", "large CSV processing", "ETL pipeline", "expression API" version: 1.0.0

Polars Fast DataFrame Library

Lightning-fast DataFrame library with lazy evaluation and parallel execution.

When to Use

Pandas is too slow for your dataset
Working with 1-100GB datasets that fit in RAM
Need lazy evaluation for query optimization
Building ETL pipelines
Want parallel execution without extra config

Lazy vs Eager Evaluation

Mode	Function	Executes	Use Case
Eager	`read_csv()`	Immediately	Small data, exploration
Lazy	`scan_csv()`	On `.collect()`	Large data, pipelines

Key concept: Lazy mode builds a query plan that gets optimized before execution. The optimizer applies predicate pushdown (filter early) and projection pushdown (select columns early).

Core Operations

Data Selection

Operation	Purpose
`select()`	Choose columns
`filter()`	Choose rows by condition
`with_columns()`	Add/modify columns
`drop()`	Remove columns
`head(n)` / `tail(n)`	First/last n rows

Aggregation

Operation	Purpose
`group_by().agg()`	Group and aggregate
`pivot()`	Reshape wide
`melt()`	Reshape long
`unique()`	Distinct values

Joins

Join Type	Description
inner	Matching rows only
left	All left + matching right
outer	All rows from both
cross	Cartesian product
semi	Left rows with match
anti	Left rows without match

Expression API

Key concept: Polars uses expressions (pl.col()) instead of indexing. Expressions are lazily evaluated and optimized.

Common Expressions

Expression	Purpose
`pl.col("name")`	Reference column
`pl.lit(value)`	Literal value
`pl.all()`	All columns
`pl.exclude(...)`	All except

Expression Methods

Category	Methods
Aggregation	`.sum()`, `.mean()`, `.min()`, `.max()`, `.count()`
String	`.str.contains()`, `.str.replace()`, `.str.to_lowercase()`
DateTime	`.dt.year()`, `.dt.month()`, `.dt.day()`
Conditional	`.when().then().otherwise()`
Window	`.over()`, `.rolling_mean()`, `.shift()`

Pandas Migration

Pandas	Polars
`df['col']`	`df.select('col')`
`df[df['col'] > 5]`	`df.filter(pl.col('col') > 5)`
`df['new'] = df['col'] * 2`	`df.with_columns((pl.col('col') * 2).alias('new'))`
`df.groupby('col').mean()`	`df.group_by('col').agg(pl.all().mean())`
`df.apply(func)`	`df.map_rows(func)` (avoid if possible)

Key concept: Polars prefers explicit operations over implicit indexing. Use .alias() to name computed columns.

File I/O

Format	Read	Write	Notes
CSV	`read_csv()` / `scan_csv()`	`write_csv()`	Human readable
Parquet	`read_parquet()` / `scan_parquet()`	`write_parquet()`	Fast, compressed
JSON	`read_json()` / `scan_ndjson()`	`write_json()`	Newline-delimited
IPC/Arrow	`read_ipc()` / `scan_ipc()`	`write_ipc()`	Zero-copy

Key concept: Use Parquet for performance. Use scan_* for large files to enable lazy optimization.

Performance Tips

Tip	Why
Use lazy mode	Query optimization
Use Parquet	Column-oriented, compressed
Select columns early	Projection pushdown
Filter early	Predicate pushdown
Avoid Python UDFs	Breaks parallelism
Use expressions	Vectorized operations
Set dtypes on read	Avoid inference overhead

vs Alternatives

Tool	Best For	Limitations
Polars	1-100GB, speed critical	Must fit in RAM
Pandas	Small data, ecosystem	Slow, memory hungry
Dask	Larger than RAM	More complex API
Spark	Cluster computing	Infrastructure overhead
DuckDB	SQL interface	Different API style

Resources

Docs: https://pola.rs/
User Guide: https://docs.pola.rs/user-guide/
Cookbook: https://docs.pola.rs/user-guide/misc/cookbook/

polars

SKILL.md