---

name: python-programming description: Python fundamentals, data structures, OOP, and data science libraries (Pandas, NumPy). Use when writing Python code, data manipulation, or algorithm implementation. sasmp_version: "1.3.0" bonded_agent: 01-python-data-science bond_type: PRIMARY_BOND

Python Programming for Data Science

Master Python from fundamentals to advanced data science applications.

Quick Start

Essential Libraries

Copyimport pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

Data Manipulation

Copy# Read data
df = pd.read_csv('data.csv')

# Explore
print(df.head())
print(df.info())
print(df.describe())

# Filter
df_filtered = df[df['age'] > 18]

# Group and aggregate
summary = df.groupby('category')['sales'].agg(['sum', 'mean', 'count'])

# Vectorized operations (FAST!)
df['new_col'] = df['col1'] * 2  # Instead of loops

Core Concepts

1. Data Structures

• Lists: [1, 2, 3] - ordered, mutable
• Dictionaries: {'key': 'value'} - key-value pairs
• Tuples: (1, 2, 3) - immutable
• Sets: {1, 2, 3} - unique elements

2. List Comprehensions

Copy# Instead of loops
squares = [x**2 for x in range(10)]
filtered = [x for x in data if x > 0]

3. NumPy Arrays

Copyarr = np.array([1, 2, 3, 4, 5])
arr * 2  # [2, 4, 6, 8, 10]
arr.mean()  # 3.0

4. Pandas DataFrames

Copydf = pd.DataFrame({
    'name': ['Alice', 'Bob'],
    'age': [25, 30],
    'salary': [50000, 60000]
})

Performance Tips

Vectorization over Loops (10-100x faster):

Copy# Bad (slow)
result = []
for x in data:
    result.append(x * 2)

# Good (fast)
result = np.array(data) * 2

Common Patterns

Reading Files

Copy# CSV
df = pd.read_csv('file.csv')

# Excel
df = pd.read_excel('file.xlsx', sheet_name='Sheet1')

# JSON
df = pd.read_json('file.json')

# SQL
import sqlite3
conn = sqlite3.connect('database.db')
df = pd.read_sql_query("SELECT * FROM table", conn)

Missing Data

Copydf.dropna()  # Remove rows
df.fillna(0)  # Fill with value
df.fillna(df.mean())  # Fill with mean

Merging Data

Copy# Join DataFrames
merged = pd.merge(df1, df2, on='id', how='left')

# Concatenate
combined = pd.concat([df1, df2], axis=0)

Best Practices

1. Use vectorized operations
2. Optimize data types
3. Avoid loops when possible
4. Use built-in functions
5. Profile before optimizing

Troubleshooting

Common Issues

Problem: MemoryError with large DataFrames

Copy# Solution 1: Use chunking
for chunk in pd.read_csv('large.csv', chunksize=10000):
    process(chunk)

# Solution 2: Optimize dtypes
df['int_col'] = df['int_col'].astype('int32')  # Instead of int64
df['cat_col'] = df['cat_col'].astype('category')  # For repeated strings

Problem: Slow DataFrame operations

Copy# Debug: Profile your code
%timeit df.apply(func)  # Compare with vectorized

# Solution: Use vectorized operations
df['result'] = np.where(df['x'] > 0, df['x'] * 2, 0)  # Instead of apply

Problem: Import errors

Copy# Solution: Check environment
pip list | grep pandas
pip install --upgrade pandas numpy

# Virtual environment best practice
python -m venv venv
source venv/bin/activate  # Linux/Mac
pip install -r requirements.txt

Problem: Data type mismatches

Copy# Debug: Check types
print(df.dtypes)

# Solution: Convert types explicitly
df['date'] = pd.to_datetime(df['date'])
df['price'] = pd.to_numeric(df['price'], errors='coerce')

Debug Checklist

• Check Python and library versions
• Verify data types with df.dtypes
• Profile with %timeit before optimizing
• Use df.info() for memory usage
• Check for NaN values with df.isna().sum()