Python Data Science with Pandas: From Raw Data to Actionable Insights

Python's data science ecosystem is unmatched. At its core sits Pandas — a library that transforms messy, real-world data into structured insights.

Understanding DataFrames

A DataFrame is a 2D labeled data structure. Think of it as a spreadsheet with superpowers.

import pandas as pd
import numpy as np

df_csv = pd.read_csv('sales.csv', parse_dates=['date'])
print(df.shape)           # (rows, columns)
print(df.dtypes)          # Column types
print(df.describe())      # Statistical summary
print(df.isnull().sum())  # Missing value counts

Handling Missing Data

# Drop rows where critical columns are null
df = df.dropna(subset=['user_id', 'timestamp'])

# Fill forward for time series
df['temperature'] = df['temperature'].fillna(method='ffill')

# Median imputation for skewed data
df['price'] = df['price'].fillna(df['price'].median())

# Category-aware imputation
df['age'] = df.groupby('gender')['age'].transform(
    lambda x: x.fillna(x.median())
)

GroupBy: The Engine of Aggregation

summary = df.groupby('product_category').agg(
    total_revenue=('revenue', 'sum'),
    avg_order_value=('revenue', 'mean'),
    order_count=('order_id', 'nunique'),
    customer_count=('customer_id', 'nunique')
).reset_index()

# Window functions
df['7d_rolling_avg'] = df.groupby('product_id')['daily_sales'].transform(
    lambda x: x.rolling(window=7, min_periods=1).mean()
)

# Rank within groups
df['rank_in_category'] = df.groupby('category')['sales'].rank(
    method='dense', ascending=False
)

Merging Datasets

# Left join
result = customers.merge(orders, on='customer_id', how='left')

# Multi-key join
result = transactions.merge(exchange_rates, on=['currency', 'date'], how='left')

# Anti-join: find customers who never ordered
merged = customers.merge(orders[['customer_id']].drop_duplicates(),
                         on='customer_id', how='left', indicator=True)
never_ordered = merged[merged['_merge'] == 'left_only']

Reproducible Pipelines

class DataPipeline:
    def load(self, path: str) -> pd.DataFrame:
        df = pd.read_parquet(path)
        required = {'user_id', 'timestamp', 'event_type'}
        missing = required - set(df.columns)
        if missing:
            raise ValueError(f"Missing columns: {missing}")
        return df

    def clean(self, df: pd.DataFrame) -> pd.DataFrame:
        df = df.copy()
        df = df.drop_duplicates(subset=['user_id', 'timestamp'])
        df = df.dropna(subset=['user_id'])
        df['timestamp'] = pd.to_datetime(df['timestamp'], utc=True)
        return df

    def run(self, path: str) -> pd.DataFrame:
        return self.clean(self.load(path))

Performance at Scale

# Categorical dtype: 10x memory reduction for string columns
df['country'] = df['country'].astype('category')
df['status'] = df['status'].astype('category')

# Process large files in chunks
results = []
for chunk in pd.read_csv('huge_file.csv', chunksize=100_000):
    results.append(process_chunk(chunk))
final_df = pd.concat(results, ignore_index=True)

# Fast filtering with query strings
df.query('revenue > 1000 and country == "US" and age >= 18')

Pandas mastery is about knowing when to use which abstraction. Build testable pipelines and always profile before optimizing.

→ Validate your data transformations with the JSON Viewer tool.