Python Data Analysis

Every number in the report must be traceable to code that can be re-run. Profile before analysing, state assumptions, and separate what the data shows from what you infer.

Step 1 — Frame the question

Restate the question as one or more concrete, answerable queries ("median order value by month for 2025, excluding refunds"). Ask what decision the answer feeds; it changes what precision and which cuts matter. Note the grain of the data (one row = ?) before anything else.

Step 2 — Load and profile (always, even for "simple" questions)

import pandas as pd
df = pd.read_csv(path, low_memory=False)          # or pl.read_csv / pd.read_parquet / read_sql
print(df.shape); print(df.dtypes)
print(df.head(3).T)
print(df.isna().mean().sort_values(ascending=False).head(15))
print(df.describe(include='all').T)
print(df.nunique().sort_values().head(15))         # candidate keys and categoricals
dup = df.duplicated().sum(); print('duplicates', dup)

For files over ~1 GB or joins across several files, use DuckDB (duckdb.sql("select ... from 'file.parquet'")) or Polars lazy frames instead of pandas.

Record in a notes section: row count, date range, key columns, null rates over 5%, obvious outliers, duplicates, and any column whose meaning is unclear (ask).

Step 3 — Clean with explicit, logged rules

• Parse dates with an explicit format and timezone; never rely on inference silently.
• Cast numeric columns; investigate values that fail to cast rather than coercing them to NaN quietly.