Understood. I will act as a data analysis expert and focus on accurate, concise, and actionable analysis. Please provide the inputs below so I can proceed efficiently.
What I need from you
- Objective and scope
- Primary question(s) and decision to support
- Problem type: classification, regression, forecasting, clustering, causal/A-B, anomaly detection, etc.
- Success metrics (e.g., ROC-AUC, RMSE, MAPE, uplift, p-value, confidence interval)
- Data access
- Data files or a sample (rows) with schema and data dictionary
- Row granularity and keys (unique IDs)
- Time columns, timezone, and aggregation level (if time-dependent)
- Known data quality issues, filters to apply, sensitive fields
- Constraints
- Compute limits, deadlines, privacy/compliance constraints
- Required interpretability or deployment environment
- Domain notes
- Definitions/units/currency, business rules, expected ranges, seasonality/events
- Known confounders or segments of interest
Proposed workflow
- Data ingestion and validation
- Type inference, schema checks, deduplication, identifier integrity, leakage scans.
- Exploratory data analysis (EDA)
- Distributions, missingness patterns, correlations/associations (Pearson/Spearman; Cramér’s V for categorical), target leakage checks, segment analysis.
- Outliers flagged via IQR/robust z-score; note: numeric summaries can suggest anomalies but precision improves with visualizations and clear thresholds.
- Data cleaning and preprocessing
- Missing handling (MCAR/MAR/MNAR assessment); imputation strategy per variable type.
- Encoding (one-hot/target), scaling (robust/standard), winsorization if justified.
- Feature creation with leakage prevention and clear provenance.
- Modeling or statistical analysis (as applicable)
- Baselines; CV scheme (stratified K-fold; time series CV with gaps); hyperparameter tuning.
- Regression/classification: regularized GLMs, tree ensembles, calibration; report effect sizes and uncertainty.
- Forecasting: decomposition, cross-validated ARIMA/ETS/Prophet; change points.
- A/B testing/causal: randomization checks, CUPED, sequential methods, diff-in-diff; state assumptions explicitly.
- Evaluation and interpretation
- Metrics with confidence intervals (bootstrap or analytic), error analysis, subgroup performance, stability checks.
- Explainability: permutation importance, SHAP (with caveats), partial dependence/ICE.
- Delivery
- Clear recommendations, risks/assumptions, next steps.
- Reproducible code, environment spec, and data versioning notes.
Minimal Python skeleton (ready to adapt once data is provided)
- General tabular (classification/regression)
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split, StratifiedKFold, KFold, cross_val_score
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.metrics import roc_auc_score, f1_score, mean_squared_error
from sklearn.linear_model import LogisticRegression, Ridge
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
def load_data(path):
df = pd.read_csv(path)
return df
def summarize(df):
out = {}
out['shape'] = df.shape
out['dtypes'] = df.dtypes.to_dict()
out['missing_by_col'] = df.isna().sum().sort_values(ascending=False).to_dict()
out['basic_stats'] = df.describe(include='all').to_dict()
return out
def flag_outliers_iqr(s, k=1.5):
q1, q3 = s.quantile([0.25, 0.75])
iqr = q3 - q1
lower, upper = q1 - kiqr, q3 + kiqr
return (s < lower) | (s > upper)
def build_pipeline(df, target, problem_type='classification'):
X = df.drop(columns=[target])
y = df[target]
num_cols = X.select_dtypes(include=['number']).columns.tolist()
cat_cols = X.select_dtypes(exclude=['number']).columns.tolist()
preproc = ColumnTransformer(
transformers=[
('num', StandardScaler(with_mean=False), num_cols),
('cat', OneHotEncoder(handle_unknown='ignore', sparse=False), cat_cols)
],
remainder='drop'
)
if problem_type=='classification':
model = LogisticRegression(max_iter=200, n_jobs=None)
pipe = Pipeline(steps=[('pre', preproc), ('clf', model)])
else:
model = Ridge(alpha=1.0)
pipe = Pipeline(steps=[('pre', preproc), ('reg', model)])
return X, y, pipe
def evaluate(pipe, X, y, problem_type='classification'):
if problem_type=='classification':
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scorer = 'roc_auc'
else:
cv = KFold(n_splits=5, shuffle=True, random_state=42)
scorer = 'neg_root_mean_squared_error'
scores = cross_val_score(pipe, X, y, cv=cv, scoring=scorer, n_jobs=-1)
return scores
- Time series (forecasting)
from sklearn.model_selection import TimeSeriesSplit
import statsmodels.api as sm
def ts_decompose(y, period):
return sm.tsa.seasonal_decompose(y, period=period, model='additive', two_sided=False)
def rolling_origin_cv(y, X=None, splits=5):
tscv = TimeSeriesSplit(n_splits=splits)
for train_idx, test_idx in tscv.split(y):
yield train_idx, test_idx
Quality and rigor practices I will follow
- Clear assumptions, versioned data and code, fixed random seeds, and leakage checks.
- Report uncertainty (confidence intervals, bootstrap), effect sizes, and practical significance.
- Robustness checks: alternative specifications, sensitivity to outliers/missingness, segment stability.
- Transparent limitations when visual support or explicit thresholds are absent.
Next step
- Share the dataset (or a sample) with schema and your objective/metrics. If sharing is not possible, provide:
- Column list with types, target variable, sample size, and brief description of each column.
- Whether data are time-dependent and any known data quality issues.
- I will then propose a tailored analysis plan and start with data validation and EDA.
