Objective
Provide an end-to-end, technically rigorous plan to evaluate a new feature via A/B testing, including an algorithm checklist, visualization recommendations, hypothesis validation steps, and five recommended analytical algorithms to drive iteration.
1) Experiment design and hypotheses
- Define primary hypothesis: e.g., H1: Treatment increases the primary metric by at least δ relative to control. H0: Δ = 0.
- Specify:
- Unit of randomization and analysis (user/account/device; match exposure).
- Primary metric and guardrails (e.g., conversion, revenue per user, latency, error rate).
- Exposure/eligibility rules and attribution window.
- Success criteria: minimum detectable effect (MDE), power, Type I error.
- Power/MDE planning (approximate, per arm):
- For means: n ≈ 2 (z_{1-α/2} + z_{1-β})^2 σ^2 / δ^2.
- For proportions: n ≈ 2 (z_{1-α/2} + z_{1-β})^2 p (1 - p) / δ^2, where p ≈ baseline rate.
- If using CUPED with pre-period correlation ρ, required n scales by (1 - ρ^2).
2) Data requirements and preprocessing
- Sessionization and unit consistency: ensure one unit maps to one assignment; deduplicate exposures and bots.
- Timestamp hygiene and time zone normalization.
- Eligibility filters: include only exposed units; enforce consistent observation windows.
- Metric construction:
- Ratios (e.g., revenue per user): define consistent denominators; consider winsorization or log transforms for heavy tails.
- Funnels: construct stepwise binary events with coherent ordering.
- Latency/error metrics: trim obvious outliers; cap at practical maxima.
- Covariates for adjustment:
- Pre-experiment metric baselines (for CUPED/ANCOVA).
- Stratification variables (geo, traffic source, device, user tenure).
- Missing data: classify MCAR/MAR/MNAR; record reasons; avoid imputation of outcomes for causal effect unless justified.
3) Assumption and integrity checks
- Sample ratio mismatch (SRM): chi-square test on assignment counts; investigate any SRM immediately.
- Covariate balance: standardized mean difference (SMD) < 0.1 for key covariates; visualize via “love plot.”
- Stable unit treatment value (no interference): verify no cross-unit spillovers; use cluster randomization if needed.
- Consistency: ensure randomized variant equals delivered experience; measure noncompliance.
- Temporal effects: check novelty and day-of-week/seasonality; ensure stable pre-trend in an A/A period if available.
- Independence/clustering: if users have repeated measures or are nested (e.g., within orgs), plan cluster-robust inference.
- Logging audits: identical event schemas and sampling across arms.
4) Core analysis algorithms (recommended 5)
1. OLS/GLM effect estimation with robust inference
- Use OLS for continuous metrics and GLM for binary/count metrics.
- Model: outcome ~ 1 + treatment + covariates, with cluster-robust (unit-level) standard errors or mixed-effects if needed.
- For ratio metrics, prefer:
- Log-transform outcome and interpret as approximate percent change, or
- GLM with appropriate link (Gamma-log for positive skew).
- Output: point estimate, 95% CI, p-value; percent lift and absolute difference.
- Use when: baseline estimation and clear interpretability are needed.
2. CUPED/ANCOVA variance reduction
- Regress outcome on pre-period baseline(s) and treatment; or compute CUPED-adjusted outcomes via Y_adj = Y - θ(X - E[X]), where θ = Cov(Y, X)/Var(X).
- Benefit: variance reduces by approximately (1 - R^2), where R^2 is from regressing Y on pre-period covariates (for single pre-period metric, ≈ 1 - ρ^2).
- Output: adjusted treatment effect with narrower CI; updated MDE.
- Use when: reliable pre-period metrics exist.
3. Bayesian hierarchical A/B modeling
- For proportions: Beta-Binomial per segment with partial pooling; for continuous: Normal with hierarchical priors on means/variances across segments.
- Outputs:
- Posterior P(Δ > 0), expected loss under rollout, posterior for segment-level effects with shrinkage.
- Decision rules: roll out if P(Δ > 0) > τ and expected loss < threshold.
- Use when: segment heterogeneity is expected and you want principled shrinkage and probability-based decisions.
4. Sequential testing (always-valid)
- Options:
- mSPRT/e-values for continuous monitoring under optional stopping.
- Group-sequential designs with alpha-spending (e.g., O’Brien–Fleming) at planned looks.
- Outputs: continuously updated evidence (likelihood ratio or e-value) and decision thresholds controlling Type I error.
- Use when: early stopping for efficacy/futility is desirable; ensure pre-specified monitoring plan.
5. Heterogeneous Treatment Effect (HTE) via uplift modeling
- Methods: causal forest, uplift gradient boosting (two-model T-learner with XGBoost/LightGBM), or doubly robust learners (R-learner).
- Outputs:
- Individual/segment uplift estimates, uplift decile curves, optimal targeting policy evaluation (Qini/area under uplift curve).
- Use when: effect varies across users and you want to target rollout or design follow-up variants.
Supplementary techniques
- Distributional effects: Quantile Treatment Effects via quantile regression; ECDF/KS tests for shape changes.
- Inference for ratios/heavy tails: percentile bootstrap with clustering; or Fieller’s theorem/delta method for ratio CIs.
- Multiple testing: control FDR with Benjamini–Hochberg across many metrics/segments; or hierarchical modeling to mitigate false positives.
5) Visualization recommendations
Integrity and setup
- SRM monitor: observed vs expected assignment share with control chart bands.
- Covariate balance “love plot”: SMD across covariates pre/post CUPED.
Primary effect and stability
- Cumulative lift over time with 95% CI; show both raw and CUPED-adjusted.
- Daily/batch difference plot with day-of-week bands to detect seasonality.
- Forest plot of segment effects (with shrinkage CIs under hierarchical model).
Distributional insights
- Overlaid ECDFs or ridgeline densities by arm; annotate median and selected quantiles.
- Quantile lift plot: Δ at p10/p25/median/p75/p90 with CIs.
Funnels and behavior
- Funnel bar charts by step with lift and propagated uncertainty.
- Sankey or stepwise drop-off comparisons.
Sequential monitoring
- Alpha-spending or e-value trajectory versus decision thresholds.
- Power curve vs elapsed samples; MDE over time.
Uplift/HTE
- Uplift decile curve (Qini); treatment policy gain vs random targeting.
- ICE/PD plots for top features driving heterogeneity.
Variance reduction diagnostics
- Pre vs post scatter with correlation and θ estimate for CUPED; R^2 and achieved variance reduction.
6) Decision framework and iteration
- Go/No-go criteria:
- Primary metric Δ with CI; ensure guardrails within pre-defined bounds.
- Bayesian: P(Δ > 0) and expected loss; Frequentist: two-sided 95% CI not crossing 0 and guardrails pass.
- Sensitivity analyses:
- Trim/winsorize extremes; re-run with/without outliers.
- Alternative denominators for ratios; user-level vs session-level aggregation.
- Re-estimate with/without CUPED; check robustness to covariate set.
- Cluster-robust vs naive SE; placebo tests on pre-period.
- If positive:
- Ramp with sequential monitoring; maintain guardrails.
- Use HTE to prioritize high-uplift segments; A/B within segments to validate targeting.
- If null/negative:
- Diagnose via funnel breakdown and distributional plots.
- Explore heterogeneity to identify subpopulations with positive lift.
- Generate follow-up hypotheses: UX adjustments, eligibility tweaks, throttling, or alternative ranking logic; design next experiments with higher power and targeted cohorts.
- If inconclusive (wide CI):
- Extend duration/samples; increase power via CUPED or stratified randomization.
- Reduce metric variance (stabilized metrics, longer aggregation windows).
7) Practical implementation checklist
- Before launch:
- Finalize hypotheses, metrics, power, randomization unit, and monitoring plan.
- Implement A/A to validate variance, CUPED correlations, and logging.
- Precompute covariates and pre-period baselines.
- During experiment:
- Daily integrity checks (SRM, logging health).
- Monitor guardrails with pre-specified sequential rules if applicable.
- After experiment:
- Run primary OLS/GLM with cluster-robust SE; report CUPED-adjusted results.
- Triangulate with bootstrap for non-normal/ratio metrics.
- If multiple metrics/segments, apply FDR control or hierarchical shrinkage.
- Conduct HTE analysis; prepare targeting recommendations.
- Produce decision memo with effect sizes, uncertainty, risk assessment, and rollout plan.
Deliverables to stakeholders
- One-page summary: primary effect, CI, guardrails, decision.
- Technical appendix: methods (models, assumptions, diagnostics), sensitivity results, code references.
- Visual package: cumulative lift, forest plots, ECDF/quantile effects, funnel lift, uplift deciles, CUPED diagnostics, and sequential evidence curves.
This plan provides a statistically sound and operationally actionable approach to evaluate the new feature, validate assumptions, and drive iterative product decisions.
