推荐算法清单生成

Objective Provide an end-to-end, technically rigorous plan to evaluate a new feature via A/B testing, including an algorithm checklist, visualization recommendations, hypothesis validation steps, and five recommended analytical algorithms to drive iteration.

1. Experiment design and hypotheses

• Define primary hypothesis: e.g., H1: Treatment increases the primary metric by at least δ relative to control. H0: Δ = 0.
• Specify:
  • Unit of randomization and analysis (user/account/device; match exposure).
  • Primary metric and guardrails (e.g., conversion, revenue per user, latency, error rate).
  • Exposure/eligibility rules and attribution window.
  • Success criteria: minimum detectable effect (MDE), power, Type I error.
• Power/MDE planning (approximate, per arm):
  • For means: n ≈ 2 (z_{1-α/2} + z_{1-β})^2 σ^2 / δ^2.
  • For proportions: n ≈ 2 (z_{1-α/2} + z_{1-β})^2 p (1 - p) / δ^2, where p ≈ baseline rate.
  • If using CUPED with pre-period correlation ρ, required n scales by (1 - ρ^2).

2. Data requirements and preprocessing

• Sessionization and unit consistency: ensure one unit maps to one assignment; deduplicate exposures and bots.
• Timestamp hygiene and time zone normalization.
• Eligibility filters: include only exposed units; enforce consistent observation windows.
• Metric construction:
  • Ratios (e.g., revenue per user): define consistent denominators; consider winsorization or log transforms for heavy tails.
  • Funnels: construct stepwise binary events with coherent ordering.
  • Latency/error metrics: trim obvious outliers; cap at practical maxima.
• Covariates for adjustment:
  • Pre-experiment metric baselines (for CUPED/ANCOVA).
  • Stratification variables (geo, traffic source, device, user tenure).
• Missing data: classify MCAR/MAR/MNAR; record reasons; avoid imputation of outcomes for causal effect unless justified.

3. Assumption and integrity checks

• Sample ratio mismatch (SRM): chi-square test on assignment counts; investigate any SRM immediately.
• Covariate balance: standardized mean difference (SMD) < 0.1 for key covariates; visualize via “love plot.”
• Stable unit treatment value (no interference): verify no cross-unit spillovers; use cluster randomization if needed.
• Consistency: ensure randomized variant equals delivered experience; measure noncompliance.
• Temporal effects: check novelty and day-of-week/seasonality; ensure stable pre-trend in an A/A period if available.
• Independence/clustering: if users have repeated measures or are nested (e.g., within orgs), plan cluster-robust inference.
• Logging audits: identical event schemas and sampling across arms.

4. Core analysis algorithms (recommended 5)

1. OLS/GLM effect estimation with robust inference

• Use OLS for continuous metrics and GLM for binary/count metrics.
• Model: outcome ~ 1 + treatment + covariates, with cluster-robust (unit-level) standard errors or mixed-effects if needed.
• For ratio metrics, prefer:
  • Log-transform outcome and interpret as approximate percent change, or
  • GLM with appropriate link (Gamma-log for positive skew).
• Output: point estimate, 95% CI, p-value; percent lift and absolute difference.
• Use when: baseline estimation and clear interpretability are needed.

2. CUPED/ANCOVA variance reduction

• Regress outcome on pre-period baseline(s) and treatment; or compute CUPED-adjusted outcomes via Y_adj = Y - θ(X - E[X]), where θ = Cov(Y, X)/Var(X).
• Benefit: variance reduces by approximately (1 - R^2), where R^2 is from regressing Y on pre-period covariates (for single pre-period metric, ≈ 1 - ρ^2).
• Output: adjusted treatment effect with narrower CI; updated MDE.
• Use when: reliable pre-period metrics exist.

3. Bayesian hierarchical A/B modeling

• For proportions: Beta-Binomial per segment with partial pooling; for continuous: Normal with hierarchical priors on means/variances across segments.
• Outputs:
  • Posterior P(Δ > 0), expected loss under rollout, posterior for segment-level effects with shrinkage.
  • Decision rules: roll out if P(Δ > 0) > τ and expected loss < threshold.
• Use when: segment heterogeneity is expected and you want principled shrinkage and probability-based decisions.

4. Sequential testing (always-valid)

• Options:
  • mSPRT/e-values for continuous monitoring under optional stopping.
  • Group-sequential designs with alpha-spending (e.g., O’Brien–Fleming) at planned looks.
• Outputs: continuously updated evidence (likelihood ratio or e-value) and decision thresholds controlling Type I error.
• Use when: early stopping for efficacy/futility is desirable; ensure pre-specified monitoring plan.

5. Heterogeneous Treatment Effect (HTE) via uplift modeling

• Methods: causal forest, uplift gradient boosting (two-model T-learner with XGBoost/LightGBM), or doubly robust learners (R-learner).
• Outputs:
  • Individual/segment uplift estimates, uplift decile curves, optimal targeting policy evaluation (Qini/area under uplift curve).
• Use when: effect varies across users and you want to target rollout or design follow-up variants.

Supplementary techniques

• Distributional effects: Quantile Treatment Effects via quantile regression; ECDF/KS tests for shape changes.
• Inference for ratios/heavy tails: percentile bootstrap with clustering; or Fieller’s theorem/delta method for ratio CIs.
• Multiple testing: control FDR with Benjamini–Hochberg across many metrics/segments; or hierarchical modeling to mitigate false positives.

5. Visualization recommendations Integrity and setup

• SRM monitor: observed vs expected assignment share with control chart bands.
• Covariate balance “love plot”: SMD across covariates pre/post CUPED.

Primary effect and stability

• Cumulative lift over time with 95% CI; show both raw and CUPED-adjusted.
• Daily/batch difference plot with day-of-week bands to detect seasonality.
• Forest plot of segment effects (with shrinkage CIs under hierarchical model).

Distributional insights

• Overlaid ECDFs or ridgeline densities by arm; annotate median and selected quantiles.
• Quantile lift plot: Δ at p10/p25/median/p75/p90 with CIs.

Funnels and behavior

• Funnel bar charts by step with lift and propagated uncertainty.
• Sankey or stepwise drop-off comparisons.

Sequential monitoring

• Alpha-spending or e-value trajectory versus decision thresholds.
• Power curve vs elapsed samples; MDE over time.

Uplift/HTE

• Uplift decile curve (Qini); treatment policy gain vs random targeting.
• ICE/PD plots for top features driving heterogeneity.

Variance reduction diagnostics

• Pre vs post scatter with correlation and θ estimate for CUPED; R^2 and achieved variance reduction.

6. Decision framework and iteration

• Go/No-go criteria:
  • Primary metric Δ with CI; ensure guardrails within pre-defined bounds.
  • Bayesian: P(Δ > 0) and expected loss; Frequentist: two-sided 95% CI not crossing 0 and guardrails pass.
• Sensitivity analyses:
  • Trim/winsorize extremes; re-run with/without outliers.
  • Alternative denominators for ratios; user-level vs session-level aggregation.
  • Re-estimate with/without CUPED; check robustness to covariate set.
  • Cluster-robust vs naive SE; placebo tests on pre-period.
• If positive:
  • Ramp with sequential monitoring; maintain guardrails.
  • Use HTE to prioritize high-uplift segments; A/B within segments to validate targeting.
• If null/negative:
  • Diagnose via funnel breakdown and distributional plots.
  • Explore heterogeneity to identify subpopulations with positive lift.
  • Generate follow-up hypotheses: UX adjustments, eligibility tweaks, throttling, or alternative ranking logic; design next experiments with higher power and targeted cohorts.
• If inconclusive (wide CI):
  • Extend duration/samples; increase power via CUPED or stratified randomization.
  • Reduce metric variance (stabilized metrics, longer aggregation windows).

7. Practical implementation checklist

• Before launch:
  • Finalize hypotheses, metrics, power, randomization unit, and monitoring plan.
  • Implement A/A to validate variance, CUPED correlations, and logging.
  • Precompute covariates and pre-period baselines.
• During experiment:
  • Daily integrity checks (SRM, logging health).
  • Monitor guardrails with pre-specified sequential rules if applicable.
• After experiment:
  • Run primary OLS/GLM with cluster-robust SE; report CUPED-adjusted results.
  • Triangulate with bootstrap for non-normal/ratio metrics.
  • If multiple metrics/segments, apply FDR control or hierarchical shrinkage.
  • Conduct HTE analysis; prepare targeting recommendations.
  • Produce decision memo with effect sizes, uncertainty, risk assessment, and rollout plan.

Deliverables to stakeholders

• One-page summary: primary effect, CI, guardrails, decision.
• Technical appendix: methods (models, assumptions, diagnostics), sensitivity results, code references.
• Visual package: cumulative lift, forest plots, ECDF/quantile effects, funnel lift, uplift deciles, CUPED diagnostics, and sequential evidence curves.

This plan provides a statistically sound and operationally actionable approach to evaluate the new feature, validate assumptions, and drive iterative product decisions.

以下为针对“广告投放波动”的人群细分与效果归因算法清单（5种），并给出用于预算优化与创意调整的落地指引。内容聚焦在可操作的建模流程、所需数据、关键评估指标与优化策略。

一、数据准备与公共特征工程

• 数据结构
  • 曝光/点击/转化日志：user_id、time、channel、placement、campaign、adgroup、creative_id、cost、impression、click、conversion、revenue、device、geo、frequency、viewability、position、bid、竞价类型。
  • 补充变量：季节性（日/周/月）、节假日、促销、竞争活动指数、库存/价格、站内转化漏斗事件、APP版本、隐私开关（ATT 等）。
• 对齐与清洗
  • 去重与跨设备ID归并（概率图谱或deterministic mapping）。
  • 窗口化（lookback/attribution window），处理延迟转化。
  • Session 切分、序列化触点路径。
  • 反作弊和异常值处理（极端CTR/CVR、非人流量）。
• 变换与特征
  • 广告时序：adstock（几何衰减）、饱和（Hill/Logistic）、滞后项。
  • 业务控制变量：价格/库存/促销强度、自然流量与品牌检索指数。
  • 受众特征：人口统计、RFM、行为序列嵌入、上下文（时间、地点、设备）、隐私合规处理。
• 标签与目标
  • 短期：转化、收入、CPA、ROAS。
  • 中长期：LTV/CLV（可用生存-收入联合模型外推），用于增量收益评估。

二、算法清单（5种）与落地指南

算法1：贝叶斯营销组合模型（MMM，分层+Adstock+饱和）

• 适用场景
  • 渠道/媒体层面的宏观归因与预算分配，稳健应对日/周级波动、季节与外部冲击。
• 核心方法
  • 贝叶斯分层回归，媒体投入经adstock与饱和函数变换；加入季节/节假日/促销等控制项。
  • 稳健先验（如马蹄或Spike-and-Slab）做变量选择与防过拟合；不同地域/品类做层级共享。
• 所需数据
  • 周/日级渠道/活动聚合数据：花费、曝光（可选）、转化/收入、控制变量。
• 输出与指标
  • 渠道贡献、弹性、边际ROI（mROAS）、饱和点、半衰期；不确定性区间。
  • 评估：后验预测检验、时间外验证、MAPE/RMSE、假设检验。
• 预算优化指引
  • 基于mROAS进行非线性约束优化（预算、上下限、最低投放、配额），按收益-成本梯度迭代更新配比。
  • 中期（月/季度）配额用MMM，结合MTA短期信号做日常微调。
• 局限与缓解
  • 不能下钻到个体路径；用地理试验或BSTS校准增量效应。

算法2：多触点归因（Shapley值变体）

• 适用场景
  • 用户级路径归因，识别跨渠道/触点的协同与助攻，解释短期波动。
• 核心方法
  • 将触点（渠道/投放位/创意）视作“参与者”，对转化增量做Shapley分摊；用采样/启发式加速。
  • 可扩展到创意维度，获取创意级贡献份额。
• 所需数据
  • 用户级触点序列、时间戳、去重规则、view-through策略、跨设备映射、转化与价值。
• 输出与指标
  • 渠道/触点/创意的边际与联合贡献、助攻网络、路径覆盖度与效能。
  • 评估：回放检验（replay accuracy）、路径重建误差、对比标杆规则（位置/时序）提升。
• 预算与创意指引
  • 在同一渠道内按创意/投放位Shapley贡献重分配预算；提升高助攻位的频控上限、保留在关键序列节点的创意。
• 局限与缓解
  • 对稀疏长尾路径敏感，需做特征聚合与采样；与因果验证（实验/BSTS）结合避免相关性误判。

算法3：Uplift建模（因果森林/DR-Learner）

• 适用场景
  • 人群细分基于“增量效应”（非相关性），用于定向与创意个性化；应对选择偏差带来的效果波动。
• 核心方法
  • 以曝光为处理、转化为结果；先估计倾向得分与结果模型（双重稳健/AIPW），再用因果森林或DR-Learner估计异质性处理效应（HTE）。
• 所需数据
  • 用户特征、历史行为、曝光/未曝光对照（随机或准实验）、转化/收入、时间窗口；合规特征。
• 输出与指标
  • 人群分箱（Top X% uplift）、政策价值（policy value）、Qini/Gini uplift、AUUC；个体或细分层的mROAS。
• 预算与创意指引
  • 将预算优先分配给高uplift人群；对不同uplift段匹配差异化创意与频控策略。
  • 对低或负uplift人群降频或排除，减少浪费。
• 局限与缓解
  • 需要对未曝光样本的可比性保证；用IPW/DR、地理随机化或自然实验增强识别性。

算法4：高维人群聚类（GMM + 表征学习）

• 适用场景
  • 建立稳定、可解释的基础细分，用于定向、创意策略与出价分层；吸收短期波动带来的噪声。
• 核心方法
  • 行为序列/特征嵌入（Autoencoder/Transformer embedding）后，用高斯混合或谱聚类进行软/硬聚类；定期重训与漂移监控。
• 所需数据
  • RFM、浏览/加购/收藏序列、上下文、内容兴趣、价格敏感、设备/地域、生命周期阶段。
• 输出与指标
  • 集群画像、规模与渗透、转化/ARPU/留存差异、稳定性（轮廓系数、调整兰德指数、人口统计一致性）。
• 预算与创意指引
  • 对高价值/高潜力集群提高出价与预算上限；为各集群配置主题创意与落地页；对敏感度高的集群做价格/优惠型创意。
• 局限与缓解
  • 聚类不代表因果；与Uplift结果交叉用于“可转化性×可影响性”的二维分层。

算法5：结构化时序因果评估（BSTS/合成控制/差分中的差分）

• 适用场景
  • 波动诊断与增量验证：新活动、突发外部因素（竞品、节假日）、算法/创意切换的效果评估。
• 核心方法
  • 使用BSTS或合成控制构建反事实基线，估计特定时间段的增量；可结合变点检测定位波动来源。
• 所需数据
  • 按地域/店铺/媒体的时间序列，包含处理组与对照组的可比控制变量（自然量、竞品指数等）。
• 输出与指标
  • 增量转化/收入与置信区间、变点时刻、受影响渠道/地域、回归期长度。
• 预算与创意指引
  • 用于校准MMM与MTA的增量尺度；在波动期快速验证策略（如降频/换素材）的真实影响，指导是否扩大或回撤。
• 局限与缓解
  • 控制组构造难；通过多控制序列与先验稳健性检验提升可信度。

三、实施顺序与协同

• 第1步：波动诊断与增量基线（算法5）
  • 识别是否由外部冲击或内部策略变更导致，并估算基线缺口。
• 第2步：宏观归因与预算框架（算法1）
  • 确定各渠道的边际ROI与饱和点，形成月/季度级预算上限与下限。
• 第3步：路径与创意微观归因（算法2）
  • 在渠道内按触点/创意重分信用，发现关键助攻位与协同组合。
• 第4步：细分与个性化（算法3+4）
  • 用Uplift对“可影响性”分层，用聚类补充“人群画像”；形成“人群×创意×频控”的策略矩阵。
• 第5步：闭环验证
  • 使用地理/时间分组实验与BSTS复核增量，校准MMM和MTA权重；持续监控和重训。

四、评估与监控指标

• 增量与收益：增量转化/收入、mROAS、LTV-CPA差值、置信区间覆盖率。
• 归因一致性：MMM与MTA在渠道汇总层的偏差、对冲期一致性。
• 细分有效性：Qini/AUUC、policy value uplift、集群稳定性与业务差异显著性。
• 波动控制：变点检测命中率、预测误差（MAPE/RMSE）、策略变更前后对比。
• 实施效果：预算执行偏差、花费迁移的边际收益、创意轮播提升（CTR/CVR/转化成本）。

五、关键风险与校准建议

• 选择偏差与漏斗偏差：使用DR-Learner/IPW、引入随机化（geo试验、持出组）、负控特征做稳健性检验。
• 延迟与观测缺失：考虑延迟分布、右删失处理，滚动窗口再归因。
• 隐私与可测性：限制视图归因权重，采用聚合报告与差分隐私；用模型校准替代逐点跟踪。
• 模型漂移：设定重训节奏（周/月）、监控特征分布漂移与后验参数变化，触发再训练。

六、快速落地模板

• 数据管道：构建日/周级汇总表（供MMM/BSTS）与用户级路径表（供MTA/Uplift/聚类）。
• 首批产出
  • MMM：输出各渠道mROAS与饱和曲线，形成预算建议区间。
  • MTA-Shapley：输出前10条高贡献路径与创意协同对。
  • Uplift：给出Top 20%可影响人群名单及其推荐创意主题。
  • 聚类：3–8个稳定画像集群及差异化频控建议。
  • BSTS：对近期波动的增量评估与变点定位。
• 执行与验证：按建议迁移10–20%预算做小步快跑；用BSTS/地理实验验证2–4周，再逐步放大。

通过上述5类算法的协同，可同时解决宏观预算分配、微观路径归因、细分人群定向以及波动期的因果验证，形成“诊断-归因-优化-验证”的闭环，提高预算效率与创意匹配度。