制定假设的统计分析

Objective Assess whether the new guidance process increases the novice task completion rate compared with the current process.

Operational definitions

• Population: Novice users (e.g., first-time users or those with ≤1 prior completed task).
• Primary outcome: Task completion rate within a defined window (e.g., completion of the first assigned task within 7 days).
• Metric: Proportion of novices who complete the task (user-level binary outcome).

Statistical hypotheses

• Null (H0): The new guidance process does not increase the novice task completion rate versus control (p_new ≤ p_control).
• Alternative (H1): The new guidance process increases the novice task completion rate (p_new > p_control).

Success criterion (practical significance)

• Minimum detectable effect: Absolute increase of at least 5 percentage points in completion rate (e.g., from 40% to ≥45%), in addition to statistical significance.

Design and analysis plan

• Design: Randomized A/B test at the user level; stratify by key factors (e.g., platform, geography), and block by cohort start date to control time effects. Prevent contamination (e.g., one user sees only one process).
• Primary test: One-sided two-proportion z-test at α = 0.05.
• Estimation: Report risk difference and 95% CI; optionally use logistic regression with covariate adjustment for precision.
• Unit of analysis: User-level indicator to avoid within-user clustering issues.

Sample size illustration

• Assumptions: Baseline completion p_control = 0.40; target p_new = 0.46 (Δ = 0.06); α = 0.05 (one-sided); power = 0.80.
• Required per group ≈ [(1.645 + 0.842)² × (0.40×0.60 + 0.46×0.54)] / 0.06² ≈ 840 users per arm.
• Inflate for expected exclusions/attrition (e.g., 10–15%).

Decision rule Reject H0 and conclude improvement if:

• The observed difference (p_new − p_control) is statistically significant (p < 0.05, one-sided), and
• The effect meets the practical threshold (≥5 percentage points) or meets pre-specified business criteria.

Key risks and controls

• Seasonality/launch timing: Use blocking and run until sample size met.
• Imbalance: Check covariate balance post-randomization.
• Measurement: Predefine novice criteria and completion window; exclude invalid users (bots/duplicates).
• Analysis integrity: Pre-register hypotheses and analysis; avoid peeking and optional stopping.

Conclusion Hypothesis: The new guidance process increases novice task completion rates. Test via a randomized A/B experiment using a one-sided proportion test, with a practical improvement threshold of at least 5 percentage points and an estimated sample size of ~840 users per group under typical assumptions.

研究问题 不同获客渠道的用户首日留存率是否存在显著差异？

指标定义 首日留存率（D1）= 某渠道在统计日新增用户中，次日仍活跃的用户数 / 该渠道新增用户总数。所有渠道需使用一致的留存口径与时间窗口。

统计假设（总体）

• 原假设 H0：各渠道的首日留存率相等（p1 = p2 = … = pk）。
• 备择假设 H1：至少有一个渠道的首日留存率不同（∃ i,j 使 pi ≠ pj）。

检验方案

• 总体检验：构建“渠道 × 是否首日留存”的列联表，使用卡方检验（α=0.05，双侧）。同时报告效应量（Cramér’s V）。
• 事后比较（在拒绝H0时）：对渠道两两比较
  • 假设：H0_ij：pi = pj；H1_ij：pi ≠ pj。
  • 方法：两比例 z 检验（样本充足）或 Fisher 精确检验（任一格期望频数<5）。
  • 多重比较调整：控制假阳性，建议 Benjamini–Hochberg（FDR=0.05）或保守的 Bonferroni。
  • 输出：差异的绝对值（Δ=pi−pj）、相对比值（风险比或胜算比）、95%置信区间与p值。

关键假设与数据要求

• 每位用户仅归因于一个渠道；样本独立。
• 渠道归因准确且无重复、机器人流量干扰（必要时做数据清洗）。
• 相同统计期与版本/市场环境，确保可比性。
• 若需控制混杂（如国家、设备、版本、投放日期等），采用逻辑回归：Retained ~ Channel + Covariates，并检验渠道系数是否显著（βi=0 vs βi≠0）。

判定与报告

• 若总体卡方检验 p<0.05：判定渠道间首日留存存在显著差异；随后通过事后比较定位差异渠道，并报告效应量与区间估计。
• 若 p≥0.05：未观察到显著差异，应报告统计功效与样本量是否足够，以避免否定之误。