统计显著性概念解读

Executive summary

• Statistical significance assesses whether the observed lift is unlikely to be due to random variation; practical significance assesses whether the lift is large enough to matter for the business.
• With 2,000 users per arm, a 1.2 percentage-point increase in next-day retention is unlikely to be statistically significant for typical mobile app baselines and is underpowered to reliably detect such a small effect.
• Paid conversion shows no material change; session length should be evaluated with appropriate methods for skewed data.
• Recommendation: do not claim a definitive win yet. Extend the experiment or consider a limited rollout with strong monitoring, depending on product risk tolerance and the ROI of a 1.2pp retention lift.

What “statistical significance” means

• Definition: Statistical significance indicates the probability of observing the effect (or larger) under a “no effect” assumption (the null). If the p-value is below a pre-specified alpha (e.g., 0.05), we consider the result statistically significant.
• For binary metrics like next-day retention, significance depends on sample size, baseline rate, and variance. With n=2,000 per group, the minimum detectable difference at 95% confidence is often around 2–3 percentage points when baseline retention is in the 20–40% range, which is larger than 1.2pp.
• Practical check: compute the z-test for two proportions or a 95% confidence interval for the difference in retention. If the CI includes 0, the result is not statistically significant.

Practical vs. statistical significance

• Statistical significance: “Is the effect likely real given sampling variation?”
• Practical significance: “Is the effect big enough to change business outcomes after accounting for costs, risks, and opportunity?”
• A 1.2pp lift can be practically meaningful at scale (e.g., thousands of additional retained users per day), even if not yet statistically significant. Conversely, a statistically significant but tiny effect may not justify engineering/design costs or rollout risk.

Applying to your experiment

• Next-day retention: +1.2pp with 2,000 users per arm is probably below the threshold for statistical significance at common baselines. You need substantially more sample to confirm a 1.2pp lift with high power.
  • Approximate planning formula for two-proportion tests: n_per_arm ≈ [2·p·(1−p)·(z_{α/2}+z_{power})^2] / Δ^2 For α=0.05, power=80% (z_{α/2}=1.96, z_{power}=0.84), Δ=0.012:
    • If baseline p=0.30: n ≈ 22.9k per arm
    • If baseline p=0.20: n ≈ 17.5k per arm
    • If baseline p=0.10: n ≈ 9.8k per arm
• Paid conversion: “no material change” suggests no adverse guardrail impact; keep monitoring.
• Session length: distribution is often skewed. Use robust estimators (median, trimmed mean) or log-transform and a t-test; pre-define whether longer sessions are a goal or a byproduct to avoid misinterpreting engagement.

Release notes (draft, factual and non-overclaiming)

• Feature: New onboarding experience for first-time users.
• Experiment summary: Observed +1.2 percentage-point increase in next-day retention; paid conversion unchanged; session length neutral-to-slightly improved (analysis ongoing).
• Confidence: Current sample (n=2,000 per arm) is insufficient to establish statistical significance at 95% confidence. Further data collection is planned.
• Next steps: Continue experiment to reach required sample size; monitor guardrail metrics (paid conversion, crash rate, support contacts).

Roadmap and trade-off recommendations

• Decision framework:
  1. If risk is low and the experience aligns with strategic goals, consider a limited rollout (e.g., 20–30%) while continuing the experiment to confirm significance. Use staged gates based on retention CI and guardrails.
  2. If risk tolerance is low or there are competing priorities, extend the experiment until powered for a 1.2pp Minimal Detectable Effect (MDE), then decide on global release.
• Measurement plan:
  • Pre-register alpha (0.05), power (≥80%), primary metric (next-day retention), guardrails (paid conversion, stability), and secondary metric (session length).
  • Compute and report 95% CIs for all key metrics; avoid binary “win/lose” calls without CIs.
  • Consider heterogeneity: segment by device, region, acquisition channel. Roll out to segments showing consistent positive lift first.
  • Adjust for multiple comparisons if making claims on several metrics.
• Resource allocation:
  • Prioritize engineering/design work that scales the onboarding change only after statistical confirmation or a clear ROI case.
  • If the onboarding change is inexpensive to maintain and shows no downside, proceed with limited rollout and invest in follow-up iterations targeting larger effect sizes (e.g., simplify steps, personalize content).
  • Maintain focus on paid conversion as a guardrail; revisit monetization tests once retention effects are clearer.

Key takeaways

• Do not conflate statistical significance with business impact. Both matter.
• Given current sample size, the 1.2pp retention lift is promising but not yet statistically confirmable.
• Extend the test or take a cautious staged rollout, backed by robust monitoring and predefined decision criteria.

以下为在随机对照试验比较两种降压方案时，对“统计显著性”与“临床显著性”的通俗解释，以及样本量与阈值设置的合理化说明。目标是帮助您把结果解读到位，并用于实际决策。

一、统计意义（统计显著性）是什么

• 定义：统计显著性表示观察到的平均差异（这里为方案B比方案A多降3 mmHg）不太可能仅由随机波动造成。常用判定标准为双侧检验的显著性水平α=0.05，即在“实际没有差异”的前提下，错误判为有差异的概率不超过5%。
• 如何解读本试验：每组样本量为120例，随访8周。对收缩压这种指标，多数降压RCT中单个受试者的“变化值”标准差通常在10–12 mmHg量级。以此为参考：
  • 若标准差≈10 mmHg，3 mmHg的差异有较大机会达到统计显著（p<0.05）。
  • 若标准差≈12 mmHg，3 mmHg的差异可能处于边缘或不显著（p≈0.05上下）。
• 关键提醒：统计显著性不衡量效果大小是否“重要”，也不等同于临床获益。更稳健的做法是同时给出并解读该差异的95%置信区间（例如在上述变异度下，3 mmHg的95%CI大致可能在约0.5–5.5或-0.0–6.0 mmHg之间，是否跨越0决定统计显著与否；区间宽度反映不确定性）。

二、临床意义（临床显著性）如何看

• 直观含义：3 mmHg的平均收缩压降幅属于“温和”改善。对单个患者，是否值得更换方案，取决于综合因素：基线风险（高危患者对小幅降压更敏感）、药物安全性与耐受性、成本与可得性、用药便利性等。
• 人群层面：在人群中持续的小幅降压可累计带来事件风险的下降；但本试验仅随访8周，无法直接评估心血管事件结局，当前证据仅限于血压这一替代终点。
• 实务建议：将统计结果与预设的“临床重要差异阈值”（例如≥4–5 mmHg常被认为更有把握带来临床收益）结合评估。若差异虽统计显著但低于临床阈值，需谨慎决策；反之亦然。

三、样本量与阈值的合理化说明

• 显著性水平（α=0.05，双侧）：双侧检验是标准做法，既考虑B优于A，也考虑B劣于A；α=0.05用于控制错误报警率在可接受范围。
• 功效（常用目标80%）：设计时通常希望在真正存在“临床重要差异”时，有≥80%的概率检出它。
• 与本试验样本量的匹配：
  • 在每组120例、标准差10–12 mmHg的典型条件下，试验对“最小可可靠检出差异（80%功效）”的门槛约为3.6–4.3 mmHg。换言之，当前样本量对≥4–5 mmHg的差异较有把握，而对3 mmHg的差异把握不足。
  • 若目标就是稳健检测3 mmHg（80%功效，α=0.05双侧），根据上述变异度，样本量大致需要每组约170–250例。
• 阈值设置的业务逻辑：在降压领域，通常将4–5 mmHg作为更具临床意义的门槛来做样本量规划；这样既能控制资源投入，又能在统计与临床上取得较好的平衡。本试验每组120例更适合验证中等幅度（≥5 mmHg）差异，观察到的3 mmHg差异可能统计上不够稳健。

四、执行建议

• 报告要点：同时报告估计差异、p值和95%置信区间，明确预先设定的临床阈值（例如≥4–5 mmHg），并给出安全性与依从性数据。
• 决策规则：若3 mmHg差异达到统计显著但低于临床阈值，应结合患者风险、药物特性与成本做综合判断；若未达统计显著，当前样本量下结论应保守，可考虑扩大样本或延长随访以提高检出能力。
• 沟通方式：面向临床与管理团队，强调“统计显著性衡量的是随机性风险；临床显著性衡量的是患者可感知的获益”。两者需要同时满足，才值得改变治疗策略。

结论

• 在每组120例、随访8周的设计下，方案B相对方案A的3 mmHg降压差异属于“小幅改善”。是否统计显著取决于实际变异度；即便统计显著，仍需对其临床意义做独立评估。当前样本量更适合检出≥4–5 mmHg的差异；若目标是稳健确认3 mmHg的真实优势，建议增加样本量或调整设计。