t检验结果解读

结论

• 在显著性水平α=0.05下，文案A与文案B的CTR存在统计学显著差异（t=2.10，df=58，p=0.040，双侧）。A显著优于B。
• 估计的绝对提升为0.3个百分点（3.2% vs 2.9%），相对提升约10.3%。

效应大小与区间估计

• 95%置信区间（基于给定t值推算）约为：0.01–0.59个百分点，排除零差异，表明差异稳健但幅度不大。
• 标准化效应大小：Cohen’s d≈0.54（Hedges’ g≈0.535），属于中等效应。

业务解读

• 在相同曝光量下，选择文案A预期可带来约10%的点击增量。鉴于绝对差异不大，收益规模取决于投放量与后续转化率。

方法与假设检查

• 使用的是两独立样本等方差t检验（df=58=nA+nB−2），关键假设包括：样本独立、组内方差相等、样本均值近似正态。
• CTR为比例数据，若每个观测对应的曝光量差异较大或方差不齐，建议采用更稳健做法：加权分析（按曝光加权）、Welch t检验，或基于二项/贝塔-二项的模型（如逻辑回归/层级模型），以避免偏差。

风险与注意事项

• p值接近0.05，属边际显著；若同时比较了多组文案或多渠道，需进行多重比较校正（如Bonferroni或FDR），校正后显著性可能减弱。
• 建议在新流量上复现实验以确认稳定性，并监控不同渠道/人群的异质性表现。

建议行动

• 短期：在控制风险的前提下优先投放文案A，并用曝光加权方式持续跟踪CTR差异与置信区间。
• 中期：在复现实验中预先设定功效。以当前效应大小（d≈0.54）为参考，要达到约80%功效、双侧α=0.05，样本量粗略需求约为每组≈50–60个观测（与当前n=30/组相比略有不足）。
• 长期：将分析升级为分渠道/分人群的模型化评估（含随机效应或协变量控制），并与后续指标（CVR、CPC、CPA）联动评估真实商业价值。

Summary

• Metric: 7-day retention (%)
• Group A: 15.0%; Group B: 15.3%
• Absolute difference (B − A): +0.30 percentage points (pp), ≈2% relative lift
• Two-sample t-test (equal variances): t = −0.85, df = 198, p = 0.397

Interpretation

• Not statistically significant at the 5% level; the data do not provide evidence that Group B differs from Group A.
• 95% CI for the difference (B − A): approximately −0.40 pp to +1.00 pp. The true effect could be a small decrease or up to a 1 pp increase.

Effect size and variance

• Cohen’s d ≈ 0.12 (small).
• Implied pooled standard deviation ≈ 2.49 pp under the equal-variance t-test used (df = nA + nB − 2).

Power and detectability

• With n = 100 per group and the observed variance, the study has roughly 80% power to detect a difference of about 1.0 pp or larger; smaller effects are underpowered.
• To detect a 0.3 pp effect with 80% power (two-sided, α = 0.05), you would need about 1,100 users per group (assuming the same variance).
• If your minimal meaningful effect is 0.5 pp, the current CI width suggests you would need roughly 2× the current sample (about 200 per group) to narrow the 95% CI half-width to ≈0.5 pp.

Assumptions and test choice

• The reported df indicate a pooled-variance t-test; this assumes equal variances and independent observations. With equal group sizes, the test is reasonably robust to mild variance differences.
• Because the metric is a proportion, a two-proportion z-test or logistic regression is also appropriate if user-level binary data are available. Results should be similar given the sample size.

Recommendation

• Do not conclude an improvement based on this test. The data are inconclusive for effects below ~1 pp.
• If your decision threshold is ≥1 pp, consider the result neutral; if you care about smaller effects, increase sample size as outlined.
• If the goal is to demonstrate “no meaningful difference,” predefine an equivalence margin and run a TOST equivalence test.