数据匿名化技术建议

建议技术：使用带密钥的确定性哈希（HMAC-SHA256）对 email 进行不可逆匿名化

核心原理
- 将标准化后的 email 使用一个受管密钥执行 HMAC-SHA256，生成固定长度的代币（token）。
- 该代币不可逆，还支持跨表或跨批次的确定性连接（同一 email 始终映射为同一 token）。
- 与无密钥哈希（如 SHA256）相比，HMAC可有效抵抗字典攻击，前提是密钥安全存储与管控。

设计要点
- 标准化输入：对 email 执行 trim、lower，并移除不可见字符，确保一致性。
- 输出编码：建议用 HEX（64字符）或 base64url（43字符，无填充）；生产中推荐 HEX，便于索引与比较。
- 可选保留域名：将 local-part 匿名化、domain保留，以支持域级分析（如 gmail.com），但会增加信息暴露面，需按业务评估。
- 密钥管理：密钥置于 KMS（如 AWS KMS、GCP KMS、Vault），服务端按需取用。密钥不落地到数据仓库或ETL日志。
- 版本与轮换：为 token增加 key_version 字段，轮换时并行计算新旧版本，逐步迁移。

实现示例（PostgreSQL，依赖扩展 pgcrypto）
1) 添加字段与索引
```
ALTER TABLE users ADD COLUMN email_token CHAR(64);
CREATE INDEX users_email_token_idx ON users(email_token);
```
2) 生成匿名化 token
```
-- :key 为服务端注入的密钥，不在SQL里常量化
UPDATE users
SET email_token = encode(hmac(lower(trim(email)), :key, 'sha256'), 'hex')
WHERE email IS NOT NULL;
```
3) 可选：保留域名做分析
```
ALTER TABLE users ADD COLUMN email_domain TEXT;
UPDATE users
SET email_domain = split_part(lower(trim(email)), '@', 2)
WHERE email IS NOT NULL;
```

实现示例（Apache Spark，Python）
```
import hmac, hashlib

def hmac_token(email: str, key: bytes) -> str:
    if email is None:
        return None
    v = email.strip().lower().encode('utf-8')
    return hmac.new(key, v, hashlib.sha256).hexdigest()  # 64-char HEX

from pyspark.sql import functions as F, types as T
key = b"<runtime-injected-kms-material>"  # 仅在驱动/执行器内存中持有

udf_hmac = F.udf(lambda e: hmac_token(e, key), T.StringType())
df = df.withColumn("email_token", udf_hmac(F.col("email"))) \
       .withColumn("email_domain", F.lower(F.element_at(F.split(F.col("email"), "@"), 2)))
```

运维与安全控制
- 访问控制：仅数据管道在计算时可访问密钥，数据仓库查询角色不得接触密钥或原始 email。
- 审计与密钥轮换：记录使用的 key_version；轮换时双写新旧 token，完成索引/下游迁移后再清理旧版。
- 性能与索引：对 email_token 建立索引用于连接；HEX字符串易于建立 B-tree 索引。
- 空值与错误：对非法 email 需设定处置策略（置空、丢弃、或记录到错误通道）。

风险与边界说明
- 不可逆性：HMAC 在密钥未知时不可逆；一旦密钥泄露，攻击者可离线验证猜测的 email。必须严格保护密钥。
- 冲突概率：SHA256 输出 256 位，理论碰撞极低；若截断长度，需评估碰撞风险（例如截断至 128 位仍极低，但应与业务可接受风险匹配）。
- 信息暴露：保留域名可能泄露部分敏感信息；若风险不可接受，应对整个 email 做 HMAC，不保留域名。
- 监管要求：HMAC 属于假名化技术，是否满足“匿名化”法律定义需结合场景与密钥隔离措施评估。

总结
- 使用 HMAC-SHA256 对 email 进行确定性不可逆代币化，是在数据工程中兼顾隐私保护与可连接性的高性价比方案。
- 关键在于输入标准化、KMS 管理与密钥轮换策略，以及对域名保留与截断长度的风险权衡。

Proposed technique: deterministic pseudonymization using HMAC-SHA256 over a normalized phone number

Overview
- Goal: Replace raw phone numbers with a non-reversible, stable token that supports joins and analytics without exposing the original value.
- Method: Compute HMAC-SHA256(phone_normalized, secret_key) and store a truncated hex digest (e.g., first 32 hex chars = 128 bits) as phone_token.
- Properties:
  - Irreversible without the secret key (unlike encryption). 
  - Deterministic for the same input, enabling de-duplication and joins across datasets.
  - Resistant to offline dictionary attacks because the key is required to compute candidate tokens.
  - Extremely low collision probability at 128 bits for typical data volumes.

Key design points
1. Canonicalization
   - Normalize all phone numbers to E.164 (e.g., +15551234567).
   - Strip spaces, hyphens, parentheses; enforce country codes; validate using a phone library at ingest.
   - Store the canonical value only in transient memory during transformation; do not persist raw values in analytical stores.

2. Token generation
   - Algorithm: HMAC-SHA256 with a secret key loaded at runtime from a secure store.
   - Output: 32 hex characters (128-bit truncation) or full 64 hex characters (256-bit) if storage is not constrained.
   - Collision risk at 128 bits is negligible for up to tens of millions of rows.

3. Secret management
   - Store the HMAC key in a managed secrets service (e.g., AWS Secrets Manager, HashiCorp Vault) or use KMS HMAC keys where supported.
   - Do not hard-code keys in SQL, code, configs, or environment variables committed to VCS.
   - Implement key rotation with versioned keys; include a key_id column or metadata to support re-tokenization over time.

4. Data model changes
   - Add column raw_user.phone_token CHAR(32) (hex) and optional key_id SMALLINT.
   - Restrict access to raw_user.phone (if retained) via column-level ACLs or move raw phone to a quarantined table with DLP controls.

5. Indexing and performance
   - Create a B-tree index on phone_token for equality lookups and joins.
   - Compute tokens in batch jobs (Spark/Beam) or in-database functions; avoid per-row network calls for keys.

Example implementations

PostgreSQL (pgcrypto)
- Prerequisite: CREATE EXTENSION IF NOT EXISTS pgcrypto;
- Sample backfill:
  - UPDATE raw_user
    SET phone_token = SUBSTRING(ENCODE(hmac(phone_normalized, :hmac_key, 'sha256'), 'hex') FOR 32)
    WHERE phone_token IS NULL AND phone_normalized IS NOT NULL;
  - Notes:
    - :hmac_key is supplied at runtime by the job, not stored in the DB.
    - phone_normalized is a canonical E.164 string prepared upstream or via a normalization function.

Apache Spark (PySpark)
- Canonicalization and tokenization:
  - from pyspark.sql import functions as F
  - import hmac, hashlib
  - SECRET = load_secret_at_runtime()  # e.g., from Vault; do not hard-code
  - def hmac_sha256_hex_32(s):
      if s is None:
          return None
      digest = hmac.new(SECRET, s.encode('utf-8'), hashlib.sha256).hexdigest()
      return digest[:32]
  - hmac_udf = F.udf(hmac_sha256_hex_32)
  - df = (
      raw_user_df
      .withColumn("phone_normalized", normalize_to_e164(F.col("phone")))  # implement normalization or use a library
      .withColumn("phone_token", hmac_udf(F.col("phone_normalized")))
  )

Operational considerations
- Key rotation:
  - Maintain key_id in the table and in logs.
  - For rotation, compute new tokens with the new key alongside old tokens, then migrate reads to the new column and backfill.
- Monitoring:
  - Track token collision counts and null rates after normalization.
  - Alert on unexpected changes in token distribution that could indicate key misuse or normalization errors.
- Access control:
  - Limit access to the HMAC key to the transformation jobs only; deny interactive queries access to the key.
  - Prevent joins from tokenized data back to raw phone sources except in tightly controlled, approved workflows.
- Documentation:
  - Record algorithm, truncation length, key_id, normalization rules, and rotation procedures in data governance artifacts.

Alternative (if format preservation is required)
- Use format-preserving encryption (e.g., AES-FF1) to generate phone-like tokens that retain structure. This is reversible and therefore pseudonymization, not anonymization. It should be used only when a phone-like format is operationally necessary.

Conclusion
- HMAC-SHA256-based deterministic pseudonymization over E.164-normalized phone numbers provides strong, irreversible anonymization suitable for analytics and joins, with low collision risk and robust protection against dictionary attacks, provided the secret key is securely managed and rotated.

技术建议：基于分位分箱的泛化（保证每箱计数≥k 的 k-匿名）

目标
- 用区间替代精确金额，消除单值可识别性。
- 保留金额分布的统计可用性，适用于报表与分析（如区间计数、分布趋势）。

原理
- 将连续型 amount 映射为有限数量的区间（bins），并保证每个区间至少包含 k 条记录（k-匿名）。
- 采用分位分箱（ntile）使每箱样本量均衡，避免尾部区间样本过少导致重识别风险。

关键参数
- k：每区间最小记录数，例如 k=50（按风险容忍度调整）。
- B：分箱数，B = ceil(n / k)，其中 n 为 orders 总行数。
- 可选顶底编码阈值 [L, U]：对异常值截断，稳定分箱与后续分析。

实施步骤
1) 预处理与审计
   - 统一单位与精度（如两位小数）。
   - 可选：对 amount 做顶底编码，amount_capped = min(max(amount, L), U)。

2) 分箱计算
   - 计算总行数 n 与分箱数 B = ceil(n / k)。
   - 按 amount（或 amount_capped）排序，使用 ntile(B) 生成 bin_id。
   - 对每个 bin 计算 bin_min、bin_max，形成区间标签。

3) 替换与存储
   - 用区间标签（如 [bin_min, bin_max)）替换原始 amount。
   - 输出到派生表 orders_anonymized，并在元数据中记录 k、B、分箱时间与阈值 [L, U]。

4) 校验
   - 统计每个 bin 的计数，确保 count ≥ k。
   - 若 n < k，无法满足 k-匿名，应降低 k 或合并为单一区间。
   - 评估信息损失（例如分布拟合、分箱后与原始分布的 KS 检验）。

参考实现（PostgreSQL 示例）
- 说明：以下示例采用 k=50，按分位进行全局分箱，并用区间标签替换 amount。

WITH
params AS (
  SELECT 50::int AS k, 0::numeric AS L, 100000::numeric AS U
),
prep AS (
  SELECT
    o.order_id,
    LEAST(GREATEST(o.amount, p.L), p.U) AS amount_capped
  FROM orders o CROSS JOIN params p
),
cnt AS (
  SELECT COUNT(*)::numeric AS n FROM prep
),
bins_param AS (
  SELECT GREATEST(1, CEIL(c.n / (SELECT k FROM params)))::int AS B
  FROM cnt c
),
ranked AS (
  SELECT
    p.*,
    ntile(bp.B) OVER (ORDER BY p.amount_capped) AS bin_id
  FROM prep p CROSS JOIN bins_param bp
),
bins AS (
  SELECT
    bin_id,
    MIN(amount_capped)::numeric(18,2) AS bin_min,
    MAX(amount_capped)::numeric(18,2) AS bin_max,
    COUNT(*) AS bin_cnt
  FROM ranked
  GROUP BY bin_id
)
INSERT INTO orders_anonymized (order_id, amount_bucket, bin_id)
SELECT
  r.order_id,
  '[' || b.bin_min || ', ' || b.bin_max || ')' AS amount_bucket,
  r.bin_id
FROM ranked r
JOIN bins b USING (bin_id);

注意事项
- 若与其他列（如用户标识、时间戳、地点）联合使用，仍可能存在链接风险。建议对其他准标识符同步进行泛化或抑制。
- 分箱区间的选择影响分析精度；分位分箱适合保证隐私，若偏好业务语义（如价格带），可结合业务定义区间并在稀疏区间合并以满足 k。
- 对外发布聚合指标时，可进一步叠加差分隐私噪声以缓解组合攻击。
- 需进行版本化与隐私预算管理（记录参数与执行时间），避免多次不同切分导致信息泄露。