数据匿名化技术建议

Proposed technique: deterministic pseudonymization using HMAC-SHA256 over a normalized phone number

Overview

• Goal: Replace raw phone numbers with a non-reversible, stable token that supports joins and analytics without exposing the original value.
• Method: Compute HMAC-SHA256(phone_normalized, secret_key) and store a truncated hex digest (e.g., first 32 hex chars = 128 bits) as phone_token.
• Properties:
  • Irreversible without the secret key (unlike encryption).
  • Deterministic for the same input, enabling de-duplication and joins across datasets.
  • Resistant to offline dictionary attacks because the key is required to compute candidate tokens.
  • Extremely low collision probability at 128 bits for typical data volumes.

Key design points

1. Canonicalization

   • Normalize all phone numbers to E.164 (e.g., +15551234567).
   • Strip spaces, hyphens, parentheses; enforce country codes; validate using a phone library at ingest.
   • Store the canonical value only in transient memory during transformation; do not persist raw values in analytical stores.

2. Token generation

   • Algorithm: HMAC-SHA256 with a secret key loaded at runtime from a secure store.
   • Output: 32 hex characters (128-bit truncation) or full 64 hex characters (256-bit) if storage is not constrained.
   • Collision risk at 128 bits is negligible for up to tens of millions of rows.

3. Secret management

   • Store the HMAC key in a managed secrets service (e.g., AWS Secrets Manager, HashiCorp Vault) or use KMS HMAC keys where supported.
   • Do not hard-code keys in SQL, code, configs, or environment variables committed to VCS.
   • Implement key rotation with versioned keys; include a key_id column or metadata to support re-tokenization over time.

4. Data model changes

   • Add column raw_user.phone_token CHAR(32) (hex) and optional key_id SMALLINT.
   • Restrict access to raw_user.phone (if retained) via column-level ACLs or move raw phone to a quarantined table with DLP controls.

5. Indexing and performance

   • Create a B-tree index on phone_token for equality lookups and joins.
   • Compute tokens in batch jobs (Spark/Beam) or in-database functions; avoid per-row network calls for keys.

Example implementations

PostgreSQL (pgcrypto)

• Prerequisite: CREATE EXTENSION IF NOT EXISTS pgcrypto;
• Sample backfill:
  • UPDATE raw_user SET phone_token = SUBSTRING(ENCODE(hmac(phone_normalized, :hmac_key, 'sha256'), 'hex') FOR 32) WHERE phone_token IS NULL AND phone_normalized IS NOT NULL;
  • Notes:
    • :hmac_key is supplied at runtime by the job, not stored in the DB.
    • phone_normalized is a canonical E.164 string prepared upstream or via a normalization function.

Apache Spark (PySpark)

• Canonicalization and tokenization:
  • from pyspark.sql import functions as F
  • import hmac, hashlib
  • SECRET = load_secret_at_runtime() # e.g., from Vault; do not hard-code
  • def hmac_sha256_hex_32(s): if s is None: return None digest = hmac.new(SECRET, s.encode('utf-8'), hashlib.sha256).hexdigest() return digest[:32]
  • hmac_udf = F.udf(hmac_sha256_hex_32)
  • df = ( raw_user_df .withColumn("phone_normalized", normalize_to_e164(F.col("phone"))) # implement normalization or use a library .withColumn("phone_token", hmac_udf(F.col("phone_normalized"))) )

Operational considerations

• Key rotation:
  • Maintain key_id in the table and in logs.
  • For rotation, compute new tokens with the new key alongside old tokens, then migrate reads to the new column and backfill.
• Monitoring:
  • Track token collision counts and null rates after normalization.
  • Alert on unexpected changes in token distribution that could indicate key misuse or normalization errors.
• Access control:
  • Limit access to the HMAC key to the transformation jobs only; deny interactive queries access to the key.
  • Prevent joins from tokenized data back to raw phone sources except in tightly controlled, approved workflows.
• Documentation:
  • Record algorithm, truncation length, key_id, normalization rules, and rotation procedures in data governance artifacts.

Alternative (if format preservation is required)

• Use format-preserving encryption (e.g., AES-FF1) to generate phone-like tokens that retain structure. This is reversible and therefore pseudonymization, not anonymization. It should be used only when a phone-like format is operationally necessary.

Conclusion

• HMAC-SHA256-based deterministic pseudonymization over E.164-normalized phone numbers provides strong, irreversible anonymization suitable for analytics and joins, with low collision risk and robust protection against dictionary attacks, provided the secret key is securely managed and rotated.

技术建议：基于分位分箱的泛化（保证每箱计数≥k 的 k-匿名）

目标

• 用区间替代精确金额，消除单值可识别性。
• 保留金额分布的统计可用性，适用于报表与分析（如区间计数、分布趋势）。

原理

• 将连续型 amount 映射为有限数量的区间（bins），并保证每个区间至少包含 k 条记录（k-匿名）。
• 采用分位分箱（ntile）使每箱样本量均衡，避免尾部区间样本过少导致重识别风险。

关键参数

• k：每区间最小记录数，例如 k=50（按风险容忍度调整）。
• B：分箱数，B = ceil(n / k)，其中 n 为 orders 总行数。
• 可选顶底编码阈值 [L, U]：对异常值截断，稳定分箱与后续分析。

实施步骤

1. 预处理与审计

   • 统一单位与精度（如两位小数）。
   • 可选：对 amount 做顶底编码，amount_capped = min(max(amount, L), U)。

2. 分箱计算

   • 计算总行数 n 与分箱数 B = ceil(n / k)。
   • 按 amount（或 amount_capped）排序，使用 ntile(B) 生成 bin_id。
   • 对每个 bin 计算 bin_min、bin_max，形成区间标签。

3. 替换与存储

   • 用区间标签（如 [bin_min, bin_max)）替换原始 amount。
   • 输出到派生表 orders_anonymized，并在元数据中记录 k、B、分箱时间与阈值 [L, U]。

4. 校验

   • 统计每个 bin 的计数，确保 count ≥ k。
   • 若 n < k，无法满足 k-匿名，应降低 k 或合并为单一区间。
   • 评估信息损失（例如分布拟合、分箱后与原始分布的 KS 检验）。

参考实现（PostgreSQL 示例）

• 说明：以下示例采用 k=50，按分位进行全局分箱，并用区间标签替换 amount。

WITH params AS ( SELECT 50::int AS k, 0::numeric AS L, 100000::numeric AS U ), prep AS ( SELECT o.order_id, LEAST(GREATEST(o.amount, p.L), p.U) AS amount_capped FROM orders o CROSS JOIN params p ), cnt AS ( SELECT COUNT()::numeric AS n FROM prep ), bins_param AS ( SELECT GREATEST(1, CEIL(c.n / (SELECT k FROM params)))::int AS B FROM cnt c ), ranked AS ( SELECT p., ntile(bp.B) OVER (ORDER BY p.amount_capped) AS bin_id FROM prep p CROSS JOIN bins_param bp ), bins AS ( SELECT bin_id, MIN(amount_capped)::numeric(18,2) AS bin_min, MAX(amount_capped)::numeric(18,2) AS bin_max, COUNT(*) AS bin_cnt FROM ranked GROUP BY bin_id ) INSERT INTO orders_anonymized (order_id, amount_bucket, bin_id) SELECT r.order_id, '[' || b.bin_min || ', ' || b.bin_max || ')' AS amount_bucket, r.bin_id FROM ranked r JOIN bins b USING (bin_id);

注意事项

• 若与其他列（如用户标识、时间戳、地点）联合使用，仍可能存在链接风险。建议对其他准标识符同步进行泛化或抑制。
• 分箱区间的选择影响分析精度；分位分箱适合保证隐私，若偏好业务语义（如价格带），可结合业务定义区间并在稀疏区间合并以满足 k。
• 对外发布聚合指标时，可进一步叠加差分隐私噪声以缓解组合攻击。
• 需进行版本化与隐私预算管理（记录参数与执行时间），避免多次不同切分导致信息泄露。