定义数据质量指标

Below are five data quality metrics tailored for transactional data. Each metric includes a precise definition and calculation approach suitable for governance dashboards and data quality SLAs.

• Mandatory Field Completeness Rate

  • Purpose: Ensure each transaction record contains all required attributes as defined by the data model and policy.
  • Definition: Percentage of transaction records for which all mandatory fields are non-null and non-empty.
  • Calculation: (Number of transactions with all mandatory fields populated ÷ Total number of transactions assessed) × 100.
  • Notes: Mandatory fields are defined in the data dictionary (e.g., transaction_id, event_timestamp, source_system_id, account/customer identifier(s), amount, currency, status). Measure also at field level (per-field null rate) to isolate problematic attributes. Exclude fields not applicable per business rules (e.g., counterparty_id for internal transfers).

• Value Validity Conformance Rate

  • Purpose: Verify adherence to format, type, range, code lists, and business rules.
  • Definition: Percentage of field-level validation checks passed across the transactional dataset.
  • Calculation: (Number of validation checks passed ÷ Total number of validation checks executed) × 100.
  • Notes: Include checks such as: ISO 4217 currency code list; amount numeric type and within allowed range; timestamp within allowed window and timezone standard; status in controlled vocabulary; instrument/channel codes valid; flags conform to schema. Implement rule severity (critical vs warning) and report separately if needed.

• Referential Integrity Error Rate

  • Purpose: Ensure all foreign keys in transactions correctly resolve to authoritative master/reference data.
  • Definition: Percentage of transaction records with at least one broken or missing foreign key reference.
  • Calculation: (Number of transactions with ≥1 invalid/missing FK reference ÷ Total number of transactions assessed) × 100.
  • Notes: Typical references include account, customer, product/instrument, merchant, and country/region tables. Measure per reference domain and in aggregate. Include “stale reference” checks where the referenced record exists but is not in an allowed status (e.g., closed account).

• Duplicate Transaction Rate

  • Purpose: Detect duplicate records that can lead to double counting, erroneous downstream processing, or compliance issues.
  • Definition: Percentage of transactions identified as duplicates within a defined matching window and business key.
  • Calculation: (Number of transactions flagged as duplicates ÷ Total number of transactions assessed) × 100.
  • Notes: Define a business key and time window—for example: exact match on transaction_id; or composite match on account_id + counterparty_id + amount + currency + near-time (e.g., ±60 seconds) + channel. Distinguish legitimate reversals/chargebacks/adjustments via status codes to avoid false positives. Report exact duplicates and near-duplicates separately.

• Timeliness (Ingestion Latency SLA Adherence)

  • Purpose: Ensure transactions are available to downstream consumers within required timeframes for operational and regulatory processes.
  • Definition: Percentage of transactions loaded, processed, and published to target systems within the SLA threshold.
  • Calculation: (Number of transactions with end-to-end latency ≤ SLA threshold ÷ Total number of transactions assessed) × 100.
  • Notes: Measure latency from event_timestamp (or source capture time) to availability in the consumption layer (e.g., data warehouse, API, reporting mart). Segment by source system and processing stage (ingest, transform, publish). Track distribution (p50/p95/p99) to identify tail latencies.

Implementation guidance:

• Establish rule ownership and stewardship per domain (e.g., Operations for timeliness, Reference Data for FK integrity).
• Document rule definitions and thresholds in the data policy and data quality standards, with risk-based targets (e.g., Completeness ≥ 99.5%, Validity ≥ 99.0%, FK Error ≤ 0.1%, Duplicate Rate ≤ 0.05%, Timeliness p95 ≤ SLA).
• Measure continuously (streaming) or in scheduled batches; maintain lineage and audit logs for all checks.
• Integrate exceptions into remediation workflows with root-cause categorization (source capture, transformation, reference data, schema changes).
• Provide drill-down reporting by field, system, product, and region to support targeted corrective actions.

以下指标面向“产品主数据（SKU/GTIN 级）”的持续性度量，适用于 MDM/PIM/ERP/渠道等系统的统一监控与治理。

1. 关键属性完整率（Completeness）

• 目的：衡量产品在其所属品类的必填属性是否齐全，支撑上架、交易、合规与履约。
• 范围：按品类配置必填集（示例）
  • 通用：SKU、GTIN、品名、品牌、类别、单位、税分类、生命周期状态、主图、长描述
  • 食品：净含量、保质期、过敏原、配料表、原产国
  • 电器：额定电压/功率、安全认证（如 CCC/CE）、能效等级
• 计算口径（属性级加权求和，建议作为默认）： 完整率% = Σ[对每个产品i、必填属性a的非空标记] / Σ[产品i的必填属性数量] × 100 即: Σ_i Σ_a 1{value(i,a) 非空} ÷ Σ_i |ReqAttr_i|
• 判定要点：仅判“是否有值”，格式/值域由“有效率”指标度量。
• 目标阈值（参考）：≥ 98%（上新/上架前关键属性可要求 100%）。

2. 标识符唯一率（Uniqueness）

• 目的：消除重复条目与冲突主键，防止库存、定价与结算差错。
• 范围：SKU（内部码）、GTIN（EAN/UPC）、供应商料号（VendorID+VendorPartNo 复合键）。
• 计算口径（按键分别度量）：
  • 唯一率_SKU = count(distinct SKU) / count(SKU)
  • 唯一率_GTIN = count(distinct GTIN where GTIN 非空) / count(GTIN 非空)
  • 唯一率_VPN = count(distinct (VendorID, VendorPartNo)) / count(VendorID, VendorPartNo)
• 判定要点：剔除作废/合并停用的历史重定向记录；对同一产品多包装、多条码需建立主从或变体关系，避免被误判为重复。
• 目标阈值（参考）：≥ 99.95%。

3. 值域与规则有效率（Validity）

• 目的：验证属性值符合标准值域、格式与业务校验规则，降低下游失败率与合规风险。
• 范围与规则示例：
  • 代码/参照：类别编码在主数据分类树内；UoM 属于允许值集（如 ISO/组织内部标准）；税分类、危化品分类在受控字典内
  • 格式/校验：GTIN 通过 GS1 校验位与长度规则；SKU 符合内部命名规范；图片分辨率/比例达标
  • 业务规则：可售=真 ⇒ 建议零售价>0；危化品=真 ⇒ 必填 UN 编码与运输限值；主包装尺寸/重量>0 且与单位匹配
• 计算口径： 有效率% = 通过全部已发布规则的属性值数量 / 被验证的属性值总数 × 100
• 目标阈值（参考）：≥ 99%（高风险合规字段≥ 99.9%）。

4. 跨系统一致率（Consistency）

• 目的：确保“单一事实来源”在 MDM/PIM/ERP/电商渠道之间对关键字段一致，避免价格、规格、状态等冲突。
• 范围：对齐到“权威源系统”的字段清单与容差（示例）
  • 文本类：品名、品牌、状态（大小写/空白规范化后严格相等）
  • 数值类：价格、尺寸、重量（设定容差，如尺寸±1 mm、价格=严格相等或币种换算后一致）
  • 枚举/参照：类别、税码（码值一致或经映射表一致）
• 计算口径： 一致率% = 符合对齐规则的字段比对数 / 参与比对的字段总数 × 100 或按产品维度统计：一致产品数 / 参与比对产品数 × 100
• 目标阈值（参考）：≥ 99%（价格/状态等交易关键字段要求 100%）。

5. 更新时效达标率（Timeliness）

• 目的：衡量新建/变更的产品信息在规定 SLA 内同步到下游与渠道的及时性，降低商机与合规滞后。
• 范围与场景示例：新商品从“审核通过”到“渠道可售”；价格变更生效；数字资产（主图/富媒体）上线。
• 计算口径（按事件类型分别度量）： 时效达标率% = 在 SLA 内完成的事件数 / 该事件总数 × 100 其中 延迟 = 发布时间（或可售生效时间）− 触发时间（审核通过/主数据变更落库）
• 目标阈值（参考）：上新 T+1 工作日内≥ 98%；价格变更 ≤ 4 小时≥ 99%。

实施与治理建议（通用）

• 分层监控：按品类、渠道、供应商、生命周期（开发/在售/停产）切片，识别结构性差异。
• 责任归属：为每个指标定义数据责任人（Data Owner/Steward）与处置流程（校验失败→工单→修复→复检）。
• 规则管理：在数据质量规则库中版本化管理必填集、值域、映射与容差；变更需经数据治理委员会审批。
• 度量频率：每日批量+关键事件实时校验；保留历史以观测趋势和回归。
• 门槛与豁免：对合规与交易关键字段设硬性阈值并阻断上线；对特例建立时间受限豁免并记录审计轨迹。