创建数据术语词条

Term: Data Lineage

Definition: Data lineage is the complete, end-to-end record of how data moves, transforms, and is used across an organization’s ecosystem. It captures the origin of data, the systems and processes that ingest, transform, and deliver it, and the downstream assets and consumers that depend on it. Lineage is expressed as connected metadata describing datasets, fields, processes, jobs, systems, and their directional relationships over time.

Purpose:

• Traceability and accountability for critical data used in reporting, analytics, and operations.
• Impact analysis for changes, incidents, and regulatory inquiries.
• Risk reduction through visibility into dependencies, controls, and data usage.
• Evidence for compliance with internal policies and external regulations.

Scope and Granularity:

• Coverage: End-to-end (from external sources to consumption) across batch, streaming, API, and manual processes.
• Levels:
  • System-level: Movement between platforms or domains.
  • Dataset-level: Flow between tables, files, topics, reports.
  • Field/column-level: Derivations and transformations for individual attributes.
  • Record-level (provenance): Event- or record-specific lineage when required by high-assurance use cases.

Lineage Types:

• Business lineage: Abstract, understandable view aligned to business terms and processes.
• Technical lineage: Detailed implementation view derived from code, jobs, and runtime metadata.
• Logical lineage: Conceptual relationships independent of specific technologies.
• Physical lineage: Concrete paths across specific systems and assets.
• Static lineage: Derived from code and configurations.
• Runtime/operational lineage: Derived from executed jobs and runtime telemetry.
• Bidirectional lineage: Upstream (origins) and downstream (impacts).

Core Components:

• Nodes:
  • Dataset/asset (e.g., table, file, topic, report, model, API).
  • Field/attribute.
  • Process/transformation (e.g., SQL query, Spark job, ELT step).
  • Job/run (execution instance with timestamps and status).
  • System/application (platforms hosting assets/processes).
  • Actor/role (human or service).
• Edges:
  • Consumes/produces.
  • Derives from (field-level).
  • Reads/writes.
  • Executes/controlled by.
• Attributes:
  • Timestamps, versions, code references, configurations, data classifications, controls applied, data contracts, data quality checks.

Representation Model:

• Directed acyclic graphs (DAGs) for deterministic flows; may include cycles for iterative workflows.
• Temporal lineage: Versioned representations per code version and per job run for point-in-time reconstruction.
• Semantics:
  • Entities and relationships must be uniquely identifiable and resolvable.
  • Transformations should encode logic where feasible (e.g., SQL parse trees, UDF references).
  • Lineage should link to business glossary terms, policies, data quality rules, and ownership.

Capture and Maintenance Methods:

• Automated harvesting:
  • Static code parsing (SQL, ETL configurations, orchestration DAGs).
  • Runtime instrumentation and event capture from executors and orchestrators.
  • Connectors to catalogs, ETL/ELT tools, BI/reporting platforms, and streaming frameworks.
• Declared/manual lineage:
  • Curated mappings for opaque processes, legacy systems, or external vendors.
• Reconciliation and validation:
  • Compare declared lineage with observed runtime events.
  • Detect breaks due to schema drift, refactors, or undocumented processes.
• Change management:
  • Version control integration and CI/CD checks for lineage-impact assertions.
  • Approval workflows for lineage updates.

Governance Controls and Roles:

• Policies:
  • Mandatory lineage for Critical Data Elements (CDEs), regulatory reports, and PII flows.
  • Minimum granularity requirements (e.g., column-level for financial metrics).
  • Freshness SLAs for lineage updates tied to deployment and execution.
  • Access controls to prevent exposure of sensitive code, keys, or PII in lineage views.
  • Vendor and data-sharing lineage obligations in contracts.
• Roles and responsibilities:
  • Data Owner: Ensures policy adherence for owned domains.
  • Data Steward: Curates and validates lineage, resolves breaks, aligns with glossary.
  • Engineering/Platform: Implements harvesters, storage, APIs, and access control.
  • Producers/Developers: Provide code annotations and data contracts.
  • Risk/Compliance/Audit: Consumes lineage as evidence and for testing controls.

Quality Metrics (examples):

• Coverage: Percentage of CDEs with end-to-end lineage to consumption.
• Granularity: Percentage of lineage at column level for scoped assets.
• Accuracy: Ratio of validated edges vs total edges; discrepancy rate vs runtime events.
• Freshness: Median lag between execution and lineage availability.
• Completeness: Number of orphan nodes or dangling edges.
• Stability: Rate of lineage-breaking changes detected per release.

Common Use Cases:

• Impact analysis for schema changes, deprecations, and platform migrations.
• Root cause analysis for data incidents and quality regressions.
• Regulatory traceability for financial reporting, risk aggregation, and privacy obligations.
• Data access reviews and least-privilege validation.
• Model governance: Track features, training data, and model outputs to sources.
• Cost and performance optimization by identifying redundant pipelines and unused assets.

Risks and Pitfalls:

• Incomplete capture leading to false assurance.
• Overexposure of sensitive logic or data in lineage UI.
• Stale lineage due to missing runtime events or failed harvesters.
• Opaque transformations (UDFs, scripts) without declared mappings.
• Fragmentation across tools without a common model or identifiers.
• Confusing business users with overly technical views; lack of business lineage.

Standards and Interoperability:

• OpenLineage: Open standard for lineage events from data pipelines.
• W3C PROV: Conceptual model for provenance (useful for record-level provenance and interchange).
• ISO/IEC 11179: Metadata registry principles relevant to entity identification.
• Align lineage metadata with enterprise glossary, data contracts, and quality frameworks.

Related Terms:

• Data provenance: Event-level or record-level history and evidence; lineage is broader and typically at dataset/field abstraction.
• Data catalog: System that often stores and visualizes lineage with metadata and glossary.
• Data contract: Formal producer–consumer agreement that can be enforced and traced via lineage.
• Critical Data Element (CDE): High-priority elements requiring stricter lineage controls.

Example (conceptual): Customer table fields are ingested nightly from CRM to a raw zone, standardized in a curated zone, joined with Orders to compute Customer_LTV in the Analytics schema, and visualized in a Finance dashboard. Column-level lineage maps Customer_LTV to its source fields and transformation logic, with job runs timestamped and linked to data quality checks and owners.

Notes for Implementation:

• Prioritize CDEs and regulated data flows for column-level, runtime-validated lineage.
• Use a central lineage store with APIs; enforce unique identifiers for systems, datasets, and fields.
• Integrate lineage checks into CI/CD and orchestrator hooks; alert on breaks and stale edges.
• Present both business and technical views, filtered by role-based access controls.

术语：敏感个人信息（个人敏感信息）

定义： 敏感个人信息是指一类个人信息，其泄露或被非法使用，容易导致对个人尊严造成侵害或对人身、财产安全造成严重危害。依据《中华人民共和国个人信息保护法》（PIPL）第28条，敏感个人信息包括但不限于：生物识别特征、宗教信仰、特定身份、医疗健康、金融账户、行踪轨迹，以及14周岁以下未成年人的个人信息。

术语边界与适用范围：

• 适用主体：所有在中国境内处理个人信息的组织与个人，以及对境内自然人个人信息进行处理的境外组织（符合PIPL适用条件）。
• 信息载体：电子与纸质形态的数据，含结构化与非结构化数据（日志、多媒体、文档、影像等）。
• 风险属性：以结果风险为导向的认定，除法定列举外，结合处理目的、方式、规模与可能影响，在具体场景中可将其他个人信息认定为敏感。

典型类别与示例（非穷尽）：

• 生物识别特征：指纹、人脸、虹膜、声纹、步态、DNA等。
• 宗教信仰：个人的宗教归属与相关活动记录。
• 特定身份信息：指能够体现个人在特定社会身份类别中的属性与标识的信息（依据具体法律场景认定）。
• 医疗健康：诊断、病历、检验结果、处方、残疾信息、心理与精神健康记录等。
• 金融账户：银行账户号、支付账户及其鉴权要素、账户余额与交易明细等。
• 行踪轨迹：定位数据、出行路径、连续位置记录（含基站、GPS、Wi‑Fi定位等）。
• 未满14周岁未成年人信息：任何与该年龄段未成年人相关的个人信息。

法律与合规要求（中国，PIPL为主）：

• 事前评估：在处理敏感个人信息前开展个人信息保护影响评估（PIPI），形成并留存评估报告与处理记录（PIPL第55条）。
• 单独同意：取得个人的“单独同意”，并向其告知必要性、处理目的、方式、范围、影响与保护措施（PIPL第29条）。
• 最小必要：遵循最小化原则，仅在达成明确、正当目的所必需的范围内收集与处理（PIPL第6条）。
• 强化保护措施：采取加密、去标识化/脱敏、访问控制、分级授权、日志审计与安全存储等措施（PIPL第51条）。
• 展示与公开：对外呈现时默认采取去标识化或必要的脱敏处理，避免可识别性与过度暴露。
• 跨境传输：开展安全评估或认证/签署标准合同，并履行告知与单独同意等要求（PIPL第38—43条及配套办法）。
• 权利保障：支持查询访问、更正、删除、撤回同意等权利；对未成年人需监护人同意与专门保护（PIPL第31、38条及相关条款）。
• 数据泄露应急：建立并演练处置流程，发生泄露时依法告知与报告（PIPL第57条）。

与其他法域的对应关系（简要）：

• 欧盟GDPR的“特殊类别个人数据”（special categories of personal data）包括：种族或民族、政治观点、宗教或哲学信仰、工会会员、遗传和生物识别数据（用于唯一识别自然人）、健康、性生活与性取向。该范围与PIPL存在交叉但不完全等同。
• 跨法域治理需进行映射与最严格规则对齐，避免合规空档。

数据治理要求与控制框架：

• 分类分级：在数据目录中建立“敏感个人信息”标签与分级规则（如S3/最高级），纳入元数据管理与血缘追踪。
• 准入与授权：实行基于角色与属性的访问控制（RBAC/ABAC），默认拒绝策略，临时授权与用途绑定，定期复核。
• 数据生命周期：从采集、传输、存储、使用、共享到销毁的端到端控制；明确保留期限与销毁标准（可审计）。
• 技术措施：加密（传输与静态、密钥分级管理）、脱敏（掩码、泛化、哈希、令牌化）、隔离（逻辑与物理）、数据泄露防护（DLP）。
• 运营与监督：设立数据所有者与数据保管人职责，建立审批与变更控制；对涉及敏感个人信息的项目强制PIPI与安全测试。
• 第三方管理：对受托处理者实施尽职审查、合同条款（保密、用途限制、子处理约束）、持续评估与现场审计。
• 监控与度量：关键指标包括分类准确率、PIPI覆盖率、单独同意获取率、访问违规率、DLP拦截事件、数据泄露MTTR、跨境合规完成率。

常见误区与校正：

• 误将所有身份证件号码一概认定为敏感：在中国法下，身份证号通常为个人信息；是否构成敏感取决于处理场景与风险（基于结果风险原则）。在高风险用途（如鉴权、金融交易）中应按敏感级别管理。
• 单独同意不足以替代最小化与评估：合规需同时满足必要性、PIPI与强化技术措施。
• 脱敏非一劳永逸：需评估重识别风险，结合数据用途动态调整技术策略。

参考依据（提名）：

• 《中华人民共和国个人信息保护法》：第6、28、29、31、38—43、51、55、57条等。
• 国家网信办等配套制度：个人信息跨境传输标准合同、出境安全评估相关规定（用于跨境场景）。

使用建议： 在组织的数据治理政策中，将“敏感个人信息”作为强管控对象进行专章阐述，明确定义、分级标准、处理准入门槛、技术与管理控制、审计与报告机制，并与安全、法务、合规职能形成闭环治理。